--- title: "Group Sequencer" description: "group and sequence TenXObjects" source: "https://github.com/log-10x/modules/tree/main/pipelines/run/units/transform/group/unit.yaml" icon: "material/select-group" --- Groups [TenXObjects](https://doc.log10x.com/api/js/#TenXObject "Provide structured, reflective access to log/trace events read from input(s).") for filtering, aggregation, and output as single units. Log events read from an input stream can serve as a part of a larger logical group. A typical example of events spanning multiple sub-events are [stack traces](https://stackoverflow.com/questions/3988788/what-is-a-stack-trace-and-how-can-i-use-it-to-debug-my-application-errors){target="\_blank"} where each line within the stack trace may be logged as a separate line. Grouping enables: - **Identify** groups consuming the most storage and analytics resources using [aggregators](https://doc.log10x.com/run/aggregate/ "Aggregate and summarize TenXObjects to publish as metrics"). This is especially valuable when storing stack traces than span 100s of lines and consume a significant amount of resources. - **Filter** unnecessary groups such as 'noisy' stack traces via [group filters](#groupfilters "JavaScript expressions TenXObject instance/group must evaluate as truthy against") and output [receivers](https://doc.log10x.com/run/output/receive "Filter and sample TenXObjects based on per-node budget sampling or a declarative field-set mute file"). - **Optimize** storage of multi-line events by [losslessly compacting](https://doc.log10x.com/run/transform/#compact) them as a composite instances to reduce storage footprint by \> **75%** when compared to storing individual events. ## :material-group: Group Heads tenXObjects which evaluate as truthy against [groupExpressions](https://doc.log10x.com/run/transform/group/#groupexpressions "JavaScript expressions TenXObject instance/group must evaluate as truthy against to be set as group head") are marked as starting of a new group (i.e. a _group head_). All subsequent TenXObjects read from the same input will join the current group until either: - Another group is started marked by a subsequent instance which evaluates as truthy against `groupExpressions`. - The number of TenXObjects in the current group exceeds [groupMaxSize](#groupmaxsize "max number of objects to place in a group"). - The [groupFlushTimeout](#groupflushtimeout "interval to flush a pending TenXObjects group") elapses. At that point the group is sealed as new composite TenXObject and flushed forward for aggregation and output. Each composite TenXObject returns the number of instances grouped within it via the [groupSize](https://doc.log10x.com/api/js/#TenXObject+groupSize "If the current instance is a logical group formed via groupExpressions returns the number of tenxObject within the group.") member. **` Example #1 - timestamps + negators (default)`** The TenXObject [constructor](https://doc.log10x.com/run/transform/script/object/ "Initialize TenXObject instances at runtime using JavaScript.") below marks an instance as the head of a group if: - its [text](https://doc.log10x.com/api/js/#text-string) field [starts with](https://doc.log10x.com/api/js/#TenXString.startsWith) an [indicator](#groupindicators) value. - its [timestamped](https://doc.log10x.com/api/js/#timestamped-boolean) field is true. ```js export class GroupTemplate extends TenXTemplate { static get isGroup() { // https://doc.log10x.com/run/initialize/group/#groupindicators if (this.startsWith(TenXEnv.get("groupIndicators"))) { return true; } return this.timestamped; } // This constructor is invoked by the engine once for each unique TenXTemplate discovered // at runtime based on log event structures constructor() { GroupTemplate.isGroup = this.isGroup(); } } ``` **` Example #2 - Group ISO_8601 events`** The TenXObject below marks an instance as the head of a group if it has an [ISO\_8601](https://stackoverflow.com/questions/3914404/how-to-get-current-moment-in-iso-8601-format-with-date-hour-and-minute){target="\_blank"} timestamp: ```js export class IsoTemplate extends TenXTemplate { constructor() { IsoTemplate.isGroup = this.timestampFormat() == "yyyy-MM-dd'T'HH:mm'Z'"; } } ``` **` Example #3 - Group Linux Call Traces`** The constructor below groups Linux Call Traces (see [example](https://syzkaller.appspot.com/text?tag=CrashLog&x=17f5743fe00000){target="\_blank"}) by folding lines that are part of a trace (e.g., an interrupt marker, or end with a memory address) into a logical group. ```js class NixTemplate extends TenXTemplate { constructor() { // is event is call trace interrupt marker if (this.contains("")) { NixTemplate.isGroup = false; } else // does event end in a '/'' + hex memory address (e.g., 0x16d0) ? // use the token() function to access the last + penultimate instance values if (this.token(-2) == "/") && (startsWith(this.token(-1), "0x")) { NixTemplate.isGroup = false; } } } ``` ## :material-filter-multiple-outline: Group Filters The [groupFilters](#groupfilters "JavaScript expressions TenXObject instance/group must evaluate as truthy against") option provides a mechanism for filtering TenXObject groups based on conditions that relate to the entire group (e.g., filter entire stack traces vs. individual lines). The [YAML](https://doc.log10x.com/config/yaml/) config below places a limit of a maximum of 'SocketException' 1000 groups per minute using a [cyclical counter](https://doc.log10x.com/api/js/#TenXCounter.inc): ```yaml group: filters: 'this.includes("SocketException") && (this.groupSize > 1) ? (TenXCounter.incAndGet("socketException", 1, "1m") > 1000) : true' ``` A filter may also be set to a function loaded from a [JavaScript config](https://doc.log10x.com/config/javascript/) file. For example: ```yaml group: filters: socketExceptionFilter() ``` Where the JavaScript file would contain: ```js // @loader: tenx public class MyFilter extends TenXObject { function socketExceptionFilter() { return this.includes("SocketException") && (this.groupSize > 1) ? (TenXCounter.incAndGet("socketException", 1, "1m") > 1000) : true } } ``` ## :material-regex: Event Source Input streams that read data from multiple locations (e.g., log files, pods, hosts) can utilize source [patterns](https://doc.log10x.com/run/input/stream/#inputsourcepattern "a regex pattern used to extract the 'source' value") and [fields](https://doc.log10x.com/run/input/stream/#inputsourcefields "JSON fields from which to extract the 'source' value") to assign each event a logical origin (e.g., log file, host address). This value ensure each TenXObject is grouped alongside instances from the same source and is accessible via the [source](https://doc.log10x.com/api/js/#TenXObject+source "Returns the source value assigned to this instance by its input source pattern") member. ## :material-wrench-outline: Config Files To configure the Group sequencer unit, [:material-cog: Edit](https://doc.log10x.com/config/app/#module-config "Learn how to edit app and module configurations") these files. Below is the default configuration from: [group/config.yaml](https://github.dev/log-10x/config/blob/main/pipelines/run/transform/group/config.yaml "group/config.yaml"){target="\_blank"}.
Edit Online
Edit config.yaml Locally
```yaml # 🔟❎ 'run' event grouping configuration # Group sequences of TenXObjects to filter, aggregate and output as a single logical unit. # To learn more see https://doc.log10x.com/run/transform/group/ # Set the 10x pipeline to 'run' tenx: run # =============================== Group Options =============================== group: # 'filters' specify JavaScript expressions an TenXObject instance/group must # evaluate as truthy against to be written to output filters: [] # 'maxSize' defines the maximum number of TenXObjects to group # before the group is sealed and forwarded into the pipeline. # Subsequent TenXObjects can form a new group. maxSize: 20000 # 'flushTimeout' defines the max interval (e.g., 10s) to wait for # new events to be read from an input stream before it flushes any # pending TenXObjects group into the pipeline. # This mechanism is designed to avoid latencies in dispatching pending event # groups to output destinations. flushTimeout: $=parseDuration("5s") # 'async' specifies whether to sequence and group TenXObjects in a dedicated thread async: true ``` ## :material-menu: Options Specify the options below to [configure](/config "configure") the Group sequencer: |Name|Description| |---|---| |[groupMaxSize](#groupmaxsize "max number of objects to place in a group")|Max number of objects to place in a group| |[groupMaxSize](#groupmaxsize "max number of objects to place in a group")|Max number of objects to place in a group| |[groupFlushTimeout](#groupflushtimeout "interval to flush a pending TenXObjects group")|Interval to flush a pending TenXObjects group| |[groupExpressions](#groupexpressions "JavaScript expressions TenXObject instance/group must evaluate as truthy against to be set as group head")|JavaScript expressions TenXObject instance/group must evaluate as truthy against to be set as group head| |[groupFilters](#groupfilters "JavaScript expressions TenXObject instance/group must evaluate as truthy against")|JavaScript expressions TenXObject instance/group must evaluate as truthy against| |[groupAsync](#groupasync "group TenXObjects in a dedicated thread")|Group TenXObjects in a dedicated thread| ### :material-menu-right-outline:**`groupMaxSize`** Max number of objects to place in a group. |Type|Default| |---|---| |Number|20000| Sets the maximum number of objects to place under one group before starting a new group. Set 0 to unlimited. ### :material-menu-right-outline:**`groupMaxSize`** Max number of objects to place in a group. |Type|Default| |---|---| |Number|20000| Sets the maximum number of objects to place under one group before starting a new group. Set 0 to unlimited. ### :material-menu-right-outline:**`groupFlushTimeout`** Interval to flush a pending TenXObjects group. |Type|Default| |---|---| |String|5sec| Sets the max interval after which to flush pending TenXObjects into the pipeline. This option provides a timeout period to flush the current group head and composed TenXObjects into the pipeline for aggregation, filtering, and encoding to output. ### :material-menu-right-outline:**`groupExpressions`** JavaScript expressions TenXObject instance/group must evaluate as truthy against to be set as group head. |Type|Default| |---|---| |String|""| Specifies a list of JavaScript expressions that must all evaluate as truthy for the target TenXObject to be classified as [group head](https://doc.log10x.com/run/transform/group/#group-heads). For example, the [group initializer](https://doc.log10x.com/run/initialize/group/ "Combine multi-line events into TenXObject group instances") module calculates a [TenxTemplate variable](https://doc.log10x.com/api/js/#TenXTemplate.set) to determine whether instances of a target `TenXTemplate` represent group heads based on a number of heuristic indicators and sets the calculated field name (e.g., `GroupTemplate.isGroup`) holding the resulting value into this array. Using TenXTemplate variables allows for computing this state once for all instances of a target event. ### :material-menu-right-outline:**`groupFilters`** JavaScript expressions TenXObject instance/group must evaluate as truthy against. |Type|Default| |---|---| |List|\[\]| Specifies a list of JavaScript expressions that must all evaluate as truthy for the target object/group to be part of a sequence aggregated/written to output. To learn more see [group filters](https://doc.log10x.com/run/transform/group/#group-filters). ### :material-menu-right-outline:**`groupAsync`** Group TenXObjects in a dedicated thread. |Type|Default| |---|---| |Boolean|true| Specifies whether to perform TenXObject grouping logic in a dedicated thread.
:material-github: This unit is defined in [group/unit.yaml](https://github.com/log-10x/modules/tree/main/pipelines/run/units/transform/group/unit.yaml "group/unit.yaml"){target="\_blank"}.