--- title: "Input Extractor" description: "filter, redact, and capture events from an input stream to transform\ \ into TenXObjects" source: "https://github.com/log-10x/modules/tree/main/pipelines/run/units/input/extract/unit.yaml" icon: "material/select-all" --- Extractors filter, redact and select text to transform into [TenXObjects](https://doc.log10x.com/api/js/#TenXObject "Provide structured, reflective access to log/trace events read from input(s).") from a stream of input events. Configured extractors scan an input event for specified JSON fields or regex [capture groups](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Regular_expressions/Named_capturing_group){target="\_blank"} values on which to perform specified actions. An extractor can be applied to an input via its [targetInput](#extractortargetinput "regex pattern identifying all inputs to which this extractor should be applied") member or by adding the extractor to a target input's [extractors](https://doc.log10x.com/run/input/stream/#inputextractors "names of extractors defining JSON fields/regex capture groups to select") list. For example, the [k8s enrichment](https://doc.log10x.com/run/initialize/k8s/ "Enrich TenXObjects with k8s pod and container information") module uses JSON extractors to add pod and container context to TenXObjects for filtering and aggregation. ## Actions Extractors can perform the following actions on input events: ### :material-select-all: Capture Capture actions control which segments of a log/trace event's text to transform into TenXObjects. JSON extractors select field values to transform into TenXObjects. Regex extractors select values using defined by the [extractorPattern](#extractorPattern "for regex extractors, the pattern for capturing named match groups. For JSON, the segment of text within events to scan for objects"). For example, to capture the `message` value of the following simple event: ```json { "event": { "origin": "localhost", "message": "some event" } } ``` A JSON extractor can specify the `message` field, while a Regex extractor can specify a capture group (see [example](https://regex101.com/r/WmGjnZ/1){target="\_blank"}). Each capture action specifies which instances of the JSON field/Regex groups within an event to transform: #### **`All`** Transform all matching JSON field/capture group values in an event into TenXObjects. #### **`First`** Transform the first matching JSON field/capture group value in an event into an TenXObject. #### **`Last`** Transform the last matching JSON field/capture group value in an event into an TenXObject. #### **`Default`** Serves as a sink for selecting events that fail to match the [extractorFilter](#extractorfilter "regex pattern to match for the extractor to be applied") pattern or do not contain JSON field names/regex capture groups specified by [extractorActions](#extractoractions "actions in the form of actionType:name. Possible values:[captureAll, captureFirst, captureLast, captureArrays, captureFirstArray, captureLastArray, captureDefault, captureField, setOuterText, drop, redact, noTransform] "). This action captures the entire text of an event to transform into an TenXObject and set its [extractorKey](https://doc.log10x.com/api/js/#TenXObject+extractorKey "If the object was extracted from a pattern match or JSON field using an input extractor,") value to the current action's name. Only one default capture action is allowed. #### **`Field`** Capture a JSON field value and attach it to every TenXObject produced by the extractor, under the alias name. Unlike the other capture actions, `captureField` does not emit its own TenXObject. For example, the [k8s enrichment](https://doc.log10x.com/run/initialize/k8s/ "Enrich TenXObjects with k8s pod and container information") module declares `captureField:kubernetes.namespace_name:namespace_name` to attach pod namespace as a context [field](https://doc.log10x.com/run/initialize/#enrichmentFields "list of field to enrich TenXObjects with") on every extracted TenXObject. Only the first matching value per field is captured. ### :material-filter-outline: Filter Filter actions allow for filtering an entire event or redacting some of its values. These actions serve a dual purpose of providing a fast mechanism for filtering out unnecessary events to save on the CPU resources as well as redacting sensitive information (i.e., HIPAA, PII). #### **`Drop`** Deletes a matched JSON field and its value(s) from the event entirely. Available only for JSON extractors. #### **`Redact`** For JSON extractors resets matching field values: |JSON Field Type|Redacted Value| |---|---| |object|{}| |array|\[\]| |number|0| |string|""| |boolean|true| For regex extractors deletes all instances of matching capture groups from an event. ### :material-brain: Advanced Advanced actions allow more granular control over how events are captured: #### **`No Transform`** Select events without transforming them into typed [TenXObjects](https://doc.log10x.com/api/js/#TenXObject "Provide structured, reflective access to log/trace events read from input(s)."). This action enables writing raw events that do not require any structure to output. Only one `noTransform` action is allowed. #### **`Outer Text`** Sets a JSON field/capture group as the `outer text` value for TenXObjects extracted by `captureAll`, `captureFirst`, or `captureLast` actions. For example, if a JSON extractor action specifies `captureFirst:message`, a `setOuterText:event` action can set the [fullText](https://doc.log10x.com/api/js/#TenXBaseObject+fullText "A text value extracted from the input stream from") value of an TenXObject to the JSON `message` field's enclosing `event` object. In the [example event](#capture), `some event` is set as the TenXObject's [text](https://doc.log10x.com/api/js/#TenXBaseObject+text "The content of the event from which this instance was structured.") value and its [fullText](https://doc.log10x.com/api/js/#TenXBaseObject+fullText "A text value extracted from the input stream from") value is the entire enclosing JSON object. The [encode](https://doc.log10x.com/api/js/#TenXObject+encode "Encodes the current object's log event into a compact, template‐based string.") function returns a compact representation of an TenXObject's text value enclosed within the outer text region. If no `setOuterText` selector is specified, an TenXObject's [fullText](https://doc.log10x.com/api/js/#TenXBaseObject+fullText "A text value extracted from the input stream from") and [text](https://doc.log10x.com/api/js/#TenXBaseObject+text "The content of the event from which this instance was structured.") fields return the same value. If `outerText` actions are defined but not matched, the extractor applies the `default` capture action (if defined), otherwise, it drops the event. ## :material-wrench-outline: Config Files To configure the Input extractor unit, [:material-cog: Edit](https://doc.log10x.com/config/app/#module-config "Learn how to edit app and module configurations") these files. === "Pattern" Below is the default configuration from: [extract/pattern.yaml](https://github.dev/log-10x/config/blob/main/pipelines/run/input/extract/pattern.yaml "extract/pattern.yaml"){target="\_blank"}.
Edit Online
Edit pattern.yaml Locally
```yaml # 🔟❎ 'run' Regex extractor configuration # Configure input regex pattern extractors. # To learn more see: https://doc.log10x.com/run/input/extract # Set the 10x pipeline to 'run' tenx: run # ============================= Extractor Options ============================= # multiple extractors can be defined below extractor: # 'name' provides a unique name for this extractor that is referenced # by any inputs to which it is applied - name: patternExtractor # 'type' controls the method for parsing input stream text (json or pattern) type: pattern # 'targetInput' sets a regex pattern to match all inputs to which the extractor is applied. # For example, to apply this extractor to an input named 'datadog', set: 'targetInput: datadog' targetInput: myInput # 'filter' specifies a regex pattern that must match input events to scan them for capture groups. # For example, the pattern below will filter out syslog events that are not errors filter: "error: " # 'pattern' sets regex pattern whose capture groups to apply extractor actions below. # To learn more about capture groups, see: https://regexone.com/lesson/capturing_groups # If the pattern does not define capture groups, matches are captured as events to transform into TenXObjects. # The example pattern below uses capture groups to parse a syslog event's 'message' field, # and drop 'password' values. # For an explanation, see: https://chat.openai.com/share/003ead39-421c-489a-a52f-ee8846028887 pattern: "/(error|info|debug): (?.+?)(\\s+password=(?\\S+))?(\\s+username=(?\\S+))?$?" # 'actions' specifies the actions taken by this extractor on a matching capture group. # Actions are defined by their type followed by ':' and the regex capture group they target. # The following action types are supported (comment in/out the ones needed for your use case): actions: # Capture # 'captureAll' transforms all instances of a matching capture group (i.e., 'message') # within the current event into TenXObjects - captureAll:message # 'captureFirst' transforms only the first instance of a matching capture group (i.e., 'message') # within the current event into an TenXObject - captureFirst:message # 'captureLast' transforms only the last instance of a matching capture group (i.e., 'message') # within the current event into an TenXObject - captureLast:message # Filter # 'redact' removes all matching capture groups (i.e., 'password') found in the event - redact:password # Advanced # 'captureDefault' serves as a sink for events that fail to match the 'filter' pattern # or do not contain any of the capture group names specified by 'actions'. These events are # transformed into TenXObjects whose 'extractorKey' field returns this action's name (i.e., 'other') - captureDefault:other # 'noTransform' is the same as 'captureDefault' (see above), except captured events are not # transformed into TenXObjects and remain as 'plain text' objects. # This is useful when the events need to be routed by the 10x pipeline to a specific # output, but do not require access to structured elements (e.g., the intrinsic 'vars' and 'timestamp' fields). - noTransform:other ``` === "Json" Below is the default configuration from: [extract/json.yaml](https://github.dev/log-10x/config/blob/main/pipelines/run/input/extract/json.yaml "extract/json.yaml"){target="\_blank"}.
Edit Online
Edit json.yaml Locally
```yaml # 🔟❎ 'run' JSON extractor configuration # Configure input JSON extractors. # To learn more see: https://doc.log10x.com/run/input/extract # Set the 10x pipeline to 'run' tenx: run # ============================= Extractor Options ============================= # multiple extractors can be defined below extractor: # 'name' provides a unique name for this extractor that is referenced # by any inputs to which it is applied - name: jsonExtractor # 'type' controls the method for parsing input stream text (json or pattern) type: json # 'targetInput' sets a regex pattern to match all inputs to which the extractor is applied. # For example, to apply this extractor to an input named 'datadog', set: 'targetInput: datadog' targetInput: myInput # 'filter' specifies a regex pattern the current event must match to scan for JSON objects. # For example, the pattern below will filter out events that contain "level":"TRACE" filter: ^((?!"level":"TRACE").)*$ # 'actions' specifies the actions taken by this extractor. # Define actions using followed by ':' and the JSON field they target. # The following action types are supported (comment in/out the ones needed for your use case): actions: # Capture # 'captureAll' transforms all instances of a matching JSON field (i.e., 'message') # within the current event into TenXObjects - captureAll:message # 'captureFirst' transforms only the first instance of a matching JSON field (i.e., 'message') # within the current event into an TenXObject - captureFirst:message # 'captureLast' transforms only the last instance matching JSON field (i.e., 'message') # within the current event into an TenXObject - captureLast:message # Filter # 'redact' resets the value of any matching JSON field (i.e., 'username') found in the event # based on its JSON value type: # - number -> 0 # - string -> "" # - boolean -> true # - object -> {} # - array -> [] - redact:username # 'drop' removes both field name and value of any matching # JSON field (i.e., 'password') found in the event. This action applies to JSON extractors only. - drop:password # Advanced # 'captureDefault' serves as a sink for events that fail to match the 'filter' pattern # or do not contain any of the capture group names specified by 'actions'. These events are # transformed into TenXObjects whose 'extractorKey' field returns this action's name (i.e., 'other') - captureDefault:other # 'noTransform' is the same as 'captureDefault' (see above), except captured events are not # transformed into TenXObjects and remain as 'plain text' objects. # This is useful when the events need to be routed by the 10x pipeline to a specific # output, but do not require access to structured elements (e.g., the intrinsic 'vars' and 'timestamp' fields). - noTransform:other ``` ## :material-menu: Options Specify the options below to [configure](/config "configure") multiple Input extractor: |Name|Description|Category| |---|---|---| |[extractorName](#extractorname "logical name identifying this extractor")|Logical name identifying this extractor|General| |[extractorEnabled](#extractorenabled "a JavaScript expression that must be evaluated as 'truthy' to enable the extractor")|A JavaScript expression that must be evaluated as 'truthy' to enable the extractor|General| |[extractorTargetInput](#extractortargetinput "regex pattern identifying all inputs to which this extractor should be applied")|Regex pattern identifying all inputs to which this extractor should be applied|General| |[extractorType](#extractortype "the method used to extract values from the event. Possible values:[json, pattern]")|The method used to extract values from the event. Possible values:\[json, pattern\]|General| |[extractorGroup](#extractorgroup "name of group of extractors which will run together")|Name of group of extractors which will run together|General| |[extractorActions](#extractoractions "actions in the form of actionType:name. Possible values:[captureAll, captureFirst, captureLast, captureArrays, captureFirstArray, captureLastArray, captureDefault, captureField, setOuterText, drop, redact, noTransform] ")|Actions in the form of actionType:name. Possible values:\[captureAll, captureFirst, captureLast, captureArrays, captureFirstArray, captureLastArray, captureDefault, captureField, setOuterText, drop, redact, noTransform\] |General| |[extractorFilter](#extractorfilter "regex pattern to match for the extractor to be applied")|Regex pattern to match for the extractor to be applied|Pattern| |[extractorPattern](#extractorpattern "for regex extractors, the pattern for capturing named match groups. For JSON, the segment of text within events to scan for objects")|For regex extractors, the pattern for capturing named match groups. For JSON, the segment of text within events to scan for objects|Pattern| |[extractorForeach](#extractorforeach "name of options group for whose instances to create matching extractors")|Name of options group for whose instances to create matching extractors|Advanced| ### General #### :material-menu-right-outline:**`extractorName`** Logical name identifying this extractor. |Type|Required|Category| |---|---|---| |String|✔|General| Provides a logical name (e.g., 'message') for this JSON/regex extractor that target input(s) can reference to apply this extractor to events which the produce. #### :material-menu-right-outline:**`extractorEnabled`** A JavaScript expression that must be evaluated as 'truthy' to enable the extractor. |Type|Default|Category| |---|---|---| |Boolean|true|General| enables/disables this extractor. If set, the JavaScript expression returns a truthy value to enable the extractor. For example, to configure this value to use a startup argument/shell variable, use: ```yaml extractor: name: myExtractor enabled: TenXEnv.get("performExtraction") ... ``` #### :material-menu-right-outline:**`extractorTargetInput`** Regex pattern identifying all inputs to which this extractor should be applied. |Type|Default|Category| |---|---|---| |String|""|General| Defines a regex pattern identifying all inputs to which this extractor should be applied. For example, to apply an extractor to a [datadog input](https://doc.log10x.com/run/input/analyzer/datadogLogs "Read events from Datadog Logs"), specify: ```yaml extractor: name: message type: json targetInput: datadog actions: - captureAll:message ``` This argument enables applying this extractor to input(s) without changing their definition. vs. directly the input referencing the extractor directly via [extractorName](https://doc.log10x.com/run/input/extract/#extractorname "logical name identifying this extractor"). #### :material-menu-right-outline:**`extractorType`** The method used to extract values from the event. Possible values:\[json, pattern\]. |Type|Required|Category| |---|---|---| |String|✔|General| Sets the type of extraction method to select, drop and redact values from a target input stream. Possible values: - json: scan events for JSON objects containing fields specified in [extractorActions](https://doc.log10x.com/run/input/extract/#extractoractions "actions in the form of actionType:name. Possible values:[captureAll, captureFirst, captureLast, captureArrays, captureFirstArray, captureLastArray, captureDefault, captureField, setOuterText, drop, redact, noTransform] "). - pattern: scan events for regex named capture groups of the pattern set by [extractorPattern](https://doc.log10x.com/run/input/extract/#extractorpattern "for regex extractors, the pattern for capturing named match groups. For JSON, the segment of text within events to scan for objects"). #### :material-menu-right-outline:**`extractorGroup`** Name of group of extractors which will run together. |Type|Default|Category| |---|---|---| |String|""|General| Defines the name of extractor group this extractor will belong to. extractors of the same group will attempt to run together on events in a single processing pass to improve performance. #### :material-menu-right-outline:**`extractorActions`** Actions in the form of actionType:name. Possible values:\[captureAll, captureFirst, captureLast, captureArrays, captureFirstArray, captureLastArray, captureDefault, captureField, setOuterText, drop, redact, noTransform\]. |Type|Default|Category| |---|---|---| |List|\[\]|General| Defines a list of actions defined as: 'actionType:name:alias' (e.g., 'captureAll:message:alias') to capture and redact values to transform into TenXObjects. For JSON extractors, 'name' refers to a field to look for in events and the value to which to apply the 'actionType'. This can be in the form of 'x.y.z' which will match internal fields in the json. For example: 'metadata.id' will match the 'id' field of the 'metadata' object in {"metadata": {"id": "1234"}} For regex extractors, 'name' refers to a [regex pattern match group](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Regular_expressions/Named_capturing_group){target="\_blank"} defined by 'extractorPattern' to which to apply the 'actionType'. This setting must be specified when [extractorType](https://doc.log10x.com/run/input/extract/#extractortype "the method used to extract values from the event. Possible values:[json, pattern]") = 'json'. If not specified and extractorType is 'pattern', the 'extractorPattern' pattern scans for regex capture groups, performing on each as'captureAll' action. The 'alias' part is optional, and if provided is used to reference the object as it's name. This is useful when multiple extractors are used to extract different things which are later used in the same way. If no capture groups are defined, any matches of the pattern within the input text line are captured. ### Pattern #### :material-menu-right-outline:**`extractorFilter`** Regex pattern to match for the extractor to be applied. |Type|Default|Category| |---|---|---| |String|""|Pattern| Specifies a regex pattern an event must match to scan for JSON fields/regex pattern capture groups. This argument provides a way to rule out events from being transformed into TenXObject. To select events failing to meet this filter, define a [captureDefault](https://doc.log10x.com/run/input/extract/#default) action. #### :material-menu-right-outline:**`extractorPattern`** For regex extractors, the pattern for capturing named match groups. For JSON, the segment of text within events to scan for objects. |Type|Default|Category| |---|---|---| |String|""|Pattern| Defines a regex pattern that applies to events read from an input stream. For [extractorType](https://doc.log10x.com/run/input/extract/#extractortype "the method used to extract values from the event. Possible values:[json, pattern]") = pattern, [matching groups](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Regular_expressions/Named_capturing_group){target="\_blank"} are used as the 'name' portions of actions specified by 'extractorActions'. If extractorType = json, only scan events for JSONs within the boundaries of the pattern's matches. ### Advanced #### :material-menu-right-outline:**`extractorForeach`** Name of options group for whose instances to create matching extractors. |Type|Default|Category| |---|---|---| |String|""|Advanced| Specifies the name of an options group for whose instances to replicate this extractor. For example, the [Elastic input](https://doc.log10x.com/run/input/analyzer/elasticsearch "Read events from an ElasticSearch hosted/on-premises cluster") defines the 'elastic' options group and configures an extractor to apply to each input stream created from its configured instances.
:material-github: This unit is defined in [extract/unit.yaml](https://github.com/log-10x/modules/tree/main/pipelines/run/units/input/extract/unit.yaml "extract/unit.yaml"){target="\_blank"}.