--- title: "Process Output" description: "write TenXObject and template values to a sub-process stdin" source: "https://github.com/log-10x/modules/tree/main/pipelines/run/modules/output/event/process/module.yaml" icon: "material/launch" --- Spawns a process and writes [TenXObject](https://doc.log10x.com/api/js/#TenXObject "Provide structured, reflective access to log/trace events read from input(s).") instance and template field values to its stdin. This output enables the runtime to leverage external processes (e.g., [FluentBit](https://fluentbit.io/){target="\_blank"}) as a means of transmitting processed data to local and remote destinations. ## :material-wrench-outline: Config Files To configure the Process output module, [:material-cog: Edit](https://doc.log10x.com/config/app/#module-config "Learn how to edit app and module configurations") these files. Below is the default configuration from: [process/config.yaml](https://github.dev/log-10x/config/blob/main/pipelines/run/output/event/process/config.yaml "process/config.yaml"){target="\_blank"} (** Required Fields*).
Edit Online
Edit config.yaml Locally
```yaml # 🔟❎ 'run' Fluent Bit output configuration # Launch a Fluent Bit process into whose stdin pipe to write TenXObject instance/template field values. # To learn more see https://doc.log10x.com/run/output/event/process/ # Set the 10x pipeline to 'run' tenx: run # =============================== Dependencies ================================ include: run/modules/output/event/process # ============================ Fluent Bit Options ============================= # Multiple sub-process outputs can be defined below procOut: # ----------------------------- Object Output ------------------------------- # 'writeObjects' controls whether to write TenXObject field values to output. - writeObjects: false # (❗ REQUIRED) # -------------------------- Process Options ------------------------------ # 'command' defines the OS-resolvable path of the sub-process to spawn command: /opt/fluent-bit/bin/fluent-bit args: - -i - stdin - -o - stdout # 'destroyMode' is set to 'cached' to allow a fluent-bit sub-process to # to be reused across multiple 10x pipeline executions within a host JVM. destroyMode: cached # 'maxCacheSize' sets the size of the cache to used for caching fluent-bit instances maxCacheSize: 1 # 'lazyLaunch' determines whether the fluent-bit instance will launch immediately # during the pipeline initialization, or only on-demand. lazyLaunch: true # 'startupWaitMs' and 'startupWaitPattern' controls the wait for the fluent-bit instance # to be ready for receiving events, before starting to emit to it. startupWaitMs: $=parseDuration("5s") startupWaitPattern: \[\s*sp\s*]\s+stream\s+processor\s+started\b # ------------------------------ Encoding --------------------------------- # 'filter' specifies an expression that any TenXObject passed to this output must # evaluate as truthy against to write its instance/template values to stdout. # For example, 'filter: this.timestamp' will only emit timestamped objects. # To learn more see: https://doc.log10x.com/api/js/#TenXBaseObject+timestamped filter: null # 'fields' specifies a list of 10x intrinsic/calculated/extracted fields # to extract from each target object passed to the output # and written to stdout in the format specified by 'encodeType' # If no fields are specified, encode/decode the target TenXObject. # This example corresponds to the stdin format defined by fluent-bit in: # https://docs.fluentbit.io/manual/pipeline/inputs/standard-input # The 'text' argument below could be replaced by any list of intrinsic/calculated/extracted fields. # The "" delimiter instructs 'joinFields' to produce JSON output. fields: # - open="[" - timestamp=dateFormat("+%s.%N") - separator="," - fields=joinFields("", "text") - close="]" # 'encodeType' specifies how 'fields' are encoded to stdout. Possible values: # - 'json': formats names and values for the current TenXObject as: {"field":"str","field2":1, "field3": true} # - 'delimited': formats values for the current TenXObject as: "str",1,true # The value separator (e.g., ',') is set by: 'encodeDelimiter' encodeType: delimited # 'encodeDelimiter' sets the char delimiter used to separate TenXObject # field values when 'encodeType' is set to: 'delimited' encodeDelimiter: ' ' # ----------------------------- Template Output ----------------------------- # 'writeTemplates' controls whether to write new templates for TenXObjects written to # this output. To learn more https://doc.log10x.com/run/template - writeTemplates: false # (❗ REQUIRED) # -------------------------- Process Options ------------------------------ # The output below writes the template values of TenXObjects # passed to this location. To learn more about TenXTemplates, see: # https://doc.log10x.com/run/template # 'command' defines the OS-resolvable path of the sub-process to spawn command: /opt/fluent-bit/bin/fluent-bit args: - -i - stdin - -o - stdout # 'destroyMode' is set to 'cached' to allow a fluent-bit sub-process to # to be reused across multiple 10x pipeline executions within a host JVM. destroyMode: cached # 'maxCacheSize' sets the size of the cache to used for caching fluent-bit instances maxCacheSize: 1 # 'lazyLaunch' determines whether the fluent-bit instance will launch immediately # during the pipeline initialization, or only on-demand. lazyLaunch: true # 'startupWaitMs' and 'startupWaitPattern' controls the wait for the fluent-bit instance # to be ready for receiving events, before starting to emit to it. startupWaitMs: $=parseDuration("5s") startupWaitPattern: \[\s*sp\s*]\s+stream\s+processor\s+started\b ``` ## :material-menu: Options Specify the options below to [configure](/config "configure") multiple Process output: |Name|Description|Category| |---|---|---| |[procOutCommand](#procoutcommand "command to launch")|Command to launch|Process| |[procOutArgs](#procoutargs "arguments to pass to the spawned sub-process")|Arguments to pass to the spawned sub-process|Process| |[procOutLazyLaunch](#procoutlazylaunch "sets whether to launch the process on-demand")|Sets whether to launch the process on-demand|Process| |[procOutStartupWaitMs](#procoutstartupwaitms "wait time for process to startup")|Wait time for process to startup|Process| |[procOutStartupWaitPattern](#procoutstartupwaitpattern "wait for an output from the process at startup")|Wait for an output from the process at startup|Process| |[procOutDestroyMode](#procoutdestroymode "control how to terminate the sub-process on output close")|Control how to terminate the sub-process on output close|Lifecycle| |[procOutDestroyWait](#procoutdestroywait "duration to wait for spawned sub-process to terminate")|Duration to wait for spawned sub-process to terminate|Lifecycle| |[procOutMaxCacheSize](#procoutmaxcachesize "max number of sub-processes to keep alive across subsequent 10x pipeline executions")|Max number of sub-processes to keep alive across subsequent 10x pipeline executions|Lifecycle| |[procOutAllowMultipleStreams](#procoutallowmultiplestreams "allow multiple concurrent outputs to share a subprocess")|Allow multiple concurrent outputs to share a subprocess|Lifecycle| |[procOutFields](#procoutfields "list of TenXObject field names to encode")|List of TenXObject field names to encode|Encoding| |[procOutFilter](#procoutfilter "a JavaScript expression an TenXObject must evaluate as 'truthy' against to be written to this output")|A JavaScript expression an TenXObject must evaluate as 'truthy' against to be written to this output|Encoding| |[procOutWriteObjects](#procoutwriteobjects "if true, TenXObject field values specified by [procOutFields](https://doc.log10x.com/run/output/event/process/#procoutfields) are encoded to this output")|If true, TenXObject field values specified by \[procOutFields\](https://doc.log10x.com/run/output/event/process/#procoutfields) are encoded to this output|Encoding| |[procOutWriteTemplates](#procoutwritetemplates "if true, write new TenXTemplates to this output")|If true, write new TenXTemplates to this output|Encoding| |[procOutEncodeType](#procoutencodetype "the output format in which to encode TenXObject values. Possible values:[json, delimited]")|The output format in which to encode TenXObject values. Possible values:\[json, delimited\]|Encoding| |[procOutEncodedLinePrefix](#procoutencodedlineprefix "the line prefix prepended to the encoded event if 'procOutEncodeType' is 'delimited'")|The line prefix prepended to the encoded event if 'procOutEncodeType' is 'delimited'|Encoding| |[procOutEncodeDelimiter](#procoutencodedelimiter "the delimiter placed between key and values pairs if 'procOutEncodeType' is 'delimited'")|The delimiter placed between key and values pairs if 'procOutEncodeType' is 'delimited'|Encoding| ### Process #### :material-menu-right-outline:**`procOutCommand`** Command to launch. |Type|Required|Category| |---|---|---| |String|✔|Process| Specifies the command to execute to launch the sub-process. This command can specify a shipper (e.g., Fluent Bit, Filebeat) or OS util (e.g., nc) process that will forward the output object values to a remote destination. #### :material-menu-right-outline:**`procOutArgs`** Arguments to pass to the spawned sub-process. |Type|Default|Category| |---|---|---| |List|\[\]|Process| List of values to pass as startup arguments to the spawned sub-process. #### :material-menu-right-outline:**`procOutLazyLaunch`** Sets whether to launch the process on-demand. |Type|Default|Category| |---|---|---| |Boolean|false|Process| Controls whether to spawn the target sub-process upon the creation of the output or on-demand. If true, launch the sub-process when the first TenXObject instance is written to its stdin. #### :material-menu-right-outline:**`procOutStartupWaitMs`** Wait time for process to startup. |Type|Default|Category| |---|---|---| |Number|0|Process| Sets a startup delay wait time in milliseconds before the process is considered available for events. if provided, will add a wait as part of starting the process before emitting events into it. max value for this argument is 10,000 (ie 10 seconds). if [procOutStartupWaitPattern](https://doc.log10x.com/run/output/event/process/#procoutstartupwaitpattern "wait for an output from the process at startup") is also provided, the process will be considered a failure if the wait time has passed before a line matching 'procOutStartupWaitPattern' was emitted. #### :material-menu-right-outline:**`procOutStartupWaitPattern`** Wait for an output from the process at startup. |Type|Default|Category| |---|---|---| |String|""|Process| Sets a regex pattern to wait for the process to emit for it to be considered available for events. if provided, will wait until the process has emitted a line to it's output that matches the regex before emitting events into it. if [procOutStartupWaitMs](https://doc.log10x.com/run/output/event/process/#procOutStartupWaitMs "wait time for process to startup") is also provided, will only wait for the specified wait time, after which if no line was emitted to match the provided regex the process will be considered a failure. ### Lifecycle #### :material-menu-right-outline:**`procOutDestroyMode`** Control how to terminate the sub-process on output close. |Type|Default|Category| |---|---|---| |String|""|Lifecycle| Controls how to treat spawned sub-process when this output closes. Possible values: |Value|Description| |---|---| |**graceful** (default)|The sub-process is destroyed gracefully with a timeout period of [procOutDestroyWait](https://doc.log10x.com/run/output/event/process/#procoutdestroywait "duration to wait for spawned sub-process to terminate").| |**forcible**|Terminates the sub-process immediately.| |**cached**|The sub-process remains alive for future pipelines that launch the same command and argument combination within the host JVM. A maximum of [procOutMaxCacheSize](https://doc.log10x.com/run/output/event/process/#procoutmaxcachesize "max number of sub-processes to keep alive across subsequent 10x pipeline executions") can be kept alive. Useful for launching shippers (e.g., Filebeat, Fluent Bit) across multiple 10x pipeline runs in a long-living server environment.| #### :material-menu-right-outline:**`procOutDestroyWait`** Duration to wait for spawned sub-process to terminate. |Type|Default|Category| |---|---|---| |String|""|Lifecycle| Controls the time duration that the output stream will wait for the sub-process to terminate gracefully (e.g., '5s'). If the process timeouts, the output will forcibly terminate it. If set to 0, the output stream will wait indefinitely for the sub-process to terminate. This value only applies when [procOutDestroyMode](https://doc.log10x.com/run/output/event/process/#procoutdestroymode "control how to terminate the sub-process on output close") is set to `graceful`. #### :material-menu-right-outline:**`procOutMaxCacheSize`** Max number of sub-processes to keep alive across subsequent 10x pipeline executions. |Type|Default|Category| |---|---|---| |Number|0|Lifecycle| Sets the global maximum number of processes that can retained globally across 10x pipeline executions within the host JVM. This value is useful when launching shippers (e.g., Filebeat, Fluent Bit) for use across multiple 10x pipeline executions within a server environment. This value only applies when [procOutDestroyMode](https://doc.log10x.com/run/output/event/process/#procoutdestroymode "control how to terminate the sub-process on output close") is set to `cached`. If [procOutMaxCacheSize](https://doc.log10x.com/run/output/event/process/#procoutmaxcachesize "max number of sub-processes to keep alive across subsequent 10x pipeline executions") is set to 0, no caching is performed, and sub-processes are destroyed gracefully. #### :material-menu-right-outline:**`procOutAllowMultipleStreams`** Allow multiple concurrent outputs to share a subprocess. |Type|Default|Category| |---|---|---| |Boolean|false|Lifecycle| If this value is true, multiple output streams sharing an identical command and arguments will emit their values to a single subprocess. Write access to the subprocess' stdin device will be synchronized between concurrent output streams. For example, if the target subprocess is a log forwarder (e.g., Fluentd/Bit) it can be configured to route TenXObjects from multiple output streams to their target destinations without requiring a separate instance of the forwarder for each configured output stream. ### Encoding #### :material-menu-right-outline:**`procOutFields`** List of TenXObject field names to encode. |Type|Default|Category| |---|---|---| |List|\[encodedOuterText\](https://doc.log10x.com/api/js/#TenXBaseObject+encodedOuterText)\]|Encoding| Specifies a list of TenXObject intrinsic/calculated/extracted field names to extract and write to the output. #### :material-menu-right-outline:**`procOutFilter`** A JavaScript expression an TenXObject must evaluate as 'truthy' against to be written to this output. |Type|Default|Category| |---|---|---| |String|""|Encoding| Specifies a JavaScript expression that an TenXObject must evaluate as truthy to write its instance/template field values to this output. For example, to only emit [timestamped](https://doc.log10x.com/api/js/#TenXBaseObject+timestamped "Return true if length of timestamp is greater than 0") TenXObjects, specify: ```yaml procOutFilter: this.timestamped ``` #### :material-menu-right-outline:**`procOutWriteObjects`** If true, TenXObject field values specified by \[procOutFields\](https://doc.log10x.com/run/output/event/process/#procoutfields) are encoded to this output. |Type|Default|Category| |---|---|---| |String|true|Encoding| Determine whether to write [procOutFields](https://doc.log10x.com/run/output/event/process/#procoutfields "list of TenXObject field names to encode") values of an TenXObject passed to this output. This configuration is typically defined using a JavaScript expression. To enable/disable the output depending on whether a launch argument, environment variable, or JVM -D option evaluates as truthy, use: ```yaml procOutWriteObjects: $=TenXEnv.get("myProcCmd") ``` To learn more see [TenXEnv.get](https://doc.log10x.com/api/js/#TenXEnv.get). #### :material-menu-right-outline:**`procOutWriteTemplates`** If true, write new TenXTemplates to this output. |Type|Default|Category| |---|---|---| |String|false|Encoding| Controls whether to write template values of TenXObjects passed to this output. The output will only emit TenXTemplates that the pipeline did not load at startup via the[templateFile](https://doc.log10x.com/run/template/#templatefiles "input files or glob patterns containing JSON-encoded TenXTemplates") argument, and will only emit an TenXTemplate once based on its [templateHash](https://doc.log10x.com/api/js/#TenXBaseObject+templateHash "An alphanumeric encoded value of a 64bit hash of the current object's template field.") value. #### :material-menu-right-outline:**`procOutEncodeType`** The output format in which to encode TenXObject values. Possible values:\[json, delimited\]. |Type|Default|Category| |---|---|---| |String|delimited|Encoding| Specifies how to encode [procOutFields](https://doc.log10x.com/run/output/event/process/#procoutfields "list of TenXObject field names to encode") values to this output. Possible values: - **json**: formats names and values for the current TenXObject as: {"field":"str","field2":1, "field3": true} - **delimited**: formats values for the current TenXObject as: "str",1,true The [procOutEncodeDelimiter](https://doc.log10x.com/run/output/event/process/#procoutencodedelimiter "the delimiter placed between key and values pairs if 'procOutEncodeType' is 'delimited'") argument determines the separator to use. #### :material-menu-right-outline:**`procOutEncodedLinePrefix`** The line prefix prepended to the encoded event if 'procOutEncodeType' is 'delimited'. |Type|Default|Category| |---|---|---| |String||Encoding| Sets the char which is prepended to the emitted TenXObject field values when [procOutEncodeType](https://doc.log10x.com/run/output/event/process/#procoutencodetype "the output format in which to encode TenXObject values. Possible values:[json, delimited]") is `delimited`. #### :material-menu-right-outline:**`procOutEncodeDelimiter`** The delimiter placed between key and values pairs if 'procOutEncodeType' is 'delimited'. |Type|Default|Category| |---|---|---| |String| |Encoding| Sets the char delimiter to separate TenXObject field values when [procOutEncodeType](https://doc.log10x.com/run/output/event/process/#procoutencodetype "the output format in which to encode TenXObject values. Possible values:[json, delimited]") is `delimited`.
:material-github: This module is defined in [process/module.yaml](https://github.com/log-10x/modules/tree/main/pipelines/run/modules/output/event/process/module.yaml "process/module.yaml"){target="\_blank"}.