--- title: "Fluent Bit Event Output" description: "write TenXObject and template values to a Fluent Bit subprocess to ship\ \ to remote log analyzers" source: "https://github.com/log-10x/modules/tree/main/pipelines/run/modules/output/event/fluentbit/module.yaml" icon: "simple/fluentbit" --- Defines a Fluent Bit process that spawns and writes [TenXObject](https://doc.log10x.com/api/js/#TenXObject "Provide structured, reflective access to log/trace events read from input(s).") instances and template field values. This output integrates a lightweight, customizable [Fluent Bit](https://fluentbit.io/){target="\_blank"} pipeline for shipping processed events to destinations such as [Datadog](https://docs.fluentbit.io/manual/pipeline/outputs/datadog){target="\_blank"}, [Splunk](https://docs.fluentbit.io/manual/pipeline/outputs/splunk){target="\_blank"}, [Elasticsearch](https://docs.fluentbit.io/manual/pipeline/outputs/elasticsearch){target="\_blank"}, and [CloudWatch](https://docs.fluentbit.io/manual/pipeline/outputs/cloudwatch){target="\_blank"}. **Use cases:** - **[Lambda launcher](https://doc.log10x.com/engine/launcher/lambda/)**: Ships optimized CloudWatch Logs events to analytics platforms. The Python handler pipes expanded events to the 10x Engine via stdin; Fluent Bit handles output shipping to the configured destination. - **[Retriever](https://doc.log10x.com/apps/retriever/#query)**: Ships query results from cloud storage (e.g., S3) to log analyzers and time-series databases. ## :material-wrench-outline: Config Files To configure the Fluent Bit event output module, [:material-cog: Edit](https://doc.log10x.com/config/app/#module-config "Learn how to edit app and module configurations") these files. === "Cloudwatch" Below is the default configuration from: [fluentbit/cloudwatch.yaml](https://github.dev/log-10x/config/blob/main/pipelines/run/output/event/fluentbit/cloudwatch.yaml "fluentbit/cloudwatch.yaml"){target="\_blank"}.
Edit Online
Edit cloudwatch.yaml Locally
```yaml # 🔟❎ 'run' Fluent Bit output configuration for events stream output to AWS Cloudwatch # Launch a Fluent Bit process into whose stdin pipe to write TenXObjects field values and TenXTemplates # To learn more see https://doc.log10x.com/run/output/event/process/ # Set the 10x pipeline to 'run' tenx: run # =============================== Dependencies ================================ include: - run/modules/output/event/fluentbit # ============================ Fluent Bit Options ============================= # Multiple Fluent Bit outputs can be defined below fluentbitOut: # ------------------------------ General ---------------------------------- # 'writeObjects' controls whether to write TenXObject field values to output. - writeObjects: true # 'writeTemplates' controls whether to write new template for TenXObjects written to # this output. To learn more about TenXTemplates, see: https://doc.log10x.com/run/template writeTemplates: false # ------------------------------ Process ---------------------------------- # 'command' specifies the command to execute to launch the Fluent Bit sub-process. command: /opt/fluent-bit/bin/fluent-bit # 'inputArgs' specifies list of values to pass as startup arguments to the spawned Fluent Bit sub-process inputArgs: # Sets the Fluent Bit input to read from stdin and route TenXTemplates # See https://github.com/log-10x/config/blob/main/pipelines/run/output/event/fluentbit/fluentbit-10x-input.conf - -c - $=path("fluentbit-10x-input.conf") # Setting the Fluent Bit output to aws cloudwatch. # for full config options of the Fluent Bit cloudwatch plugin, see - # https://docs.fluentbit.io/manual/pipeline/outputs/cloudwatch # # NOTE - authenticating with AWS is handled by Fluent Bit using credentials providers # as defined in - https://docs.fluentbit.io/manual/administration/aws-credentials # outputArgs: ############### Events output ############### # Set event output to Cloudwatch - "-o" - "cloudwatch_logs" - "-m" - "input" # Setting destination within AWS Cloudwatch - "-p" - "log_group_name=my-tenx-logs-stream" - "-p" - "log_stream_prefix=events-" - "-p" - "auto_create_group=true" - "-p" - "$=\"region=\" + TenXEnv.get(\"AWS_REGION\", \"us-east-1\")" # Flush every second - "-f" - "1" ############### Templates output ############### # Set templates output to Cloudwatch - "-o" - "cloudwatch_logs" - "-m" - "templates" # Setting destination within AWS Cloudwatch - "-p" - "log_group_name=my-tenx-logs-stream" - "-p" - "log_stream_prefix=templates-" - "-p" - "auto_create_group=true" - "-p" - "$=\"region=\" + TenXEnv.get(\"AWS_REGION\", \"us-east-1\")" # Flush every second - "-f" - "1" # 'destroyMode' is set to 'cached' to allow a fluent-bit sub-process to # to be reused across multiple 10x pipeline executions within a host JVM. destroyMode: cached # 'maxCacheSize' sets the size of the cache to used for caching fluent-bit instances maxCacheSize: 1 # 'lazyLaunch' determines whether the fluent-bit instance will launch immediately # during the pipeline initialization, or only on-demand. lazyLaunch: true # ------------------------------ Encoding --------------------------------- # 'filter' specifies an expression that any TenXObject passed to this output must # evaluate as truthy against to write its instance/template values to stdout. # For example, 'filter: this.timestamp' will only emit timestamped objects. # To learn more see: https://doc.log10x.com/api/js/#TenXBaseObject+timestamped filter: null # This corresponds to the stdin format defined by fluent-bit in: # https://docs.fluentbit.io/manual/pipeline/inputs/standard-input # The 'text' argument below could be replaced by any list of intrinsic/calculated/extracted fields. # The "" delimiter instructs 'joinFields' to produce JSON output. fields: # - open="[" - timestamp=dateFormat("+%s.%N") - separator="," - fields=joinFields("", "text") - close="]" ``` === "Datadog" Below is the default configuration from: [fluentbit/datadog.yaml](https://github.dev/log-10x/config/blob/main/pipelines/run/output/event/fluentbit/datadog.yaml "fluentbit/datadog.yaml"){target="\_blank"} (** Required Fields*).
Edit Online
Edit datadog.yaml Locally
```yaml # 🔟❎ 'run' Fluent Bit output configuration for events stream output to Datadog # Launch a Fluent Bit process into whose stdin pipe to write TenXObjects field values and TenXTemplates # To learn more see http://doc.log10x.com/run/output/event/process/ # Set the 10x pipeline to 'run' tenx: run # =============================== Dependencies ================================ include: - run/modules/output/event/fluentbit # ============================ Fluent Bit Options ============================= # Multiple Fluent Bit outputs can be defined below fluentbitOut: # ------------------------------ General ---------------------------------- # 'writeObjects' controls whether to write TenXObject field values to output. - writeObjects: true # 'writeTemplates' controls whether to write new template for TenXObjects written to # this output. To learn more about TenXTemplates, see: http://doc.log10x.com/run/template writeTemplates: false # ------------------------------ Process ---------------------------------- # 'command' specifies the command to execute to launch the Fluent Bit sub-process. command: /opt/fluent-bit/bin/fluent-bit # 'inputArgs' specifies list of values to pass as startup arguments to the spawned Fluent Bit sub-process inputArgs: # Sets the Fluent Bit input to read from stdin and route TenXTemplates # See https://github.com/log-10x/config/blob/main/pipelines/run/output/event/fluentbit/fluentbit-10x-input.conf - -c - $=path("fluentbit-10x-input.conf") # Setting the Fluent Bit output to Datadog # for full config options of the Fluent Bit datadog plugin, see - # https://docs.fluentbit.io/manual/pipeline/outputs/datadog # # Authentication requires the DD_API_KEY environment variable. # Set DD_SITE to your Datadog region (default: datadoghq.com). # outputArgs: ############### Events output ############### # Set event output to Datadog - "-o" - "datadog" - "-m" - "input" # Setting API key - "-p" - "$=\"apikey=\" + TenXEnv.get(\"DD_API_KEY\")" # (❗ REQUIRED) # Setting intake host (derived from DD_SITE, e.g. datadoghq.com, datadoghq.eu, us5.datadoghq.com) - "-p" - "$=\"Host=http-intake.logs.\" + TenXEnv.get(\"DD_SITE\", \"datadoghq.com\")" # TLS and compression - "-p" - "TLS=on" - "-p" - "compress=gzip" # Source and tag metadata - "-p" - "dd_source=log10x" - "-p" - "$=\"dd_tags=\" + TenXEnv.get(\"DD_TAGS\", \"\")" # Use 'text' field as the message body - "-p" - "dd_message_key=text" # Flush every second - "-f" - "1" ############### Templates output ############### # Set templates output to Datadog - "-o" - "datadog" - "-m" - "templates" # Setting API key - "-p" - "$=\"apikey=\" + TenXEnv.get(\"DD_API_KEY\")" # (❗ REQUIRED) # Setting intake host - "-p" - "$=\"Host=http-intake.logs.\" + TenXEnv.get(\"DD_SITE\", \"datadoghq.com\")" # TLS and compression - "-p" - "TLS=on" - "-p" - "compress=gzip" # Source and tag metadata for templates - "-p" - "dd_source=log10x-templates" - "-p" - "$=\"dd_tags=\" + TenXEnv.get(\"DD_TAGS\", \"\")" # Flush every second - "-f" - "1" # 'destroyMode' is set to 'cached' to allow a fluent-bit sub-process to # to be reused across multiple 10x pipeline executions within a host JVM. destroyMode: cached # 'maxCacheSize' sets the size of the cache to used for caching fluent-bit instances maxCacheSize: 1 # 'lazyLaunch' determines whether the fluent-bit instance will launch immediately # during the pipeline initialization, or only on-demand. lazyLaunch: true # ------------------------------ Encoding --------------------------------- # 'filter' specifies an expression that any TenXObject passed to this output must # evaluate as truthy against to write its instance/template values to stdout. # For example, 'filter: this.timestamp' will only emit timestamped objects. # To learn more see: http://doc.log10x.com/api/js/#TenXBaseObject+timestamped filter: null # This corresponds to the stdin format defined by fluent-bit in: # https://docs.fluentbit.io/manual/pipeline/inputs/standard-input # The 'text' argument below could be replaced by any list of intrinsic/calculated/extracted fields. # The "" delimiter instructs 'joinFields' to produce JSON output. fields: # - open="[" - timestamp=dateFormat("+%s.%N") - separator="," - fields=joinFields("", "text") - close="]" ``` === "Elasticsearch" Below is the default configuration from: [fluentbit/elasticsearch.yaml](https://github.dev/log-10x/config/blob/main/pipelines/run/output/event/fluentbit/elasticsearch.yaml "fluentbit/elasticsearch.yaml"){target="\_blank"} (** Required Fields*).
Edit Online
Edit elasticsearch.yaml Locally
```yaml # 🔟❎ 'run' Fluent Bit output configuration for events stream output to Elasticsearch # Launch a Fluent Bit process into whose stdin pipe to write TenXObjects field values and TenXTemplates # To learn more see https://doc.log10x.com/run/output/event/process/ # Set the 10x pipeline to 'run' tenx: run # =============================== Dependencies ================================ include: - run/modules/output/event/fluentbit # ============================ Fluent Bit Options ============================= # Multiple Fluent Bit outputs can be defined below fluentbitOut: # ------------------------------ General ---------------------------------- # 'writeObjects' controls whether to write TenXObject field values to output. - writeObjects: true # 'writeTemplates' controls whether to write new template for TenXObjects written to # this output. To learn more about TenXTemplates, see: https://doc.log10x.com/run/template writeTemplates: false # ------------------------------ Process ---------------------------------- # 'command' specifies the command to execute to launch the Fluent Bit sub-process. command: /opt/fluent-bit/bin/fluent-bit # 'inputArgs' specifies list of values to pass as startup arguments to the spawned Fluent Bit sub-process inputArgs: # Sets the Fluent Bit input to read from stdin and route TenXTemplates # See https://github.com/log-10x/config/blob/main/pipelines/run/output/event/fluentbit/fluentbit-10x-input.conf - -c - $=path("fluentbit-10x-input.conf") # Setting the Fluent Bit output to Elasticsearch # for full config options of the Fluent Bit elasticsearch plugin, see - # https://docs.fluentbit.io/manual/pipeline/outputs/elasticsearch # outputArgs: ############### Events output ############### # Set event output to Elasticsearch - "-o" - "es" - "-m" - "input" # Setting host - "-p" - "$=\"host=\" + TenXEnv.get(\"ELASTICSEARCH_HOST\", \"elasticsearch-master\")" # (❗ REQUIRED) - "-p" - "$=\"port=\" + TenXEnv.get(\"ELASTICSEARCH_PORT\", \"9200\")" # (❗ REQUIRED) # Setting up credentials - "-p" - "$=\"HTTP_User=\" + TenXEnv.get(\"ELASTICSEARCH_USERNAME\", \"elastic\")" # (❗ REQUIRED) - "-p" - "$=\"HTTP_Passwd=\" + TenXEnv.get(\"ELASTICSEARCH_PASSWORD\")" # (❗ REQUIRED) # Setting the index destination - "-p" - "Logstash_Format=On" # Config retry - "-p" - "retry_limit=False" # Support for removal of types from elasticsearch - "-p" - "Suppress_Type_Name=On" ############### Templates output ############### # Set templates output to Elasticsearch - "-o" - "es" - "-m" - "templates" # Setting host - "-p" - "$=\"host=\" + TenXEnv.get(\"ELASTICSEARCH_HOST\", \"elasticsearch-master\")" # (❗ REQUIRED) - "-p" - "$=\"port=\" + TenXEnv.get(\"ELASTICSEARCH_PORT\", \"9200\")" # (❗ REQUIRED) # Setting up credentials - "-p" - "$=\"HTTP_User=\" + TenXEnv.get(\"ELASTICSEARCH_USERNAME\", \"elastic\")" # (❗ REQUIRED) - "-p" - "$=\"HTTP_Passwd=\" + TenXEnv.get(\"ELASTICSEARCH_PASSWORD\")" # (❗ REQUIRED) # Setting the templates index destination - "-p" - "index=l1es_dml" - "-p" - "id_key=templateHash" # Config retry - "-p" - "retry_limit=False" # Support for removal of types from elasticsearch - "-p" - "Suppress_Type_Name=On" # 'destroyMode' is set to 'cached' to allow a fluent-bit sub-process to # to be reused across multiple 10x pipeline executions within a host JVM. destroyMode: cached # 'maxCacheSize' sets the size of the cache to used for caching fluent-bit instances maxCacheSize: 1 # 'lazyLaunch' determines whether the fluent-bit instance will launch immediately # during the pipeline initialization, or only on-demand. lazyLaunch: true # ------------------------------ Encoding --------------------------------- # 'filter' specifies an expression that any TenXObject passed to this output must # evaluate as truthy against to write its instance/template values to stdout. # For example, 'filter: this.timestamp' will only emit timestamped objects. # To learn more see: https://doc.log10x.com/api/js/#TenXBaseObject+timestamped filter: null # This corresponds to the stdin format defined by fluent-bit in: # https://docs.fluentbit.io/manual/pipeline/inputs/standard-input # The 'text' argument below could be replaced by any list of intrinsic/calculated/extracted fields. # The "" delimiter instructs 'joinFields' to produce JSON output. fields: # - open="[" - timestamp=dateFormat("+%s.%N") - separator="," - fields=joinFields("", "text") - close="]" ``` === "Splunk" Below is the default configuration from: [fluentbit/splunk.yaml](https://github.dev/log-10x/config/blob/main/pipelines/run/output/event/fluentbit/splunk.yaml "fluentbit/splunk.yaml"){target="\_blank"} (** Required Fields*).
Edit Online
Edit splunk.yaml Locally
```yaml # 🔟❎ 'run' Fluent Bit output configuration for events stream output to Splunk # Launch a Fluent Bit process into whose stdin pipe to write TenXObjects field values and TenXTemplates # To learn more see https://doc.log10x.com/run/output/event/process/ # Set the 10x pipeline to 'run' tenx: run # =============================== Dependencies ================================ include: - run/modules/output/event/fluentbit # ============================ Fluent Bit Options ============================= # Multiple Fluent Bit outputs can be defined below fluentbitOut: # ------------------------------ General ---------------------------------- # 'writeObjects' controls whether to write TenXObject field values to output. - writeObjects: true # 'writeTemplates' controls whether to write new template for TenXObjects written to # this output. To learn more about TenXTemplates, see: https://doc.log10x.com/run/template writeTemplates: false # ------------------------------ Process ---------------------------------- # 'command' specifies the command to execute to launch the Fluent Bit sub-process. command: /opt/fluent-bit/bin/fluent-bit # 'inputArgs' specifies list of values to pass as startup arguments to the spawned Fluent Bit sub-process inputArgs: # Sets the Fluent Bit input to read from stdin and route TenXTemplates # See https://github.com/log-10x/config/blob/main/pipelines/run/output/event/fluentbit/fluentbit-10x-input.conf - -c - $=path("fluentbit-10x-input.conf") # Setting the Fluent Bit output to Splunk # for full config options of the Fluent Bit splunk plugin, see - # https://docs.fluentbit.io/manual/pipeline/outputs/splunk # # Fluent Bit is using Splunk's HTTP event collector (HEC) for emitting events # for instructions on setting up HEC on splunk, see - # https://dev.splunk.com/enterprise/docs/devtools/httpeventcollector # outputArgs: ############### Events output ############### # Set event output to Splunk - "-o" - "splunk" - "-m" - "input" # Setting host - "-p" - "$=\"host=\" + TenXEnv.get(\"SPLUNK_HOST\", \"127.0.0.1\")" # (❗ REQUIRED) - "-p" - "$=\"port=\" + TenXEnv.get(\"SPLUNK_PORT\", \"8088\")" # (❗ REQUIRED) # Setting up credentials - "-p" - "$=\"splunk_token=\" + TenXEnv.get(\"SPLUNK_HEC_TOKEN\")" # (❗ REQUIRED) # Setting the index destination - "-p" - "event_index=main" - "-p" - "event_source=fluent-bit-tenx-stream" - "-p" - "event_sourcetype=tenx_encoded" # Setting TLS - "-p" - "tls=on" - "-p" - "tls.verify=on" # Don't send raw - "-p" - "splunk_send_raw=off" ############### Templates output ############### # Set templates output to Elasticsearch - "-o" - "es" - "-m" - "templates" # Setting host - "-p" - "$=\"host=\" + TenXEnv.get(\"SPLUNK_HOST\", \"127.0.0.1\")" # (❗ REQUIRED) - "-p" - "$=\"port=\" + TenXEnv.get(\"SPLUNK_PORT\", \"8088\")" # (❗ REQUIRED) # Setting up credentials - "-p" - "$=\"splunk_token=\" + TenXEnv.get(\"SPLUNK_HEC_TOKEN\")" # (❗ REQUIRED) # Setting the templates index destination - "-p" - "event_index=main" - "-p" - "event_source=fluent-bit-tenx-stream" - "-p" - "event_sourcetype=tenx_dml_raw_json" # Setting TLS - "-p" - "tls=on" - "-p" - "tls.verify=on" # Don't send raw - "-p" - "splunk_send_raw=off" # 'destroyMode' is set to 'cached' to allow a fluent-bit sub-process to # to be reused across multiple 10x pipeline executions within a host JVM. destroyMode: cached # 'maxCacheSize' sets the size of the cache to used for caching fluent-bit instances maxCacheSize: 1 # 'lazyLaunch' determines whether the fluent-bit instance will launch immediately # during the pipeline initialization, or only on-demand. lazyLaunch: true # ------------------------------ Encoding --------------------------------- # 'filter' specifies an expression that any TenXObject passed to this output must # evaluate as truthy against to write its instance/template values to stdout. # For example, 'filter: this.timestamp' will only emit timestamped objects. # To learn more see: https://doc.log10x.com/api/js/#TenXBaseObject+timestamped filter: null # This corresponds to the stdin format defined by fluent-bit in: # https://docs.fluentbit.io/manual/pipeline/inputs/standard-input # The 'text' argument below could be replaced by any list of intrinsic/calculated/extracted fields. # The "" delimiter instructs 'joinFields' to produce JSON output. fields: # - open="[" - timestamp=dateFormat("+%s.%N") - separator="," - fields=joinFields("", "text") - close="]" ``` === "Stdout" Below is the default configuration from: [fluentbit/stdout.yaml](https://github.dev/log-10x/config/blob/main/pipelines/run/output/event/fluentbit/stdout.yaml "fluentbit/stdout.yaml"){target="\_blank"}.
Edit Online
Edit stdout.yaml Locally
```yaml # 🔟❎ 'run' Fluent Bit output configuration for events stream output to Standard output # Launch a Fluent Bit process into whose stdin pipe to write TenXObjects field values and TenXTemplates # To learn more see https://doc.log10x.com/run/output/event/process/ # Set the 10x pipeline to 'run' tenx: run # =============================== Dependencies ================================ include: - run/modules/output/event/fluentbit # ============================ Fluent Bit Options ============================= # Multiple Fluent Bit outputs can be defined below fluentbitOut: # ------------------------------ General ---------------------------------- # 'writeObjects' controls whether to write TenXObject field values to output. - writeObjects: true # 'writeTemplates' controls whether to write new template for TenXObjects written to # this output. To learn more about TenXTemplates, see: https://doc.log10x.com/run/template writeTemplates: false # ------------------------------ Process ---------------------------------- # 'command' specifies the command to execute to launch the Fluent Bit sub-process. command: /opt/fluent-bit/bin/fluent-bit # 'inputArgs' specifies list of values to pass as startup arguments to the spawned Fluent Bit sub-process inputArgs: # Sets the Fluent Bit input to read from stdin and route TenXTemplates # See https://github.com/log-10x/config/blob/main/pipelines/run/output/event/fluentbit/fluentbit-10x-input.conf - -c - $=path("fluentbit-10x-input.conf") # Setting the Fluent Bit output to stdout outputArgs: ############### Events output ############### # Set event output to stdout - "-o" - "stdout" - "-m" - "input" ############### Templates output ############### # Set templates output to stdout - "-o" - "stdout" - "-m" - "templates" # 'destroyMode' is set to 'cached' to allow a fluent-bit sub-process to # to be reused across multiple 10x pipeline executions within a host JVM. destroyMode: cached # 'maxCacheSize' sets the size of the cache to used for caching fluent-bit instances maxCacheSize: 1 # 'lazyLaunch' determines whether the fluent-bit instance will launch immediately # during the pipeline initialization, or only on-demand. lazyLaunch: true # ------------------------------ Encoding --------------------------------- # 'filter' specifies an expression that any TenXObject passed to this output must # evaluate as truthy against to write its instance/template values to stdout. # For example, 'filter: this.timestamp' will only emit timestamped objects. # To learn more see: https://doc.log10x.com/api/js/#TenXBaseObject+timestamped filter: null # This corresponds to the stdin format defined by fluent-bit in: # https://docs.fluentbit.io/manual/pipeline/inputs/standard-input # The 'text' argument below could be replaced by any list of intrinsic/calculated/extracted fields. # The "" delimiter instructs 'joinFields' to produce JSON output. fields: # - open="[" - timestamp=dateFormat("+%s.%N") - separator="," - fields=joinFields("", "text") - close="]" ``` ## :material-menu: Options Specify the options below to [configure](/config "configure") multiple Fluent Bit event output: |Name|Description|Category| |---|---|---| |[fluentbitOutCommand](#fluentbitoutcommand "command to launch Fluent Bit")|Command to launch Fluent Bit|Process| |[fluentbitOutOutputArgs](#fluentbitoutoutputargs "arguments to pass to the spawned Fluent Bit process setting up output")|Arguments to pass to the spawned Fluent Bit process setting up output|Process| |[fluentbitOutLazyLaunch](#fluentbitoutlazylaunch "sets whether to launch the process on-demand")|Sets whether to launch the process on-demand|Process| |[fluentbitOutInputArgs](#fluentbitoutinputargs "arguments to pass to the spawned Fluent Bit process setting up input")|Arguments to pass to the spawned Fluent Bit process setting up input|Advanced| |[fluentbitOutStartupWaitMs](#fluentbitoutstartupwaitms "wait time for process to startup")|Wait time for process to startup|Advanced| |[fluentbitOutStartupWaitPattern](#fluentbitoutstartupwaitpattern "wait for an output from the process at startup")|Wait for an output from the process at startup|Advanced| |[fluentbitOutDestroyMode](#fluentbitoutdestroymode "control how to terminate the sub-process on output close")|Control how to terminate the sub-process on output close|Lifecycle| |[fluentbitOutDestroyWait](#fluentbitoutdestroywait "duration to wait for spawned sub-process to terminate")|Duration to wait for spawned sub-process to terminate|Lifecycle| |[fluentbitOutMaxCacheSize](#fluentbitoutmaxcachesize "max number of sub-processes to keep alive across subsequent 10x pipeline executions")|Max number of sub-processes to keep alive across subsequent 10x pipeline executions|Lifecycle| |[fluentbitOutFields](#fluentbitoutfields "list of TenXObject field names to encode")|List of TenXObject field names to encode|Encoding| |[fluentbitOutFilter](#fluentbitoutfilter "a JavaScript expression an TenXObject must evaluate as 'truthy' against to be written to this output")|A JavaScript expression an TenXObject must evaluate as 'truthy' against to be written to this output|Encoding| |[fluentbitOutWriteObjects](#fluentbitoutwriteobjects "if true, TenXObject field values specified by [fluentbitOutFields](https://doc.log10x.com/run/output/event/fluentbit/#fluentbitoutfields) are encoded to this output")|If true, TenXObject field values specified by \[fluentbitOutFields\](https://doc.log10x.com/run/output/event/fluentbit/#fluentbitoutfields) are encoded to this output|Encoding| |[fluentbitOutWriteTemplates](#fluentbitoutwritetemplates "if true, write new TenXTemplates to this output")|If true, write new TenXTemplates to this output|Encoding| |[fluentbitOutEncodeType](#fluentbitoutencodetype "the output format in which to encode TenXObject values. Possible values:[json, delimited]")|The output format in which to encode TenXObject values. Possible values:\[json, delimited\]|Encoding| |[fluentbitOutEncodedLinePrefix](#fluentbitoutencodedlineprefix "the line prefix prepended to the encoded event if 'fluentbitOutEncodeType' is 'delimited'")|The line prefix prepended to the encoded event if 'fluentbitOutEncodeType' is 'delimited'|Encoding| |[fluentbitOutEncodeDelimiter](#fluentbitoutencodedelimiter "the delimiter placed between key and values pairs if 'fluentbitOutEncodeType' is 'delimited'")|The delimiter placed between key and values pairs if 'fluentbitOutEncodeType' is 'delimited'|Encoding| ### Process #### :material-menu-right-outline:**`fluentbitOutCommand`** Command to launch Fluent Bit. |Type|Required|Category| |---|---|---| |String|✔|Process| Specifies the command to execute to launch the Fluent Bit sub-process. #### :material-menu-right-outline:**`fluentbitOutOutputArgs`** Arguments to pass to the spawned Fluent Bit process setting up output. |Type|Default|Category| |---|---|---| |List|\[\]|Process| List of values to pass as startup arguments to the spawned Fluent Bit sub-process intended to set up it's output plugin(s). #### :material-menu-right-outline:**`fluentbitOutLazyLaunch`** Sets whether to launch the process on-demand. |Type|Default|Category| |---|---|---| |Boolean|false|Process| Controls whether to spawn the target sub-process upon the creation of the output or on-demand. If true, launch the sub-process when the first TenXObject instance is written to its stdin. ### Advanced #### :material-menu-right-outline:**`fluentbitOutInputArgs`** Arguments to pass to the spawned Fluent Bit process setting up input. |Type|Default|Category| |---|---|---| |List|\[-c, fluentbit-10x-input.conf\]|Advanced| List of values to pass as startup arguments to the spawned Fluent Bit sub-process intended to set up it's stdin input. #### :material-menu-right-outline:**`fluentbitOutStartupWaitMs`** Wait time for process to startup. |Type|Default|Category| |---|---|---| |Number|5000|Advanced| Sets a startup delay wait time in milliseconds before the process is considered available for events. if provided, will add a wait as part of starting the process before emitting events into it. max value for this argument is 10,000 (ie 10 seconds). if [fluentbitOutStartupWaitPattern](https://doc.log10x.com/run/output/event/fluentbit/#fluentbitOutstartupwaitpattern "wait for an output from the process at startup") is also provided, the process will be considered a failure if the wait time has passed before a line matching [fluentbitOutStartupWaitPattern](https://doc.log10x.com/run/output/event/fluentbit/#fluentbitoutstartupwaitpattern "wait for an output from the process at startup") was emitted. #### :material-menu-right-outline:**`fluentbitOutStartupWaitPattern`** Wait for an output from the process at startup. |Type|Default|Category| |---|---|---| |String|\\\[\\s\*sp\\s\*\]\\s+stream\\s+processor\\s+started\\b|Advanced| Sets a regex pattern to wait for the process to emit for it to be considered available for events. if provided, will wait until the process has emitted a line to it's output that matches the regex before emitting events into it. if [fluentbitOutStartupWaitMs](https://doc.log10x.com/run/output/event/fluentbit/#fluentbitOutStartupWaitMs "wait time for process to startup") is also provided, will only wait for the specified wait time, after which if no line was emitted to match the provided regex the process will be considered a failure. ### Lifecycle #### :material-menu-right-outline:**`fluentbitOutDestroyMode`** Control how to terminate the sub-process on output close. |Type|Default|Category| |---|---|---| |String|graceful|Lifecycle| Controls how to treat spawned sub-process when this output closes. Possible values: |Value|Description| |---|---| |**graceful** (default)|The sub-process is destroyed gracefully with a timeout period of [fluentbitOutDestroyWait](https://doc.log10x.com/run/output/event/fluentbit/#fluentbitoutdestroywait "duration to wait for spawned sub-process to terminate").| |**forcible**|Terminates the sub-process immediately.| |**cached**|The sub-process remains alive for future pipelines that launch the same command and argument combination within the host JVM. A maximum of [fluentbitOutMaxCacheSize](https://doc.log10x.com/run/output/event/fluentbit/#fluentbitoutmaxcachesize "max number of sub-processes to keep alive across subsequent 10x pipeline executions") can be kept alive. Useful for launching shippers (e.g., Filebeat, Fluent Bit) across multiple 10x pipeline runs in a long-living server environment.| #### :material-menu-right-outline:**`fluentbitOutDestroyWait`** Duration to wait for spawned sub-process to terminate. |Type|Default|Category| |---|---|---| |String|""|Lifecycle| Controls the time duration that the output stream will wait for the sub-process to terminate gracefully (e.g., '5s'). If the process timeouts, the output will forcibly terminate it. If set to 0, the output stream will wait indefinitely for the sub-process to terminate. This value only applies when [fluentbitOutDestroyMode](https://doc.log10x.com/run/output/event/fluentbit/#fluentbitoutdestroymode "control how to terminate the sub-process on output close") is set to `graceful`. #### :material-menu-right-outline:**`fluentbitOutMaxCacheSize`** Max number of sub-processes to keep alive across subsequent 10x pipeline executions. |Type|Default|Category| |---|---|---| |Number|0|Lifecycle| Sets the global maximum number of processes that can retained globally across 10x pipeline executions within the host JVM. This value is useful when launching shippers (e.g., Filebeat, Fluent Bit) for use across multiple 10x pipeline executions within a server environment. This value only applies when [fluentbitOutDestroyMode](https://doc.log10x.com/run/output/event/fluentbit/#fluentbitoutdestroymode "control how to terminate the sub-process on output close") is set to `cached`. If [fluentbitOutMaxCacheSize](https://doc.log10x.com/run/output/event/fluentbit/#fluentbitoutmaxcachesize "max number of sub-processes to keep alive across subsequent 10x pipeline executions") is set to 0, no caching is performed, and sub-processes are destroyed gracefully. ### Encoding #### :material-menu-right-outline:**`fluentbitOutFields`** List of TenXObject field names to encode. |Type|Default|Category| |---|---|---| |List|\[\]|Encoding| Specifies a list of TenXObject intrinsic/calculated/extracted field names to extract and write to the output. #### :material-menu-right-outline:**`fluentbitOutFilter`** A JavaScript expression an TenXObject must evaluate as 'truthy' against to be written to this output. |Type|Default|Category| |---|---|---| |String|""|Encoding| Specifies a JavaScript expression that an TenXObject must evaluate as truthy to write its instance/template field values to this output. For example, to only emit [timestamped](https://doc.log10x.com/api/js/#TenXBaseObject+timestamped "Return true if length of timestamp is greater than 0") TenXObjects, specify: ```yaml fluentbitOutFilter: this.timestamped ``` #### :material-menu-right-outline:**`fluentbitOutWriteObjects`** If true, TenXObject field values specified by \[fluentbitOutFields\](https://doc.log10x.com/run/output/event/fluentbit/#fluentbitoutfields) are encoded to this output. |Type|Default|Category| |---|---|---| |String|true|Encoding| Determine whether to write [fluentbitOutFields](https://doc.log10x.com/run/output/event/fluentbit/#fluentbitoutfields "list of TenXObject field names to encode") values of an TenXObject passed to this output. This configuration is typically defined using a JavaScript expression. To enable/disable the output depending on whether a launch argument, environment variable, or JVM -D option evaluates as truthy, use: ```yaml fluentbitOutWriteObjects: $=TenXEnv.get("myProcCmd") ``` To learn more see [TenXEnv.get](https://doc.log10x.com/api/js/#TenXEnv.get). #### :material-menu-right-outline:**`fluentbitOutWriteTemplates`** If true, write new TenXTemplates to this output. |Type|Default|Category| |---|---|---| |String|false|Encoding| Controls whether to write template values of TenXObjects passed to this output. The output will only emit TenXTemplates that the pipeline did not load at startup via the[templateFile](https://doc.log10x.com/run/template/#templatefiles "input files or glob patterns containing JSON-encoded TenXTemplates") argument, and will only emit an TenXTemplate once based on its [templateHash](https://doc.log10x.com/api/js/#TenXBaseObject+templateHash "An alphanumeric encoded value of a 64bit hash of the current object's template field.") value. #### :material-menu-right-outline:**`fluentbitOutEncodeType`** The output format in which to encode TenXObject values. Possible values:\[json, delimited\]. |Type|Default|Category| |---|---|---| |String|delimited|Encoding| Specifies how to encode [fluentbitOutFields](https://doc.log10x.com/run/output/event/fluentbit/#fluentbitoutfields "list of TenXObject field names to encode") values to this output. Possible values: - **json**: formats names and values for the current TenXObject as: {"field":"str","field2":1, "field3": true} - **delimited**: formats values for the current TenXObject as: "str",1,true The [fluentbitOutEncodeDelimiter](https://doc.log10x.com/run/output/event/fluentbit/#fluentbitOutencodedelimiter "the delimiter placed between key and values pairs if 'fluentbitOutEncodeType' is 'delimited'") argument determines the separator to use. #### :material-menu-right-outline:**`fluentbitOutEncodedLinePrefix`** The line prefix prepended to the encoded event if 'fluentbitOutEncodeType' is 'delimited'. |Type|Default|Category| |---|---|---| |String||Encoding| Sets the char which is prepended to the emitted TenXObject field values when [fluentbitOutEncodeType](https://doc.log10x.com/run/output/event/fluentbit/#fluentbitoutencodetype "the output format in which to encode TenXObject values. Possible values:[json, delimited]") is `delimited`. #### :material-menu-right-outline:**`fluentbitOutEncodeDelimiter`** The delimiter placed between key and values pairs if 'fluentbitOutEncodeType' is 'delimited'. |Type|Default|Category| |---|---|---| |String| |Encoding| Sets the char delimiter to separate TenXObject field values when [fluentbitOutEncodeType](https://doc.log10x.com/run/output/event/fluentbit/#fluentbitoutencodetype "the output format in which to encode TenXObject values. Possible values:[json, delimited]") is `delimited`.
:material-github: This module is defined in [fluentbit/module.yaml](https://github.com/log-10x/modules/tree/main/pipelines/run/modules/output/event/fluentbit/module.yaml "fluentbit/module.yaml"){target="\_blank"}.