--- title: "Datadog Logs Input" description: "read events from Datadog Logs" source: "https://github.com/log-10x/modules/tree/main/pipelines/run/modules/input/analyzer/datadogLogs/module.yaml" icon: "simple/datadog" --- Configures a Datadog Logs input from which to read events to transform into typed [TenXObjects](https://doc.log10x.com/api/js/#TenXObject "Provide structured, reflective access to log/trace events read from input(s)."). Instances of this [module](https://doc.log10x.com/engine/module/) define a connection to a Datadog Logs service from which to retrieve log messages, as well as the querying logic used such as chronological direction, start values, time ranges, and page size of each API request sent. Datadog Logs inputs commonly run within scheduled jobs (e.g., k8s CronJob) to retrieve a recent sample amount of events (e.g., 200MB in the last 10min) to [transform](https://doc.log10x.com/run/transform/ "Transform log and trace events into well-defined TenXObjects") into TenXObjects as part of the [Dev app](https://doc.log10x.com/apps/dev/) app. ## :material-wrench-outline: Config Files To configure the Datadog Logs input module, [:material-cog: Edit](https://doc.log10x.com/config/app/#module-config "Learn how to edit app and module configurations") these files. Below is the default configuration from: [datadogLogs/config.yaml](https://github.dev/log-10x/config/blob/main/pipelines/run/input/analyzer/datadogLogs/config.yaml "datadogLogs/config.yaml"){target="\_blank"} (** Required Fields*).
Edit Online
Edit config.yaml Locally
```yaml # 🔟❎ 'run' Datadog Logs input configuration # Configure a Datadog Logs event input # To learn more see https://doc.log10x.com/run/input/analyzer/datadogLogs/ # Set the 10x pipeline to 'run' tenx: run # =============================== Dependencies ================================ include: run/modules/input/analyzer/datadogLogs # ============================== Datadog Inputs =============================== # Multiple Datadog inputs can be defined below datadogLogs: # 'name' sets a unique logical name across all pipeline inputs - name: DatadogLogs # -------------------------- Connection Options --------------------------- # 'apiKey' and 'appKey' must be set to authenticate with the logs endpoint # To learn more see: https://docs.datadoghq.com/account_management/api-app-keys/ apiKey: $=TenXEnv.get("DD_API_KEY") # (❗ REQUIRED) appKey: $=TenXEnv.get("DD_APP_KEY") # (❗ REQUIRED) # 'host' points to the target Datadog site # To learn more see: https://docs.datadoghq.com/getting_started/site/ host: us5.datadoghq.com # ----------------------------- Query Options ----------------------------- # 'query' sets the Datadog query used to request paginated documents. # The default query (defined in 'run/modules/input/analyzer/datadog/query.txt') retrieves # values sorted in desc order by @timestamp, starting from the value of 'searchFrom' query: null # 'searchFrom sets the initial value used to request the first page of results. searchFrom: now-1000h # 'querySize' sets the number of documents to retrieve with each paginated request querySize: 500 # 'indexes' sets the lists of data indices to search in # If not provided, will use a default of '*' indexes: null # --------------------------- Backpressure Options ----------------------- # 'queryInterval' sets the interval between queries to the remote API queryInterval: $=parseDuration("5s") # 'totalDuration' sets the max duration to try reading from the the remote input totalDuration: $=parseDuration("5min") # 'totalBytesLimit' sets the max total bytes to read from the remote input totalBytesLimit: $=parseBytes("50MB") # 'totalEventsLimit' sets the max number of events to read the remote input totalEventsLimit: 10000 # --------------------------- Ancillary Options --------------------------- # 'printProgress' controls whether to print a progress gage to the console # This option helps test the input printProgress: $=!TenXEnv.get("quiet") ``` ## :material-menu: Options Specify the options below to [configure](/config "configure") multiple Datadog Logs input: |Name|Description|Category| |---|---|---| |[datadogLogsName](#datadoglogsname "logical name for this DatadogLogs input")|Logical name for this DatadogLogs input|General| |[datadogLogsEnabled](#datadoglogsenabled "sets whether this input is enabled")|Sets whether this input is enabled|General| |[datadogLogsPrintProgress](#datadoglogsprintprogress "sets whether this input prints throughput stats to the console")|Sets whether this input prints throughput stats to the console|General| |[datadogLogsTotalDuration](#datadoglogstotalduration "sets maximum time to query the backend")|Sets maximum time to query the backend|General| |[datadogLogsExtractors](#datadoglogsextractors "list of extractors to apply on the events returning from the query")|List of extractors to apply on the events returning from the query|Advanced| |[datadogLogsHost](#datadoglogshost "DatadogLogs host address")|DatadogLogs host address|Authentication| |[datadogLogsApiKey](#datadoglogsapikey "Datadog API key token")|Datadog API key token|Authentication| |[datadogLogsAppKey](#datadoglogsappkey "Datadog app key")|Datadog app key|Authentication| |[datadogLogsQuery](#datadoglogsquery "query to pass to the Datadog 'api/v2/logs/events/search' REST endpoint")|Query to pass to the Datadog 'api/v2/logs/events/search' REST endpoint|Query| |[datadogLogsIndexes](#datadoglogsindexes "list of indices to search in")|List of indices to search in|Query| |[datadogLogsQuerySize](#datadoglogsquerysize "number of documents to read per batch from the Datadog server")|Number of documents to read per batch from the Datadog server|Query| |[datadogLogsSearchFrom](#datadoglogssearchfrom "timestamp from which to begin the search")|Timestamp from which to begin the search|Query| |[datadogLogsRetryTimeout](#datadoglogsretrytimeout "a timeout period (in milliseconds) to wait for querying of new data")|A timeout period (in milliseconds) to wait for querying of new data|Query| |[datadogLogsTotalBytesLimit](#datadoglogstotalbyteslimit "maximum total bytes to read from input before closing")|Maximum total bytes to read from input before closing|Backpressure| |[datadogLogsTotalEventsLimit](#datadoglogstotaleventslimit "maximum total events to read from input before closing")|Maximum total events to read from input before closing|Backpressure| |[datadogLogsQueryInterval](#datadoglogsqueryinterval "query interval (in milliseconds) for checking new data from remote source")|Query interval (in milliseconds) for checking new data from remote source|Backpressure| ### General #### :material-menu-right-outline:**`datadogLogsName`** Logical name for this DatadogLogs input. |Type|Required|Category| |---|---|---| |String|✔|General| Sets a logical name (e.g., 'myDatadogLogs') for this input. The [inputName](https://doc.log10x.com/api/js/#TenXBaseObject+inputName "Returns the context in which the current object, template or summary was created.") field returns this value at run time to allow for identifying and operating on instances originating from this input. #### :material-menu-right-outline:**`datadogLogsEnabled`** Sets whether this input is enabled. |Type|Default|Category| |---|---|---| |Boolean|true|General| Sets whether to open the input stream. To enable this input only when a 'datadogLogsHost' startup argument value is truthy, use: ```yaml datadogLogsEnabled: $=TenXEnv.get("datadogLogsHost") ``` To learn more see [TenXEnv.get](https://doc.log10x.com/api/js/#TenXEnv.get). #### :material-menu-right-outline:**`datadogLogsPrintProgress`** Sets whether this input prints throughput stats to the console. |Type|Default|Category| |---|---|---| |Boolean|false|General| Sets whether this input prints throughput stats to the console for testing an integration to a remote endpoint. #### :material-menu-right-outline:**`datadogLogsTotalDuration`** Sets maximum time to query the backend. |Type|Default|Category| |---|---|---| |String|5min|General| Sets the maximum time the pipeline will continue to query the datadog api backend before exiting. For example, setting this to '1min' means that queries to the backend will stop after 1 minute has passed. ### Advanced #### :material-menu-right-outline:**`datadogLogsExtractors`** List of extractors to apply on the events returning from the query. |Type|Default|Category| |---|---|---| |List|\[\]|Advanced| Sets list of extractor names which will work on the events returning from the query. For more info on extractors, see: https://doc.log10x.com/run/input/extract. ### Authentication #### :material-menu-right-outline:**`datadogLogsHost`** DatadogLogs host address. |Type|Required|Category| |---|---|---| |String|✔|Authentication| Sets the Datadog site address to connect to (e.g., 'us5.datadoghq.com'). To learn more see [Datadog sites](https://docs.datadoghq.com/getting_started/site){target="\_blank"}. #### :material-menu-right-outline:**`datadogLogsApiKey`** Datadog API key token. |Type|Required|Category| |---|---|---| |String|✔|Authentication| Sets the 'DD-API-KEY' API key authentication request header. To learn more see: [API keys](https://docs.datadoghq.com/account_management/api-app-keys/){target="\_blank"}. #### :material-menu-right-outline:**`datadogLogsAppKey`** Datadog app key. |Type|Required|Category| |---|---|---| |String|✔|Authentication| Sets the 'DD-APPLICATION-KEY' app key authentication request header. To learn more see: [API keys](https://docs.datadoghq.com/account_management/api-app-keys/){target="\_blank"}. ### Query #### :material-menu-right-outline:**`datadogLogsQuery`** Query to pass to the Datadog 'api/v2/logs/events/search' REST endpoint. |Type|Default|Category| |---|---|---| |String|""|Query| Sets the Datadog query used to request paginated documents from which to extract events to transform into TenXObjects. The [default query](https://github.com/log-10x/modules/blob/main/pipelines/run/modules/input/analyzer/datadogLogs/query.txt){target="\_blank"} retrieves paginated values sorted in desc order by @timestamp, starting from the value of 'now-15m'. #### :material-menu-right-outline:**`datadogLogsIndexes`** List of indices to search in. |Type|Default|Category| |---|---|---| |List|\[\]|Query| Sets the lists of data indices to search in. To search all data streams and indices, omit this parameter or use \* or \_all. #### :material-menu-right-outline:**`datadogLogsQuerySize`** Number of documents to read per batch from the Datadog server. |Type|Default|Category| |---|---|---| |Number|500|Query| Sets the number of documents to retrieve with each page of results. #### :material-menu-right-outline:**`datadogLogsSearchFrom`** Timestamp from which to begin the search. |Type|Default|Category| |---|---|---| |String|now() in nano.|Query| This value sets the initial value used to request the first page of results. #### :material-menu-right-outline:**`datadogLogsRetryTimeout`** A timeout period (in milliseconds) to wait for querying of new data. |Type|Default|Category| |---|---|---| |Number|10000|Query| Used by [datadog-input.js](https://github.com/log-10x/modules/blob/main/pipelines/run/modules/input/analyzer/datadogLogs/datadog-input.js){target="\_blank"} to determine how long to wait until the next time the search endpoint API is invoked to query for new data. ### Backpressure #### :material-menu-right-outline:**`datadogLogsTotalBytesLimit`** Maximum total bytes to read from input before closing. |Type|Default|Category| |---|---|---| |Number|50000000|Backpressure| sets the maximum number of bytes a target pipeline input will read into the pipeline. This value limits the volume of events to read from a local/remote source (e.g., log analyzer). For example: ```yaml datadogLogsTotalBytesLimit: $=parseBytes("1GB") ``` #### :material-menu-right-outline:**`datadogLogsTotalEventsLimit`** Maximum total events to read from input before closing. |Type|Default|Category| |---|---|---| |Number|10000|Backpressure| Sets the maximum number of events a target pipeline input will read into the pipeline. This value limits the volume of events to read from a local/remote source (e.g., log analyzer). #### :material-menu-right-outline:**`datadogLogsQueryInterval`** Query interval (in milliseconds) for checking new data from remote source. |Type|Default|Category| |---|---|---| |Number|2000|Backpressure| sets the interval between queries to the remote Datadog Logs API. This controls how frequently the input polls for new log data. For example: ```yaml datadogLogsQueryInterval: $=parseDuration("5s") ```
:material-github: This module is defined in [datadogLogs/module.yaml](https://github.com/log-10x/modules/tree/main/pipelines/run/modules/input/analyzer/datadogLogs/module.yaml "datadogLogs/module.yaml"){target="\_blank"}.