--- title: "ElasticSearch Input" description: "read events from an ElasticSearch hosted/on-premises cluster" source: "https://github.com/log-10x/modules/tree/main/pipelines/run/modules/input/analyzer/elasticsearch/module.yaml" icon: "simple/elasticsearch" --- Reads events from Elasticsearch clusters and transforms them into TenXObjects. Instances of this [module](https://doc.log10x.com/engine/module/) define a connection to a hosted/on-premises ElasticSearch cluster from which events to retrieve, as well as the querying logic used such as chronological direction, start values, time ranges, and page size of each API request sent. ElasticSearch inputs commonly run within scheduled jobs (e.g., k8s CronJob) to retrieve a recent sample amount of events (e.g., 200MB in the last 10min) to [transform](https://doc.log10x.com/run/transform/ "Transform log and trace events into well-defined TenXObjects") into TenXObjects as part of the [Dev app](https://doc.log10x.com/apps/dev/) app. ## :material-wrench-outline: Config Files To configure the ElasticSearch input module, [:material-cog: Edit](https://doc.log10x.com/config/app/#module-config "Learn how to edit app and module configurations") these files: ??? tenx-elastic "Elastic" ### Elastic {.hidden-heading} Activate [ElasticSearch inputs](/run/input/analyzer/elasticsearch "Read events from an ElasticSearch hosted/on-premises cluster") to read events from an ElasticSearch hosted/on-premises cluster. Below is the default configuration from: [elastic/config.yaml](https://github.dev/log-10x/config/blob/main/pipelines/run/input/analyzer/elastic/config.yaml "elastic/config.yaml"){target="\_blank"} ().

Edit Online

```yaml # 🔟❎ 'run' Elastic input configuration # Configure an Elastic event input # To learn more see https://doc.log10x.com/run/input/analyzer/elasticsearch/ # Set the 10x pipeline to 'run' tenx: run # =============================== Dependencies ================================ include: run/modules/input/analyzer/elasticsearch # =============================== Elastic Options ============================== # Multiple Elastic inputs can be defined below: elasticSearch: # 'name' sets a unique logical name across all pipeline inputs - name: ElasticSearch # --------------------------- Connection Options -------------------------- # 'host' and 'port' set the ElasticSearch API endpoint to connect to # For hosted Elastic (e.g. my-project-ec5189.es.us-central1.gcp.elastic.cloud). # See https://cloud.elastic.co/deployments and https://i.stack.imgur.com/jXjLJ.png host: null # (❗ REQUIRED) port: "" # (not mandatory if the host already encapsulates it) # 'protocol' sets the connection protocol ('https' or 'http' for local testing) protocol: https # 'token' used to authenticate with. To learn more see: # https://www.elastic.co/guide/en/cloud-enterprise/current/ece-restful-api-authentication.html token: $=TenXEnv.get("ELASTIC_TOKEN") # (❗ REQUIRED) # ----------------------------- Query Options ----------------------------- # 'targets' lists data streams, indices, and aliases to search. # To search all data streams and indices, use '*' or '_all'. targets: - _all # 'startTime' and 'endTime' set the time frame of the query. startTime: now-1h endTime: now # --------------------------- Backpressure Options ----------------------- # 'queryInterval' sets the interval between queries to the remote API queryInterval: $=parseDuration("2s") # 'totalDuration' sets the max duration to try reading from the the remote input totalDuration: $=parseDuration("5min") # 'totalBytesLimit' sets the max total bytes to read from the remote input totalBytesLimit: $=parseBytes("50MB") # 'totalEventsLimit' sets the max number of events to read the remote input totalEventsLimit: 10000 # --------------------------- Ancillary Options --------------------------- # 'printProgress' controls whether to print a progress gage to the console # This option helps debug and test the input printProgress: $=!TenXEnv.get("quiet") # --------------------------- Advanced Options --------------------------- # 'querySize' sets the number of documents retrieved with each results page. querySize: 500 # 'query' sets the elastic query to request paginated documents from the '/_search' REST endpoint. # The default query (defined in 'input/analyzer/elasticsearch/query.txt') # retrieves paginated values sorted in desc order by @timestamp query: null ``` ??? tenx-logzio "Logz.io" ### Logz.io {.hidden-heading} Below is the default configuration from: [logzio/config.yaml](https://github.dev/log-10x/config/blob/main/pipelines/run/input/analyzer/logzio/config.yaml "logzio/config.yaml"){target="\_blank"} ().

Edit Online

```yaml # 🔟❎ 'run' Logz.io ElasticSearch input configuration # Configures a Logz.io ElasticSearch event input # To learn more see https://doc.log10x.com/run/input/analyzer/elasticsearch/ # Set the 10x pipeline to 'run' tenx: run # =============================== Dependencies ================================ include: run/modules/input/analyzer/elasticsearch # =============================== Logzio Options ============================== # Multiple Logz.io inputs can be defined below: elasticSearch: # 'name' sets a unique logical name across all pipeline inputs - name: logzioElasticSearch # --------------------------- Connection Options -------------------------- # 'token' must be set to the Logz.io API token. # See: https://docs.logz.io/user-guide/tokens/api-tokens.html token: null # (❗ REQUIRED) # 'host' and 'port' set the Logz.io service address to connect to host: api.logz.io port: 443 # ----------------------------- Query Options ----------------------------- # 'targets' lists data streams, indices, and aliases to search. # To search all data streams and indices, use '*' or '_all'. targets: - v1 # 'startTime' and 'endTime' set the time frame of the query. startTime: now-5m endTime: now # --------------------------- Backpressure Options ----------------------- # 'queryInterval' sets the interval between queries to the remote API queryInterval: $=parseDuration("2s") # 'totalDuration' sets the max duration to try reading from the the remote input totalDuration: $=parseDuration("5min") # 'totalBytesLimit' sets the max total bytes to read from the remote input totalBytesLimit: $=parseBytes("50MB") # 'totalEventsLimit' sets the max number of events to read the remote input totalEventsLimit: 10000 # --------------------------- Ancillary Options --------------------------- # 'printProgress' controls whether to print a progress gage to the console # This option helps debug and test the input printProgress: $=!TenXEnv.get("quiet") # --------------------------- Advanced Options --------------------------- # 'querySize' sets the number of documents retrieved with each results page. querySize: 500 # 'query' sets the elastic query to request paginated documents from the '/_search' REST endpoint. # The default query (defined in 'input/analyzer/elasticsearch/query.txt') # retrieves paginated values sorted in desc order by @timestamp query: null # 'endpoint' sets the REST API endpoint to query endpoint: search # 'messageField' defines the name of the field to extract from the retrieved # documents containing the log message. messageField: message ``` ??? tenx-coralogix "Coralogix" ### Coralogix {.hidden-heading} Below is the default configuration from: [coralogix/config.yaml](https://github.dev/log-10x/config/blob/main/pipelines/run/input/analyzer/coralogix/config.yaml "coralogix/config.yaml"){target="\_blank"} ().

Edit Online

```yaml # 🔟❎ 'run' Coralogix ElasticSearch input configuration # This configuration defines a Coralogix ElasticSearch input from # which events are read and transformed into TenXObjects for aggregation and reporting. # Set the 10x pipeline to 'run' tenx: run # =============================== Dependencies ================================ include: run/modules/input/analyzer/elasticsearch # ============================== Coralogix Options ============================ # Multiple Coralogix inputs can be defined below elasticSearch: # 'name' sets a unique logical name across all pipeline inputs - name: coralogixElasticSearch # ------------------------- Connection Options ---------------------------- # 'token' must be set to the Coralogix send data API key # To learn more see: https://coralogix.com/docs/user-guides/account-management/api-keys/send-your-data-api-key/ token: null # (❗ REQUIRED) # 'authorizationKeyName' must be set to match Coralogix auth type # To learn more see: https://coralogix.com/docs/user-guides/visualizations/hosted-opensearch-view/opensearch-api/ authorizationKeyName: Bearer # 'host' and 'port' set the Coralogix service address to connect to host: api.coralogix.us/data/os-api # --------------------------- Query Options ------------------------------- # 'targets' lists data streams, indices, and aliases to search. # To search all data streams and indices, use '*' or '_all.' targets: - '*' # 'startTime' and 'endTime' set the time frame of the query. startTime: now-5m endTime: now # --------------------------- Backpressure Options ----------------------- # 'queryInterval' sets the interval between queries to the remote API queryInterval: $=parseDuration("2s") # 'totalDuration' sets the max duration to try reading from the the remote input totalDuration: $=parseDuration("5min") # 'totalBytesLimit' sets the max total bytes to read from the remote input totalBytesLimit: $=parseBytes("50MB") # 'totalEventsLimit' sets the max number of events to read the remote input totalEventsLimit: 10000 # --------------------------- Ancillary Options --------------------------- # 'printProgress' controls whether to print a progress gage to the console # This option helps debug and test the input printProgress: $=!TenXEnv.get("quiet") # --------------------------- Advanced Options ---------------------------- # 'querySize' sets the number of documents retrieved with each results page. querySize: 500 # 'query' sets the elastic query to request paginated documents from the '/_search' REST endpoint. # The default query (defined in 'input/analyzer/elasticsearch/query.txt') # retrieves paginated values sorted in desc order by @timestamp query: null # 'timestampField' specifies the name of the field to get the event timestamp from. timestampField: coralogix.timestamp # 'messageField' defines the name of the field to extract from the retrieved # documents containing the log message. messageField: log ``` ## :material-menu: Options Specify the options below to [configure](/config "configure") multiple ElasticSearch input: |Name|Description|Category| |---|---|---| |[elasticSearchName](#elasticsearchname "logical name for this elastic input")|Logical name for this elastic input|General| |[elasticSearchEnabled](#elasticsearchenabled "sets whether this input is enabled")|Sets whether this input is enabled|General| |[elasticSearchPrintProgress](#elasticsearchprintprogress "sets whether this input prints throughput stats to the console")|Sets whether this input prints throughput stats to the console|General| |[elasticSearchTotalDuration](#elasticsearchtotalduration "sets maximum time to query the backend")|Sets maximum time to query the backend|General| |[elasticSearchExtractors](#elasticsearchextractors "list of extractors to apply on the events returning from the query")|List of extractors to apply on the events returning from the query|General| |[elasticSearchHost](#elasticsearchhost "elastic host address")|Elastic host address|Authentication| |[elasticSearchPort](#elasticsearchport "elastic server port")|Elastic server port|Authentication| |[elasticSearchProtocol](#elasticsearchprotocol "defines the protocol to connect to ElasticSearch")|Defines the protocol to connect to ElasticSearch|Authentication| |[elasticSearchToken](#elasticsearchtoken "elastic authentication token")|Elastic authentication token|Authentication| |[elasticSearchAuthorizationKeyName](#elasticsearchauthorizationkeyname "prefix for the elastic authentication token")|Prefix for the elastic authentication token|Authentication| |[elasticSearchQuery](#elasticsearchquery "query to pass to the elastic server")|Query to pass to the elastic server|Query| |[elasticSearchMessageField](#elasticsearchmessagefield "name of the field used to extract the log message from retrieved documents")|Name of the field used to extract the log message from retrieved documents|Query| |[elasticSearchTimestampField](#elasticsearchtimestampfield "name of the field used to sort retrieved documents by")|Name of the field used to sort retrieved documents by|Query| |[elasticSearchStartTime](#elasticsearchstarttime "The start of the time range, expressed as a relative string (i.e., \"now-7d\") or the number of milliseconds after Jan 1, 1970 00:00:00 UTC.")|The start of the time range, expressed as a relative string (i.e., "now-7d") or the number of milliseconds after Jan 1, 1970 00:00:00 UTC.|Query| |[elasticSearchEndTime](#elasticsearchendtime "The end of the time range, expressed as a relative string (i.e., \"now-7d\") or the number of milliseconds after Jan 1, 1970 00:00:00 UTC")|The end of the time range, expressed as a relative string (i.e., "now-7d") or the number of milliseconds after Jan 1, 1970 00:00:00 UTC|Query| |[elasticSearchTargets](#elasticsearchtargets "list of data streams, indices, and aliases to search")|List of data streams, indices, and aliases to search|Query| |[elasticSearchEndpoint](#elasticsearchendpoint "name of search REST endpoint")|Name of search REST endpoint|Query| |[elasticSearchQuerySize](#elasticsearchquerysize "number of documents to read per batch from the elastic server")|Number of documents to read per batch from the elastic server|Query| |[elasticSearchRetryTimeout](#elasticsearchretrytimeout "a timeout period (in milliseconds) to wait for querying of new data")|A timeout period (in milliseconds) to wait for querying of new data|Query| |[elasticSearchTotalBytesLimit](#elasticsearchtotalbyteslimit "maximum total bytes to read from input before closing")|Maximum total bytes to read from input before closing|Backpressure| |[elasticSearchTotalEventsLimit](#elasticsearchtotaleventslimit "maximum total events to read from input before closing")|Maximum total events to read from input before closing|Backpressure| |[elasticSearchQueryInterval](#elasticsearchqueryinterval "query interval (in milliseconds) for checking new data from remote source")|Query interval (in milliseconds) for checking new data from remote source|Backpressure| ### General #### :material-menu-right-outline:**`elasticSearchName`** Logical name for this elastic input. |Type|Default|Category| |---|---|---| |String|""|General| Sets a logical name (e.g., 'myElastic') for this input. The [inputName](https://doc.log10x.com/api/js/#TenXBaseObject+inputName "Returns the context in which the current object, template or summary was created.") field returns this value at run time to allow for identifying and operating on instances originating from this input. #### :material-menu-right-outline:**`elasticSearchEnabled`** Sets whether this input is enabled. |Type|Default|Category| |---|---|---| |Boolean|true|General| Sets whether to open the input stream. To enable this input only when a 'elasticSearchHost' startup argument value is truthy, use: ```yaml elasticSearchEnabled: $=TenXEnv.get("elasticSearchHost") ``` To learn more see [TenXEnv.get](https://doc.log10x.com/api/js/#TenXEnv.get). #### :material-menu-right-outline:**`elasticSearchPrintProgress`** Sets whether this input prints throughput stats to the console. |Type|Default|Category| |---|---|---| |Boolean|false|General| Sets whether this input prints throughput stats to the console for testing an integration to a remote endpoint. #### :material-menu-right-outline:**`elasticSearchTotalDuration`** Sets maximum time to query the backend. |Type|Default|Category| |---|---|---| |String|5min|General| Sets the maximum time the pipeline will continue to query the backend before exiting. For example, setting this to '1min' means that queries to the backend will stop after 1 minute has passed. #### :material-menu-right-outline:**`elasticSearchExtractors`** List of extractors to apply on the events returning from the query. |Type|Default|Category| |---|---|---| |List|\[\]|General| Sets list of extractor names which will work on the events returning from the query. For more info on extractors, see: https://doc.log10x.com/run/input/extract. ### Authentication #### :material-menu-right-outline:**`elasticSearchHost`** Elastic host address. |Type|Required|Category| |---|---|---| |String|✔|Authentication| Set the elastic host address to connect to. Examples: coralogix-esapi.coralogix.com api.logz.io .us-central1.gcp.cloud.es.io. #### :material-menu-right-outline:**`elasticSearchPort`** Elastic server port. |Type|Default|Category| |---|---|---| |Number|0|Authentication| Sets the elastic server port to connect to (e.g., '9443') The port value is not needed if [elasticSearchHost](https://doc.log10x.com/run/input/analyzer/elasticsearch/#elasticsearchhost "elastic host address") value encapsulates the port. #### :material-menu-right-outline:**`elasticSearchProtocol`** Defines the protocol to connect to ElasticSearch. |Type|Default|Category| |---|---|---| |String|https|Authentication| Sets the protocol to connect to elastic with (e.g., `https`). #### :material-menu-right-outline:**`elasticSearchToken`** Elastic authentication token. |Type|Required|Category| |---|---|---| |String|✔|Authentication| Sets the API token used to authenticate against the target elastic cluster. To learn more see: [api tokens](https://www.elastic.co/guide/en/elasticsearch/reference/current/security-api-get-token.html){target="\_blank"}. #### :material-menu-right-outline:**`elasticSearchAuthorizationKeyName`** Prefix for the elastic authentication token. |Type|Default|Category| |---|---|---| |String|ApiKey|Authentication| Sets the prefix for the provided [elasticSearchToken](https://doc.log10x.com/run/input/analyzer/elasticsearch/#elasticsearchtoken "elastic authentication token") when using it in an 'Authorization' header. commonly used values by different elastic vendors are 'ApiKey' and 'Bearer'. ### Query #### :material-menu-right-outline:**`elasticSearchQuery`** Query to pass to the elastic server. |Type|Default|Category| |---|---|---| |String|""|Query| Sets the elastic query to request paginated documents from the '/\_search' REST endpoint. The default query (defined in 'elasticsearch/query.txt') retrieves paginated values sorted in desc order by @timestamp, starting from the value of 'now()'. #### :material-menu-right-outline:**`elasticSearchMessageField`** Name of the field used to extract the log message from retrieved documents. |Type|Default|Category| |---|---|---| |String|log|Query| Sets the name of the field used to extract the log message from retrieved documents. #### :material-menu-right-outline:**`elasticSearchTimestampField`** Name of the field used to sort retrieved documents by. |Type|Default|Category| |---|---|---| |String|@timestamp|Query| Sets the name of the field used to sort retrieved documents by. #### :material-menu-right-outline:**`elasticSearchStartTime`** The start of the time range, expressed as a relative string (i.e., "now-7d") or the number of milliseconds after Jan 1, 1970 00:00:00 UTC. |Type|Default|Category| |---|---|---| |String|""|Query| Sets the start of the time range, expressed as a relative string (i.e., "now-7d") or the number of milliseconds after Jan 1, 1970 00:00:00 UTC. The query includes events with a timestamp equal or later to this value. #### :material-menu-right-outline:**`elasticSearchEndTime`** The end of the time range, expressed as a relative string (i.e., "now-7d") or the number of milliseconds after Jan 1, 1970 00:00:00 UTC. |Type|Default|Category| |---|---|---| |String|""|Query| Sets the end of the time range, expressed as a relative string (i.e., "now-7d") or the number of milliseconds after Jan 1, 1970 00:00:00 UTC. The query excludes events with a timestamp later than this value. #### :material-menu-right-outline:**`elasticSearchTargets`** List of data streams, indices, and aliases to search. |Type|Default|Category| |---|---|---| |List|\[\]|Query| Sets list of data streams, indices, and aliases to search. To search all data streams and indices, omit this parameter or use '\*' or '\_all'. #### :material-menu-right-outline:**`elasticSearchEndpoint`** Name of search REST endpoint. |Type|Default|Category| |---|---|---| |String|\_search|Query| Sets the name of the search REST endpoint. #### :material-menu-right-outline:**`elasticSearchQuerySize`** Number of documents to read per batch from the elastic server. |Type|Default|Category| |---|---|---| |Number|500|Query| Sets the number of documents to retrieve with each page of results. #### :material-menu-right-outline:**`elasticSearchRetryTimeout`** A timeout period (in milliseconds) to wait for querying of new data. |Type|Default|Category| |---|---|---| |Number|10000|Query| Determines how long to wait to invoke the search endpoint API again to query for new data. See [elastic-input.js](https://github.com/log-10x/modules/blob/main/pipelines/run/modules/input/analyzer/elasticsearch/elastic-input.js){target="\_blank"}. ### Backpressure #### :material-menu-right-outline:**`elasticSearchTotalBytesLimit`** Maximum total bytes to read from input before closing. |Type|Default|Category| |---|---|---| |Number|50000000|Backpressure| sets the maximum number of bytes a target pipeline input will read into the pipeline. This value limits the volume of events to read from a local/remote source (e.g., log analyzer). For example: ```yaml elasticSearchTotalBytesLimit: $=parseBytes("1GB") ``` #### :material-menu-right-outline:**`elasticSearchTotalEventsLimit`** Maximum total events to read from input before closing. |Type|Default|Category| |---|---|---| |Number|10000|Backpressure| Sets the maximum number of events a target pipeline input will read into the pipeline. This value limits the volume of events to read from a local/remote source (e.g., log analyzer). #### :material-menu-right-outline:**`elasticSearchQueryInterval`** Query interval (in milliseconds) for checking new data from remote source. |Type|Default|Category| |---|---|---| |Number|2000|Backpressure| sets the interval between queries to the remote Elasticsearch API. This controls how frequently the input polls for new log data. For example: ```yaml elasticSearchQueryInterval: $=parseDuration("5s") ```
:material-github: This module is defined in [elasticsearch/module.yaml](https://github.com/log-10x/modules/tree/main/pipelines/run/modules/input/analyzer/elasticsearch/module.yaml "elasticsearch/module.yaml"){target="\_blank"}.