--- title: "AWS Cloudwatch Logs Input" description: "read events from AWS CloudWatch Logs" source: "https://github.com/log-10x/modules/tree/main/pipelines/run/modules/input/analyzer/cloudwatchLogs/module.yaml" icon: "fontawesome/brands/aws" --- Configures an AWS CloudWatch Logs input from which to read events to transform into typed [TenXObjects](https://doc.log10x.com/api/js/#TenXObject "Provide structured, reflective access to log/trace events read from input(s)."). Instances of this [module](https://doc.log10x.com/engine/module/) define a connection to an AWS CloudWatch Logs service from which to retrieve log messages, as well as the querying logic used such as chronological direction, start values, time ranges, and page size of each API request sent. Cloudwatch Logs inputs commonly run within scheduled jobs (e.g., k8s CronJob) to retrieve a recent sample amount of events (e.g., 200MB in the last 10min) to [transform](https://doc.log10x.com/run/transform/ "Transform log and trace events into well-defined TenXObjects") into TenXObjects as part of the [Dev app](https://doc.log10x.com/apps/dev/) app. ## :material-wrench-outline: Config Files To configure the AWS Cloudwatch Logs input module, [:material-cog: Edit](https://doc.log10x.com/config/app/#module-config "Learn how to edit app and module configurations") these files. Below is the default configuration from: [cloudwatchLogs/config.yaml](https://github.dev/log-10x/config/blob/main/pipelines/run/input/analyzer/cloudwatchLogs/config.yaml "cloudwatchLogs/config.yaml"){target="\_blank"} ().

Edit Online

```yaml # 🔟❎ 'run' AWS CloudWatch Logs input configuration # Configure an AWS CloudWatch Logs event input # To learn more see https://doc.log10x.com/run/input/analyzer/cloudwatchLogs/ # Set the 10x pipeline to 'run' tenx: run # =============================== Dependencies ================================ include: run/modules/input/analyzer/cloudwatchLogs # ============================ CloudWatch Options ============================= # Multiple CloudWatch inputs can be defined below cloudwatchLogs: # 'name' sets a unique logical name across all pipeline inputs - name: CloudwatchLogs # --------------------------- Connection Options -------------------------- # 'awsAccessKeyID' sets the AWS access key (e.g. 'AKIAIOSFODNN7EXAMPLE') # To learn more see: https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_access-keys.html awsAccessKeyID: $=TenXEnv.get("AWS_ACCESS_KEY_ID") # (❗ EnvVar REQUIRED) # 'awsSecretKey' sets the AWS access secret key (e.g. 'wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY') # To learn more see: https://docs.aws.amazon.com/general/latest/gr/aws-sec-cred-types.html awsSecretKey: $=TenXEnv.get("AWS_SECRET_ACCESS_KEY") # (❗ EnvVar REQUIRED) # 'groupName' sets the log group name (e.g., 'my-log-group'). # To learn more see: https://docs.aws.amazon.com/AmazonCloudWatchLogs/latest/APIReference/API_GetLogEvents.html#CWL-GetLogEvents-request-logGroupName groupName: "" # (❗ REQUIRED) # 'streamName' sets the log stream name (e.g., 'my-log-stream'). # To learn more see: https://docs.aws.amazon.com/AmazonCloudWatchLogs/latest/APIReference/API_GetLogEvents.html#CWL-GetLogEvents-request-logStreamName streamName: "" # (❗ REQUIRED) # 'awsRegion' sets the region hosting the CloudWatch Logs stream # To learn more see: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-regions-availability-zones.html awsRegion: $=TenXEnv.get("AWS_DEFAULT_REGION", "us-east-1") # ----------------------------- Query Options ----------------------------- # 'limit' sets the number of documents retrieved with each results page. # To learn more see: https://docs.aws.amazon.com/AmazonCloudWatchLogs/latest/APIReference/API_GetLogEvents.html#CWL-GetLogEvents-request-limit limit: 500 # 'startFromHead' controls the direction of the search (old <-> new events). # To learn more see: https://docs.aws.amazon.com/AmazonCloudWatchLogs/latest/APIReference/API_GetLogEvents.html#CWL-GetLogEvents-request-startFromHead startFromHead: true # 'startTime' sets the start of the time range, expressed as the number of milliseconds after Jan 1, 1970 00:00:00 UTC. # The search includes events with a timestamp equal to or later than this value. # The search excludes events with a timestamp earlier than this value. # To learn more see: https://docs.aws.amazon.com/AmazonCloudWatchLogs/latest/APIReference/API_GetLogEvents.html#CWL-GetLogEvents-request-startTime startTime: $=now("-240h") # 'endTime' sets the end of the time range, expressed as the number of milliseconds after Jan 1, 1970 00:00:00 UTC. # The search excludes events with a timestamp equal to or later than this value. # To learn more see: https://docs.aws.amazon.com/AmazonCloudWatchLogs/latest/APIReference/API_GetLogEvents.html#CWL-GetLogEvents-request-endTime endTime: $=now() # --------------------------- Backpressure Options ----------------------- # 'queryInterval' sets the interval between queries to the remote API queryInterval: $=parseDuration("2s") # 'totalDuration' sets the max duration to try reading from the the remote input totalDuration: $=parseDuration("5min") # 'totalBytesLimit' sets the max total bytes to read from the remote input totalBytesLimit: $=parseBytes("50MB") # 'totalEventsLimit' sets the max number of events to read the remote input totalEventsLimit: 10000 # --------------------------- Ancillary Options --------------------------- # 'printProgress' controls whether to print a gage to the console # This option helps debug and test the input printProgress: $=!TenXEnv.get("quiet") # 'messageField' sets the JSON field to extract from CloudWatch log messages # For K8s/Docker logs with 10x encoding, use 'log' to extract the encoded log field messageField: log ``` ## :material-menu: Options Specify the options below to [configure](/config "configure") multiple AWS Cloudwatch Logs input: |Name|Description|Category| |---|---|---| |[cloudwatchLogsName](#cloudwatchlogsname "logical name for this AWS CloudWatch Logs input")|Logical name for this AWS CloudWatch Logs input|General| |[cloudwatchLogsEnabled](#cloudwatchlogsenabled "sets whether this input is enabled")|Sets whether this input is enabled|General| |[cloudwatchLogsPrintProgress](#cloudwatchlogsprintprogress "sets whether this input prints throughput stats to the console")|Sets whether this input prints throughput stats to the console|General| |[cloudwatchLogsAwsRegion](#cloudwatchlogsawsregion "AWS region")|AWS region|Authentication| |[cloudwatchLogsAwsAccessKeyID](#cloudwatchlogsawsaccesskeyid "AWS authentication access key ID")|AWS authentication access key ID|Authentication| |[cloudwatchLogsAwsSecretKey](#cloudwatchlogsawssecretkey "AWS authentication secret key")|AWS authentication secret key|Authentication| |[cloudwatchLogsGroupName](#cloudwatchlogsgroupname "AWS CloudWatch Logs Group name")|AWS CloudWatch Logs Group name|Log group| |[cloudwatchLogsStreamName](#cloudwatchlogsstreamname "AWS CloudWatch Logs Stream name")|AWS CloudWatch Logs Stream name|Log group| |[cloudwatchLogsLimit](#cloudwatchlogslimit "number of events to read per batch from the AWS CloudWatch Logs service")|Number of events to read per batch from the AWS CloudWatch Logs service|Query| |[cloudwatchLogsNextToken](#cloudwatchlogsnexttoken "The token for the next set of items to return (received this token from a previous call)")|The token for the next set of items to return (received this token from a previous call)|Query| |[cloudwatchLogsStartFromHead](#cloudwatchlogsstartfromhead "If true, return the earliest log events first. If false, return the most recent log events first.")|If true, return the earliest log events first. If false, return the most recent log events first.|Query| |[cloudwatchLogsStartTime](#cloudwatchlogsstarttime "The start of the time range, expressed as the number of milliseconds after Jan 1, 1970 00:00:00 UTC.")|The start of the time range, expressed as the number of milliseconds after Jan 1, 1970 00:00:00 UTC.|Query| |[cloudwatchLogsEndTime](#cloudwatchlogsendtime "The end of the time range, expressed as the number of milliseconds after Jan 1, 1970 00:00:00 UTC")|The end of the time range, expressed as the number of milliseconds after Jan 1, 1970 00:00:00 UTC|Query| |[cloudwatchLogsRetryTimeout](#cloudwatchlogsretrytimeout "a timeout period (in milliseconds) to wait for querying of new data")|A timeout period (in milliseconds) to wait for querying of new data|Query| |[cloudwatchLogsTotalBytesLimit](#cloudwatchlogstotalbyteslimit "maximum total bytes to read from input before closing")|Maximum total bytes to read from input before closing|Backpressure| |[cloudwatchLogsTotalEventsLimit](#cloudwatchlogstotaleventslimit "maximum total events to read from input before closing")|Maximum total events to read from input before closing|Backpressure| |[cloudwatchLogsTotalDuration](#cloudwatchlogstotalduration "maximum duration to keep input open before closing")|Maximum duration to keep input open before closing|Backpressure| |[cloudwatchLogsQueryInterval](#cloudwatchlogsqueryinterval "query interval (in milliseconds) for checking new data from remote source")|Query interval (in milliseconds) for checking new data from remote source|Backpressure| |[cloudwatchLogsMessageField](#cloudwatchlogsmessagefield "JSON field name to extract from CloudWatch log messages")|JSON field name to extract from CloudWatch log messages|Extraction| ### General #### :material-menu-right-outline:**`cloudwatchLogsName`** Logical name for this AWS CloudWatch Logs input. |Type|Required|Category| |---|---|---| |String|✔|General| Sets a logical name (e.g., 'myCloudWatchLogs') for this input. The [inputName](https://doc.log10x.com/api/js/#TenXBaseObject+inputName "Returns the context in which the current object, template or summary was created.") field returns this value at run time to allow for identifying and operating on instances originating from this input. #### :material-menu-right-outline:**`cloudwatchLogsEnabled`** Sets whether this input is enabled. |Type|Default|Category| |---|---|---| |Boolean|false|General| Sets whether to open the input stream. To enable this input only when a 'cloudwatchLogsName' startup argument value is truthy, use: ```yaml cloudwatchLogsEnabled: $=TenXEnv.get("cloudwatchLogsName") ``` To learn more see [TenXEnv.get](https://doc.log10x.com/api/js/#TenXEnv.get). #### :material-menu-right-outline:**`cloudwatchLogsPrintProgress`** Sets whether this input prints throughput stats to the console. |Type|Default|Category| |---|---|---| |Boolean|false|General| Sets whether this input prints throughput stats to the console for testing an integration to a remote endpoint. ### Authentication #### :material-menu-right-outline:**`cloudwatchLogsAwsRegion`** AWS region. |Type|Default|Category| |---|---|---| |String|""|Authentication| Sets the AWS region (e.g., 'us-east-1') hosting the target CloudWatch logs stream. To learn more see [AWS Regions](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-regions-availability-zones.html){target="\_blank"}. #### :material-menu-right-outline:**`cloudwatchLogsAwsAccessKeyID`** AWS authentication access key ID. |Type|Default|Category| |---|---|---| |String|""|Authentication| Sets the AWS access key (e.g., 'AKIAIOSFODNN7EXAMPLE') used as part of the request signing process. **If not specified**, the system automatically falls back to the [AWS default credential provider chain](https://docs.aws.amazon.com/sdk-for-java/latest/developer-guide/credentials-chain.html){target="\_blank"}, which checks (in order): 1. Environment variables (`AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`) 2. Java system properties 3. Web identity token (for EKS) 4. Shared credentials file (`~/.aws/credentials`) 5. ECS container credentials 6. EC2 instance profile credentials To learn more see [AWS access keys](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_access-keys.html){target="\_blank"}. #### :material-menu-right-outline:**`cloudwatchLogsAwsSecretKey`** AWS authentication secret key. |Type|Default|Category| |---|---|---| |String|""|Authentication| Sets the AWS access secret key (e.g. 'wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY') used as part of the request signing process. **If not specified**, the system automatically falls back to the [AWS default credential provider chain](https://docs.aws.amazon.com/sdk-for-java/latest/developer-guide/credentials-chain.html){target="\_blank"}. This allows using credentials from `~/.aws/credentials`, IAM roles, or environment variables without explicit configuration. To learn more see [AWS security credentials](https://docs.aws.amazon.com/general/latest/gr/aws-sec-cred-types.html){target="\_blank"}. ### Log group #### :material-menu-right-outline:**`cloudwatchLogsGroupName`** AWS CloudWatch Logs Group name. |Type|Required|Category| |---|---|---| |String|✔|Log group| Used within the body of the 'GetLogEvents' AWS request to specify the log group name (e.g., 'my-log-group'). To learn more see [logGroupName](https://docs.aws.amazon.com/AmazonCloudWatchLogs/latest/APIReference/API_GetLogEvents.html#CWL-GetLogEvents-request-logGroupName){target="\_blank"}. #### :material-menu-right-outline:**`cloudwatchLogsStreamName`** AWS CloudWatch Logs Stream name. |Type|Required|Category| |---|---|---| |String|✔|Log group| Used within the body of the AWS 'AGetLogEvents' request to specify the log stream name (e.g., 'my-log-stream'). To learn more see [logStreamName](https://docs.aws.amazon.com/AmazonCloudWatchLogs/latest/APIReference/API_GetLogEvents.html#CWL-GetLogEvents-request-logStreamName){target="\_blank"}. ### Query #### :material-menu-right-outline:**`cloudwatchLogsLimit`** Number of events to read per batch from the AWS CloudWatch Logs service. |Type|Default|Category| |---|---|---| |Number|500|Query| Sets the number of documents to retrieve with each page of results. To learn more see: [request limit](https://docs.aws.amazon.com/AmazonCloudWatchLogs/latest/APIReference/API_GetLogEvents.html#CWL-GetLogEvents-request-limit){target="\_blank"}. #### :material-menu-right-outline:**`cloudwatchLogsNextToken`** The token for the next set of items to return (received this token from a previous call). |Type|Default|Category| |---|---|---| |String|""|Query| Sets the token from which the requested result page is to start. This value is returned from a previous call to the AWS 'GetLogEvents' command. To learn more see: [nextToken](https://docs.aws.amazon.com/AmazonCloudWatchLogs/latest/APIReference/API_GetLogEvents.html#CWL-GetLogEvents-request-nextToken){target="\_blank"}. #### :material-menu-right-outline:**`cloudwatchLogsStartFromHead`** If true, return the earliest log events first. If false, return the most recent log events first. |Type|Default|Category| |---|---|---| |Boolean|false|Query| Controls the direction of the search (old \<-\> new events). By default, the search starts from the most recent events. To learn more see: [startFromHead](https://docs.aws.amazon.com/AmazonCloudWatchLogs/latest/APIReference/API_GetLogEvents.html#CWL-GetLogEvents-request-startFromHead){target="\_blank"}. #### :material-menu-right-outline:**`cloudwatchLogsStartTime`** The start of the time range, expressed as the number of milliseconds after Jan 1, 1970 00:00:00 UTC. |Type|Default|Category| |---|---|---| |Number|0|Query| Sets the start of the time range, expressed as milliseconds after Jan 1, 1970 00:00:00 UTC. Events with a timestamp equal to this time or later than this time are included. Events with a timestamp earlier than this time are excluded: To learn more see: [startTime](https://docs.aws.amazon.com/AmazonCloudWatchLogs/latest/APIReference/API_GetLogEvents.html#CWL-GetLogEvents-request-startTime){target="\_blank"}. #### :material-menu-right-outline:**`cloudwatchLogsEndTime`** The end of the time range, expressed as the number of milliseconds after Jan 1, 1970 00:00:00 UTC. |Type|Default|Category| |---|---|---| |Number|0|Query| Sets the end of the time range, expressed as milliseconds after Jan 1, 1970 00:00:00 UTC. Events with a timestamp equal to or later than this time are not included: To learn more see: [endTime](https://docs.aws.amazon.com/AmazonCloudWatchLogs/latest/APIReference/API_GetLogEvents.html#CWL-GetLogEvents-request-endTime){target="\_blank"}. #### :material-menu-right-outline:**`cloudwatchLogsRetryTimeout`** A timeout period (in milliseconds) to wait for querying of new data. |Type|Default|Category| |---|---|---| |Number|20000|Query| Used by [cloudwatchLogs-input.js](https://github.com/log-10x/modules/blob/main/pipelines/run/modules/input/analyzer/cloudwatchLogs/cloudwatchLogs-input.js){target="\_blank"} to determine how long to wait to invoke the 'GetLogEvents' API again to query for new data. ### Backpressure #### :material-menu-right-outline:**`cloudwatchLogsTotalBytesLimit`** Maximum total bytes to read from input before closing. |Type|Default|Category| |---|---|---| |Number|50000000|Backpressure| sets the maximum number of bytes a target pipeline input will read into the pipeline. This value limits the volume of events to read from a local/remote source (e.g., log analyzer). For example: ```yaml totalBytesLimit: $=parseBytes("1GB") ``` #### :material-menu-right-outline:**`cloudwatchLogsTotalEventsLimit`** Maximum total events to read from input before closing. |Type|Default|Category| |---|---|---| |Number|10000|Backpressure| Sets the maximum number of events a target pipeline input will read into the pipeline. This value limits the volume of events to read from a local/remote source (e.g., log analyzer). #### :material-menu-right-outline:**`cloudwatchLogsTotalDuration`** Maximum duration to keep input open before closing. |Type|Default|Category| |---|---|---| |String|parseDuration("5min")|Backpressure| sets the maximum duration a target pipeline input will remain open. When reached, the input will close and no more data will be read. For example: ```yaml cloudwatchLogsTotalDuration: $=parseDuration("10min") ``` #### :material-menu-right-outline:**`cloudwatchLogsQueryInterval`** Query interval (in milliseconds) for checking new data from remote source. |Type|Default|Category| |---|---|---| |Number|2000|Backpressure| sets the interval between queries to the remote CloudWatch Logs API. This controls how frequently the input polls for new log data. For example: ```yaml cloudwatchLogsQueryInterval: $=parseDuration("5s") ``` ### Extraction #### :material-menu-right-outline:**`cloudwatchLogsMessageField`** JSON field name to extract from CloudWatch log messages. |Type|Default|Category| |---|---|---| |String||Extraction| sets the JSON field name to extract from each CloudWatch log message. When set, the extractor will look for this field in each message's JSON payload. When empty, the message extraction step is skipped. For example, to extract the 'log' field from Kubernetes/Docker logs: ```yaml messageField: log ```
:material-github: This module is defined in [cloudwatchLogs/module.yaml](https://github.com/log-10x/modules/tree/main/pipelines/run/modules/input/analyzer/cloudwatchLogs/module.yaml "cloudwatchLogs/module.yaml"){target="\_blank"}.