--- title: "AWS S3 Logs Input" description: "read log events from a single AWS S3 object" source: "https://github.com/log-10x/modules/tree/main/pipelines/run/modules/input/analyzer/s3Logs/module.yaml" icon: "material/bucket-outline" --- The S3 Logs module enables reading log and trace events from a single [AWS S3](https://aws.amazon.com/s3/){target="\_blank"} object for analysis and transformation into [TenXObjects](https://doc.log10x.com/api/js/#TenXObject "Provide structured, reflective access to log/trace events read from input(s)."). ### :material-hammer-wrench: How It Works The S3 Logs input fetches a single S3 object by key using the [GetObject](https://docs.aws.amazon.com/AmazonS3/latest/API/API_GetObject.html){target="\_blank"} API. The object's line-delimited content is then split into individual events for processing. ```mermaid graph LR A["Timer
Trigger"] --> B["GetObject
API"] B --> C["Stream
Content"] C --> D["Split by
Newlines"] D --> E["Transform to
TenXObjects"] E --> F["Close
Route"] classDef trigger fill:#2563eb88,stroke:#1d4ed8,color:#ffffff classDef api fill:#ea580c88,stroke:#c2410c,color:#ffffff classDef process fill:#05966988,stroke:#047857,color:#ffffff class A trigger class B api class C,D,E process class F trigger ``` ### :material-key: Authentication The module supports the [AWS default credential provider chain](https://docs.aws.amazon.com/sdk-for-java/latest/developer-guide/credentials-chain.html){target="\_blank"}: 1. Environment variables (`AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`) 2. Java system properties 3. Web identity token (for EKS/IRSA) 4. Shared credentials file (`~/.aws/credentials`) 5. ECS container credentials 6. EC2 instance profile credentials Explicit credentials can also be provided via configuration: ```yaml s3Logs: - name: MyS3LogsInput bucket: my-bucket key: logs/app.log awsAccessKeyID: AKIAIOSFODNN7EXAMPLE awsSecretKey: wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY ``` ### :material-speedometer: Backpressure Controls Control resource usage with backpressure options: |Option|Description|Default| |---|---|---| |`totalBytesLimit`|Max bytes to read|50MB| |`totalEventsLimit`|Max events to read|10000| |`totalDuration`|Max time to keep input open|5min| ### :material-code-tags: Example Configuration ```yaml tenx: run include: run/modules/input/analyzer/s3Logs s3Logs: - name: ProductionLogs bucket: my-company-logs key: logs/2024/01/15/application.log awsRegion: us-west-2 totalBytesLimit: $=parseBytes("100MB") totalEventsLimit: 50000 printProgress: true ``` ### :material-link-variant: Related Modules - [CloudWatch Logs Input](https://doc.log10x.com/run/input/analyzer/cloudwatchLogs/ "Read events from AWS CloudWatch Logs") - Read from CloudWatch Logs - [Object Storage Query](https://doc.log10x.com/run/input/objectStorage/query/ "Query an object storage container for events matching target criteria") - Query-based object storage access ## :material-wrench-outline: Config Files To configure the AWS S3 Logs input module, [:material-cog: Edit](https://doc.log10x.com/config/app/#module-config "Learn how to edit app and module configurations") these files. Below is the default configuration from: [s3Logs/config.yaml](https://github.dev/log-10x/config/blob/main/pipelines/run/input/analyzer/s3Logs/config.yaml "s3Logs/config.yaml"){target="\_blank"} (** Required Fields*).
Edit Online
Edit config.yaml Locally
```yaml # 'run' AWS S3 Logs input configuration # Configure an AWS S3 Logs event input to read a single S3 object # To learn more see https://doc.log10x.com/run/input/analyzer/s3Logs/ # Set the 10x pipeline to 'run' tenx: run # =============================== Dependencies ================================ include: run/modules/input/analyzer/s3Logs # ============================== S3 Logs Options ============================== # Multiple S3 Logs inputs can be defined below s3Logs: # 'name' sets a unique logical name across all pipeline inputs - name: S3LogsInput # --------------------------- Connection Options -------------------------- # 'awsAccessKeyID' sets the AWS access key (e.g. 'AKIAIOSFODNN7EXAMPLE') # To learn more see: https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_access-keys.html awsAccessKeyID: $=TenXEnv.get("AWS_ACCESS_KEY_ID") # (EnvVar or default credential chain) # 'awsSecretKey' sets the AWS access secret key (e.g. 'wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY') # To learn more see: https://docs.aws.amazon.com/general/latest/gr/aws-sec-cred-types.html awsSecretKey: $=TenXEnv.get("AWS_SECRET_ACCESS_KEY") # (EnvVar or default credential chain) # 'awsRegion' sets the region hosting the S3 bucket # To learn more see: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-regions-availability-zones.html awsRegion: $=TenXEnv.get("AWS_DEFAULT_REGION", "us-east-1") # ---------------------------- S3 Object Options --------------------------- # 'bucket' sets the S3 bucket name (e.g., 'my-log-bucket'). # To learn more see: https://docs.aws.amazon.com/AmazonS3/latest/userguide/bucketnamingrules.html bucket: "" # (❗ REQUIRED) # 'key' sets the S3 object key to fetch (e.g., 'logs/app.log'). # To learn more see: https://docs.aws.amazon.com/AmazonS3/latest/API/API_GetObject.html key: "" # (❗ REQUIRED) # --------------------------- Backpressure Options ----------------------- # 'totalDuration' sets the max duration to try reading from the remote input totalDuration: $=parseDuration("5min") # 'totalBytesLimit' sets the max total bytes to read from the remote input totalBytesLimit: $=parseBytes("50MB") # 'totalEventsLimit' sets the max number of events to read the remote input totalEventsLimit: 10000 # --------------------------- Ancillary Options --------------------------- # 'printProgress' controls whether to print a gauge to the console # This option helps debug and test the input printProgress: $=!TenXEnv.get("quiet") ``` ## :material-menu: Options Specify the options below to [configure](/config "configure") multiple AWS S3 Logs input: |Name|Description|Category| |---|---|---| |[s3LogsName](#s3logsname "logical name for this AWS S3 input")|Logical name for this AWS S3 input|General| |[s3LogsEnabled](#s3logsenabled "sets whether this input is enabled")|Sets whether this input is enabled|General| |[s3LogsPrintProgress](#s3logsprintprogress "sets whether this input prints throughput stats to the console")|Sets whether this input prints throughput stats to the console|General| |[s3LogsAwsRegion](#s3logsawsregion "AWS region")|AWS region|Authentication| |[s3LogsAwsAccessKeyID](#s3logsawsaccesskeyid "AWS authentication access key ID")|AWS authentication access key ID|Authentication| |[s3LogsAwsSecretKey](#s3logsawssecretkey "AWS authentication secret key")|AWS authentication secret key|Authentication| |[s3LogsBucket](#s3logsbucket "AWS S3 bucket name")|AWS S3 bucket name|Object| |[s3LogsKey](#s3logskey "S3 object key to fetch")|S3 object key to fetch|Object| |[s3LogsTotalBytesLimit](#s3logstotalbyteslimit "maximum total bytes to read from input before closing")|Maximum total bytes to read from input before closing|Backpressure| |[s3LogsTotalEventsLimit](#s3logstotaleventslimit "maximum total events to read from input before closing")|Maximum total events to read from input before closing|Backpressure| |[s3LogsTotalDuration](#s3logstotalduration "maximum duration to keep input open before closing")|Maximum duration to keep input open before closing|Backpressure| ### General #### :material-menu-right-outline:**`s3LogsName`** Logical name for this AWS S3 input. |Type|Required|Category| |---|---|---| |String|✔|General| Sets a logical name (e.g., 'myS3Input') for this input. The [inputName](https://doc.log10x.com/api/js/#TenXBaseObject+inputName "Returns the context in which the current object, template or summary was created.") field returns this value at run time to allow for identifying and operating on instances originating from this input. #### :material-menu-right-outline:**`s3LogsEnabled`** Sets whether this input is enabled. |Type|Default|Category| |---|---|---| |Boolean|false|General| Sets whether to open the input stream. To enable this input only when an 's3LogsBucket' startup argument value is truthy, use: ```yaml s3LogsEnabled: $=TenXEnv.get("s3LogsBucket") ``` To learn more see [TenXEnv.get](https://doc.log10x.com/api/js/#TenXEnv.get). #### :material-menu-right-outline:**`s3LogsPrintProgress`** Sets whether this input prints throughput stats to the console. |Type|Default|Category| |---|---|---| |Boolean|false|General| Sets whether this input prints throughput stats to the console for testing an integration to a remote endpoint. ### Authentication #### :material-menu-right-outline:**`s3LogsAwsRegion`** AWS region. |Type|Default|Category| |---|---|---| |String|""|Authentication| Sets the AWS region (e.g., 'us-east-1') hosting the target S3 bucket. To learn more see [AWS Regions](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-regions-availability-zones.html){target="\_blank"}. #### :material-menu-right-outline:**`s3LogsAwsAccessKeyID`** AWS authentication access key ID. |Type|Default|Category| |---|---|---| |String|""|Authentication| Sets the AWS access key (e.g., 'AKIAIOSFODNN7EXAMPLE') used as part of the request signing process. **If not specified**, the system automatically falls back to the [AWS default credential provider chain](https://docs.aws.amazon.com/sdk-for-java/latest/developer-guide/credentials-chain.html){target="\_blank"}, which checks (in order): 1. Environment variables (`AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`) 2. Java system properties 3. Web identity token (for EKS) 4. Shared credentials file (`~/.aws/credentials`) 5. ECS container credentials 6. EC2 instance profile credentials To learn more see [AWS access keys](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_access-keys.html){target="\_blank"}. #### :material-menu-right-outline:**`s3LogsAwsSecretKey`** AWS authentication secret key. |Type|Default|Category| |---|---|---| |String|""|Authentication| Sets the AWS access secret key (e.g. 'wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY') used as part of the request signing process. **If not specified**, the system automatically falls back to the [AWS default credential provider chain](https://docs.aws.amazon.com/sdk-for-java/latest/developer-guide/credentials-chain.html){target="\_blank"}. This allows using credentials from `~/.aws/credentials`, IAM roles, or environment variables without explicit configuration. To learn more see [AWS security credentials](https://docs.aws.amazon.com/general/latest/gr/aws-sec-cred-types.html){target="\_blank"}. ### Object #### :material-menu-right-outline:**`s3LogsBucket`** AWS S3 bucket name. |Type|Required|Category| |---|---|---| |String|✔|Object| Sets the S3 bucket name (e.g., 'my-log-bucket') containing the object to read. To learn more see [S3 Bucket Naming](https://docs.aws.amazon.com/AmazonS3/latest/userguide/bucketnamingrules.html){target="\_blank"}. #### :material-menu-right-outline:**`s3LogsKey`** S3 object key to fetch. |Type|Required|Category| |---|---|---| |String|✔|Object| sets the S3 object key (e.g., 'logs/2024/01/15/app.log') to fetch. This input reads a single S3 object and transforms its line-delimited content into TenXObjects for analysis. For example: ```yaml s3Logs: - name: MyLogs bucket: my-bucket key: logs/application.log ``` ### Backpressure #### :material-menu-right-outline:**`s3LogsTotalBytesLimit`** Maximum total bytes to read from input before closing. |Type|Default|Category| |---|---|---| |String|50000000|Backpressure| sets the maximum number of bytes a target pipeline input will read into the pipeline. This value limits the volume of events to read from the S3 object. For example: ```yaml s3LogsTotalBytesLimit: $=parseBytes("1GB") ``` #### :material-menu-right-outline:**`s3LogsTotalEventsLimit`** Maximum total events to read from input before closing. |Type|Default|Category| |---|---|---| |String|10000|Backpressure| Sets the maximum number of events a target pipeline input will read into the pipeline. This value limits the volume of events to read from the S3 object. #### :material-menu-right-outline:**`s3LogsTotalDuration`** Maximum duration to keep input open before closing. |Type|Default|Category| |---|---|---| |String|parseDuration("5min")|Backpressure| sets the maximum duration a target pipeline input will remain open. When reached, the input will close and no more data will be read. For example: ```yaml s3LogsTotalDuration: $=parseDuration("10min") ```
:material-github: This module is defined in [s3Logs/module.yaml](https://github.com/log-10x/modules/tree/main/pipelines/run/modules/input/analyzer/s3Logs/module.yaml "s3Logs/module.yaml"){target="\_blank"}.