--- icon: material/lambda --- Deploy the [Retriever](../) app to AWS Lambda with the [terraform-aws-tenx-retriever-lambda](https://registry.terraform.io/modules/log-10x/tenx-retriever-lambda/aws){target="\_blank"} module. ???+ tenx-bootstrap "Step 1: Prerequisites" | Requirement | Description | |-------------|-------------| | 10x License | Your license key ([get one](https://doc.log10x.com/run/bootstrap/#apikey)) | | AWS CLI | Configured with credentials that can create Lambda, SQS, IAM, and S3 notifications | | Terraform | [Terraform](https://www.terraform.io/downloads){target="\_blank"} >= 1.5 | | Docker | For mirroring the published image into ECR | | ECR Repository | Any ECR repo in the same AWS account and region as the Lambda. You'll push the retriever image here | | S3 Bucket(s) | One for raw log uploads (source), one for index artifacts (can be the same bucket) | ??? tenx-cloud "Step 2: Mirror the Published Image into ECR" The retriever ships as a prebuilt Lambda container image, published to Docker Hub and GHCR. There is nothing to build: ``` docker.io/log10x/lambda-10x:1.1.69-native ghcr.io/log-10x/lambda-10x:1.1.69-native ``` Tags track the engine release. Pin one, never `:latest`, which the module rejects at plan time. Two packagings ship per release: | Tag | Payload | Architectures | Cold start | |---|---|---|---| | `-native` (default) | GraalVM binary on `provided:al2023` | `x86_64` only | sub-second | | `` (jvm) | shadow jar on `lambda/java:21` | multi-arch manifest | seconds | The module's `lambda_packaging` input defaults to `native` and is cross-checked against the tag suffix at plan time, so the `-native` tag is the one that pairs with the defaults. For the JVM packaging, mirror the version-only tag and set `lambda_packaging = "jvm"`. Lambda pulls container images only from a private ECR repository in the same AWS account as the function, so mirror the tag you picked: ```bash REGION=us-east-1 ACCOUNT=$(aws sts get-caller-identity --query Account --output text) TAG=1.1.69-native aws ecr create-repository --repository-name lambda-10x --region $REGION # --platform linux/amd64 matters on an arm64 workstation: the native tag has # no arm64 variant to fall back to, and a silent emulated pull would push an # image Lambda cannot run. docker pull --platform linux/amd64 docker.io/log10x/lambda-10x:${TAG} docker tag docker.io/log10x/lambda-10x:${TAG} \ ${ACCOUNT}.dkr.ecr.${REGION}.amazonaws.com/lambda-10x:${TAG} aws ecr get-login-password --region $REGION \ | docker login --username AWS --password-stdin ${ACCOUNT}.dkr.ecr.${REGION}.amazonaws.com docker push ${ACCOUNT}.dkr.ecr.${REGION}.amazonaws.com/lambda-10x:${TAG} ``` The resulting private-ECR URI is what you pass to the module as `image_uri`. ??? tenx-config "Step 3: Configure Terraform" ``` { .hcl title="main.tf"} provider "aws" { region = "us-east-1" } data "aws_caller_identity" "current" {} module "retriever" { source = "log-10x/tenx-retriever-lambda/aws" version = "~> 3.0" name_prefix = "my-retriever" # Private ECR in the same AWS account as the Lambda, from Step 2. # `lambda_packaging` defaults to "native", so the tag carries the # -native suffix and the plan-time packaging check passes. For the # JVM image, set lambda_packaging = "jvm" and drop the suffix. image_uri = "${data.aws_caller_identity.current.account_id}.dkr.ecr.us-east-1.amazonaws.com/lambda-10x:1.1.69-native" # Bring-your-own buckets. Source holds raw logs; index holds the bloom/reverse artifacts. # Can be the same bucket (EKS-style single-bucket layout). source_bucket_name = "my-raw-log-bucket" index_bucket_name = "my-raw-log-bucket" # Scope the S3 event trigger to the prefix/suffix where uploads land, # so the indexer's own writes cannot re-trigger the indexer via the # S3 notification (recursive-invocation loop). When # source_bucket_name == index_bucket_name the module also requires a # non-empty `index_bucket_path`, which defaults to "indexing-results/". source_prefix = "app/" source_suffix = ".log" tenx_api_key = var.tenx_api_key tags = { Environment = "production" ManagedBy = "terraform" } } variable "tenx_api_key" { type = string sensitive = true } output "query_function_url" { value = module.retriever.query_function_url } ``` Apply: ```bash terraform init terraform apply -var="tenx_api_key=YOUR_API_KEY" ``` ??? tenx-objectstorageindex "Step 4: Verify Indexing" Upload a test file matching the trigger prefix/suffix: ```bash echo "{\"timestamp\":\"$(date -u +%Y-%m-%dT%H:%M:%SZ)\",\"level\":\"ERROR\",\"message\":\"Test error\"}" > test.log aws s3 cp test.log s3://my-raw-log-bucket/app/test.log ``` Within 10-20 s the indexer Lambda fires and writes bloom + reverse-index artifacts under `tenx/` keys in the index bucket: ```bash aws s3 ls s3://my-raw-log-bucket/tenx/ --recursive ``` Check CloudWatch Logs for the indexer Lambda (`/aws/lambda/my-retriever-indexer`): ``` INFO IndexFilterStats - index complete. Bytes: 21, filters.size: 1, ... INFO ExecutionPipeline - execution of pipeline.yaml (myObjectStorageIndex) completed in: 7501ms ``` ??? tenx-objectstoragequery "Step 5: Verify Querying" Submit a query against the Function URL. A Function URL invokes its Lambda on every path, and the handler selects its role from the `ROLE` environment variable rather than from the request path, so the path carries no meaning here: `POST` to the URL root. `/streamer/query` is the Quarkus retriever's HTTP route and is reserved for it. The in-engine app path is `@apps/retriever/query`. Request body fields are `processingTimeMs`, `resultSizeBytes`, and `name`. Requests are signed with SigV4 when `query_url_auth = "AWS_IAM"` (default): ```bash URL=$(terraform output -raw query_function_url) awscurl --service lambda --region us-east-1 -X POST \ "$URL" \ -H "Content-Type: application/json" \ -d '{"from": "now(\"-5m\")", "to": "now()", "search": "severity_level == \"ERROR\"", "name": "my_query"}' ``` The query Lambda writes `_DONE.json` on completion: ```bash aws s3 ls s3://my-raw-log-bucket/tenx/my-raw-log-bucket/qr/ --recursive | grep _DONE.json ``` Matched events are visible in the stream Lambda's CloudWatch log group (`/aws/lambda/my-retriever-stream`). Fluent Bit is not used in the Lambda deployment. The pipeline emits directly from the stream worker. ??? tenx-objectstoragequery-output "Step 6: Front with API Gateway (Optional)" The built-in Function URL is the fastest path to a public query endpoint. If you need custom auth, routing, or rate limiting, disable it and front with API Gateway: ```hcl module "retriever" { # ... other inputs enable_query_url = false } resource "aws_apigatewayv2_api" "query" { name = "tenx-retriever-query" protocol_type = "HTTP" } resource "aws_apigatewayv2_integration" "query" { api_id = aws_apigatewayv2_api.query.id integration_type = "AWS_PROXY" integration_uri = module.retriever.lambda_function_arns["query"] payload_format_version = "2.0" } ``` ??? tenx-monitoring "Step 7: Monitor Operations" Useful CloudWatch alarms: | Metric | Source | Threshold | Meaning | |---|---|---|---| | `ApproximateAgeOfOldestMessage` | SQS `*-index-queue` | > 300 s | Indexing is falling behind. Concurrency limit? | | `ApproximateNumberOfMessages` | SQS `*-index-queue-dlq` | > 0 | Failed indexing messages. Inspect DLQ body | | `Errors` | Lambda `*-query` | > 0 | Query handler crashes | | `Duration` p95 | Lambda `*-query` | > 8 s | Cold starts dominating. Consider provisioned concurrency | Inspect a DLQ message: ```bash aws sqs receive-message \ --queue-url $(terraform output -raw dlq_urls | jq -r '."index"') ``` ??? tenx-delete "Step 8: Teardown" ```bash terraform destroy -var="tenx_api_key=YOUR_API_KEY" ``` The module removes the Lambdas, queues, DLQs, IAM role, and S3 event notification. S3 buckets and the ECR image are left alone (you own them). To remove indexed artifacts: ```bash aws s3 rm s3://my-raw-log-bucket/tenx/ --recursive ``` ## :material-cube-outline: Components What `terraform apply` creates. Tabs describe each component and what it's for. === ":material-application-outline: Compute" Four Lambda functions run from a single container image. The `ROLE` environment variable tells each one which role to play. - `indexer` runs once per log-file upload. It builds the Bloom filter and reverse-index, then writes both back to S3. - `query` is the public entry point. It receives a query, picks the sub-windows to scan, and fans them out. - `subquery` scans one time-slice of the Bloom index. It finds matching byte-ranges and fans those out to `stream`. - `stream` fetches matched byte-ranges from S3, decodes the events, and emits them to the log analyzer. === ":material-inbox-multiple-outline: Queues" Three SQS queues (`index`, `subquery`, `stream`) buffer work between roles. Each has a dead-letter queue. When a consumer Lambda fails `max_receive_count` times on a message, the message moves to the DLQ instead of being silently dropped. DLQs retain messages for 14 days so you can inspect failures. === ":material-shield-lock-outline: IAM" One role shared by all four Lambdas. It grants S3 read and write on the source and index buckets, SQS send on the three queues, and CloudWatch Logs access for diagnostics. The role also includes `s3:PutObjectTagging`. The engine tags its bloom-filter writes with S3 object tags. Without this permission, tagged PUTs silently return HTTP 400 and bloom filters never land. === ":material-flash-outline: Triggers" The source bucket's `ObjectCreated:*` notification sends messages to the index queue, and the index queue invokes the indexer. The notification is scoped by `source_prefix` and `source_suffix` so the indexer's own writes under `tenx/` don't re-trigger it. A `terraform plan` precondition refuses any configuration that would create a recursive loop. Three Lambda event-source mappings connect each SQS queue to its consumer. === ":material-api: HTTP entry" A Lambda Function URL on the `query` Lambda accepts `POST` on any path. It is cheaper than API Gateway and simpler to wire up. Callers must sign requests with SigV4 (`AWS_IAM`) by default. Set `enable_query_url = false` to front the query Lambda with API Gateway instead, or `query_url_auth = "NONE"` to make the endpoint publicly invocable for demos. ## :material-speedometer: Performance Measured against retriever v1.0 (engine merge `a32bd0a3`) on AWS `us-east-1`, Lambda x86_64 at 6144 MB, April 2026. Corpus: `otel-sample` JSON-lines files, ~21 MB each. Queries target a single file's byte-range with a simple severity filter. Figures are wall-clock from the caller's perspective: HTTP POST to last event in the analyzer for queries, S3 PUT to bloom-filter-written for indexing. === ":material-timer-outline: Query latency" | Condition | p50 | p95 | |---|---|---| | Warm | 1.2 s | 1.4 s | | Cold | 6.7 s | 10 s | Enable [Provisioned Concurrency](https://docs.aws.amazon.com/lambda/latest/dg/provisioned-concurrency.html){target="\_blank"} on `query` and `stream` to eliminate cold starts. 3–5 warm instances per Lambda at ~$15/month each cover ~1 query/min. === ":material-timer-outline: Indexing latency" | File size | p50 | p95 | Throughput | |---|---|---|---| | 21 MB | 15.4 s | 18.3 s | 1.4 MB/s | Fixed ~5 s bootstrap + linear work phase. === ":material-chart-line: Sizing projection" | File size | Expected p50 | Dominant cost | |---|---|---| | 1 MB | 5–7 s | Bootstrap | | 21 MB | 15 s | Measured | | 100 MB | 60–80 s | Work phase | For files under 5 MB, batch producer-side before S3 PUT. One 20 MB file indexes ~3x faster than twenty 1 MB files serially. ## :material-currency-usd: Cost Unit rates (Lambda at 6144 MB, us-east-1). Multiply by your offloaded volume. | Line item | Rate | |---|---| | Indexer compute | ~$0.074 per GB of logs indexed | | Query chain compute (warm) | ~$0.0005 per query | | S3 storage | $0.023 per GB-month (Standard) | | SQS messages | $0.40 per million (one per indexed file) | Scales roughly linearly by daily offloaded volume (what lands in the bucket, not total ingest). AWS Lambda default account concurrency (1000) covers up to ~100 TB/day of offload. For mode-vs-mode and market comparisons, see the [cost table on the picker page](index.md#cost-at-1-tbday). ## :material-tune-variant: Tunables | Variable | Default | Effect | |---|---|---| | `memory_size` | 6144 | CPU scales linearly with memory. 6144 MB is measured optimal. 10240 MB plateaus. Lower memory is dramatically slower. | | `pipeline_shutdown_grace_ms` | 250 | How long the engine waits for sequencer queues to drain on pipeline close. The engine default of 5000 adds a flat 5 s to every warm Lambda invocation, because sequencer queues are already empty at close time in a single-shot invocation. 250 ms bounds the wait safely. Override upward only if you observe dropped events on a long-running workload (EKS). | | `indexer_batch_size` | 1 | SQS batch size for the indexer. A batch of 1 is safest (ordered, no redelivery). Increase it to trade latency for throughput under backlog. | | `enable_query_url` | true | Whether to create the Lambda Function URL that exposes the `query` Lambda over HTTPS. Set to false if you front the query Lambda with API Gateway instead. | | `query_url_auth` | `AWS_IAM` | Function URL auth mode. `AWS_IAM` requires SigV4-signed requests. `NONE` makes the URL publicly invocable (use only for demos). | See the [module README](https://github.com/log-10x/terraform-aws-tenx-retriever-lambda/tree/v3.0.0){target="\_blank"} for the full input and output reference.