--- title: "Object Storage Query Input" description: "query an object storage container for events matching target criteria" source: "https://github.com/log-10x/modules/tree/main/pipelines/run/modules/input/objectStorage/query/module.yaml" icon: "octicons/filter-24" --- Defines a search query against an object storage container (e.g., AWS S3 bucket) [index](https://doc.log10x.com/run/input/objectStorage/index "Index files uploaded to a object storage container (e.g. AWS S3 bucket)."). Queries can execute periodically (e.g., k8s CronJob) or ad-demand via the [REST API](https://doc.log10x.com/api/launch/#quarkus) to populate log analytics dashboards, alerts, and ad-hoc queries. Each query specifies _criteria_ and _actions_. **Criteria** select events to fetch and transform into [TenXObjects](https://doc.log10x.com/api/js/#tenxobject): - [**Target**](#queryfiltertarget): target app/service whose events to query (e.g., 'myApp'). - [**Time range**](#queryfrom "beginning of the search time range (inclusive)"): of log/trace events to fetch. - [**Terms**](#querysearch "list of terms to search for"): query to execute against the underling storage - [**Filters**](#queryfilters "list of terms to search for"): list of JavaScript expressions which allows for more complex filtering of which TenXObjects matching the search terms to select. **Actions** specify how to further filter, [enrich](https://doc.log10x.com/run/transform/script/object/#enrich), aggregate, and stream TenXObjects to log analyzers and metric outputs (e.g., AWS CloudWatch, Prometheus). ## :material-floor-plan: Architecture A distributed [Stream Processing Architecture](https://docs.aws.amazon.com/whitepapers/latest/build-modern-data-streaming-analytics-architectures/what-is-a-modern-streaming-data-architecture.html){target="\_blank"} executes queries via parallel [scan](#scan) and [stream](#stream) workers to select, transform and stream TenXObjects to target output(s). ### :material-filter-variant: Scan Scan workers process [TenXTemplate Filters](https://doc.log10x.com/run/input/objectStorage/index/#tenxtemplate-filters) to identify blobs matching a target prefix (i.e., app/service name), time frame (e.g., last 10min) and search terms (e.g., ERROR). Multiple scan workers perform this workload in parallel, submitting matching byte ranges to stream workers via SQS. ### :material-set-split: Stream Stream workers read object storage blob byte ranges identified by the scan workers and transform them into TenXObjects on which they perform the [actions](#queryactions) specified by the query. ### :material-counter: Summaries Stream workers optionally write per-pattern rollups to a parallel S3 prefix `qrs/{queryId}/{sliceFrom}_{sliceTo}/{worker}.jsonl`, alongside or instead of raw events at `qr/{queryId}/{sliceFrom}_{sliceTo}/{worker}.jsonl`. Each summary record carries `summaryVolume` (event count), `summaryBytes` (UTF-8 byte total), and the named enrichment fields the pipeline initialized. Toggle via [`queryWriteSummaries`](#querywritesummaries "sets whether per-slice TenXSummary aggregates are buffered to JSONL and uploaded") on the query module, or by setting `writeSummaries: true` on the `POST /retriever/query` REST request. Raw events (`qr/`) carry forensic payloads; summaries (`qrs/`) keep counts exact across all matched events instead of capped per worker. ## :material-sitemap-outline: Architecture Flow The following diagram illustrates the complete distributed, parallel architecture for query execution: ```mermaid graph TD Client["🔍 Query Request
API /search"] --> S3["☁️ S3 Object Storage
Log Files + Bloom Filters"] S3 --> ScanLayer subgraph ScanLayer["⚡ Parallel Scan Functions
"] Scan1["Scan #1
Time: 00:00-05:00
• Filter prefix
• Test Bloom filters
• Find byte ranges"] Scan2["Scan #2
Time: 05:00-10:00
• Filter prefix
• Test Bloom filters
• Find byte ranges"] Scan3["Scan #3
Time: 10:00-15:00
• Filter prefix
• Test Bloom filters
• Find byte ranges"] ScanN["Scan #N
Time: N+5:00
• Filter prefix
• Test Bloom filters
• Find byte ranges"] end ScanLayer --> StreamLayer subgraph StreamLayer["🌊 Parallel Stream Functions
"] Stream1["Stream #1
• Fetch ranges
• Parse logs
• Create TenXObjects
• Filter & enrich"] Stream2["Stream #2
• Fetch ranges
• Parse logs
• Create TenXObjects
• Filter & enrich"] Stream3["Stream #3
• Fetch ranges
• Parse logs
• Create TenXObjects
• Filter & enrich"] StreamM["Stream #M
• Fetch ranges
• Parse logs
• Create TenXObjects
• Filter & enrich"] end StreamLayer --> OutputLayer subgraph OutputLayer["📊 Output Targets
"] Elastic["Elasticsearch
Events & Logs"] Prometheus["Prometheus
Metrics & Gauges"] Datadog["Datadog
Custom Metrics"] Splunk["Splunk
Search Indexes"] end %% Elegant dark theme styling classDef default fill:#2d3748,stroke:#4a5568,stroke-width:2px,color:#e2e8f0 classDef client fill:#2b6cb0,stroke:#3182ce,stroke-width:2px,color:#ffffff classDef storage fill:#d69e2e,stroke:#ed8936,stroke-width:2px,color:#ffffff classDef scan fill:#38a169,stroke:#48bb78,stroke-width:2px,color:#ffffff classDef stream fill:#d53f8c,stroke:#ed64a6,stroke-width:2px,color:#ffffff classDef output fill:#805ad5,stroke:#9f7aea,stroke-width:2px,color:#ffffff class Client client class S3 storage class Scan1,Scan2,Scan3,ScanN scan class Stream1,Stream2,Stream3,StreamM stream class Elastic,Prometheus,Datadog,Splunk output ``` ### :material-check: Benefits - **Scalability**: Functions auto-scale based on query complexity and data volume - **Parallelism**: Time-based partitioning enables concurrent processing across multiple workers - **Efficiency**: Bloom filters minimize unnecessary data reads from object storage - **Flexibility**: Multiple output targets support different analytics use cases (logs, metrics, alerts) - **Cost-Effective**: SQS-based execution model scales with demand ## :material-speedometer: Performance Index lookups identify matching files in under 1 second. Fetching and streaming events depends on result set size and parallel worker configuration. ### Baseline - **~10K events:** 10–30 seconds with default parallel scan/stream configuration - **Index filters:** Identify matching files in \<1 second ([accuracy](https://doc.log10x.com/run/input/objectStorage/index/#accuracy) is configurable, default ~1% false positive rate) - **Network limits:** S3 API throughput and network bandwidth are the practical limits at scale ### Parallelism Parallel [scan](#scan) and [stream](#stream) workers fetch and parse events concurrently. Three configuration options control parallelism: - [`queryScanFunctionParallelTimeslice`](#queryscanfunctionparalleltimeslice "max time range to process by a scan worker instance"), max time range per scan worker (e.g., `1m` = each worker scans 1 minute of index) - [`queryScanFunctionParallelMaxInstances`](#queryscanfunctionparallelmaxinstances "max number of instances of the scan worker"), max number of parallel scan workers (default 1000) - [`queryStreamFunctionParallelObjects`](#querystreamfunctionparallelobjects "max number of input object byte ranges to process in a single 'stream' instance"), max byte ranges per stream worker (default 50) **Example:** 100K events over a 10-minute time range with a 1-minute timeslice produces ~10 parallel scan workers. Total query time approaches the single-worker baseline (~10–30 sec) rather than scaling linearly with result set size. Actual time depends on file size distribution, S3 API rate limits, network bandwidth, and log parsing overhead. ### Optimization Tips - Use narrow [time ranges](#queryfrom "beginning of the search time range (inclusive)") to reduce scan volume - Add service/host filters to narrow scope - For massive result sets (\>10M events), split into smaller time windows - Use [query limits](#querylimitprocessingtime "max milliseconds alloted for this query") to prevent runaway queries ## :material-wrench-outline: Config Files To configure the Object storage query input module, [:material-cog: Edit](https://doc.log10x.com/config/app/#module-config "Learn how to edit app and module configurations") these files. Below is the default configuration from: [query/config.yaml](https://github.dev/log-10x/config/blob/main/pipelines/run/input/objectStorage/query/config.yaml "query/config.yaml"){target="\_blank"} ().

Edit Online

```yaml # 🔟❎ 'run' Object storage query AWS S3 configuration # Configure query AWS S3 bucket settings # To learn more see https://doc.log10x.com/run/input/objectStorage/query/ # Set the 10x pipeline to 'run' tenx: run # =============================== Dependencies ================================ include: - run/modules/input/objectStorage/input - run/modules/input/objectStorage/query # =============================== Query Options =============================== query: # 'name' specifies a logical name associated with this query - name: my-query # 'readPrintProgress' sets whether this input prints throughput stats to the console readPrintProgress: $=!TenXEnv.get("quiet") # ---------------------------- Storage Options ---------------------------- # 'objectStorage' selects the underlying storage (e.g., S3). To learn more see: https://doc.log10x.com/run/input/objectStorage # 'objectStorageName' selects the storage accessor alias registered by 'objectStorageClassName' (AWS or Azure). # Set TENX_OBJECT_STORAGE_NAME to Azure to target Azure Blob Storage. Defaults to AWS. # AWS authentication is made via the default provider credentials chain: # https://sdk.amazonaws.com/java/api/latest/software/amazon/awssdk/auth/credentials/DefaultCredentialsProvider.html # AWS region selection is made via the default region provider chain: # https://sdk.amazonaws.com/java/api/latest/software/amazon/awssdk/regions/providers/DefaultAwsRegionProviderChain.html # Azure resolves the storage account from the first segment of 'indexContainer' (account/container/path), # or from AZURE_STORAGE_ACCOUNT when 'indexContainer' is container/path. 'readContainer' is a bare container name. objectStorageName: $=TenXEnv.get("TENX_OBJECT_STORAGE_NAME", "AWS") # 'readContainer' specifies the container (e.g., S3 bucket, Azure Blob container) in which to search readContainer: $=TenXEnv.get("TENX_STREAMER_INPUT_BUCKET", "") # (❗ REQUIRED) # 'indexContainer' specifies the container (e.g., S3 bucket, Azure account/container/path) in which index objects created by the 'index' module reside. indexContainer: $=TenXEnv.get("TENX_STREAMER_INDEX_BUCKET", "") # (❗ REQUIRED) # ----------------------------- Search Options ---------------------------- # 'target' specifies a user app/service name (e.g., 'acme-client') to search in. target: app # (❗ REQUIRED) # 'from' sets the query's beginning epoch time range (inclusive). # The 'now' function returns the current epoch value with an optional positive/negative offset. from: now("-5m") # (❗ REQUIRED) # 'to' sets the end epoch time range (exclusive), must be greater than 'from'. to: now() # (❗ REQUIRED) # 'search' Specifies a search expression applied to raw events in the underlying storage # Supports basic operations: `&&` (AND), `||` (OR), == (equality), and includes(field, text) (substring match). search: (severity_level == "ERROR") || (severity_level == "FATAL") # 'id' specifies a unique query identifier. Automatically assigned if not set. id: "" # 'logLevels' specifies which log levels are sent to CloudWatch Logs for query progress tracking. # Default (when empty): ERROR, INFO, PERF. Set to include DEBUG for diagnostic output. # NOTE: PERF is required for console progress tracking (gauges, completion detection). # Disabling it will cause the console to not detect query completion. logLevels: "" # 'logGroup' specifies the CloudWatch Logs log group for query event logging. # The log group must be pre-created (e.g. by Terraform). If empty, query event logging is disabled. logGroup: $=TenXEnv.get("TENX_QUERY_LOG_GROUP", "") # 'writeResults' gates the stream pipeline's JSONL results writer, which # buffers matched TenXObjects to object storage under # `{indexObjectPath(queryResults)}/{queryId}/{sliceFrom}_{sliceTo}/*.jsonl` # alongside the byte-count marker backstop used by the query coordinator. # Defaults to false so the extra JSONL upload only runs when a caller # explicitly opts in. Out-of-band consumers like the Log10x MCP server # pass `writeResults=true` on their query request to retrieve full event # payloads from S3; other deployments leave this off to avoid the # results-writer overhead. writeResults: false # 'writeSummaries' gates the stream pipeline's JSONL summaries writer # (TenXSummary records produced by run/aggregate/stream). Defaults to # false. When true, per-slice summaries are buffered to object storage # under `{indexObjectPath(querySummaries)}/{queryId}/{sliceFrom}_{sliceTo}/*.jsonl` # so consumers can fetch a low-volume per-slice rollup (count, bytesum, # grouping fields) for high-cardinality patterns instead of pulling # raw events. Independent of `writeResults`; either can be on in # isolation or both together. writeSummaries: false # 'filters' specify expressions TenXObjects transformed from matching events in storage must meet. # The `$?queryFilters` macro lets the CLI/REST `queryFilters` override # plumb through to the worker pipeline. With ACCEPT_SINGLE_VALUE_AS_ARRAY # in the engine's Jackson config, a single-string override value also # hydrates into the List queryFilters field correctly. # To learn more see https://doc.log10x.com/run/output/receive/#filter-expressions filters: $?queryFilters # ----------------------------- Limit Options ------------------------------ # 'limit' options caps the query's processing time and volume of results it will return limit: # 'processingTime' specifies the time in ms allotted for this query to complete. # Once this period has elapsed the query will terminate regardless of # whether 'resultSize' has not been exceeded. processingTime: parseDuration("1m") # 'resultSize' specifies the volume in bytes of events matching the target app name, terms and time range to fetch. # Once this volume has been reached the query will terminate. resultSize: parseBytes("10MB") ``` ## :material-menu: Options Specify the options below to [configure](/config "configure") multiple Object storage query input: |Name|Description|Category| |---|---|---| |[queryName](#queryname "logical name for the query")|Logical name for the query|General| |[queryReadPrintProgress](#queryreadprintprogress "sets whether this input prints throughput stats to the console")|Sets whether this input prints throughput stats to the console|General| |[queryLogLevels](#queryloglevels "log levels to send to CloudWatch Logs for query progress tracking")|Log levels to send to CloudWatch Logs for query progress tracking|General| |[queryLogGroup](#queryloggroup "CloudWatch Logs log group for query event logging")|CloudWatch Logs log group for query event logging|General| |[queryWriteResults](#querywriteresults "sets whether matched events are buffered to JSONL and uploaded to object storage")|Sets whether matched events are buffered to JSONL and uploaded to object storage|General| |[queryWriteSummaries](#querywritesummaries "sets whether per-slice TenXSummary aggregates are buffered to JSONL and uploaded")|Sets whether per-slice TenXSummary aggregates are buffered to JSONL and uploaded|General| |[queryTarget](#querytarget "logical prefix to filter index objects by")|Logical prefix to filter index objects by|Filter| |[queryResultTarget](#queryresulttarget "optional Tier-1 result-sink redirect prefix for query OUTPUT")|Optional Tier-1 result-sink redirect prefix for query OUTPUT|Filter| |[querySearch](#querysearch "list of terms to search for")|List of terms to search for|Filter| |[queryFilters](#queryfilters "list of terms to search for")|List of terms to search for|Filter| |[queryFrom](#queryfrom "beginning of the search time range (inclusive)")|Beginning of the search time range (inclusive)|Filter| |[queryTo](#queryto "end of the search time range (exclusive)")|End of the search time range (exclusive)|Filter| |[queryObjectStorageName](#queryobjectstoragename "object storage logical name")|Object storage logical name|Access| |[queryIndexContainer](#queryindexcontainer "Object storage container containing index objects")|Object storage container containing index objects|Scan| |[queryScanFunctionParallelTimeslice](#queryscanfunctionparalleltimeslice "max time range to process by a scan worker instance")|Max time range to process by a scan worker instance|Scan| |[queryScanFunctionParallelMaxInstances](#queryscanfunctionparallelmaxinstances "max number of instances of the scan worker")|Max number of instances of the scan worker|Scan| |[queryScanFunctionParallelThreads](#queryscanfunctionparallelthreads "max number of threads allocated per each 'scan' worker instance")|Max number of threads allocated per each 'scan' worker instance|Scan| |[queryScanFlushInterval](#queryscanflushinterval "interval to wait before posting pending 'stream' worker requests")|Interval to wait before posting pending 'stream' worker requests|Scan| |[queryScanFunctionUrl](#queryscanfunctionurl "scan worker endpoint URL")|Scan worker endpoint URL|Scan| |[queryReadContainer](#queryreadcontainer "location of the output file for writing TenXObject field values")|Location of the output file for writing TenXObject field values|Stream| |[queryStreamFunctionParallelObjects](#querystreamfunctionparallelobjects "max number of input object byte ranges to process in a single 'stream' instance")|Max number of input object byte ranges to process in a single 'stream' instance|Stream| |[queryStreamFunctionParallelByteRange](#querystreamfunctionparallelbyterange "max input object bytes for a single fetch request")|Max input object bytes for a single fetch request|Stream| |[queryStreamFunctionUrl](#querystreamfunctionurl "stream worker endpoint URL")|Stream worker endpoint URL|Stream| |[queryObjectStorageArgs](#queryobjectstorageargs "custom object storage args")|Custom object storage args|Advanced| |[queryFilterVars](#queryfiltervars "specifies a list of variable values to scan for")|Specifies a list of variable values to scan for|Advanced| |[queryFilterTemplateHashes](#queryfiltertemplatehashes "specifies a list of TenXTemplate hash values to scan for")|Specifies a list of TenXTemplate hash values to scan for|Advanced| |[queryId](#queryid "unique ID of host query")|Unique ID of host query|Advanced| |[queryElapseTime](#queryelapsetime "epoch after which processing times out")|Epoch after which processing times out|Advanced| |[queryScanFunctionLimitResultSizeInterval](#queryscanfunctionlimitresultsizeinterval "interval by which to check whether 'queryLimitResultSize' is exceeded")|Interval by which to check whether 'queryLimitResultSize' is exceeded|Advanced| |[queryLimitProcessingTime](#querylimitprocessingtime "max milliseconds alloted for this query")|Max milliseconds alloted for this query|Limit| |[queryLimitResultSize](#querylimitresultsize "max byte volume of matching events to fetch")|Max byte volume of matching events to fetch|Limit| ### General #### :material-menu-right-outline:**`queryName`** Logical name for the query. |Type|Default|Category| |---|---|---| |String|""|General| Specifies a logical name associated with the query (e.g., `errorsQuery`) This value identifies this query operation in aggregated metrics reports. If not specified, defaults to a concatenation of [queryTarget](https://doc.log10x.com/run/input/objectStorage/query/#querytarget "logical prefix to filter index objects by") and [querySearch](https://doc.log10x.com/run/input/objectStorage/query/#querysearchs). #### :material-menu-right-outline:**`queryReadPrintProgress`** Sets whether this input prints throughput stats to the console. |Type|Default|Category| |---|---|---| |Boolean|false|General| Sets whether this input prints throughput stats to the console This value is commonly used when testing an integration to a remote endpoint. #### :material-menu-right-outline:**`queryLogLevels`** Log levels to send to CloudWatch Logs for query progress tracking. |Type|Default|Category| |---|---|---| |String|""|General| Specifies which query event log levels are sent to CloudWatch Logs for tracking query progress. Valid values: `ERROR`, `INFO`, `DEBUG`, `PERF`. If not specified, defaults to `ERROR`, `INFO`, `PERF`. **Note:** `PERF` is required for console progress tracking (gauges, completion detection). Disabling it will cause the console to not detect query completion. ```yaml queryLogLevels: ERROR,INFO,PERF ``` #### :material-menu-right-outline:**`queryLogGroup`** CloudWatch Logs log group for query event logging. |Type|Default|Category| |---|---|---| |String|""|General| Specifies the CloudWatch Logs log group name for logging query lifecycle events. The log group must be pre-created (e.g. by Terraform). If not specified or empty, query event logging is disabled. ```yaml queryLogGroup: /log10x/query ``` #### :material-menu-right-outline:**`queryWriteResults`** Sets whether matched events are buffered to JSONL and uploaded to object storage. |Type|Default|Category| |---|---|---| |Boolean|false|General| Controls whether the stream pipeline launches the JSONL results writer in addition to the byte-count marker backstop used by the query coordinator. Defaults to `false` so the extra JSONL upload only runs when a caller explicitly opts in. The query coordinator forwards this flag to each stream worker it dispatches via a `queryObjectWriteResults=` bootstrap arg on the stream pipeline launch. Out-of-band consumers like the Log10x MCP server set this to `true` so they can retrieve full event payloads from S3 under `{indexObjectPath(queryResults)}/{queryId}/{sliceFrom}_{sliceTo}/*.jsonl`. Deployments that only rely on the coordinator's numeric-key poll leave this off. ```yaml queryWriteResults: true ``` #### :material-menu-right-outline:**`queryWriteSummaries`** Sets whether per-slice TenXSummary aggregates are buffered to JSONL and uploaded. |Type|Default|Category| |---|---|---| |Boolean|false|General| Controls whether the stream pipeline launches a TenXSummary-mode results writer in addition to the events writer (gated by `queryWriteResults`) and the byte-count marker backstop. Defaults to `false`. The query coordinator forwards this flag to each stream worker it dispatches via a `queryObjectWriteSummaries=` bootstrap arg on the stream pipeline launch. When set to `true`, per-slice aggregated summaries (count, bytesum, grouping fields produced by `run/aggregate/stream`) are written to `{indexObjectPath(querySummaries)}/{queryId}/{sliceFrom}_{sliceTo}/*.jsonl`. Consumers (e.g. log10x\_pattern\_trend, log10x\_top\_patterns) can poll these per-slice summaries to render time-spread results for high-cardinality patterns without paying the bandwidth cost of full event JSONL. Independent of `queryWriteResults`; either can be on in isolation or both together. ```yaml queryWriteSummaries: true ``` ### Filter #### :material-menu-right-outline:**`queryTarget`** Logical prefix to filter index objects by. |Type|Required|Category| |---|---|---| |String|✔|Filter| Specifies a logical name associated with the search target. This value identifies the app that produced the queried events (e.g., `acme-client`). This value corresponds with the [indexWriteTarget](https://doc.log10x.com/run/input/objectStorage/index/#indexwritetarget "logical name identifying the origin of 'indexReadObject'") argument. #### :material-menu-right-outline:**`queryResultTarget`** Optional Tier-1 result-sink redirect prefix for query OUTPUT. |Type|Default|Category| |---|---|---| |String|""|Filter| Optional bare-token prefix under which query RESULT objects (`qr/`, `qrs/`, `q/` markers, `_DONE.json`) are written, inside the SAME index container as `queryTarget`. When omitted or blank, results are written under `queryTarget`, byte-identical to legacy behavior. Must match `[A-Za-z0-9_-]+` (no slashes, separators, or `..`); a violating value fails the query closed. The READ path for the durable index (templates, byte ranges, reverse index) is unaffected and always uses `queryTarget`. #### :material-menu-right-outline:**`querySearch`** List of terms to search for. |Type|Default|Category| |---|---|---| |String|""|Filter| Specifies a search expression applied to the underlying storage via [TenXTemplate filters](https://doc.log10x.com/run/input/objectStorage/index/#tenxtemplate-filters) for efficient pre-filtering. Supports only basic operations: `&&` (AND), `||` (OR), == (equality), and [includes](https://doc.log10x.com/api/js/#TenXString.includes) (substring match). Example: ```js (level==INFO || level==WARN) && includes(text, "POST") ``` This narrows down data fetched from storage before in-memory processing. To perform more complex in-memory filtering, specify [queryFilters](https://doc.log10x.com/run/input/objectStorage/query/#queryfilters "list of terms to search for"). #### :material-menu-right-outline:**`queryFilters`** List of terms to search for. |Type|Default|Category| |---|---|---| |List|\[\]|Filter| Specifies JavaScript expressions applied in-memory to all TenXObjects that match the [querySearch](https://doc.log10x.com/run/input/objectStorage/query/#querysearch "list of terms to search for") expression. Supports all valid [TenX JavaScript](https://doc.log10x.com/api/js/) expressions for complex filtering, enrichment, and processing. This provides precise, exact filtering after data is fetched from storage. ```yaml queryFilters: - LevelTemplate.level == "ERROR" - Math.max(this.price, 100) ``` #### :material-menu-right-outline:**`queryFrom`** Beginning of the search time range (inclusive). |Type|Required|Category| |---|---|---| |String|✔|Filter| defines the beginning of the search time range (inclusive) expressed as a UNIX millisecond epoch value. This value commonly uses the [now](https://doc.log10x.com/api/js/#TenXDate.now) function. For example, to express the value of one hour before the current millisecond epoch, specify: ```yaml queryFrom: now("-1h") ``` #### :material-menu-right-outline:**`queryTo`** End of the search time range (exclusive). |Type|Required|Category| |---|---|---| |String|✔|Filter| defines the end of the search time range expressed as a UNIX millisecond epoch value (exclusive). This value commonly uses the [now](https://doc.log10x.com/api/js/#TenXDate.now) function. For example, to express the value of the current millisecond epoch, specify: ```yaml queryTo: now() ``` ### Access #### :material-menu-right-outline:**`queryObjectStorageName`** Object storage logical name. |Type|Required|Category| |---|---|---| |String|✔|Access| Identifies the [Object Storage](https://doc.log10x.com/run/input/objectStorage/#objectstorageclassname) against which this query is made (e.g., `AWS`). ### Scan #### :material-menu-right-outline:**`queryIndexContainer`** Object storage container containing index objects. |Type|Required|Category| |---|---|---| |String|✔|Scan| Specifies the object storage container (e.g., AWS S3 bucket) name in which to store [TenXTemplate Filters](https://doc.log10x.com/run/input/objectStorage/index/#tenxtemplate-filters)'. This value corresponds with the [indexWriteContainer](https://doc.log10x.com/run/input/objectStorage/index "Index files uploaded to a object storage container (e.g. AWS S3 bucket)."). #### :material-menu-right-outline:**`queryScanFunctionParallelTimeslice`** Max time range to process by a scan worker instance. |Type|Default|Category| |---|---|---| |String|""|Scan| Specifies the max time duration (e.g., 1min) an instance of the [scan](https://doc.log10x.com/run/input/objectStorage/query/#scan) worker will traverse within the index. For example, if a query specifies a time range of `15min` and this value is set to `1m`, 15 instances of the scan worker will execute in parallel to scan through the index. If this value is 0, a single worker instance will sequentially scan through the index. #### :material-menu-right-outline:**`queryScanFunctionParallelMaxInstances`** Max number of instances of the scan worker. |Type|Default|Category| |---|---|---| |Number|1000|Scan| Specifies the maximum number of [scan](https://doc.log10x.com/run/input/objectStorage/query/#scan) worker instances that will traverse the index. if the provided time range and [timeSlice](https://doc.log10x.com/run/input/objectStorage/query/#queryscanfunctionparalleltimeslice "max time range to process by a scan worker instance") provided will result in more instances than the provided value, the time slice will increase its size until the amount of workers needed will be less than this value. For example, if a query specifies a time range of `60min` and a time slice of `1m`, while this value is set to `25`, the time slice will be effectively increased to `3m` to accommodate this. The maximum value for this option is: 10,000. #### :material-menu-right-outline:**`queryScanFunctionParallelThreads`** Max number of threads allocated per each 'scan' worker instance. |Type|Default|Category| |---|---|---| |Number|0|Scan| Sets the number of JVM threads to scan through a time slice of the index within a [scan](https://doc.log10x.com/run/input/objectStorage/query/#scan) worker instance. For example, if [timeSlice](https://doc.log10x.com/run/input/objectStorage/query/#queryscanfunctionparalleltimeslice "max time range to process by a scan worker instance") is `10min` and this value is set to 10, scan each 1m index time range using a dedicated thread. If set to 1 or 0, a single thread will scan through the entire index time range (e.g., 10min). #### :material-menu-right-outline:**`queryScanFlushInterval`** Interval to wait before posting pending 'stream' worker requests. |Type|Default|Category| |---|---|---| |String|""|Scan| Specifies a maximum interval (e.g., `2sec`) when scanning an index time slice to wait to submit pending [stream](https://doc.log10x.com/run/input/objectStorage/query/#stream) worker requests. If not set, all stream worker requests are sent after the index scanning process for a target index time slice has finished. #### :material-menu-right-outline:**`queryScanFunctionUrl`** Scan worker endpoint URL. |Type|Required|Category| |---|---|---| |String|✔|Scan| Specifies the endpoint used to invoke the [scan](https://doc.log10x.com/run/input/objectStorage/query/#scan) worker. ### Stream #### :material-menu-right-outline:**`queryReadContainer`** Location of the output file for writing TenXObject field values. |Type|Required|Category| |---|---|---| |String|✔|Stream| Specifies the object storage container (e.g., S3 bucket) name where input objects (e.g., log file) to scan are stored. This value corresponds with the [indexWriteContainer](https://doc.log10x.com/run/input/objectStorage/index "Index files uploaded to a object storage container (e.g. AWS S3 bucket)."). #### :material-menu-right-outline:**`queryStreamFunctionParallelObjects`** Max number of input object byte ranges to process in a single 'stream' instance. |Type|Default|Category| |---|---|---| |Number|0|Stream| Defines the maximum number of Storage objects to process by a single instance of the [stream](https://doc.log10x.com/run/input/objectStorage/query/#stream) worker. For example, if set to 50 and the [scan](https://doc.log10x.com/run/input/objectStorage/query/#scan) phase has identified 200 potentially matching input object byte ranges, four instances of the stream worker will split the workload in parallel. #### :material-menu-right-outline:**`queryStreamFunctionParallelByteRange`** Max input object bytes for a single fetch request. |Type|Required|Category| |---|---|---| |String|✔|Stream| Defines the max byte range within an input object (e.g., log file) that can fetched via a single request. For example, if this value is `5Mb` and two matching 1Mb byte ranges spaced 2Mb apart are identified within a target object fetch both from the storage via a single operation. If there is a 2Mb gap, launch two instances of the [stream](https://doc.log10x.com/run/input/objectStorage/query/#stream) worker. #### :material-menu-right-outline:**`queryStreamFunctionUrl`** Stream worker endpoint URL. |Type|Required|Category| |---|---|---| |String|✔|Stream| Specifies the endpoint used to invoke the [stream](https://doc.log10x.com/run/input/objectStorage/query/#stream) worker. ### Advanced #### :material-menu-right-outline:**`queryObjectStorageArgs`** Custom object storage args. |Type|Default|Category| |---|---|---| |List|\[\]|Advanced| Custom arguments passed as a map to the constructor of the underlying [object storage](https://doc.log10x.com/run/input/objectStorage/query/#indexobjectstoragename). This list is expected to hold pairs of key values (e.g., args: \[key1, value1, key2, value2\]). #### :material-menu-right-outline:**`queryFilterVars`** Specifies a list of variable values to scan for. |Type|Default|Category| |---|---|---| |List|\[\]|Advanced| Specifies a list of variable values to scan for. #### :material-menu-right-outline:**`queryFilterTemplateHashes`** Specifies a list of TenXTemplate hash values to scan for. |Type|Default|Category| |---|---|---| |List|\[\]|Advanced| Specifies a list of TenXTemplate hash values to scan for. #### :material-menu-right-outline:**`queryId`** Unique ID of host query. |Type|Default|Category| |---|---|---| |String|""|Advanced| UUID identifying this query. This value is automatically assigned if not explicitly set. #### :material-menu-right-outline:**`queryElapseTime`** Epoch after which processing times out. |Type|Default|Category| |---|---|---| |Number|0|Advanced| Epoch value which if exceeds the current system time will result in this request being cancelled. #### :material-menu-right-outline:**`queryScanFunctionLimitResultSizeInterval`** Interval by which to check whether 'queryLimitResultSize' is exceeded. |Type|Default|Category| |---|---|---| |String|""|Advanced| Interval by which to check whether [queryLimitResultSize](https://doc.log10x.com/run/input/objectStorage/query/#querylimitresultsize "max byte volume of matching events to fetch") that is the query has returned \>= the request result volume. ### Limit #### :material-menu-right-outline:**`queryLimitProcessingTime`** Max milliseconds alloted for this query. |Type|Required|Category| |---|---|---| |String|✔|Limit| Milliseconds allotted for this query to complete. Once this period has elapsed the query will terminate regardless of whether [queryLimitResultSize](https://doc.log10x.com/run/input/objectStorage/query/#querylimitresultsize "max byte volume of matching events to fetch") has not been exceeded. #### :material-menu-right-outline:**`queryLimitResultSize`** Max byte volume of matching events to fetch. |Type|Required|Category| |---|---|---| |String|✔|Limit| Volume in bytes of events matching the app prefix, terms and time range to fetch. Once this volume has been reached the query will terminate. If [queryLimitProcessingTime](https://doc.log10x.com/run/input/objectStorage/query/#querylimitprocessingtime "max milliseconds alloted for this query") has elapsed before the requested result volume has been fetched, the query will still terminate.
:material-github: This module is defined in [query/module.yaml](https://github.com/log-10x/modules/tree/main/pipelines/run/modules/input/objectStorage/query/module.yaml "query/module.yaml"){target="\_blank"}.