---
title: "Object Storage Fetch Blob"
description: "read events from Storage blob byte range(s)"
source: "https://github.com/log-10x/modules/tree/main/pipelines/run/modules/input/objectStorage/object/module.yaml"
icon: "octicons/filter-24"
hidden: "true"

---
Fetch specified byte range(s) from an object storage container (e.g., AWS S3 bucket) to transform
into TenXObjects on which to perform a set of [actions](https://doc.log10x.com/run/input/objectStorage/query/#queryactions).

End-users do not commonly configure fetch object directly but instead use [queries](https://doc.log10x.com/run/input/objectStorage/query/ "Query an object storage container for events matching target criteria") to scan through a target container for matching byte ranges to fetch.

The query coordinator dispatches each stream worker with bootstrap args carrying its slice bounds ([`queryObjectSliceFrom`](#queryobjectslicefrom "slice from-bound (epoch ms) for this dispatch"), [`queryObjectSliceTo`](#queryobjectsliceto "slice to-bound (epoch ms) for this dispatch")) and output-mode flags ([`queryObjectWriteResults`](#queryobjectwriteresults "sets whether matched events are buffered to JSONL and uploaded to object storage"), [`queryObjectWriteSummaries`](#queryobjectwritesummaries "sets whether per-slice TenXSummary aggregates are buffered to JSONL and uploaded")). Slice bounds are encoded into output S3 keys: events at `qr/{queryId}/{sliceFrom}_{sliceTo}/{worker}.jsonl`, summaries at `qrs/{queryId}/{sliceFrom}_{sliceTo}/{worker}.jsonl`.

## :material-menu: Options

Specify the options below to [configure](/config "configure") multiple Object storage fetch blob:

|Name|Description|Category|
|---|---|---|
|[queryObjectQueryName](#queryobjectqueryname "logical name to give this query")|Logical name to give this query|General|
|[queryObjectPrintProgress](#queryobjectprintprogress "sets whether this input prints throughput stats to the console")|Sets whether this input prints throughput stats to the console|General|
|[queryObjectWriteResults](#queryobjectwriteresults "sets whether matched events are buffered to JSONL and uploaded to object storage")|Sets whether matched events are buffered to JSONL and uploaded to object storage|General|
|[queryObjectWriteSummaries](#queryobjectwritesummaries "sets whether per-slice TenXSummary aggregates are buffered to JSONL and uploaded")|Sets whether per-slice TenXSummary aggregates are buffered to JSONL and uploaded|General|
|[queryObjectSliceFrom](#queryobjectslicefrom "slice from-bound (epoch ms) for this dispatch")|Slice from-bound (epoch ms) for this dispatch|General|
|[queryObjectSliceTo](#queryobjectsliceto "slice to-bound (epoch ms) for this dispatch")|Slice to-bound (epoch ms) for this dispatch|General|
|[queryObjectObjectStorageName](#queryobjectobjectstoragename "object storage logical name")|Object storage logical name|Container|
|[queryObjectContainer](#queryobjectcontainer "storage container of the target blob")|Storage container of the target blob|Container|
|[queryObjectIndexContainer](#queryobjectindexcontainer "object storage container containing index objects")|Object storage container containing index objects|Container|
|[queryObjectTarget](#queryobjecttarget "logical name identifying the origin of 'queryObjectTargetObject'")|Logical name identifying the origin of 'queryObjectTargetObject'|Container|
|[queryObjectResultTarget](#queryobjectresulttarget "Tier-1 result-sink redirect prefix forwarded by the coordinator")|Tier-1 result-sink redirect prefix forwarded by the coordinator|Container|
|[queryObjectTargetObject](#queryobjecttargetobject "the target object to retrieve from storage")|The target object to retrieve from storage|Container|
|[queryObjectByteRanges](#queryobjectbyteranges "array of byte ranges to fetch")|Array of byte ranges to fetch|Container|
|[queryObjectFilter](#queryobjectfilter "filter applied on objects in range")|Filter applied on objects in range|Filter|
|[queryObjectFrom](#queryobjectfrom "beginning of the search time range (inclusive)")|Beginning of the search time range (inclusive)|Filter|
|[queryObjectTo](#queryobjectto "end of the search time range (exclusive)")|End of the search time range (exclusive)|Filter|
|[queryObjectObjectStorageArgs](#queryobjectobjectstorageargs "custom object storage args")|Custom object storage args|Advanced|
|[queryObjectID](#queryobjectid "unique ID of host query")|Unique ID of host query|Advanced|
|[queryObjectElapseTime](#queryobjectelapsetime "epoch after which processing times out")|Epoch after which processing times out|Advanced|
|[queryObjectLogLevels](#queryobjectloglevels "log levels to send to CloudWatch Logs for query progress tracking")|Log levels to send to CloudWatch Logs for query progress tracking|Advanced|
|[queryObjectLogGroup](#queryobjectloggroup "CloudWatch Logs log group for query event logging")|CloudWatch Logs log group for query event logging|Advanced|

### General

#### :material-menu-right-outline:**`queryObjectQueryName`**

Logical name to give this query.

|Type|Required|Category|
|---|---|---|
|String|✔|General|

Specifies a logical name associated with the query.
This value identifies this query operation in aggregated metrics reports.


#### :material-menu-right-outline:**`queryObjectPrintProgress`**

Sets whether this input prints throughput stats to the console.

|Type|Default|Category|
|---|---|---|
|Boolean|false|General|

Sets whether this input prints throughput stats to the console
This value is commonly used when testing an integration to a remote endpoint.


#### :material-menu-right-outline:**`queryObjectWriteResults`**

Sets whether matched events are buffered to JSONL and uploaded to object storage.

|Type|Default|Category|
|---|---|---|
|Boolean|false|General|

Controls whether a second output alongside the byte-count marker writer
buffers matched TenXObject events to a local temp file as JSONL and
uploads them to object storage on close.

Defaults to `false` so the extra JSONL upload only runs when a caller
explicitly opts in. When set to `true`, query results are written
under a slice-keyed prefix
`{indexObjectPath(queryResults)}/{queryId}/{sliceFrom}_{sliceTo}/*.jsonl`
so out-of-band consumers (e.g. the Log10x MCP server) can retrieve
partial progress by polling per-slice subdirectories. The fields
serialized per event default to `timestamp` + `text` plus every name
registered in the global `enrichmentFields` option.

The MCP server opts in by sending `writeResults=true` on its query
request; the coordinator forwards the flag to stream workers as a
`queryObjectWriteResults=true` bootstrap arg on the stream pipeline
launch. Deployments that rely solely on the byte-count marker
backstop used by the query coordinator leave this off and pay no
results-writer overhead.


#### :material-menu-right-outline:**`queryObjectWriteSummaries`**

Sets whether per-slice TenXSummary aggregates are buffered to JSONL and uploaded.

|Type|Default|Category|
|---|---|---|
|Boolean|false|General|

Controls whether a second results-style output buffers TenXSummary
records (produced by the optional `run/aggregate/stream` aggregator)
to a local temp file as JSONL and uploads them to object storage on
close.

Defaults to `false`. When set to `true`, summaries are written under
`{indexObjectPath(querySummaries)}/{queryId}/{sliceFrom}_{sliceTo}/*.jsonl`
so consumers can fetch a low-volume per-slice rollup (count,
bytesum, grouping fields) for noisy patterns instead of pulling raw
events. Independent of `queryObjectWriteResults`; either can be on
in isolation or both together.


#### :material-menu-right-outline:**`queryObjectSliceFrom`**

Slice from-bound (epoch ms) for this dispatch.

|Type|Default|Category|
|---|---|---|
|Number|0|General|

Set by the query coordinator to the lower bound of the timeslice
being processed by this stream worker. Always populated regardless
of `queryObjectFrom`/`queryObjectTo` (which may be zero when the
byte range fully contains the slice and per-event timestamp filtering
is unnecessary). Used by writers to key slice-grouped S3 outputs.


#### :material-menu-right-outline:**`queryObjectSliceTo`**

Slice to-bound (epoch ms) for this dispatch.

|Type|Default|Category|
|---|---|---|
|Number|0|General|

Set by the query coordinator to the upper bound of the timeslice
being processed by this stream worker. See `queryObjectSliceFrom`.


### Container

#### :material-menu-right-outline:**`queryObjectObjectStorageName`**

Object storage logical name.

|Type|Required|Category|
|---|---|---|
|String|✔|Container|

Identifies the [object storage](https://doc.log10x.com/run/input/objectStorage#objectstorageccessclassname) against which this query is made (e.g., `AWS`).


#### :material-menu-right-outline:**`queryObjectContainer`**

Storage container of the target blob.

|Type|Required|Category|
|---|---|---|
|String|✔|Container|

Specifies the object storage container (e.g., AWS S3 bucket) of the target
blob (e.g., log file) byte range to fetch.


#### :material-menu-right-outline:**`queryObjectIndexContainer`**

Object storage container containing index objects.

|Type|Required|Category|
|---|---|---|
|String|✔|Container|

Specifies the object storage container (e.g., AWS S3 bucket)
name in which to store [TenXTemplate Filters](https://doc.log10x.com/run/input/objectStorage/index/#tenxtemplate-filters).


#### :material-menu-right-outline:**`queryObjectTarget`**

Logical name identifying the origin of 'queryObjectTargetObject'.

|Type|Required|Category|
|---|---|---|
|String|✔|Container|

Specifies the logical name used to store index objects produced by the index app.


#### :material-menu-right-outline:**`queryObjectResultTarget`**

Tier-1 result-sink redirect prefix forwarded by the coordinator.

|Type|Default|Category|
|---|---|---|
|String|""|Container|

Bare-token prefix under which this stream worker writes query OUTPUT
(`qr/`, `qrs/`, `q/` markers) inside the index container. Forwarded by
the query coordinator from `queryResultTarget`. Blank/absent falls back
to `queryObjectTarget`, byte-identical to legacy behavior.


#### :material-menu-right-outline:**`queryObjectTargetObject`**

The target object to retrieve from storage.

|Type|Required|Category|
|---|---|---|
|String|✔|Container|

Sets the name of the object (e.g., log file) to retrieve from [queryObjectContainer](https://doc.log10x.com/run/input/objectStorage/query/object/#queryobjectcontainer).


#### :material-menu-right-outline:**`queryObjectByteRanges`**

Array of byte ranges to fetch.

|Type|Default|Category|
|---|---|---|
|List|\[\]|Container|

Defines an array of byte-range pairs (e.g., start1,length1..startN,lengthN)
to fetch from [queryObjectTargetObject](https://doc.log10x.com/run/input/objectStorage/query/object/#queryobjecttargetobject).

To learn more see [byte range fetches](https://docs.aws.amazon.com/whitepapers/latest/s3-optimizing-performance-best-practices/use-byte-range-fetches.html){target="\_blank"}.

If not set, fetch the entire object.


### Filter

#### :material-menu-right-outline:**`queryObjectFilter`**

Filter applied on objects in range.

|Type|Default|Category|
|---|---|---|
|String|""|Filter|

Specifies filtering logic for TenXObjects in this query object.
Combines both the querySearch (Bloom filter pre-filtering from storage) and queryFilters (in-memory JavaScript expressions) from the originating query.
This ensures consistent application of both storage-level and in-memory filtering.


#### :material-menu-right-outline:**`queryObjectFrom`**

Beginning of the search time range (inclusive).

|Type|Required|Category|
|---|---|---|
|Number|✔|Filter|

Defines the beginning of the search time range (inclusive)
expressed as a UNIX millisecond epoch value. This value commonly uses the
[now](https://doc.log10x.com/api/js/#TenXDate.now) function.

For example, setting this value to:

```yaml
queryObjectFrom: $=now("-1h")
````

express the value of one hour before the current millisecond epoch.


#### :material-menu-right-outline:**`queryObjectTo`**

End of the search time range (exclusive).

|Type|Required|Category|
|---|---|---|
|Number|✔|Filter|

Defines the end of the search time range expressed as an
UNIX millisecond epoch value (exclusive). This value commonly uses the [now](https://doc.log10x.com/api/js/#TenXDate.now)
function.


### Advanced

#### :material-menu-right-outline:**`queryObjectObjectStorageArgs`**

Custom object storage args.

|Type|Default|Category|
|---|---|---|
|List|\[\]|Advanced|

Custom arguments passed as a map to the constructor of the underlying [object storage](https://doc.log10x.com/run/input/objectStorage/query/object/#queryobjectobjectstoragename).
This list is expected to hold pairs of key values (e.g., args: \[key1, value1, key2, value2\]).


#### :material-menu-right-outline:**`queryObjectID`**

Unique ID of host query.

|Type|Required|Category|
|---|---|---|
|String|✔|Advanced|

UUID of the query which has submitted this fetch request.


#### :material-menu-right-outline:**`queryObjectElapseTime`**

Epoch after which processing times out.

|Type|Required|Category|
|---|---|---|
|Number|✔|Advanced|

Epoch value which if exceeds the current system time will result in this request being cancelled.


#### :material-menu-right-outline:**`queryObjectLogLevels`**

Log levels to send to CloudWatch Logs for query progress tracking.

|Type|Default|Category|
|---|---|---|
|List|\[\]|Advanced|

Specifies which query event log levels are sent to CloudWatch Logs for tracking query progress.
Valid values: `ERROR`, `INFO`, `DEBUG`, `PERF`.

If not specified, defaults to `ERROR`, `INFO`, `PERF`.


#### :material-menu-right-outline:**`queryObjectLogGroup`**

CloudWatch Logs log group for query event logging.

|Type|Default|Category|
|---|---|---|
|String|""|Advanced|

Specifies the CloudWatch Logs log group name for logging query lifecycle events.
The log group must be pre-created (e.g. by Terraform). If not specified or empty,
query event logging is disabled.


<br/>:material-github: This module is defined in [object/module.yaml](https://github.com/log-10x/modules/tree/main/pipelines/run/modules/input/objectStorage/object/module.yaml "object/module.yaml"){target="\_blank"}.