---
title: "Input Stream"
description: "read events from a variety of sources to transform into TenXObjects"
source: "https://github.com/log-10x/modules/tree/main/pipelines/run/units/input/stream/unit.yaml"
icon: "material/waves-arrow-right"

---
Defines network, disk, and IPC locations for reading events into [TenXObjects](https://doc.log10x.com/api/js/#TenXObject "Provide structured, reflective access to log/trace events read from input(s)."). Pipelines support multiple concurrent streams.

Input [modules](https://doc.log10x.com/engine/module/#run) read from log analyzers, forwarders, and other sources.

Each stream provides:

- [:material-select-all: Extractors](#inputextractors "names of extractors defining JSON fields/regex capture groups to select"): Select and redact input data before transformation
- [:material-tag-outline: Source patterns](#inputsourcepattern "a regex pattern used to extract the 'source' value"): Associate events with their origin (log file, network resource)
- [:material-pipe-valve: Backpressure](#backpressure): Throttle data volume

### :octicons-package-dependents-24: Extensions

The [input extensions](https://doc.log10x.com/api/input/) API reads events from custom local and remote sources.

## :material-menu: Options

Specify the options below to [configure](/config "configure") multiple Input stream:

|Name|Description|Category|
|---|---|---|
|[inputName](#inputname "logical name for the input stream")|Logical name for the input stream|General|
|[inputEnabled](#inputenabled "a JavaScript expression that must be evaluated as 'truthy' to enable the input")|A JavaScript expression that must be evaluated as 'truthy' to enable the input|General|
|[inputPrintProgress](#inputprintprogress "sets whether this input reports progress to the console")|Sets whether this input reports progress to the console|General|
|[inputType](#inputtype "type of stream. Possible values:[stream, file, inline, stdin]")|Type of stream. Possible values:\[stream, file, inline, stdin\]|Stream|
|[inputPath](#inputpath "input stream location")|Input stream location|Stream|
|[inputArgs](#inputargs "custom arguments passed to InputStream")|Custom arguments passed to InputStream|Stream|
|[inputExtractors](#inputextractors "names of extractors defining JSON fields/regex capture groups to select")|Names of extractors defining JSON fields/regex capture groups to select|Parse|
|[inputEscapeScheme](#inputescapescheme "unescape scheme to apply to characters read from the underlying input")|Unescape scheme to apply to characters read from the underlying input|Parse|
|[inputCharset](#inputcharset "charset used to decode bytes read from the underlying input")|Charset used to decode bytes read from the underlying input|Parse|
|[inputSourcePattern](#inputsourcepattern "a regex pattern used to extract the 'source' value")|A regex pattern used to extract the 'source' value|Parse|
|[inputSourceFields](#inputsourcefields "JSON fields from which to extract the 'source' value")|JSON fields from which to extract the 'source' value|Parse|
|[inputSourceFilter](#inputsourcefilter "a regex pattern used to filter out input events from applying 'inputSourcePattern'")|A regex pattern used to filter out input events from applying 'inputSourcePattern'|Parse|
|[inputBackpressureIntervalBytesLimit](#inputbackpressureintervalbyteslimit "the max allowed number of bytes to read within 'inputBackpressureIntervalBytesDuration'")|The max allowed number of bytes to read within 'inputBackpressureIntervalBytesDuration'|Backpressure|
|[inputBackpressureIntervalBytesDuration](#inputbackpressureintervalbytesduration "the time interval after which to reset the backpressure input quota")|The time interval after which to reset the backpressure input quota|Backpressure|
|[inputBackpressureTotalEventsLimit](#inputbackpressuretotaleventslimit "the total number of bytes to read from input")|The total number of bytes to read from input|Backpressure|
|[inputBackpressureTotalBytesLimit](#inputbackpressuretotalbyteslimit "the total number of bytes to read from input")|The total number of bytes to read from input|Backpressure|
|[inputBackpressureTotalDuration](#inputbackpressuretotalduration "a JavaScript expression that evaluates the max duration in which an input can read events before closing")|A JavaScript expression that evaluates the max duration in which an input can read events before closing|Backpressure|
|[inputForeach](#inputforeach "name of option group for whose instances to create input streams")|Name of option group for whose instances to create input streams|Advanced|

### General

#### :material-menu-right-outline:**`inputName`**

Logical name for the input stream.

|Type|Default|Category|
|---|---|---|
|String|an automatically generated input name|General|

Defines a unique logical name for the input (e.g., `myFluentd`).
Each TenXObject instantiated from events read from this input
will return this value via its [inputName](https://doc.log10x.com/api/js/#TenXBaseObject+inputName "Returns the context in which the current object, template or summary was created.") field.


#### :material-menu-right-outline:**`inputEnabled`**

A JavaScript expression that must be evaluated as 'truthy' to enable the input.

|Type|Default|Category|
|---|---|---|
|String|true|General|

enables/disables this input stream. If set, the JavaScript expression returns a truthy value
to open the input. For example, to configure this value to use a startup argument/shell variable, use:

```yaml
input:
   name: myInput
   enabled: TenXEnv.get("readFromStdin") 
  ... 
```


#### :material-menu-right-outline:**`inputPrintProgress`**

Sets whether this input reports progress to the console.

|Type|Default|Category|
|---|---|---|
|String|""|General|

Controls whether to print information on the volume of events read input to the console.
The [printProgress](https://doc.log10x.com/run/console/#printprogress "report i/o progress to the console") launch argument must also be set for this value to take effect.


### Stream

#### :material-menu-right-outline:**`inputType`**

Type of stream. Possible values:\[stream, file, inline, stdin\].

|Type|Required|Category|
|---|---|---|
|String|✔|Stream|

Determines the type of input. Possible values:

- file: contents of a target file specified by [inputPath](https://doc.log10x.com/run/input/stream/#inputpath "input stream location"). The file is read once and NOT tailed.
- stdin: read from the process stdin device.
- inline: read JSON-escaped event(s) from the value of `inputPath` separated by `\n`.
- stream: read events from an [InputStream](https://docs.oracle.com/javase/8/docs/api/java/io/InputStream.html){target="\_blank"}
  or [Reader](https://docs.oracle.com/javase/8/docs/api/java/io/Reader.html?is-external=true){target="\_blank"} sub-class specified by by 'inputPath'.


#### :material-menu-right-outline:**`inputPath`**

Input stream location.

|Type|Required|Category|
|---|---|---|
|String|✔|Stream|

Specifies the location from which to read events. This value depends on [inputType](https://doc.log10x.com/run/input/stream/#inputtype "type of stream. Possible values:[stream, file, inline, stdin]"). For:

- file: file path\\glob on disk (relative or canonical), or \`stdin\`\` to read from the process stdin device
- inline: JSON-escaped content events separated by `\n`.
- stream: fully qualified [InputStream](https://docs.oracle.com/javase/8/docs/api/java/io/InputStream.html){target="\_blank"}/
  [Reader](https://docs.oracle.com/javase/8/docs/api/java/io/Reader.html?is-external=true){target="\_blank"} sub-class to instantiate.
- stdin: [inputPath](https://doc.log10x.com/run/input/stream/#inputpath "input stream location") is ignored.


#### :material-menu-right-outline:**`inputArgs`**

Custom arguments passed to InputStream.

|Type|Default|Category|
|---|---|---|
|List|\[\]|Stream|

If [inputType](https://doc.log10x.com/run/input/stream/#inputtype "type of stream. Possible values:[stream, file, inline, stdin]") is `stream`, this argument specifies a list of values pairs (key1,value1...) passed to the constructor of the
InputStream/Reader sub-class specified by [inputPath](https://doc.log10x.com/run/input/stream/#inputpath "input stream location").

A Map combining these values and those of the input's module option group
specified by the (inputForeach)\[#inputforeach\] argument (if present) provides the
stream with module-specific context (e.g., auth tokens, host addresses) 'config' map.

An [EvaluatorBean](https://github.com/log-10x/pipeline-extensions/blob/main/edge-extensions/src/main/java/com/log10x/api/bean/EvaluatorBean.java){target="\_blank"} reference
provides an interface to the 10x JavaScript engine.

For example:

```java
  import com.log10x.api.bean.EvaluatorBean;
  import java.io.Reader;
  import java.util.Map;

  public class MyReader extends Reader {

    /**
    * If a constructor with this signature is found, it is used. 
    * 
    * @param config	combines launch argument provided to the 10x Engine
    * 					    for this input with those specified by 'inputArgs' 	
    * @param bean		provides an interface to the 10x Engine
    */
    public MyReader(Map<String, Object> config, EvaluatorBean bean) {
    }

    /**
    * If the above constructor is not defined this signature is used. 
    * 
    * @param config	combines launch argument provided to the 10x Engine
    * 					    for this input with those specified by 'inputArgs'
    */

    public MyReader(Map<String, Object> config) {
    }

    /**
    * 
    * If the two signatures above are not defined a parameterless
    * constructor must be defined.
    */
    public MyReader(){
    } 

    @Override
    public int read(char[] cbuf, int off, int len) throws IOException {
      //Read data from the underlying input 
      return charsRead;
    }

    @Override
    public void close() throws IOException {
    // close input
    }
}
```

If the search for a matching constructor fails, the pipeline halts.


### Parse

#### :material-menu-right-outline:**`inputExtractors`**

Names of extractors defining JSON fields/regex capture groups to select.

|Type|Default|Category|
|---|---|---|
|List|\[\]|Parse|

Specifies a list of [input extractors](https://doc.log10x.com/run/input/extract "Filter, redact, and capture events from an input stream to transform into TenXObjects") to apply on events read from this input stream.

If this value is empty, each line read from the stream of up to 65536 characters
is used to instantiate an TenXObject. Lines that exceed that length force
a virtual line break, and the following characters will form the next event until a line break is detected.


#### :material-menu-right-outline:**`inputEscapeScheme`**

Unescape scheme to apply to characters read from the underlying input.

|Type|Default|Category|
|---|---|---|
|String|""|Parse|

Specifies an unescape algorithm to apply to characters read from the underlying input.
Supported values: \[json,java,xml,html,javaScript\]. If not specified, characters are treated as not escaped.

To learn more see [Unescape algorithms](https://www.unbescape.org/usingunbescape.html){target="\_blank"}.


#### :material-menu-right-outline:**`inputCharset`**

Charset used to decode bytes read from the underlying input.

|Type|Default|Category|
|---|---|---|
|String|""|Parse|

Sets the charset used to decode bytes read from this input into characters.
Defaults to UTF-8.


#### :material-menu-right-outline:**`inputSourcePattern`**

A regex pattern used to extract the 'source' value.

|Type|Default|Category|
|---|---|---|
|String|""|Parse|

Defines a regex pattern that, if matched for the current event text
is attached to each TenXObject extracted from it as its 'source' value.

The source value identifies the logical location (e.g., log file, host address) from which an event originated. For example,
a [forwarder input](https://doc.log10x.com/run/input/forwarder "Report, receive and optimize events collected by log forwarders") may read events from multiple log files (i.e., sources) concurrently.

Assigning source values ensures that single and multiline events (e.g., stack traces) read from a target input (e.g., Filebeat, Fluent Bit)
originating from a source location (e.g., /var/log/a.log) are sequenced and grouped
without interlacing with events from a different source (e.g., /var/log/b.log).

For example, when reading events originating from different log files,
the `sourcePattern` can be set to: `\"fileName\":"(.*?)\"`
to extract a 'fileName' JSON field within each event to ensure the resulting
TenXObjects are sequenced and grouped (as in the case of stack traces)
with other TenXObjects based on the actual log file from which they originated.

The TenXObject intrinsic [source](https://doc.log10x.com/api/js/#TenXObject+source "Returns the source value assigned to this instance by its input source pattern") field provides access to this value at runtime
to control how to sequence an instance with other TenXObjects read from the current input stream.

To learn more see [event source](https://doc.log10x.com/run/transform/group/#event-source).


#### :material-menu-right-outline:**`inputSourceFields`**

JSON fields from which to extract the 'source' value.

|Type|Default|Category|
|---|---|---|
|String|""|Parse|

Specifies a list of JSON fields whose value are captured from every matching capture group within [inputSourcePattern](https://doc.log10x.com/run/input/stream/#inputsourcepattern "a regex pattern used to extract the 'source' value").
If no source pattern is set the entire event is scanned for the target JSON field values.
The TenXObject intrinsic [source](https://doc.log10x.com/api/js/#TenXObject+source "Returns the source value assigned to this instance by its input source pattern") field provides access to joint value at runtime to control how to sequence an instance with other TenXObjects read from the current input stream.
To learn more see [event source](https://doc.log10x.com/run/transform/group/#event-source).


#### :material-menu-right-outline:**`inputSourceFilter`**

A regex pattern used to filter out input events from applying 'inputSourcePattern'.

|Type|Default|Category|
|---|---|---|
|String|""|Parse|

Specifies a regex pattern to match for [inputSourcePattern](https://doc.log10x.com/run/input/stream/#inputsourcepattern "a regex pattern used to extract the 'source' value") to be applied.

This pattern provides an option to specify a fast-performing pattern to determine
whether inputSourcePattern is applied to capture 'source' values for TenXObjects extracted for an event read from the input.

For example, `inputSourceFilter` can determine whether an input event
begins with a target character sequence before a more complex `inputSourcePattern` is
applied to select its 'source' value.


### Backpressure

#### :material-menu-right-outline:**`inputBackpressureIntervalBytesLimit`**

The max allowed number of bytes to read within 'inputBackpressureIntervalBytesDuration'.

|Type|Default|Category|
|---|---|---|
|String|""|Backpressure|

Sets the maximum number of bytes that can be read by the
input within the target [inputBackpressureIntervalBytesDuration](https://doc.log10x.com/run/input/stream/#inputbackpressureintervalbytesduration "the time interval after which to reset the backpressure input quota"). If exceeded,
the input blocks further reading (if supported); otherwise, the input
drops events from the pipeline for the remainder of the interval.

For example:

```yaml
inputBackpressureIntervalBytesLimit: $=ParseBytes("10MB")
```

If set, [inputBackpressureIntervalBytesDuration](https://doc.log10x.com/run/input/stream/#inputbackpressureintervalbytesduration "the time interval after which to reset the backpressure input quota") must also be set.

This option aims to prevent a situation where the inflow of data exceeds the
memory allocated to the pipeline's host JVM via the XMX argument.

To learn more see [JVM heap](https://www.baeldung.com/jvm-parameters#explicit-heap-memory---xms-and-xmx-options){target="\_blank"}.


#### :material-menu-right-outline:**`inputBackpressureIntervalBytesDuration`**

The time interval after which to reset the backpressure input quota.

|Type|Default|Category|
|---|---|---|
|String|""|Backpressure|

sets the interval in milliseconds for which [inputBackpressureIntervalBytesLimit](https://doc.log10x.com/run/input/stream/#inputbackpressureintervalbyteslimit "the max allowed number of bytes to read within 'inputBackpressureIntervalBytesDuration'") is enforced.

When the duration elapses, the input resets the counter against which `inputBackpressureIntervalBytesLimit` is measured.

If set, `inputBackpressureIntervalBytesLimit` must also be set.

For example:

```yaml
inputBackpressureIntervalBytesDuration: $=parseDuration("1m")
```


#### :material-menu-right-outline:**`inputBackpressureTotalEventsLimit`**

The total number of bytes to read from input.

|Type|Default|Category|
|---|---|---|
|String|""|Backpressure|

Set the max number of events a target pipeline input will read into the pipeline.
This value limits the volume of events to read from a local/remote source (e.g., log analyzer).


#### :material-menu-right-outline:**`inputBackpressureTotalBytesLimit`**

The total number of bytes to read from input.

|Type|Default|Category|
|---|---|---|
|String|""|Backpressure|

set the max number of bytes a target pipeline input will read into the pipeline.
This value limits the volume of events to read from a local/remote source (e.g., log analyzer).

For example:

```yaml
inputBackpressureTotalBytesLimit: $=ParseBytes("1GB")
```


#### :material-menu-right-outline:**`inputBackpressureTotalDuration`**

A JavaScript expression that evaluates the max duration in which an input can read events before closing.

|Type|Default|Category|
|---|---|---|
|String|""|Backpressure|

set the max duration (e.g. `1m`) an input can read events from a local/remote
source before closing.

For example:

```yaml
inputBackpressureTotalDuration: $=parseDuration("5m")
```


### Advanced

#### :material-menu-right-outline:**`inputForeach`**

Name of option group for whose instances to create input streams.

|Type|Default|Category|
|---|---|---|
|String|""|Advanced|

Specifies the name of an options group for whose instances
to replicate this input stream object. This value enables 10x modules
to create multiple input instances, each receiving a unique set of arguments.

For a full example, see [Elastic input](https://doc.log10x.com/run/input/analyzer/elasticsearch "Read events from an ElasticSearch hosted/on-premises cluster").


<br/>:material-github: This unit is defined in [stream/unit.yaml](https://github.com/log-10x/modules/tree/main/pipelines/run/units/input/stream/unit.yaml "stream/unit.yaml"){target="\_blank"}.