--- title: "Input" description: "read events into the pipeline from local and remote sources" source: "https://github.com/log-10x/modules/tree/main/pipelines/run/units/input/unit.yaml" icon: "material/set-merge" --- Reads log/trace events from local and remote sources, transforming them into [TenXObjects](https://doc.log10x.com/api/js/#TenXObject "Provide structured, reflective access to log/trace events read from input(s)."). Inputs comprise two core components: _streams_ and _extractors_ to construct [modules](https://doc.log10x.com/engine/module/) which read events from edge/cloud sources such as log analyzers, forwarders and object storage. Multiple input streams can execute in conjunction, where each stream provides [backpressure](https://doc.log10x.com/run/input/stream/#backpressure) controls to ensure the pipeline can process the information received. ## :octicons-package-24: Modules
- :octicons-command-palette-24:{ .lg .middle } __Stdin Device Input__ --- Read events from stdin. [:octicons-arrow-right-24: More info](/run/input/stdin) - :material-math-log:{ .lg .middle } __File Inputs__ --- Read events from an input file/glob path. [:octicons-arrow-right-24: More info](/run/input/file) - :material-text-search:{ .lg .middle } __Log Analyzer Inputs__ --- Read events from log analyzers via REST. [:octicons-arrow-right-24: More info](/run/input/analyzer) - :simple-fluentbit:{ .lg .middle } __Log Forwarder Inputs__ --- Report, receive and optimize events collected by log forwarders. [:octicons-arrow-right-24: More info](/run/input/forwarder) - :material-cloud-arrow-right-outline:{ .lg .middle } __Object Storage Input__ --- Stream events from object storage. [:octicons-arrow-right-24: More info](/run/input/objectStorage) - :material-waves-arrow-right:{ .lg .middle } __Input Streams__ --- Read events from a variety of sources to transform into TenXObjects. [:octicons-arrow-right-24: More info](/run/input/stream) - :material-select-all:{ .lg .middle } __Input Extractors__ --- Filter, redact, and capture events from an input stream to transform into TenXObjects. [:octicons-arrow-right-24: More info](/run/input/extract)
## :material-wrench-outline: Config Files To configure the Input unit, [:material-cog: Edit](https://doc.log10x.com/config/app/#module-config "Learn how to edit app and module configurations") these files. Below is the default configuration from: [backpressure/config.yaml](https://github.dev/log-10x/config/blob/main/pipelines/run/input/backpressure/config.yaml "backpressure/config.yaml"){target="\_blank"}.
Edit Online
Edit config.yaml Locally
```yaml # 🔟❎ run input configuration # This config file specifies general pipeline input settings # To learn more see https://doc.log10x.com/run/input tenx: run # ============================== Concurrency Settings ========================= # 'inputThreadPoolSize' controls the maximum number of threads allocated to concurrently # read events from inputs source (e.g., stream, stdin, file). inputThreadPoolSize: 16 # ============================= Backpressure Settings ========================= objectLimit: # 'maxInstances' sets the max number of TenXObjects the pipeline can contain at any given moment # Default to 5K objects per 100MB ram of JVM -XMX (if specified), otherwise no limit maxInstances: $=TenXEnv.get("xmx") ? (parseInt(TenXEnv.get("xmx")) / 104857600 * 5000):0 # 'interval' sets the duration time between repeated backpressure tests once backpressure is detected. interval: $=parseDuration("1s") ``` ## :material-menu: Options Specify the options below to [configure](/config "configure") the Input: |Name|Description| |---|---| |[objectLimitMaxInstances](#objectlimitmaxinstances "max number of TenXObjects that the pipeline can hold")|Max number of TenXObjects that the pipeline can hold| |[objectLimitInterval](#objectlimitinterval "Defines interval between consecutive positive backpressure tests")|Defines interval between consecutive positive backpressure tests| |[inputThreadPoolSize](#inputthreadpoolsize "sets the maximum number of threads to allocate for concurrent inputs.")|Sets the maximum number of threads to allocate for concurrent inputs.| ### :material-menu-right-outline:**`objectLimitMaxInstances`** Max number of TenXObjects that the pipeline can hold. |Type|Default| |---|---| |Number|10000| Sets the max number of TenXObjects that the pipeline can contain at any given moment, beyond which input(s) either block (if supported) or drop events from the pipeline. Once an object reaches the end of the pipeline, space becomes available for made for instantiating TenXObjects from events read input. This value prevents scenarios where writing data to output is slower than reading events from input, which may cause backpressure inside the pipeline. Setting this value can avoid an out-of-mem error because of the object backlog. Increasing the host JVM [XMX ](https://www.baeldung.com/jvm-parameters#explicit-heap-memory---xms-and-xmx-options){target="\_blank"} argument can help address potential OOM issues. ### :material-menu-right-outline:**`objectLimitInterval`** Defines interval between consecutive positive backpressure tests. |Type|Default| |---|---| |Number|1000| Sets the interval between backpressure tests after detection to allow time for clearing TenXObjects from the pipeline before attempting additional tests. ### :material-menu-right-outline:**`inputThreadPoolSize`** Sets the maximum number of threads to allocate for concurrent inputs. |Type|Default| |---|---| |Number|16| Sets the maximum number of threads to allocate for concurrent inputs. If the value is - - 0: open a dedicated thread for each input - Less than 1: interpreted as a percentage of the number of available cores (e.g., 0.5 = use up to 50% of available cores) - 1: allocate a single async thread - Greater than 1: interpreted as a fixed number of threads (e.g., 10 = 10 threads) The number of threads should be limited when a large number of short-lived inputs are defined to avoid out-of-memory errors.
:material-github: This unit is defined in [input/unit.yaml](https://github.com/log-10x/modules/tree/main/pipelines/run/units/input/unit.yaml "input/unit.yaml"){target="\_blank"}.