--- title: "Parallel Processor" description: "distribute event parsing and transformation workloads across multiple\ \ cores." source: "https://github.com/log-10x/modules/tree/main/pipelines/run/units/transform/parallelize/unit.yaml" icon: "octicons/cpu-24" --- Transform log/trace events read from [inputs](https://doc.log10x.com/run/input/ "Read events into the pipeline from local and remote sources") into typed [TenXObjects](https://doc.log10x.com/api/js/#TenXObject "Provide structured, reflective access to log/trace events read from input(s).") using multiple cores. ## :material-wrench-outline: Config Files To configure the Parallel processor unit, [:material-cog: Edit](https://doc.log10x.com/config/app/#module-config "Learn how to edit app and module configurations") these files. Below is the default configuration from: [parallelize/config.yaml](https://github.dev/log-10x/config/blob/main/pipelines/run/transform/parallelize/config.yaml "parallelize/config.yaml"){target="\_blank"}.
Edit Online
Edit config.yaml Locally
```yaml # 🔟❎ 'run' event parallel processing configuration # Transform log/trace events read from inputs into well-defined TenXObjects using multiple cores. # To learn more see https://doc.log10x.com/run/transform/parallelize/ # Set the 10x pipeline to 'run' tenx: run # =============================== Parallel Options ============================ parallelEvent: # 'threadPoolSize' specifies the number of threads allocated to transform events into TenXObjects concurrently. # If the value is - # - = 0: transform events into TenXObjects synchronously using their input stream calling thread. # - < 1: interpreted as a percentage of the number of available cores (e.g., 0.5 = use up to 50% of available cores) # - = 1: allocate a single dedicated to transform events. # - > 1: interpreted as a fixed number of threads (e.g., 10 = 10 threads) threadPoolSize: "0.5" # 'batchSize' specifies the maximum number of events to queue for concurrent processing before flushing. # If 'threadPoolSize' is 1, this value is unused, and events are transformed into TenXObject synchronously # to their calling input's thread. If 0, flush pending events after 'flushInterval' expires or the # source input reaches end-of-file. batchSize: 1000 # 'flushInterval' specifies the maximum wait duration before flushing queued events # If 'threadPoolSize' is 1, this value is unused, and events are transformed into TenXObject synchronously # to their input thread. If 0, no wait flush interval is applied. flushInterval: 2s # 'processingTimeout' specifies the maximum wait duration before dropping un-processed queued events # This value provides a backstop for overflowing the heap if the pipeline cannot dequeue # pending events to transform into TenXObjects. If 0, no timeout is applied. processingTimeout: 30s ``` ## :material-menu: Options Specify the options below to [configure](/config "configure") the Parallel processor: |Name|Description| |---|---| |[parallelEventThreadPoolSize](#paralleleventthreadpoolsize "number of threads to allocate to transform events into TenXObjects concurrently")|Number of threads to allocate to transform events into TenXObjects concurrently| |[parallelEventBatchSize](#paralleleventbatchsize "maximum number of events to queue for concurrent processing before flushing")|Maximum number of events to queue for concurrent processing before flushing| |[parallelEventFlushInterval](#paralleleventflushinterval "maximum wait duration before flushing queued events")|Maximum wait duration before flushing queued events| |[parallelEventProcessingTimeout](#paralleleventprocessingtimeout "maximum wait duration before dropping un-processed queued events")|Maximum wait duration before dropping un-processed queued events| ### :material-menu-right-outline:**`parallelEventThreadPoolSize`** Number of threads to allocate to transform events into TenXObjects concurrently. |Type|Default| |---|---| |String|""| Controls the number of threads to allocate to transform events into TenXObjects concurrently. If the value is - |Setting|Allocated Threads| |---|---| |:material-equal: **0**|Transform events into TenXObjects synchronously using their input stream calling thread| |:material-less-than: **1**|Interpreted as a percentage of the number of available cores (e.g., 0.5 = use up to 50% of available cores)| |:material-equal: **1**|Allocate a single dedicated to transform events| |:material-greater-than: **1**|Interpreted as a fixed number of threads (e.g., 10 = 10 threads)| ### :material-menu-right-outline:**`parallelEventBatchSize`** Maximum number of events to queue for concurrent processing before flushing. |Type|Default| |---|---| |Number|1000.0| Sets the number of events to queue before flushing them to transform into TenXObjects. If [threadPoolSize](https://doc.log10x.com/run/transform/parallelize/#paralleleventthreadpoolsize "number of threads to allocate to transform events into TenXObjects concurrently") is 1, this value is unused, and events are transformed into TenXObject synchronously to their calling input's thread. If 0, flush pending events after [flushInterval](https://doc.log10x.com/run/transform/parallelize/#paralleleventflushinterval "maximum wait duration before flushing queued events") expires or the source input reaches end-of-file. ### :material-menu-right-outline:**`parallelEventFlushInterval`** Maximum wait duration before flushing queued events. |Type|Default| |---|---| |String|2s| Sets an interval period for (e.g. '2s) before flushing pending events to transform into TenXObjects. If [threadPoolSize](https://doc.log10x.com/run/transform/parallelize/#paralleleventthreadpoolsize "number of threads to allocate to transform events into TenXObjects concurrently") is 1, this value is unused, and events are transformed into TenXObject synchronously to their input thread. If 0, no wait flush interval is applied. ### :material-menu-right-outline:**`parallelEventProcessingTimeout`** Maximum wait duration before dropping un-processed queued events. |Type|Default| |---|---| |String|30s| Sets a timeout period (e.g., '30s'), after which to drop queued events. This value provides a backstop for overflowing the heap if the pipeline cannot dequeue pending events to transform into TenXObject. If 0, no timeout is applied.
:material-github: This unit is defined in [parallelize/unit.yaml](https://github.com/log-10x/modules/tree/main/pipelines/run/units/transform/parallelize/unit.yaml "parallelize/unit.yaml"){target="\_blank"}.