# Input concurrency | Modal Docs

- **URL:** https://modal.com/docs/guide/concurrent-inputs
- **Summary:** This guide documents the use of the modal.concurrent decorator to process multiple inputs at the same time in a single Modal container.

Copy page

Input concurrency
=================

This guide documents the use of the `modal.concurrent` decorator to process multiple inputs at the same time in a single Modal container.

This page is a high-level guide to input concurrency. For reference documentation of the `modal.concurrent` decorator, see [this page](https://modal.com/docs/sdk/py/latest/modal.concurrent)
.

Overview 

As traffic to your application increases, Modal will automatically scale up the number of containers running your Function:

By default, each container will be assigned one input at a time. Autoscaling across containers allows your Function to process inputs in parallel. This is ideal when the operations performed by your Function are CPU-bound.

For some workloads, though, it is inefficient for containers to process inputs one-by-one. Modal supports these workloads with its _input concurrency_ feature, which allows individual containers to process multiple inputs at the same time:

When used effectively, input concurrency can reduce latency and lower costs.

Use cases 

Input concurrency can be especially effective for workloads that are primarily I/O-bound, e.g.:

*   Querying a database
*   Making external API requests
*   Making remote calls to other Modal Functions

For such workloads, individual containers may be able to concurrently process large numbers of inputs with minimal additional latency. This means that your Modal application will be more efficient overall, as it won’t need to scale containers up and down as traffic ebbs and flows.

Another use case is to leverage _continuous batching_ on GPU-accelerated containers. Frameworks such as [vLLM](https://modal.com/docs/examples/llm_inference)
 can achieve the benefits of batching across multiple inputs even when those inputs do not arrive simultaneously (because new batches are formed for each forward pass of the model).

Note that for CPU-bound workloads, input concurrency will likely not be as effective (or will even be counterproductive), and you may want to use Modal’s [_dynamic batching_ feature](https://modal.com/docs/guide/dynamic-batching)
 instead.

Enabling input concurrency 

To enable input concurrency, add the `@modal.concurrent` decorator:

    @app.function()
    @modal.concurrent(max_inputs=100)
    def my_function(input: str):
        ...

When using the class pattern, the decorator should be applied at the level of the _class_, not on individual methods:

    @app.cls()
    @modal.concurrent(max_inputs=100)
    class MyCls:
    
        @modal.method()
        def my_method(self, input: str):
            ...

Because all methods on a class will be served by the same containers, a class with input concurrency enabled will concurrently run distinct methods in addition to multiple inputs for the same method.

Setting a concurrency target 

When using the `@modal.concurrent` decorator, you must always configure the maximum number of inputs that each container will concurrently process. If demand exceeds this limit, Modal will automatically scale up more containers.

Additional inputs may need to queue up while these additional containers cold start. To help avoid degraded latency during scaleup, the `@modal.concurrent` decorator has a separate `target_inputs` parameter. When set, Modal’s autoscaler will aim for this target as it provisions resources. If demand increases faster than new containers can spin up, the active containers will be allowed to burst above the target up to the `max_inputs` limit:

    @app.function()
    @modal.concurrent(max_inputs=96, target_inputs=80)  # Allow a 20% burst
    def my_function(input: str):
        ...

It may take some experimentation to find the right settings for these parameters in your particular application. Our suggestion is to set the `target_inputs` based on your desired latency and the `max_inputs` based on resource constraints (i.e., to avoid GPU OOM). You may also consider the relative latency cost of scaling up a new container versus overloading the existing containers.

Concurrency mechanisms 

Modal uses different concurrency mechanisms to execute your Function depending on whether it is defined as synchronous or asynchronous. Each mechanism imposes certain requirements on the Function implementation. Input concurrency is an advanced feature, and it’s important to make sure that your implementation complies with these requirements to avoid unexpected behavior.

For synchronous Functions, Modal will execute concurrent inputs on separate threads. _This means that the Function implementation must be thread-safe._

    # Each container can execute up to 10 inputs in separate threads
    @app.function()
    @modal.concurrent(max_inputs=10)
    def sleep_sync():
        # Function must be thread-safe
        time.sleep(1)

For asynchronous Functions, Modal will execute concurrent inputs using separate `asyncio` tasks on a single thread. This does not require thread safety, but it does mean that the Function needs to participate in collaborative multitasking (i.e., it should not block the event loop).

    # Each container can execute up to 10 inputs with separate async tasks
    @app.function()
    @modal.concurrent(max_inputs=10)
    async def sleep_async():
        # Function must not block the event loop
        await asyncio.sleep(1)

Gotchas 

Input concurrency is a powerful feature, but there are a few caveats that can be useful to be aware of before adopting it.

### Input cancellations 

Synchronous and asynchronous Functions handle input cancellations differently. Modal will raise a `modal.exception.InputCancellation` exception in synchronous Functions and an `asyncio.CancelledError` in asynchronous Functions.

When using input concurrency with a synchronous Function, a single input cancellation will terminate the entire container. If your workflow depends on graceful input cancellations, we recommend using an asynchronous implementation.

### Concurrent logging 

The separate threads or tasks that are executing the concurrent inputs will write any logs to the same stream. This makes it difficult to associate logs with a specific input, and filtering for a specific function call in Modal’s web dashboard will show logs for all inputs running at the same time.

To work around this, we recommend including a unique identifier in the messages you log (either your own identifier or the `modal.current_input_id()`) so that you can use the search functionality to surface logs for a specific input:

    @app.function()
    @modal.concurrent(max_inputs=10)
    async def better_concurrent_logging(x: int):
        logger.info(f"{modal.current_input_id()}: Starting work with {x}")

[Input concurrency](https://modal.com/docs/guide/concurrent-inputs#input-concurrency)
[Overview](https://modal.com/docs/guide/concurrent-inputs#overview)
[Use cases](https://modal.com/docs/guide/concurrent-inputs#use-cases)
[Enabling input concurrency](https://modal.com/docs/guide/concurrent-inputs#enabling-input-concurrency)
[Setting a concurrency target](https://modal.com/docs/guide/concurrent-inputs#setting-a-concurrency-target)
[Concurrency mechanisms](https://modal.com/docs/guide/concurrent-inputs#concurrency-mechanisms)
[Gotchas](https://modal.com/docs/guide/concurrent-inputs#gotchas)
[Input cancellations](https://modal.com/docs/guide/concurrent-inputs#input-cancellations)
[Concurrent logging](https://modal.com/docs/guide/concurrent-inputs#concurrent-logging)

See it in action

[Single GPU serving concurrent requests](https://modal.com/docs/examples/llm_inference)

[Responsive Stable Diffusion web UI](https://modal.com/docs/examples/text_to_image)