import CodeBlock from "@theme/CodeBlock";

# Fallbacks

When working with language models, you may often encounter issues from the underlying APIs, e.g. rate limits or downtime.
Therefore, as you move your LLM applications into production it becomes more and more important to have contingencies for errors.
That's why we've introduced the concept of fallbacks.

Crucially, fallbacks can be applied not only on the LLM level but on the whole runnable level.
This is important because often times different models require different prompts. So if your call to OpenAI fails, you don't just want to send the same prompt to Anthropic - you probably want want to use e.g. a different prompt template.

## Handling LLM API errors

This is maybe the most common use case for fallbacks. A request to an LLM API can fail for a variety of reasons - the API could be down,
you could have hit a rate limit, or any number of things.

**IMPORTANT:** By default, many of LangChain's LLM wrappers catch errors and retry.
You will most likely want to turn those off when working with fallbacks. Otherwise the first wrapper will keep on retrying rather than failing.

import ModelExample from "@examples/guides/fallbacks/model.ts";

<CodeBlock language="typescript">{ModelExample}</CodeBlock>

## Fallbacks for RunnableSequences

We can also create fallbacks for sequences, that are sequences themselves.
Here we do that with two different models: ChatOpenAI and then normal OpenAI (which does not use a chat model).
Because OpenAI is NOT a chat model, you likely want a different prompt.

import ChainExample from "@examples/guides/fallbacks/chain.ts";

<CodeBlock language="typescript">{ChainExample}</CodeBlock>

## Handling long inputs

One of the big limiting factors of LLMs in their context window.
Sometimes you can count and track the length of prompts before sending them to an LLM,
but in situations where that is hard/complicated you can fallback to a model with longer context length.

import LongInputExample from "@examples/guides/fallbacks/long_inputs.ts";

<CodeBlock language="typescript">{LongInputExample}</CodeBlock>

## Fallback to a better model

Often times we ask models to output format in a specific format (like JSON). Models like GPT-3.5 can do this okay, but sometimes struggle.
This naturally points to fallbacks - we can try with a faster and cheaper model, but then if parsing fails we can use GPT-4.

import BetterModelExample from "@examples/guides/fallbacks/better_model.ts";

<CodeBlock language="typescript">{BetterModelExample}</CodeBlock>