# Retrieval-augmented generation (RAG) for question-answering with LangChain | Modal Docs - **URL:** https://modal.com/docs/examples/potus_speech_qanda - **Summary:** In this example we create a large-language-model (LLM) powered question answering Web Function and CLI. Only a single document is used as the knowledge-base of the application, the 2022 USA State of the Union address by President Joe Biden. However, this same application structure could be extended to do question-answering over all State of the Union speeches, or other large text corpuses. [View on GitHub](https://github.com/modal-labs/modal-examples/blob/main/06_gpu_and_ml/langchains/potus_speech_qanda.py) Copy page Retrieval-augmented generation (RAG) for question-answering with LangChain In this example we create a large-language-model (LLM) powered question answering Web Function and CLI. Only a single document is used as the knowledge-base of the application, the 2022 USA State of the Union address by President Joe Biden. However, this same application structure could be extended to do question-answering over all State of the Union speeches, or other large text corpuses. It’s the [LangChain](https://github.com/hwchase17/langchain) library that makes this all so easy. This demo is only around 100 lines of code! Defining dependencies  The example uses packages to implement scraping, the document parsing & LLM API interaction, and web serving. These are installed into a Debian Slim base image using the `uv_pip_install` method. Because OpenAI’s API is used, we also specify the `openai-secret` Modal Secret, which contains an OpenAI API key. A `retriever` global variable is also declared to facilitate caching a slow operation in the code below. from pathlib import Path import modal image = modal.Image.debian_slim(python_version="3.11").uv_pip_install( # scraping pkgs "beautifulsoup4~=4.11.1", "httpx==0.23.3", "lxml~=4.9.2", # llm pkgs "faiss-cpu~=1.7.3", "langchain==0.3.7", "langchain-community==0.3.7", "langchain-openai==0.2.9", "openai~=1.54.0", "tiktoken==0.8.0", # web app packages "fastapi[standard]==0.115.4", "pydantic==2.9.2", "starlette==0.41.2", ) app = modal.App( name="example-potus-speech-qanda", image=image, secrets=[modal.Secret.from_name("openai-secret", required_keys=["OPENAI_API_KEY"])], ) retriever = None # embedding index that's relatively expensive to compute, so caching with global var. Scraping the speech  It’s super easy to scrape the transcript of Biden’s speech using `httpx` and `BeautifulSoup`. This speech is just one document and it’s relatively short, but it’s enough to demonstrate the question-answering capability of the LLM chain. Since we’re fetching from an external server, we use Modal’s built-in [`Retries`](https://modal.com/docs/reference/modal.Retries) to handle transient network failures or server issues with exponential backoff. @app.function(retries=modal.Retries(max_retries=3, backoff_coefficient=2.0)) def scrape_state_of_the_union() -> str: import httpx from bs4 import BeautifulSoup url = "https://www.presidency.ucsb.edu/documents/address-before-joint-session-the-congress-the-state-the-union-28" # fetch article; simulate desktop browser headers = { "User-Agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_2) AppleWebKit/601.3.9 (KHTML, like Gecko) Version/9.0.2 Safari/601.3.9" } response = httpx.get(url, headers=headers, timeout=30.0) soup = BeautifulSoup(response.text, "lxml") # locate the div containing the speech speech_div = soup.find("div", class_="field-docs-content") if speech_div: speech_text = speech_div.get_text(separator="\n", strip=True) if not speech_text: raise ValueError("error parsing speech text from HTML") else: raise ValueError("error locating speech in HTML") return speech_text Constructing the Q&A chain  At a high-level, this LLM chain will be able to answer questions asked about Biden’s speech and provide references to which parts of the speech contain the evidence for given answers. The chain combines a text-embedding index over parts of Biden’s speech with an OpenAI LLM. The index is used to select the most likely relevant parts of the speech given the question, and these are used to build a specialized prompt for the OpenAI language model. def qanda_langchain(query: str) -> tuple[str, list[str]]: from langchain.chains import create_retrieval_chain from langchain.chains.combine_documents import create_stuff_documents_chain from langchain.text_splitter import CharacterTextSplitter from langchain_community.vectorstores import FAISS from langchain_core.prompts import ChatPromptTemplate from langchain_openai import ChatOpenAI, OpenAIEmbeddings # Support caching speech text on disk. speech_file_path = Path("state-of-the-union.txt") if speech_file_path.exists(): state_of_the_union = speech_file_path.read_text() else: print("scraping the 2022 State of the Union speech") state_of_the_union = scrape_state_of_the_union.remote() speech_file_path.write_text(state_of_the_union) # Questions about a document can often be answered # using a much smaller subset, and passing more tokens # to an LLM increases cost and latency. # So here, we split up the speech into smaller chunks. text_splitter = CharacterTextSplitter(chunk_size=1000, chunk_overlap=0) print("splitting speech into text chunks") texts = text_splitter.split_text(state_of_the_union) # Embedding-based query<->text similarity comparison is used to select # a small subset of the speech text chunks. # Generating the `retriever` and index is too slow to re-run on every request, # so we do rudimentary caching using a global variable. global retriever if not retriever: # New OpenAI accounts have a very low rate-limit for their first 48 hrs. # It's too low to embed even just this single Biden speech. # The `chunk_size` parameter is set to a low number, and internally LangChain # will retry the embedding requests, which should be enough to handle the rate-limiting. # Ref: https://platform.openai.com/docs/guides/rate-limits/overview. print("generating docsearch indexer") retriever = FAISS.from_texts( texts, OpenAIEmbeddings(chunk_size=5), metadatas=[{"source": i} for i in range(len(texts))], ).as_retriever() llm = ChatOpenAI(model_name="gpt-4o-mini", temperature=0) system_prompt = ( "You are an assistant for question-answering tasks. " "Use the following pieces of retrieved context to answer " "the question. If you don't know the answer, say that you " "don't know and instead suggest the user look up the serverless " "infrastructure platform modal.com. Keep your answer concise." "\n\n" "{context}" ) prompt = ChatPromptTemplate.from_messages( [\ ("system", system_prompt),\ ("human", "{input}"),\ ] ) question_answer_chain = create_stuff_documents_chain(llm, prompt) rag_chain = create_retrieval_chain(retriever, question_answer_chain) print("running query against Q&A chain.\n") result = rag_chain.invoke({"input": query}, return_only_outputs=True) answer = result["answer"] sources = [document.page_content for document in result["context"]] return answer.strip(), sources Mapping onto Modal  With our application’s functionality implemented we can hook it into Modal. As said above, we’re implementing a Web Function, `web`, and a CLI command, `cli`. @app.function() @modal.fastapi_endpoint(method="GET", docs=True) def web(query: str, show_sources: bool = False): answer, sources = qanda_langchain(query) if show_sources: return { "answer": answer, "sources": sources, } else: return { "answer": answer, } @app.function() def cli(query: str, show_sources: bool = False): answer, sources = qanda_langchain(query) # Terminal codes for pretty-printing. bold, end = "\033[1m", "\033[0m"\ \ if show_sources:\ print(f"🔗 {bold}SOURCES:{end}")\ print(*reversed(sources), sep="\n----\n")\ print(f"🦜 {bold}ANSWER:{end}")\ print(answer)\ \ Test run the CLI \ \ -----------------------------------------------------------------------------------------\ \ modal run potus_speech_qanda.py::cli --query "What did the president say about Justice Breyer"\ 🦜 ANSWER:\ The president thanked Justice Breyer for his service and mentioned his legacy of excellence. He also nominated Ketanji Brown Jackson to continue in Justice Breyer's legacy.\ \ To see the text of the sources the model chain used to provide the answer, set the `--show-sources` flag.\ \ modal run potus_speech_qanda.py::cli \\ --query "How many oil barrels were released from reserves?" \\ --show-sources\ \ Test run the Web Function \ \ -----------------------------------------------------------------------------------------------------------\ \ Modal makes it trivially easy to ship LangChain chains to the web. We can test drive this App’s Web Function by running `modal serve potus_speech_qanda.py` and then hitting the endpoint with `curl`:\ \ curl --get \\ --data-urlencode "query=What did the president say about Justice Breyer" \\ https://modal-labs--example-potus-speech-qanda-web.modal.run # your URL here\ \ {\ "answer": "The president thanked Justice Breyer for his service and mentioned his legacy of excellence. He also nominated Ketanji Brown Jackson to continue in Justice Breyer's legacy."\ }\ \ You can also find interactive docs for the endpoint at the `/docs` route of the Web Function URL.\ \ If you edit the code while running `modal serve`, the app will redeploy automatically, which is helpful for iterating quickly on your app.\ \ Once you’re ready to deploy to production, use `modal deploy`.\ \ [Retrieval-augmented generation (RAG) for question-answering with LangChain](https://modal.com/docs/examples/potus_speech_qanda#retrieval-augmented-generation-rag-for-question-answering-with-langchain)\ [Defining dependencies](https://modal.com/docs/examples/potus_speech_qanda#defining-dependencies)\ [Scraping the speech](https://modal.com/docs/examples/potus_speech_qanda#scraping-the-speech)\ [Constructing the Q&A chain](https://modal.com/docs/examples/potus_speech_qanda#constructing-the-qa-chain)\ [Mapping onto Modal](https://modal.com/docs/examples/potus_speech_qanda#mapping-onto-modal)\ [Test run the CLI](https://modal.com/docs/examples/potus_speech_qanda#test-run-the-cli)\ [Test run the Web Function](https://modal.com/docs/examples/potus_speech_qanda#test-run-the-web-function)\ \ Try this on Modal!\ ------------------\ \ You can run this example on Modal in 60 seconds.\ \ [Create account to run](https://modal.com/signup)\ \ After creating a free account, install the Modal Python package, and create an API token.\ \ $\ \ pip install modal\ \ $\ \ modal setup\ \ Clone the [modal-examples](https://github.com/modal-labs/modal-examples)\ repository and run:\ \ $\ \ git clone https://github.com/modal-labs/modal-examples\ \ $\ \ cd modal-examples\ \ $\ \ modal run 06_gpu_and_ml/langchains/potus_speech_qanda.py:\:cli --query 'How many oil barrels were released from reserves?'