RAG for Finance: Automating Document Analysis with LLMs

With our use case defined, we will now walk through the full RAG pipeline to see how it handles real-world proxy statements.

Step 1. Data Ingestion and Parsing Data

The first step in any RAG workflow is data ingestion—collecting, parsing, and preparing the data for retrieval. In our case, that means downloading proxy statement PDFs and converting them into a format suitable for embedding. For this case study, we will be using three illustrative portfolio companies—Apple, Amazon, and Microsoft. We first download their proxy statements (Form DEF 14A) from the SEC’s EDGAR website and save them to our DEF14A PDFs folder.

from langchain.document_loaders import PyPDFLoader
import os
# Directory where PDFs are stored
pdf_folder = "Retrieval Augmented Generation/DEF14A-PDFs"
# Get list of PDF files in the folder
pdf_files = [os.path.join(pdf_folder, f) for f in os.listdir(pdf_folder) if f.lower().endswith(".pdf")]
# Load each PDF into a list of documents
documents = []
for file_path in pdf_files:
    loader = PyPDFLoader(file_path)
    docs = loader.load()
    documents.extend(docs)

Once the documents are collected, the next step is parsing them into a usable format. PDFs often contain a mix of text, images, and tables, making it difficult to extract structured data efficiently. Traditional PDF parsers primarily convert PDFs into plain text, which can lead to loss of important metadata, such as section headers and table structures, that are crucial for accurate information retrieval.

To address this, we will use Mistral’s Optical Character Recognition (OCR) API, which converts PDFs into markdown files with preserved structure, including section headers and table formatting. This approach provides several advantages:

• Improved text splitting: Section headers allow for more logical segmentation of content.
• Enhanced metadata extraction: Structured outputs enable easier classification of different sections of the document.
• Better query accuracy: By preserving document hierarchy, RAG models can retrieve relevant information with greater precision.

Later, we will compare this markdown-based approach with a more naive text extraction strategy that does not preserve structure, highlighting the benefits of structured parsing in enhancing retrieval accuracy and efficiency.

from pathlib import Path
import json
def parse_pdf(client: object, pdf_path: Path) -> str:
    # Upload PDF to Mistral's OCR service
    uploaded_file = client.files.upload(
        file={
            "file_name": pdf_path.stem,
            "content": pdf_path.read_bytes()
        },
        purpose="ocr"
    )
    
    # Get URL for the uploaded file from Mistral API
    signed_url = client.files.get_signed_url(file_id=uploaded_file.id, expiry=1)
    # Process PDF with OCR, returning markdown text
    pdf_response = client.ocr.process(
        document=DocumentURLChunk(document_url=signed_url.url),
        model="mistral-ocr-latest"
    )
    # Extract and return the markdown content
    response_dict = json.loads(pdf_response.model_dump_json())
    markdown_text = "\n\n".join(page["markdown"] for page in response_dict["pages"])
    
    return markdown_text

The following is a markdown output example:

# UNITED STATES 
 # UNITED STATES 
 SECURITIES AND EXCHANGE COMMISSION 
Washington, DC 20549
## SCHEDULE 14A
PROXY STATEMENT PURSUANT TO SECTION 14(a)
OF THE SECURITIES EXCHANGE ACT OF 1934\n(Amendment No. )
Filed by the Registrant
Filed by a Party other than the Registrant
## Check the appropriate box:
Preliminary Proxy Statement
Confidential

Note that for simplicity, we are not parsing images in this example, even though Mistral’s OCR API supports it. We have included table extraction, but image processing is beyond the scope of this demonstration⁶.

Step 2. Chunking the Markdown Files

Now that we have converted our PDFs into structured markdown, the next step is to split them into smaller, meaningful chunks. Instead of using naive character-based splitting, which can break the semantic structure of text, we will use markdown headers to ensure logical segmentation.

from langchain_text_splitters import MarkdownHeaderTextSplitter
def split_markdown_by_headers(pdf_name: str, markdown: str) -> list:
    # Define header levels to split on
    headers_to_split_on = [
        ("#", "Header 1"),
        ("##", "Header 2"),
        ("###", "Header 3")
    ]
    
    # Initialize the Markdown splitter
    markdown_splitter = MarkdownHeaderTextSplitter(headers_to_split_on)
    
    # Split the markdown content based on headers
    md_header_splits = markdown_splitter.split_text(markdown)
    
    # Attach the original file name as metadata for traceability
    for split in md_header_splits:
        split.metadata['file_name'] = pdf_name
    
    return md_header_splits

With this function, our markdown content is now divided into semantically meaningful chunks, preserving the structure of the proxy statements. This will improve retrieval performance when querying our RAG system in later steps.

Step 3. Embedding and Storing Documents in the Vector Database

With the markdown chunks prepared, the next step is to convert them into numerical vector embeddings and store them in a vector database. We will use Chroma, a lightweight and high-performance vector store, to store these embeddings. The embedding process is handled by OpenAIʼs OpenAIEmbeddings model, which converts the text into numerical representations optimized for retrieval.

from langchain_chroma import Chroma
from langchain_openai import OpenAIEmbeddings
# Store document embeddings in a vector database
vectorstore = Chroma.from_documents(
    documents=md_header_splits,
    embedding=OpenAIEmbeddings(),
    persist_directory="vector_db_md_split"
)

Step 4. Define the Retriever and Set Up the Prompt

Now that our structured markdown content has been embedded and stored, we need to define a retriever. The retriever is responsible for searching the vector database and returning the most relevant chunks of text to be added to the modelʼs context window when responding to queries.

We will use k-NN for retrieval and set k equal to 100, which identifies the top 100 most semantically similar text chunks to the input query. This is done by using a similarity measure, such as Euclidean distance, which measures the straight line distance of vectors, or such measures as cosine similarity, which measures the angular similarities of the vectors. This ensures that the model has enough relevant context for accurate analysis.

# Create a retriever from the vectorstore
retriever = vectorstore.as_retriever(search_kwargs={"k": 100})

Next, we need to set up the prompt to send to the retriever. We do this by creating a prompt template in which we define the instructions for the LLM and the inputs that will change based on each question. In our case, the inputs are the question we are asking and the relevant context the retriever finds in the database based on our question.

from langchain_core.prompts import ChatPromptTemplate
# Define your prompt template explicitly
template = """Execute the following query based only on the following context.
Query: \n {query}
Context: \n {context}
"""
def create_rag_prompt(query: str, retriever, template: str) -> str:
		# Create the prompt object from the prompt template
    prompt = ChatPromptTemplate.from_template(template)
    
    # Retrieve relevant documents (context) for a given query
    context_docs = retriever.get_relevant_documents(query)
    
    # Format context documents into a single string
		context_str = str(context_docs)
		
		# Format the final prompt we will send to the Openai chat completions API with context and query
		final_prompt = prompt.format(context=context_str, query=query)
    return final_prompt

We are now ready to query the model for response generation.

Step 5. Response Generation

The final step in the RAG workflow is response generation, where we use the context-enriched prompt to obtain answers from the large language model. For this example, we are leveraging OpenAIʼs API to generate structured responses. Structured outputs ensure the modelʼs responses conform precisely to a specified format. This will be very useful for the second approach, in which we are requesting the LLM to output the executive compensation details into a table for cross comparison. For the general interfacing of the documents, we will use a general model structure to force the LLM to explain its thoughts before providing an answer.

from openai import OpenAI
from dotenv import load_dotenv
from pydantic import BaseModel
# Define the structured output
class ExplainOutputs(BaseModel):
	explanation: str
	answer: str
# Define the OpenAI API client and load environment API keys
client = OpenAI()
load_dotenv()
# Send structured prompt to OpenAI's ChatCompletion API
completion = client.beta.chat.completions.parse(
    model="gpt-4o-mini",
    messages=[
        {"role": "system", "content": "You are an expert at interfacing documents and providing responses in a structured format."},
        {"role": "user", "content": final_prompt}
    ],
    response_format=ExplainOutputs,  # Enforces structured output
)
structured_response = completion.choices[0].message.parsed

We now have all the components of our RAG pipeline in place—from ingestion through structured response generation—and we are ready to move into the next phase: testing the RAG workflow.