6 Ways to Improve the Performance of RAG (Retrieval Augmented Generation) Systems

LLMs (Large Language Models) are a remarkable invention, but they tend to run into one significant problem: AI hallucinations. RAG (Retrieval Augmented Generation) was developed to address this, making LLMs more useful by providing fact-based context for its responses.

This article explains how to improve the performance of Retrieval Augmented Generation (RAG) systems, which use large language models (LLMs) to generate high-quality answers to user queries. RAG systems leverage external information to strengthen LLM-generated responses; this article focuses on six practical strategies for enhancing their effectiveness.

1. Retrieval-Augmented Generation (RAG) Quick Start Guide

RAG (retrieval-augmented generation) is the process of supplementing a user's input to a large language model (LLM) such as ChatGPT with additional information retrieved from elsewhere. The LLM uses that information to enrich the response it generates.

This article focuses on improving RAG system quality and examines strategies for doing so in detail. For those building with RAG, we want to help close the gap between a basic setup and production-level performance. Here, improvement means increasing the share of queries for which the system successfully does the following:

1. Find the right context
2. Generate the right response

If you'd like to learn more about how RAG works, we recommend reading this article.

2. 6 Ways to Improve the Performance of RAG (Retrieval-Augmented Generation)

Here, we won't go into detail on implementing each strategy; instead, we'll explain when and why it's useful. Given how quickly this space evolves, an exhaustive or fully up-to-date list of best practices is impossible. Instead, our goal is to outline things to consider and try when improving your RAG application.

2.1. Organizing and Cleaning Your Data

RAG links an LLM's capabilities to your data. If the content or layout of your data is disorganized, problems arise in the system. Data with conflicting or redundant information makes it harder for retrieval to find the right context, so the LLM's generation step may not be optimal.

Suppose you're building a chatbot for a startup's help documentation and notice it isn't working well. First, check the data you're feeding into the system. Are topics divided logically? Is each covered in one place, or spread across several? If you, as a human, can't easily tell which document answers a common query, the retrieval system won't be able to either.

This can be as simple as manually merging documents on the same topic, but you can take it further. One creative approach: use an LLM to generate a summary of every document provided. The retrieval step then searches these summaries first, examining details only when needed. Some frameworks even offer this as a built-in abstraction.

2.2. Different Types of Indexes

Indexes are a core component of LlamaIndex and LangChain — the objects that hold your retrieval system. The standard RAG approach involves embeddings and similarity search: split and embed your context data, then find similar sections when a query comes in. This is highly effective, but not always the best fit. Is a query related to a specific item, such as a product in an e-commerce store? A keyword-based search may be worth considering. It doesn't have to be either/or — many applications use a hybrid, e.g. keyword-based indexes for product-specific queries and embeddings for general customer support.

2.3. Experimenting with the Base Prompt

Here's an example of a base prompt used in LlamaIndex.

"The context information is as follows. Please answer the question using the context information rather than prior knowledge."

You can override this and try other options — for instance, letting the LLM fall back on its own knowledge when it can't find a good answer in the context, or adjusting the prompt to control the types of queries it accepts, such as responding to subjective questions in a specific way. At minimum, overriding the prompt so the LLM has context about its job is useful. For example:

"You are a customer support agent. You are designed to be as helpful as possible while only providing factual information. You should be friendly, but avoid being overly chatty. The context information is as follows. Please answer the question using the context information rather than prior knowledge."

2.4. Using Query Routing

In many cases, it's useful to have multiple indexes and route each incoming query to the appropriate one — for example, separate indexes for summary questions, specific questions, and date-dependent questions. Optimizing a single index for all these behaviors hurts performance across the board; routing queries to the right index instead avoids this. Another use case is sending certain queries to a keyword-based index.

Once you've created your indexes, just define in text what each one is used for; the LLM then picks the right option at query time. Both LlamaIndex and LangChain offer tools for this.

2.5. Considering Query Transformations

A user's query is already being modified by being placed within the base prompt. It makes sense to modify it further still. Below are a few examples.

Paraphrasing: If the system can't find context relevant to a query, you can have the LLM rephrase the query and try again. Two questions that look the same to a human don't necessarily look the same in embedding space.

HyDE: HyDE is a strategy that takes a query, generates a hypothetical response, and uses both for embedding search. Research has shown this can dramatically improve performance.

Sub-queries: LLMs tend to perform better when complex queries are broken down. You can build this into your RAG system to decompose a query into multiple questions.

2.6. Fine-Tuning the Embedding Model

Embedding-based similarity is the standard retrieval mechanism for RAG. Data is split and embedded within the index. When a query comes in, it's also embedded so it can be compared against the embeddings in the index. But what exactly are embeddings? Typically, they come from a pre-trained model such as OpenAI's text-embedding-ada-002.

The problem is that a pre-trained model's notion of similarity in embedding space may not align with similarity in your context. Imagine working with legal documents: you'd want embedding similarity judged by domain-specific terms like "intellectual property" or "breach of contract," not generic terms like "herein" or "agree."

You can solve this by fine-tuning your embedding model, which can improve retrieval metrics by 5-10%. This takes a bit more effort, but it can make a significant difference in retrieval performance. LlamaIndex helps generate training sets, which makes this process easier than it sounds.

3. Frequently Asked Questions (FAQ)

4. Conclusion

Building with RAG can be frustrating: it's easy to get started, but hard to make it work well. We hope the strategies above offer hints for closing that gap. None of these ideas work every time — it's a process of experimentation and trial and error. This article hasn't covered evaluation in detail, i.e. how you measure system performance. At this stage evaluation is more art than science, but it's important to set up some system you can consistently check in on — the only way to know whether your changes are actually making a difference.

Miichisoft has extensive experience developing solutions that leverage RAG technology. For more details on how RAG works, please see this article on our website.