RAG technique to provide external knowledge to LLMs

A query cannot be answered correctly without having knowledge about the intention of the query.

RAG is a technique that retrieves relevant information from external sources and gives it to an LLM (Large Language Model) as context when answering the question. It helps the LLM to get some knowledge about the intention of the query. In business, this technique will help us make the LLM work for our specific needs.

I don’t want to keep you waiting long, so let’s get started…

What is RAG? Why RAG?

Before understanding what RAG is, we need to know WHY RAG and what the business problem it’s going to solve.

Let’s assume you have a great finance assistant to work for your organisation, but he doesn’t have any knowledge of your real data. So, when you ask ‘Explain the reasons given in 2025 annual report for the fall in revenue’ do you think that he will be able to answer that? Without knowing the data, he couldn’t answer the question. This is where RAG comes into the picture.

You might have the best LLM, but they are trained on various source of data, and they don’t have any knowledge of your organisation’s data. So, we need to provide the respective data to the LLM to answer your query. RAG helps us to provide the data which are needed to answer the query.

The Three Stages: Retrieve, Augment and Generate

The name Retrieval-Augmented Generation (RAG) describes the three core stages involved in producing a response. Rather than relying solely on the knowledge learned during training, a RAG system first retrieves relevant information from external sources, augments the LLM’s prompt with that information, and then generates a response based on both the user’s query and the retrieved context.

Retrieve: Search the knowledge base and retrieve the most relevant information for the user’s query.

Augment: Add the retrieved information to the LLM’s prompt as additional context.

Generate: The LLM uses the augmented prompt to generate a grounded and context-aware response.

The following sections explain each of these stages in detail, with a primary focus on retrieval, as it is the foundation of an effective RAG system.

Retrieval

This is the part where the data that is relevant to the query are to be populated. There are many types of retrieval strategies which can be used alone or in a combination based on our business needs. Let’s assume we have our data in the form of documents, which makes it much easier to understand.

1. Keyword-Based Retrieval (Sparse Retrieval)

This is the simplest and traditional search approach. It will match the keywords in the query with documents and retrieve the matched ones.

How does it work?

Example Query: How much did we spend on machinery repairs in 2025?

I. Identify keywords in the query,

• machinery
• repair
• spent

II. Search all the populated keywords in the documents. We can use effective search like an inverted index.

III. Retrieve the documents/chunks which have the keywords

IV. Attach the context with the query and send it to the LLM. (Sample input to the LLM)

V. LLM will respond to the query using the attached context.

Limitation:

This approach will not work if the query doesn’t have the exact keywords. For example, if a user forms the query like

How much was the equipment maintenance expense in 2025?

The above query has the same intent, but it doesn’t have the exact keywords which are present in the document. Basic keyword search may fail or rank the correct document poorly when the query uses different terminology. More advanced techniques like fuzzy or phonetic search can partially bridge this gap, but they still struggle with true semantic differences in meaning. These kinds of issues are solved using semantic retrieval.

2. Semantic Retrieval (Dense Retrieval)

This is the most common retrieval method in modern RAG. Instead of comparing the keywords, this strategy will compare the meaning/intent of the query with the data.

The Main Idea Behind Semantic Retrieval

Semantic retrieval converts text into vectors. A vector is just a list of numbers that represents the meaning of the text.

Notice the vectors are very similar. The embedding model has learnt that that phrases such as “machine repair cost” and “equipment maintenance expense” most often express similar meanings, so it places their vectors close together.

Embedding Model

An embedding model is a machine learning model trained to convert text into vectors.

Input:    “equipment maintenance expense”

Output: [0.124, 0.762, 0.442, … 1536 numbers]

Depending on the embedding model, a vector may contain several hundred or several thousand numbers.

Example of embedding models:

• OpenAI text embedding models
• BERT-based embedding models
• Sentence Transformers

Document Storage Process (Indexing Phase)

We got the idea of semantic search. Now the question is how to convert our raw data to vectors. We need some pre-processing to enable the semantic search. R or the database which support vector search.

Let’s take a PDF:

Step 1: Document loading

The system extracts text from the document.

Example: The total spent on machine repair costs in 2025 was $2 million.

Step 2: Chunking

Large documents are split into smaller pieces called chunks. Choosing an effective chunking strategy is crucial, as it directly impacts retrieval quality. Depending on the nature of the data, strategies such as fixed-size chunking, semantic chunking, recursive character splitting, and others can be used.

Step 3: Generate Embeddings

Each chunk goes to the embedding model:

Example:
Chunk: The equipment maintenance cost was $2 million.

Vector: [0.27, 0.60, 0.91, 0.44, …]

Step 4: Store in Vector Database

The vector database stores something like:

So, a vector database usually stores:

This entire process is called indexing or ingestion.

[

Query Time (Retrieval Phase)

Query: How much was the equipment maintenance expense in 2025?

Step 1: Convert Query to Vector

The same embedding model is used:

Example: [0.26, 0.61, 0.90, 0.45]

Step 2: Vector Similarity Search

The query vector is compared against the millions of stored vectors in a vector database.

Example:

The closest vectors are selected.

Step 3: Return Top-K Chunks

The best matches are selected. Top selection will be fixed based on the output quality.

Example:

Top 2 results:

1. Total spent on machine repair cost in 2025 was $2 million.
2. Machinery replacement budget was $1M.

Step 4: Create a prompt – Augmentation

Augmentation simply means adding the retrieved information to the LLM’s prompt.

Step 5: LLM response – Generation

Now the LLM finally does what it is best at.

It reads the user’s question and augmented context and generated the natural language answer.

3. Hybrid Retrieval (Keyword + Semantic)

Interestingly, many production RAG systems use both.

Why?

Because semantic search can sometimes miss exact details. Example:

Query: Show invoice INV-98234

Semantic search may not treat INV-98234 as highly meaningful. Keyword search will match since the same invoice number will be present in the document. So, production RAG systems often combine keyword and semantic retrieval to leverage the strengths of both approaches. The retrieved candidates are then merged, deduplicated, and reranked so that only the most relevant passages are sent to the LLM.

This helps reduce irrelevant context, improves answer quality, and makes better use of the LLM’s context window.

4. Multi-Hop RAG

A hop means one step of retrieving information. A single hop which we have seen so far can solve a direct query.

Example:

Query: Who is the author of “Harry Potter”?

Consider the following query:

Query: In which country was the author of “Harry Potter” born?

Consider the following query:

Query: In which country was the author of “Harry Potter” born?

To answer this question, the system must connect information from multiple pieces of evidence:

• Identify the author of Harry Potter.
• Determine the country where the author was born.

Although a single retrieval may sometimes return both documents, this cannot always be guaranteed, especially in large knowledge bases. In such cases, the system first retrieves the author’s name, then performs another retrieval using that information to find the author’s birthplace. This step-by-step retrieval process is known as Multi-hop Retrieval.

Hop1:

Query: In which country was the author of “Harry Potter” born?

Retrieval:

Based on the above query, Let’s assume the system retrieved only the following context.

Document 1:
Harry Potter was written by J.K. Rowling.

Augmentation:

The retrieved context and the user’s query are then sent to the LLM. The prompt is carefully designed (augmented) to help the LLM determine whether the available context is sufficient to answer the question. If additional information is required, the LLM identifies the missing information and generates the next search query, enabling another retrieval hop.

Example Input

Generation:

The LLM reasons

The system recognises that it still needs one more fact and performs another search using the author’s name.

Hop2:

Query: Which country was J.K. Rowling born in?

Retrieval:

Context retrieved

Document 2:
J.K. Rowling was born in England, part of the United Kingdom.

Augmentation:

LLM Input

Generation:

LLM Output

Limitation:

It is difficult to handle multiple retrieval and reasoning steps.

Limitations:

Retrieval Quality Issues

The quality of a RAG system heavily depends on the quality of the retrieved context. Poor retrieval can result in missing relevant information or including irrelevant context, which directly impacts the accuracy of the generated response.

Source Quality

RAG grounds answers in source documents, but cannot guarantee the sources themselves are accurate. If the underlying documents are outdated, incorrect, duplicated, contradictory, or incomplete, the answer will reflect those flaws.

Irrelevant Context (Context Pollution)

Even when relevant information is retrieved, additional unrelated chunks may also be included, which can distract the LLM and reduce answer accuracy.

No Guarantee of Correct Answers (Hallucination)

RAG reduces hallucinations by providing external context, but it does not completely eliminate them. The LLM may still generate incorrect or misleading answers.

Security and Access Control

In a business context, not all documents should be accessible to all users. A RAG system must ensure users retrieve only documents they are authorised to see, sensitive data is protected, and queries and responses are appropriately logged and governed. Simply uploading documents to a vector database is not enough — access control must be part of the design.

Context Window Limitation

LLMs have a limited context window (the maximum number of tokens they can process). Large amounts of retrieved context may exceed this limit, causing important information to be excluded.

Difficulty with Multi-Hop Reasoning

Questions requiring information from multiple documents can be difficult to answer, as they may require multiple retrieval and reasoning steps.

Key Feature: Citations and Traceability

A well-designed RAG system should always indicate which document or passage supported the answer, allowing users to verify it independently. This builds trust and enables audit trails.

When RAG is Appropriate and When Another Tool Is Better

RAG is an excellent choice when the information required to answer a question exists in unstructured or semi-structured sources such as documents, PDFs, manuals, knowledge bases, policies, or FAQs. It enables LLMs to retrieve relevant information from these sources and generate grounded responses without retraining the model.

However, RAG is not the best solution for every use case. When information is stored in structured databases, it is often more efficient to query the database directly using SQL or another query language. Similarly, tasks involving calculations, business workflows, API integrations, or real-time operations are better handled by dedicated tools or agents rather than document retrieval.

In practice, enterprise AI applications often combine RAG with other techniques. For example, a system may use RAG to retrieve information from documents, execute SQL queries against a database, call APIs to fetch live data, or invoke external tools to perform actions. Choosing the right approach depends on the nature of the data and the problem being solved.

Conclusion

RAG is a practical and effective way to enable LLMs to answer questions using private, up-to-date, or domain-specific information without retraining the model. By retrieving relevant information, augmenting the prompt with that context, and generating a response, RAG helps ground answers in external knowledge while keeping the model’s knowledge separate from business data.

However, building an effective RAG system involves much more than retrieval alone. The quality of the response depends on the quality of the source data, document processing, chunking, retrieval strategy, ranking, prompt design, and the relevance of the retrieved context. Depending on the use case, keyword, semantic, hybrid, or multi-hop retrieval may be the most suitable approach.

Although RAG can significantly reduce hallucinations and improve answer traceability through source citations, it cannot guarantee correctness. An effective RAG solution should also consider security, access control, and the specific needs of its users. Ultimately, there is no one-size-fits-all approach; understanding your data, business requirements, and user queries is the key to designing an accurate, reliable, and scalable RAG system.

Frequently asked questions on RAG