RAG (Retrieval-Augmented Generation)

Retrieval-Augmented Generation (RAG) is an AI architecture that retrieves relevant information from an external knowledge source and provides that information to a language model as context before generating an answer. Instead of relying only on knowledge encoded in model parameters, the application can ground responses in documents, databases, knowledge bases, or other current and domain-specific data.

RAG is especially useful when an AI application must answer questions about private documentation, frequently changing information, or a dataset too large to place entirely inside the model's context window. The central idea is simple: retrieve the most useful evidence first, then ask the LLM to answer using that evidence.

Table of Contents

• Why RAG Is Needed
• How RAG Works
  • Indexing
  • Retrieval
  • Generation
• A Simple RAG Example
• Chunking Documents for RAG
• Retrieval Strategies
  • Semantic Search
  • Hybrid Search
  • Reranking
• Building the LLM Context
• RAG and Hallucinations
• Common RAG Failures
• Production RAG Architecture
  • Document Ingestion
  • Query Processing
• Monitoring and Evaluating RAG
• Conclusion
  • More Articles to Read

Why RAG Is Needed

Large language models learn patterns from their training data, but model parameters are not a live database. An LLM may not know private company information, recently updated policies, current inventory, or documentation created after its training process.

Consider an internal support assistant asked:

How long are failed payment events retained before permanent deletion?

The answer may exist in an internal operations manual that the base model has never seen. Without access to that document, the model can either admit that it does not know or generate an answer based on unrelated patterns learned during training.

One approach would be to place the entire documentation repository into every prompt. That quickly becomes impractical. A company might have hundreds of thousands of pages, while the LLM has a finite context window.

RAG solves this by selecting only information relevant to the current question:

Large Knowledge Base
        ↓
Retrieve Relevant Information
        ↓
Small Context
        ↓
LLM
        ↓
Grounded Answer

This provides several important benefits. Knowledge can be updated without retraining the model, private information can remain in application-controlled storage, and only a small subset of the knowledge base needs to consume model context for each request.

Context-window constraints and their production implications are covered in AI Tokens and Context Windows.

How RAG Works

A basic RAG system has two broad phases. First, source information is prepared and indexed. Later, when a request arrives, relevant information is retrieved and supplied to the language model.

A typical flow looks like:

Documents → Chunk → Embed → Index

Question → Retrieve → Build Context → LLM → Answer

The retrieval implementation can vary substantially. Vector search is common, but RAG does not require a vector database. Traditional full-text search, SQL queries, graph traversal, APIs, or combinations of retrieval methods can all supply external information to the model.

Indexing

For document-based RAG, source documents are commonly divided into smaller pieces called chunks.

Each chunk can then be converted into an embedding:

def index_chunk(chunk):
    vector = embedding_model.embed(chunk.text).vector

    vector_store.upsert(
        id=chunk.id,
        vector=vector,
        metadata={
            "document_id": chunk.document_id,
            "tenant_id": chunk.tenant_id,
            "title": chunk.title,
            "updated_at": chunk.updated_at,
        },
    )

The embedding represents the semantic characteristics of the chunk. The vector and metadata are stored in an index that supports similarity search.

Embeddings and similarity metrics are explained in AI Embeddings.

Retrieval

When a question arrives, the application converts it into a compatible query embedding and searches the index.

def retrieve(query: str, tenant_id: str):
    query_vector = embedding_model.embed(query).vector

    return vector_store.search(
        vector=query_vector,
        filters={
            "tenant_id": tenant_id,
            "status": "published",
        },
        limit=10,
    )

The result is a small set of candidate chunks that appear semantically related to the question.

Retrieval is one of the most important stages in the entire architecture. If the correct evidence is not retrieved, the language model cannot reliably use it.

A vector database can make this search efficient across large collections. Vector Databases for AI covers indexing, filtering, approximate nearest-neighbor search, and scaling in more detail.

Generation

After retrieval, the application places the selected text into the LLM context together with instructions and the user's question.

A simplified prompt might be:

Answer the question using only the provided context.

If the context does not contain enough information,
say that the available information is insufficient.

Context:
---
Failed payment events remain replayable for 14 days.
After 14 days they are archived for 90 days before deletion.
---

Question:
How long can a failed payment event be replayed?

The model now has evidence supporting the answer.

The LLM still generates probabilistically. RAG does not turn generation into a deterministic database lookup. The model can misunderstand retrieved text, combine information incorrectly, or ignore an instruction.

The application should therefore distinguish between retrieving evidence and generating an answer from evidence.

A Simple RAG Example

Consider a documentation assistant for an engineering organization.

The knowledge base contains these chunks:

Chunk A:
Production database backups are created every six hours
and retained for 30 days.

Chunk B:
Application logs are retained for 14 days.

Chunk C:
Redis is used for temporary session storage.

A user asks:

How long do production database backups remain available?

The application embeds the question and searches the indexed chunks. Chunk A should receive a high similarity score because its meaning closely matches the question.

The application then constructs the model context:

Documentation:

[1]
Production database backups are created every six hours
and retained for 30 days.

Question:
How long do production database backups remain available?

Answer using the documentation above.

The model can answer:

Production database backups are retained for 30 days.

Notice the division of responsibilities. The vector search finds the evidence. The application decides which evidence is allowed into the context. The LLM converts that evidence into a natural-language response.

This separation makes RAG easier to debug. If the answer is wrong, the first question should be: was the correct evidence retrieved?

Chunking Documents for RAG

Chunking is one of the most important RAG design decisions because retrieval operates on the indexed units.

Imagine a 40-page document covering authentication, billing, deployments, monitoring, and disaster recovery. Embedding the entire document as one vector compresses many unrelated topics into one representation.

Instead, the document can be split into meaningful sections.

Very small chunks provide precise retrieval but may lose necessary surrounding context. Very large chunks preserve context but may contain several unrelated topics and consume more LLM tokens when retrieved.

Structure-aware chunking often works better than splitting purely by character count. Technical documentation can be divided by headings and paragraphs. Source code can be divided by functions or classes. Support conversations can be divided by messages or conversation stages.

Useful metadata should also be preserved. A retrieved paragraph may be difficult to interpret without its document title or section heading.

For example:

{
  "text": "Backups are retained for 30 days.",
  "document": "Database Operations",
  "section": "Backup Retention",
  "version": 7,
  "updated_at": "2026-07-15"
}

Chunk size should be evaluated using real queries. There is no universally correct value such as 500 or 1,000 tokens for every RAG system.

Retrieval Strategies

Basic RAG is often presented as vector similarity search, but production retrieval can contain several stages. The goal is not to find vectors that look mathematically interesting. The goal is to deliver the evidence most useful for answering the question.

Semantic Search

Semantic search uses embeddings to find conceptually similar content.

A query such as:

Why is the standby database falling behind?

can retrieve a chunk containing:

Replication lag occurs when a read replica cannot replay changes as quickly as the primary generates them.

This works even though the query and document use different terminology.

Semantic retrieval is especially useful for natural-language questions, concept matching, and cases where users may describe the same problem in many different ways.

Hybrid Search

Embeddings are weaker when exact strings carry important meaning. Error codes, product identifiers, function names, SKUs, ticket IDs, and uncommon technical terms may be better handled by lexical search.

A query such as:

SQLSTATE 40001 transaction retry

benefits from preserving the exact error code.

Hybrid search combines semantic vector search with traditional lexical search:

Query
 ├─→ Vector Search
 └─→ Keyword Search
          ↓
      Merge Results
          ↓
       Candidates

The candidate rankings can be combined using weighted scoring or rank-fusion algorithms.

This approach often works well for enterprise and technical search because real queries contain both semantic intent and exact identifiers.

Reranking

Fast retrieval indexes are optimized to find a useful candidate set quickly. They do not necessarily produce the best possible final ranking.

A common architecture retrieves more results than the LLM will receive:

Retrieve 50 Candidates → Rerank → Select Best 5 → LLM

A reranker evaluates the query and candidate text more carefully than the first-stage vector search. This requires additional computation but is applied only to a small candidate set.

This separates two goals:

• Candidate retrieval: avoid missing potentially relevant evidence.
• Final ranking: identify which candidates are most useful.

Increasing the number of retrieved chunks without reranking is not always helpful. Sending many weakly related chunks to the LLM can consume context and introduce contradictory or irrelevant information.

Building the LLM Context

Retrieval produces candidates, but those candidates still need to be assembled into a useful model context.

A context builder should consider relevance, token budget, document authority, freshness, duplication, and permissions.

Suppose retrieval returns ten chunks but the application has space for only four. Selecting the four highest vector scores may work, but production systems may also consider whether several results contain duplicate text or come from outdated versions of the same document.

A context builder might prioritize:

1. Authorized content
2. Current document versions
3. High relevance
4. Diverse supporting evidence
5. Available token budget

The source should also remain identifiable. Instead of concatenating text without boundaries, context can preserve document metadata:

[Source 1]
Document: Database Operations
Section: Backup Retention
Updated: 2026-07-15

Backups are retained for 30 days.

[Source 2]
Document: Disaster Recovery
Section: Recovery Policy
Updated: 2026-08-02

Production backups are replicated to the recovery region.

This structure helps the model distinguish sources and makes citations possible in the final answer.

The total retrieved content should remain bounded. Context-window capacity is a resource, and retrieval should compete for a defined portion rather than consume everything available.

The objective is not to maximize retrieved tokens. It is to maximize useful evidence per token.

RAG and Hallucinations

RAG can reduce some hallucinations by providing authoritative evidence at inference time. It is especially useful when the model would otherwise need to rely on incomplete or outdated internal knowledge.

However, RAG does not eliminate hallucinations.

Several failure paths remain possible:

• the correct document is not retrieved;
• an irrelevant document receives a high retrieval score;
• retrieved sources contradict each other;
• the source itself contains incorrect information;
• the LLM misinterprets correct evidence;
• the model adds unsupported details to an otherwise correct answer.

A grounded prompt can instruct the model to answer only from supplied sources and explicitly state when evidence is insufficient.

Use only the provided sources.

If the sources do not contain enough information
to answer the question, respond that the available
information is insufficient.

Do not infer missing values.

This can improve behavior but remains an instruction rather than a deterministic guarantee.

For high-stakes workflows, application logic can require citations, verify that cited passages actually exist, constrain output schemas, or route uncertain cases for review.

The broader problem is covered in AI Hallucinations.

Common RAG Failures

RAG systems can fail at many stages, and blaming the language model for every poor answer hides the actual cause.

Poor chunking is a common problem. If the answer spans two chunks and neither contains enough context independently, retrieval may return incomplete evidence.

Weak retrieval occurs when the relevant document exists but does not appear among the candidates. This may come from unsuitable embeddings, incorrect similarity configuration, poor query formulation, or an approximate index tuned too aggressively for speed.

Too much retrieval can be as harmful as too little. Returning 30 loosely related chunks may overwhelm the important evidence with noise.

Stale indexes occur when source documents change but embeddings are not updated. The RAG system can then confidently retrieve outdated information.

Missing metadata filters can cause content from the wrong tenant, product, region, language, or document version to appear in results.

Exact-term failures happen when vector search is used for identifiers or technical strings that lexical search would handle better.

Generation failures happen even after correct retrieval. The model may misread evidence, combine unrelated facts, or introduce unsupported information.

These failure modes suggest a useful debugging sequence:

Was the source indexed?
        ↓
Was the correct chunk retrieved?
        ↓
Was it ranked high enough?
        ↓
Was it included in the prompt?
        ↓
Did the LLM interpret it correctly?
        ↓
Did the final output pass validation?

Observing each stage separately makes RAG much easier to operate.

Production RAG Architecture

A production RAG system should separate offline or asynchronous document processing from latency-sensitive query processing.

Document Ingestion

Document ingestion prepares information for retrieval:

Source Documents
      ↓
   Parsing
      ↓
  Cleaning
      ↓
  Chunking
      ↓
 Metadata
      ↓
 Embeddings
      ↓
 Search Index

The source may be a document repository, database, object storage system, CMS, support platform, source-code repository, or external API.

Ingestion should be idempotent. Reprocessing the same source version should not create duplicate chunks.

A chunk record can carry source-version information:

{
  "chunk_id": "policy-42-v7-003",
  "document_id": "policy-42",
  "source_version": 7,
  "embedding_version": 3,
  "checksum": "71f6...",
  "updated_at": "2026-08-20"
}

Checksums can help determine whether content actually changed before spending resources generating another embedding.

Failed ingestion should be retryable, observable, and eventually visible as an indexing backlog rather than silently leaving documents missing from search.

Query Processing

The online query path can include several retrieval and validation stages:

User Question
      ↓
Authentication
      ↓
Query Processing
      ↓
Vector + Lexical Retrieval
      ↓
Permission / Metadata Filters
      ↓
Reranking
      ↓
Context Builder
      ↓
LLM
      ↓
Output Validation
      ↓
Response

Authentication and authorization should happen before protected content is exposed to the model.

The retrieval layer can apply tenant and document-level access constraints. The model should never receive unauthorized chunks and then be asked not to mention them.

Timeouts should also be budgeted across the complete pipeline. If an application requires a response within three seconds, retrieval cannot independently consume three seconds and then leave unlimited time for reranking and generation.

A simplified latency budget might be:

These numbers are only illustrative, but the architectural principle is important: RAG latency is the sum of several systems, not only the LLM call.

Monitoring and Evaluating RAG

A RAG system needs both traditional infrastructure monitoring and AI-specific quality evaluation.

Infrastructure metrics include embedding latency, vector-search latency, reranking latency, index size, indexing backlog, LLM latency, token usage, and error rates.

Retrieval metrics answer a different question: did the system find the correct evidence?

Useful metrics include:

• Recall@k. Measures whether expected relevant documents appear among the top k retrieved results.
• Precision@k. Measures how many retrieved results are actually relevant.
• Ranking quality. Measures whether the strongest evidence appears near the top.
• Empty retrieval rate. Tracks queries for which no acceptable evidence is found.
• Stale-document rate. Detects retrieval of outdated content.

Generation should be evaluated separately. A system can retrieve the perfect document and still produce an incorrect answer.

Useful generation-level checks include:

• Groundedness. Are factual claims supported by retrieved evidence?
• Answer correctness. Does the response correctly answer the question?
• Citation correctness. Do citations point to sources that actually support the associated claims?
• Abstention quality. Does the model decline to answer when retrieval provides insufficient evidence?

An evaluation dataset should contain representative questions, expected answers where practical, and known relevant documents. This makes it possible to compare changes to embedding models, chunk sizes, search indexes, rerankers, prompts, and language models.

For example, if changing chunk size improves retrieval recall but increases context tokens by 80%, the trade-off becomes measurable instead of subjective.

End-to-end metrics also matter. A RAG assistant may have excellent retrieval recall but still fail to resolve customer requests. The most useful measurements ultimately connect retrieval and generation quality to the product outcome.

AI Monitoring and Evaluation covers production evaluation in greater detail.

Conclusion

Retrieval-Augmented Generation connects language models with external knowledge. Instead of expecting an LLM to contain every fact in its parameters, a RAG system retrieves relevant evidence and places that evidence into the model's context at inference time.

A strong RAG architecture depends on much more than a vector database. Document parsing, chunking, embeddings, lexical and semantic retrieval, metadata filters, reranking, context construction, authorization, generation, and evaluation all contribute to the final result.

The most important design principle is to treat retrieval and generation as separate systems. When an answer fails, determine whether the evidence was missing, retrieved incorrectly, ranked poorly, excluded from context, or misunderstood by the model.

RAG works best when the retrieval system finds a small amount of high-quality evidence and the language model is given a clear, bounded task for using that evidence.

More Articles to Read

• AI Embeddings
• Vector Databases for AI
• AI Tokens and Context Windows
• AI Prompt Engineering
• RAG vs Fine-Tuning