Production RAG Architecture: Ingestion, Retrieval, Citations, and Evaluation

A RAG prototype can look convincing with a folder of documents, an embedding call, a vector search, and a prompt that asks the model to answer from retrieved context. That proves the idea. It does not yet prove that the knowledge system will stay correct when data changes, permissions differ by user, indexes are rebuilt, or retrieval strategies evolve.

This guide treats production RAG as two independently operated systems: a knowledge plane and an online answer plane. The knowledge plane discovers, parses, chunks, versions, embeds, indexes, updates, and deletes knowledge. The online plane authorizes a query, retrieves candidates, fuses and reranks them, builds context, generates an answer, validates citations, and records evaluation and observability data.

This extends Production AI System Design: the model remains one component, while the knowledge system around it must be deterministic, observable, testable, and replaceable.

The companion implementation uses a deterministic local/offline harness. Any measurements reported below are local harness results, not benchmarks of OpenAI, Anthropic, AWS, Azure, Elastic, pgvector, embedding providers, vector databases, or production infrastructure.

A RAG Demo Is Not a Knowledge System#

A model call can be healthy while the knowledge path is wrong. The index can be stale. A deleted document can remain retrievable. A vector query can miss an exact identifier that lexical search would have found. A user can receive a chunk from another tenant. A generated answer can contain polished citations that do not actually support its claims.

These failures need their own contracts, metrics, and release gates.

A useful production mental model is:

Knowledge plane

Source → Parse → Normalize → Chunk → Metadata/Provenance → Embed → Index → Verify

Online answer plane

Query → Authorize/Filter → Retrieve → Fuse → Rerank → Build Context → Generate → Validate Citations → Return

Retrieval quality should be measured before answer quality so that a bad search result cannot hide behind fluent generation.

Define the Production RAG Contract First#

Start with application-level contracts rather than a vector-store SDK.

A retrieved chunk needs stable identity, source provenance, authorization attributes, version information, and retrieval metadata. A useful contract includes:

• source ID and source version
• chunk ID and chunk version
• content hash
• source URI
• title, section, page, or offsets when available
• authorization attributes
• retrieval method
• retrieval score
• index version

A citation should reference that evidence by stable identity. Rendering [1], footnotes, or source chips is a presentation decision. The underlying provenance should not depend on the display format.

The tested reference implementation keeps these boundaries in:

• src/contracts.ts
• src/ingestion/provenance.ts

The product contract should remain stable even if the search engine, embedding provider, reranker, or model provider changes.

Reference Production RAG Architecture#

A useful baseline architecture separates the knowledge pipeline from the online serving path.

The exact storage and provider choices can vary. The important property is that ingestion, retrieval, authorization, context assembly, generation, citations, and evaluation remain separate enough to test and observe independently.

Make Ingestion Replayable and Versioned#

Ingestion should be replayable and idempotent.

Reprocessing the same source should not create duplicate chunks. Transient failures should be retryable. Malformed documents should be quarantined. Updates and deletes should propagate deliberately.

A production path should expose explicit stages:

1. discover or receive a source change
2. fetch the source version
3. parse
4. normalize
5. chunk
6. attach metadata and provenance
7. create embeddings where required
8. write lexical and vector representations
9. verify retrieval
10. publish the intended index version

The reference implementation uses:

• src/ingestion/idempotency.ts
• src/ingestion/chunker.ts
• src/ingestion/provenance.ts

Version the knowledge you retrieve#

Track the transformations that can change retrieval behavior:

• parser version
• chunker version
• embedding version
• metadata-schema version
• index version

A content hash identifies unchanged input. A chunk version distinguishes the same source processed under a new strategy. An index version supports shadow evaluation, controlled cutover, and rollback.

Changing any of these can alter retrieval behavior even when the source documents are unchanged, so migrations should run through evaluation before cutover.

Treat Chunking as a Retrieval Decision#

There is no universally correct chunk size.

Small chunks can improve citation granularity but lose surrounding meaning. Large chunks can preserve context while diluting the matching signal and consuming more context space. Overlap can recover boundary information while also duplicating evidence.

Evaluate chunking against real queries using retrieval recall, duplicate-context rate, context size, and citation granularity.

Contextual metadata can restore meaning that is lost when a chunk is isolated from its parent document.

What the chunking experiment showed#

The deterministic local harness compared chunk configurations on the same controlled corpus.

Measured locally:

• Recall@3: 1.0
• duplicate-context rate: 0
• average context size: 38 units

No single chunk configuration is claimed as universally best. These numbers describe the local fixture only.

Start with Keyword and Vector Baselines#

Lexical retrieval remains useful for exact identifiers, API names, error codes, and domain terms.

Vector retrieval helps when the query and source express the same concept using different wording.

The tested implementations are:

• src/retrieval/keyword.ts
• src/retrieval/vector.ts

Measure both independently before deciding what additional complexity is justified.

What the retrieval experiment showed#

The same deterministic query set was evaluated with keyword, vector, and hybrid retrieval.

All three strategies retrieved the expected evidence on this small controlled corpus.

That result does not prove that the strategies are equivalent in production, and it does not prove that hybrid retrieval is inherently superior. It demonstrates why retrieval strategies should be independently measurable on the workload that matters to the product.

Combine Retrieval with Reciprocal Rank Fusion#

Hybrid retrieval combines independently ranked candidate sets.

Reciprocal Rank Fusion is useful as a baseline because it combines ranked lists without requiring their raw scores to share the same scale.

The reference implementation uses:

• src/retrieval/rrf.ts
• src/retrieval/hybrid.ts

A practical serving path is:

keyword candidates
        +
vector candidates
        ↓
      RRF
        ↓
candidate set
        ↓
optional reranker

In the local experiment, hybrid retrieval matched the simpler baselines instead of outperforming them. The production lesson is to keep retrieval modes observable and to enable additional complexity only when evaluation demonstrates a meaningful benefit.

Filter Before You Rank#

Authorization is a retrieval boundary.

Tenant, project, visibility, date, language, region, and confidentiality constraints should be applied before evidence reaches generation.

A chunk that the current user cannot access must not enter the answer context.

The tested implementation lives in:

src/retrieval/authorization.ts

What the authorization experiment showed#

The multi-tenant local harness tested authorized and unauthorized retrieval scenarios.

Measured result:

• unauthorized retrieval count: 0

This is a release-critical correctness check, not a retrieval-quality optimization.

A nonzero result should fail the release gate.

Rerank a Small Candidate Set#

Reranking belongs after inexpensive candidate generation.

The reranker should receive a small candidate set, and the system should measure incremental quality, latency, and cost rather than assuming that reranking is always beneficial.

The tested implementation lives in:

src/retrieval/reranker.ts

What the reranking experiment showed#

Measured locally:

The deterministic reranker did not improve MRR on the current corpus.

The measured timing is local harness overhead, not a provider benchmark.

The useful finding is architectural: a reranking stage should earn its place by improving the target workload. If it does not, the extra dependency and latency may not be justified.

Build Context, Do Not Dump Search Results#

Search results are not yet model context.

A deterministic context builder should:

• enforce a context budget
• deduplicate overlapping evidence
• preserve source diversity where appropriate
• control ordering
• retain evidence IDs
• support an insufficient-evidence outcome

The tested implementation lives in:

src/context/context-builder.ts

What the context-budget experiment showed#

The local harness compared context budgets of 20, 40, and 80 units.

The duplicate-context rate remained 0 in the exercised fixture.

Increasing the budget from 40 to 80 did not add evidence because the selected context already occupied 38 units. A larger context budget does not automatically produce better evidence.

These units are deterministic local harness units, not provider billing tokens.

Citations Are a Data Contract#

Citations should be built from evidence provenance, not invented by the model as formatting.

Keep source ID, source version, chunk ID, source URI, page/section/offset when available, and evidence identity through the pipeline.

Then validate that:

1. the cited source was actually retrieved
2. the citation references the correct source version
3. the source/chunk identity is valid
4. supported claims receive the required evidence mapping

The tested implementation uses:

• src/citations/citation-builder.ts
• src/citations/citation-validator.ts

What the citation experiment showed#

The deterministic evidence-ID checks produced:

• citation precision: 1.0
• citation coverage: 1.0

The test cases included a correct citation, a non-retrieved source, a wrong source version, and an uncited supported claim.

These are local deterministic checks of the fixture and citation contract, not model citation-quality benchmarks.

Evaluate Retrieval Before Answer Quality#

Retrieval and generation should have separate scorecards.

AWS Bedrock's RAG evaluation documentation separates retrieve-only evaluation from retrieve-and-generate evaluation. That distinction is useful even when a different stack is used.

Useful retrieval metrics can include:

• Recall@k
• MRR
• nDCG
• context relevance
• context coverage
• zero-result rate
• authorization correctness
• freshness correctness

The reference implementation includes:

src/eval/retrieval-metrics.ts

A failed retrieval test should not be hidden by a model that happens to produce a fluent answer from prior knowledge.

Evaluate the Grounded Answer#

Once retrieval passes, evaluate the answer separately.

Useful dimensions include:

• correctness
• completeness
• faithfulness
• citation precision
• citation coverage
• refusal behavior when evidence is insufficient

Some checks are deterministic. Some require human review. Model-based graders can also be used, but they should themselves be evaluated rather than treated as unquestioned truth.

The reference implementation includes:

• src/eval/citation-metrics.ts
• src/eval/release-gate.ts

Release criteria should remain independent enough that a strong score in one area cannot hide an authorization, freshness, citation, or retrieval regression.

Observe the Whole RAG Request#

A production trace should show more than model latency.

Capture structured events or spans for:

• query normalization
• authorization filters
• keyword retrieval
• vector retrieval
• candidate counts
• fusion
• reranking
• context assembly
• context size
• generation
• citation validation

Also record identifiers such as:

• request ID
• index version
• retrieval strategy
• top-k
• reranker version
• prompt/configuration version
• model route

This makes regressions diagnosable. A team should be able to distinguish a generation slowdown from a retrieval slowdown, or an answer regression from an index migration.

The end-to-end reference path is implemented in:

src/pipeline/rag-pipeline.ts

Freshness, Deletes, and Rebuilds Are Reliability Features#

Knowledge changes.

A production RAG system needs an explicit definition of freshness and deliberate delete propagation.

Track source timestamps and ingestion timestamps. Use durable delete signals so a rebuild cannot accidentally resurrect removed content.

For major parser, chunker, metadata, or embedding changes, build a shadow index, replay ingestion, run retrieval and end-to-end evaluation, compare the candidate version with the current version, and cut over only after the release gate passes.

What the freshness and delete experiment showed#

The deterministic harness exercised source update, source delete, replay, and shadow-index rebuild scenarios.

Measured locally:

• stale result count: 0
• deleted result count: 0
• local ingestion timing: ~0.41 ms

The timing is local harness timing, not a production freshness SLA or external indexing benchmark.

The important verified behavior is that stale and deleted content were not returned in the exercised scenarios.

Production Readiness Checklist#

Before shipping a RAG feature, verify the following.

Ingestion#

• [ ] Source identity and change detection are explicit.
• [ ] Ingestion is idempotent and replayable.
• [ ] Parser, chunker, embedding, metadata, and index versions are recorded.
• [ ] Update and delete propagation are tested.

Retrieval#

• [ ] Keyword retrieval has a measured baseline.
• [ ] Vector retrieval has a measured baseline.
• [ ] Hybrid retrieval is measured rather than assumed.
• [ ] Reranking is justified by measured incremental value.

Authorization#

• [ ] Access-control filters run before evidence reaches generation.
• [ ] Cross-tenant and restricted-content leakage tests exist.
• [ ] Unauthorized retrieval is a release-gating failure.

Context and citations#

• [ ] Context assembly has deterministic budget and deduplication rules.
• [ ] Evidence identity survives context assembly.
• [ ] Citations map to retrieved evidence.
• [ ] Wrong-version and non-retrieved citations are rejected.

Evaluation#

• [ ] Retrieval has its own evaluation set.
• [ ] Grounded answers have correctness, faithfulness, and citation checks.
• [ ] Release gates can fail independently on retrieval, citations, authorization, freshness, latency, and other configured thresholds.

Observability#

• [ ] Traces include retrieval strategy and index version.
• [ ] Candidate counts and reranking are visible.
• [ ] Context size and citation validation outcomes are observable.

Operations#

• [ ] Freshness expectations are explicit.
• [ ] Delete propagation is tested.
• [ ] Index migrations run through evaluation before cutover.
• [ ] Rollback remains possible.

If several of these answers are unknown, the system may still be a useful RAG prototype—but it is not yet a well-defined production knowledge system.

Final Reference Architecture#

The complete reference architecture combines knowledge ingestion, online retrieval, provenance, authorization, evaluation, and observability:

The key design choice is not any individual database, embedding model, or reranker.

It is the separation between mutable knowledge, deterministic retrieval controls, nondeterministic generation, evidence provenance, and release criteria.

The surrounding production pattern should remain recognizable:

versioned ingestion → authorized retrieval → measured ranking → deterministic context → grounded generation → validated citations → evaluation → controlled release

Reference Implementation#

This article is backed by the tested Production RAG Architecture Lab in ProdAI Stack Labs.

Lab source: Production RAG Architecture Lab

The implementation includes:

• src/contracts.ts
• src/ingestion/idempotency.ts
• src/ingestion/chunker.ts
• src/ingestion/provenance.ts
• src/retrieval/keyword.ts
• src/retrieval/vector.ts
• src/retrieval/rrf.ts
• src/retrieval/hybrid.ts
• src/retrieval/authorization.ts
• src/retrieval/reranker.ts
• src/context/context-builder.ts
• src/citations/citation-builder.ts
• src/citations/citation-validator.ts
• src/pipeline/rag-pipeline.ts
• src/eval/retrieval-metrics.ts
• src/eval/citation-metrics.ts
• src/eval/release-gate.ts

The verified commands recorded for the Lab are:

npm run typecheck
npm run lint
npm run test
npm run example:rag
npm run example:experiments
npm run example:diagrams

The local harness results reported in this article come from that implementation. They should be interpreted as reproducible implementation evidence, not external-provider benchmarks.

Sources and Further Reading#

• OpenAI — Retrieval
• OpenAI — File search
• OpenAI — Working with evals
• Microsoft — Hybrid search using vectors and full text in Azure AI Search
• Microsoft — Relevance scoring in hybrid search using Reciprocal Rank Fusion
• Elastic — Hybrid search
• pgvector — pgvector
• AWS — Evaluate the performance of RAG sources using Amazon Bedrock evaluations
• AWS — Review metrics for RAG evaluations that use LLMs
• Anthropic — Contextual Retrieval

git clone https://github.com/sarvab-dev/prodai-stack-labs
cd prodai-stack-labs/production-rag-architecture
npm install
npm test