A production-ready RAG system reliably finds the right passages in your content and turns them into accurate, cited answers, while respecting permissions, staying fast and keeping costs predictable. Getting there depends on five things: clean document parsing, sensible chunking, hybrid retrieval with reranking, careful prompting, and continuous evaluation. New to RAG? Read what RAG is first.
Reference architecture
Ingestion pipeline
- Connect sources: file storage, help centre, wiki, ticketing system, database.
- Parse: extract text, headings and tables; run OCR on scanned PDFs and images.
- Chunk: split along the document's structure and attach metadata such as source, section, date, language and access level.
- Embed and index: store vectors for meaning-based search and a keyword index for exact matches.
- Sync: re-index changed documents automatically and remove deleted ones.
Query pipeline
- Understand the question: rewrite follow-ups into standalone questions and detect filters such as product or date.
- Retrieve: run vector and keyword search together, filtered by the user's permissions.
- Rerank: reorder the top results with a reranking model and keep the best few.
- Generate: answer only from the supplied sources, cite them, and say so when the answer isn't there.
Chunking strategies that work
- Structure-aware chunks that follow headings and keep lists and tables whole.
- Contextual headers: prepend the document title and section path to each chunk so it still makes sense on its own.
- Parent-child retrieval: search small chunks for precision, then pass the larger surrounding section to the model.
- Overlap sparingly: a little overlap helps continuity; too much wastes tokens and duplicates results.
Retrieval: why hybrid search and reranking matter
Vector search finds passages that mean the same thing in different words. Keyword search (BM25) finds exact terms like product codes, names and error messages. Combining them, then reranking, usually gives noticeably better results than either alone. For questions that need several lookups or comparisons, agentic RAG lets an agent run multiple searches and combine the results.
Choosing a vector database
| Option | Consider it when |
|---|---|
| PostgreSQL + pgvector | You already run Postgres and want one database for data and vectors |
| Qdrant / Weaviate | You want an open-source, self-hostable vector engine with filtering and hybrid search |
| Pinecone | You prefer a fully managed service |
| Elasticsearch / OpenSearch | You already use them for keyword search and want to add vectors |
Evaluating a RAG system
- Retrieval quality: is the correct passage in the top results (recall@k)?
- Faithfulness: is every claim in the answer supported by the retrieved sources?
- Answer quality: is the answer correct, complete and useful?
- Operations: latency, cost per question and the rate of "I don't know" answers.
Build a test set from real questions, use automated scoring (tools like Ragas help) plus human review, and run it on every change.
Security and privacy
- Filter by permissions at retrieval time so users never see content they can't access.
- Treat retrieved text as data, not instructions, to reduce prompt-injection risk.
- Mask or exclude personal data that isn't needed, and log access.
- Use open-weight models on your own servers when data must stay in-house.
Keeping costs under control
- Send the model only the few best passages after reranking.
- Cache answers to frequent questions and reuse embeddings.
- Route simple questions to a smaller, cheaper model.
Key takeaways
- Most RAG quality problems start in parsing and chunking, not in the language model.
- Hybrid search with reranking and permission filtering is a strong default.
- Measure retrieval, faithfulness and answers separately, on every change.
Need RAG over your documents? Talk to AppifyDevs about RAG development, or see how RAG fits into building an AI agent.