Architecture - rag-pipeline-hallucination-mitigation

Executive Summary

A production-grade Retrieval Augmented Generation (RAG) pipeline with comprehensive hallucination detection and mitigation. Built from real-world deployments processing 2M+ claims/month (Cigna) and 500K+ tax transactions (Ceretax). The pipeline implements a four-stage architecture: document ingestion with intelligent chunking, hybrid retrieval (BM25 + dense vectors), grounded LLM generation with citation attribution, and multi-layer faithfulness validation using RAGAS, SelfCheckGPT, and NLI entailment. Supports deployment across AWS (Bedrock/OpenSearch), Azure (OpenAI/AI Search), and GCP (Vertex AI/Matching Engine).

Overview

Python

Language

AI/ML

Architecture Diagram

Documents
PDF, HTML, Markdown

Document Processor
Chunking + Embedding

Ingestion Layer

Vector Store
pgvector / OpenSearch / Matching Engine

BM25 Index
Sparse Retrieval

Hybrid Retriever
Dense + Sparse + Cross-Encoder Reranking

Retrieval Layer

Grounded Generator
GPT-4 / Claude / Gemini

<-->

Semantic Cache
Validated Answer Reuse

Generation Layer

Faithfulness Validator
RAGAS + SelfCheckGPT + NLI

<-->

Human-in-the-Loop
Low-Confidence Review

Hallucination Mitigation Layer

Prometheus Metrics
Latency, Tokens, Drift

Grafana Dashboards
Observability

Alerting
Quality Gates

Observability Layer

Component Breakdown

Component	File	Purpose	Key Dependencies
Document Processor	`src/ingestion/document_processor.py`	Intelligent document chunking with overlap, embedding generation, and vector store indexing	langchain, sentence-transformers
Hybrid Retriever	`src/retrieval/hybrid_retriever.py`	Combined BM25 sparse and dense vector search with cross-encoder reranking and relevance filtering	rank-bm25, pgvector, sentence-transformers
Grounded Generator	`src/generation/grounded_generator.py`	LLM generation with context injection, citation grounding, and source attribution	langchain-openai, anthropic, vertexai
Faithfulness Validator	`src/validation/faithfulness_validator.py`	Multi-method hallucination detection: RAGAS faithfulness scoring, SelfCheckGPT consistency, NLI entailment verification	ragas, nltk, transformers

Data Flow / Request Flow

Document Ingestion -- Documents are processed through intelligent chunking (configurable size/overlap), embedded using dense models (Titan/Ada/Gecko), and indexed in both vector and sparse stores.
Query Processing -- User queries are classified, optionally rephrased using HyDE (Hypothetical Document Embeddings), and expanded for better recall.
Hybrid Retrieval -- The retriever runs parallel BM25 sparse and dense vector searches, fuses results with reciprocal rank fusion, then reranks with a cross-encoder model.
Grounded Generation -- Retrieved context is injected into the LLM prompt with explicit grounding instructions. The generator produces responses with inline citations to source chunks.
Faithfulness Validation -- Responses undergo three-layer validation: RAGAS faithfulness scoring, SelfCheckGPT consistency sampling, and NLI entailment checking. Low-confidence responses are routed to human review.
Observability -- Token usage, latency, groundedness scores, and retrieval quality metrics are exported to Prometheus and visualized in Grafana dashboards with configurable alerting.

Security Controls

Control	Implementation
API Key Management	Environment variables via python-dotenv; no hardcoded secrets
Data Encryption	AES-256 at rest for vector stores; TLS 1.3 in transit for all API calls
PHI/PII Handling	Configurable redaction pipeline for healthcare and financial data
Access Control	RBAC for document corpus and query endpoints
Audit Logging	Structured logging of all queries, retrievals, and generation events
Input Validation	Pydantic models for all request/response schemas; prompt injection prevention
Quality Gates	Configurable faithfulness thresholds prevent low-confidence responses from reaching users
Compliance	HIPAA/HITRUST, FedRAMP High, SOC 2, NIST 800-53 alignment

Industry Adaptation

Healthcare

HIPAA-compliant clinical RAG with PHI redaction, medical terminology understanding, clinical trial document retrieval, and HITRUST-aligned audit trails.

Finance

Tax compliance RAG (Ceretax pattern) with SOX audit support, PCI-DSS data handling, regulatory filing assistance, and financial document analysis.

Legal

Contract analysis RAG with clause extraction, precedent retrieval, regulatory change tracking, and attorney-client privilege preservation.

Government

FedRAMP-authorized RAG with CUI handling, NIST 800-53 compliance, classified document retrieval controls, and FOIA response automation.

Insurance

Claims processing RAG (Cigna pattern) with policy document retrieval, claims adjudication support, and fraud detection integration.

Production Readiness Checklist

Configure vector store backend (pgvector/OpenSearch/Matching Engine) and verify connectivity
Load document corpus and validate chunking quality and embedding coverage
Set LLM API keys and test generation across target models (GPT-4/Claude/Gemini)
Calibrate faithfulness thresholds using labeled evaluation dataset
Configure cross-encoder reranking model and validate retrieval accuracy
Set up Prometheus metrics endpoint and Grafana dashboards
Configure alerting for groundedness score degradation and latency spikes
Enable semantic caching and validate cache hit rates
Run end-to-end evaluation with RAGAS benchmark suite
Deploy behind TLS-terminating reverse proxy with rate limiting
Configure log retention and audit trail for compliance requirements

Configuration / Environment Variables

Variable	Required	Default	Description
`OPENAI_API_KEY`	Yes	--	OpenAI API key for generation and embeddings
`ANTHROPIC_API_KEY`	No	--	Anthropic API key for Claude models
`VECTOR_DB_URL`	Yes	--	Vector store connection string
`EMBEDDING_MODEL`	No	text-embedding-3-small	Embedding model for document and query encoding
`RERANKER_MODEL`	No	cross-encoder/ms-marco-MiniLM-L-6-v2	Cross-encoder model for result reranking
`CHUNK_SIZE`	No	512	Document chunk size in tokens
`CHUNK_OVERLAP`	No	50	Overlap between chunks in tokens
`FAITHFULNESS_THRESHOLD`	No	0.85	Minimum faithfulness score for response delivery
`ENABLE_CACHE`	No	true	Enable semantic caching for validated responses
`PROMETHEUS_PORT`	No	9090	Prometheus metrics endpoint port

Deployment

Local Development

git clone https://github.com/kogunlowo123/rag-pipeline-hallucination-mitigation.git
cd rag-pipeline-hallucination-mitigation
python -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt
cp .env.example .env
# Edit .env with API keys and DB connection
python -m src.ingestion.document_processor
python -m src.main

Production Deployment

# AWS: Deploy with Bedrock + OpenSearch Serverless
# Azure: Deploy with Azure OpenAI + AI Search
# GCP: Deploy with Vertex AI + Matching Engine

# Docker deployment
docker build -t rag-pipeline .
docker run -e OPENAI_API_KEY=sk-xxx \
  -e VECTOR_DB_URL=postgresql://... \
  rag-pipeline

# Monitoring: Prometheus + Grafana stack
# See monitoring/dashboards/ for pre-built dashboards

Links

Repository	github.com/kogunlowo123/rag-pipeline-hallucination-mitigation
README	README.md
Hallucination Mitigation Guide	HALLUCINATION-MITIGATION.md
License	MIT License