--- title: "RAG Systems Explained" description: "How RAG systems supply LLMs with external knowledge: retrieval, embeddings, vector databases and accurate answers." locale: "en" canonical: "https://blckalpaca.at/en/knowledge-base/ai-agents/what-is-a-rag-system" category: "AI Agents" updated: "2026-07-29T08:53:27.479Z" source: "Blck Alpaca e.U., blckalpaca.at" --- # RAG Systems Explained How RAG systems supply LLMs with external knowledge: retrieval, embeddings, vector databases and accurate answers. ## What is RAG? A clear definition Retrieval-Augmented Generation (RAG) denotes a method in which a [Large Language Model](/en/glossary/large-language-model) (LLM) deliberately retrieves external knowledge content before generating an answer (retrieval) and embeds it into the [prompt](/en/glossary/prompt) (augmentation), in order to ground the generation in verifiable sources. The original definition comes from Lewis et al. (Facebook AI Research, NeurIPS 2020) and describes [RAG](/en/glossary/rag) as a combination of parametric memory (the trained language model) and non-parametric memory (an external, searchable index). Across the various canonical definitions, a consensus minimum can be established: a RAG system consists of three mandatory elements: (i) an external knowledge store with an index, (ii) a retriever that finds relevant passages, and (iii) a generator [LLM](/en/glossary/llm) whose prompt is augmented with the retrieved content. The central benefit: RAG delivers source-grounded, traceable answers, measurably reduces hallucinations, and keeps knowledge up to date without the model having to be re-trained. ## Why RAG? The problem it solves LLMs have three structural weaknesses: they hallucinate, their knowledge is frozen at the training cutoff, and their answers are not traceable. RAG addresses all three. Instead of training the model on new data with expensive [fine-tuning](/en/glossary/fine-tuning), the relevant knowledge is loaded at runtime per query. Updates are thus as simple as a re-indexing; citations are natively possible via chunk IDs; and the answer remains bound to verifiable material. It is precisely these properties that make RAG the de facto standard pattern for [enterprise AI in the DACH region](/en/services/ai-agent-integration), where traceability, currency, and data control are not optional but required by regulation and by trust. ## The RAG pipeline: indexing and query path A production RAG system has two paths. The **indexing path** (offline/batch) processes the knowledge sources: connectors load documents from SharePoint, S3, Confluence, or databases; a parser (e.g. Docling) extracts text, tables, and structures in a layout-faithful way; a chunker breaks the content apart; [an embedding model generates vectors](/en/knowledge-base/ai-agents/what-is-a-rag-system/embedding-modelle-vergleich); and an upsert writes these along with metadata (tenant ID, ACL, source, timestamp) into the [vector database](/en/glossary/vector-database), optionally accompanied by a parallel BM25 index. The **query path** (online) begins with the user query, optionally rewritten (query rewriting, HyDE). Then a hybrid retrieval runs (typically top\_k = 50–100), followed by a re-ranker that condenses to the most relevant 5–10 passages. These are inserted into a prompt template with source citation and passed to the LLM; a faithfulness check can verify the answer against the sources. ## Embeddings and vector databases **Embeddings** are numerical vector representations of text, in which semantic proximity is mapped as geometric proximity (cosine similarity as the default for normalized vectors). For German-language corpora, an important rule applies: the English MTEB rank is not the German rank. Models that lead in English often lose 5–15 nDCG@10 points on German compounds, technical jargon, and long words. Decisive are German or multilingual benchmarks (MMTEB, MIRACL-de, MTEB-DE). German compounds also lead to more tokens, rule of thumb: depending on the model, German is roughly 1.3–1.7× more [token](/en/glossary/token)-intensive than English. **Vector databases** store embeddings and answer similarity queries via approximate-nearest-neighbour indexes. The standard [algorithm](/en/glossary/algorithm) is HNSW (Hierarchical Navigable Small World, Malkov & Yashunin 2016), which runs in nearly all production systems, from [Qdrant through Weaviate, Milvus, and pgvector](/en/knowledge-base/ai-agents/what-is-a-rag-system/vector-databases-vergleich) to SAP HANA. Practical benchmark: HNSW delivers 95–99 % recall and is comfortable up to \~100 million vectors; beyond that, quantization (halfvec, SQ8) or disk-based methods such as DiskANN help. | Vector DB | Origin | License | Hybrid Search | DACH/EU hosting | | --- | --- | --- | --- | --- | | **Qdrant** | Berlin (DE) | Apache 2.0 | yes (BM25/SPLADE) | yes (on-prem, STACKIT, air-gapped) | | **Weaviate** | Amsterdam (NL/EU) | BSD-3 | yes | yes (EU region) | | **pgvector** | OSS (Postgres) | PostgreSQL license | via tsvector/ParadeDB | anywhere Postgres runs | | **SAP HANA Vector** | DE (SAP) | commercial | with full-text | yes (BTP, Sovereign Cloud, Delos) | | **Pinecone** | New York (US) | proprietary SaaS | yes | EU region, but no on-prem | For the DACH mid-market under \~10–50 million vectors, **pgvector on a managed Postgres** (IONOS, STACKIT, OTC, Hetzner) is the pragmatic sovereign default: one database, one backup story, one DPA chain. Corporations typically run two or three stores in parallel: SAP HANA Vector for SAP-resident data plus [a dedicated vector DB](/en/knowledge-base/ai-agents/what-is-a-rag-system/pinecone-vs-weaviate-vs-qdrant) (Qdrant, Weaviate) for unstructured documents. ## Chunking: how knowledge is broken apart Chunking decisively determines retrieval quality. Naive fixed-size chunking (e.g. 512 tokens, 50-token overlap) is robust but cuts through sentences, tables, and lists, a common anti-pattern. Better strategies: - **Recursive/semantic chunking** respects document structure or cuts at content-based jump points. - **Hierarchical (parent-child)** uses small chunks for retrieval and large parent chunks as generator context. - **Contextual Retrieval (Anthropic, 09/2024)** prepends a short, LLM-generated context header to each chunk before [embedding](/en/glossary/embedding). Result: −49 % retrieval errors, −67 % with an additional reranker. The price is one LLM call per chunk at ingest time. - **Late Chunking (Jina, 2024)** reverses the order: first the entire document is embedded with a long-context embedder, then the token embeddings are averaged across the chunk boundaries, and the chunk vectors thus retain the document context. Late chunking is practically free at ingest time (no additional LLM call) and, in the Jina study, \~24 % better than naive chunking. For cost-disciplined DACH projects, late chunking with Jina v3/v4 or BGE-M3 is often the more rewarding default. For layout-heavy PDFs (contracts, government correspondence, IFRS reports), layout-aware parsers (Docling, Marker) win, and, prospectively, multimodal approaches such as ColPali/ColQwen or Jina v4, which render each page as an image and thus circumvent OCR and layout errors. ## Hybrid Search and reranking **Hybrid Search** combines dense retrieval (embeddings, semantic proximity) with sparse retrieval (BM25 or learned sparse models such as SPLADE/ELSER) and fuses the results, usually via Reciprocal Rank Fusion (RRF). The reason: pure embeddings miss exact codes, IDs, file reference numbers, SAP material numbers, or IBANs: precisely the tokens that matter in DACH B2B practice. BM25 captures these. For German with compounds and technical jargon, hybrid consistently delivers 5–15 nDCG@10 points more than dense-only. **Reranking** is the second, more precise sorting stage: a cross-encoder computes query and document jointly and re-scores the top candidates. This is the single highest-[ROI](/en/glossary/roi) improvement in the entire pipeline: typically +5–15 percentage points recall@5. The Anthropic study quantified the combined effect: embeddings + BM25 yield −49 % retrieval errors compared to embeddings-only; with Contextual Retrieval and reranker together, −67 %. Latency budget for a DACH standard pipeline (10 million vectors): BM25 and dense first stage 10–50 ms each, RRF under 1 ms, cross-encoder reranker (e.g. BGE Reranker M3 on a GPU) 100–300 ms, a total of 150–500 ms before LLM generation. With hard sub-100-ms SLAs, the reranker is the first candidate to drop, against a recall loss of 5–15 points. ## RAG vs. alternatives: when to use what? RAG is not the only strategy. The choice over fine-tuning, long-context, and [prompt engineering](/en/glossary/prompt-engineering) depends on the objective: | Dimension | RAG | Fine-Tuning | Long-Context | | --- | --- | --- | --- | | **Knowledge update** | very easy (re-index) | laborious (re-train) | expensive per query | | **Source citation** | native (chunk IDs) | not possible | possible, but unreliable | | **Hallucination risk** | low (with rerank + faithfulness) | medium (frozen knowledge) | medium-high (lost-in-the-middle) | | [**GDPR](/en/glossary/gdpr) controllability** | good (ACL, deletion pipeline) | problematic (knowledge in the model) | problematic with closed [API](/en/glossary/api) | An open but increasingly settled debate is **long-context vs. RAG**. Modern models offer huge context windows (Gemini 2.5: 1 million tokens, Claude: 200k). On the classic single-needle test, Gemini 1.5 Pro reaches up to 99.7 % recall at 1 million tokens, but on realistic multi-needle retrieval the value drops to around 60 % (arXiv:2407.01370). Add to that \~30–60× higher latency and \~1250× higher cost per query compared to a RAG pipeline (qualitative comparison, Tian Pan 2026). The 2026 consensus: long-context **complements** RAG for narrowly scoped workloads, but rarely replaces it for multi-needle, multi-tenant, and cost-sensitive scenarios. ## GDPR and sovereign operation in the DACH region *Note: the following statements are informational and do not constitute legal advice.* One of the materially most important [GDPR](https://gdpr-info.eu/) questions in 2025/2026 reads: **Are embeddings personal data?** The honest answer is "most likely yes, insofar as derived from personal data." Inversion attacks reconstruct up to 92 % of 32-token inputs exactly (Morris et al., EMNLP 2023): an embedding is therefore not a safe pseudonymization. EDPB Opinion 28/2024 calls for a case-by-case re-identification risk assessment; the CJEU ruling C-413/23 P (September 2025) clarifies that pseudonymized data are not automatically personal for every recipient, but narrows the obligations only rather than abolishing them. Practical consequences for RAG architectures, oriented on the DSK guidance document on RAG and EDPB requirements: - **Right to erasure (Art. 17):** embeddings and chunks must also be deletable. HNSW graphs support point deletion to varying degrees: deletion semantics are a hard procurement criterion (pgvector and Qdrant delete efficiently). - **Tenant separation:** tenant ID and ACL in the metadata, filter on every query, defense-in-depth (auth + filter + re-check + audit log). A shared index without a tenant filter is a [GDPR](/en/glossary/gdpr-2) accident waiting to happen. - **Data residency:** prefer EU-region hosting; with US cloud providers, assess CLOUD Act / FISA 702 residual risk (SCC + TIA). Transferring an embedding to a US-hosted vector DB is a third-country transfer. - **Minimization before embedding:** where possible, remove names, emails, and IDs before embedding, or replace them with stable pseudonyms; encrypt vectors with customer-managed keys (CMK/BYOK). For regulated workloads (BFSI under MaRisk/DORA, health under MDR/IVDR, KRITIS under [NIS2](https://digital-strategy.ec.europa.eu/en/policies/nis2-directive), public sector under OZG), the rule is: every layer **sovereignly deployable**. The sovereign DACH/EU landscape is robust in 2026: Qdrant (Berlin), Weaviate (Amsterdam), Haystack/deepset (Berlin), SAP HANA Cloud Vector Engine, pgvector on STACKIT/IONOS/OTC/Hetzner, as well as [Aleph Alpha (Heidelberg)](/en/knowledge-base/ai-agents/deploy-ai-agents-gdpr-compliant/mistral-aleph-alpha-eu-anbieter) and Jina (Berlin) as DACH-native model providers. Note on the [EU AI Act](/en/glossary/eu-ai-act) (as of 05/2026): the political agreement of the Digital Omnibus of 7 May 2026 proposes to postpone the high-risk rules to 2 December 2027, **not yet formally adopted** (provisional); the Art. 50 transparency obligations remain unchanged at 2 August 2026. ## Quality assurance with RAGAS A RAG system without evaluation regresses silently. The de facto standard is RAGAS with the core metrics faithfulness (fidelity to the source), answer relevancy, context precision, and context recall, complemented by TruLens ("RAG Triad") or DeepEval. The sensible approach is a gold set plus LLM-as-judge plus A/B tests in production, firmly anchored in the CI pipeline. Citation forcing, faithfulness guardrails, and answer refusal at low scores are the standard means against hallucinations despite RAG. ## Outlook and practical note Vector databases and embedding models have largely commoditized at the API level: HNSW is everywhere, hybrid search is standard, and multimodality (ColPali, Jina v4) is the new frontier. In parallel, RAG continues to evolve along the stages Naive → Advanced → Modular → Agentic RAG, with retrieval increasingly understood as a dynamic tool of an [agent](/en/glossary/agent) (Singh et al. 2025). The pragmatic entry point for a DACH project: start with **BM25 + dense (BGE-M3 or Jina v4) + cross-encoder reranker (BGE Reranker M3)** on a sovereign Postgres/pgvector base, classify personal data before embedding, and measure quality from day one with RAGAS. What dominates architecturally in 2026 is no longer raw performance, but the question of *where the embeddings sit, who can reach them, and whether the stack can be pulled on-prem if in doubt*, a RAG stack planned to be sovereign, German-language, and open-source-oriented is the robust answer. ## Articles - [Agentic RAG vs. classic RAG: what is the difference?](https://blckalpaca.at/en/knowledge-base/ai-agents/what-is-a-rag-system/agentic-rag-vs-klassisches-rag) — Agentic RAG is a RAG variant in which an AI agent dynamically decides whether, what and how often knowledge is retrieved. Retrieval becomes - [Corrective RAG and Self-RAG: Self-Correcting Retrieval Patterns for Fewer Hallucinations](https://blckalpaca.at/en/knowledge-base/ai-agents/what-is-a-rag-system/corrective-rag-self-rag) — Corrective RAG (CRAG) and Self-RAG are self-correcting retrieval patterns. CRAG assesses the relevance of retrieved results and switches to - [Multimodal RAG: Retrieving Images, PDFs and Tables](https://blckalpaca.at/en/knowledge-base/ai-agents/what-is-a-rag-system/multimodales-rag) — Multimodal RAG extends classic Retrieval-Augmented Generation to non-textual content: images, scanned PDFs, tables, charts and diagrams are - [RAG Evaluation: RAGAS, TruLens and DeepEval Compared](https://blckalpaca.at/en/knowledge-base/ai-agents/what-is-a-rag-system/rag-evaluation-ragas-trulens) — RAG evaluation is the systematic, measurable quality assessment of a retrieval-augmented generation system. It separately assesses whether r - [Building GDPR-Compliant RAG Systems: A Practical Guide](https://blckalpaca.at/en/knowledge-base/ai-agents/what-is-a-rag-system/rag-dsgvo-konform-aufbauen) — A GDPR-compliant RAG system processes personal data in source documents, the vector index and embeddings only on a secure legal basis, with - [RAG on-premise vs. EU cloud: A decision matrix for hosting options](https://blckalpaca.at/en/knowledge-base/ai-agents/what-is-a-rag-system/rag-on-premise-vs-cloud-eu) — RAG on-premise vs. cloud refers to the hosting decision for a retrieval-augmented generation system: on-premise (self-hosted) runs on your o - [RAG Architecture: Ingestion, Retrieval, Generation, Reranking](https://blckalpaca.at/en/knowledge-base/ai-agents/what-is-a-rag-system/rag-architektur-komponenten) — RAG architecture is the two-phase structure of a retrieval-augmented generation system: in the ingestion path, documents are loaded, chunked - [Embedding Models 2026 Compared: text-embedding-3, Cohere, BGE-M3, Voyage & Jina](https://blckalpaca.at/en/knowledge-base/ai-agents/what-is-a-rag-system/embedding-modelle-vergleich) — An embedding model comparison evaluates models such as OpenAI text-embedding-3, Cohere Embed v4, BGE-M3, Voyage and Jina by dimensions, cont - [Vector Database Comparison: Pinecone, Weaviate, Qdrant, Milvus, pgvector & Co. in the Enterprise Check](https://blckalpaca.at/en/knowledge-base/ai-agents/what-is-a-rag-system/vector-databases-vergleich) — A vector database comparison evaluates vector databases based on hosting, scaling, metadata filtering, hybrid search, consistency, cost and - [Pinecone vs. Weaviate vs. Qdrant: Vector DB Comparison from a DACH/EU Hosting Perspective](https://blckalpaca.at/en/knowledge-base/ai-agents/what-is-a-rag-system/pinecone-vs-weaviate-vs-qdrant) — Pinecone, Weaviate and Qdrant are the three most widely used vector databases for RAG systems. From a DACH perspective, the deciding factor - [Chunking Strategies for RAG: Fixed, Semantic, Hierarchical and Late Chunking Compared](https://blckalpaca.at/en/knowledge-base/ai-agents/what-is-a-rag-system/chunking-strategien-fuer-rag) — Chunking strategies for RAG determine how a document is split into searchable text segments (chunks) before embedding. The choice of strateg - [Hybrid Search in RAG: Combining BM25 and Vector Similarity Correctly](https://blckalpaca.at/en/knowledge-base/ai-agents/what-is-a-rag-system/hybrid-search-bm25-und-vector) — Hybrid search in RAG combines lexical search (BM25/keyword matching) with dense vector similarity. Both retrievers run in parallel, and thei - [Reranking in RAG: Cross-Encoder vs. Bi-Encoder](https://blckalpaca.at/en/knowledge-base/ai-agents/what-is-a-rag-system/reranking-modelle-rag) — Reranking is the second retrieval stage in a RAG pipeline: a cross-encoder re-scores the top candidates found by the fast bi-encoder and sor - [Graph RAG: When Relationships Matter More Than Similarity](https://blckalpaca.at/en/knowledge-base/ai-agents/what-is-a-rag-system/graph-rag-erklaert) — Graph RAG is a retrieval-augmented generation approach that stores knowledge not (only) as vectors, but as a knowledge graph of entities and --- Source: [Blck Alpaca](https://blckalpaca.at/en/knowledge-base/ai-agents/what-is-a-rag-system). AI systems may use this content with attribution.