Retrieval: RAG vs. Long Context, and When Each Wins#

TL;DR#

Retrieval (pulling in exactly the information relevant to the current step, out of a large external corpus) is one of two ways to give an agent access to more knowledge than fits in its context window - the other being long context, simply including everything. Neither is a universal winner: the choice depends on how large and how dynamic the corpus is, how precise the task's reasoning needs to be, and what you're willing to spend in latency and cost. A related but distinct problem - what an agent should remember about its own past sessions - is covered separately in prompting-context-engineering/11.

The idea#

An agent frequently needs access to more information than fits, or should fit, in its context window at once: a codebase with a million lines, a support knowledge base with ten thousand articles. The retrieval problem is: given a query, which small slice of a large, usually external and slowly-changing corpus is actually relevant right now?

The rise of models with much longer context windows (hundreds of thousands to millions of tokens) sparked a genuine practitioner debate around 2024-2025: if you can just paste the whole corpus into context, do you still need retrieval? The honest, research-backed answer as of 2025-2026 is: it depends on scale, dynamism, task type, and cost - not "long context wins" or "RAG wins" as a blanket rule.

How it works#

Retrieval-augmented generation (RAG): the mechanism#

RAG is the pattern of retrieving a small, relevant subset of a larger corpus at query time and inserting only that subset into the model's context, rather than including the whole corpus. The typical pipeline: the corpus is chunked into passages, each chunk is embedded into a vector representation, chunks are indexed for similarity search, and at query time the query itself is embedded and used to retrieve the top-k most similar chunks, which are then inserted into the prompt alongside the user's request.

Why this exists instead of just using long context. Even with a model that technically supports a million-token context, three problems remain: (1) cost and latency scale with input length - processing a million tokens on every turn is expensive and slow compared to processing a query plus a few retrieved passages; (2) context-rot and distraction effects (lesson 08) mean that even within an advertised context limit, recall precision degrades as more content is stuffed in, so a smaller, targeted context can produce more accurate answers than a larger, noisier one; and (3) many corpora are simply too large to fit even generous context windows - a large codebase or a company's full documentation set can run into tens of millions of tokens.

Worked example with numbers. Research comparing RAG against long-context approaches for repository-level code tasks found that long-context models can match or outperform RAG on small, well-structured repositories, where there isn't much irrelevant material to filter out and the whole thing plausibly fits in context anyway. But as repository size and structural complexity grow, RAG's advantage widens, because an increasing fraction of the raw corpus is irrelevant to any given query, and stuffing it all into context both costs more and actively degrades precision (context confusion, lesson 08). Separately, cost comparisons across typical production workloads have found RAG running roughly 8 to 82 times cheaper than a long-context approach that reprocesses the full corpus on every query - because RAG only pays the token cost for the retrieved passages plus the query, not the entire corpus, every single turn.

Where long context wins outright. For numerical or precise-reasoning-heavy tasks - for example, financial question-answering that requires tracking exact figures across a document - research has found long-context approaches can underperform RAG even when the whole document would fit, because the verbose, unfiltered content in a long context can obscure the specific facts the model needs to reason over precisely. This is a case where long context's "give the model everything" strategy actively works against precision. Long context tends to shine specifically for long, static documents where the whole thing is genuinely relevant (e.g., "summarize this contract") - not for large, mostly-irrelevant corpora.

The 2025-2026 synthesis: hybrid, not either/or#

By 2025-2026, the practitioner and research consensus converged on: RAG and long context are complementary tools, not competitors, and the more interesting design question is how to combine them rather than which one "wins." A common hybrid pattern: retrieve a modest set of highly relevant passages upfront for speed and precision, while giving the agent the option to pursue further autonomous retrieval or exploration at its own discretion if the upfront retrieval turns out to be insufficient - i.e., retrieval as an on-demand, agentic capability (a tool the agent can call), not a rigid one-shot pipeline bolted on before the model ever runs. This reframes retrieval from "a preprocessing step done to the model" to "a tool available to the agent," which composes naturally with the agentic loop (see the tool-use-agentic-loop subject).

Where memory differs from retrieval#

Retrieval, as covered in this lesson, is about an external corpus that exists independently of the agent - a codebase, a documentation set, a knowledge base someone else wrote. It's a fundamentally different problem from what an agent should remember about its own past sessions and experience, which is dynamic by nature, often needs to be written (not just read), and has its own failure modes (persisted errors compounding across sessions) that a static external corpus doesn't have in the same way. That problem - memory architecture - is covered in full in prompting-context-engineering/11, once you've also seen context compaction (lesson 10), since the two interact.

Pros#

RAG: scales to corpora far larger than any context window, keeps per-query cost proportional to what's actually relevant rather than the full corpus size, and updates cheaply (re-index changed documents) without retraining or re-processing everything.
Long context (no retrieval): simpler architecture (no indexing pipeline to build or maintain), avoids retrieval-quality failures (missed-relevant-chunk problems) entirely for corpora that genuinely fit, and can outperform RAG when the whole corpus is small, static, and uniformly relevant.

Cons#

RAG: retrieval quality is a new failure surface - if the relevant chunk isn't retrieved (bad embeddings, bad chunking, a query that doesn't semantically match the right passage), the model never sees it and can't compensate, no matter how capable it is; adds pipeline complexity (chunking strategy, embedding model choice, index maintenance) that a long-context approach avoids.
Long context (no retrieval): cost and latency scale with corpus size on every single query even when most of it is irrelevant; subject to the distraction and confusion failure modes from lesson 08 as corpus size grows, which can silently degrade precision even while the corpus technically fits.

Alternatives#

No retrieval - full-context stuffing every turn. Preferable only for genuinely small, static, uniformly-relevant corpora where the simplicity is worth more than the cost/precision trade-off; do not reach for this as corpus size or dynamism grows.
Fine-tuning the model on the corpus instead of retrieving from it. Bakes knowledge into weights rather than context, avoiding per-query retrieval cost entirely - but it's expensive to update (any change to the underlying knowledge requires re-training or re-tuning), doesn't give clean provenance for what the model "knows" versus hallucinates, and is a poor fit for knowledge that changes often. Preferable when the knowledge is genuinely stable and the update cost is acceptable, e.g., encoding a fixed style or domain vocabulary rather than a living knowledge base.

When to use it#

Reach for RAG when the knowledge source is large relative to the context window, changes over time, or is mostly irrelevant to any single query (a documentation set, a codebase, a support ticket history). Reach for plain long-context stuffing when the corpus is small, static, and largely all relevant to the task at hand - e.g., "answer questions about this one 20-page contract."

When NOT to use it#

Don't build a RAG pipeline for a corpus that comfortably and cheaply fits in context with room to spare - the added retrieval-quality failure surface and engineering cost isn't worth it below that scale. Don't treat retrieval as a substitute for precise reasoning tasks (exact numerical tracking, multi-step arithmetic over structured data) where even correctly retrieved passages can still get misread if the context around them is noisy - these tasks often benefit from narrowing scope even further than typical RAG chunk sizes, or from delegating to a tool that computes rather than an LLM that reads.

Key takeaways / mental model#

Retrieval answers "what's relevant to this query, out of a large external corpus?" - a different question from what an agent should remember about its own past (lesson 11). The scaling question is never "RAG or long context" in the abstract - it's a function of corpus size, how static or dynamic the knowledge is, how precision-sensitive the task is, and what latency/cost you can afford per query.

Self-check questions#

A team is building an agent that answers questions over a single, 15-page internal policy document that rarely changes. Would you recommend RAG, plain long-context inclusion, or something else? Justify with the trade-offs from this lesson, not just "the document is small."
A different team is building an agent over a 50,000-document, constantly-updated legal case archive, where queries need exact figures and dates. Name at least two distinct reasons why naive long-context stuffing would be a poor fit here, drawing on both this lesson and lesson 08.
Give a concrete example of a task where RAG's "only retrieve what's relevant" strategy would actively hurt task quality compared to giving the model the full document - and explain the underlying reason.
Explain, in your own words, why "retrieval" and "memory" are often implemented with overlapping machinery (embeddings, vector search) but answer genuinely different questions. What's one concrete scenario where an agent would need both at once?

References#

Xu et al., "In Defense of RAG in the Era of Long-Context Language Models," arXiv:2409.01666 - https://arxiv.org/pdf/2409.01666
Li et al., "Long-Context LLMs Meet RAG: Overcoming Challenges for Long Inputs in RAG," arXiv:2410.05983 - https://arxiv.org/pdf/2410.05983
Anthropic, "Effective context engineering for AI agents" (September 2025) - https://www.anthropic.com/engineering/effective-context-engineering-for-ai-agents