Every few months someone declares RAG dead. The argument is always the same: context windows keep growing, so just stuff the whole corpus into the prompt and let the model sort it out. No chunking, no embeddings, no vector database, no retrieval logic. Delete half your pipeline.

I have built both. I have shipped a system that dumped 150K tokens of documentation into every request, and I have shipped chunked retrieval over the same corpus. The long-context version was simpler to build and worse to run. It was slower, it cost roughly 30x more per query, and on a specific class of questions it was measurably less accurate.

Long context did not make RAG obsolete. It moved the line. Below a certain corpus size and query volume, stuffing the context is now the right call and RAG is over-engineering. Above that line, retrieval still wins, and the reasons are economic and accuracy-based, not nostalgic. This post is about where that line sits and the four scenarios where you should still reach for chunked RAG.

The Naive Take: Just Put Everything in Context

Start with the argument at its strongest, because it is genuinely good for small problems.

You have a 40-page product manual. A user asks questions about it. The naive long-context approach:

manual = load_document("product_manual.md")  # ~25K tokens

def answer(question: str) -> str:
    return llm.generate(
        system="Answer using only the manual below.\n\n" + manual,
        messages=[{"role": "user", "content": question}],
    )

That is the whole system. No chunking strategy to tune, no embedding model to pick, no reranker, no chunk-boundary bugs, no “why did retrieval miss the obvious paragraph” debugging sessions. For a 25K-token manual answered a few hundred times a day, this is correct. Building a vector pipeline here is resume-driven development.

The problem is that people take this pattern, which is right at 25K tokens, and assume it scales linearly to 250K tokens and a million queries a day. It does not. Two things break: cost and accuracy. Both break quietly, which is the dangerous part.

Where It Breaks, Part One: Cost at Full Context

Long-context pricing is where the naive approach falls apart first, and the math is not subtle once you write it down.

The key fact people miss: you pay for every input token on every request. If you put 200K tokens of context in front of a one-line question, you are billed for 200K input tokens, not for the 3K tokens the answer actually depended on. RAG that retrieves the 8 relevant chunks sends maybe 6K input tokens for the same question.

Here is the comparison at illustrative rates. Take an input price of a few dollars per million tokens (typical for a mid-tier 2026 model) and 100,000 queries per day.

Approach Input tokens/query Tokens/day Relative cost
Full context (200K) ~200,000 20B ~30x
Full context (200K) + prompt cache ~200,000 (cached) 20B ~3-4x
Chunked RAG (top-8) ~6,000 0.6B 1x (baseline)

Prompt caching is the honest counterargument, so address it directly. If your 200K-token corpus is identical across requests, provider prompt caching lets you pay full price once and a large discount on cache hits after. That collapses the 30x gap to roughly 3-4x. Good, and for some workloads that is acceptable.

But caching has two failure modes people forget:

  • The corpus has to be stable. Cache entries have a short lifetime (minutes, not hours) and are evicted. If your corpus changes per user, per tenant, or per session, you rarely hit the cache and you are back to paying full freight. A per-customer knowledge base defeats caching entirely.
  • You still pay for cache reads. Cached input is cheaper, not free. At 200K tokens per query and high volume, “cheaper” is still a large bill for tokens the model mostly ignored.

RAG’s cost advantage is not a trick. You are sending fewer tokens because you retrieved only what matters. That advantage compounds with every single query, and it does not depend on your corpus staying frozen.

Where It Breaks, Part Two: Accuracy in the Middle

Cost you can model on a spreadsheet. The accuracy problem is worse because it is invisible until it bites you in production.

Long-context models do not attend uniformly across their window. The well-documented pattern, confirmed by needle-in-haystack testing across every major model generation, is that recall is high for information near the start and end of the context and sags in the middle. Put a critical fact at the 55% depth mark of a 200K-token window and the model is meaningfully more likely to miss it than if the same fact sat in the first 10K tokens.

It gets worse when there are multiple needles. Single-fact retrieval from long context is close to solved on modern models. But real questions often need three or four facts scattered across the corpus, plus reasoning over them. Multi-needle recall degrades faster than single-needle recall as context grows, and degrades further when the needles are similar to each other and to surrounding distractor text.

Needle recall vs position in a 200K-token window (schematic)

recall
100% |####                                    ####
     |#######                              #######
 90% |##########                        ##########
     |#############                  #############
 80% |################            ################
     |###################      ###################
 70% |##########################################
     +--------------------------------------------
      0%        25%       50%       75%      100%
                      depth in context

This is the part the “just use long context” crowd waves away. A benchmark that asks one clean question about one clearly stated fact will look great. Your production traffic, full of ambiguous multi-fact questions over a corpus stuffed with near-duplicate boilerplate, will not. Retrieval sidesteps the whole problem: if you only put the 8 most relevant chunks in front of the model, there is no lost middle, because there is no middle. Everything in the window is relevant and short.

The Four Scenarios Where Chunked RAG Still Wins

So when specifically should you keep RAG rather than stuff the context? Four cases, in rough order of how often they show up.

1. High query volume over a large, stable corpus

This is the pure cost case. If your corpus is large (hundreds of thousands of tokens or more) and you answer many queries against it, the per-query token savings from retrieval dominate everything else.

Do the arithmetic before you decide. The break-even is roughly: does the engineering cost of a retrieval pipeline get paid back by token savings within a quarter?

def full_context_cost(corpus_tokens, queries_per_day, price_per_mtok, cache_discount=1.0):
    # cache_discount: 1.0 = no caching, 0.1 = 90% cheaper on cache hits
    return corpus_tokens * queries_per_day * (price_per_mtok / 1_000_000) * cache_discount

def rag_cost(retrieved_tokens, queries_per_day, price_per_mtok):
    return retrieved_tokens * queries_per_day * (price_per_mtok / 1_000_000)

# 300K-token corpus, 50K queries/day, $3/Mtok input
fc = full_context_cost(300_000, 50_000, 3.0, cache_discount=0.15)  # generous caching
rag = rag_cost(7_000, 50_000, 3.0)
print(round(fc / rag, 1), "x more expensive")  # still multiples, even cached

Even with aggressive caching assumptions, high volume over a big corpus keeps RAG multiples cheaper. At scale, “multiples cheaper” is the difference between a viable product and a demo you cannot afford to launch.

2. Corpus that does not fit, or that changes per request

The context window is a hard ceiling. A 200K-token window cannot hold a 2M-token knowledge base, and enterprise corpora are routinely tens of millions of tokens. Even 1M-token windows do not hold a real company wiki plus its ticket history plus its codebase.

The subtler version of this case is the per-request corpus. If every user, tenant, or session sees a different slice of documents (their own files, their own permissions, their own history), you cannot precompute one big cached context. Retrieval is the natural fit: you filter to the documents this user is allowed to see, then retrieve within them. This is also how you enforce access control - you never put a document in the context that the requester should not see, which is much harder to guarantee if you are dumping a shared corpus.

3. Precision-critical multi-fact questions

When the answer depends on pulling several specific facts out of a large body of similar-looking text, the lost-middle and multi-needle degradation described above becomes a correctness bug, not a cost line. Legal clauses, medical records, financial disclosures, compliance rules - domains where retrieving the wrong-but-similar paragraph produces a confidently wrong answer.

Here retrieval plus reranking beats long context on accuracy, not just cost. A good reranker over retrieved candidates puts the genuinely relevant passages at the top of a short context, exactly where recall is highest, and drops the near-duplicate distractors entirely.

def retrieve_and_rerank(query, k_retrieve=50, k_final=8):
    candidates = vector_store.search(query, top_k=k_retrieve)
    scored = reranker.score(query, [c.text for c in candidates])
    ranked = sorted(zip(candidates, scored), key=lambda x: x[1], reverse=True)
    return [c for c, _ in ranked[:k_final]]  # short, dense, relevant

You are not fighting the model’s attention curve. You are handing it a small window where every token earns its place.

4. Low, predictable latency at scale

Time to first token scales with input length. Prefilling 200K tokens is real compute even when it is cached, and it shows up as latency. If you are serving an interactive product where p95 latency matters, sending 6K tokens instead of 200K is the difference between a snappy response and a visible pause on every query. Retrieval adds a vector lookup (single-digit milliseconds) and removes hundreds of thousands of tokens of prefill. That trade almost always favors retrieval for interactive workloads.

A Simple Decision Diagram

Put the four scenarios together and the routing logic is not complicated.

                 corpus fits in context?
                    /            \
                  no              yes
                  |                |
                RAG        query volume high
                             /          \
                          yes            no
                           |              |
                  changes per request?   |
                     /        \           |
                   yes         no         |
                    |          |          |
                   RAG   precision-       |
                          critical?       |
                          /     \         |
                        yes      no       |
                         |        \       |
                        RAG   latency-sensitive?
                                  /      \
                                yes       no
                                 |         |
                                RAG   long context is fine

The “long context is fine” leaf is real and it is bigger than it used to be. Small corpus, low-to-moderate volume, single-fact or synthesis questions, no hard latency budget: skip the vector database and stuff the context. You will ship faster and the accuracy will be fine.

The Hybrid That Most Teams Actually End Up With

In practice the interesting production systems are not pure. They retrieve to narrow a huge corpus down to a few thousand candidate tokens, then use a generous context window to reason over those candidates without aggressive chunk-size penny-pinching. Long context made RAG better here, not obsolete. You no longer have to cram everything into 512-token chunks and pray the boundary did not split the answer. You can retrieve larger, more complete passages because the window can hold them.

So the honest framing is not “long context vs RAG.” It is “retrieval decides what goes in the window, long context decides how much room you have once you are there.” They are layers, not competitors.

Honest Assessment

What actually changed with big context windows: the floor moved up. Problems that used to require a retrieval pipeline now do not. If your corpus is under about 100K tokens, your volume is modest, and you are not doing precision-critical multi-fact lookups, building RAG in 2026 is often over-engineering. Just put it in the context. This is a real and useful shift, and the RAG-forever crowd undersells it.

What did not change: at scale, retrieval is cheaper by multiples on every query, and caching narrows but does not close that gap because caching needs a stable, shared corpus that many real workloads do not have. Long context also still has a soft spot in the middle of the window and degrades on multi-needle recall, so for correctness-sensitive lookups over large, similar-looking corpora, retrieval plus reranking is more accurate, not just cheaper.

What to actually do: do not pick a side ideologically. Estimate your corpus size, your query volume, and whether your questions are single-fact or multi-fact and precision-critical. Run the cost arithmetic with your real numbers and honest caching assumptions. If you land in the small-corpus, moderate-volume, low-stakes quadrant, delete the vector database and enjoy the simpler system. If you land anywhere else - big corpus, high volume, per-tenant data, precision-critical, or latency-bound - keep RAG, and use the bigger context window to hold richer retrieved passages rather than to replace retrieval. The tool did not die. It moved.