Do AI Models Really Need a 1M Context Window? What NVIDIA's Study Found

Do AI Models Really Need a 1M Context Window? What NVIDIA's Study Found

AI developers are locked in a race to expand context windows: the more information a model can process in one go, the better. But researchers at NVIDIA have challenged this logic. In their experiments, neural networks solved complex tasks just as well even when conversation history was compacted after every single step. Here is how that is possible, what other studies reveal, and how much context AI actually uses in production.

Why AI Context Windows Keep Expanding

Imagine working with ChatGPT on a complex engineering problem for hours. You discuss system architecture, fix bugs, upload documents, refine constraints, and gradually move toward the result.

To answer each new question, the model may need information from previous turns. The longer the dialogue runs, the more data it has to take into account.

That is context — the information accessible to a language model while processing a given request. The context window is measured in tokens: small chunks of text. Context includes user messages, instructions, documents, tool outputs, and an agent's history of prior actions.

A large window allows processing voluminous material without prior compression or slicing. Expanding its size is naturally seen as a major technological advantage. But does an AI model genuinely need to keep its entire history in front of it at all times?

In November 2026, NVIDIA researchers published a paper titled You Don't Need Long Context, attempting to answer this question experimentally.

The NVIDIA Study: Models Can Thrive on Compact Context

The authors tested how drastically shrinking context impacts performance on long and complex tasks. The benchmarks included SWE-bench Verified and Terminal-Bench 2.1, where autonomous agents must explore codebases, patch bugs, execute terminal commands, and build on prior outputs.

Three operational modes were compared:

  1. Full History. The model receives the entire cumulative conversation and every prior step's output.
  2. Periodic Compaction. Once a token threshold is reached, history is compressed into key facts and task state.
  3. Single-Action Compaction (SAC). After every single step, the agent saves a concise state summary and continues with a fresh history.

The third mode, single-action compaction (SAC), is particularly interesting. Instead of carrying an ever-growing chain of messages, the agent only needs to maintain a lightweight working memory: what was done, what conclusion was reached, and what next step is required.

Results: 7–9k Tokens Instead of a Massive Transcript

Model & BenchmarkFull HistorySingle-Action CompactionMedian Peak Context (SAC)
Claude Opus 4.5, SWE-bench Verified70.0%70.2%8k
Claude Opus 4.5, Terminal-Bench 2.147.2%51.8%9k
GPT-5.6 Sol, SWE-bench Verified84.4%82.4%7k
GPT-5.6 Sol, Terminal-Bench 2.187.6%84.3%8k

Source: NVIDIA, Table 2. SAC context represents the median peak active context per task, not cumulative token consumption across the session.

SWE-bench and Terminal-Bench accuracy with full history vs single-action compaction

Figure 1. In NVIDIA's experiments, compacting after every action left Claude Opus 4.5 performance virtually unchanged while GPT-5.6 Sol experienced a minor drop of 2.0 to 3.3 percentage points. Data source: NVIDIA, Table 2.

Claude Opus 4.5 maintained its score on SWE-bench Verified and improved by 4.6 percentage points on Terminal-Bench 2.1 under SAC. GPT-5.6 Sol proved slightly more sensitive, with scores dropping by 2.0 and 3.3 points respectively.

Throughout these experiments, the median peak active context with compaction remained between 7k and 9k tokens, even though individual tasks in uncompressed mode accumulated much longer histories. Maximum peak context under SAC dropped to 15k–37k tokens, compared to 130k–200k tokens under full history.

This is not proof that all tasks can be solved within an 8k token context window. Rather, it shows that with appropriate task organization, an agent rarely needs to see its entire operational trajectory at once.

How a Model Works Without Remembering the Entire Chat

Consider an AI agent debugging a checkout error in an e-commerce platform. During execution, it reads files, inspects logs, runs tests, and edits code multiple times. Full history preserves useful solutions, failed attempts, intermediate tool responses, and verbose technical logs.

Yet for the next step, the agent often needs nothing more than a brief note:

  • Goal: Fix checkout flow error.
  • Root Cause: Incorrect shipping cost calculation.
  • Fix: Updated calculation logic in the required module.
  • Validation: Primary test suite passes.
  • Next Step: Verify checkout with promotional discounts.

Diagram showing full agent history compressed into a concise state while maintaining access to source files

Figure 2. Context compaction principle: the agent maintains current task state and the next action, while raw code files, documents, and logs remain accessible for re-reading as needed. Schema prepared based on NVIDIA's approach.

An essential distinction here: underlying data is not deleted. Code, documents, and test results reside in an external environment where the model can reopen any file when necessary.

This approach mirrors a human engineer's scratchpad: we write down the main points and pull specific details from reference documents only when they are needed again.

Why Dumping an Entire Document Into Context Isn't Always Optimal

Compacting dialogue was only part of the research. NVIDIA's authors also evaluated selective reading of large documents. Rather than loading thousands of pages directly into context, the agent could inspect source files using search and read tools.

Consider a query about a company's revenue over five years. One could feed every financial report into the model at once. Alternatively, the agent can first locate relevant documents, open the specific revenue tables, and then compare the numbers.

In one experimental setup on the LOONG benchmark, Claude Opus 4.5's average score rose from 70.1 to 79.8, and GPT-5.6 Sol's rose from 79.3 to 84.6 when using selective reading and active context management.

This outcome does not imply that selective reading is universally superior — retrieval algorithms can overlook critical fragments. But it demonstrates that the total volume of source information and the amount of information that must be actively held in context at once are two distinct quantities.

When Long Context Degrades Output Quality

Lost in the Middle: Information Buried in Deep Prompts

In Lost in the Middle (2024), researchers studied how language models utilize long documents. Across many tested models, performance depended heavily on where relevant information appeared: models performed best when target facts sat near the beginning or end of the input text, and erred much more frequently when facts were placed in the middle.

Consequently, having the capacity to ingest large context does not guarantee uniform quality across all its parts.

NoLiMa: Complex Reasoning Suffers Without Direct Keywords

The NoLiMa benchmark (2025) examines tasks where one cannot simply match words in the text with the query. True semantic reasoning is required. At 32k tokens, ten out of twelve tested models retained less than half their baseline score on short context. In that experiment, GPT-4o's score dropped from 99.3% to 69.7%.

This test reflects specific model checkpoints and tasks. Nonetheless, it underscores the gap between technical input capacity and effective information synthesis.

Context Rot: Irrelevant Text Dilutes Focus

In Context Rot (Chroma, 2025), researchers evaluated 18 models and observed degradation in response quality as input length expanded. Difficulties became especially pronounced when distractor information and interdependent reasoning steps were present.

The collective takeaway from all three studies: information relevance, document structure, and data positioning strongly influence the final answer.

How Much Context Do Real Applications Actually Use? OpenRouter Data

Laboratory benchmarks assess model capabilities, but provider telemetry illustrates real-world workload dynamics.

In December 2025, OpenRouter and a16z published the State of AI study based on metadata from more than 100 trillion tokens. The authors demonstrated that average input tokens per request climbed from roughly 1.5k in early 2024 to over 6k by late 2025. Coding-related requests regularly exceeded 20k input tokens.

Separately, researchers measured the average sequence length of input + output: by late 2025, it exceeded 5.4k tokens. Because these metrics measure different things by definition, they should not be conflated.

The OpenRouter Rankings page provides sections dedicated to context sizes and session costs for coding agents. However, aggregate public metrics do not provide a basis for identifying the exact context length covering 90% of all user sessions.

Evaluating the 90th percentile of sessions requires completed dialogue-level data, not just individual API calls. A single conversational session can involve hundreds of requests repeatedly sending preceding history.

Harvard: Real-World Coding Agents Rely on Much Larger Contexts

At first glance, OpenRouter data might suggest long context is barely needed. But more specialized research reveals a different dynamic.

In November 2026, the Harvard Systems Group published From Requests to Sessions — an analysis of roughly 12k real-world coding agent sessions comprising 1.2 million model calls and 209 billion input tokens.

In these agentic traces, the median request contained 129.1k input tokens. The median volume of new, uncached tokens was just 491, with a median output length of 227 tokens.

This is explained by prompt caching: an AI agent repeatedly resends prior context to the model, but the vast majority can reuse a previously processed prefix. Among requests with caching metadata, cached tokens accounted for 97.4% of total input volume and 72.9% of estimated API costs.

Three different context measurements: OpenRouter, Harvard, and NVIDIA

Figure 3. According to OpenRouter, average input requests in late 2025 exceeded 6k tokens. In Harvard's coding agent sample, the median input request was 129.1k. In NVIDIA's experiment, median peak active context with compaction was 7–9k. These represent distinct samples and metrics; this chart does not depict an overall session size distribution.

On the surface, there appears to be a contradiction: NVIDIA demonstrates the viability of working with dramatically shorter active context, while production agents transmit massive histories.

In practice, this is a question of both quality and compute architecture. Reusing a cached prefix can be cost-effective, whereas frequent compaction can disrupt caching and require additional computation. There is currently no universally optimal approach for all models and workloads.

One cannot directly compare Harvard's 129.1k median token count in real queries with NVIDIA's 7–9k median peak in a controlled experiment as values from the same distribution.

Why Long Context Is Expensive

Context size directly impacts latency, compute load, and memory bandwidth. When generating new tokens, the model continues to attend over previous ones, and the cost of that computation depends on context depth and architectural details.

In NVIDIA's study, researchers measured the throughput of several open-weight models on H100 accelerators. Moving from a 16k to a 256k window caused the estimated compute cost per generated output token to multiply:

ModelCompute Cost Increase at 256k vs 16k
GLM 5.23.6x
Kimi K2.65.6x
Qwen 3 Coder Next7.0x

Relative compute cost of long context for three open-weight models

Figure 4. NVIDIA estimates based on measured H100 GPU throughput and GPU-hour costs. Displays relative token generation costs at 256k vs 16k context, not consumer API pricing tiers. Source: NVIDIA, Figure 1.

These figures cannot be interpreted as a promised subscription price drop or the cost of a specific API request. Actual bills depend on input and output tokens, caching, pricing tiers, hardware, and the number of additional agent steps.

However, for neural network providers, the potential effect is substantial: with effective active context management, one can reduce compute costs or serve significantly more requests on existing infrastructure.

Where Short Context Falls Short

NVIDIA's study also highlights counterexamples. On SWE-bench Verified, Qwen 3 Coder Next achieved 53.0% with full history, but dropped to 3.6% under an aggressive periodic compaction mode with a 16k limit and 12k threshold. This was a separate EoCC mode, distinct from the primary SAC mode benchmarked on Claude and GPT.

The authors attribute this performance collapse to lengthy agent trajectories: the model ran out of steps before making useful progress prior to the next compaction cycle. This finding emphasizes that context management strategies must be tailored to the specific model.

Large context also retains its value where an agent must simultaneously cross-reference numerous precise fragments and details. Any summary risks omitting a fact that becomes essential later. And selective search may fail to locate the target document.

Next-Gen AI: Evolving State Over Infinite Transcripts

NVIDIA researchers introduced the concept of Semantic State Space Models (SSSM). The core concept: the model receives a concise description of task state, makes a decision, executes an action, records the outcome, and updates its semantic state for the next step.

In schematic terms: current state → decision → action → result → updated state.

This remains an architectural hypothesis and a proposal for future model training. The paper does not present a fully trained standalone SSSM proving universal superiority over conventional long-context models.

Tiered memory architectures have been explored before. For example, MemGPT (2023) proposed separating active context from external storage, retrieving information on demand.

What This Means for Everyday ChatGPT and AI Users

The practical takeaway: long dialogues do not need to run indefinitely. When working on a project across hours or days, periodic state checkpoints help maintain output quality.

For example, upon concluding a milestone, one can ask the AI:

Provide a structured summary of our work so far: main objective, decisions made, key facts, constraints, identified issues, and next steps. Separately list details that must not be lost and questions that remain unresolved.

Such a summary is ideal for carrying over into a fresh chat. But remember: a summary does not replace the originals. Code, documents, source calculations, and complete test logs should always be preserved independently.

For AI application developers, the conclusion is even more vital. Teams should measure not just raw context window capacity and input token prices, but also retrieval accuracy, the frequency of re-reading source files, caching efficiency, and end-to-end task completion cost.

The Bottom Line: AI Needs More Useful Context, Not Just More Tokens

NVIDIA's research does not negate the advantages of long-context models. What it shows is that for a variety of complex tasks, the volume of information actively held by a model can be significantly reduced without catastrophic quality degradation.

Other studies confirm that massive context does not guarantee equally reliable utilization across all its parts. Meanwhile, real-world production traces from Harvard show that coding agents today still transmit hundreds of thousands of tokens per request.

The most promising paradigm combines multiple mechanisms: a lean current state, on-demand access to source materials, precise search, and efficient prompt caching.

In the future, the defining capability of leading AI systems may not be how many millions of tokens they can ingest at once, but how intelligently they select what is worth remembering at any given moment.

Sources

  1. NVIDIA (2026). You Don't Need Long Context.
  2. Harvard Systems Group (2026). From Requests to Sessions.
  3. OpenRouter / a16z (2025). State of AI: 100 Trillion Token Study.
  4. OpenRouter. Rankings.
  5. Liu et al. (2024). Lost in the Middle.
  6. NoLiMa (2025).
  7. Chroma (2025). Context Rot.
  8. MemGPT (2023).