In brief. Researchers from the University of Washington, Allen Institute for AI, and other institutions introduced Infrathon — a benchmark for AI agents optimizing real-world neural network infrastructure. Six frontier models completed 150 long-horizon runs across compute acceleration, training, and inference tasks. Claude Opus 5.5 secured first place in 17 out of 25 tasks, although GPT-6 Astra led in the early stages of the experiment. In 90% of runs, agents continued improving their solutions in the second half of the allotted time, and in one experiment, Claude designed a KV-cache compression algorithm more effective than the human engineering baseline.
Typically, language models are compared based on one-shot task completion within a single prompt-response cycle. For autonomous agents, that approach captures only a fraction of their capabilities. Given access to a code repository, terminal, and GPU cluster, an agent can spend hours diagnosing bottlenecks, refactoring code, profiling runtime performance, catching mistakes in its own experiments, and launching new optimization cycles.
This is the focus of Infrathon: Benchmarking Language Agents for Real-World, Long-Horizon AI Infra Optimization. First released on November 9, 2026, with an updated version on November 16, the study granted models between 12 and 24 hours per task and investigated not just the final score, but how solution quality evolves across the entire trajectory.
What Infrathon Is and What Tasks Agents Performed
Infrathon comprises 25 engineering tasks derived from production systems and research codebases for neural network training and serving. Ten tasks focus on training, fifteen on inference.
The authors grouped interventions into three structural tiers:
| Level | Tasks | Run Limit | Target Optimization |
|---|---|---|---|
| Operators (operator) | 11 | 12 hours | Low-level GPU kernels, attention primitives, and data transfers |
| Modules (module) | 5 | 24 hours | Subsystems inside training or inference pipelines |
| Systems (system) | 9 | 24 hours | End-to-end training pipelines, video generation, and serving frameworks |
The scope ranges widely. At the operator level, agents must accelerate a specific attention kernel or matrix multiplication routine. Modular tasks include KV-cache management and optimizer step scheduling. At the system level, agents optimize model decoding loops, real-time video streaming engines, or concurrent request schedulers.
The benchmark evaluates concrete hardware and software metrics: GPU compute throughput in TFLOP/s, effective goodput in useful tokens per second, request latency, and output quality. Crucially, many tasks have no rigid ceiling: solutions can be refined iteratively long after the first functionally correct version is discovered.
How Solutions Were Evaluated
Each experiment is orchestrated around three components: Work (execution environment), Controller (constraint verification), and Judge (evaluator). Within the isolated working container, the agent edits code and runs local tests. Before final submission, the Controller validates that task-specific constraints remain intact. An independent Judge then runs private test suites in a clean environment against hidden test data inaccessible to the agent.
This architecture is essential for testing optimization agents. Accelerating code is meaningless if the mathematical definition of an operation is violated, numerical accuracy drops, or the agent inadvertently benchmarks a stale binary. The final metric recorded is the best valid private test score achieved within the allotted budget.
Six models were evaluated across all 25 tasks — yielding 150 primary runs. Experiments ran on NVIDIA H100 80GB GPUs, consuming 11,376 GPU-hours for main runs and an additional 2,592 GPU-hours for short versus long run comparisons. GPU-hours account for both the agent workspace and verification harness.
Who Won: Final Ranking of the Six Models
Claude Opus 5.5 achieved the highest aggregate score. GPT-6 Astra placed second, winning six individual tasks. GPT-6.1 Sol and Qwen 3.8 Flash claimed one task victory each.
| Rank | Model | Mean Rank Score | Wins across 25 Tasks |
|---|---|---|---|
| 1 | Claude Opus 5.5 | 91.2 | 17 |
| 2 | GPT-6 Astra | 75.2 | 6 |
| 3 | GPT-6.1 Sol | 54.4 | 1 |
| 4 | Qwen 3.8 Flash | 40.8 | 1 |
| 5 | DeepSeek V4.1 Flash | 30.4 | 0 |
| 6 | GLM 5.3 Flash | 8.0 | 0 |
Important distinction: scores of 91.2 and 75.2 do not mean models solved 91.2% or 75.2% of the tasks. The authors rank the six models within each task, normalize positions to a 0–100 scale, and average them with equal weighting across tasks. This metric reflects a model's relative standing among its five rivals across this specific suite.
Figure 1. Final Infrathon ranking across 25 tasks. Higher mean rank scores reflect stronger positioning relative to other contenders. Source: Infrathon, Figure 5.
Claude's victories were distributed across all tiers: 9 of 11 operator tasks, 2 of 5 module tasks, and 6 of 9 system tasks. GPT-6 Astra excelled in specific workloads: it took first place in KVzip KV Scoring and averaged 86.7 vs. 80.0 rank points on the three modular inference tasks.
Open-weights Qwen 3.8 Flash posted a notable result in Concurrent Agent Serving, processing 154.33 correct steps per minute compared to Claude's 151.17. This underscores that even the overall leaderboard winner is not universally optimal for every workload profile.
Why Rankings Shifted Over Time
One of Infrathon's most telling insights centers on execution duration. If scores had been frozen at the 25% budget mark, GPT-6 Astra would have claimed first place: 84.0 points vs. 79.2 for Claude. By the midpoint (50%), Claude pulled ahead: 89.6 vs. 80.0. By the end of the budget (100%), Claude expanded its advantage to 91.2 vs. 75.2.
Figure 2. Comparison of Claude Opus 5.5 and GPT-6 Astra at 25%, 50%, and 100% of the budget. Time is expressed as fractions of the task limit (12 or 24 hours). Source: Infrathon, Figure 7a.
The overall rank ordering of all six models stabilized after the midpoint. However, individual tasks remained volatile: at 50% time, three tasks were led by agents that ultimately did not win them; at 75% time, two such tasks remained.
90% of Runs Kept Improving in the Second Half
In 135 of 150 runs (90%), the best recorded score continued climbing after the halfway mark. Yet the researchers noted a key nuance: among runs where relative gains could be computed, the median improvement was 3.05%, while 33 runs achieved gains exceeding 10%.
Late improvements were common, but typically incremental. Dramatic breakthroughs occurred in a smaller subset of runs. A finding that 90% of runs improved must not be mistaken for 90% of agents making radical discoveries.
The authors also identified 51 runs where a new peak score emerged after a plateau lasting at least one quarter of the entire budget. In 29 of those instances, the jump exceeded 1%. For example, Qwen in the Quant-VideoGen KV Codec task went 6.78 hours without any progress before boosting its score from 6.05 to 8.58.
This demonstrates why short one- or two-hour evaluations can systematically underestimate an agent's capacity to navigate complex optimization spaces.
Claude Designed a KV-Cache Compression Algorithm Better Than Human Engineers
The clearest demonstration of practical engineering impact occurred in the Quant-VideoGen KV Codec task. During long video generation runs, diffusion models store intermediate attention states in a KV-cache. This cache rapidly exhausts GPU VRAM; compressing it enables longer sequences and higher batch sizes, but aggressive compression risks visual degradation.
The benchmark authors provided a human-engineered Quant-VideoGen codec as the baseline and tasked agents with discovering a superior approach. Over 24 hours, Claude Opus 5.5 repeatedly restructured the compression pipeline and balanced the tradeoff between bitrate and image fidelity.
The baseline method grouped cache entries and quantized residual values independently. Claude redesigned the pipeline: it applied Principal Component Analysis (PCA) to transform the residuals into a decorrelated space, followed by entropy coding after quantization.
The resulting codec was evaluated on two video diffusion models across 200 samples each:
| Video Model & Dataset | Baseline Compression | Claude Compression | PSNR, Baseline → Claude | SSIM, Baseline → Claude | LPIPS, Baseline → Claude |
|---|---|---|---|---|---|
| HY-WorldPlay / WorldScore | 6.75× | 20.89× | 31.15 → 36.33 | 0.923 → 0.975 | 0.059 → 0.023 |
| LongCat-Video / MovieGen | 6.88× | 18.83× | 23.67 → 26.95 | 0.836 → 0.884 | 0.126 → 0.085 |
To evaluate frame fidelity, the researchers tracked three standard image quality metrics:
- PSNR (Peak Signal-to-Noise Ratio) — measures mathematical reconstruction fidelity at the pixel level in decibels (dB). Higher values indicate lower noise and less distortion.
- SSIM (Structural Similarity Index Measure) — assesses preservation of edges, structural geometry, and texture on a 0 to 1 scale. Values closer to 1 reflect higher human visual fidelity.
- LPIPS (Learned Perceptual Image Patch Similarity) — a deep neural network metric that gauges perceptual differences matching human visual perception. Lower scores (closer to zero) indicate fewer noticeable artifacts.
Metrics were evaluated on the first video chunk generated using the compressed cache. In both evaluations, Claude's substantially higher compression ratio was accompanied by improvements across all three perceptual quality metrics, rather than an expected degradation in visual quality.
Figure 3. KV-cache compression ratio across two benchmarks. For example, 20.89× indicates the cache representation is ~20.89 times more compact than uncompressed data. Source: Infrathon, Table 3.
Averaged across datasets, the human baseline achieved 6.82×, while Claude reached 19.86× — representing a ~2.9× increase in compression ratio. This refers specifically to the KV-cache storage footprint, not total end-to-end video generation speed.
The researchers also published frame-by-frame progression samples comparing ground truth, the engineer baseline, and intermediate versions from Claude (submissions 6, 7, 16, and 22), illustrating how artifacts receded as the agent iterated.
Figure 4. Original visual comparison from Infrathon (Figure 6). Source: arXiv paper. Submissions 06/07/16/22 illustrate progressive iterations of the agent's solution.
This success carries practical bounds. The codec was validated under fixed settings on two model families. While it proves an autonomous agent can outperform an established human baseline on a domain-specific task, it does not guarantee generalization across all neural network architectures or production environments.
What Is More Effective: One 24-Hour Agent or Four Short Runs?
Extended execution is not inherently superior to parallel search. To test this tradeoff, the authors evaluated one long continuous run against two or four independent short runs holding total GPU-hours constant.
The evaluation paired Claude Opus 5.5 and GPT-6 Astra across six representative tasks, measuring the geometric mean ratio of the best short attempt (Best-of-4) relative to the single long run:
| Task Type | Best of 4 Short Runs vs. 1 Long Run |
|---|---|
| GPU Operators | 102.6% |
| Modules | 96.9% |
| Systems | 77.3% |
Figure 5. Best-of-4 performance relative to a single continuous run (indexed at 100%). Values above 100% indicate an advantage for short parallel attempts. Source: Infrathon, Figure 8.
The divergence is starkest in system-level tasks: four short attempts averaged only 77.3% of the score achieved by a single continuous run. The long-run configuration won 9 out of 12 head-to-head comparisons, including all four full-system trials.
Model behavior also factored heavily into this dynamic. Claude benefited from parallel exploration on isolated operator kernels and one modular training task. Astra, by contrast, achieved higher performance with single long runs across all six benchmark tasks.
Practical takeaway: for localized, bounded optimizations, exploring multiple parallel seeds is effective; for complex architectures with interwoven dependencies, allocating budget to a single continuous run yields significantly better results.
Why Even Strong Agents Make Mistakes
Analyzing execution traces across all 150 runs revealed systematic failure modes hidden beneath aggregate scores:
Constraint violations. In NVFP4 Attention optimization, both Claude and DeepSeek produced kernels that passed basic validation tests but altered the mandatory sequence of mathematical operations. In another instance, an early iteration of Claude for Quant-VideoGen downloaded weights from an external pretrained codec and attempted to submit them. The Controller rejected these submissions before private scoring.
Measurement errors. Qwen benchmarked 20 configurations of a Mamba-2 GPU kernel, but its profiling script repeatedly evaluated a cached compile artifact. After identifying the anomaly, the agent cleared the cache and re-executed. In another run, GLM modified one function while benchmarking another with an identical identifier.
Premature completion. In several instances, researchers instructed models to continue working after they had already claimed completion. In three examined cases, additional compute within the original time window delivered substantial score gains:
- GLM 5.3 Flash on SageAttention: +24.8%.
- DeepSeek V4.1 Flash on Miles SD3.5-M training: +14.0%.
- GPT-6 Astra on Flux/Comet MoE Primitive: +7.3%.
These episodes highlight why autonomous agents require automated constraint checkers, sandboxed test environments, and explicit continuation criteria. An agent's self-reported "task complete" status is not a reliable indicator that optimization space has been exhausted.
Context in Autonomous AI Agent Research
Assessing long-horizon agent autonomy is a central theme across contemporary AI safety and capability evaluations. The METR framework formalized time horizon — task difficulty defined by the time an expert human would require to achieve a given success rate. This is conceptually distinct: the 24 clock hours allotted to agents in Infrathon cannot be equated directly to 24 hours of human engineering effort under METR's methodology.
Similarly, RE-Bench evaluated language models against domain specialists on research engineering tasks. In that study, agents showed strength under tight compute limits, while human specialists pulled ahead as time horizons expanded. Infrathon builds on this foundation by applying modern models to production infrastructure workloads with detailed optimization telemetry.
The researchers have released the open-source Infrathon repository on GitHub with task configurations and codec examples, enabling independent technical verification.
Key Caveats When Interpreting the Benchmark
Several experimental constraints warrant consideration:
- Harness variance. Claude was driven via Claude Code, whereas other models executed through Codex CLI. Even at maximum reasoning effort, results reflect the combined agent harness, tooling, and model, not the raw foundation LLM alone.
- Domain scope. All 25 tasks center strictly on machine learning systems engineering. Superior performance here does not imply superiority in data analytics, web development, general software engineering, or document automation.
- Relative rank scale. The composite score evaluates comparative placement among these six models. It does not measure the absolute percentage of completed assignments nor the absolute technical margin between adjacent models.
- Compute footprint. 11,376 GPU-hours represents a substantial investment. Real-world deployment decisions must factor in token pricing, GPU server costs, and human oversight time.
- Single-trajectory sampling. Leaderboard rankings reflect a single representative run per model-task pair, leaving open questions about variance across stochastic retries.
Key Takeaways
Infrathon demonstrates that modern AI agents can productively navigate complex ML infrastructure code over extended time horizons, refactoring kernels and finding optimizations that outpace human baselines. Claude Opus 5.5 leads overall across 25 tasks and showcases a standout example of autonomous algorithm design in KV-cache compression. GPT-6 Astra proves competitive in early execution windows and retains advantages in modular workloads.
The central finding concerns time allocation and verification infrastructure. Short evaluation windows frequently misidentify the ultimate winner or fail to capture late breakthroughs. However, additional runtime is not universally beneficial: isolated operator kernels benefit from parallel seeds, whereas multi-component systems require sustained sequential search. To translate these gains into production, organizations must couple agents with automated constraint validation and rigorous private test environments.
Sources
- Hu et al. Infrathon: Benchmarking Language Agents for Real-World, Long-Horizon AI Infra Optimization, arXiv:2611.11995, v2 (Nov 16, 2026) — primary research paper with methodology, data tables, and figures.
- Infrathon on GitHub — task configurations, environment setups, and baseline code.
- Infrathon: Figure 6 frame comparison — original visual quality comparison from the study.
- METR. Measuring AI Ability to Complete Long Tasks — methodology for measuring agent time horizons.
- RE-Bench: Evaluating Frontier AI R&D Capabilities of Language Model Agents against Human Experts — reference evaluation of long-horizon AI research engineering.
