In six months, AI learned to predict events twice as well: 25% → 50%

In six months, AI learned to predict events twice as well: 25% → 50%

Imagine someone handed you a newspaper dated December 24, 2025 and asked you to write down what will happen before the end of March. Every day brings a fresh issue, and you can change your mind: rewrite the percentages, admit that yesterday you were talking nonsense, bet on someone else. At the end you are shown the answers, and what gets scored is not how many you got right, but how well your confidence matched reality. Confidently wrong, minus. Silent, zero. Right and sure, one.

That is exactly how FutureSim works, a benchmark built by Shashwat Goel, Nikhil Chandak and Arvind Arun with colleagues from ELLIS Tübingen, the Max Planck Institute for Intelligent Systems and the University of Stuttgart. The paper came out in May 2026 and won Best Paper at the forecasting workshop held as part of ICML 2026. On November 15 the authors posted the third-quarter results, and they are more interesting than just a new line on a leaderboard: the best model scores 50% accuracy there against 25% in the first run, though the question set is different and the two cannot be compared directly.

This is a chronicle, not a crystal ball

The first thing to understand: the agent does not predict the future. The world is rewound to December 24, 2025 and played forward one day at a time. News arrives in batches, questions resolve, and the researchers already hold the right answers: the events were picked from what happened between January and March.

To keep that from turning into a hint, the benchmark has a date gate. The corpus is built from Common Crawl News: 7.36 million articles from 141 sources, with another 244,000 added over the 88 days of simulation. An article published after the current simulation day is out of reach, through search, through the terminal, through persuasion.

There are only two actions: submit_forecasts() to set or rewrite probabilities, and next_day() to move to the next day. What to forecast, how often to update, when to move on, the agent decides. That is the whole point: the test is not whether a model can answer, but whether it can change its answer in time.

The FutureSim experiment diagram: a timeline from January 1 to March 28, articles up to the current day are open and the rest are locked; a table of tasks with resolution dates; current predictions; and a loop of submit() and next_day()

The diagram: FutureSim. It shows why this is not a quiz: you can change a forecast until you move to the next day, and the right answer appears in the task table only when the resolution date arrives, before that it reads "N/A yet". To the right of the dotted line sits what the agent cannot reach yet: both the articles and the answers.

330 questions, 88 days and 25%

The first set is 330 short questions collected from Al Jazeera news. The agent invents the outcome options itself: not yes or no, but a list of candidates with probabilities, like a bookmaker, except percentages instead of odds.

First-quarter accuracy chart: GPT 5.5 - 27%, Fable 5 - 26%, Opus 4.8 - 21%, Opus 4.7 - 19%, Opus 4.6 - 18%, GLM 5.2 - 13%, GLM 5.1 - 12%, DeepSeek V4 Pro - 9%, Qwen 3.6 Plus - 5%

Chart: FutureSim, a snapshot from November 16, 2026. The vertical axis is accuracy in percent: by March 28 the leader sits at 27% and the bottom model at 5%. The arXiv paper lists 25% for the best agent: the preprint figure and the updated leaderboard figure differ slightly, and the headline uses the published value.

Next comes the Brier skill score. The formula is simple: for every option take the gap between the stated probability and the fact (one if the option was right, zero if not), square it and add it up. The result is subtracted from one. One, confident and correct. Zero, staying silent. Minus, confidently wrong.

And this is where it gets interesting. In the first run the best agent hit 25% accuracy, meaning that in a quarter of the questions the option with the highest probability turned out to be right. And many models went negative: their Brier skill score came out below zero, worse than not forecasting at all.

Why "below zero" is not a paradox

At first glance it sounds absurd: how can you be worse than not answering? Is zero percent not rock bottom?

No. Zero on this scale is not "no hits" but a specific action: submitting no probabilities at all. Then the only thing you are penalised for is the right answer that received zero. The gap from one is one, squared makes one. Score = 1 − 1 = 0. The reference point is "I said nothing".

Now the model puts 0.8 on option B and 0.3 on option A, and A turns out to be right. Penalty: 0.8² + (0.3 − 1)² = 0.64 + 0.49 = 1.13. Score: −0.13. It did not merely make a mistake, it spent its confidence for nothing, and staying silent would have been more honest. And probabilities do not have to sum to one, so you can confidently back two wrong options at once and sink even deeper into the negative.

Hence the main takeaway of the test, which matters more than any table: the problem is not that models know too little. The problem is that they do not feel the edge of their own knowledge. They say 80% where they should say 30%, and they do not revise when a fresh batch of news arrives.

A separate experiment confirmed this from another angle. All agents were given the same weak starting forecast, and none beat the zero level. Events around them changed, news kept coming, and the models clung to their first version. Changing your mind is a skill they did not have.

Second quarter: calibration, but no adaptation

The run over April to June events flipped the picture again. The most accurate model is GPT 5.6 Sol: 31% by June 28, having started at 15%. Almost all of the gain came during the simulation rather than from starting knowledge.

Second-quarter accuracy chart: GPT 5.6 Sol - 31%, GPT 5.5 - 28%, Opus 5 - 24%, Fable 5 - 23%, GLM 5.2 - 14%, DeepSeek V4 Flash 0731 - 13%

Chart: FutureSim, a snapshot from November 16, 2026. The vertical axis is accuracy in percent. Two different strategies are visible: Claude starts higher but its line is almost flat, while GPT starts low and climbs steeply, and by the middle of the simulation GPT has passed Claude on accuracy.

It was in this run that DeepSeek showed negative calibration: the model assigned high probabilities where it was wrong. And GPT 5.6 Sol made about 15,000 tool calls, more than any other model, and beat the previous version precisely through active search.

By the third quarter, models had learned to change their minds

The update of November 15 replays July to September events on 252 open questions, and the picture is different. Search now runs through Exa Snapshot with a cap of 3,000 calls per run, not through the corpus by hand.

The key point: instead of the flat curves of the second quarter, most models gain 5-15% during the simulation. Not because they were tipped off, but because they read the news they themselves receive and adjust the forecast. All models, open ones included, became positively calibrated: they say 70% and land at about 70%.

Accuracy rose together with calibration: 50% for Opus 5.5, 47% for GPT 6.1 Sol, 41% for DeepSeek V4.1 Flash, 38% for Muse Wonder Full, 30% for GLM 5.3 Flash. On the Brier scale the spread in this quarter runs from 0.13 to 0.35, and every value is positive. Comparing directly with the 25% of the first run is not possible: different question set, different search, different models. But the direction is visible: in six months the best result has doubled.

Third-quarter accuracy chart: Opus 5.5 - 50%, GPT 6.1 Sol - 47%, DeepSeek V4.1 Flash - 40.9%, Muse Wonder Full 1.0 - 37.9%, GLM 5.3 Flash - 30.2%

Chart: FutureSim, a snapshot from November 16, 2026. The vertical axis is accuracy in percent, not Brier: on July 1 the top models sit at 33-35%, and by September 30 Opus 5.5 reaches 50%. The leader adds 17 percentage points, the bottom model about 7.

First place on Brier skill score, both at the start and at the finish, belongs to Claude Opus 5.5. The authors noted a curious detail: the model quickly worked out how the scoring works and almost stopped betting on "nothing will happen" and "other", because such answers almost never win in this test. That is not cheating, but it is not quite forecasting either: rather, the agent figured out the rules of the game and adapted to them.

The gap between models does not match the familiar coding leaderboard: in the third quarter the last place is GLM 5.3 Flash at 0.13, while DeepSeek V4.1 Flash holds the middle at 0.25. The accuracy leader changed as well: GPT led in the second quarter, and in the third Opus 5.5 moved ahead.

What matters to me here is not the ranking, but the fact that adaptation turned out to be a skill of its own. You cannot buy it with model size, and you cannot see it on a coding leaderboard.

The test itself got caught peeking

The most human story happened in the first quarter. There was one loud topic in the world, and a lot of questions about it. And models started deriving answers from the wording of neighbouring questions: if you do not know who wins, look at what is being asked nearby.

This was called cross-task leakage. The fix was honest: for each question a strong model was asked to forecast with access to the other questions and without it, and the ones with a suspiciously large gap were thrown out. The lesson is universal: as soon as an agent benchmark contains many related tasks, "independent questions" stop being independent, and the agent receives a hint nobody intended to give.

I like this more than the numbers themselves. The people who build benchmarks said plainly where their measuring instrument had drifted and showed how they fixed it. That is what proper work on mistakes looks like, unlike tables that live for years with a hole in the methodology.

What to do about it if you do not build benchmarks

It helps that the test has an external check: prediction markets. Where a question overlapped with Polymarket, the AI behaved differently: in the Nepal elections it followed the crowd and offered reasonable arguments, on the Super Bowl with 700 million dollars in trading volume it was sometimes ahead, and on the Grammy and UK district elections it lost to people by a visible margin.

The practical takeaway is simple. In narrow, well-studied stories an AI forecast is useful as a second opinion. In broad ones, where taste, chance or a sudden event decides, it is not.

And if you run an agent on a stream of news, logs or reports, check one thing that costs less than any benchmark. Give it a task, then add a new fact that contradicts the first conclusion, and see whether it changes its answer. A model that stubbornly repeats its first version will lie to you confidently for months, and you will not see it in any general leaderboard.

What this test does not show

In fairness, 25% is not a verdict. The authors themselves call the results a lower bound: the corpus is limited, the tools are limited, and search with reliable date filtering is a separate unsolved problem. With greater access to data, agents would have shown more.

There are three more caveats. First: the numbers are tied to the harness. Codex, Claude Code and OpenCode are different wrappers, and part of the gap between models may be a gap between wrappers. Second: the "5-15% during simulation" gain is mixed with search quality, so it is unclear whether the model got better at thinking or simply better at searching. Third: one score for all domains. Politics, sports and elections are not equally hard, and the table blends them together.

So no pretty picture of "AI predicts the future" follows from these numbers. Nor the opposite, that AI cannot do it at all.

Instead of a conclusion

The ability to change your mind is not a bonus on top of intelligence, it is part of it. We are used to measuring models by what they know, and we forget to measure how they react when they learn they were wrong. FutureSim, with all its caveats, measures exactly that, and shows that even the best models are still learning it with difficulty.

Sources