Opus 5.5 vs DeepSeek and GPT-6.1 Sol in Poker: Who Is Stronger and Cheaper?

On October 5, developer Chrystian Schutz published the HoldemBench benchmark: 17 players, including Claude Opus 5.5, GPT-6.1 Sol and DeepSeek V4.1 Flash, played 7,729 hands of heads-up poker. On strength, the frontier models drew with the cheap ones: DeepSeek scored the most (+5.4 big blinds per hand), Sol +3.4, Opus +2.7, but the gap is within the margin of error. On price, the gap is huge: 1,000 hands cost $22 with Opus, $6.45 with DeepSeek and $0.71 with GPT-6 Luna. And a 30-line Python script played Opus to a draw.

How to read these numbers if you don't play poker. The big blind is a forced bet a player puts in before the hand starts; in poker it serves as the unit of money. In this test it equals 100 chips, and every player starts each hand with 10,000. "+5.4 big blinds per hand" means DeepSeek won 540 chips per hand on average, and "-4.0" means a player lost 400 per hand on average. Zero is break-even. The higher the number, the better the player does against the rest.

How the test works

The format is heads-up no-limit hold'em: blinds of 50/100, 100 big blinds each, and stacks reset after every hand. All AI players get the same English text: their cards, the board, the pot, the stacks, the betting so far, the opponent's statistics and a list of legal actions. There are nine actions: fold, check, call, five bet sizes from a third of the pot to one and a half pots, and all-in. The player picks an action and states its chance to win.

The rating is the average result per hand in big blinds. It is fitted over every match at once, so players who never met can still be compared through common opponents. Each result comes with a 95% confidence interval. The API calls cost $16.44 in total.

There are four kinds of players: two frontier models, five cheap reasoning models, four "decision models" (they return a choice with probabilities instead of text), one local model, and scripted bots.

How luck was taken out of the result

Poker has enormous variance: a good player can lose for a thousand hands because the opponent got better cards. Most AI poker comparisons online are a few hundred hands of raw chip counts, and they mostly measure luck. To make the ranking reflect play rather than luck, the author did three things:

1. Every deal is played twice with the seats swapped. If one player gets aces and the other kings in the first hand, they swap cards in the second. What remains is the difference in how each played the same cards.
2. The same deals for everyone. Every pairing in a league uses one shared list of deals, so everyone faces identical cards.
3. All-ins are scored by odds, not by outcome. When both players are all-in before the last card, the hand is scored by each player's chance to win at that moment, not by the cards that happened to come. This alone cut the spread of results by a factor of 2.7.

Stacks also reset after every hand: nobody busts, and every hand counts the same.

Even so, one duplicate deal between two players swings by about 9 big blinds. Pinning a single pairing to within one big blind per hand takes about 320 deals. That is why the cheap models played many opponents, while the expensive Sol and Opus only played those whose ratings were already pinned down: the script, Jev, pplx, Luna, Muse Spark Contributor, and each other.

Results table

Models available on GPTunneL are in bold. The top five places all went to LLMs: three cheap and two frontier. The intervals of DeepSeek and Opus overlap, so first place is an estimate, not a proven lead. The difference in cost per hand is not an estimate, though: $6.45 vs $21.97, a factor of 3.4.

Frontier vs cheap: a draw

The overall rating is measured against the whole field. The author also put Sol and Opus head to head with the others:

The author's conclusion: the frontier models judge their chances best (a 0.87-0.88 correlation with the real odds) and never produced an invalid answer. They beat the decision models, mostly by small margins. But against the cheap reasoning models they only drew, even though those cost 10 to 30 times less. The author calls 20-deal results hints, not verdicts, but the picture is consistent: Luna loses fractions of a big blind to Sol and Opus, Muse Spark Contributor draws with Sol and beats Opus.

The two frontier models play differently. Sol is tighter: it enters 58% of pots and folds to a bet 34% of the time. Opus enters 67% and folds to a bet only 18% of the time, and spends no reasoning tokens at all on easy spots.

Who plays like whom

Why the cheap models ended up next to the frontier ones, or even above them, the author shows with a separate experiment. He froze 40 spots from real hands and asked every player what it would do. Two players "agree" on a spot if they choose the same kind of action.

Sol, Opus, Luna, DeepSeek and Muse Spark Contributor pick the same kind of action in 65-80% of spots, even though they come from four different companies. The author's takeaway: good play seems to converge on one style. The cheap reasoning models play close to textbook poker: they enter many pots, raise more often than they call, and bet between half a pot and a full pot.

The rest split into "families." The decision models Jev and pplx play like the calling station, a bot that always calls. Solar and the local Bielik land closest to the random bot and the maniac, which raises every hand.

Where they differ is consistency. Asked the same question twice, GLM gives the same answer 55% of the time, MiMo 65%, DeepSeek 82%. Opus answered the same way 40 times out of 40 and kept its answer in 38 cases when the card suits were renamed, which changes nothing about the hand. Sol managed 39 and 40. The decision models are deterministic and always give the same answer, but renaming the suits still flips about one answer in ten for them too.

The 30-line script

The equity_tag bot estimates its chance to win with a quick Monte Carlo simulation on every move and bets when it is ahead. It has no reasoning and no memory of past hands. In the overall rating its +2.4 is level with MiMo.

Sol beat the script convincingly: +2.4 big blinds, with the whole interval above zero. Opus did not: +0.5 with an interval from -1.4 to +2.6, which is a draw. The author uses this match to show how small samples mislead. After 20 deals, Opus led the script by 3.5 big blinds, and the interval did not include zero. The author admits he nearly wrote that up as a result. The next 50 deals went the other way, and the combined result is a draw.

The script beat the decision models: Jev 1.13 by +2.7 over 100 deals, Solar Decide by +5.3. As the author puts it, the cheapest way to play decent poker in this test is still free: the script finished level with Opus, MiMo and GLM.

Where they go wrong

Decision models are a new kind of model: instead of text, they take a state and a list of options and return a choice with a probability for each. They are cheap (5 to 10 cents per 1,000 decisions) and fast: Jev answers in 0.3 seconds. But they play strange poker:

• Solar Decide was dealt a pair of aces, the best starting hand, facing a small raise from the random bot, put its chances at 48% and folded. With both hands known, the aces were an 84% favorite. This is not a one-off glitch: in 13 hands with a premium hand (AA, KK, QQ or AK) it folded four times, while the random bot in the same league did so 3 times in 18. Solar also goes all-in in 56% of its hands.
• Jev 1.13, the best of the decision models, raises before the flop in only 3% of hands, calls almost everything, and after the flop its median bet is six times the pot. That works against players who fold too much, but against a script that simply calls when its odds justify it, those bets cost whole stacks.
• pplx-decider folded in 7 of the 35 river spots where its chance to win was at least 90%.

All decision models underestimate their hands: pplx by 19 points on average, Jev and Solar by 17, Mercury by 15. Language models are off by 4 to 9 points, the scripts by less than one.

The local Bielik 11B is an open Polish model from the SpeakLeash project, built for the Polish language rather than for reasoning, with no thinking mode. The author ran it on his own machine (Q8, llama.cpp, one RTX 3090 Ti). It finished last, below the random bot: it put money in 96% of pots, re-raised 77% of the time it could, went all-in in 31% of hands, and never once folded to a bet. It is the only player that overestimates its chances, by 20 points on average. This says little about Bielik as a Polish assistant, the author notes. But a small model without reasoning does not turn a poker prompt into poker.

DeepSeek's weak spot is time. It thinks for about 100 seconds per decision on average and up to five minutes on hard spots, against 6-7 seconds for Opus and Sol. A single hand against it takes 3 to 8 minutes, and its last hands kept the whole cheap league waiting. On 3.6% of decisions it ran past its 32,768-token output limit, and those answers counted as errors. It took first place anyway.

What went wrong for the author

The author writes that most of the work was not poker. Some of his mistakes are worth knowing for anyone comparing models themselves:

• Elo crowned the maniac. The first leaderboard used Elo on single deals. A bot that raises every hand wins many small pots and loses a few huge ones, so it collected the most "wins" while losing chips. The ranking now uses chips per hand.
• Reasoning settings don't carry over between providers. GLM on default settings thought until it hit the 8,192-token limit and returned nothing; on medium effort it did not reason at all; only high effort worked. DeepSeek ran out of tokens in a third of its answers until the limit was raised to 32,768.
• Rate limits look like bad play. pplx returned more than a thousand "too many requests" errors in one run, and the fallback quietly turned them into check or fold. The result looked like a very passive player. Such hands are now replayed.
• A sign error. For a few minutes the author read the script's loss to Opus as a win. The charts computed it correctly; the headline was wrong.

Limitations

The author lists what limits the conclusions:

• heads-up only, with 100-big-blind stacks reset every hand;
• five fixed bet sizes;
• one English prompt for everyone; a different one could change the order;
• the frontier and local models played fewer deals, so their intervals are wider, and he calls 20-deal matches hints, not verdicts;
• no comparison with a real solver.

The code and every decision with its estimate, latency and cost are public: the HoldemBench repository under the MIT license and the dataset on Hugging Face.

What this means for you

Poker is not your work task, but the lesson is broader: more expensive does not mean stronger. Opus 5.5 was no better than the cheap models in this test, and a hand with it costs 31 times as much as with Luna. The price per token is not the price per task either: the total depends on how long the model "thinks" about its answer. So run your own task on two or three models and compare the cost of an accepted result, not the price list.

Per-token prices on GPTunneL for the four models from the test:

Luna and Sol are currently 50% off; see the pricing page for current rates.

Where to try

All four models are on GPTunneL, with one balance for any calls: DeepSeek V4.1 Flash, GPT-6 Luna, GPT-6.1 Sol and Claude Opus 5.5. How DeepSeek V4.1 Flash did in other tests is covered in DeepSeek V4.1 Flash 1017, and Luna was compared with Haiku 5.5 in a separate review.

Sources

• HoldemBench: I Made 17 AIs Play 7,700 Hands of Poker, Chrystian Schutz, 10/05/2026 - format, model behavior, limitations.
• HoldemBench repository, MIT - results table, exact hand count (7,729; the author rounded it to 7,700 in the article title) and $16.44 in API costs.
• HoldemBench dataset - decisions, estimates, latency and cost of every hand.