The LLM was better at building a solver than playing the game

I asked a very strong LLM to play 322, a small Dota 2 drafting roguelite. I expected it to be better than me because the task looked as if it should suit a computer. It wasn't.

I was disappointed by how well the LLM did. The choices looked incredibly statistical: select players and heroes from repeated offers, balance rating, familiarity and chemistry, decide when to reroll, then choose an opponent field.

I am not even particularly knowledgeable about Dota 2. I had only started watching it occasionally again during the previous six months or year. I did not think I should be the difficult benchmark.

The first experiment eventually produced a championship. It also produced a more interesting question. We try to solve everything with LLMs now, and we do not necessarily need to. Could a cheaper deterministic system make these decisions more reliably?

It could. The final solver made each recommendation in milliseconds without an LLM call. Its best policy reached a 27.45% title rate in the largest declared opponent-field budget, against 2.30% for a policy that simply chased player rating. Getting to that result turned a browser game into quite a cool little statistical project.

A win did not settle it

322 asks you to draft five Dota players and five heroes. Player event ratings determine the base score. Hero familiarity adds another component. Certain combinations of former teammates create chemistry. The completed roster then enters a simulated tournament.

The LLM played nine runs before winning. Some early rosters looked stronger than the winner: teams rated 93 finished second or fourth, while the championship roster was rated 90. That roster had sensible player-to-hero comfort, a useful chemistry link and a favourable final field. It went 14-2 in the group stage, dropped into the lower bracket, then won the grand final 3-1.

That was enough to show why the visible rating was not the whole problem. It was not enough to tell us which decisions caused the win.

Draft quality and tournament luck were tangled together. The same strength could produce different finishes. A good roster could lose a match; a weaker one could get a fortunate bracket. My earlier manual history had a 14% title rate across 50 runs, while the LLM had one title from nine attempts, but those were not controlled samples. The random offers, rejected packs and field choices were not preserved in a way that let us compare the runs fairly.

The feeling that the LLM was doing worse than me motivated the project. It was not the evidence that eventually showed the deterministic policies producing higher title rates.

Make the randomness replayable

The first proper step was not to invent a clever policy. It was to reproduce the game offline.

We froze the public client and its seven data files, recorded their SHA-256 hashes, and ported the drafting, scoring and tournament mechanics into a deterministic Python engine. Automatic hero allocation was reproduced as the game actually implemented it, including checking all 120 player-to-hero permutations. The opponent draw, group stage, playoffs and grand final were all seeded and replayable.

This created an important boundary. A completed roster could be scored exactly by the copied game rules. An unfinished draft could not. Policies still needed estimates for questions such as whether a chemistry opportunity was worth more than the strongest player in the current pack, but those partial-state values stayed labelled as estimates rather than quietly becoming “exact” because the program produced a number.

Every random choice came from an indexed tape. Given the same snapshot, policy identity, seed and episode index, the system could reconstruct the same player offers, hero samples, opponent fields and tournament randomness. Results were written to append-only logs with enough identity information to reject an attempted resume if a policy, parameter or data snapshot had changed.

That sounds like infrastructure around the interesting bit. It was the thing that made the interesting bit measurable.

Give each policy the same luck

The core comparison used common random numbers. Instead of running policy A on one collection of random games and policy B on another, both policies received the same indexed tapes.

For each episode we could ask a narrower question: under this exact stream of opportunities and tournament randomness, did the new policy win where the old one lost? Pairing the episodes removes a large amount of irrelevant variance. The comparison is about the changed policy rather than one policy happening to receive stronger packs.

Opponent selection needed its own boundary. The browser allowed repeated field draws, which would make an unlimited solver degenerate into “keep drawing until the field is unusually weak.” We declared three finite budgets instead:

• K=1: accept the first field.
• K=4: draw four fields and select one.
• K=16: draw sixteen fields and select one.

The first field in K=1 was also the first field in K=4 and K=16. The first four were shared too. That nesting let us compare budgets without changing the underlying luck.

The reports used Wilson intervals for an individual success rate and paired Newcombe intervals for the difference between two policies. The distinction matters. “This policy won 27.45% of its games” is a different statistical claim from “this policy improved the title rate by 5.12 percentage points on the same episodes.”

Keep the final test boring

Repeated improvement creates another risk: if you keep looking at the same results and changing the policy, the supposed test data gradually becomes training data.

The solver separated train, validation and test tapes. Candidate parameters were frozen before comparison. Successive halving evaluated several candidates on a small shared training budget, discarded the weak ones, then spent more episodes on the survivors. A fresh validation set decided whether the winner was eligible for the final test.

The test partition sat behind a deliberate release guard. Before it could be opened, the candidate needed positive title-rate results, no meaningful top-four or top-eight regression, zero failures, deterministic replay and an acceptable runtime. Once opened, the policy source, model, features, parameters, seeds and episode count could not change.

This process was not magically correct because the files had serious names. A later independent audit found that an early replay check only compared duplicate in-memory runs and that the first runtime gate benefited from warm caches. Both controls were repaired: the public command-line replay path was exercised, and runtime was measured with a fresh policy build for every episode. The audit also recomputed the final statistics from the raw evidence rather than trusting the generated report.

The process made improvement slightly inconvenient. That inconvenience was deliberate evidence protection.

Four generations of policy

The policies became progressively less naive.

raw_ovr_greedy picked the highest-rated legal player and then a common hero. current_heuristic added hand-written rules for chemistry, hero comfort and rerolls. fast_component valued base strength, active chemistry, partial automatic assignment and teammate opportunities on a common scale.

The final value_model_v2 was a deterministic ridge-linear model over 37 frozen features. Those features represented things such as role-feasible future strength, current and potential chemistry, partial hero assignment, familiarity, draft stage and remaining rerolls. It was trained offline on 8,511 action rows, then frozen. It did not learn during play and did not call an LLM.

The untouched final test ran 10,000 paired episodes for each of four policies at each field budget: 120,000 outcomes with zero failures.

| Policy | K=1 titles | K=4 titles | K=16 titles | |---|---:|---:|---:| | Raw rating | 1.18% | 1.98% | 2.30% | | Hand-written heuristic | 8.07% | 11.26% | 14.77% | | Fast component model | 13.03% | 17.92% | 22.33% | | Frozen 37-feature value model | 16.95% | 22.71% | 27.45% |

At K=16, the value model improved the title rate over fast_component by 5.12 percentage points. The paired 95% interval ran from +4.24 to +6.00 points. Its top-four and top-eight rates improved at all three budgets too, with every paired interval remaining above zero.

The complete draft loop took a median 47.1 milliseconds with a cold value-model build, compared with 26.6 milliseconds for fast_component. That is roughly 1.8 times the latency, but still a very different cost profile from asking a large language model to reason through every offer.

It gave up some hero bonus

One of the most interesting changes was not simply that every score component increased.

Compared with fast_component, the value model raised mean base strength from 80.47 to 81.02 and chemistry from 2.47 to 3.12. It used more of its two draft rerolls: 1.73 on average rather than 1.29. Its mean assigned hero bonus fell from 7.07 to 6.82.

Those component changes are consistent with giving up a little immediate hero familiarity when the surrounding draft state suggested stronger players or chemistry later. The median final overall moved from 90 to 91. A greedy policy could see the lost bonus. The value model could price the trade.

It still was not an optimal solution to 322. It estimated continuation under a baseline policy, and its future chemistry features described bounded opportunities rather than the actual probability of drawing a particular card. The result applies to the frozen July 2026 game data, the declared format, two draft rerolls and those finite opponent-field budgets.

It was good, not magic.

The LLM found its role

The satisfying part was that, compared with those first browser runs, it kept improving and kept getting better to the point that it was good. I had started from “this just isn't very impressive” and ended up very pleasantly surprised.

The change was not that the LLM eventually became brilliant at choosing cards directly. The runtime policy stopped being an LLM. The language model was useful around the decision: inspecting the game, reverse-engineering its rules, building the exact engine, designing policies, running controlled experiments, finding weak controls, auditing evidence and creating visualisations that made the progress visible.

This was also a good fit for the long-running Codex work I had started doing with stronger models. There was no single clever prompt that solved it. The task could be improved and improved and improved, with each result changing what was worth trying next.

We try to solve everything with LLMs now, and we do not necessarily need to. For 322, deterministic code made the decisions. The LLM was impressive at building the player and showing me why it had improved.

If I reopened the project tomorrow, I would not ask the LLM which card to pick. I would ask it which experiment to run next.