How TRACE teaches an agent which of its turns actually mattered

TRACE is a new training method for language-model agents that solves a long-standing problem: when an agent takes ten steps and finally succeeds or fails, which of the ten steps deserves the credit or the blame? This piece walks through what TRACE does, why the credit-assignment problem has been a bottleneck for long-horizon agents, and why a small open-weight model going from 7.2% to 35.6% accuracy under this method is a bigger deal than it looks.

The paper was posted to arXiv on 15 July 2026 by researchers at the University of Wisconsin-Madison and Microsoft Research.¹

The problem, in one sentence

Imagine trying to learn chess by only being told, at the end of the game, who won. No commentary on which moves were brilliant and which were disasters. Just: white won, or black won. You could eventually get better, but it would take an enormous number of games, because every move in the winning game gets credited equally, including the bad ones the opponent failed to punish.

This is roughly what training a language-model agent looks like today. The agent takes a sequence of turns, search the web, read a document, ask a follow-up, synthesise an answer, and at the end, a reward signal arrives: correct or incorrect. That single number has to somehow teach the model which of those turns was the good one.

The technical name for this is the credit assignment problem, and it is the reason agent training has been stubborn. The reward is sparse (one signal per trajectory) when what you actually want is a dense signal (one signal per turn). Sparse-reward reinforcement learning works, but it is slow, high-variance, and gets worse as trajectories get longer.

The standard fix, and why it is annoying

The textbook response is to train a second neural network, called a critic, whose job is to look at any intermediate state and estimate "how well is this trajectory going to end up?" You then use the critic's estimates to hand out per-turn rewards. This is the actor-critic pattern that underlies most modern RL.

It works, but it has real costs. You have to train the critic, which is itself a hard learning problem. The critic can learn the wrong value function, overfit, or introduce variance that destabilises the actor it is supposed to guide. And in a production setting, you now have two models to maintain instead of one.

For frontier labs with enormous engineering budgets, this is annoying but tractable. For the small-model deployments most teams are actually running, it is a serious tax.

What TRACE does instead

The move is to skip the critic entirely and use the base model itself as the reward oracle.

Here is the mechanism concretely. You keep a frozen copy of the reference model — the same one you started training from, held fixed. At each turn of the agent's trajectory, you ask the frozen model a specific question: given the conversation so far, how likely is the gold answer?

That likelihood is just a number the frozen model can compute directly: the probability it assigns to the known correct answer, conditioned on everything the agent has said and done up to this point. You do not need to train anything to get it. It falls out of a forward pass.

Now compare that number across two consecutive turns. If, after turn 5, the frozen model thinks the gold answer is more predictable than it did after turn 4, then turn 5 helped: the agent gathered useful information, or narrowed the search, or asked a clarifying question that made the answer more reachable. That turn gets a positive reward. If the number went down, turn 5 hurt, and the reward is negative.

You have converted a single terminal reward into a dense, per-turn signal, without training a critic. The frozen reference model does the work, at inference cost only.

The result on Qwen3-4B

The headline number: on the BrowseComp-Plus benchmark, a Qwen3-4B agent trained with TRACE jumps from 7.2% accuracy to 35.6%, roughly a 4.9x multiplier, using pure reinforcement learning with no critic, no supervised warm-up, and no live-web training.¹ A similar pattern shows up on the larger Qwen3-30B-A3B, which rises from 8.4% to 42.6%.¹

The 4B result is the one I find most interesting, and here is why. Most production agent systems are not running frontier-scale models. They are running 4B–8B open-weight models because that is what fits the latency and cost budget of a real deployment. A method that lifts a small model by 28 percentage points on a long-horizon benchmark is commercially more consequential than an equivalent gain at 70B, because that is where the deployments actually live. Qwen3-4B is also open-weight, which means anyone can pick this up.

Why this is structurally interesting, not just empirically better

Two things are worth naming.

The critic-free framing removes a failure mode, not just a cost. Critics are a known source of instability in actor-critic RL. Getting rid of the critic entirely means one fewer thing to break, one fewer thing to serve, one fewer thing to keep in sync during training. The cost story is real, but the reliability story may matter more.

Using the model's own predictions as a reward is a clean idea with legs. Once you see it — the frozen reference model as a cheap, always-available proxy for "are we getting closer?" — you can imagine it applied wherever there is a known target the model can score its own progress against. That is a broad space.

What the paper does not yet tell us

I want to flag the honest limits.

The benchmark is search-centric: BrowseComp-Plus and similar tasks where the "gold answer" is a document or fact the frozen model can probe against. This is exactly the setting where the oracle trick works cleanly. It is less obvious how it extends to open-ended agent tasks, code execution, multi-agent workflows, creative synthesis, where there is no single gold answer to score predictability against. TRACE may generalise; the paper does not yet demonstrate that it does.

The 7.2% baseline is very low. Some of the reported gain will reflect how much headroom sparse-reward training left on the table, rather than a purely intrinsic property of the method. A stronger sparse-reward baseline might narrow the gap.

And the frozen model must run at every turn to score the reward, which adds inference cost during training. At long enough trajectories, or large enough models, that overhead could start to eat the savings from skipping a critic. This is an empirical question the paper's scale does not fully answer.

None of this diminishes the result. It just marks where the frontier of the claim actually sits.

The one thing worth remembering

The credit assignment problem, figuring out which turn in a long trajectory actually mattered, has been the quiet bottleneck for training agents that persist across many steps. TRACE's move is to notice that you already have a model that can tell you, turn by turn, whether the agent is getting warmer or colder: the frozen reference model itself. Ask it the right question, and it hands you a dense reward signal for free.

Footnotes and links

Further reading

• alphaXiv agent paper tracker: https://www.alphaxiv.org
• Prior work on step-level reward estimation: Lightman et al., "Let's Verify Step by Step," arXiv 2305.20050
• "Math-Shepherd: Verify and Reinforce LLMs Step-by-step," arXiv 2312.08935