Your agent regression suite ran Tuesday and scored eighty-two. Wednesday morning, same suite, same configuration, same model: seventy-nine. Nothing changed between the two runs. You now have a promotion decision to make – and no reliable information to make it with.
That is the failing signal. A score that shifts between identical runs tells you something happened, but not what your prompt change did. One run cannot answer that question.
Run-to-Run Variance Is the Problem You Are Not Measuring
Run-to-run variance is the spread in scores you get from running the exact same configuration multiple times – same prompt, same model, same harness, same task. No changes. Different results. Language models sample probabilistically, and an agent's earlier outputs shape what comes next in ways that compound across a long task.
Most teams treat this variance as a rounding error. It swallows real signal instead. Researchers at TU Munich and partner institutions ran a coding agent benchmark across five configuration axes on four scientific tasks and released 18,240 recorded agent trajectories so others can recheck the work. On that benchmark, about 54 percent of the score variance among genuine attempts came from repeating the identical configuration – not from any configuration choice the researchers made. This is a non-peer-reviewed arXiv preprint, and the number belongs to one benchmark's four scientific tasks; read it as a warning to test your own suite, not as a law about all agents. [1]
When variance of that size is present, a three-point drop between Tuesday and Wednesday is almost certainly noise.
The Promotion Gate: Mean, Spread, Completion
The fix is operational, not statistical. Run each scenario several times, not once. Report three numbers next to every score: the mean across runs, the spread – how far individual runs land from that mean – and the completion rate, the fraction of runs that finished rather than timing out or erroring.
A delta smaller than your spread is not a result. If repeated runs give you a spread of five points and your change moved the score by two, you do not know which direction you actually moved. The promotion gate becomes one rule: a score change counts only when the delta exceeds the observed spread. Anything below that threshold means run more runs, not ship.
Treat completion rate as a leading indicator. A change that lifts the score while dropping completion is usually gaming the metric, not improving the agent.
Spend the Next Improvement on Information, Not a Model
Once the gate exists, where do you spend? On the measured benchmark, the task-context information handed to the agent – the domain knowledge, schemas, and constraints specific to each task – had a larger effect than either time budget or model size, while also cutting cost. Improve the information before paying for a model upgrade.
The same study found a verification trap. Prompting the agent to check its own answer barely changed verification behavior on these tasks, while giving the agent a dedicated verification tool changed it substantially. Build verification into the harness as a callable tool, structured and bounded, rather than asking for it in prose.
Deterministic Constraints Help – If You Measure Them
A second non-peer-reviewed preprint, by Saransh Dhage, tested deterministic execution constraints – finite-state control, forced tool selection, output validation, bounded retry, and schema-validated structured planning – on two open-weight models across two synthetic tasks, one finance and one legal. Reproducibility here means the same task yields the same outcome on every run. With structured planning in place, the harness achieved a reproducibility rate and determinism index of 1.000 in three of four model-task cells at 100 runs per cell, with task success holding at 100 percent in three of those cells . [2]
The caveat matters as much as the result: a first-pass harness without structured planning made reproducibility significantly worse in two of four cells, and the latency effect was model-dependent. Constraints must be measured, not assumed. Apply them, then pass the repeated-run gate to confirm they helped.
The Rule: No Spread, No Promotion
Put the rule in the reporting template: no single-run number enters a promotion decision without its spread. Every suite result that triggers a go-or-no-go call carries three values – mean, spread, completion rate – and the delta must exceed the spread before anyone acts on it.
Single-run scores are not worthless; a score that collapses tells you something real. What one run cannot do is resolve a small change against background noise, and most meaningful changes are small. Build the spread into the template now, before the next decision lands.