REVIEW 3 major objections 3 minor 15 references
The Replay Gap: Static Evaluation of Model Switching in LLM Agents Scores the Wrong World
T0 review · 3 major / 3 minor · reviewed 2026-08-12 · deepseek-v4-flash
Pith's one-line read Replay evaluation scores LLM agent model switching against states that never occur.
desk verdict Branching-rollout protocol gives a solid action-level replay-gap measurement; outcome-level claims and abstract overreach, but the paper deserves a serious referee. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing mechanism is the branching-rollout protocol: at a chosen fork step, the recorded action prefix is re-executed in a fresh container, the message history is seeded with the recorded prefix, and the trajectory continues with a different (or, for control, the same) model. The comparison between swap arms and same-model control arms isolates sampling and serving-stack noise, so divergence attributable to the swap is read relative to the control. Three metrics carry the result: normalized action edit distance between post-fork suffixes, replay validity (the prefix-match fraction of base post-fork actions), and patch similarity between the replay-predicted and actually-produced patches.
What would settle it
Run the same branching protocol with a competent, unquantized model pair and a base success rate above 20%; if a log-stitching replay evaluator correctly predicts most success-relevant switches, or if swap arms no longer exceed same-model control floors in edit distance, the claim that replay scores the wrong world would be contradicted.
Extended reading notes
Core claim
The central claim is that the assumption underlying replay evaluation of agentic routers—that substituting another model's recorded output at step k leaves the rest of the trajectory unaffected—is wrong in a measurable, structured way. Using branching rollouts on SWE-bench with a bash-only agent scaffold, the paper finds that model swaps rewrite 61–94% of post-fork actions, with first-action divergence in up to 77% of early swaps. Replay validity, the fraction of post-fork states a replay evaluator would score correctly, falls to 3–8% for early swaps; a replay evaluator of an early swap scores 92–97% of post-fork decisions against a state that never occurs. All five observed outcome flips occur in swap arms, with none in 359 control forks, and a log-stitching replay evaluator mispredicts every success-relevant outcome while achieving only 0.00–0.11 patch similarity.
Load-bearing premise
The pilot pool—one bash-only scaffold, two quantized models from one family, 30 instances per run pair, and 0–3% base success rates—is representative of agentic routing generally, so that the measured divergence and outcome misprediction transfer to competent, larger-scale regimes.
Editorial extensions
If this is right
- Replay-based benchmarks for agentic routing that look up logged outputs cannot observe outcome flips; any evaluation of per-step switching needs live branching rollouts.
- Early swaps at 30% of a trajectory are the most invalid for replay evaluation, while late handoffs at 70% inherit more completed work and diverge less.
- Upgrades diverge immediately because the stronger model re-decides at once, whereas downgrades diverge later and less, so handoff policy should depend on direction and position.
- Under tight budgets, the stronger model may exhaust its step limit without submitting more often than the weak one, so routers should price step consumption, not only per-step quality.
- Off-policy estimators such as importance sampling and doubly robust estimation need branched ground truth to validate them, because per-step action-probability ratios are far from one immediately after a swap.
Reading between the lines
- The structural invalidity likely extends beyond bash coding agents to any closed-loop environment—web navigation, computer use, tool use—where the next observation depends on the previous action; the paper tests one scaffold, so this is an extrapolation.
- A testable extension: with a third fork position at 50% and a competent unquantized model pair, the depth-dependence of divergence should saturate or invert, which would pin down where handoffs are safe.
- The paper's 'thoroughness tax' observation suggests that upgrading under a fixed step budget can reduce completion rate; if it generalizes, routing cost models must include expected step consumption, otherwise reported efficiency gains will be overstated.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper asks whether replay-based static evaluation of per-step model switching in LLM agents is valid. On SWE-bench Verified with a mini-SWE-agent scaffold, the authors fork live trajectories at 30% and 70% of the base trajectory, re-execute the prefix in a fresh container, and continue with either a different model (swap arm) or the same model (control arm). The two models are Qwen3-4B-Instruct (FP8) and Qwen3-14B (AWQ). The action-level comparisons show that swaps diverge from the logged base trajectory far more than same-model controls, with bootstrap CIs excluding zero in all four arms and the swap-above-control ordering holding in all 24 run-pair cells. The paper also reports five outcome flips in swap arms, zero in 359 control forks, and that a log-stitching replay evaluator mispredicts five success-relevant calls and produces near-orthogonal patches. The conclusion is that replay-based benchmarks for agentic routing evaluate decisions against states that never occur.
Significance. The action-level result is a meaningful, well-controlled negative result. Its strengths are substantive: paired bootstrap inference at the instance level, same-model control arms for the noise floor, prefix-replay fidelity checked at 99.99% return-code agreement over 11,702 actions, replication across six run pairs and all 24 cells, and release of the harness and full trajectory dataset. If the structural claim 'replay scores states that never occur' holds, it directly challenges the validity of a growing class of agentic router benchmarks. The evidence is less secure for the outcome-level claims: the success-relevant calls number five, the stitching rule is bespoke to the paper, and the pilot's absolute success rates are 0-3%, so the abstract's wording 'replay mispredicts every success-relevant outcome call' overreaches what the data can show. The paper is a rigorous pilot whose main claims need to be re-scoped rather than discarded.
major comments (3)
- [§4.5, Table 2, Abstract] The claim that replay 'mispredicts every success-relevant outcome call' is not supported by five decisive events. With five Bernoulli trials, an evaluator with true success-detection accuracy as high as 45% would still be wrong on all five calls with probability about 5%, and the exact 95% confidence interval for 0/5 successes has upper bound near 45%. The five events can indicate a sign, but not a systematic 0-for-5 failure. In addition, the stitching rule is defined by the authors and the paper explicitly disclaims that any published benchmark implements it. A real per-step router evaluator might collect the target model's decision at the fork point from the base prefix rather than from the target model's full standalone run. Please downgrade this result to a pilot-level observation, remove 'mispredicts every' from the abstract, or add a binomial analysis that quantifies what the five calls can actually exclude.
- [§6, Abstract, Title] The Limitations section states that the paper shows the inputs to replay evaluation are invalid 'rather than demonstrating end-to-end router mis-ranking,' yet the title and abstract assert generally that replay-based benchmarks 'score the wrong world for agentic routing.' The action-level evidence does establish that many replayed states never occur, but the broader claim about benchmark scores and router rankings goes beyond what the pilot can demonstrate. Please align the title and abstract with the Section 6 scope, or add an explicit bridge explaining why the structural invalidity of replayed states is sufficient for the broader conclusion despite the pilot's narrow regime.
- [§4.2] The outcome-flip evidence is reported as 'all five flips in swap arms, zero in 359 control forks,' but these events are selected post hoc from 180 instances across three difficulty tiers and two fork positions, with no multiplicity control. With base success rates of 0-3%, the counts cannot distinguish a genuine swap-only flip tendency from a rare-event null. The fork-position invariance of two instances is a useful case-study observation, but the flip counts should be presented as illustrative rather than as statistical evidence, and they should not appear in the abstract's headline as a stand-alone result.
minor comments (3)
- [Abstract vs. §4.1] The abstract says '74-77% of early swaps diverge at the first post-fork action,' but Section 4.1 reports 73.9% (up) and 76.7% (down); please make the numbers consistent.
- [§4.4] The text alternates between 'patch-identity metrics' and 'patch similarity.' Since the reported metric is a SequenceMatcher ratio over raw unified diffs, use 'patch similarity' consistently to avoid implying exact identity is being measured.
- [§4.5] The parenthetical explaining that the five decisive calls differ from the five flip events of Section 4.2 is easy to misread. A short worked example of one missed rescue, one false success, and one correctly-called downgrade loss would clarify the overlap and the direction of each error.
Circularity Check
No significant circularity: the central claim is a direct branching-rollout measurement compared against same-model controls, and the replay-stitch evaluator is an explicit baseline tested against branched ground truth, not a fitted or self-referential quantity.
full rationale
The paper's load-bearing result is an empirical comparison between branched live rollouts and replay-style predictions, not a quantity recovered from the data it is used to validate. Section 3 defines the branching protocol: forks are paired with same-model control arms that absorb sampling and replay noise, and divergence attributable to the swap is read relative to control. The four headline deltas (e.g., down@early +0.663 with bootstrap CI [0.580, 0.744]) are computed from paired per-instance differences with instance-level resampling, and are not fitted parameters. The replay-validity statistic in Table 1 is a definitional metric of prefix agreement, but using it to conclude that early swaps score 92-97% of decisions against never-occurring states is a measurement, not a derivation: nothing forces the measured 3.2% validity. The Section 4.5 log-stitching evaluator is the only quantity that could resemble a predicted outcome in the paper, and it is explicit and non-circular: it predicts a switch's result from the target model's logged standalone run and is compared to branched truth; the paper even compares it to an always-predict-failure constant, which scores 2/5 versus replay's 0/5, so the failure is not imposed by construction. No fitted input is relabeled as a prediction, no load-bearing claim rests on a self-citation (the author's own prior work is not invoked as evidence), and no uniqueness or ansatz is imported. The main legitimate concerns are statistical power and external validity, which the paper itself flags in Section 6 ('one scaffold, one benchmark family, one model family... n=30 instances per run pair') and in its disclaimer that it demonstrates invalid replay inputs 'rather than demonstrating end-to-end router mis-ranking'; those are correctness and representativeness risks, not circularity. The paper is self-contained as a controlled measurement, so a score of 0 is appropriate.
Assumptions & free parameters
assumptions (3)
- domain assumption Prefix replay of recorded commands in a fresh Docker container reproduces the same environment state the base model observed.
- domain assumption The tested pool (mini-swe-agent, SWE-bench Verified, Qwen3-4B FP8 and Qwen3-14B AWQ, n=30 per run pair, temperature 0) is representative enough to support the general conclusion that replay-based benchmarks score the wrong world for agentic routing.
- domain assumption Command-level actions in the scaffold are the relevant decision units for routing, and Levenshtein edit distance over exact command strings captures meaningful trajectory divergence.
Cite this review
Pith. "Pith review of The Replay Gap: Static Evaluation of Model Switching in LLM Agents Scores the Wrong World." pith.science (2026). https://pith.science/paper/YJHUIHNR
@misc{pith2026260808239,
author = {Pith},
title = {Pith review of: The Replay Gap: Static Evaluation of Model Switching in LLM Agents Scores the Wrong World},
year = {2026},
howpublished = {\url{https://pith.science/paper/YJHUIHNR}},
note = {Machine review of arXiv:2608.08239}
}
read the original abstract
LLM routers promise efficiency by matching each request to the cheapest adequate model, and are increasingly applied per step inside multi-step agents. Yet agentic routers are evaluated like single-turn routers: by replaying logged trajectories and substituting another model's recorded outputs, assuming the rest of the trajectory is unaffected. We test this assumption with branching rollouts: we fork live SWE-bench agent trajectories at controlled points, rebuild the environment, continue each fork with a different model, and compare against same-model control forks that isolate sampling and replay noise. Across six paired runs (~900 rollouts), swaps exceed their matched control floors by +0.25 to +0.66 normalized edit distance (multiplicity-corrected CIs exclude zero), rewriting 61-94% of post-fork actions; 74-77% of early swaps diverge at the first post-fork action, versus 6-35% of controls, leaving only 3% of replayed states valid. Divergence decreases with fork depth in both directions. All five outcome flips we observe occur in swap arms, upgrades rescuing unsolved instances and a downgrade losing the sole solve, and zero occur across 359 control forks. Scoring these same swaps with a log-stitching replay evaluator, replay mispredicts every success-relevant outcome call and predicts patches with 0.00-0.11 similarity to reality. Auditing the noise floor, temperature-0 "determinism" is configuration-dependent: FP8-served controls diverge on over 90% of forks while AWQ-served ones remain near-identical; and under tight budgets the stronger model more often exhausts its steps without submitting. Replay-based benchmarks score the wrong world for agentic routing; we release our harness and all trajectories.
Figures
Reference graph
Works this paper leans on
-
[1]
Switchcraft: AI Model Router for Agentic Tool Calling
Sharad Agarwal, Pooria Namyar, Alec Wolman, Rahul Ambavat, Ankur Gupta, and Qizheng Zhang. Switchcraft: Ai model router for agentic tool calling.arXiv preprint arXiv:2605.07112,
-
[4]
Routerbench: A benchmark for multi- llm routing systems.arXiv preprint arXiv:2403.12031,
Qitian Jason Hu, Jacob Bieker, Xiuyu Li, Nan Jiang, Benjamin Keigwin, Gaurav Ranganath, Kurt Keutzer, and Shriyash Kaustubh Upadhyay. Routerbench: A benchmark for multi- llm routing systems.arXiv preprint arXiv:2403.12031,
-
[5]
Zhongzhan Huang, Guoming Ye, Guowen Liang, Yuanyuan Zeng, Yingjie Chen, Yang Zhong, Xiao Yang, Wei Zheng, and Liang Lin. Routereval: A comprehensive benchmark for routing llms to explore model-level scaling up in llms.arXiv preprint arXiv:2503.10657,
-
[6]
Universal model routing for efficient llm inference.arXiv preprint arXiv:2502.08773,
Wittawat Jitkrittum, Harikrishna Narasimhan, Ankit Singh Rawat, Jeevesh Juneja, Congchao Wang, Chen-Yu Lee, Pradeep Shenoy, Rina Panigrahy, Aditya Krishna Menon, and Sanjiv Kumar. Universal model routing for efficient llm inference.arXiv preprint arXiv:2502.08773,
-
[9]
Yifan Lu, Rixin Liu, Jiayi Yuan, Xingqi Cui, Shenrun Zhang, Hongyi Liu, and Jiarong Xing. Routerarena: An open platform for comprehensive comparison of llm routers.arXiv preprint arXiv:2510.00202,
-
[10]
Odar: Principled adaptive routing for llm reasoning via active inference
Siyuan Ma, Bo Gao, Xiaojun Jia, Simeng Qin, Tianlin Li, Ke Ma, Xiaoshuang Jia, Wenqi Ren, and Yang Liu. Odar: Principled adaptive routing for llm reasoning via active inference. arXiv preprint arXiv:2602.23681,
-
[11]
Isaac Ong, Amjad Almahairi, Vincent Wu, Wei-Lin Chiang, Tianhao Wu, Joseph E. Gonzalez, M. Waleed Kadous, and Ion Stoica. Routellm: Learning to route llms with preference data. arXiv preprint arXiv:2406.18665,
-
[12]
Zhihong Pan, Kai Zhang, Yuze Zhao, and Yupeng Han. Route to reason: Adaptive routing for llm and reasoning strategy selection.arXiv preprint arXiv:2505.19435,
Show all 15 references
-
[14]
Qwen3 technical report.arXiv preprint arXiv:2505.09388,
An Yang, Anfeng Li, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Gao, Chengen Huang, Chenxu Lv, et al. Qwen3 technical report.arXiv preprint arXiv:2505.09388,
-
[15]
Twinrouterbench: Fast static and live dynamic evaluation for realistic agentic llm routing.arXiv preprint arXiv:2605.18859,
Pei Yang, Wanyi Chen, Tongyun Yang, Pengbin Feng, Jiarong Xing, Wentao Guo, Yuhang Yao, Yuhang Han, Hanchen Li, Xu Wang, Zeyu Wang, Jie Xiao, Anjie Yang, Liang Tian, Lynn Ai, Eric Yang, and Tianyu Shi. Twinrouterbench: Fast static and live dynamic evaluation for realistic agen...
-
[2016]
Step-level optimization for efficient computer-use agents.arXiv preprint arXiv:2604.27151,
Jinbiao Wei, Kangqi Ni, Yilun Zhao, Guo Gan, and Arman Cohan. Step-level optimization for efficient computer-use agents.arXiv preprint arXiv:2604.27151,
-
[2023]
Routejudge: An open platform for reproducible and preference-aware llm routing.arXiv preprint arXiv:2606.18774,
Guannan Lai, Haoran Hu, and Han-Jia Ye. Routejudge: An open platform for reproducible and preference-aware llm routing.arXiv preprint arXiv:2606.18774,
-
[2024]
Frugalgpt: How to use large language models while reducing cost and improving performance.arXiv preprint arXiv:2305.05176,
Lingjiao Chen, Matei Zaharia, and James Zou. Frugalgpt: How to use large language models while reducing cost and improving performance.arXiv preprint arXiv:2305.05176,
-
[2025]
Session-aware agentic routing: Continuity- aware model selection for long-horizon llm agents
Xunzhuo Liu, Bowei He, Huamin Chen, Haichen Zhang, Andy Luo, and the vLLM Semantic Router Team. Session-aware agentic routing: Continuity- aware model selection for long-horizon llm agents. https://vllm.ai/blog/ 2026-06-02-session-aware-agentic-routing,
2026
-
[2026]
Automix: Automatically mixing language models
Pranjal Aggarwal, Aman Madaan, et al. Automix: Automatically mixing language models. arXiv preprint arXiv:2310.12963,
Reviewed August 12, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.