Pith. sign in

REVIEW 2 major objections 5 minor 18 references

Exact-token fresh prefill is not a faithful stand-in for live decoder state; the boundary KV cache carries the divergent trajectory.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-31 05:33 UTC pith:CVJE6N7X

load-bearing objection Tight empirical audit: exact-token prefill is not live state in ordinary BF16, and full boundary KV is enough to carry the divergent trajectory. the 2 major comments →

arxiv 2607.28495 v1 pith:CVJE6N7X submitted 2026-07-30 cs.LG cs.CL

Stage-Replay Divergence Follows the KV Cache: Fixed-Prefix Precision Controls and Bidirectional Cache Transplantation

Alexander Boesgaard Lorup This is my paper
classification cs.LG cs.CL
keywords stage replayKV cacheprefill vs incremental decodenumerical precisioncache transplantationLLM inference fidelityreasoning-stage boundary
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Stage-replay tools often rebuild an intermediate token prefix and continue from a fresh prefill, treating that as if the model were still in the state it originally reached. This paper audits that assumption at a whole reasoning-stage boundary in a multi-branch Qwen2.5-derived system. With identical integer tokens, exact within-construction replicas, and a shared suffix decoder, BF16 live and prefill states still produce different long trajectories on most items, while net accuracy barely moves. Holding the token prefix fixed and crossing precision shows the behavioral split recurs in BF16 and vanishes in FP32. Swapping the full boundary key/value cache in both directions makes every tested divergent continuation follow the cache donor, so the cache is a causally sufficient carrier of the split on the tested states. The practical stake is measurement: without a live-state control, replayed intermediate text can be perfectly repeatable and still describe a state the end-to-end decoder never occupied.

Core claim

Under the tested ordinary BF16 surface, one-shot prefill of identical tokens is not a faithful substitute for incremental decoder state at a whole-stage boundary: replicas stay exact while live and prefill diverge on suffixes and labels, FP32 removes the decoded disagreement on fixed prefixes, and bidirectional transplantation of all boundary K/V layers makes every tested divergent continuation follow its cache donor.

What carries the argument

Bidirectional whole-cache transplantation at the merge boundary: all 48 key/value layer pairs are swapped while recipient tokens, roles, masks, positions, and pre-intervention boundary logits are held fixed, so any trajectory switch must be mediated by the donated cache.

Load-bearing premise

That the live-versus-prefill split and KV causal sufficiency seen under one greedy Hugging Face setup, one GPU, and one multi-branch model family are an operational warning for ordinary stage-replay practice elsewhere.

What would settle it

On the same fixed prefixes and matched suffix contract, either FP32 would still show decoded live/prefill disagreement, or full bidirectional K/V swaps on divergent rows would fail to make continuations follow the cache donor.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Replica stability and identical token IDs do not by themselves prove live-state fidelity for stage replay.
  • Aggregate accuracy can mask large trajectory and item-level churn from construction differences.
  • Merge-stage probes, counterfactual token credit, privileged-context distillation, and stage ablations need a retained live reference before treating reconstructed text as the original decoder state.
  • Numerical precision can suppress or reveal construction-driven behavioral divergence even when caches remain non-bit-exact.
  • A compact fidelity protocol should separately check replica floors, token/role/mask contracts, live-state match, and causal state sufficiency.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Batch-invariant or higher-precision inference stacks may shrink this failure mode in practice, but only a live-versus-prefill control on the target stack would confirm it.
  • Partial-layer or key-only versus value-only transplants could localize which cache components carry the trajectory without changing the paper’s full-cache sufficiency claim.
  • The same audit pattern likely applies to any tool that re-feeds stored intermediate text as if it were a paused decode, not only multi-branch merge boundaries.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 5 minor

Summary. The paper audits a common assumption in stage-replay diagnostics: that fresh one-shot prefill of an identical integer-token prefix continues from the same decoder state that live autoregressive decoding occupied. At a whole-stage merge boundary in a Qwen2.5-derived multi-branch system, a matched 200-item live-vs-prefill matrix with exact within-construction replicas finds large BF16 trajectory divergence (166/200 suffixes, 20 correctness flips) with near-zero net accuracy change. A fixed-prefix 2×2 holds all 200 token states constant across construction and precision: BF16 disagreements recur, while FP32 yields 0/200 decoded disagreements. A prospective bridge shows token-by-token incremental construction is bit-exact to retained live cache on 12/12 newly reached prefixes, with a 200-row saved-ledger audit reproducing trajectories and comparison fingerprints. Bidirectional transplantation of all 48 K/V layers makes every tested divergent continuation follow its cache donor (24/24 selected; 43/43 outcome-blind later-checkpoint replication). The operational claim is that exact-token replay can be repeatable without live-state fidelity, that boundary K/V is a causally sufficient carrier on the tested divergent states, and that numerical precision moderates behavioral expression.

Significance. If the result holds under the stated scope, it is a load-bearing methodological correction for stage-replay, counterfactual token-credit, privileged-context distillation, and stage-level ablation work that currently treats reconstructed text prefixes as live decoder states. The contribution is not a new accuracy number but a fidelity protocol: replica floors, fixed discrete prefixes, precision crossing, prospective live/incremental bridge, and direct bidirectional K/V intervention. Strengths that raise the bar for this literature include conjunctive integrity gates, storage-isolated replicas on both constructions, the fixed-prefix removal of the discrete-state confound, bit-exact prospective bridge evidence, full reproducibility digests and frozen contracts (Table 3), and an outcome-blind checkpoint replication of the transplant result. The paper is carefully non-overclaiming on kernel cause, accuracy harm, and architecture-general prevalence.

major comments (2)
  1. [§3.1, §6, Abstract] §3.1–3.2 and §6: The central operational warning is framed for an “ordinary BF16 surface,” yet the primary system uses custom six-role embeddings, a role-visibility attention contract (Table 2), a structural logits mask, and a multi-branch merge scheduler. The adjacent vLLM stock-model diagnostic (§6) addresses batch-composition sensitivity, not live-vs-prefill cache construction. The claims are scoped to tested states, but the manuscript should state more explicitly in the abstract/introduction that the audited surface is ordinary BF16 arithmetic under this HF SDPA contract and custom role/mask stack, not a generic stock decoder path, so readers do not over-generalize the prevalence language.
  2. [§4.4–4.5, Tables 5–6] §4.4–4.5, Tables 5–6: Bidirectional donor recovery is complete (24/24 and 43/43), which supports causal sufficiency of the full boundary K/V on divergent rows. The exact-control strata remain small (0/8 and 0/5), with Wilson upper bounds of 32.4% and 43.4%, and the later panel misses the pre-registered six-control threshold by one. This does not undermine the recovery endpoint, but the manuscript’s “no third trajectory” language should remain strictly stratified and should not be read as a tight population bound on transplant artifacts; a brief power/precision note next to those intervals would prevent over-reading.
minor comments (5)
  1. [Table 1] Table 1 is very useful; consider adding a one-line row for “paired task correctness / long greedy horizon” so the endpoint contrast with Matteson’s short probes is visible at a glance.
  2. [§3.4] §3.4: The relative-drift definition d_rel is clear; briefly state whether the reported medians aggregate over layers first or over the concatenated K/V inventory, to aid reimplementation.
  3. [§4.1] §4.1: The median first-divergence at 88 tokens (range 4–453) is important; a small histogram or quartile note in the supplement or main text would help readers judge how often divergence is early versus late.
  4. [§3.9, Table 3] §3.9 / Table 3: The reproducibility contract is exemplary. If space allows, point readers to which released artifact reconstructs each primary statistic (suffix disagreement, transplant recovery, ledger audit) to reduce friction for independent checks.
  5. [§3.1, Abstract] Typos/clarity: “Alivecache” in §3.1 appears to be a missing space (“A live cache”); “whole reasoning- stage” in the abstract has a stray hyphen/space.

Circularity Check

0 steps flagged

No significant circularity: empirical audit with independent experimental arms; results are measured, not forced by definition or self-citation.

full rationale

The paper's load-bearing claims (BF16 live/prefill suffix divergence on fixed tokens, FP32 elimination of decoded disagreement, prospective live=incremental bit-exact bridge, and bidirectional KV donor recovery 24/24 and 43/43) are established by matched experimental arms with replica floors, integrity gates, and direct state intervention. None of these endpoints is defined in terms of the quantity being 'predicted,' and there is no fitted parameter renamed as a prediction. Self-citations (Lorup 2026 on counterfactual likelihood; the disclosed invalidated self-training motivation) supply context and origin story only; the divergence and transplant results do not rest on them. Related-work citations (Matteson 2026; Chodavarapu and Xu 2026) are used comparatively, not as uniqueness theorems or smuggled ansatzes that force the present findings. The derivation chain is observational and interventional measurement under a stated execution contract, not a closed definitional loop. Score 0 is appropriate.

Axiom & Free-Parameter Ledger

2 free parameters · 5 axioms · 0 invented entities

The central claims are experimental, not derived from a fitted theory. Load-bearing background is standard autoregressive transformer caching plus the authors’ fixed multi-role visibility/structural-mask contract and a specific HF greedy execution surface. No new physical entities. Free choices are design parameters (caps, batch-of-two, selection strata) that bound scope rather than fit a predictive constant.

free parameters (2)
  • branch/merge/answer content caps (768/512/64) = 768/512/64 tokens
    Hard length limits that force closing delimiters and shape trajectory length; chosen by system design, not fit to the fidelity claim, but they bound the observed suffix horizon.
  • transplant panel sizes and strata (24 divergent + 8 exact; 48 outcome-blind) = 32 and 48 rows
    Hand-chosen panel sizes and length-spread order statistics determine power of recovery and control bounds; not fitted to maximize recovery after seeing transplant outcomes (selection frozen beforehand), but they are free design knobs.
axioms (5)
  • domain assumption Standard transformer KV-cache semantics: cached keys/values plus current token state determine subsequent greedy decoding under a fixed mask/position contract.
    Assumed throughout Experiments 1–5; transplantation interprets donor following as evidence the cache carries the trajectory.
  • domain assumption The six-role visibility and structural logit mask (Table 2) correctly implements the intended multi-branch merge/answer information flow.
    Section 3.1; custom to this system. Fidelity results are conditional on this contract matching deployment.
  • ad hoc to paper Greedy decoding (T=0, top-p=1, top-k=0) with HF SDPA and physical duplicate batch-of-two is a legitimate ordinary BF16 surface for the audit.
    Table 3 execution contract; no certified batch-invariant kernel. Claim is scoped to this surface but offered as an operational warning.
  • domain assumption Teacher-forced incremental cache construction implements live-style cache updates when prefixes match.
    Used in Experiment 2; directly tested by Experiment 3 bridge (12/12 bit-exact) rather than left purely assumed.
  • domain assumption Deterministic answer matcher errors do not drive primary trajectory/cache endpoints.
    Section 3.8 and Limitations; correctness is secondary to suffix/cache identity.

pith-pipeline@v1.2.0-daily-grok45 · 18659 in / 3311 out tokens · 72015 ms · 2026-07-31T05:33:21.149946+00:00 · methodology

0 comments
read the original abstract

Stage-replay diagnostics reconstruct intermediate token prefixes and treat fresh-prefill continuation as continuation from the decoder state that originally reached the prefix. We audit that assumption at a whole reasoning-stage boundary in a Qwen2.5-derived system. A matched 200-item experiment compares retained live cache with one-shot prefill of identical integer tokens and places an exact replica on both sides. In BF16, replicas remain exact while the constructions differ on 166 suffixes and 20 correctness labels; the accuracy difference is only one point (paired 95% CI [-3.5, +5.5]). A fixed-prefix 2x2 holds all 200 token states constant while crossing construction and precision. The BF16 disagreements recur, whereas FP32 produces no decoded disagreement (95% Wilson upper bound 1.88%). A prospective bridge makes token-by-token incremental and retained live caches bit-exact on 12/12 rows; an all-200 saved-ledger audit reproduces every retained trajectory and comparison fingerprint. Bidirectional transplantation of all 48 key/value layers makes every tested divergent continuation follow its cache donor, both on a selected set at the primary checkpoint (24/24) and an outcome-blind replication at a later checkpoint (43/43). Exact-token replay can therefore be repeatable without preserving live-state fidelity. On the tested states, boundary K/V cache is a causally sufficient carrier of the divergent trajectory, while numerical precision moderates its behavioral expression.

Figures

Figures reproduced from arXiv: 2607.28495 by Alexander Boesgaard Lorup.

Figure 1
Figure 1. Figure 1: Matched live-cache/fresh-prefill design. The duplicates measure replica floors. Conditional [PITH_FULL_IMAGE:figures/full_fig_p005_1.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

18 extracted references · 6 canonical work pages · 4 internal anchors

  1. [1]

    The illusion of equivalence: Systematic FP16 divergence in KV-cached autoregressive inference.arXiv preprint arXiv:2604.15409,

    Ranjith Chodavarapu and Lei Xu. The illusion of equivalence: Systematic FP16 divergence in KV-cached autoregressive inference.arXiv preprint arXiv:2604.15409,

    Pith/arXiv arXiv

  2. [3]

    MarginGate: Sparse Margin-Triggered Verification for Batch-Invariant LLM Inference

    doi: 10.48550/arXiv.2605.30218. URLhttps://arxiv.org/abs/2605.30218. Shreyas Fadnavis, Praitayini Kanakaraj, and Felix Wyss. Beyond consensus: Trace-level synthesis in mixture of agents.arXiv preprint arXiv:2605.29116,

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2605.30218

  3. [4]

    Beyond Consensus: Trace-Level Synthesis in Mixture of Agents

    doi: 10.48550/arXiv.2605.29116. URLhttps://arxiv.org/abs/2605.29116. 14 Jubayer Ibn Hamid, Ifdita Hasan Orney, Michael Y. Li, Omar Shaikh, Yoonho Lee, Dorsa Sadigh, Chelsea Finn, and Noah Goodman. SPIRAL: Learning to search and aggregate.arXiv preprint arXiv:2606.23595,

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2605.29116

  4. [5]

    URL https://arxiv.org/abs/2606

    doi: 10.48550/arXiv.2606.23595. URL https://arxiv.org/abs/2606. 23595. Zesheng Hong, Jiadong Yu, and Hui Pan. Amortized reasoning tree search: Decoupling proposal and decision in large language models.arXiv preprint arXiv:2602.12846,

    Pith/arXiv arXiv doi:10.48550/arxiv.2606.23595

  5. [6]

    URLhttps://arxiv.org/abs/2602.12846

    doi: 10.48550/ arXiv.2602.12846. URLhttps://arxiv.org/abs/2602.12846. Jingcheng Hu, Yinmin Zhang, Shijie Shang, Xiaobo Yang, Yue Peng, Zhewei Huang, Hebin Zhou, Xin Wu, Jie Cheng, Fanqi Wan, Xiangwen Kong, Chengyuan Yao, Kaiwen Yan, Ailin Huang, Hongyu Zhou, Qi Han, Zheng Ge, Daxin Jiang, Xiangyu Zhang, and Heung-Yeung Shum. PaCoRe: Learning to scale test...

    work page doi:10.48550/arxiv.2602.12846

  6. [7]

    URL https://arxiv.org/abs/2601

    doi: 10.48550/arXiv.2601.05593. URL https://arxiv.org/abs/2601. 05593. Alexander Boesgaard Lorup. Counterfactual likelihood tests for indirect influence in private reasoning channels.arXiv preprint arXiv:2605.19092,

    doi:10.48550/arxiv.2601.05593

  7. [8]

    Counterfactual Likelihood Tests for Indirect Influence in Private Reasoning Channels

    doi: 10.48550/arXiv.2605.19092. URLhttps://arxiv.org/abs/2605.19092. Nils Matteson. Re-feeding is not replaying: Measuring replay noise in counterfactual token-credit estimation.arXiv preprint arXiv:2606.15621,

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2605.19092

  8. [9]

    URL https://arxiv.org/abs/2606.15621

    doi: 10.48550/arXiv.2606.15621. URL https://arxiv.org/abs/2606.15621. Jianing Qi, Xi Ye, Hao Tang, Zhigang Zhu, and Eunsol Choi. Learning to reason across parallel samples for LLM reasoning.arXiv preprint arXiv:2506.09014,

    work page doi:10.48550/arxiv.2606.15621

  9. [12]

    URL https://aclanthology.org/2024.naacl-long.462/

    doi: 10.18653/v1/2024.naacl-long.462. URL https://aclanthology.org/2024.naacl-long.462/. Siddarth Venkatraman, Vineet Jain, Sarthak Mittal, Vedant Shah, Johan Obando-Ceron, Yoshua Bengio, Brian R. Bartoldson, Bhavya Kailkhura, Guillaume Lajoie, Glen Berseth, Nikolay Malkin, and Moksh Jain. Recursive self-aggregation unlocks deep thinking in large language...

    arXiv 2024

  10. [13]

    URL https: //arxiv.org/abs/2509.26626

    doi: 10.48550/arXiv.2509.26626. URL https: //arxiv.org/abs/2509.26626. vLLM Project. Batch invariance. vLLM 0.24.0 documentation,

    doi:10.48550/arxiv.2509.26626

  11. [14]

    ai/en/v0.24.0/features/batch_invariance/

    URL https://docs.vllm. ai/en/v0.24.0/features/batch_invariance/. Beta feature; accessed 2026-07-30. Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V. Le, Ed H. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in lan- 15 guage models. InInternational Conference on Learning Representations,

    2026

  12. [17]

    URL https: //arxiv.org/abs/2412.15115

    doi: 10.48550/arXiv.2412.15115. URL https: //arxiv.org/abs/2412.15115. Jiayi Yuan, Hao Li, Xinheng Ding, Wenya Xie, Yu-Jhe Li, Wentian Zhao, Kun Wan, Jing Shi, Xia Hu, and Zirui Liu. Understanding and mitigating numerical sources of non- determinism in LLM inference. InAdvances in Neural Information Processing Systems, volume 38,

    Pith/arXiv arXiv doi:10.48550/arxiv.2412.15115

  13. [18]

    Ziyang Zhang, Xinheng Ding, Jiayi Yuan, Rixin Liu, Huizi Mao, Jiarong Xing, and Zirui Liu

    URL https://proceedings.neurips.cc/paper_files/paper/2025/hash/ f80094a824ba5912d4a2de169c404a40-Abstract-Conference.html. Ziyang Zhang, Xinheng Ding, Jiayi Yuan, Rixin Liu, Huizi Mao, Jiarong Xing, and Zirui Liu. Deterministic inference across tensor parallel sizes that eliminates training–inference mismatch. InProceedings of the 43rd International Confe...

    2025

  14. [2020]

    doi: 10.18653/v1/2020.emnlp-demos.6

    Association for Computational Linguistics. doi: 10.18653/v1/2020.emnlp-demos.6. URLhttps: //aclanthology.org/2020.emnlp-demos.6. An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai ...

    Pith/arXiv arXiv 2020

  15. [2023]

    URL https: //openreview.net/forum?id=1PL1NIMMrw. Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, R´ emi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, an...

    2020

  16. [2024]

    Swarnadeep Saha, Omer Levy, Asli Celikyilmaz, Mohit Bansal, Jason Weston, and Xian Li

    URL https://openreview.net/ forum?id=Ti67584b98. Swarnadeep Saha, Omer Levy, Asli Celikyilmaz, Mohit Bansal, Jason Weston, and Xian Li. Branch-Solve-Merge improves large language model evaluation and generation. InProceed- ings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologie...

    2024

  17. [2025]

    doi: 10.48550/arXiv.2506. 09014. URLhttps://arxiv.org/abs/2506.09014. David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R. Bowman. GPQA: A graduate-level Google-Proof Q&A benchmark. InFirst Conference on Language Modeling,

    doi:10.48550/arxiv.2506.09014

  18. [2026]

    The Illusion of Equivalence: Systematic FP16 Divergence in KV-Cached Autoregressive Inference

    doi: 10.48550/arXiv. 2604.15409. URLhttps://arxiv.org/abs/2604.15409. Kexin Chu, Yang Zhou, and Wei Zhang. MarginGate: Sparse margin-triggered verification for batch- invariant LLM inference.arXiv preprint arXiv:2605.30218,

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2604.15409