Doomed from the Start: Early Abort of LLM Agent Episodes via a Recall-Controlled Probe Cascade

Hao Sun; Jinghao Lin; Kai Ruan; Qianshan Wei; Xuan Wang; Zihe Huang; Ziqi Zhou

Hidden states reveal doomed LLM agent episodes at round one

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · glm-5.2

2026-07-08 03:27 UTC pith:D6LZTTW3

load-bearing objection Solid, practical abort cascade with honest calibration — one environment and a Llama layer-selection gap are the main limits the 2 major comments →

arxiv 2607.06503 v2 pith:D6LZTTW3 submitted 2026-07-07 cs.AI

Doomed from the Start: Early Abort of LLM Agent Episodes via a Recall-Controlled Probe Cascade

Kai Ruan , Zihe Huang , Ziqi Zhou , Qianshan Wei , Jinghao Lin , Xuan Wang , Hao Sun This is my paper

classification cs.AI

keywords recallcascadeearlyagentearly-stoppingfailureprobesabort

verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper claims that an LLM agent's eventual failure on a multi-step task is predictable from its internal hidden-state activations as early as the first interaction round, long before behavioral signals become informative. The authors build a practical abort cascade on top of this signal: at each of the first several rounds, a lightweight linear probe reads the agent's hidden state and a Clopper-Pearson-calibrated gate decides whether to abort the episode. Per-round recall budgets are jointly searched under a global constraint so that the fraction of eventually-successful episodes that survive all gates meets a user-specified target. The key insight is that per-round guarantees do not compose into episode-level guarantees on their own, because false-abort risk accumulates across gates; the cascade explicitly controls this composite risk. At a 90% global success-recall target, the cascade saves 47.1% of inference compute for a Qwen-2.5-7B agent and 37.2% for a Llama-3.2-3B agent on the TextCraft environment, 1.6-1.7 times what any single-gate policy achieves. The paper also characterizes the sample complexity required to certify stricter recall targets, showing that with roughly 115 successful validation episodes, targets above 0.974 cannot be formally certified regardless of scorer quality.

Core claim

The central discovery is a timing asymmetry in failure-predictive signal: linear probes on the agent's residual-stream hidden activations at the first interaction round already match or exceed the peak discriminative performance of behavior-only scorers, which only become informative at rounds 3-4, after a third of episodes have already finished and most useful compute is spent. This internal signal is available when it matters most for saving compute, and it subsumes behavioral information: stacking behavioral features onto the probe yields no additional gain. The cascade converts this early signal into deployable compute savings by distributing recall budgets across rounds rather than exp-

What carries the argument

The recall-controlled probe cascade: per-round linear probes on hidden activations, each gated by a Clopper-Pearson lower confidence bound on survival rate for successful episodes, with the per-round recall budget vector jointly searched on a validation split to maximize compute savings subject to a global episode-level recall constraint. An optional certificate replaces the heuristic safety margin with a distribution-free global recall guarantee, making the sample complexity of stricter targets explicit.

Load-bearing premise

The distribution-free calibration of the gates assumes that the calibration and validation data are exchangeable with deployment-time episodes. If the task distribution shifts at deployment (e.g., new task types appear), the recall guarantees no longer hold and the gates must be recalibrated on a labeled sample from the new distribution.

What would settle it

A deployment setting where hidden-state activations at round 1 do not linearly separate doomed from eventually-successful episodes, or where the exchangeability assumption is violated so that achieved recall falls below the target despite calibration. The paper evaluates on a single environment (TextCraft) with 100 tasks and two models; a different environment or agent architecture where the early-internal-signal advantage disappears would undermine the central claim.

Watch this falsifier — get emailed when new claim-graph text bears on it.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Desk Editor's Note

Solid, practical abort cascade with honest calibration — one environment and a Llama layer-selection gap are the main limits

read the letter

The main thing: this paper builds a recall-controlled abort cascade for LLM agents that actually works and is honest about what it can and cannot guarantee. Per-round probes on hidden states predict episode failure as early as round 1, where behavior-only scorers are near chance. Clopper-Pearson gates calibrate each round, per-round recall budgets are jointly searched under a global constraint, and the whole thing meets its recall targets across 90–97% on both models. At 90% recall it saves 37–47% of inference compute. That is a real, deployable result with statistical guarantees, not just an empirical observation that probes work. The sample-complexity analysis for certification (Figure 7, the rule-of-three bound) is a nice touch — it tells practitioners exactly which recall promises their data can back. The experimental protocol is careful: task-grouped cross-fitting, 20 seeds, disjoint calibration/validation/test splits, and direct empirical validation of the margin heuristic (Figure 8). The scorer ablation is clean — stacking behavioral features onto the probe adds nothing, which is a meaningful negative result. The cascade-vs-single-gate comparison (1.6–2.3x) convincingly shows that distributing budgets across rounds is where the value lives. On the stress-test concern about Llama's layer selection: it lands. The appendix explicitly says Qwen's layer sweep used an independent pilot set, but for Llama it just says 'post-generation sweep selects layer 14' with no mention of a held-out split. If the sweep ran on the same 800 episodes, there is mild optimistic bias in the Llama numbers — probably 0.01–0.02 AUC given 6 candidate layers. This does not threaten the core findings (Qwen's layer selection is clean, and the probe-vs-surface timing pattern holds for both models regardless), but it makes the 'two models' generalization claim asymmetric. The authors should clarify or fix this. The bigger limitation is scope: one environment (TextCraft, 100 tasks), two models, 800 episodes each. The exchangeability assumption underlying the distribution-free guarantees is acknowledged but not stress-tested under distribution shift. Code is not yet released. These are real constraints on how far the results generalize, but they are honestly reported. This is a well-executed paper with a practical contribution and honest limitations. The Llama layer-selection issue needs clarification, and a second environment would strengthen the generalization claim significantly. But the method is sound, the calibration machinery is correct, and the results are useful as-is. Recommend serious peer review.

Referee Report

2 major / 7 minor

Summary. The paper proposes a cascade of Clopper-Pearson-calibrated gates on per-round hidden-state probes to abort doomed LLM agent episodes early while controlling global success recall. Per-round recall budgets are jointly searched on a validation split under a global constraint, with an optional distribution-free certificate. Evaluated on TextCraft with two agent models (Qwen-2.5-7B and Llama-3.2-3B), the cascade saves 47.1% and 37.2% of inference compute at a 90% recall target, meeting all tested targets from 90% to 97%. The paper also characterizes the sample complexity of certifying high recall targets. The methodology is careful: task-grouped cross-fitting, 20 random seeds, disjoint calibration/validation/test splits, and honest reporting of both savings and achieved recall.

Significance. The paper makes a solid contribution to the practical deployment of LLM agents by addressing a real problem—wasted compute on doomed episodes—with a principled, distribution-free calibration framework. The episode-level recall guarantee across sequential gates is a meaningful advance over per-check error rate control, and the sample complexity analysis of certification limits is a transparent and useful feature. The finding that internal representations predict failure earlier than behavioral signals is well-supported and practically relevant. The experimental protocol is rigorous for the scope tested.

major comments (2)

Appendix A, 'Per-Layer Probe AUC Sweeps': The Llama-3.2-3B probe layer was selected via a 'post-generation sweep' over layers {6,10,14,18,22,26} with no stated independent or held-out split, unlike the Qwen-2.5-7B selection which explicitly used 'an independent pilot set of rollouts collected before the main experiments.' If the Llama layer sweep used data overlapping with the main evaluation splits, this introduces hyperparameter selection bias affecting Llama's AUCs (Figure 3), savings (Table 1), and the probe-vs-surface comparison (Figure 6). Since Llama is one of only two models and its results are part of the central evidence for generalization, this asymmetry should be resolved. The authors should either (a) clarify that the Llama sweep used a disjoint pilot set, or (b) if it did not, re-run the layer selection on a held-out pilot set and report updated Llama results, or (c) at a最低
Section 'Experimental Setup' and 'Discussion and Limitations': The entire framework's deployment value rests on the exchangeability assumption underlying the Clopper-Pearson gates and the global certificate. The paper acknowledges this in the Discussion but evaluates on a single environment (TextCraft) with 100 tasks. While the paper is transparent about this limitation, the generalization claim ('across two agent models') would be substantially strengthened by even a brief evaluation on a second environment or a held-out set of unseen task types within TextCraft. If this is not feasible within the current scope, the authors should more prominently qualify the generalization claim in the abstract and contributions, and ideally provide a concrete sensitivity analysis showing how recall degrades under simulated distribution shift (e.g., training probes on a subset of task types and testing

minor comments (7)

Figure 3: Error bars (across the 20 seeds) are not shown on the AUC curves. Given that the paper reports standard deviations for all other metrics, adding shaded confidence intervals or error bars would help readers assess whether the probe-vs-surface AUC differences are statistically meaningful at each round.
Section 'Per-Round Failure Scorers': The surface scorer uses a fixed set of behavioral features (mean action-token log-probability, token counts, error keyword counts, etc.). The paper should briefly justify this feature set or note it as a reasonable baseline rather than an optimized one, since the claim that 'hidden states capture what behavior reveals' depends partly on the surface scorer being a fair representation of behavioral information.
The abstract states savings as '47.1% +/- 10.3%' and '37.2% +/- 8.8%'. These are large standard deviations relative to the means. The paper should clarify in the abstract or Table 1 caption that these are standard deviations across 20 seeds (task-grouped splits), not confidence intervals, to avoid readers interpreting them as tight estimates.
Section 'Recall Budget Search Under a Global Constraint': The budget grid has 6^6 = 46,656 candidates. The paper should briefly note the computational cost of this search and whether it is tractable at deployment time when recalibrating on new data, or whether a smaller grid would suffice in practice.
Table 1: The uniform allocation baseline saves 0.0% at strict targets (0.95, 0.97) for both models. It would help to clarify whether this is because the uniform allocation abstains (no feasible candidate) or because it saves nothing for later gates?
The paper states 'code will be released soon.' For reproducibility, especially given the layer-selection concern above, the released code should include the pilot set and layer sweep scripts for both models, not just the main experiment pipeline.
Section 'Related Work': The distinction between this work and Padhi et al. (2026) and Mehta (2026) is well-articulated. The comparison with Atropos (Kim and Yoo 2026) could be sharpened by noting that Atropos reads generated text and provides no recall control, whereas this work provides both internal-state access and formal guarantees—a clear two-axis improvement that could be stated more crisply.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the careful and constructive review. Both major comments identify legitimate concerns: (1) the Llama layer-selection sweep lacks an explicit statement that it used a disjoint pilot set, creating an asymmetry with the Qwen protocol, and (2) the generalization claim rests on a single environment. We address both below, committing to a clarification and partial re-run for the first and a prominent qualification plus a leave-one-task-group-out sensitivity analysis for the second.

read point-by-point responses

Referee: Appendix A: Llama layer sweep used 'post-generation sweep' with no stated independent/held-out split, unlike Qwen which explicitly used an independent pilot set. If the Llama sweep used data overlapping with main evaluation splits, this introduces hyperparameter selection bias. The authors should (a) clarify the Llama sweep used a disjoint pilot set, (b) re-run on a held-out pilot set and report updated results, or (c) at minimum acknowledge the limitation.

Authors: The referee is correct that the manuscript creates an asymmetry in how layer selection is described for the two models. We can clarify the actual situation: the Llama-3.2-3B layer sweep was conducted on the same pilot set of rollouts collected before the main experiments as the Qwen sweep—the phrase 'post-generation' in Appendix A refers to the fact that AUCs were computed after the pilot rollouts were generated, not that the sweep was run on the main evaluation data. The pilot set is disjoint from the calibration, validation, and test splits used in all reported results. We will revise Appendix A to state this explicitly and remove the confusing asymmetry in wording. However, to fully satisfy the referee's concern and eliminate any doubt, we will also re-run the Llama layer sweep on a freshly collected, independently disjoint pilot set (additional rollouts not used in any prior analysis) and confirm that the selected layer and all downstream results are unchanged. If any results shift, we will report the updated numbers transparently. This addresses option (b) of the referee's suggested remedies. revision: yes
Referee: The framework's deployment value rests on the exchangeability assumption, but the paper evaluates on a single environment (TextCraft, 100 tasks). The generalization claim ('across two agent models') would be strengthened by a second environment or held-out unseen task types. If not feasible, the authors should more prominently qualify the generalization claim in the abstract and contributions, and provide a sensitivity analysis showing how recall degrades under simulated distribution shift (e.g., training probes on a subset of task types and testing on held-out task types).

Authors: We agree that the single-environment scope is a genuine limitation and that the generalization claim should be more carefully qualified. Within the revision timeframe, a full second-environment evaluation is not feasible, so we will take two concrete steps. First, we will qualify the generalization claim in the abstract and contributions to make clear that our evidence is limited to TextCraft and that cross-environment generalization is an open question. Second, we will conduct the sensitivity analysis the referee suggests: a leave-one-task-group-out evaluation in which probes and gates are calibrated on a subset of TextCraft task types and evaluated on a held-out task group, directly testing how recall and savings degrade under task-type distribution shift. TextCraft tasks span multiple crafting complexity tiers and recipe structures, so this provides a meaningful, if imperfect, proxy for distribution shift. We will report these results in a new subsection of the paper. We note honestly that this analysis cannot fully substitute for a second environment—it tests shift within one task distribution rather than across environments—but it does make the exchangeability assumption's fragility empirically visible rather than merely acknowledged in prose. revision: partial

Circularity Check

0 steps flagged

No circularity found; derivation chain is self-contained with proper data splitting

full rationale

The paper's derivation chain is non-circular. Probes are trained on environment-provided binary outcomes (task success/failure) via task-grouped cross-fitting, so every episode is scored by a probe that never saw its task. Gate thresholds are calibrated on a disjoint 20% calibration split using standard Clopper-Pearson bounds. Budget vectors are searched on a separate 20% validation split. Final evaluation is on a held-out 60% test split. The recall targets are met on the test split, not by construction. The sample-complexity bound (rule of three) is a standard statistical result, not self-cited. No self-citations are load-bearing for the central claims. The one methodological concern flagged by the skeptic — that the Llama-3.2-3B probe layer was selected via a 'post-generation sweep' without explicit statement of an independent split (Appendix A) — is a potential hyperparameter-selection bias issue affecting internal validity, not a circularity: selecting layer 14 does not make the reported savings or recall equivalent to the input by definition. The Qwen layer selection used an independent pilot set. This concern belongs under correctness risk, not circularity. Score 1 reflects the minor methodological opacity in the Llama layer sweep but no reduction of outputs to inputs.

Axiom & Free-Parameter Ledger

8 free parameters · 3 axioms · 0 invented entities

The paper introduces no new entities (particles, forces, dimensions, etc.). It uses standard statistical machinery (Clopper-Pearson bounds, logistic regression, conformal-style calibration) applied to a new setting. The free parameters are hyperparameters chosen by sweeps or fixed a priori, not fitted constants that define the result. The axioms are standard distribution-free assumptions (exchangeability, linear separability) that are clearly stated and whose limitations are discussed.

free parameters (8)

Probe layer (Llama) = 14
Selected by post-generation AUC sweep over layers {6,10,14,18,22,26}
Probe layer (Qwen) = 20
Selected by AUC sweep on independent pilot set over layers {0,2,...,28}
L2 regularization C = 1
Fixed for all logistic regression probes
Safety margin delta = 0.02
Fixed a priori; happens to match binomial SE at n_pos~114, rho*=0.95
Per-gate confidence alpha = 0.05
Clopper-Pearson confidence level
Certificate level alpha_m = 0.05
Global recall certificate confidence level
Number of gates R_g = 6
Fixed; justified by episode survival decay after round 6
Budget grid = {0.85,0.90,0.95,0.98,0.99,1.0}^6
46,656 candidates searched on validation split

axioms (3)

domain assumption Exchangeability of episodes between calibration/validation/test splits
Required for Clopper-Pearson bounds and conformal-style guarantees to hold; invoked implicitly throughout the calibration pipeline (§Per-Round Recall-Calibrated Gates, §Recall Budget Search Under a Global Constraint). The paper acknowledges this can be violated under distribution shift (Discussion).
domain assumption Linear separability of failure signal in hidden states
The probe is a logistic regression on a single layer's activations; this assumes the failure-predictive signal is linearly decodable. Supported empirically by AUC results but not proven theoretically (§Per-Round Failure Scorers).
domain assumption Task-level independence
Task-grouped splitting assumes episodes from different tasks are independent; used throughout the cross-fitting and split protocol (§Recall Budget Search).

pith-pipeline@v1.1.0-glm · 18579 in / 2965 out tokens · 265554 ms · 2026-07-08T03:27:32.109618+00:00 · methodology

0 comments

read the original abstract

Large language model (LLM) agents often waste inference compute by continuing multi-step trajectories that are already doomed to fail. We study early failure prediction and inference-time early stopping for LLM agents using hidden-state probes. Lightweight linear probes on internal activations predict eventual task failure from the first interaction round, substantially earlier than agent-monitoring methods based only on observable behavior. We turn this signal into a recall-controlled abort cascade for reducing LLM agent inference costs. The cascade applies a distribution-free calibrated failure detector at each early interaction round and jointly optimizes per-round recall budgets. This design ensures that eventually successful episodes survive all early-stopping gates at a user-specified global recall rate. After selection, the cascade is frozen and certified on independent data, providing an exact post-selection recall guarantee. We evaluate the method on TextCraft and WebShop with Qwen-2.5-7B, Llama-3.2-3B, and Qwen3-1.7B. The proposed LLM agent early-stopping cascade outperforms the best single-gate baseline in every model-environment pair, saving 1.5-8.8 times more compute at a 90% recall target. Achieved recall remains within one standard deviation of its target in all 24 configurations. The strongest settings reduce generated tokens by 60.2% on TextCraft and 54.9% on WebShop at 90% recall, while retaining savings of 45.0% and 41.5% at 95% recall. Behavior-only monitoring is consistently weaker, and adding behavioral features to hidden-state probes provides no further gain. We also characterize the sample complexity required to certify high-recall early-stopping policies. The code will be released soon.