{"id":"fcc641e2-e35f-44d5-b124-e0a0ce758301","arxiv_id":"2608.09168","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A lightweight post-retrieval gate trained on verifier feedback can skip most low-value LLM agent executions while retaining most of the achievable reward.","lead":"This paper proposes RADEG, a cheap gate that decides whether a retrieved set of agent skills is likely to produce a useful result before the LLM agent is invoked. It reports skipping 68 percent of agent calls while keeping 61 percent of verifier reward on a 288-rollout SkillsBench study.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Headline AUROC may hinge on within-query online feedback: the main five-split RADEG evaluation updates the logistic head on rewards from one bundle variant before scoring other variants of the same held-out query, a leakage the paper itself flags in TS C.3.","rationale":"The most load-bearing threat to the central claim is not that each pair is executed once (that adds label noise, which usually attenuates rather than inflates ranking metrics), but that the online head can consume same-query reward information before scoring later variants of that query. The main comparison in Table 3 is the evidence for the claim 'predicts execution utility before the rollout is launched,' and RADEG's advantage over Static Head is large (0.717 vs 0.517 AUROC). Since static prediction is near chance, the reported utility signal appears to come from the online updates described in §4.4. The update protocol is not fully specified for the main five-split evaluation, and TS C.3 explicitly acknowledges the same leakage shape for cross-agent evaluation. If the main evaluation permits updating the head on the first variant of a held-out query before scoring the other three, then the 'held-out' numbers measure within-query correlation rather than pre-rollout prediction for fresh pairs. The paper deserves credit for disclosing the limitation in the supplement and for providing query-level splits, 30-split robustness, and bootstrap comparisons; those do not remove the leakage because they preserve the same within-query feedback structure. A strict single-bundle-per-query evaluation is the decisive check. If it reproduces the headline, the central claim stands; if it collapses toward Static Head, the paper's contribution is substantially weakened. This does not change the reader's CONDITIONAL verdict, but it sharpens the primary condition that must be met before acceptance.","tokens_in":18131,"tokens_out":8244,"duration_ms":78024,"concrete_test":"Re-run the common five-split comparison under a strict single-bundle-per-query protocol: for each held-out query, score all four bundle variants before revealing any of their rewards, then update the head only on completed queries; or equivalently, evaluate only the GoS Original bundle per query with updates occurring between queries. If RADEG's AUROC falls toward Static Head's 0.517 and RR@20% toward ~27%, the headline 0.717/61% result is an artifact of within-query feedback; if it remains around 0.7+, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"RADEG's central claim is that execution utility can be predicted before the expensive rollout is launched, with headline evidence at Table 3: AUROC 0.717, 68% calls saved, 61% reward retained under a 'query-level held-out evaluation.' The deployed method in §4.4 updates the logistic head whenever verifier feedback is observed. The five-split evaluation keeps all four bundle variants of each query in the same partition and processes the held-out query stream prequentially, so after the first variant of a query is scored its reward is used to update the head before the remaining three variants are scored. The four variants share a query and are reward-correlated, so this within-query feedback can inflate predictive and operational metrics relative to the actual deployment case of one retrieved bundle per query. The paper itself concedes the issue for the cross-agent protocol in TS C.3: 'feedback from an earlier bundle variant may influence predictions for later variants of the same query... A stricter evaluation should score all variants of a query before revealing any corresponding labels, or evaluate only one retrieved bundle per query.' Nothing in the main evaluation rules out the same structural leakage, and Static Head's AUROC of 0.517 shows that most of RADEG's reported advantage comes from the online head. If that advantage depends on same-query reward updates, the central claim is not established by the current protocol.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes RADEG, a post-retrieval, pre-execution gate for skill-based LLM agents. RADEG encodes a query and a retrieved skill bundle, predicts the probability of non-zero verifier reward, and decides whether to launch the expensive agent rollout; after deployment it updates only a logistic head from observed verifier feedback. The authors construct a 288-rollout dataset on SkillsBench by locally perturbing GoS bundles (delete, add, or replace one skill), find that aggregate PPR relevance has little predictive signal for reward, and report that RADEG achieves AUROC 0.717, skips 68% of calls, and retains 61% of verifier reward at threshold 0.5, outperforming PPR-based and random gating under matched execution budgets. Additional experiments cover cross-agent adaptation, generalization across retrievers and tool benchmarks, selective feedback with epsilon-greedy exploration, feature ablations, and runtime overhead.","tokens_in":18346,"tokens_out":6540,"duration_ms":66166,"significance":"The relevance-utility gap is a real and practically important problem, and the controlled perturbation design is a useful way to create supervision without hand-labeling. The paper shows good methodological hygiene in several respects: query-level splits, a 30-split uncertainty analysis for RADEG, a permutation test for PPR relevance, query-level bootstrap resampling for fixed-budget comparisons, and explicit measurements of the gate's runtime overhead. If the headline results survive a stricter evaluation protocol, RADEG would be a simple, retriever-agnostic, and cost-effective contribution to skill-based LLM agents. As reported, however, the central claim that execution utility can be predicted before a rollout is launched is not yet established, because the main evaluation allows the online head to receive same-query feedback within the held-out stream and because each query-bundle pair has only a single rollout whose reward is treated as a deterministic label.","major_comments":[{"comment":"The five-split evaluation is described as leakage-free, but the online update protocol in §4.4 lets the logistic head be updated with verifier feedback from one bundle variant of a held-out query before the remaining variants of the same query are scored. Since the four variants share a query and are reward-correlated, this within-query feedback can inflate the AUROC and the operational metrics in Table 3 relative to the deployment scenario in which only one bundle is retrieved per query. The paper itself concedes the issue for the cross-agent protocol in Technical Supplement C.3 ('feedback from an earlier bundle variant may influence predictions for later variants of the same query... a stricter evaluation should score all variants of a query before revealing any corresponding labels, or evaluate only one retrieved bundle per query'), but it does not address the same structural issue in the main five-split evaluation. Given that Static Head achieves AUROC 0.517, most of RADEG's reported advantage appears to come from the online head; the central claim therefore requires an evaluation in which all variants of a held-out query are scored before any label is revealed, or a single-bundle-per-query evaluation.","section":"§5.1, §5.2, Table 3"},{"comment":"Each query-bundle pair is executed exactly once, and the resulting verifier reward is used as a deterministic label for both training and evaluation. If LLM agent rollouts are stochastic, a single rollout is a noisy draw from the distribution of outcomes for that pair, and the gate may be learning to predict one trajectory's luck rather than the pair's expected execution utility. The manuscript acknowledges stochastic execution variation only in Technical Supplement D.2 as a possible source of error in the case studies, not as a primary limitation of the learning protocol. The authors should either provide evidence that rewards are stable across repeated rollouts (for example, by repeating a subset of pairs with multiple independent rollouts and reporting agreement) or explicitly redefine the learning target as the reward of a single rollout under the fixed agent and verifier configuration, in which case the central claim about 'execution utility' should be qualified accordingly.","section":"§3, §5.1"}],"minor_comments":[{"comment":"The RADEG row contains a malformed entry ('0.5706861 0.483') and no uncertainty estimates for the baselines; since the paper's own 30-split analysis reports a standard deviation of 0.059 for RADEG's AUROC, reporting bootstrap or split-level intervals for all methods would clarify whether the gaps in Tables 3 and 4 are meaningful.","section":"Table 3"},{"comment":"The limitations paragraph does not mention the two most consequential threats identified above: same-query online updates within the held-out protocol and single-run rollout labels. These should be acknowledged explicitly even after a stricter evaluation is added.","section":"§6"},{"comment":"Several figure captions and inline equations contain garbled or overlapping symbols, which makes the precise definitions hard to check; please regenerate the figure and proofread the tables.","section":"Figure 2 and inline equations"},{"comment":"The cross-benchmark baselines 'Size-only' and 'GoS-based Gate' are not defined in the main text; the reader must infer their construction, so a one-sentence definition of each baseline belongs with the main results or in the supplement's setup.","section":"Technical Supplement F"}],"recommendation":"major_revision","confidential_remarks":"To the editor: the manuscript addresses a timely problem and the empirical material is useful. The main blocker is the evaluation protocol: the within-query online feedback issue is acknowledged by the authors themselves in the technical supplement for the cross-agent experiment, so I expect it can be fixed by re-running the main evaluation with a stricter protocol. The single-rollout-label issue is also addressable with a repetition study or a careful redefinition of the target. I therefore recommend major revision rather than rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick read of arXiv:2608.09168. The core idea is sound and the framing is genuinely new: separate \"is this bundle relevant\" from \"is this bundle worth executing,\" and learn a cheap gate on verifier feedback. The controlled perturbation study (delete/add/replace one skill, 288 rollouts) is a real diagnostic, and the PPR permutation test nicely shows relevance scores carry no signal for positive reward. I credit the query-level splitting and the 30-split robustness check for RADEG; those are the right instincts.\n\nThe problem is the headline evaluation. RADEG's advantage comes almost entirely from the online logistic head — Static Head gets AUROC 0.517 vs 0.717 — and the five-split protocol updates that head on verifier feedback as the held-out stream is processed. Since all four bundle variants of a query sit in the same partition, the head can see the reward from the first variant before scoring the remaining three. Those rewards are correlated through the shared query, so the reported 0.717 AUROC and 61% reward retention may be inflated by within-query label leakage. The paper itself concedes exactly this issue for the cross-agent protocol in TS C.3, but never addresses it for the main evaluation. That's the load-bearing caveat. A stricter protocol — score all variants of a query before revealing any of their labels, or evaluate one bundle per query — is needed before I trust the central claim.\n\nThere are secondary issues. Each rollout is executed once, so the binary label is a single noisy draw; stochastic agent execution could mean the gate is predicting one trajectory's luck. No code or data are released, and the main tables give point estimates without baseline error bars. The 30-split analysis only covers RADEG, not the baselines. All of these are fixable.\n\nWho's this for? People building cost-saving layers for agent pipelines, and anyone designing evaluations of retrieval-to-execution systems. It deserves a serious referee — the idea is worth engaging — but only with a request for major revision: leak-free online protocol, repeated rollouts or variance acknowledgment, error bars everywhere, and public artifacts. If the numbers survive the stricter protocol, this is a useful paper.","headline":"Useful framing and a clean diagnostic, but the headline AUROC likely leaks same-query reward through the online head update; needs a stricter protocol before the central claim holds.","tokens_in":18954,"tokens_out":2571,"would_cite":false,"duration_ms":26042,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper introduces RADEG, a lightweight post-retrieval gate that predicts whether a query–bundle pair will earn verifier reward, and shows it can skip 68% of agent calls while retaining 61% of reward.","keywords":["LLM agents","skill retrieval","execution gating","reward prediction","relevance-utility gap","surrogate model","cost-aware routing","selective execution"],"falsifier":"Re-run a sample of the 288 query-bundle pairs multiple times with non-zero sampling temperature under the same agent and verifier, and compute per-pair variance of verifier reward; if rewards flip between zero and positive across runs, or if a gate trained on single-run labels loses AUROC when evaluated against the averaged rewards, the single-rollout labeling assumption is violated.","tokens_in":17876,"feed_emoji":"🤖","tokens_out":7847,"duration_ms":66001,"temperature":0.7,"pith_summary":"Skill-based LLM agents retrieve a bundle of skills for each query, but a bundle that looks relevant may still waste expensive agent execution when it produces no verifier reward. This paper argues that relevance and execution utility are different decisions, and that a lightweight post-retrieval gate can predict the utility of a query–bundle pair before the agent is launched. The proposed RADEG learns a low-cost surrogate from logged (query, bundle, reward) rollouts, using locally perturbed bundles of the same query to isolate the effect of bundle composition. On 288 held-out rollouts it skips 68% of agent calls while retaining 61% of verifier reward, and under a fixed 20% execution budget it retains 40% of reward versus 15% for relevance-based gating and 19% for random gating. The paper's point is that execution-aware gating, not better retrieval alone, is what prevents paying for irrelevant-looking bundles.","feed_headline":"A cheap gate skips 68% of agent runs, keeping 61% of reward","feed_subtitle":"A retriever-agnostic layer predicts execution value before rollout, beating relevance and random gates.","key_machinery":"The load-bearing object is RADEG, a retriever-agnostic \"execution gate\" placed between skill retrieval and agent invocation: it outputs $g_{\\theta}(q,b;\\tau)=\\mathbb{I}[s_{\\theta}(q,b)\\ge\\tau]$, where the score $s_{\\theta}(q,b)$ approximates $\\Pr(r(q,b)>0\\mid q,b)$ and the threshold $\\tau$ controls the execute/skip trade-off. The score is produced by a frozen text encoder that maps the query and each skill's textual specification into embeddings, a mean-pooled bundle representation, a 776-dimension feature vector (query embedding, bundle embedding, a four-dimensional bundle-condition indicator, and four Graph-of-Skills PPR summary statistics), an MLP utility encoder, and a logistic execution head. Offline, the encoder and head are trained jointly with class-weighted binary cross-entropy on logged rollouts; online, only the warm-started logistic head is updated with observed verifier feedback, so the decision boundary adapts without retraining the retriever or the agent. The training data are generated by locally perturbing each retrieved bundle (delete top-PPR skill, add an irrelevant skill, replace with a graph-similar skill) to create matched same-query rollouts that isolate the effect of bundle composition on reward.","core_discovery":"RADEG's central claim is that whether a retrieved skill bundle is worth executing can be scored before execution begins, using only the query, the bundle's text, and previously observed verifier feedback. The gate encodes the query and the mean-pooled bundle representation, feeds the concatenation through a small MLP, and trains a logistic head to estimate $\\Pr(r(q,b)>0)$, the probability of non-zero verifier reward. Offline training uses class-weighted binary cross-entropy on logged rollouts; at deployment the encoder is frozen and only the logistic head is updated online as new verifier rewards arrive. To obtain supervision that controls for task difficulty, the authors construct matched same-query variants by deleting, adding, or replacing one skill in each retrieved bundle, so reward differences reflect bundle composition rather than query difficulty. The evaluation on 288 rollouts reports AUROC 0.717 and AUPRC 0.570, and at threshold 0.5 RADEG skips 68% of calls while retaining 61% of available reward, raising reward per executed call from 0.309 to 0.483.","pith_inferences":["Beyond the paper: if single-run rewards are noisy samples of a pair's true utility, the gate could be learning to predict one trajectory's luck; re-executing a sample of pairs multiple times and comparing AUROC against averaged rewards would test this directly.","Beyond the paper: the matched-perturbation data-generation scheme could be reused as a self-supervised curriculum — an agent could generate its own delete, add, or replace variants online to bootstrap a gate when no logged rollouts exist.","Beyond the paper: the fixed-budget results suggest a deployment policy that uses the gate's predicted scores to schedule executions rather than merely skip them, reserving scarce budget for the highest-scoring pairs while lower-scoring pairs wait.","Beyond the paper: since exploration reveals labels for skipped calls, an epsilon-greedy deployment that occasionally executes low-scoring bundles trades a small cost for better future gating; the paper's reported degradation under selective feedback (AUROC 0.557 at epsilon 0) is consistent with this trade-off."],"forward_implications":["Execution-utility prediction is a learnable, retriever-agnostic layer: RADEG can sit on top of any existing skill retriever and is evaluated with hybrid, BM25, and embedding retrieval.","Fixed execution budgets become a ranking problem: given predicted utility scores, choosing the top K calls retains 40% of reward at a 20% budget, more than double the relevance-based gate's 15%.","The gate transfers, with adaptation, across downstream agents: from a Claude-trained checkpoint, per-target AUROC reaches 0.733 on Qwen3.7-Max and 0.752 on GLM-5.2 after updating only the head.","Because only the logistic head is updated online, the system can adapt to new verifier feedback at microsecond cost (about 75 microseconds per decision) instead of retraining the agent.","Relevance scores such as Graph-of-Skills PPR carry almost no signal for non-zero reward (AUROC 0.435), so retrieval quality and execution utility should be optimized separately."],"supporting_citations":[{"why":"Supplies the SkillsBench tasks and deterministic verifier rewards used to build the 288-rollout evaluation.","marker":"(Li et al. 2026b)"},{"why":"Supplies the Graph-of-Skills retrieved bundles, PPR relevance scores, and dependency graph used in the motivating study and input features.","marker":"(Liu et al. 2026)"},{"why":"Supplies the learned-routing fixed-budget cost-quality evaluation setup that the matched-budget comparison follows.","marker":"(Ong et al. 2025)"},{"why":"Supplies the paired bootstrap resampling method used to assess significance of fixed-budget reward-retention differences.","marker":"(Koehn 2004)"},{"why":"Supplies the cost-aware LLM routing motivation that selective execution inherits.","marker":"(Chen, Zaharia, and Zou 2024)"},{"why":"Supplies the early embedding-similarity skill retrieval paradigm that motivates the relevance-to-utility distinction.","marker":"(Wang et al. 2024)"}],"fun_headline_variants":["RADEG predicts execution value, skipping 68% of runs while keeping 61% reward","Before rollout, RADEG gates skill bundles: 68% fewer runs, 61% reward kept","Cheap gate forecasts execution utility, boosting reward per call to 0.483","Execution-aware gating skips 68% of rollouts, preserving 61% of verifier reward"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method assumes that each query-bundle pair's single observed verifier reward is a reliable label for how useful that pair really is; if agent rollouts are stochastic, the gate may be predicting one run's luck rather than the bundle's worth.","fun_headline_variants_meta":{"raw":{"variants":["RADEG predicts execution value, skipping 68% of runs while keeping 61% reward","Before rollout, RADEG gates skill bundles: 68% fewer runs, 61% reward kept","Cheap gate forecasts execution utility, boosting reward per call to 0.483","Execution-aware gating skips 68% of rollouts, preserving 61% of verifier reward"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000717,"raw_usage":{"total_tokens":3269,"prompt_tokens":1040,"completion_tokens":2229,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":656,"completion_tokens_details":{"reasoning_tokens":2130}},"tokens_in":656,"tokens_out":2229,"duration_ms":15026,"temperature":1.0,"reasoning_tokens":2130,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T22:11:41.768260+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run a sample of the 288 query-bundle pairs multiple times with non-zero sampling temperature under the same agent and verifier, and compute per-pair variance of verifier reward; if rewards flip between zero and positive across runs, or if a gate trained on single-run labels loses AUROC when evaluated against the averaged rewards, the single-rollout labeling assumption is violated.","supporting_citations":[],"review_version":1}