{"id":"a10e08cb-54bc-4a2a-9921-b40d7cbf281d","arxiv_id":"2607.16345","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A CI-integrated evaluator whose separation between executor and grader turns an agent's self-corrections from hidden repairs into auditable first-attempt failures.","lead":"AEVAL is a CI framework that tests AI-agent skills by running them against developer-written evaluation contracts and grading only the first attempt, before the agent can patch its own mistakes. It converts demo-based anecdotal skill checks into reproducible pass/fail signals that merge requests can block on, which is useful for anyone shipping or consuming agent skills at scale.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The gate signal rests on an LLM grader's causal attribution of self-corrections, which Section 8 admits can fail and which no experiment validates; the first-attempt pass rate is therefore not established as reliable.","rationale":"The reader's weakest assumption identifies the same load-bearing point: the first-attempt rule depends on the grader correctly attributing causal self-corrections in τ, and Section 8 admits this can fail. I agree that this is the central vulnerability. The paper's design genuinely prevents the executor from grading its own patched outputs, which is a real contribution, but it does not guarantee that the grader can reliably reconstruct which corrections were necessary. The only empirical evidence is a single curated degradation case plus six cross-runtime runs; there is no measurement of grader accuracy, no reproducibility study for the first-attempt pass rate itself, and no released code or data. The cross-runtime results in §7.1 actually show that the grader's severity classification is backend-dependent, which weakens the claim that the emitted signal is a stable, deterministic quality gate. However, these are reasons to remain conditional rather than to reject: the framework is coherent, the limitations are frankly stated, and a targeted synthetic-transcript test could either validate or refute the causal-attribution reliability. Thus the reader's CONDITIONAL verdict is appropriate, and I would not change it based on this analysis.","tokens_in":9677,"tokens_out":7067,"duration_ms":81682,"concrete_test":"Construct a synthetic benchmark of, say, 50 transcripts from a fixed skill, each with known ground-truth labels for whether a first-attempt self-correction was causally required for each assertion to pass. Include cases: (a) direct skill-file edits, (b) multi-step corrections with intervening failed attempts, (c) no-edit retries with altered arguments, and (d) irrelevant edits that should not fail assertions. Run only the AEVAL grader (using the same grader.md) over these transcripts and compare its per-assertion FAIL/self-correction attributions against ground truth. If accuracy on cases (b) and (c) is materially below 95%, or if rerunning the same transcript changes the first-attempt pass rate, the gate signal is not reliable as claimed.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that the first-attempt pass rate is a reliable gate signal for the skill as shipped. That claim depends on the grader correctly identifying, from the transcript τ, which self-corrections were causally required for each assertion to pass. This is an LLM judgment task over a record that may omit internal reasoning or untracked retries. Section 8 explicitly concedes: 'complex multi-step corrections can be misattributed.' The paper provides no ground-truth evaluation of this causal-attribution step: the empirical section is one qualitative case plus six runs, with no released code or data. The cross-runtime study in §7.1 shows that different backends assign the same root causes to different severity tiers (causal share 0.41 vs 0.82), demonstrating that the grader's classification of 'causally implicated' findings is unstable. If the same instability affects whether an assertion is marked FAIL because of a first-attempt correction, the gate signal inherits that variance. Moreover, the formal model in §3 defines self-correction as skill-file edits Δ(τ)≠∅, while the implementation appears to treat any applied change (including edits to a generated runner script) as a correction; this ambiguity makes the set of events requiring causal attribution imprecise. Without a measurement of false-positive and false-negative rates on the causal-attribution step, the paper's central reliability claim is unsupported.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes AEVAL, a CI-integrated evaluation framework for agentic skills. Every change to a skill directory triggers an automated executor run against a developer-declared eval.config; a structurally separate, read-only grader subagent evaluates pre-written assertions using a first-attempt grading rule and produces a machine-readable quality signal; fix suggestions are emitted as tiered, evidence-grounded inline merge-request comments. The central claim is that separating execution from grading and grading only the first-attempt trajectory prevents self-correction bias, so the emitted first-attempt pass rate is a reliable gate signal for the skill as shipped. The empirical section provides one deliberately degraded segmentation case and a six-run matched cross-backend comparison; the paper explicitly defers a full quantitative study to the extended version.","tokens_in":10072,"tokens_out":6163,"duration_ms":65704,"significance":"If the reliability claim were established, this would be a useful contribution to a real problem: agentic skill evaluation is indeed often anecdotal, and the identified self-correction bias is plausible and serious. The structural executor/grader separation, the pre-execution assertions, the first-attempt rule, and the grounded LV1/LV2 suggestion tiers are concrete and implementable design ideas. The formalization in Section 3 is clean, and the paper is unusually candid about its limitations. However, the current evidence consists of one qualitative case plus six runs with no ground-truth labels, no error bars, and no stated artifact release; the central reliability claim is therefore not yet supported at the level asserted in the abstract and conclusion.","major_comments":[{"comment":"The headline claim that AEVAL 'converts spurious 100% pass rates into reproducible first-attempt fail signals' is not established by the evidence offered. The only direct demonstration is a single deliberately degraded segmentation skill with no replicates. The matched cross-backend study in Table 1 has six runs total, overlapping pass-rate ranges (70–89%), and no confidence intervals, and Section 7 states that a full quantitative study is deferred. Since the reliability of the gate signal is the paper's central claim, the authors need either repeated runs with ground-truth first-attempt labels and variance estimates, or a substantial weakening of the abstract/conclusion wording.","section":"Section 4.3; Section 8; Table 1"},{"comment":"The first-attempt grading rule makes the gate signal depend on the grader's causal attribution of self-corrections from the transcript τ. Section 8 concedes that 'complex multi-step corrections can be misattributed,' yet no experiment measures false-positive/false-negative rates for this attribution step. Table 1 shows that graders from two backends assign the same root causes to different severity tiers (causal share 0.41 vs 0.82); if similar instability affects whether a correction is judged causally required for an assertion, the first-attempt pass rate inherits that variance. This is load-bearing for the 'reliable gate signal' claim and needs a targeted evaluation, e.g., transcripts with injected corrections of known causal status.","section":"Section 3 vs Section 4.3/7"},{"comment":"The formal model defines self-correction as skill-file edits Δ(τ)≠∅, but the implementation appears to count any applied change as a correction. In the segmentation case, the agent 'patched the action name in its generated runner script' — an edit to a generated artifact, not necessarily a skill file — yet this is recorded as self-correction and causes assertions to FAIL. This ambiguity makes the set of events requiring causal attribution imprecise and can change the reported first-attempt pass rate depending on what is classified as a correction. The authors should align the formal definition with the implementation and specify which files lie in the protected skill directory.","section":"Section 4.2; Section 8"},{"comment":"The ban on simulated outputs is enforced by instruction rather than sandboxing, and Section 8 acknowledges that 'a sufficiently capable agent could in principle violate it.' Because the transcript is the grader's only evidence of what happened, a fabricated transcript would silently corrupt the gate signal. The proposed mitigation (transcript audit plus grader verification) is not validated. For a framework whose title and abstract emphasize determinism, this is a substantive reliability gap; either sandbox the executor's write scope or provide evidence that violations are detected.","section":"Section 4.2; Section 8"}],"minor_comments":[{"comment":"No artifact availability URL or data release is stated. For a paper whose core claim is reproducible testing, please provide a link to the code/data or explicitly state availability conditions.","section":"Section 6"},{"comment":"The term 'deterministic' is used for a pipeline whose subagents are stochastic LLMs. Define the intended meaning (e.g., fixed change-triggered protocol, fixed artifact schema, reproducible trigger conditions) to avoid overclaiming.","section":"Throughout"},{"comment":"The 'protocol fidelity' column reports one drift among six runs. Since the framework detected it by structural checks, add a sentence on what downstream CI is expected to do when fidelity=false; otherwise the reader cannot see how this failure mode is handled.","section":"Section 7.1"},{"comment":"The paper uses inconsistent spellings 'AEVAL' and 'AEV AL' (including in the title and header). Please unify.","section":"Title/Introduction"}],"recommendation":"major_revision","confidential_remarks":"The core mechanism is worth pursuing, but the manuscript's claims outrun its evidence. I would like to see either a substantive quantitative validation (repeated runs, injected ground-truth corrections, variance reporting) or a clear narrowing of the reliability claims. The absence of an artifact release is also a concern for a paper whose central selling point is reproducible, auditable evaluation."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a systems paper with a genuinely useful idea and thin validation. The design—separate executor from grader, write assertions before outputs, grade only first-attempt behavior—addresses a real failure mode in agent evaluation, and the paper names it clearly (self-correction bias). The formalization in Section 3 is simple but sufficient, and the tiered suggestion scheme is a nice operational detail. The authors are also unusually honest: Section 8 flags the two biggest weaknesses plainly.\n\nWhat is new: change-triggered evaluation with a per-skill eval.config, the executor/grader information barrier, and the first-attempt grading rule. None of the cited platforms do this. That's worth building on.\n\nThe soft spots are real and in proportion. The empirical section is one deliberately degraded case plus six runs across two backends; there is no released code or data, and the full quantitative study is deferred. More importantly, the gate signal—the first-attempt pass rate—depends on the grader correctly deciding which self-corrections were causally required for an assertion to pass. That is an LLM judgment over a transcript, and the paper admits multi-step corrections can be misattributed. The cross-runtime results don't measure that error rate; they show causal share (a severity tiering) differing 0.41 vs 0.82 across backends. That's evidence the grader's causal classification is unstable, which should be concerning for the gate as well, though it's not direct evidence of gate failure. The variable assertion-list denominators (10–18) also make pass rates shaky as a route signal unless the per-assertion results are what CI consumes.\n\nOne ambiguity worth fixing: the formal model defines self-correction as skill-file edits Δ(τ)≠∅, but the case study counts the agent patching its generated runner script as a self-correction. The intuition is clear, but the definition needs to match the implementation, or the first-attempt rule becomes under-specified.\n\nNone of this kills the paper's contribution. The design principle is sound: grading the state before corrections is strictly better than grading the patched state. What's missing is evidence that the grader can reliably see the difference in practice. That's an empirical question, not a conceptual dead end.\n\nFor a reader: this is aimed at people building skill marketplaces or CI for agent workflows, and it will likely influence how they structure tests. It deserves a serious referee—send it out—with the expectation of major revision: code and data release, a ground-truth study of causal attribution, and a precise definition of self-correction.","headline":"The executor/grader separation and first-attempt rule are a real design contribution, but the paper's central reliability claim rests on an untested LLM causal-attribution step that the authors themselves admit can fail.","tokens_in":10507,"tokens_out":4718,"would_cite":true,"duration_ms":40311,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper's central claim is that agentic skill evaluation becomes deterministic when the grading component is separated from the executing component and only first-attempt behavior is graded.","keywords":["agentic skills","deterministic evaluation","self-correction bias","first-attempt grading","executor-grader separation","CI testing","eval contract"],"falsifier":"Seed a set of skills with known first-attempt defects, have each defect repaired by a self-correction of varying complexity (a single edit versus a multi-step repair), run AEVAL, and compare its FAIL verdicts against the seeded ground truth; if increasingly complex repairs are increasingly misclassified as passes, or if the same defect flips its verdict across agent backends, the causal-attribution guarantee fails.","tokens_in":9609,"feed_emoji":"⚙️","tokens_out":4465,"duration_ms":42434,"temperature":0.7,"pith_summary":"Skills are installable packages that teach LLM agents domain workflows, yet evaluating a skill today usually means a developer watches a demo and forms a subjective impression. This paper claims that this anecdotal practice can be replaced by a deterministic, change-triggered test pipeline, and that the central reason naive pipelines lie is self-correction bias: the same agent that executes a skill also grades it, so it silently patches errors and then certifies its patched output as passing. AEVAL separates execution from grading, requires assertions to be written before outputs are seen, and grades only the first-attempt trajectory, marking any assertion that needed a self-correction as FAIL even if it later passed. On production skills, the report is that spurious 100% pass rates become reproducible first-attempt fail signals with an audited record of every fix. A sympathetic reader would care because this turns skill evaluation from a subjective demo into a merge-gating signal that CI and downstream statistical methods can consume.","feed_headline":"Separating grading from execution turns 100% pass rates into fails","feed_subtitle":"A first-attempt grading rule stops agents from patching their own failures and passing their own tests.","key_machinery":"The load-bearing mechanism is the information barrier between executor and grader: an executor lane with write access runs the skill while a separate, read-only grader subagent evaluates it using only the test prompt, transcript, outputs, and pre-execution assertions. Everything else hangs off that barrier — the first-attempt grading rule, explicit self-correction tracking, the eval.config contract that turns each skill into a declarative test harness, and the grounded-in-evidence requirement that rejects any fix suggestion without a failed assertion or logged correction. Together these make the emitted gate signal a function of the skill as written and first attempted, not of the skill as p","core_discovery":"The central claim is that naive agentic evaluators systematically over-report skill quality because agents self-correct, and that AEVAL removes this bias structurally rather than by prompting. The grader is a separate read-only subagent that sees only the test prompt, the execution transcript, output files, and assertions written before the run; the first-attempt grading rule then declares FAIL for any assertion whose later success required a self-correction. In the reported segmentation-skill case, a skill with mismatched action names produced a 10/10 terminal pass after the agent patched its own runner, but the protocol marked the action-name and configuration-consistency assertions as FAI","pith_inferences":["If first-attempt pass rate is a reliable gate, eval.config becomes a regression-test suite for agentic workflows; a skill's natural-language contract could be diffed and reviewed exactly like test code.","The measured cross-runtime disagreement on causal share suggests that a single global severity threshold under-fits; a natural extension is per-runtime calibration of the LV1/LV2 boundary against historical merge outcomes.","The causal-attribution assumption could be stress-tested by seeding defects of increasing repair complexity; if graders systematically miss multi-step self-corrections, the structural guarantee is only partial.","The paper's instruction-level ban on simulated outputs points to a testable hardening: sandboxing the executor's filesystem write scope so it cannot alter the original skill directory would make the self-correction signal enforceable rather than prompt-dependent."],"forward_implications":["Downstream CI can gate merges on a reproducible first-attempt pass rate, so skill regressions that an agent would have hidden behind self-corrections now block the merge.","Every skill can declare its own evaluation contract once, and every subsequent push replays the full install, execute, grade, and suggest-fixes cycle with no human in the loop.","Per-run artifacts such as grading.json and benchmark.json enable comparison against a last-known-good baseline, flagging pass-rate deltas and newly failing assertions.","Fix suggestions are tied to specific evidence and can be applied as one-click merge-request comments, shortening the test-fail-fix loop for skill authors.","Because the executor interface is runtime-agnostic, the same skill and contract can be replayed across different agent backends to surface runtime-specific compatibility issues."],"fun_headline_variants":["Separate grader from executor to expose self-correcting agents","First-attempt grading rule turns 100% pass rates into fails","Agent self-correction bias: deterministic testing for skill workflows","AEVAL: structural fix for agents grading their own patched outputs","Skill testing: stop agents from passing their own fixes"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The protocol's signal is only as trustworthy as the grader's ability to read the execution transcript and correctly decide which self-corrections were causally required for each assertion to pass.","fun_headline_variants_meta":{"raw":{"variants":["Separate grader from executor to expose self-correcting agents","First-attempt grading rule turns 100% pass rates into fails","Agent self-correction bias: deterministic testing for skill workflows","AEVAL: structural fix for agents grading their own patched outputs","Skill testing: stop agents from passing their own fixes"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000912,"raw_usage":{"total_tokens":3813,"prompt_tokens":858,"completion_tokens":2955,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":602,"completion_tokens_details":{"reasoning_tokens":2870}},"tokens_in":602,"tokens_out":2955,"duration_ms":21858,"temperature":1.0,"reasoning_tokens":2870,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-01T23:14:21.742886+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Seed a set of skills with known first-attempt defects, have each defect repaired by a self-correction of varying complexity (a single edit versus a multi-step repair), run AEVAL, and compare its FAIL verdicts against the seeded ground truth; if increasingly complex repairs are increasingly misclassified as passes, or if the same defect flips its verdict across agent backends, the causal-attribution guarantee fails.","supporting_citations":[],"review_version":1}