Pith. sign in

REVIEW 3 major objections 5 minor 87 references

SkillTV-Bench: Benchmarking How Well Judges Perform on Skill-Augmented Agentic Execution

T0 review · 3 major / 5 minor · reviewed 2026-08-08 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A reusable text skill, evolved on disjoint examples, lifts an agent judge's trajectory-verification accuracy by 14.8 points.

desk verdict Solid benchmark and careful study, but the inherited source-verifier labels are the load-bearing assumption; the +14.8 pp is improvement against that verifier, not absolute ground truth. read the letter →

arxiv 2608.05573 v1 pith:YGNS2Q2L submitted 2026-08-06 cs.AI

Zhi Han , Chenxi Zeng , Liuhaichen Yang , Zihan Guo , Ming Zhou , Yang Li This is my paper · ORCID
classification cs.AI
keywords LLM-as-a-Judgeagenttrajectoryverificationskill-augmentedagentsJudgeSkillbenchmarkconstructionrolloutselectiontest-timescalingverifierevolution
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

SkillTV-Bench is a 681-case benchmark of real skill-augmented agent executions across 50 tasks and eleven domains, each packaged as instruction, normalized trajectory, task-time skills, and inspectable artifacts with a hidden source label. The paper claims that existing judges—direct LLMs, ensembles, rubric-based judges, and agent judges without a skill—systematically fail on this benchmark, mostly by accepting plausible-looking failed executions. It proposes SkillTV-Evolve, which externalizes verification knowledge as a reusable JudgeSkill that plans inspections, logs evidence, and issues verdicts, and refines that skill in a gated loop on a disjoint development pool. The central result is that the refined skill lifts the same agent judge's accuracy from 43.8% to 58.6% on the held-out set, and lifts selected-trajectory success at ten rollouts from 22.9% to 45.5%. If correct, this shows verification of long-horizon agent executions can be substantially improved without updating the model.

What carries the argument

The central objects are the judge-case tuple (I, τ, S, A)—instruction, normalized trajectory, task-time skills, and artifacts—and the JudgeSkill, a human-readable procedure that decomposes judging into an inspection plan, an executed inspection log, and a final evidence-grounded verdict. The JudgeSkill carries the argument because it converts abstract rubric criteria into concrete evidence-seeking actions across artifact checks, skill-usage checks, and reasoning-process checks. The second mechanism is the gated evolution loop: misjudged development cases are compressed into rewrite briefs, an optimizer proposes revised skill text, and a fixed 36-case gate with weighted score G = 0.5*BalAcc + 0.3*Recall_fail + 0.2*Precision_pass decides promotion, so the skill improves without changing model weights.

What would settle it

Take a random sample of the 681 SkillTV-Bench cases and have independent human annotators (or a second verifier) relabel pass/fail from the packaged instruction, trajectory, skills, and artifacts; if a substantial fraction of labels flip, recompute the 14.8-point accuracy gain and the 45.5% success@10 against the corrected labels.

Watch

Extended reading notes

Core claim

The core discovery is that verification of skill-augmented agent executions is a joint consistency problem over instruction, trajectory, skill use, and artifact, and that the dominant failure of existing judges is false acceptance of failed trajectories that look plausible. SkillTV-Bench operationalizes this by packaging each case as (I, τ, S, A) with a hidden source-verifier label, and the paper shows that frontier direct judges, ensembles, rubrics, and agent judges without a skill all land near or below 0.586 balanced accuracy, with high recall but low precision on the pass class. The paper's own mechanism, an explicit JudgeSkill that guides an agent judge through an inspection plan, an inspection log, and an evidence-grounded verdict, and that is refined on misjudged development cases through a gated evolution loop, reaches 58.6% accuracy and 0.634 balanced accuracy, and reduces false accepts from 105 to 71 on the evaluation set. The paper therefore claims that benchmark-grounded evolution can convert verification errors into reusable procedural knowledge, and that this knowledge transfers to offline rollout-pool selection, where the refined skill lifts selected-trajectory success to 45.5% at ten rollouts.

Load-bearing premise

The load-bearing premise is that the source verifier labels from SkillsBench are correct for all 681 cases; the paper adopts them without human validation or label-error analysis, so every accuracy and selection number stands or falls with that verifier.

Editorial extensions

If this is right

  • Judging long-horizon agent runs can be treated as a trainable artifact: a textual skill, not model weights, carries the improvement.
  • SkillTV-Bench's error profile implies that future judges should be evaluated for false-accept rate, not just overall accuracy, since plausible failures dominate.
  • Verification gains translate directly to test-time scaling: a cleaner predicted-pass set makes rollout selection more reliable, so investing in verifiers can be an alternative to adding more rollouts.
  • The evolution protocol is reusable: any judge that emits plans, logs, and verdicts can be improved on a disjoint development pool without touching the underlying LLM.
  • The paper's stated limitation means the method today filters candidates rather than improving task execution; returning judge feedback to task agents is the natural next step.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the source-verifier labels contain noise, the +14.8-percentage-point gain could partly reflect label-noise patterns rather than true task success; a human relabeling study on a sample would clarify this.
  • The JudgeSkill may generalize to other backbone models or other trajectory sources, since the paper does not test this and the mechanism suggests domain-agnostic inspection heuristics.
  • The same evolution loop could be driven by other error signals—human preference judgments or downstream task outcomes—turning the verifier into a general learning signal rather than only a filter.
  • A closed loop that returns evidence-grounded feedback to task agents could make SkillTV-Bench self-expanding, because newly validated trajectories would refresh the development pool.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. SkillTV-Bench is a 681-case trajectory-verification benchmark derived from the authors' prior SkillsBench executions, spanning 50 source tasks and eleven domains, and split into a 478-case evolution pool and a task-disjoint 203-case evaluation set. Each judge case packages the original instruction, a normalized step-wise trajectory, the task-time skills, and inspectable artifacts, with pass/fail labels inherited from SkillsBench's automated source verifier. The paper reports that existing judges (GPT-5.2, Claude Sonnet 4.6, Gemini 3.1 Pro, majority vote, rubric-based judging, and a tool-using agent judge) all perform poorly on the benchmark, primarily by accepting plausible-but-failed executions. The proposed SkillTV-Evolve method externalizes verification knowledge as a human-readable JudgeSkill that structures judging into an inspection plan, an inspection log, and an evidence-grounded verdict, and iteratively rewrites the skill with an LLM-based optimizer whose candidates are promoted only if they improve a fixed 36-case development gate. The refined skill v4 improves the same Claude-Sonnet-4.6-based agent judge from 43.8% to 58.6% accuracy (McNemar p < 10^-4) and from 0.568 to 0.634 balanced accuracy on the held-out set, and increases verifier-guided selected-trajectory success at ten rollouts from a 22.9% single-rollout base rate to 45.5%, with all intermediate skill versions reported.

Significance. If the headline numbers survive an independent label check, this is a solid, well-scoped contribution. The benchmark fills a genuine gap - skill-aware, evidence-grounded, agentic trajectory verification - and the +14.8 pp gain from an explicit, externally evolvable skill, with the underlying model held fixed, is a concrete and falsifiable result. I specifically credit the task-disjoint held-out split, the fixed development gate that prevents evaluation-label feedback, the paired significance test, the monotone held-out improvement across four skill versions, the transparent per-domain table that shows two zero-gain domains, the honest limitation paragraph that confines the claims to offline verification and selection, and the release of code, data, and named harness versions (Claude Code v2.1.19, Harbor). The main risk is external validity: every number in Tables 2-5 is measured against source-verifier labels whose fidelity to true task success is never tested, so the magnitude of the reported improvements should be read as conditional on label quality.

major comments (3)
  1. [Benchmark construction; Eq. (1)] Eq. (1) and the 'Benchmark construction' section define y_i as the source-verifier label inherited from SkillsBench, and the five-stage construction pipeline validates schema conformance, token limits, and cross-field consistency but never the correctness of the pass/fail decision itself. This is the load-bearing premise for the headline numbers: all of Table 2, Table 5, and the evolution objective in Eq. (4) are computed against these labels, and the benchmark's stated purpose is precisely to expose 'plausible-but-failed' executions, which is the class most vulnerable to verifier blind spots. If the source verifier mislabels a substantial fraction of the 681 cases, then the measured judge weaknesses and the +14.8 pp gain could partly reflect the skill learning the verifier's biases on 36 task families of the same SkillsBench lineage, rather than learning to detect true task failure. I do not regard this as a circularity problem, because the evaluation set is task-disjoint and the evaluation labels never enter evolution; it is an untested external-validity assumption. Please add a human-annotation audit on a stratified sample of the 681 labels (I would suggest 100-200 cases judged with the source-verifier output blinded), report agreement and disagreement patterns, and recompute the headline results on the human-confirmed subset, or alternatively state the inherited-label assumption explicitly and bound how label error would move the 14.8 pp estimate.
  2. [SkillTV-Evolve development gate; Eq. (4); Table 4] Two properties of the development gate in SkillTV-Evolve need clarification and strengthening. First, the sentence describing the gate as 'a fixed 36-case development gate that preserves all 14 task families' is ambiguous in a way that touches the paper's non-circularity guarantee: the evaluation set also contains 14 source tasks, and if any of the gate's 36 cases come from those tasks, then skill promotion uses evaluation-task labels and the claimed disjointness is compromised. Please state explicitly that the gate is drawn exclusively from the 478-case evolution pool and is task-disjoint from the 203-case evaluation set, and define the phrase 'matches the full-set baseline behavior,' which is currently undefined. Second, the promotions between adjacent versions in Table 4 rest on gate-score differences of at most 1.1 points (G = 49.2, 50.3, 52.6, 52.7) on a 36-case gate, and the v3-to-v4 promotion in particular is a 0.1-point difference that is well within the sampling noise of such a small set. The monotone held-out improvement from 50.8% to 58.6% across versions is reassuring evidence that the progression is not pure selection noise, but the paper should report bootstrap confidence intervals for G at each version, or enlarge the gate, before claiming that the loop reliably promotes better skills.
  3. [Automated JudgeSkill evolution section; RQ2] The evolution loop is not reproducible from the paper as written. The 'Automated JudgeSkill evolution' section does not specify the optimizer (it says only that 'an optimizer receives the current skill... and proposes a revised skill'), the model and configuration behind it, the number of evolution rounds, the sampling scheme for the 'sampled development executions,' or the per-round budget. Most importantly, the full texts of JudgeSkill v1-v4, and especially the final v4 skill that produces the headline numbers, are not included in the paper or appendix, even though the method's contribution is exactly an explicit, auditable, human-readable verification skill. The same gap applies to the three judge-facing failure-prior references synthesized from prior failure analyses, whose content is never described. Without the skill text and the optimizer configuration, readers cannot reproduce SkillTV-Evolve, cannot audit what was learned, and cannot test the claim that verification knowledge has been externalized; please include the full skill texts and complete loop hyperparameters, or point precisely to where they are released in the repository.
minor comments (5)
  1. [RQ3 protocol; Table 5] In the RQ3 protocol, the definition of the 'Best rollout' column in Table 5 is not given, and the selection rule for the case where no trajectory is predicted as pass is not stated; the paper should also confirm that every one of the 14 source tasks has at least 10 stored trajectories, since 'sampling n without replacement' for n = 10 requires pool sizes of at least 10.
  2. [RQ2; Table 2] For the main paired comparison (43.8% vs 58.6%, Table 2), please report the McNemar discordant-pair counts (b and c) and a confidence interval for the 14.8 pp difference, rather than the p-value alone.
  3. [Table 2 and RQ1] Because 156 of 203 evaluation cases are failures (76.8%), a trivial all-fail classifier would achieve 76.8% accuracy, above every method in Table 2; adding this majority-class baseline to the Table 2 discussion and noting that the balanced-accuracy and rollout-selection results are the more robust evidence would help readers interpret the accuracy numbers.
  4. [Benchmark construction] In 'Benchmark construction,' the sentence 'We believe there is no need to introduce verifiers to select pass candidate if the agentic trajectories of the task are completely correct or wrong' is ungrammatical, and the underlying design choice (dropping all-pass and all-fail tasks) deliberately inflates benchmark difficulty; please state the rationale explicitly and note that the 22.9% single-rollout success rate is a construction-dependent quantity rather than a natural base rate.
  5. [Automated JudgeSkill evolution] In 'Automated JudgeSkill evolution,' the relationship between the gate's 'all 14 task families' and the evaluation set's '14 source tasks' should be clarified in one unambiguous sentence (this overlaps with major comment 2); likewise, 'prior empirical analyses paper' and a few other phrases in the same section need copyediting.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the 14.8-pp improvement is measured on a disjoint held-out set against fixed external labels, and the evolution loop's training signal is confined to the development pool.

full rationale

SkillTV-Bench's construction inherits trajectories and source-verifier labels from SkillsBench (a prior work by overlapping authors), but that is a data-provenance citation, not a load-bearing circular argument. The benchmark's labels y_i are fixed before judging (Eq. 1), the judge sees only x_i = (I, tau, S, A) (Eq. 2), and the JudgeSkill is evolved on a 478-case development pool explicitly disjoint from the 203-case evaluation set. Candidate rewrites are promoted only through a fixed 36-case development gate (Eq. 4), and the paper states that evaluation labels are never used for skill evolution. The headline +14.8-pp accuracy gain and the Success@10 gain from 22.9% to 45.5% are empirical comparisons on the held-out set against these fixed labels; no fitted parameter is renamed as a prediction, and no equation defines the reported improvement in terms of the JudgeSkill's own training objective. The main caveat is external validity rather than circularity: the ground-truth labels come verbatim from SkillsBench's source verifier without human validation, so if that verifier is noisy or biased, the measured improvement may partly reflect learning those biases. That is a correctness and validity risk, not a circularity, because the labels are external to the judge-evaluation pipeline and the held-out split prevents the evolution loop from directly optimizing the reported numbers.

Assumptions & free parameters 4 free parameters · 4 assumptions · 1 invented entities

The central empirical claim rests on the JudgeSkill text (a learned artifact), the gate weights, and the source-verifier labels. The labels are the most consequential unvalidated input: they come from a prior automated verifier and are never checked by humans. The JudgeSkill content is omitted from the manuscript, so the main intervention is not directly inspectable.

free parameters (4)
  • JudgeSkill v4 text (final evolved skill) = Not included in manuscript
    The skill is repeatedly revised by the optimizer on development misjudged cases until it maximizes the gate score G. It is the central artifact whose content determines the reported accuracy.
  • Gate objective weights (G = 0.5*BalAcc + 0.3*Recall_fail + 0.2*Precision_pass) = 0.5, 0.3, 0.2
    Chosen by the authors to balance false-pass rejection against calibration; no sensitivity analysis is reported.
  • Development gate composition (36 cases, 27 fail, 9 pass) = 36 / 27 / 9
    The fixed 36-case subset is used for candidate promotion and is, in effect, a validation set whose composition is tuned to preserve task families and match baseline behavior.
  • Judge-facing failure-prior references = Synthesized from prior failure analyses
    The three compact references supply failure priors to the judge; their content and length are chosen by the authors and not ablated.
assumptions (4)
  • domain assumption Source-verifier labels in SkillsBench are correct and sufficiently complete to serve as ground truth for skill-augmented task success.
    All benchmark labels y_i are taken verbatim from SkillsBench's source verifier; the paper provides no human re-annotation or label error analysis. The headline accuracy numbers are computed against these labels.
  • domain assumption The normalized step-wise trajectory representation preserves all evidence necessary for verification.
    The rule-based converter maps raw agent event streams to a fixed schema; if the converter discards signal needed to detect failure, the benchmark understates judge capability.
  • domain assumption Task-time skills S_i are correctly matched to trajectories and are the same skills available to the task agent during execution.
    The benchmark joins trajectories with the skill sets from SkillsBench; any mismatch between recorded and actually used skills would distort skill-aware evaluation.
  • domain assumption The 203-case evaluation set is a representative, unbiased sample of the 50-task, 681-case pool.
    Domain-balanced sampling is used, but the sampling algorithm and stratification procedure are not detailed, so representativeness cannot be verified from the manuscript.
invented entities (1)
  • JudgeSkill independent evidence
    purpose: A reusable, human-readable textual skill that externalizes verification knowledge, guiding the agent judge through inspection planning, evidence logging, and verdict issuance.
    The skill is the intervention being tested; its value is measured by held-out accuracy and rollout-selection success, which are falsifiable quantitative outcomes.

how reviews work

0 comments
Cite this review

Pith. "Pith review of SkillTV-Bench: Benchmarking How Well Judges Perform on Skill-Augmented Agentic Execution." pith.science (2026). https://pith.science/paper/YGNS2Q2L

@misc{pith2026260805573,
  author       = {Pith},
  title        = {Pith review of: SkillTV-Bench: Benchmarking How Well Judges Perform on Skill-Augmented Agentic Execution},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/YGNS2Q2L}},
  note         = {Machine review of arXiv:2608.05573}
}
read the original abstract

LLM agents increasingly execute long-horizon tasks through tool use and environment interaction, shifting evaluation from final-response scoring to verification of complete executions. For skill-augmented agents, verification additionally requires the procedural knowledge encoded in task-time skills, because this knowledge indicates what evidence to inspect and which failures are task-critical. However, existing judge benchmarks often expose final responses or static trajectories, and rarely combine task-time skills with directly inspectable artifacts and environments. We therefore introduce SkillTV-Bench, a 681-case benchmark of real agent trajectories from 50 tasks across eleven domains, designed to evaluate skill-aware trajectory verification for both LLM-as-a-Judge and Agent-as-a-Judge methods. Additionally, we propose SkillTV-Evolve, which externalizes verification knowledge as a reusable JudgeSkill that guides an agent judge to plan targeted inspections and issue evidence-grounded verdicts. On a disjoint development pool, an automated evolution loop further refines the JudgeSkill using misjudged cases. On SkillTV-Bench, the refined skill increases the same agent judge's accuracy by 14.8 percentage points. In offline rollout-pool selection, it increases selected-trajectory success from 22.9% with one rollout to 45.5% with ten rollouts. The code and data are available at https://github.com/HanZhi306/SkillTV-Bench

Figures

Figures reproduced from arXiv: 2608.05573 by the authors.

Figure 1
Figure 1. Overview of SkillTV-Bench and SkillTV-Evolve. [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Overview of SkillTV-Bench construction and SkillTV-Evolve, including gated JudgeSkill refinement and plan-driven, [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Offline verifier-guided rollout selection over 14 [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Agent-as-a-judge comparison at ten rollouts over [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

87 extracted references · 21 canonical work pages

  1. [1]

    2023 , eprint=

    ReAct: Synergizing Reasoning and Acting in Language Models , author=. 2023 , eprint=

    2023

  2. [2]

    2023 , eprint=

    Toolformer: Language Models Can Teach Themselves to Use Tools , author=. 2023 , eprint=

    2023

  3. [3]

    2023 , eprint=

    Tree of Thoughts: Deliberate Problem Solving with Large Language Models , author=. 2023 , eprint=

    2023

  4. [4]

    2023 , eprint=

    Reflexion: Language Agents with Verbal Reinforcement Learning , author=. 2023 , eprint=

    2023

  5. [5]

    2023 , eprint=

    Self-Refine: Iterative Refinement with Self-Feedback , author=. 2023 , eprint=

    2023

  6. [6]

    2023 , eprint=

    G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment , author=. 2023 , eprint=

    2023

  7. [7]

    2024 , eprint=

    Prometheus 2: An Open Source Language Model Specialized in Evaluating Other Language Models , author=. 2024 , eprint=

    2024

  8. [8]

    2024 , eprint=

    RewardBench: Evaluating Reward Models for Language Modeling , author=. 2024 , eprint=

    2024

Show all 87 references
  1. [9]

    2025 , eprint=

    JudgeBench: A Benchmark for Evaluating LLM-based Judges , author=. 2025 , eprint=

    2025

  2. [10]

    2023 , eprint=

    Let's Verify Step by Step , author=. 2023 , eprint=

    2023

  3. [11]

    2023 , eprint=

    LEVER: Learning to Verify Language-to-Code Generation with Execution , author=. 2023 , eprint=

    2023

  4. [12]

    2024 , eprint=

    -bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains , author=. 2024 , eprint=

    2024

  5. [13]

    2024 , eprint=

    WorkArena: How Capable Are Web Agents at Solving Common Knowledge Work Tasks? , author=. 2024 , eprint=

    2024

  6. [14]

    2024 , eprint=

    Agent-as-a-Judge: Evaluate Agents with Agents , author=. 2024 , eprint=

    2024

  7. [15]

    2026 , eprint=

    SkillsBench: Benchmarking How Well Agent Skills Work Across Diverse Tasks , author=. 2026 , eprint=

    2026

  8. [16]

    American Journal of Physics , volume=

    Interactive-engagement versus traditional methods: A six-thousand-student survey of mechanics test data for introductory physics courses , author=. American Journal of Physics , volume=

  9. [17]

    2025 , eprint=

    TRAIL: Trace Reasoning and Agentic Issue Localization , author=. 2025 , eprint=

    2025

  10. [18]

    2026 , eprint=

    Demystifying the Lifecycle of Failures in Platform-Orchestrated Agentic Workflows , author=. 2026 , eprint=

    2026

  11. [19]

    2026 , eprint=

    Characterizing Faults in Agentic AI: A Taxonomy of Types, Symptoms, and Root Causes , author=. 2026 , eprint=

    2026

  12. [20]

    2025 , eprint=

    Where LLM Agents Fail and How They can Learn From Failures , author=. 2025 , eprint=

    2025

  13. [21]

    2026 , eprint=

    AgentRx: Diagnosing AI Agent Failures from Execution Trajectories , author=. 2026 , eprint=

    2026

  14. [22]

    2025 , eprint=

    Aegis: Taxonomy and Optimizations for Overcoming Agent-Environment Failures in LLM Agents , author=. 2025 , eprint=

    2025

  15. [23]

    2025 , eprint=

    Can Agents Fix Agent Issues? , author=. 2025 , eprint=

    2025

  16. [24]

    2026 , eprint=

    An Empirical Study of Bugs in Modern LLM Agent Frameworks , author=. 2026 , eprint=

    2026

  17. [25]

    2025 , eprint=

    Exploring Autonomous Agents: A Closer Look at Why They Fail When Completing Tasks , author=. 2025 , eprint=

    2025

  18. [26]

    2026 , eprint=

    Aligning Agents via Planning: A Benchmark for Trajectory-Level Reward Modeling , author=. 2026 , eprint=

    2026

  19. [27]

    2026 , eprint=

    OS-Themis: A Scalable Critic Framework for Generalist GUI Rewards , author=. 2026 , eprint=

    2026

  20. [28]

    2026 , eprint=

    AJ-Bench: Benchmarking Agent-as-a-Judge for Environment-Aware Evaluation , author=. 2026 , eprint=

    2026

  21. [29]

    2025 , eprint=

    AgentRewardBench: Evaluating Automatic Evaluations of Web Agent Trajectories , author=. 2025 , eprint=

    2025

  22. [30]

    2024 , eprint=

    Large Language Monkeys: Scaling Inference Compute with Repeated Sampling , author=. 2024 , eprint=

    2024

  23. [31]

    2024 , eprint=

    Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters , author=. 2024 , eprint=

    2024

  24. [33]

    Harbor: A Framework for Evaluating and Optimizing Agents and Models in Container Environments , year =

  25. [34]

    2026 , eprint=

    Terminal-Bench: Benchmarking Agents on Hard, Realistic Tasks in Command Line Interfaces , author=. 2026 , eprint=

    2026

  26. [35]

    2025 , month = dec, howpublished =

    Introducing. 2025 , month = dec, howpublished =

    2025

  27. [36]

    2026 , month = feb, howpublished =

    Introducing. 2026 , month = feb, howpublished =

    2026

  28. [37]

    2026 , month = feb, howpublished =

    2026

  29. [38]

    2026 , eprint=

    GEPA: Reflective Prompt Evolution Can Outperform Reinforcement Learning , author=. 2026 , eprint=

    2026

  30. [39]

    2026 , eprint=

    SkillLearnBench: Benchmarking Continual Learning Methods for Agent Skill Generation on Real-World Tasks , author=. 2026 , eprint=

    2026

  31. [40]

    2026 , eprint=

    SkillRevise: Improving LLM-Authored Agent Skills via Trace-Conditioned Skill Revision , author=. 2026 , eprint=

    2026

  32. [41]

    2026 , eprint=

    Skill-RM: Unifying Heterogeneous Evaluation Criteria via Agent Skill , author=. 2026 , eprint=

    2026

  33. [42]

    2026 , eprint=

    Inference-Time Scaling of Verification: Self-Evolving Deep Research Agents via Test-Time Rubric-Guided Verification , author=. 2026 , eprint=

    2026

  34. [43]

    2024 , eprint=

    Large Language Models as Optimizers , author=. 2024 , eprint=

    2024

  35. [44]

    2023 , eprint=

    Promptbreeder: Self-Referential Self-Improvement Via Prompt Evolution , author=. 2023 , eprint=

    2023

  36. [45]

    2025 , eprint=

    TRAJECT-Bench:A Trajectory-Aware Benchmark for Evaluating Agentic Tool Use , author=. 2025 , eprint=

    2025

  37. [46]

    2026 , eprint=

    SkillJuror: Measuring How Agent Skill Organization Changes Runtime Behavior , author=. 2026 , eprint=

    2026

  38. [47]

    2026 , eprint=

    Agent Skill Evaluation and Evolution: Frameworks and Benchmarks , author=. 2026 , eprint=

    2026

  39. [48]

    A.; Tan, S.; Soylu, D.; Ziems, N.; Khare, R.; Opsahl-Ong, K.; Singhvi, A.; Shandilya, H.; Ryan, M

    Agrawal, L. A.; Tan, S.; Soylu, D.; Ziems, N.; Khare, R.; Opsahl-Ong, K.; Singhvi, A.; Shandilya, H.; Ryan, M. J.; Jiang, M.; Potts, C.; Sen, K.; Dimakis, A. G.; Stoica, I.; Klein, D.; Zaharia, M.; and Khattab, O. 2026. GEPA: Reflective Prompt Evolution Can Outperform Reinforc...

    2026 arXiv

  40. [49]

    Anthropic . 2026. Introducing Claude Sonnet 4.6 . https://www.anthropic.com/news/claude-sonnet-4-6. Accessed: 2026-07-18

    2026

  41. [50]

    Barke, S.; Goyal, A.; Khare, A.; Singh, A.; Nath, S.; and Bansal, C. 2026. AgentRx: Diagnosing AI Agent Failures from Execution Trajectories. arXiv:2602.02475

    2026

  42. [51]

    V.; Ré, C.; and Mirhoseini, A

    Brown, B.; Juravsky, J.; Ehrlich, R.; Clark, R.; Le, Q. V.; Ré, C.; and Mirhoseini, A. 2024. Large Language Monkeys: Scaling Inference Compute with Repeated Sampling. arXiv:2407.21787

    2024 arXiv

  43. [52]

    Chen, T.; Jiang, G.; Cheng, P.; Huang, S.; Liu, Y.; Ni, J.; Guo, J.; Zhou, M.; Tang, K.; Liu, J.; Su, Q.; Jiang, X.; and Jiang, G. 2026 a . Skill-RM: Unifying Heterogeneous Evaluation Criteria via Agent Skill. arXiv:2606.03980

    2026 arXiv

  44. [53]

    Chen, Z.; Guo, Z.; Huang, B.; Lu, B.; Lin, J.; Zhou, Y.; and Zhang, W. 2026 b . SkillJuror: Measuring How Agent Skill Organization Changes Runtime Behavior. arXiv:2606.11543

    2026 arXiv

  45. [54]

    Deshpande, D.; Gangal, V.; Mehta, H.; Krishnan, J.; Kannappan, A.; and Qian, R. 2025. TRAIL: Trace Reasoning and Agentic Issue Localization. arXiv:2505.08638

    2025 arXiv

  46. [55]

    Ding, K.; Zhou, Y.; Jin, C.; Tong, F.; Zhou, M.; and Metaxas, D. N. 2026. Agent Skill Evaluation and Evolution: Frameworks and Benchmarks. arXiv:2606.11435

    2026 arXiv

  47. [56]

    H.; Verme, M

    Drouin, A.; Gasse, M.; Caccia, M.; Laradji, I. H.; Verme, M. D.; Marty, T.; Boisvert, L.; Thakkar, M.; Cappart, Q.; Vazquez, D.; Chapados, N.; and Lacoste, A. 2024. WorkArena: How Capable Are Web Agents at Solving Common Knowledge Work Tasks? arXiv:2403.07718

    2024 arXiv

  48. [57]

    Fernando, C.; Banarse, D.; Michalewski, H.; Osindero, S.; and Rocktäschel, T. 2023. Promptbreeder: Self-Referential Self-Improvement Via Prompt Evolution. arXiv:2309.16797

    2023 arXiv

  49. [58]

    Harbor Framework Team . 2026. Harbor: A Framework for Evaluating and Optimizing Agents and Models in Container Environments. Software, version 0.19.0, GitHub repository. https://github.com/harbor-framework/harbor; archived at Zenodo: https://doi.org/10.5281/zenodo.20953922

    2026 doi

  50. [59]

    He, P.; Dai, Z.; He, B.; Liu, H.; Tang, X.; Lu, H.; Li, J.; Ding, J.; Mukherjee, S.; Wang, S.; Xing, Y.; Tang, J.; and Dumoulin, B. 2025. TRAJECT-Bench:A Trajectory-Aware Benchmark for Evaluating Agentic Tool Use. arXiv:2510.04550

    2025

  51. [60]

    Y.; Shin, J.; Welleck, S.; Neubig, G.; Lee, M.; Lee, K.; and Seo, M

    Kim, S.; Suk, J.; Longpre, S.; Lin, B. Y.; Shin, J.; Welleck, S.; Neubig, G.; Lee, M.; Lee, K.; and Seo, M. 2024. Prometheus 2: An Open Source Language Model Specialized in Evaluating Other Language Models. arXiv:2405.01535

    2024 arXiv

  52. [61]

    Y.; Chandu, K.; Dziri, N.; Kumar, S.; Zick, T.; Choi, Y.; Smith, N

    Lambert, N.; Pyatkin, V.; Morrison, J.; Miranda, L.; Lin, B. Y.; Chandu, K.; Dziri, N.; Kumar, S.; Zick, T.; Choi, Y.; Smith, N. A.; and Hajishirzi, H. 2024. RewardBench: Evaluating Reward Models for Language Modeling. arXiv:2403.13787

    2024 arXiv

  53. [62]

    W.; Sun, J.; Wang, S.; Tao, C.; Li, B.; Zhao, X.; Geng, H.; Wu, X.; Zhou, J.; Chen, X.; Xing, H.; Li, Y.; Zeng, Q.; Wang, D.; Wang, Y.; Chaim, R

    Li, X.; Liu, Y.; Chen, W.; You, B.; Di, Z.; He, Y.; Zheng, S.; Choe, K. W.; Sun, J.; Wang, S.; Tao, C.; Li, B.; Zhao, X.; Geng, H.; Wu, X.; Zhou, J.; Chen, X.; Xing, H.; Li, Y.; Zeng, Q.; Wang, D.; Wang, Y.; Chaim, R. B.; Jiang, P.; Shen, H.; Kong, L.; Liu, X.; Wang, R.; Liu, ...

    2026 arXiv

  54. [63]

    Li, Z.; Wu, Z.; Zhao, Y.; Yang, B.; Xie, J.; Liu, Z.; Liu, Z.; Jin, K.; Liang, J.; Li, Z.; Wu, F.; Zhou, B.; Wang, Z.; and Ding, Z. 2026 b . OS-Themis: A Scalable Critic Framework for Generalist GUI Rewards. arXiv:2603.19191

    2026

  55. [64]

    Liu, Y.; Iter, D.; Xu, Y.; Wang, S.; Xu, R.; and Zhu, C. 2023. G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment. arXiv:2303.16634

    2023 arXiv

  56. [65]

    Liu, Y.; Su, Z.; Xie, L.; Zhang, Y.; Zong, Q.; Guo, J.; Xie, Z.; Ji, Y.; Yim, Y.; Luo, H.; Ren, X.; Chenyu, R.; Li, H.; and Song, Y. 2026. SkillRevise: Improving LLM-Authored Agent Skills via Trace-Conditioned Skill Revision. arXiv:2606.01139

    2026 arXiv

  57. [66]

    Lu, R.; Li, Y.; and Huo, Y. 2025. Exploring Autonomous Agents: A Closer Look at Why They Fail When Completing Tasks. arXiv:2508.13143

    2025 arXiv

  58. [67]

    H.; Kazemnejad, A.; Meade, N.; Patel, A.; Shin, D.; Zambrano, A.; Stańczak, K.; Shaw, P.; Pal, C

    Lù, X. H.; Kazemnejad, A.; Meade, N.; Patel, A.; Shin, D.; Zambrano, A.; Stańczak, K.; Shaw, P.; Pal, C. J.; and Reddy, S. 2025. AgentRewardBench: Evaluating Automatic Evaluations of Web Agent Trajectories. arXiv:2504.08942

    2025

  59. [68]

    Ma, X.; Xie, X.; Wang, Y.; Wang, J.; Wu, B.; Li, M.; and Wang, Q. 2026. Demystifying the Lifecycle of Failures in Platform-Orchestrated Agentic Workflows. arXiv:2509.23735

    2026

  60. [69]

    A.; Shaw, A

    Merrill, M. A.; Shaw, A. G.; Carlini, N.; Li, B.; Raj, H.; Bercovich, I.; Shi, L.; Shin, J. Y.; Walshe, T.; Buchanan, E. K.; Shen, J.; Ye, G.; Lin, H.; Poulos, J.; Wang, M.; Nezhurina, M.; Jitsev, J.; Lu, D.; Mastromichalakis, O. M.; Xu, Z.; Chen, Z.; Liu, Y.; Zhang, R.; Chen,...

    2026 arXiv

  61. [70]

    OpenAI . 2025. Introducing GPT -5.2. https://openai.com/index/introducing-gpt-5-2/. Accessed: 2026-07-18

    2025

  62. [71]

    W.; Liu, J.; Chen, W.; Chen, Z.; and Lou, Y

    Rahardja, A. W.; Liu, J.; Chen, W.; Chen, Z.; and Lou, Y. 2025. Can Agents Fix Agent Issues? arXiv:2505.20749

    2025

  63. [72]

    Schick, T.; Dwivedi-Yu, J.; Dessì, R.; Raileanu, R.; Lomeli, M.; Zettlemoyer, L.; Cancedda, N.; and Scialom, T. 2023. Toolformer: Language Models Can Teach Themselves to Use Tools. arXiv:2302.04761

    2023 arXiv

  64. [73]

    B.; Morovati, M

    Shah, M. B.; Morovati, M. M.; Rahman, M. M.; and Khomh, F. 2026. Characterizing Faults in Agentic AI: A Taxonomy of Types, Symptoms, and Root Causes. arXiv:2603.06847

    2026 arXiv

  65. [74]

    Shi, W.; Wang, Y.; Zhao, Y.; Chen, Y.; Feng, F.; Hao, X.; Su, X.; Gu, Q.; Su, H.; Cai, X.; and He, X. 2026. AJ-Bench: Benchmarking Agent-as-a-Judge for Environment-Aware Evaluation. arXiv:2604.18240

    2026 arXiv

  66. [75]

    Snell, C.; Lee, J.; Xu, K.; and Kumar, A. 2024. Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters. arXiv:2408.03314

    2024 arXiv

  67. [76]

    Song, K.; Jayarajan, A.; Ding, Y.; Su, Q.; Zhu, Z.; Liu, S.; and Pekhimenko, G. 2025. Aegis: Taxonomy and Optimizations for Overcoming Agent-Environment Failures in LLM Agents. arXiv:2508.19504

    2025 arXiv

  68. [77]

    Y.; Cuadron, A.; Wang, C.; Popa, R

    Tan, S.; Zhuang, S.; Montgomery, K.; Tang, W. Y.; Cuadron, A.; Wang, C.; Popa, R. A.; and Stoica, I. 2025. JudgeBench: A Benchmark for Evaluating LLM-based Judges. arXiv:2410.12784

    2025 arXiv

  69. [78]

    The Gemini Team . 2026. Gemini 3.1 Pro : A Smarter Model for Your Most Complex Tasks. https://blog.google/innovation-and-ai/models-and-research/gemini-models/gemini-3-1-pro/. Accessed: 2026-07-18

    2026

  70. [79]

    Wan, Y.; Fang, T.; Li, Z.; Huo, Y.; Wang, W.; Mi, H.; Yu, D.; and Lyu, M. R. 2026. Inference-Time Scaling of Verification: Self-Evolving Deep Research Agents via Test-Time Rubric-Guided Verification. arXiv:2601.15808

    2026 arXiv

  71. [80]

    Wang, J.; Hu, Y.; Yang, W.; Pan, Z.; Li, X.; and Guo, L.-Z. 2026. Aligning Agents via Planning: A Benchmark for Trajectory-Level Reward Modeling. arXiv:2604.08178

    2026 arXiv

  72. [81]

    V.; Zhou, D.; and Chen, X

    Yang, C.; Wang, X.; Lu, Y.; Liu, H.; Le, Q. V.; Zhou, D.; and Chen, X. 2024. Large Language Models as Optimizers. arXiv:2309.03409

    2024 arXiv

  73. [82]

    Yao, S.; Shinn, N.; Razavi, P.; and Narasimhan, K. 2024. -bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains. arXiv:2406.12045

    2024 arXiv

  74. [83]

    Yao, S.; Zhao, J.; Yu, D.; Du, N.; Shafran, I.; Narasimhan, K.; and Cao, Y. 2023. ReAct: Synergizing Reasoning and Acting in Language Models. arXiv:2210.03629

    2023 arXiv

  75. [84]

    Zhong, S.; Lu, Y.; Ning, J.; Wan, Y.; Feng, L.; Ao, Y.; Ribeiro, L. F. R.; Dreyer, M.; Ammirati, S.; and Xiong, C. 2026. SkillLearnBench: Benchmarking Continual Learning Methods for Agent Skill Generation on Real-World Tasks. arXiv:2604.20087

    2026 arXiv

  76. [85]

    E.; Zhang, C.; Lin, C.; Wang, J.; Zhang, G.; and Zhou, W

    Zhu, K.; Li, H.; Wu, S.; Xing, T.; Ma, D.; Tang, X.; Liu, M.; Yang, J.; Liu, J.; Jiang, Y. E.; Zhang, C.; Lin, C.; Wang, J.; Zhang, G.; and Zhou, W. 2025 a . Scaling Test-time Compute for LLM Agents. arXiv preprint arXiv:2506.12928

    2025 arXiv

  77. [86]

    Zhu, K.; Liu, Z.; Li, B.; Tian, M.; Yang, Y.; Zhang, J.; Han, P.; Xie, Q.; Cui, F.; Zhang, W.; Ma, X.; Yu, X.; Ramesh, G.; Wu, J.; Liu, Z.; Lu, P.; Zou, J.; and You, J. 2025 b . Where LLM Agents Fail and How They can Learn From Failures. arXiv:2509.25370

    2025

  78. [87]

    Zhu, X.; Wu, J.; Zhang, X.; Li, T.; Mu, Y.; Zhai, J.; Shen, C.; Fang, C.; and Liu, Y. 2026. An Empirical Study of Bugs in Modern LLM Agent Frameworks. arXiv:2602.21806

    2026 arXiv

  79. [88]

    Zhuge, M.; Zhao, C.; Ashley, D.; Wang, W.; Khizbullin, D.; Xiong, Y.; Liu, Z.; Chang, E.; Krishnamoorthi, R.; Tian, Y.; Shi, Y.; Chandra, V.; and Schmidhuber, J. 2024. Agent-as-a-Judge: Evaluate Agents with Agents. arXiv:2410.10934

    2024 arXiv

Pith tools

Reviewed August 8, 2026 · model on record in the stance chip above.