Pith. sign in

REVIEW 5 major objections 5 minor 4 cited by

A minimal three-component pipeline of file-based trajectory exploration, consensus mining, and independent validation optimizes agent skills faster and better than complex methods, and extends directly to full harness code.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-12 02:22 UTC pith:D3LD2W5B

load-bearing objection Useful systems simplification with real tables and a practical IDE hook; the “theory-justified minimal pipeline” packaging outruns the ablations. the 5 major comments →

arxiv 2607.03451 v1 pith:D3LD2W5B submitted 2026-07-03 cs.SE cs.AIcs.LG

SkillOpt-Lite: Better and Faster Agent Self-evolution via One Line of Vibe

Yifei Shen , Bo Li , Xinjie Zhang This is my paper
classification cs.SE cs.AIcs.LG
keywords skill optimizationagent self-evolutionzeroth-order optimizationPAC learningharness optimizationfile-system toolsminimal viable pipelinecoding agents
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper asks what is the smallest set of components actually required for reliable skill optimization of autonomous agents. It casts the problem as zeroth-order optimization, notes that agent trajectories supply readable debugging traces rather than blind scalar feedback, and derives three principles from PAC stability and file-centric systems practice: store each rollout as a flat file and explore it with ordinary tools, mine only the attributes that appear across many trajectories, and accept an edit only when an independent validation set improves. Everything else (mini-batch pooling, slow update damping, rejected-edit buffers) is removed. The resulting SkillOpt-Lite converges faster and reaches higher accuracy than the full SkillOpt pipeline on six benchmarks, sometimes allowing a much smaller base model to surpass a larger one that used the heavier method. Because every agent artifact is treated as ordinary editable code, the identical loop becomes HarnessOpt and can rewrite the execution scaffolding itself, again letting a nano-scale model beat a frontier model on SpreadsheetBench. The practical payoff is that a developer can trigger the entire evolution cycle from a single slash command inside a coding IDE.

Core claim

A minimal viable skill-optimization pipeline needs only file-system trajectory exploration, consensus attribute mining across trajectories, and independent validation gating. Once base models are sufficiently capable, the extra machinery of mini-batch reflection pooling, textual learning-rate schedules, and rejection buffers is redundant; stripping it yields both faster convergence and higher final scores than the full SkillOpt design, and the same stripped loop generalizes unchanged to full harness optimization.

What carries the argument

The three-pillar loop: (1) dump every rollout as an independent flat file and let the optimizer navigate it with ordinary list/read/shell tools, (2) mine only the cross-task invariants (consensus attributes) rather than single-trial anomalies, and (3) accept a candidate edit solely when an independent held-out validation set improves. These three operations turn skill (and later harness) optimization into language-mediated program compilation that uses readable traces as debugging feedback.

Load-bearing premise

The claim rests on the premise that, once base models are large enough, ordinary file-system inspection of raw trajectories plus independent validation is already sufficient to replace mini-batch pooling, slow updates, and rejection buffers without harming stability or generalization.

What would settle it

Restore the original mini-batch pooling, slow-update, and rejection-buffer modules, re-run the LiveMath and Spreadsheet comparisons under identical model scales and the paper's original (unadjusted) validation splits, and check whether SkillOpt-Lite still converges faster and finishes higher; if it does not, the redundancy claim fails.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Complex ZO-style controls (mini-batch averaging, edit-budget decay, rejected-edit buffers) can be discarded once models scale without loss of final accuracy.
  • A smaller base model can outperform a larger model on the same task once the execution harness is co-optimized with the skill files.
  • The identical three-pillar loop applies to any agent component that can be stored as editable code, turning skill optimization into full harness optimization.
  • Developers can launch closed-loop skill or harness evolution from a single slash command inside a production coding IDE.
  • Strict statistical independence of the validation set is required; any reuse of training failures re-introduces the stability penalty the PAC bound removes.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If readable trajectories remain the dominant signal, future optimizers may treat the entire workspace (skills, harness scripts, and data loaders) as one compilable program rather than layered modules.
  • The pilot result that a single unvalidated coding-agent batch can already beat multi-epoch SkillOpt suggests raw model scale is already substituting for many classical variance-reduction operators.
  • Closing the same loop over data-collection controllers, as sketched in the paper's future work, would turn foundation-model training itself into an instance of the three-pillar architecture.
  • Once editing power is unconstrained, independent validation capacity—not gradient machinery—becomes the practical bottleneck for open-ended agent self-evolution.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper asks what constitutes a minimal viable pipeline for agent skill optimization. It maps existing skill-optimization heuristics to classical zeroth-order operators (Table 1), argues that rollout trajectories are interpretable debugging traces rather than black-box scalars, and derives three design principles from PAC-style stability and a Claude-Code/file-centric pilot: file-system trajectory exploration, consensus attribute mining, and independent validation gating. SkillOpt-Lite drops mini-batch reflection pooling, epoch-level slow updates, and rejected-edit buffers; on six benchmarks it reports faster convergence and higher scores than full SkillOpt (e.g., LiveMath +8.8 on GPT-5.5, +25.4 on GPT-5.4-nano). The same file-centric loop is extended to HarnessOpt; on SpreadsheetBench, GPT-5.4-nano with HarnessOpt reaches 0.7758, above GPT-5.5 under SkillOpt (0.7620). The pipeline is packaged as a VS Code slash command.

Significance. If the empirical claims hold under controlled conditions, the work is a useful systems contribution to agent engineering: it argues that much of the machinery in recent skill optimizers is redundant once base models are strong, shows a practical IDE-integrated loop, and demonstrates that co-designing harness code with skills can invert model-size rankings on a hard spreadsheet task. Strengths include multi-model multi-benchmark tables, explicit pipeline diagrams (Fig. 3), convergence curves (Fig. 4), open code, and a clear extension path from skills to harnesses. The ZO/PAC framing is mainly organizational rather than a new theorem, but the engineering result—if robust—would matter for practitioners building self-evolving agents.

major comments (5)
  1. [§3.1–3.2, Insights 2–4, Fig. 3, Table 2] The load-bearing claim that mini-batch pooling, slow updates, and rejection buffers are generally redundant (Insights 2–4, §3.1–3.2, Fig. 3) is supported only by full-package SkillOpt-Lite vs full SkillOpt comparisons (Table 2, Fig. 4) plus a single-batch Copilot pilot (Fig. 2). There is no controlled ablation that re-introduces each discarded module under matched batch budgets, the same optimizer agent, and fixed data splits. Without that, the “minimal viable pipeline justified by theory or empirical necessity” framing overstates what the evidence isolates.
  2. [§4.1, Table 2] §4.1 states that LiveMath and OfficeQA were re-split from 2:1:7 to 2:2:6 “to mitigate validation variance,” while SkillOpt numbers are reported under the same table. It is unclear whether the SkillOpt baselines were re-run on the new splits or carried over from the original protocol. If the latter, the large LiveMath gains (e.g., GPT-5.4-nano 30.3→55.7) are not cleanly attributable to the lite pipeline. Please re-run both methods on identical fixed splits and report both original and re-split results.
  3. [Table 2, Table 3, Fig. 4] All main results (Tables 2–3, Figs. 1 and 4) are single-run point estimates with no error bars, multi-seed variance, or confidence intervals. Agent rollouts and coding-agent edits are highly stochastic; without variance, claims such as “nano surpasses GPT-5.4 optimized by SkillOpt” and the SpreadsheetBench inversion (0.7758 vs 0.7620) cannot be assessed for statistical reliability. At minimum, report mean±std over several seeds for the headline LiveMath and Spreadsheet settings.
  4. [§5.1–5.2, Table 3] HarnessOpt is evaluated only on SpreadsheetBench (§5.2, Table 3), after the authors note that other datasets showed little harness headroom. The claim that the minimal skill pipeline “naturally generalizes to full harness optimization” therefore rests on one task family. Either expand HarnessOpt to at least one additional benchmark where harness edits matter, or clearly scope the harness claims to Spreadsheet-style tool/control-flow bottlenecks.
  5. [§2.1–2.2, Table 1] The PAC stability sketch (§2.2) motivates consensus mining and independent validation but does not derive that pooling/slow-update/rejection buffers specifically inflate β_exp or are unnecessary once models scale. The ZO mapping (Table 1) is a useful taxonomy, not a proof of minimality. The paper should either (i) treat the three principles as empirically motivated design choices rather than theory-justified necessity, or (ii) add a short controlled study that links each discarded component to measured stability/generalization under the stated bound.
minor comments (5)
  1. [Abstract, Table 2] Abstract and intro cite LiveMath +8.8 on GPT-5.5 and +25.4 on GPT-5.4-nano relative to SkillOpt; Table 2 shows 64.8→73.6 (+8.8) and 30.3→55.7 (+25.4). Good consistency, but the abstract’s “allowing the nano model to surpass standard GPT-5.4 optimized by SkillOpt” should cite the exact SkillOpt GPT-5.4 LiveMath number (54.0) for the reader.
  2. [Figure 1] Figure 1a radar uses macro averages across model scales without listing which models enter the average; a short caption note would help.
  3. [§3.1, §4.1] §4.1 mentions “identical optimizer models” and GitHub Copilot as the coding agent, but does not specify temperature, tool budget, or whether the same Copilot configuration is used for SkillOpt-Lite edits and the §3.1 pilot. A short reproducibility paragraph would strengthen the code release.
  4. [§4.2, Figure 4] Typo/style: “Toevaluatetheoptimizationefficiency” missing spaces in §4.2; “Best valhard so far” vs “Best val_hard so far” inconsistent notation between text and figures.
  5. [§5.1, References] Related work on harness optimization [6,26–28] is cited but not contrasted in depth with Meta-Harness / Self-Harness; a short paragraph on what is unique to the file-centric three-pillar loop would help positioning.

Circularity Check

0 steps flagged

Empirical systems paper: performance claims rest on held-out benchmarks, not on identities or self-citations that force the result by construction.

full rationale

SkillOpt-Lite’s load-bearing claims are empirical (Table 2, Fig. 1/4, Table 3): accuracy gains vs. Init skill and full SkillOpt, and HarnessOpt nano vs. larger models on SpreadsheetBench. The ZO mapping (Table 1, §2.1) is an analogy that organizes prior agent methods; it does not define the Lite pipeline’s scores. The PAC stability sketch (§2.2) and Insights 2–3 motivate consensus mining and independent validation in the usual way (β_exp reduced by cross-task compression; removed from the selection bound by a disjoint val set)—they do not fit a parameter and rename it a prediction, nor define the method in terms of the reported LiveMath/Spreadsheet numbers. The pilot (§3.1, Fig. 2) is an empirical motivation that sometimes beats 4-epoch SkillOpt and sometimes degrades without gating; it is not used as a uniqueness theorem. SkillOpt [4] is the baseline being ablated/outperformed, not a self-citation that forbids alternatives. No equation equates a claimed “prediction” to a fitted input by construction; no uniqueness theorem is imported from overlapping authors to force the three-pillar design. Correctness concerns (missing component-wise ablations, re-split LiveMath/OfficeQA, proprietary optimizers) are outside circularity. Score 0 with empty steps is the appropriate finding.

Axiom & Free-Parameter Ledger

5 free parameters · 5 axioms · 3 invented entities

Central claims rest on treating skill/harness text as ZO-editable programs, on PAC-style stability justifying consensus mining and independent validation, and on the empirical bitter lesson that file-native coding agents beat engineered topologies. Free parameters are operational (batch size, rounds, smoke N, dead band). Invented entities are the named pipelines and principles; they are engineering constructs with empirical handles, not new physical entities.

free parameters (5)
  • optimization batch size / rounds = e.g. batchsize=40, rounds=10 (skills) / 2 (harness demos)
    Examples use batchsize=40 and 10 batches (or harness rounds=2); these control exploration budget and are chosen by the experimenter, not derived.
  • validation dead band Δ = Δ depending on sample number (unspecified numeric schedule)
    HarnessOpt accepts patches only above a statistical dead band depending on sample size; the exact Δ is a design choice that filters noise vs. progress.
  • smoke test size N = N=5
    Candidate harnesses must pass compile + early smoke test before full validation; N=5 is a hand-set safety parameter.
  • train/val/test re-split for LiveMath and OfficeQA = 2:2:6
    Authors change 2:1:7 to 2:2:6 to reduce validation variance; the new ratio is a free experimental choice that affects reported gains.
  • optimizer token budget for file selection
    Trajectory exploration selects files under a fixed token budget; budget size is an unstated free control on what the miner sees.
axioms (5)
  • domain assumption Skill optimization can be analyzed as zeroth-order optimization of f(s)=E_z[R(H(M,z,s))] with text edits as discrete analogues of ZO operators.
    §2.1 uses this structural analogy to organize prior work; it is not a proven equivalence of rates or estimators in discrete language space.
  • domain assumption Expected on-average stability β_exp bounds generalization of skill libraries, and consensus mining across trajectories reduces β_exp.
    §2.2 invokes PAC/stability bounds from learning theory and interprets them as requiring cross-task invariants rather than single-trace edits.
  • standard math A strictly independent validation set removes β_exp from the model-selection bound and is required for unbiased gating.
    Standard model-selection bound cited via Zhang (2023); applied to skill selection in Insight 3.
  • ad hoc to paper As base LLMs scale, primitive file-system exploration of raw logs outperforms bespoke multi-agent topologies (bitter lesson / Claude Code philosophy).
    Insight 4 and §3.1 pilot with Copilot on flat trajectory files; empirical premise used to drop mini-batch pooling, LR schedules, and rejection buffers.
  • domain assumption All agent components (skills and harness) can be treated as standard editable code files under the same three-pillar loop.
    Used to justify HarnessOpt generalization in §5; necessary for the nano-over-larger SpreadsheetBench claim.
invented entities (3)
  • SkillOpt-Lite pipeline independent evidence
    purpose: Minimal skill optimizer: trajectory staging as files, exploration tools, consensus mining + minimal edit, independent validation gate.
    Named contribution; evaluated empirically against SkillOpt. Independent evidence is the public code and reported benchmarks, not external theory.
  • HarnessOpt pipeline independent evidence
    purpose: Lift of the same loop to editable harness code with Round-0 human gate, allowlists, smoke tests, and reversible toggles.
    Introduced to show co-design of environment scaffolding; primary evidence is SpreadsheetBench table only.
  • Three principles (file-system trajectory exploration, consensus attribute mining, independent validation gating) no independent evidence
    purpose: Claimed necessary and sufficient design pillars for convergence and generalization of skill optimization.
    Synthesized from ZO/PAC framing and pilot; necessity is argued, not exhaustively ablated component-by-component against all alternatives.

pith-pipeline@v1.1.0-grok45 · 19138 in / 3845 out tokens · 38085 ms · 2026-07-12T02:22:58.795177+00:00 · methodology

0 comments
read the original abstract

While skill optimization for autonomous agents has gained traction, existing methods rely on complex pipelines. This leaves a fundamental question unaddressed: What constitutes a minimal viable pipeline for skill optimization, where every component is justified by theory or empirical necessity? We formalize skill optimization via Zeroth-Order (ZO) optimization, mapping classical counterparts (central difference, trust regions) to recent literature. Noting that unlike blind numerical perturbations in classical ZO, skill trajectories serve as interpretable debugging feedback. Grounded in Claude Code philosophy and PAC learning, we establish three principles for convergence and generalization: file-system-based trajectory exploration, consensus attribute mining, and independent validation gating. Eliminating redundancies, we propose SkillOpt-Lite. It accelerates convergence and outperforms full SkillOpt: improving LiveMath by +8.8 points on GPT-5.5 and +25.4 points on GPT-5.4-nano, allowing the nano model to surpass standard GPT-5.4 optimized by SkillOpt. Finally, we integrate our framework into production coding agents like VSCode Copilot, enabling developers to evolve agent skills via one line of vibe. Because our framework treats all agent components simply as standard editable code, this minimal pipeline naturally generalizes to full harness optimization (HarnessOpt). On SpreadsheetBench, HarnessOpt enables GPT-5.4-nano to achieve 0.7758 accuracy, outperforming the larger GPT-5.5 running standard pipelines (0.7620). Code is available at https://github.com/EvolvingLMMs-Lab/SkillOpt-Lite.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 4 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. VeriSkill: A Self-Evolution Framework for Program Verification Skills

    cs.AI 2026-07 conditional novelty 6.0

    VeriSkill turns failed program-verification attempts into validated, reusable skill updates, reporting PASS-rate gains of 17.6 to 51.3 percentage points over no-skill agents across Dafny, Frama-C, and VeriFast.

  2. Training Skills Like Parameters via Self-Supervised Semantic Diffusion

    cs.CL 2026-07 conditional novelty 6.0

    Screenwriting skills can be learned from human scripts alone by compressing and reconstructing them, storing the lessons as text cards that improve generation without changing model weights.

  3. Rethinking Self-Evolution: A Constrained Exploration-Exploitation Process for Mitigating Skill Overfitting

    cs.AI 2026-07 conditional novelty 5.0

    SkillBoost reduces skill overfitting in self-evolving LLM agents by combining failure-localized editing, multi-candidate generation, and an anti-regression acceptance gate.

  4. Mage-VL: An Efficient Codec-Native Streaming Multimodal Foundation Model

    cs.CV 2026-07 conditional novelty 5.0

    Codec-guided sparse patch selection plus a lightweight speak/silent gate yields a 4B streaming VLM that is competitive on static tasks, stronger on video/spatial benchmarks, and much cheaper at inference.

Reference graph

Works this paper leans on

30 extracted references · 14 linked inside Pith · cited by 4 Pith papers

  1. [1]

    Claude code: An agentic command line tool.https://docs.anthropic.com/ en/docs/agents-and-tools/claude-code/overview, 2025

    Anthropic. Claude code: An agentic command line tool.https://docs.anthropic.com/ en/docs/agents-and-tools/claude-code/overview, 2025. Accessed: 2026-06-24

    2025

  2. [2]

    Openclaw: An open-source agentic command-line assistant.https: //github.com/openclaw/openclaw, 2025

    OpenClaw Contributors. Openclaw: An open-source agentic command-line assistant.https: //github.com/openclaw/openclaw, 2025. GitHub Repository

    2025

  3. [3]

    Hermes agent

    Nous Research. Hermes agent. https://nousresearch.com/hermes3/, 2026. Advanced Agentic and Function-Calling Foundation Models

    2026

  4. [4]

    Skillopt: Executive strategy for self-evolving agent skills

    Yifan Yang, Ziyang Gong, Weiquan Huang, Qihao Yang, Ziwei Zhou, Zisu Huang, Yan Li, Xuemei Gao, Qi Dai, Bei Liu, et al. Skillopt: Executive strategy for self-evolving agent skills. arXiv preprint arXiv:2605.23904, 2026

    Pith/arXiv arXiv 2026

  5. [5]

    Swe-agent: Agent-computer interfaces enable automated software engineering.Advances in Neural Information Processing Systems, 37:50528–50652, 2024

    John Yang, Carlos Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press. Swe-agent: Agent-computer interfaces enable automated software engineering.Advances in Neural Information Processing Systems, 37:50528–50652, 2024

    2024

  6. [6]

    Meta-harness: End-to-end optimization of model harnesses.arXiv preprint arXiv:2603.28052, 2026

    Yoonho Lee, Roshen Nair, Qizheng Zhang, Kangwook Lee, Omar Khattab, and Chelsea Finn. Meta-harness: End-to-end optimization of model harnesses.arXiv preprint arXiv:2603.28052, 2026. 14

    Pith/arXiv arXiv 2026

  7. [7]

    Agentic harness engineering: Observability-driven automatic evolution of coding-agent harnesses.arXiv preprint arXiv:2604.25850, 2026

    Jiahang Lin, Shichun Liu, Chengjun Pan, Lizhi Lin, Shihan Dou, Zhiheng Xi, Xuanjing Huang, Hang Yan, Zhenhua Han, Tao Gui, et al. Agentic harness engineering: Observability-driven automatic evolution of coding-agent harnesses.arXiv preprint arXiv:2604.25850, 2026

    Pith/arXiv arXiv 2026

  8. [8]

    Trace2skill: Distill trajectory-local lessons into transferable agent skills.arXiv preprint arXiv:2603.25158, 2026

    Jingwei Ni, Yihao Liu, Xinpeng Liu, Yutao Sun, Mengyu Zhou, Pengyu Cheng, Dexin Wang, Erchao Zhao, Xiaoxi Jiang, and Guanjun Jiang. Trace2skill: Distill trajectory-local lessons into transferable agent skills.arXiv preprint arXiv:2603.25158, 2026

    Pith/arXiv arXiv 2026

  9. [9]

    Re- flexion: Language agents with verbal reinforcement learning.Advances in neural information processing systems, 36:8634–8652, 2023

    Noah Shinn, Federico Cassano, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Re- flexion: Language agents with verbal reinforcement learning.Advances in neural information processing systems, 36:8634–8652, 2023

    2023

  10. [10]

    Voyager: An open-ended embodied agent with large language models.arXiv preprint arXiv:2305.16291, 2023

    Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and Anima Anandkumar. Voyager: An open-ended embodied agent with large language models.arXiv preprint arXiv:2305.16291, 2023

    Pith/arXiv arXiv 2023

  11. [11]

    Skilladaptor: Self-adapting skills for llm agents from trajectories.arXiv preprint arXiv:2606.01311, 2026

    Zhuoyun Yu, Xin Xie, Wuguannan Yao, Chenxi Wang, Lei Liang, Xiang Qi, and Shumin Deng. Skilladaptor: Self-adapting skills for llm agents from trajectories.arXiv preprint arXiv:2606.01311, 2026

    Pith/arXiv arXiv 2026

  12. [12]

    Skillforge: Forging domain-specific, self-evolving agent skills in cloud technical support.arXiv preprint arXiv:2604.08618, 2026

    Xingyan Liu, Xiyue Luo, Linyu Li, Ganghong Huang, Jianfeng Liu, and Honglin Qiao. Skillforge: Forging domain-specific, self-evolving agent skills in cloud technical support.arXiv preprint arXiv:2604.08618, 2026

    Pith/arXiv arXiv 2026

  13. [13]

    Skillcat: Contrastive assessment and topology-aware skill self-evolution for llm agents.arXiv preprint arXiv:2606.13317, 2026

    Kunfeng Chen, Qihuang Zhong, Juhua Liu, and Bo Du. Skillcat: Contrastive assessment and topology-aware skill self-evolution for llm agents.arXiv preprint arXiv:2606.13317, 2026

    Pith/arXiv arXiv 2026

  14. [14]

    Claude Code Source Code

    Anthropic. Claude Code Source Code. https://github.com/codeaashu/claude-code, 2026

    2026

  15. [15]

    A primer on zeroth-order optimization in signal processing and machine learning: Principals, recent advances, and applications.IEEE Signal Processing Magazine, 37(5):43–54, 2020

    Sijia Liu, Pin-Yu Chen, Bhavya Kailkhura, Gaoyuan Zhang, Alfred O Hero III, and Pramod K Varshney. A primer on zeroth-order optimization in signal processing and machine learning: Principals, recent advances, and applications.IEEE Signal Processing Magazine, 37(5):43–54, 2020

    2020

  16. [16]

    Softskill: Behavioral compression for contextual adaptation.arXiv preprint arXiv:2606.20333, 2026

    Xijia Tao, Yihua Teng, Xinyu Fu, Ziru Liu, Kecheng Chen, Yuzhi Zhao, Suiyun Zhang, Rui Liu, and Lingpeng Kong. Softskill: Behavioral compression for contextual adaptation.arXiv preprint arXiv:2606.20333, 2026

    Pith/arXiv arXiv 2026

  17. [17]

    Learnability and stability in the vapnik-chervonenkis sense.The Annals of Statistics, 38(5):2642–2696, 2010

    Shai Shalev-Shwartz, Ohad Shamir, Nathan Srebro, and Karthik Sridharan. Learnability and stability in the vapnik-chervonenkis sense.The Annals of Statistics, 38(5):2642–2696, 2010

    2010

  18. [18]

    Cambridge University Press, 2023

    Tong Zhang.Mathematical Analysis of Machine Learning Algorithms. Cambridge University Press, 2023. doi: 10.1017/9781009093057

    doi:10.1017/9781009093057 2023

  19. [19]

    GitHub Copilot.https://github.com/features/copilot, 2021

    GitHub. GitHub Copilot.https://github.com/features/copilot, 2021. Accessed: 2026

    2021

  20. [20]

    Spreadsheetbench: Towards challenging real world spreadsheet manipulation.Advances in Neural Information Processing Systems, 37:94871–94908, 2024

    Zeyao Ma, Bohan Zhang, Jing Zhang, Jifan Yu, Xiaokang Zhang, Xiaohan Zhang, Sijia Luo, Xi Wang, and Jie Tang. Spreadsheetbench: Towards challenging real world spreadsheet manipulation.Advances in Neural Information Processing Systems, 37:94871–94908, 2024

    2024

  21. [21]

    Alfworld: Aligning text and embodied environments for interactive learning.arXiv preprint arXiv:2010.03768, 2020

    Mohit Shridhar, Xingdi Yuan, Marc-Alexandre Côté, Yonatan Bisk, Adam Trischler, and Matthew Hausknecht. Alfworld: Aligning text and embodied environments for interactive learning.arXiv preprint arXiv:2010.03768, 2020. 15

    Pith/arXiv arXiv 2010

  22. [22]

    Livemathematicianbench: A live benchmark for mathematician-level reasoning with proof sketches.arXiv preprint arXiv:2604.01754, 2026

    Linyang He, Qiyao Yu, Hanze Dong, Baohao Liao, Xinxing Xu, Micah Goldblum, Jiang Bian, and Nima Mesgarani. Livemathematicianbench: A live benchmark for mathematician-level reasoning with proof sketches.arXiv preprint arXiv:2604.01754, 2026

    arXiv 2026

  23. [23]

    Docvqa: A dataset for vqa on document images

    Minesh Mathew, Dimosthenis Karatzas, and CV Jawahar. Docvqa: A dataset for vqa on document images. InProceedings of the IEEE/CVF winter conference on applications of computer vision, pages 2200–2209, 2021

    2021

  24. [24]

    Searchqa: A new q&a dataset augmented with context from a search engine.arXiv preprint arXiv:1704.05179, 2017

    Matthew Dunn, Levent Sagun, Mike Higgins, V Ugur Guney, Volkan Cirik, and Kyunghyun Cho. Searchqa: A new q&a dataset augmented with context from a search engine.arXiv preprint arXiv:1704.05179, 2017

    Pith/arXiv arXiv 2017

  25. [25]

    Officeqa pro: An enterprise benchmark for end-to-end grounded reasoning.arXiv preprint arXiv:2603.08655, 2026

    Krista Opsahl-Ong, Arnav Singhvi, Jasmine Collins, Ivan Zhou, Cindy Wang, Ashutosh Baheti, Owen Oertell, Jacob Portes, Sam Havens, Erich Elsen, et al. Officeqa pro: An enterprise benchmark for end-to-end grounded reasoning.arXiv preprint arXiv:2603.08655, 2026

    arXiv 2026

  26. [26]

    Code as agent harness.arXiv preprint arXiv:2605.18747, 2026

    Xuying Ning, Katherine Tieu, Dongqi Fu, Tianxin Wei, Zihao Li, Yuanchen Bei, Jiaru Zou, Mengting Ai, Zhining Liu, Ting-Wei Li, et al. Code as agent harness.arXiv preprint arXiv:2605.18747, 2026

    Pith/arXiv arXiv 2026

  27. [27]

    Self-harness: Harnesses that improve themselves.arXiv preprint arXiv:2606.09498, 2026

    Hangfan Zhang, Shao Zhang, Kangcong Li, Chen Zhang, Yang Chen, Yiqun Zhang, Lei Bai, and Shuyue Hu. Self-harness: Harnesses that improve themselves.arXiv preprint arXiv:2606.09498, 2026

    Pith/arXiv arXiv 2026

  28. [28]

    Harness updating is not harness benefit: Disentangling evolution capabilities in self-evolving llm agents

    Harness Self-Evolution. Harness updating is not harness benefit: Disentangling evolution capabilities in self-evolving llm agents. 2026

    2026

  29. [29]

    Learning beyond gradients

    Jiayi Weng. Learning beyond gradients. https://trinkle23897.github.io/ learning-beyond-gradients/, May 2026. Accessed: 2026-06-30

    2026

  30. [30]

    Llava-onevision-2: Towards next-generation perceptual intelligence.arXiv preprint arXiv:2605.25979, 2026

    Xiang An, Yin Xie, Feilong Tang, Yunyao Yan, Huajie Tan, Didi Zhu, Changrui Chen, Xiuwei Zhao, Bin Qin, Kaicheng Yang, et al. Llava-onevision-2: Towards next-generation perceptual intelligence.arXiv preprint arXiv:2605.25979, 2026. 16

    Pith/arXiv arXiv 2026