Pith. sign in

REVIEW 4 major objections 6 minor 1 cited by

DocOps claims that even the strongest AI agent configurations fail about a third of realistic document-editing tasks, with near-collapse on long, cross-file workflows.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 11:28 UTC pith:SMZW4L6H

load-bearing objection A useful, well-engineered benchmark for document-agent reliability, with a verifier-fidelity audit that is thinner than the central claim wants—still worth publishing after a round of revisions. the 4 major comments →

arxiv 2607.19865 v1 pith:SMZW4L6H submitted 2026-07-22 cs.AI cs.CLcs.LG

DocOps: A Verifiable Benchmark for Autonomous Agents in Complex Document Operations

Jiazhen Jiang , Boxi Cao , Lingyong Yan , Yaojie Lu , Hongyu Lin , Shuaiqiang Wang , Dawei Yin , Xianpei Han
show 1 more author
Le Sun
This is my paper
classification cs.AI cs.CLcs.LG
keywords document manipulationLLM agentsbenchmarkdeterministic verificationnative file formatsstate trackingworkflow difficultyoffice documents
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper argues that reliable document manipulation—editing real Word, Excel, PowerPoint, and PDF files while keeping hidden state intact—is a distinct capability that existing benchmarks do not measure, and that current agents are far from mastering it. To make the point, it introduces DocOps, a benchmark of 210 tasks organized along two axes: atomic operations (content, format, structure) and workflow depth (from a single edit to cross-document projects), each paired with a deterministic verifier that inspects the final native file. The empirical centerpiece is that the strongest tested configuration—a frontier model in an open-ended coding harness with document skills—passes only 67.1% of tasks, and that workflow-level tasks (L3 and L4) see pass rates fall to roughly 24% or lower. The paper attributes most failures to three modes: losing track of global document state, accepting surface-plausible output without checking underlying semantics, and destructively flattening native structure. If right, this redirects agent research toward state-aware, non-destructive editing rather than better tool invocation.

Core claim

DocOps's central claim is that agent performance on document tasks collapses when tasks require maintaining document state across coupled steps, and that this is a systematic weakness of current systems, not an artifact of any single model or harness. The paper demonstrates this with verifiers that read submitted files through native libraries and check three types of predicates: structural predicates for hidden native state (executable formulas, heading hierarchies, real table objects), linguistic anchors for requested content, and preservation predicates for out-of-scope elements that must remain untouched. On this standard, the best configuration achieves an overall pass rate of 0.671, dr

What carries the argument

The deterministic verifier is the load-bearing mechanism: a per-task program that inspects the final artifact through document-native libraries, using structural predicates (e.g., a cell must contain an executable formula that recomputes the expected total from the workbook's own code map), linguistic anchors (diagnostic keywords that allow paraphrasing), and preservation predicates (out-of-scope sheets, styles, bookmarks, or page order must be unchanged). This artifact-level check is what separates a genuinely correct edit from a visually plausible one, and it grounds the taxonomy's categories in measurable failure modes.

Load-bearing premise

The load-bearing premise is that the 210 tasks, constructed by one main reviewer from filtered community and workflow seeds, and their hand-written verifier predicates capture what 'correct, globally consistent, non-destructive document editing' means outside the benchmark—if the predicates miss a hidden class of state, the reported pass rates and failure-mode proportions are shifted.

What would settle it

Run the full 210-task suite with expert human audit of every verifier decision, watching specifically for uncaught state corruption such as altered page-setup dimensions, changed document metadata, or broken cross-references that the current predicates do not inspect; if the false-pass rate exceeds the reported 3/128 (≈2.3%) or includes such hidden-state corruptions, the headline pass rates overstate agent reliability.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Agents running in open-ended programming harnesses with file-system feedback substantially outperform those restricted to fixed document-tool interfaces, so harness design is a first-order determinant of document-operation reliability.
  • Pass rates degrade as task coupling increases, not merely as operation count increases: Excel workflows collapse while PDF workflows remain comparatively stable, suggesting that state-coupling is the key difficulty axis.
  • Explicit document skills improve some mid-tier open models but can hurt or have no effect on frontier models, so skill injection is an empirical design choice, not a guaranteed improvement.
  • Any evaluation that checks only task-level success or rendered output will miss the dominant failure modes; artifact-level verification of structural validity and preservation is necessary to measure true reliability.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The verifier-fidelity audit itself reports three false passes (e.g., a slide-reorder task that also changed canvas dimensions went undetected), so extending the predicates to cover peripheral document properties would likely push the reported pass rates down slightly.
  • If state-tracking collapse is the general bottleneck, the same two-axis taxonomy could transfer to other structured artifacts—codebases, CAD files, or HTML/CSS layouts—where a correct edit also means preserving hidden dependent state.
  • The near-zero Excel L3 results single out formula-reference and validation-boundary preservation as a concrete stress test for any agent deployed in finance or accounting; an agent that cannot pass that subset should not be trusted with real spreadsheets.
  • Because DocOps tasks are compact and hand-built, real-world files are likely messier and harder; the benchmark's numbers are plausibly an upper bound on current agent reliability in production document workflows.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper introduces DocOps, a benchmark of 210 document-manipulation tasks (XLSX, DOCX, PPTX, PDF) organized along an operation taxonomy (content/format/structure) and a four-level difficulty gradient (L1 atomic, L2 composite, L3 workflow, L4 cross-document). Each task ships with a deterministic verifier that inspects the final native artifact via structural predicates, linguistic anchors, and preservation predicates. The authors evaluate 12 models across four harnesses (DocTools, Terminus-2, Codex, Claude Code) with and without document skills. Main findings: the best configuration (GPT-5.5 + Codex + skills) reaches only 0.671 overall pass rate; performance collapses on L3/L4, especially for Excel; three failure modes (long-term state tracking, semantic verification gap, destructive editing) account for a majority of failures; harness choice strongly modulates performance; and skills help mid-tier open-source models more than frontier models. Verifier fidelity is reported as 95.31% agreement with a human audit of 128 decisions and 96.67% detection on 180 injected mutations.

Significance. If the results hold, DocOps is a valuable and well-scoped contribution. Its strengths are concrete: a released dataset and codebase, deterministic per-task verifiers over native file formats, an openly described construction pipeline, an honest limitations section, and a fidelity audit with both human judgments and mutation-based stress testing. The empirical pattern—frontier agents failing roughly one-third of tasks and degrading sharply on coupled, long-horizon workflows—is plausible and likely to be influential for the agent-evaluation community. The resource should be genuinely reusable. However, the strength of the quantitative claims currently exceeds the strength of the supporting verifier-fidelity evidence: several load-bearing, fixable gaps in the audit design and uncertainty reporting should be addressed before the benchmark's numbers are taken as definitive.

major comments (4)
  1. [Appendix F / §3.3] The verifier-fidelity audit is under-powered for the load it carries in §4.2.1. Agreement of 95.31% is computed over 128 decisions with no pass-conditional false-pass rate, no difficulty-stratified agreement, and no confidence interval; the 3/128 false-pass rate alone has a Wilson upper bound near 7%. The one representative false pass described in Appendix F altered the presentation's global canvas dimensions—a persistent, out-of-scope state property—yet the verifier was not designed to inspect it. Since deterministic preservation-aware verification is the benchmark's core novelty, an unmeasured class of omitted state dimensions directly affects the headline 0.671 and the L3/L4 drop magnitudes in Figure 2. False passes inflate reported pass rates; false fails (3/128) deflate them, so the net bias is unknown. The mutation stress test (96.67%) cannot detect state dimensions the verifier ne
  2. [Table 2 / §4.2.1] Table 2 reports the central results—the headline 0.671 and all cross-harness/cross-model comparisons—without uncertainty quantification. With 210 binary trials, the sampling standard error for a 0.67 rate is about 3.2 percentage points, so differences such as GPT-5.5 Codex w/skill (0.671) vs w/o skill (0.648) and vs GPT-5.4 (0.662) are within noise. The claim in §4.2.1 and the abstract that skills offer only 'marginal utility for frontier models' rests on a +0.024 delta (Table 5) with no confidence interval; the Figure 7 bootstrap procedure is applied only to open-source models. Please add bootstrap CIs (or equivalent) to Table 2 or the headline text, and run the paired-bootstrap analysis for the frontier-model skill comparisons as well.
  3. [§4.2.3 / Figure 5] The three failure-mode definitions overlap. 'Semantic verification gap' includes violations of 'required formulas, value ranges, hierarchy, or structural conditions,' while 'destructive editing' includes 'formulas, validation rules, styles, heading hierarchy, native tables, bookmarks.' A single failed run can trigger assertions in multiple categories, yet Figure 5 reports a single distribution with no stated rule for assigning a run to one mode. In addition, the mode labels impute causes (the agent 'accepts' plausible outputs without verification) from verifier-assertion outcomes; the quantified distribution is based on 'verifier-grounded signals,' not on the trajectory evidence used in Figure 3. Please state the assignment rule (e.g., priority or primary-trigger) and either restrict causal wording to the trajectory case studies or add trajectory-based coding for the quantified distribut
  4. [§3.2 Stage 4 / Appendix F] Both the task-construction review (Stage 4) and the verifier-fidelity audit rely on a single human judge: one PhD reviewer for the 210-task pipeline and one PhD candidate for the 128-decision audit. No inter-rater reliability is reported for either. Since the benchmark's construct—'correct, globally consistent, non-destructive document editing'—is ultimately defined by human judgment, a second independent rater on a stratified sample is needed to bound systematic bias in task selection and in the gold labels used to measure the verifier. This matters because the audit's three false passes and three false fails are dismissed as 'peripheral' without a systematic argument that such cases cannot concentrate in L3/L4.
minor comments (6)
  1. [Appendix F] Report the full 2×2 contingency table (manual pass/fail × verifier pass/fail) and the sampling stratification over formats and difficulty levels for the 128 decisions; currently only aggregate agreement is given.
  2. [Figure 5] The legend order and the percentages do not make the mapping explicit (e.g., 'Other failures' is 1.13% in panel (a) but 11.49% in panel (c)). Label categories directly on the bars or align the legend order with the segments.
  3. [Author block / Figure 8 / Appendix C] Typos: 'ὑ7' appears before the project URL in the author block; 'T able/Sheet operations' in Figure 8(a); 'We report s/task' in Appendix C should be 'seconds/task'.
  4. [Table 8 / Figure 2] Within-level heterogeneity is large: L3_v2_011 and L3_v2_016 list all 13 operation labels, while L3_v2_012-013 list two. The L3/L4 aggregates in Figure 2 thus mix tasks of very different width and composition; consider reporting the L3_v2 sub-family separately or controlling for operation count in the coupling analysis of §4.2.2.
  5. [§4.2.2] The FormatCV bootstrap confidence intervals reflect sampling variability only, not verifier disagreement or benchmark-construction variability. State this caveat when interpreting the non-overlapping intervals between L1 and L3.
  6. [Footnotes] The abstract carries a footnote marker ('ecosystems.1') but the footnote text appears at the start of the Introduction; move the footnote to the title/authors or the abstract itself.

Circularity Check

1 steps flagged

Central pass-rate claims are self-contained external measurements; only the failure-mode distribution is mildly design-inherited from verifier predicate categories.

specific steps
  1. self definitional [Section 4.2.3 'Key Failure Modes' and Figure 5; definitions and the method sentence 'we extract verifier-grounded signals from failed runs and group them by failure mode.']
    "We define this failure mode as cases where the violated assertions concern persistent document state, such as page, slide, or sheet order, preserved objects, hidden sheets, or cross-document mappings. ... we extract verifier-grounded signals from failed runs and group them by failure mode. Figure 5 shows that semantic verification gaps dominate across representative models and overall settings."

    The failure-mode distribution is computed by grouping the authors' own verifier assertion categories, so each mode's prevalence is inherited by construction from which predicates the verifier suite includes. A failure type with no predicate cannot register at all: Appendix F's acknowledged false pass (slide order correct but global canvas dimensions altered, 'not covered by the verifier') is exactly a state-tracking violation the taxonomy cannot count. The claim that state-tracking/semantic/destructive failures 'account for a majority of failed runs' therefore largely restates the predicate mix the authors designed rather than independently measuring agent behavior. This is secondary: the headline pass rate 0.671 and the L3/L4 collapse remain external, verifier-grounded measurements, not f

full rationale

DocOps's central empirical claims—pass rates of real models under different harnesses (Table 2), difficulty-level gradients (Figure 2), format-dependent degradation (Table 4), and skill-injection deltas (Figure 7)—are external measurements against hand-written, deterministic verifier predicates. No parameter is fitted to benchmark labels, no 'prediction' is derived from benchmark outcomes, and the pass/fail signal comes from inspecting final artifacts, not from the agent trajectories that produced them. The verifier predicates are authored before any agent run, so the headline results are self-contained empirical findings. The Appendix F fidelity audit (128 decisions, 95.31% agreement, 3 false passes) is limited in power, and one acknowledged false pass altered a property ('global canvas dimensions') that is the very kind of global state the benchmark claims to measure; this is a real validity threat to the headline 0.671, but it is a correctness/coverage concern, not circularity. The only mild circular element is the failure-mode analysis (Section 4.2.3, Figure 5): each mode is defined as 'cases where the violated assertions concern [category X]', so the reported proportions (~40% semantic, ~30% state-tracking, ~25% destructive) are by construction a projection of the predicate categories the authors chose to write. This affects the interpretive 'which failure mode dominates' claim, but the paper's primary contribution—the deterministic verifier and the measured agent pass rates—does not reduce to its own inputs. No load-bearing self-citation, unique-theorem import, or ansatz-smuggling is present. Score 2.

Axiom & Free-Parameter Ledger

4 free parameters · 5 axioms · 0 invented entities

The benchmark's conclusions depend on design choices and assumptions about verifier fidelity, task representativeness, and comparable serving; no fitted parameters are used to produce the main results. The main free parameters are the hand-chosen task distribution and verifier predicates, which affect pass rates but not the qualitative direction of the findings.

free parameters (4)
  • Difficulty distribution (L1/L2/L3/L4 task counts) = 50/40/60/60
    Chosen by hand. Aggregate pass rates and the L1-to-L4 degradation claim depend on this mix.
  • Harness hyperparameters (temperature, turn limits) = temp 0.0/0.2; 200-turn limit; fixed small step budget for DocTools
    Chosen by hand. Harness comparisons could shift with different budgets or temperatures.
  • Verifier-specific thresholds and expected values = task-specific
    Hand-written per task. Pass/fail outcomes depend on choices such as formula prefixes, outline equality, and exact header sequences.
  • Seed filtering rate = 118/600 excluded (19.7%)
    Task-pool composition and benchmark scope are determined by this exclusion rule.
axioms (5)
  • domain assumption Deterministic verifier predicates correspond to task success and preservation of out-of-scope state.
    Section 3.3; fidelity audit shows 95.31% agreement and 96.67% mutation detection, so this is approximate rather than exact.
  • domain assumption Synthetic compact source artifacts and filtered seeds are representative of real-world document operations.
    Section 3.2; tasks are deliberately synthetic and filtered, so external validity is assumed.
  • domain assumption A single PhD-level human reviewer provides sufficient quality control for instructions and artifacts.
    Section 3.2 Stage 4; no inter-rater reliability is reported.
  • domain assumption Provider API and vLLM serving configurations produce comparable agent behavior across models.
    Appendix H; closed models ran on provider APIs, open models on vLLM with different parsers/templates. Serving differences could affect comparative results.
  • domain assumption Harbor containerization faithfully instantiates each harness without distorting behavior.
    Appendix I; the assumption is that container images and preinstalled tooling do not alter agent behavior.

pith-pipeline@v1.3.0-alltime-deepseek · 26548 in / 13358 out tokens · 143516 ms · 2026-08-01T11:28:01.242290+00:00 · methodology

0 comments
read the original abstract

As autonomous agents rapidly evolve, their ability to reliably manipulate ubiquitous digital documents has become critical for enabling general-purpose AI assistants and automating complex workspace workflows. In this paper, we introduce DocOps, a deterministically verifiable evaluation framework underpinned by a hierarchical taxonomy that deconstructs document operations inspired by real-world practices into atomic dimensions and escalating workflow complexities. Based on DocOps, we systematically evaluate representative closed- and open-source models across various agentic harnesses, revealing that even the most advanced frontier configurations still exhibit profound limitations when handling highly coupled, long-range tasks. Furthermore, a fine-grained analysis of existing agents' manipulation behaviors uncovers 3 key failure modes: long-term state tracking collapse, shallow semantic verification, and destructive editing of structural metadata. Ultimately, our work exposes the capability boundaries of agents in maintaining global document consistency, shedding light on the future design of robust, non-destructive agents for complex digital ecosystems.

Figures

Figures reproduced from arXiv: 2607.19865 by Boxi Cao, Dawei Yin, Hongyu Lin, Jiazhen Jiang, Le Sun, Lingyong Yan, Shuaiqiang Wang, Xianpei Han, Yaojie Lu.

Figure 1
Figure 1. Figure 1: The overview framework of DocOps. lation. As illustrated in [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Pass rates across difficulty levels for represen [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Representative trajectory-level failure cases showing typical agent behaviors under three failure modes: (a) long-term state tracking failure. (b) semantic verification gap. (c) destructive editing. Extraction Editing Generation Computation Reasoning Style consistency Highlighting Layout control Theme transfer Insert/Delete Reorder Hierarchy editing Table/Sheet ops GPT-5.4 Claude Sonnet 4.6 Qwen3.5 122B-A1… view at source ↗
Figure 5
Figure 5. Figure 5: Distribution of different failure modes across [PITH_FULL_IMAGE:figures/full_fig_p008_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Harness-level pass-rate ranges over open [PITH_FULL_IMAGE:figures/full_fig_p009_6.png] view at source ↗
Figure 8
Figure 8. Figure 8: DocOps benchmark statistics. tory. Unlike runtime, token accounting may differ across harnesses because different runtimes expose different usage metadata. Specifically: • DocTools. We use the agent-reported usage fields when available. For trajectories where the LLM￾call records preserve the visible interaction but do not expose reliable token-usage metadata, we es￾timate token usage from the visible LLM-… view at source ↗
Figure 9
Figure 9. Figure 9: Prompt template used to formalize informal document-editing seeds into the metadata schema used by [PITH_FULL_IMAGE:figures/full_fig_p014_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Atomic operation accuracy by model on L1 tasks for all evaluated models. Scores are averaged over the [PITH_FULL_IMAGE:figures/full_fig_p014_10.png] view at source ↗
Figure 11
Figure 11. Figure 11: Representative Excel verifier snippet. The [PITH_FULL_IMAGE:figures/full_fig_p016_11.png] view at source ↗
Figure 14
Figure 14. Figure 14: Representative PDF verifier snippet. The ver [PITH_FULL_IMAGE:figures/full_fig_p016_14.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. OmegaUse-OfficeVal: Benchmarking LLM Agents on Long-Horizon Office-Suite Tasks with Economic Grounding

    cs.AI 2026-07 conditional novelty 6.0

    On 100 long-horizon office-suite tasks with task-level economic labels and code verifiers, frontier LLMs are cheaper and faster than humans but lag substantially in deliverable quality.

Reference graph

Works this paper leans on

57 extracted references · 3 canonical work pages · cited by 1 Pith paper

  1. [1]

    , title =

    Mathew, Minesh and Karatzas, Dimosthenis and Jawahar, C.V. , title =. Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV) , month =. 2021 , pages =

    2021

  2. [2]

    2026 , eprint=

    Finch: Benchmarking Finance & Accounting across Spreadsheet-Centric Enterprise Workflows , author=. 2026 , eprint=

    2026

  3. [3]

    , title =

    Mathew, Minesh and Bagal, Viraj and Tito, Rub\`en and Karatzas, Dimosthenis and Valveny, Ernest and Jawahar, C.V. , title =. Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV) , month =. 2022 , pages =

    2022

  4. [4]

    C hart QA : A Benchmark for Question Answering about Charts with Visual and Logical Reasoning

    Masry, Ahmed and Long, Do Xuan and Tan, Jia Qing and Joty, Shafiq and Hoque, Enamul. C hart QA : A Benchmark for Question Answering about Charts with Visual and Logical Reasoning. Findings of the Association for Computational Linguistics: ACL 2022. 2022. doi:10.18653/v1/2022.findings-acl.177

    doi:10.18653/v1/2022.findings-acl.177 2022

  5. [5]

    2026 , howpublished =

    Terminus-2: Harbor's Reference Agent Implementation , author =. 2026 , howpublished =

    2026

  6. [6]

    2026 , howpublished =

    Introducing GPT-5.4 , author =. 2026 , howpublished =

    2026

  7. [7]

    2026 , howpublished =

    Introducing Claude Sonnet 4.6 , author =. 2026 , howpublished =

    2026

  8. [8]

    2026 , howpublished =

    Introducing GPT-5.5 , author =. 2026 , howpublished =

    2026

  9. [9]

    2024 , eprint=

    SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering , author=. 2024 , eprint=

    2024

  10. [10]

    MMLONGBENCH-DOC: Benchmarking Long-context Document Understanding with Visualizations , url =

    Ma, Yubo and Zang, Yuhang and Chen, Liangyu and Chen, Meiqi and Jiao, Yizhu and Li, Xinze and Lu, Xinyuan and Liu, Ziyu and Ma, Yan and Dong, Xiaoyi and Zhang, Pan and Pan, Liangming and Jiang, Yu-Gang and Wang, Jiaqi and Cao, Yixin and Sun, Aixin , booktitle =. MMLONGBENCH-DOC: Benchmarking Long-context Document Understanding with Visualizations , url =....

    work page doi:10.52202/079017-3041

  11. [11]

    2024 , eprint=

    DOCBENCH: A Benchmark for Evaluating LLM-based Document Reading Systems , author=. 2024 , eprint=

    2024

  12. [12]

    Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , month =

    Ouyang, Linke and Qu, Yuan and Zhou, Hongbin and Zhu, Jiawei and Zhang, Rui and Lin, Qunshu and Wang, Bin and Zhao, Zhiyuan and Jiang, Man and Zhao, Xiaomeng and Shi, Jin and Wu, Fan and Chu, Pei and Liu, Minghao and Li, Zhenxiang and Xu, Chao and Zhang, Bo and Shi, Botian and Tu, Zhongying and He, Conghui , title =. Proceedings of the IEEE/CVF Conference...

    2025

  13. [13]

    SheetCopilot: Bringing Software Productivity to the Next Level through Large Language Models , url =

    Li, Hongxin and Su, Jingran and Chen, Yuntao and Li, Qing and ZHANG, ZHAO-XIANG , booktitle =. SheetCopilot: Bringing Software Productivity to the Next Level through Large Language Models , url =

  14. [14]

    I nstruct E xcel: A Benchmark for Natural Language Instruction in Excel

    Payan, Justin and Mishra, Swaroop and Singh, Mukul and Negreanu, Carina and Poelitz, Christian and Baral, Chitta and Roy, Subhro and Chakravarthy, Rasika and Van Durme, Benjamin and Nouri, Elnaz. I nstruct E xcel: A Benchmark for Natural Language Instruction in Excel. Findings of the Association for Computational Linguistics: EMNLP 2023. 2023. doi:10.1865...

    work page doi:10.18653/v1/2023.findings-emnlp.265 2023

  15. [15]

    SpreadsheetBench: Towards Challenging Real World Spreadsheet Manipulation , url =

    Ma, Zeyao and Zhang, Bohan and Zhang, Jing and Yu, Jifan and Zhang, Xiaokang and Zhang, Xiaohan and Luo, Sijia and Wang, Xi and Tang, Jie , booktitle =. SpreadsheetBench: Towards Challenging Real World Spreadsheet Manipulation , url =. doi:10.52202/079017-3007 , editor =

    doi:10.52202/079017-3007

  16. [16]

    PPTA gent: Generating and Evaluating Presentations Beyond Text-to-Slides

    Zheng, Hao and Guan, Xinyan and Kong, Hao and Zhang, Wenkai and Zheng, Jia and Zhou, Weixiang and Lin, Hongyu and Lu, Yaojie and Han, Xianpei and Sun, Le. PPTA gent: Generating and Evaluating Presentations Beyond Text-to-Slides. Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing. 2025. doi:10.18653/v1/2025.emnlp-main.728

    doi:10.18653/v1/2025.emnlp-main.728 2025

  17. [17]

    2026 , eprint=

    Talk to Your Slides: High-Efficiency Slide Editing via Language-Driven Structured Data Manipulation , author=. 2026 , eprint=

    2026

  18. [18]

    2025 , eprint=

    PPTArena: A Benchmark for Agentic PowerPoint Editing , author=. 2025 , eprint=

    2025

  19. [19]

    2024 , eprint=

    OfficeBench: Benchmarking Language Agents across Multiple Applications for Office Automation , author=. 2024 , eprint=

    2024

  20. [20]

    2025 , eprint=

    OdysseyBench: Evaluating LLM Agents on Long-Horizon Complex Office Application Workflows , author=. 2025 , eprint=

    2025

  21. [21]

    AgentBench: Evaluating LLMs as Agents , url =

    Liu, Xiao and Yu, Hao and Zhang, Hanchen and Xu, Yifan and Lei, Xuanyu and Lai, Hanyu and Gu, Yu and Ding, Hangliang and Men, Kaiwen and Yang, Kejuan and Zhang, Shudan and Deng, Xiang and Zeng, Aohan and Du, Zhengxiao and Zhang, Chenhui and Shen, Sheng and Zhang, Tianjun and Su, Yu and Sun, Huan and Huang, Minlie and Dong, Yuxiao and Tang, Jie , booktitle...

  22. [22]

    GAIA: a benchmark for General AI Assistants , url =

    Mialon, Gr\'. GAIA: a benchmark for General AI Assistants , url =. International Conference on Learning Representations , editor =

  23. [23]

    2025 , eprint=

    A-MemGuard: A Proactive Defense Framework for LLM-Based Agent Memory , author=. 2025 , eprint=

    2025

  24. [24]

    Unveiling Privacy Risks in LLM Agent Memory

    Wang, Bo and He, Weiyi and Zeng, Shenglai and Xiang, Zhen and Xing, Yue and Tang, Jiliang and He, Pengfei. Unveiling Privacy Risks in LLM Agent Memory. Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2025. doi:10.18653/v1/2025.acl-long.1227

    doi:10.18653/v1/2025.acl-long.1227 2025

  25. [25]

    ACM Trans

    He, Junda and Treude, Christoph and Lo, David , title =. ACM Trans. Softw. Eng. Methodol. , month = may, articleno =. 2025 , issue_date =. doi:10.1145/3712003 , abstract =

    doi:10.1145/3712003 2025

  26. [26]

    Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence,

    Large Language Model Based Multi-agents: A Survey of Progress and Challenges , author =. Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence,. 2024 , month =. doi:10.24963/ijcai.2024/890 , url =

    doi:10.24963/ijcai.2024/890 2024

  27. [27]

    WebArena: A Realistic Web Environment for Building Autonomous Agents , url =

    Zhou, Shuyan and Xu, Frank F and Zhu, Hao and Zhou, Xuhui and Lo, Robert and Sridhar, Abishek and Cheng, Xianyi and Ou, Tianyue and Bisk, Yonatan and Fried, Daniel and Alon, Uri and Neubig, Graham , booktitle =. WebArena: A Realistic Web Environment for Building Autonomous Agents , url =

  28. [28]

    V isual W eb A rena: Evaluating Multimodal Agents on Realistic Visual Web Tasks

    Koh, Jing Yu and Lo, Robert and Jang, Lawrence and Duvvur, Vikram and Lim, Ming and Huang, Po-Yu and Neubig, Graham and Zhou, Shuyan and Salakhutdinov, Russ and Fried, Daniel. V isual W eb A rena: Evaluating Multimodal Agents on Realistic Visual Web Tasks. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: L...

    doi:10.18653/v1/2024.acl-long.50 2024

  29. [29]

    2026 , eprint=

    LLMs Corrupt Your Documents When You Delegate , author=. 2026 , eprint=

    2026

  30. [30]

    2024 , eprint=

    WorkArena: How Capable Are Web Agents at Solving Common Knowledge Work Tasks? , author=. 2024 , eprint=

    2024

  31. [31]

    OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments , url =

    Xie, Tianbao and Zhang, Danyang and Chen, Jixuan and Li, Xiaochuan and Zhao, Siheng and Cao, Ruisheng and Hua, Toh Jing and Cheng, Zhoujun and Shin, Dongchan and Lei, Fangyu and Liu, Yitao and Xu, Yiheng and Zhou, Shuyan and Savarese, Silvio and Xiong, Caiming and Zhong, Victor and Yu, Tao , booktitle =. OSWorld: Benchmarking Multimodal Agents for Open-En...

    doi:10.52202/079017-1650

  32. [32]

    2024 , eprint=

    Windows Agent Arena: Evaluating Multi-Modal OS Agents at Scale , author=. 2024 , eprint=

    2024

  33. [33]

    A pp W orld: A Controllable World of Apps and People for Benchmarking Interactive Coding Agents

    Trivedi, Harsh and Khot, Tushar and Hartmann, Mareike and Manku, Ruskin and Dong, Vinty and Li, Edward and Gupta, Shashank and Sabharwal, Ashish and Balasubramanian, Niranjan. A pp W orld: A Controllable World of Apps and People for Benchmarking Interactive Coding Agents. Proceedings of the 62nd Annual Meeting of the Association for Computational Linguist...

    doi:10.18653/v1/2024.acl-long.850 2024

  34. [34]

    2025 , eprint=

    MCPWorld: A Unified Benchmarking Testbed for API, GUI, and Hybrid Computer Use Agents , author=. 2025 , eprint=

    2025

  35. [35]

    Equipping agents for the real world with Agent Skills , year =

  36. [36]

    Claude Code: an agentic coding tool , year =

  37. [37]

    OpenAI Codex CLI , year =

  38. [38]

    Harness engineering: leveraging Codex in an agent-first world , year =

  39. [39]

    2026 , eprint=

    SkillsBench: Benchmarking How Well Agent Skills Work Across Diverse Tasks , author=. 2026 , eprint=

    2026

  40. [40]

    2026 , eprint=

    Terminal-Bench: Benchmarking Agents on Hard, Realistic Tasks in Command Line Interfaces , author=. 2026 , eprint=

    2026

  41. [41]

    Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles , year=

    Efficient Memory Management for Large Language Model Serving with PagedAttention , author=. Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles , year=

  42. [42]

    Qwen3.5: Accelerating Productivity with Native Multimodal Agents , url =

  43. [43]

    2026 , eprint=

    Gemma 4 Technical Report , author=. 2026 , eprint=

    2026

  44. [44]

    Document Recognition and Retrieval X , editor =

    Song Mao and Azriel Rosenfeld and Tapas Kanungo , title =. Document Recognition and Retrieval X , editor =. 2003 , doi =

    2003

  45. [45]

    Task complexity: A review and conceptualization framework , journal =

    Peng Liu and Zhizhong Li , keywords =. Task complexity: A review and conceptualization framework , journal =. 2012 , issn =. doi:https://doi.org/10.1016/j.ergon.2012.09.001 , url =

    doi:10.1016/j.ergon.2012.09.001 2012

  46. [46]

    Stanton , abstract =

    Neville A. Stanton , abstract =. Hierarchical task analysis: Developments, applications, and extensions , journal =. 2006 , note =. doi:https://doi.org/10.1016/j.apergo.2005.06.003 , url =

    work page doi:10.1016/j.apergo.2005.06.003 2006

  47. [47]

    2025 , howpublished =

    Anthropic Skills , author =. 2025 , howpublished =

    2025

  48. [48]

    DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence , author=

  49. [49]

    2025 , eprint=

    GLM-4.5: Agentic, Reasoning, and Coding (ARC) Foundation Models , author=. 2025 , eprint=

    2025

  50. [50]

    and Verme, Manuel Del and Marty, Tom and Vazquez, David and Chapados, Nicolas and Lacoste, Alexandre , title =

    Drouin, Alexandre and Gasse, Maxime and Caccia, Massimo and Laradji, Issam H. and Verme, Manuel Del and Marty, Tom and Vazquez, David and Chapados, Nicolas and Lacoste, Alexandre , title =. Proceedings of the 41st International Conference on Machine Learning , articleno =. 2024 , publisher =

    2024

  51. [51]

    2026 , eprint=

    Memory in the Age of AI Agents , author=. 2026 , eprint=

    2026

  52. [52]

    2024 , eprint=

    MemGPT: Towards LLMs as Operating Systems , author=. 2024 , eprint=

    2024

  53. [53]

    2023 , eprint=

    ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIs , author=. 2023 , eprint=

    2023

  54. [54]

    API -Bank: A Comprehensive Benchmark for Tool-Augmented LLM s

    Li, Minghao and Zhao, Yingxiu and Yu, Bowen and Song, Feifan and Li, Hangyu and Yu, Haiyang and Li, Zhoujun and Huang, Fei and Li, Yongbin. API -Bank: A Comprehensive Benchmark for Tool-Augmented LLM s. Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. 2023. doi:10.18653/v1/2023.emnlp-main.187

    doi:10.18653/v1/2023.emnlp-main.187 2023

  55. [55]

    2023 , eprint=

    LLM as OS, Agents as Apps: Envisioning AIOS, Agents and the AIOS-Agent Ecosystem , author=. 2023 , eprint=

    2023

  56. [56]

    A survey on large language model based autonomous agents , volume=

    Wang, Lei and Ma, Chen and Feng, Xueyang and Zhang, Zeyu and Yang, Hao and Zhang, Jingsen and Chen, Zhiyuan and Tang, Jiakai and Chen, Xu and Lin, Yankai and Zhao, Wayne Xin and Wei, Zhewei and Wen, Jirong , year=. A survey on large language model based autonomous agents , volume=. Frontiers of Computer Science , publisher=. doi:10.1007/s11704-024-40231-1...

    doi:10.1007/s11704-024-40231-1

  57. [57]

    Document Understanding Dataset and Evaluation

    Jordy Van Landeghem and Rafal Powalski and Rub. Document Understanding Dataset and Evaluation. 2023 , url =. doi:10.1109/ICCV51070.2023.01789 , timestamp =

    arXiv 2023