Pith. sign in

REVIEW 3 major objections 51 references

ToFu is an open, white-box agent harness that fixes more real software issues than leading coding agents while using far fewer tokens.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

T0 review · grok-4.5

2026-07-14 05:40 UTC pith:B3O3HNTQ

load-bearing objection Solid open harness paper: real SWE-bench gains and token savings vs Claude Code, dual research-assistant pitch still thinner than the coding evidence. the 3 major comments →

arxiv 2607.11423 v1 pith:B3O3HNTQ submitted 2026-07-13 cs.CL

ToFu: A White-Box, Token-Efficient Agent Harness for Researchers

classification cs.CL
keywords agent harnesstoken efficiencycontext compactionSWE-benchmultilingual agentsagent orchestrationopen-source agentsresearch assistants
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper argues that agent systems succeed or fail not only because of the language model, but because of the harness: the orchestration code that plans, calls tools, manages context, and recovers from errors. Existing harnesses force a trade-off between open research prototypes that are hard to use as products and polished closed products that are poor instruments for science. ToFu is offered as a single open-source system that researchers can inspect, modify, and run locally, yet still delivers application-level workflows and stronger coding results than commercial and open baselines. Its main technical bet is three-layer context compaction plus multilingual translate-then-reason and a dependency-aware swarm of sub-agents. A sympathetic reader cares because cheaper, inspectable agent loops could turn research assistants from advice-givers into systems that actually finish goals without leaking private codebases.

Core claim

On the full SWE-bench Verified suite of 500 instances, ToFu achieves higher Pass@1 than a leading commercial coding agent and an open coding baseline across three backbone models, while using 28.4 percent fewer tokens on average than the commercial agent (up to 43.6 percent fewer), with competitive cost and turn counts. The same harness is meant to serve dual roles: a practical research assistant that edits code, papers, and tools, and a white-box object for studying harness design itself.

What carries the argument

Three-layer context compaction: size-aware budgeting that replaces large tool outputs with previews and recoverable references; cache-aware micro-compaction that shortens cold history without rewriting cached prefixes; and query-aware semantic summarization of older turns when the context nears its limit, while keeping the current turn intact.

Load-bearing premise

That strong coding-benchmark results plus a small human preference study are enough to support the dual claim of a general research assistant and a reusable harness-research baseline.

What would settle it

Independent full runs of SWE-bench Verified with the same three backbone models showing ToFu at equal or higher average token use than the commercial baseline while matching or underperforming its Pass@1 would falsify the accuracy-plus-efficiency claim.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Privacy-sensitive researchers can run capable agent loops locally under an open license instead of shipping codebases to closed services.
  • Harness-evolution research gains a baseline that is both strong on coding benchmarks and usable as a product surface.
  • Non-English users can keep their interface language while the agent reasons in English, improving multilingual coding outcomes on average.
  • Longer test-time token budgets are not automatically better; orchestration quality can beat simply spending more tokens.
  • Protocol-based tool adapters can turn manuscript, version-control, and note systems into first-class agent environments.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If three-layer compaction generalizes past coding, the same pattern may cut cost for long-horizon research agents that browse papers, edit manuscripts, and run experiments overnight.
  • Future harness comparisons may need to report recovery under stream failures and cache-hit stability as first-class metrics alongside Pass@1.
  • Translate-then-reason may need language-specific safeguards when technical terms or code-adjacent prose are mistranslated.
  • Open harnesses with structured tool-error return and automatic retry may matter more for unattended work than small gains on curated benchmarks alone.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 0 minor

Summary. The paper presents TOFU, an open-source (MIT) agentic harness for research workflows that combines a modular architecture (UI, orchestration core, model abstraction, capability runtime, state/knowledge, context management) with three-layer context compaction, BM25 memory retrieval, multilingual translate-then-reason, and a dependency-aware swarm scheduler. Empirically, on the full 500-instance SWE-bench Verified suite graded by the official harness in udocker, TOFU reports higher Pass@1 than Claude Code and OpenCode across Claude Opus 4.6, GLM 5.1, and DeepSeek-v4-pro, with ~28.4% fewer average tokens than Claude Code (Table 1). Multilingual gains are shown on MAPS:SWE-bench (Fig. 3), and Appendix A reports a blind human preference study favoring TOFU. The dual claim is that TOFU is both a practical research assistant and a white-box baseline for harness research with application-level UX.

Significance. If the coding results hold under independent reproduction, the work is a useful systems contribution: an open, inspectable harness that improves Pass@1 while reducing tokens relative to a strong commercial baseline across three backbones is rare and directly relevant to harness-evolution research. Strengths include full-suite SWE-bench Verified evaluation with official grading, multi-backbone comparison, open-source release, and a concrete design comparison to OpenCode (Table 3). The dual positioning as research assistant plus research object is timely for the community studying harness design, provided the non-coding claims are scoped to the evidence actually provided.

major comments (3)
  1. Abstract and §1 claim superior token efficiency, lower cost, and practical research-assistant support relative to existing harnesses. Table 1 supports higher Pass@1 and lower average tokens vs Claude Code, but cost is not uniformly lower (Claude Opus 4.6: $5.13 vs $4.97; OpenCode is cheaper on two backbones). The dual research-assistant claim therefore overstates cost superiority and rests mainly on coding benchmarks. Limitations already note that broader research-assistant evaluation is limited; the abstract/intro should be tightened to match Table 1 (token efficiency + Pass@1) rather than blanket 'lower cost' and general research workflows.
  2. §4 and Limitations vs Appendix A: the main text states broader evaluation is limited to a small human preference study with only three participants, while Appendix A reports seven annotators × 100 tasks (700 pairwise comparisons) with Claude Opus 4.6 only, no token/cost metrics, and no non-coding agentic baselines beyond an unspecified baseline. This size inconsistency and the coding-only proxy for the dual claim are load-bearing for the 'practical research assistant' half. Please reconcile participant counts, name the baseline system, and either add non-coding efficiency metrics or explicitly demote the dual claim to coding-centric evidence plus a preference pilot.
  3. §2.2 three-layer compaction and §4.2 efficiency claims: the manuscript attributes the 28.4% token reduction primarily to harness design, but free parameters (preview budgets, micro-compaction rules, ~80% semantic-compaction trigger, BM25 retrieval) are not ablated. Without at least a minimal ablation or sensitivity analysis on SWE-bench (or a subset), it is hard to attribute gains to the layered design versus other orchestration differences in Table 3 (retry/recovery, batched tools, planner-worker-critic). A compact ablation or controlled disable of one compaction layer would substantially strengthen the central efficiency claim.

Circularity Check

0 steps flagged

No circular derivation: ToFu's claims are empirical harness comparisons on external benchmarks, not results forced by definition or self-citation.

full rationale

ToFu is a systems paper whose load-bearing claims are measured outcomes, not closed-form derivations. Pass@1, token counts, cost, and turns on full SWE-bench Verified (Table 1) are obtained by running ToFu, Claude Code, and OpenCode under the official SWE-bench harness with three external backbone LLMs; those numbers are not defined in terms of the design knobs they are used to justify. Multilingual gains (Fig. 3) are likewise measured on MAPS:SWE-bench, and the human preference study (Appendix A) is a blind pairwise rating against a baseline—not a fit renamed as a prediction. Context compaction, BM25 memory retrieval, swarm scheduling, and the translate-then-reason layer are engineering choices whose success is evaluated after the fact; none of the reported metrics reduce by construction to those choices. Self-citations (e.g., Ge et al. 2024, Mu et al. 2024) appear only as background on multilingual/instruction work and do not underwrite uniqueness theorems or forbid alternatives. There is no self-definitional loop, no fitted parameter re-labeled as a prediction, and no uniqueness claim imported from the authors' prior work. Scope concerns about coding benchmarks proxying a broader research-assistant claim are evaluation-coverage issues, not circularity. Score 0 with empty steps is therefore the correct finding.

Axiom & Free-Parameter Ledger

4 free parameters · 4 axioms · 2 invented entities

This is an engineering systems paper. The load-bearing content is design choices plus empirical evaluation, not a formal derivation. Free parameters are operational thresholds and retrieval/orchestration settings. Axioms are standard agent-evaluation and systems assumptions. Invented entities are named design modules of ToFu rather than physical postulates.

free parameters (4)
  • semantic_compaction_trigger_threshold = ~80% of context limit
    Query-aware LLM summarization fires only when conversation approaches usable context limit (paper cites about 80%); this threshold is a design choice that affects token counts and behavior.
  • tool_output_preview_budget
    Size-aware budgeting replaces large tool outputs with compact previews and references; the preview size is an engineering free parameter controlling measured token use.
  • BM25_memory_retrieval_ranking
    Memory retrieval uses BM25 over metadata/content rather than embeddings; ranking and injection policy are chosen design parameters that affect which knowledge enters context.
  • swarm_concurrency_and_retry_bounds = retry up to 16 (interrupted stream)
    Dependency-aware swarm uses concurrency control, rate-limit-aware execution, and automatic retries (interrupted streams re-run up to 16 times with backoff in the OpenCode comparison); these bounds affect success and cost.
axioms (4)
  • domain assumption SWE-bench Verified Pass@1 is a valid primary proxy for agentic coding harness quality and a useful signal for research-assistant utility.
    Section 4 frames agentic coding as the primary harness task; dual-role claims partly rest on this proxy.
  • domain assumption Comparing harnesses with the same backbone LLMs isolates harness effects from model intelligence.
    Stated in evaluation setup; true only if tool surfaces, retries, approvals, and context policies are otherwise comparable.
  • domain assumption Translate-then-reason preserves task semantics for non-English users while exploiting English-centric model strength.
    Section 2.4; MAPS gains of ~2.5 points average support it partially but not universally across languages.
  • ad hoc to paper Deterministic micro-compaction that avoids rewriting cached prefixes reduces tokens without material loss of operational context.
    Core of three-layer design in §2.2; not independently proven beyond end-to-end benchmark outcomes.
invented entities (2)
  • ToFu three-layer context compaction stack no independent evidence
    purpose: Reduce redundant context while preserving tool-use continuity via size-aware tool budgets, cache-aware micro-compaction, and query-aware semantic summarization.
    Presented as ToFu’s central efficiency mechanism; evidence is end-to-end token/Pass@1 comparison rather than isolated causal ablations.
  • ToFu dependency-aware swarm scheduler no independent evidence
    purpose: Coordinate parallel sub-agents with DAG dependencies, result injection, and reactive spawning.
    Described in §2.5 as an adaptive execution mechanism; not separately benchmarked against alternative multi-agent schedulers in the main results table.

pith-pipeline@v1.1.0-grok45 · 16182 in / 3333 out tokens · 48722 ms · 2026-07-14T05:40:55.404386+00:00 · methodology

0 comments
read the original abstract

Agentic coding tools present new opportunities to transform research workflows. The performance of agent systems built depends on both large language models (LLMs) and the harness around LLMs, which is the orchestration code that determines an agent's behavior. We present ToFu, an agentic harness for researchers that reads your codebase, edits files, runs commands, and integrates with your development tools. ToFu plays a dual role in research. As a research assistant, it supports practical research workflows with superior token efficiency, lower cost, and multilingual capability compared with existing agentic harnesses. Its release under the MIT License further enables local deployment for privacy-sensitive users. As a research object, ToFu provides a white-box agentic harness that allows researchers to inspect, modify, and evaluate its orchestration logic, tool-use behavior, and harness design, while retaining strong benchmark performance and an application-level user experience.

Figures

Figures reproduced from arXiv: 2607.11423 by Bei Li, Chenglong Wang, Jingang Wang, Jingbo Zhu, Junhao Ruan, Tong Xiao, Xin Chen, Yongjing Yin, Yuan Ge, Yuchun Fan.

Figure 1
Figure 1. Figure 1: Human interaction with TOFU harness. a degree of autonomy (Anthropic, 2025; OpenAI, 2025; OpenClaw, 2026). Harness enables the agent to read documents or codebase, plan sequences of actions, execute them using real development tools, evaluate the results, and adjust its approach. The widespread adoption of current agentic sys￾tems is constrained by the following challenges of agentic harness. (i) Token eff… view at source ↗
Figure 2
Figure 2. Figure 2: The overview of TOFU harness. ing the tool call process and returning the results of the tool call along with any resulting effects. Capability Runtime module includes native capa￾bilities such as code/project tools, shell execution, web search, desktop/browser access, document pro￾cessing, image generation, and scheduling, as well as extension adapters such as MCPs and external agent APIs. TOFU supports a… view at source ↗
Figure 3
Figure 3. Figure 3: Performance of TOFU with multilingual en￾hancement (M-Enh) on MAPS: SWE-bench. ments in 7 languages, while the remaining three are nearly unchanged or exhibit slight declines. 5 Related work Agentic coding tools. An agentic system acts toward a goal with a degree of autonomy, rather than LLMs that respond to one prompt at a time or an agent workflow designed for specific tasks (Anthropic, 2025; OpenAI, 202… view at source ↗
Figure 4
Figure 4. Figure 4: Demonstration UI page of TOFU and paper reader. Design axis OpenCode TOFU Context management Long tool outputs Every result is hard truncated to a fixed size, and the overflow is saved to a file the agent must re-open Keeps a bounded preview plus a re-expandable reference, and leaves file reads intact to avoid re-read loops Reusing old context Rewrites old tool results in place, which invalidates the provi… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

51 extracted references · 3 canonical work pages

  1. [1]

    Aho and Jeffrey D

    Alfred V. Aho and Jeffrey D. Ullman , title =. 1972

  2. [2]

    Publications Manual , year = "1983", publisher =

  3. [3]

    Chandra and Dexter C

    Ashok K. Chandra and Dexter C. Kozen and Larry J. Stockmeyer , year = "1981", title =. doi:10.1145/322234.322243

  4. [4]

    Scalable training of

    Andrew, Galen and Gao, Jianfeng , booktitle=. Scalable training of

  5. [5]

    Dan Gusfield , title =. 1997

  6. [6]

    Tetreault , title =

    Mohammad Sadegh Rasooli and Joel R. Tetreault , title =. Computing Research Repository , volume =. 2015 , url =

  7. [7]

    A Framework for Learning Predictive Structures from Multiple Tasks and Unlabeled Data , Volume =

    Ando, Rie Kubota and Zhang, Tong , Issn =. A Framework for Learning Predictive Structures from Multiple Tasks and Unlabeled Data , Volume =. Journal of Machine Learning Research , Month = dec, Numpages =

  8. [8]

    The Claude 3 Model Family: Opus, Sonnet, Haiku , author=

  9. [9]

    2025 , howpublished=

    Claude Code: Anthropic's agentic coding system , author=. 2025 , howpublished=

  10. [10]

    2025 , howpublished=

    Codex: AI Coding Partner from OpenAI , author=. 2025 , howpublished=

  11. [11]

    2024 , howpublished=

    Cursor: The AI Code Editor , author=. 2024 , howpublished=

  12. [12]

    2023 , howpublished=

  13. [13]

    International Conference on Learning Representations (ICLR) , year =

    ReAct: Synergizing Reasoning and Acting in Language Models , author =. International Conference on Learning Representations (ICLR) , year =

  14. [14]

    2024 , howpublished=

    Introducing the Model Context Protocol , author=. 2024 , howpublished=

  15. [15]

    Proceedings of the 29th Symposium on Operating Systems Principles , pages=

    Efficient memory management for large language model serving with PagedAttention , author=. Proceedings of the 29th Symposium on Operating Systems Principles , pages=

  16. [16]

    The Twelfth International Conference on Learning Representations , year=

    SWE-bench: Can Language Models Resolve Real-world Github Issues? , author=. The Twelfth International Conference on Learning Representations , year=

  17. [17]

    Advances in neural information processing systems , volume=

    Autosurvey: Large language models can automatically write surveys , author=. Advances in neural information processing systems , volume=

  18. [18]

    arXiv preprint arXiv:2510.24701 , year=

    Tongyi deepresearch technical report , author=. arXiv preprint arXiv:2510.24701 , year=

  19. [19]

    OpenHands: An Open Platform for AI Software Developers as Generalist Agents , author=

  20. [20]

    arXiv preprint arXiv:2412.19437 , year=

    Deepseek-v3 technical report , author=. arXiv preprint arXiv:2412.19437 , year=

  21. [21]

    arXiv preprint arXiv:2601.03267 , year=

    Openai gpt-5 system card , author=. arXiv preprint arXiv:2601.03267 , year=

  22. [22]

    arXiv preprint arXiv:2507.02592 , year=

    Websailor: Navigating super-human reasoning for web agent , author=. arXiv preprint arXiv:2507.02592 , year=

  23. [23]

    , author =

    OpenClaw: Your own personal AI assistant. , author =. 2026 , howpublished =

  24. [24]

    MaxClaw: Build Your AI Agent with Skills and Subagents , author =

  25. [25]

    The Eleventh International Conference on Learning Representations , year =

    Language models are multilingual chain-of-thought reasoners , author=. The Eleventh International Conference on Learning Representations , year =

  26. [26]

    arXiv preprint arXiv:2404.04925 , year=

    Multilingual large language model: A survey of resources, taxonomy and frontiers , author=. arXiv preprint arXiv:2404.04925 , year=

  27. [27]

    Revealing the Parallel Multilingual Learning within Large Language Models

    Mu, Yongyu and Feng, Peinan and Cao, Zhiquan and Wu, Yuzhang and Li, Bei and Wang, Chenglong and Xiao, Tong and Song, Kai and Liu, Tongran and Zhang, Chunliang and Zhu, JingBo. Revealing the Parallel Multilingual Learning within Large Language Models. Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing. 2024. doi:10.1865...

  28. [28]

    MAPS : A Multilingual Benchmark for Agent Performance and Security

    Hofman, Omer and Brokman, Jonathan and Rachmil, Oren and Bose, Shamik and Pahuja, Vikas and Shimizu, Toshiya and Starostina, Trisha and Marchisio, Kelly and Goldfarb-Tarrant, Seraphina and Vainshtein, Roman. MAPS : A Multilingual Benchmark for Agent Performance and Security. Findings of the A ssociation for C omputational L inguistics: EACL 2026. 2026. do...

  29. [29]

    arXiv preprint arXiv:2409.00088 , year=

    On-device language models: A comprehensive review , author=. arXiv preprint arXiv:2409.00088 , year=

  30. [30]

    Clustering and Ranking: Diversity-preserved Instruction Selection through Expert-aligned Quality Estimation

    Ge, Yuan and Liu, Yilun and Hu, Chi and Meng, Weibin and Tao, Shimin and Zhao, Xiaofeng and Xia, Mahong and Li, Zhang and Chen, Boxing and Yang, Hao and Li, Bei and Xiao, Tong and Zhu, JingBo. Clustering and Ranking: Diversity-preserved Instruction Selection through Expert-aligned Quality Estimation. Proceedings of the 2024 Conference on Empirical Methods...

  31. [31]

    PAPILLON: Privacy preservation from Internet-based and local language model ensembles , author=. Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers) , pages=

  32. [32]

    ICLR 2025 Workshop on Building Trust in Language Models and Applications , year=

    On-premises LLM deployment demands a middle path: Preserving privacy without sacrificing model confidentiality , author=. ICLR 2025 Workshop on Building Trust in Language Models and Applications , year=

  33. [33]

    Advances in neural information processing systems , volume=

    Toolformer: Language models can teach themselves to use tools , author=. Advances in neural information processing systems , volume=

  34. [34]

    Advances in neural information processing systems , volume=

    Reflexion: Language agents with verbal reinforcement learning , author=. Advances in neural information processing systems , volume=

  35. [35]

    Sirui Hong and Mingchen Zhuge and Jonathan Chen and Xiawu Zheng and Yuheng Cheng and Jinlin Wang and Ceyao Zhang and Zili Wang and Steven Ka Shing Yau and Zijuan Lin and Liyang Zhou and Chenyu Ran and Lingfeng Xiao and Chenglin Wu and J. Meta. The Twelfth International Conference on Learning Representations , year=

  36. [36]

    First conference on language modeling , year=

    Autogen: Enabling next-gen LLM applications via multi-agent conversations , author=. First conference on language modeling , year=

  37. [37]

    T iny S cientist: An Interactive, Extensible, and Controllable Framework for Building Research Agents

    Yu, Haofei and Xuan, Keyang and Li, Fenghai and Zhu, Kunlun and Lei, Zijie and Zhang, Jiaxun and Qi, Ziheng and Richardson, Kyle and You, Jiaxuan. T iny S cientist: An Interactive, Extensible, and Controllable Framework for Building Research Agents. Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing: System Demonstratio...

  38. [38]

    R esear S tudio: A Human-intervenable Framework for Building Controllable Deep Research Agents

    Yang, Linyi and Weng, Yixuan. R esear S tudio: A Human-intervenable Framework for Building Controllable Deep Research Agents. Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing: System Demonstrations. 2025. doi:10.18653/v1/2025.emnlp-demos.69

  39. [39]

    2026 , howpublished =

    Anthropic , title =. 2026 , howpublished =

  40. [40]

    2026 , howpublished =

    Zhipu-AI , title =. 2026 , howpublished =

  41. [41]

    GitHub repository , howpublished =

    MiniMax-AI , title =. GitHub repository , howpublished =. 2026 , publisher =

  42. [42]

    2024 , howpublished =

    OpenAI , title =. 2024 , howpublished =

  43. [43]

    Proceedings of the 29th symposium on operating systems principles , pages=

    Efficient memory management for large language model serving with pagedattention , author=. Proceedings of the 29th symposium on operating systems principles , pages=

  44. [44]

    , author=

    Ollama: Ollama is the easiest way to automate your work using open models, while keeping your data safe. , author=. 2024 , howpublished=

  45. [45]

    2026 , howpublished=

  46. [46]

    arXiv preprint arXiv:2603.28052 , year=

    Meta-harness: End-to-end optimization of model harnesses , author=. arXiv preprint arXiv:2603.28052 , year=

  47. [47]

    arXiv preprint arXiv:2604.25850 , year=

    Agentic harness engineering: Observability-driven automatic evolution of coding-agent harnesses , author=. arXiv preprint arXiv:2604.25850 , year=

  48. [48]

    arXiv preprint arXiv:2605.30621 , year=

    Harness Updating Is Not Harness Benefit: Disentangling Evolution Capabilities in Self-Evolving LLM Agents , author=. arXiv preprint arXiv:2605.30621 , year=

  49. [49]

    arXiv preprint arXiv:2606.01779 , year=

    HarnessForge: Joint Harness and Policy Evolution for Adaptive Agent Systems , author=. arXiv preprint arXiv:2606.01779 , year=

  50. [50]

    arXiv preprint arXiv:2604.21003 , year=

    The Last Harness You'll Ever Build , author=. arXiv preprint arXiv:2604.21003 , year=

  51. [51]

    Yang, John and Jimenez, Carlos E and Wettig, Alexander and Lieret, Kilian and Yao, Shunyu and Narasimhan, Karthik and Press, Ofir , year=