REVIEW 3 major objections 51 references
ToFu is an open, white-box agent harness that fixes more real software issues than leading coding agents while using far fewer tokens.
Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →
T0 review · grok-4.5
2026-07-14 05:40 UTC pith:B3O3HNTQ
load-bearing objection Solid open harness paper: real SWE-bench gains and token savings vs Claude Code, dual research-assistant pitch still thinner than the coding evidence. the 3 major comments →
ToFu: A White-Box, Token-Efficient Agent Harness for Researchers
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
On the full SWE-bench Verified suite of 500 instances, ToFu achieves higher Pass@1 than a leading commercial coding agent and an open coding baseline across three backbone models, while using 28.4 percent fewer tokens on average than the commercial agent (up to 43.6 percent fewer), with competitive cost and turn counts. The same harness is meant to serve dual roles: a practical research assistant that edits code, papers, and tools, and a white-box object for studying harness design itself.
What carries the argument
Three-layer context compaction: size-aware budgeting that replaces large tool outputs with previews and recoverable references; cache-aware micro-compaction that shortens cold history without rewriting cached prefixes; and query-aware semantic summarization of older turns when the context nears its limit, while keeping the current turn intact.
Load-bearing premise
That strong coding-benchmark results plus a small human preference study are enough to support the dual claim of a general research assistant and a reusable harness-research baseline.
What would settle it
Independent full runs of SWE-bench Verified with the same three backbone models showing ToFu at equal or higher average token use than the commercial baseline while matching or underperforming its Pass@1 would falsify the accuracy-plus-efficiency claim.
If this is right
- Privacy-sensitive researchers can run capable agent loops locally under an open license instead of shipping codebases to closed services.
- Harness-evolution research gains a baseline that is both strong on coding benchmarks and usable as a product surface.
- Non-English users can keep their interface language while the agent reasons in English, improving multilingual coding outcomes on average.
- Longer test-time token budgets are not automatically better; orchestration quality can beat simply spending more tokens.
- Protocol-based tool adapters can turn manuscript, version-control, and note systems into first-class agent environments.
Where Pith is reading between the lines
- If three-layer compaction generalizes past coding, the same pattern may cut cost for long-horizon research agents that browse papers, edit manuscripts, and run experiments overnight.
- Future harness comparisons may need to report recovery under stream failures and cache-hit stability as first-class metrics alongside Pass@1.
- Translate-then-reason may need language-specific safeguards when technical terms or code-adjacent prose are mistranslated.
- Open harnesses with structured tool-error return and automatic retry may matter more for unattended work than small gains on curated benchmarks alone.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper presents TOFU, an open-source (MIT) agentic harness for research workflows that combines a modular architecture (UI, orchestration core, model abstraction, capability runtime, state/knowledge, context management) with three-layer context compaction, BM25 memory retrieval, multilingual translate-then-reason, and a dependency-aware swarm scheduler. Empirically, on the full 500-instance SWE-bench Verified suite graded by the official harness in udocker, TOFU reports higher Pass@1 than Claude Code and OpenCode across Claude Opus 4.6, GLM 5.1, and DeepSeek-v4-pro, with ~28.4% fewer average tokens than Claude Code (Table 1). Multilingual gains are shown on MAPS:SWE-bench (Fig. 3), and Appendix A reports a blind human preference study favoring TOFU. The dual claim is that TOFU is both a practical research assistant and a white-box baseline for harness research with application-level UX.
Significance. If the coding results hold under independent reproduction, the work is a useful systems contribution: an open, inspectable harness that improves Pass@1 while reducing tokens relative to a strong commercial baseline across three backbones is rare and directly relevant to harness-evolution research. Strengths include full-suite SWE-bench Verified evaluation with official grading, multi-backbone comparison, open-source release, and a concrete design comparison to OpenCode (Table 3). The dual positioning as research assistant plus research object is timely for the community studying harness design, provided the non-coding claims are scoped to the evidence actually provided.
major comments (3)
- Abstract and §1 claim superior token efficiency, lower cost, and practical research-assistant support relative to existing harnesses. Table 1 supports higher Pass@1 and lower average tokens vs Claude Code, but cost is not uniformly lower (Claude Opus 4.6: $5.13 vs $4.97; OpenCode is cheaper on two backbones). The dual research-assistant claim therefore overstates cost superiority and rests mainly on coding benchmarks. Limitations already note that broader research-assistant evaluation is limited; the abstract/intro should be tightened to match Table 1 (token efficiency + Pass@1) rather than blanket 'lower cost' and general research workflows.
- §4 and Limitations vs Appendix A: the main text states broader evaluation is limited to a small human preference study with only three participants, while Appendix A reports seven annotators × 100 tasks (700 pairwise comparisons) with Claude Opus 4.6 only, no token/cost metrics, and no non-coding agentic baselines beyond an unspecified baseline. This size inconsistency and the coding-only proxy for the dual claim are load-bearing for the 'practical research assistant' half. Please reconcile participant counts, name the baseline system, and either add non-coding efficiency metrics or explicitly demote the dual claim to coding-centric evidence plus a preference pilot.
- §2.2 three-layer compaction and §4.2 efficiency claims: the manuscript attributes the 28.4% token reduction primarily to harness design, but free parameters (preview budgets, micro-compaction rules, ~80% semantic-compaction trigger, BM25 retrieval) are not ablated. Without at least a minimal ablation or sensitivity analysis on SWE-bench (or a subset), it is hard to attribute gains to the layered design versus other orchestration differences in Table 3 (retry/recovery, batched tools, planner-worker-critic). A compact ablation or controlled disable of one compaction layer would substantially strengthen the central efficiency claim.
Circularity Check
No circular derivation: ToFu's claims are empirical harness comparisons on external benchmarks, not results forced by definition or self-citation.
full rationale
ToFu is a systems paper whose load-bearing claims are measured outcomes, not closed-form derivations. Pass@1, token counts, cost, and turns on full SWE-bench Verified (Table 1) are obtained by running ToFu, Claude Code, and OpenCode under the official SWE-bench harness with three external backbone LLMs; those numbers are not defined in terms of the design knobs they are used to justify. Multilingual gains (Fig. 3) are likewise measured on MAPS:SWE-bench, and the human preference study (Appendix A) is a blind pairwise rating against a baseline—not a fit renamed as a prediction. Context compaction, BM25 memory retrieval, swarm scheduling, and the translate-then-reason layer are engineering choices whose success is evaluated after the fact; none of the reported metrics reduce by construction to those choices. Self-citations (e.g., Ge et al. 2024, Mu et al. 2024) appear only as background on multilingual/instruction work and do not underwrite uniqueness theorems or forbid alternatives. There is no self-definitional loop, no fitted parameter re-labeled as a prediction, and no uniqueness claim imported from the authors' prior work. Scope concerns about coding benchmarks proxying a broader research-assistant claim are evaluation-coverage issues, not circularity. Score 0 with empty steps is therefore the correct finding.
Axiom & Free-Parameter Ledger
free parameters (4)
- semantic_compaction_trigger_threshold =
~80% of context limit
- tool_output_preview_budget
- BM25_memory_retrieval_ranking
- swarm_concurrency_and_retry_bounds =
retry up to 16 (interrupted stream)
axioms (4)
- domain assumption SWE-bench Verified Pass@1 is a valid primary proxy for agentic coding harness quality and a useful signal for research-assistant utility.
- domain assumption Comparing harnesses with the same backbone LLMs isolates harness effects from model intelligence.
- domain assumption Translate-then-reason preserves task semantics for non-English users while exploiting English-centric model strength.
- ad hoc to paper Deterministic micro-compaction that avoids rewriting cached prefixes reduces tokens without material loss of operational context.
invented entities (2)
-
ToFu three-layer context compaction stack
no independent evidence
-
ToFu dependency-aware swarm scheduler
no independent evidence
read the original abstract
Agentic coding tools present new opportunities to transform research workflows. The performance of agent systems built depends on both large language models (LLMs) and the harness around LLMs, which is the orchestration code that determines an agent's behavior. We present ToFu, an agentic harness for researchers that reads your codebase, edits files, runs commands, and integrates with your development tools. ToFu plays a dual role in research. As a research assistant, it supports practical research workflows with superior token efficiency, lower cost, and multilingual capability compared with existing agentic harnesses. Its release under the MIT License further enables local deployment for privacy-sensitive users. As a research object, ToFu provides a white-box agentic harness that allows researchers to inspect, modify, and evaluate its orchestration logic, tool-use behavior, and harness design, while retaining strong benchmark performance and an application-level user experience.
Figures
Reference graph
Works this paper leans on
-
[1]
Aho and Jeffrey D
Alfred V. Aho and Jeffrey D. Ullman , title =. 1972
1972
-
[2]
Publications Manual , year = "1983", publisher =
1983
-
[3]
Ashok K. Chandra and Dexter C. Kozen and Larry J. Stockmeyer , year = "1981", title =. doi:10.1145/322234.322243
-
[4]
Scalable training of
Andrew, Galen and Gao, Jianfeng , booktitle=. Scalable training of
-
[5]
Dan Gusfield , title =. 1997
1997
-
[6]
Tetreault , title =
Mohammad Sadegh Rasooli and Joel R. Tetreault , title =. Computing Research Repository , volume =. 2015 , url =
2015
-
[7]
A Framework for Learning Predictive Structures from Multiple Tasks and Unlabeled Data , Volume =
Ando, Rie Kubota and Zhang, Tong , Issn =. A Framework for Learning Predictive Structures from Multiple Tasks and Unlabeled Data , Volume =. Journal of Machine Learning Research , Month = dec, Numpages =
-
[8]
The Claude 3 Model Family: Opus, Sonnet, Haiku , author=
-
[9]
2025 , howpublished=
Claude Code: Anthropic's agentic coding system , author=. 2025 , howpublished=
2025
-
[10]
2025 , howpublished=
Codex: AI Coding Partner from OpenAI , author=. 2025 , howpublished=
2025
-
[11]
2024 , howpublished=
Cursor: The AI Code Editor , author=. 2024 , howpublished=
2024
-
[12]
2023 , howpublished=
2023
-
[13]
International Conference on Learning Representations (ICLR) , year =
ReAct: Synergizing Reasoning and Acting in Language Models , author =. International Conference on Learning Representations (ICLR) , year =
-
[14]
2024 , howpublished=
Introducing the Model Context Protocol , author=. 2024 , howpublished=
2024
-
[15]
Proceedings of the 29th Symposium on Operating Systems Principles , pages=
Efficient memory management for large language model serving with PagedAttention , author=. Proceedings of the 29th Symposium on Operating Systems Principles , pages=
-
[16]
The Twelfth International Conference on Learning Representations , year=
SWE-bench: Can Language Models Resolve Real-world Github Issues? , author=. The Twelfth International Conference on Learning Representations , year=
-
[17]
Advances in neural information processing systems , volume=
Autosurvey: Large language models can automatically write surveys , author=. Advances in neural information processing systems , volume=
-
[18]
arXiv preprint arXiv:2510.24701 , year=
Tongyi deepresearch technical report , author=. arXiv preprint arXiv:2510.24701 , year=
-
[19]
OpenHands: An Open Platform for AI Software Developers as Generalist Agents , author=
-
[20]
arXiv preprint arXiv:2412.19437 , year=
Deepseek-v3 technical report , author=. arXiv preprint arXiv:2412.19437 , year=
-
[21]
arXiv preprint arXiv:2601.03267 , year=
Openai gpt-5 system card , author=. arXiv preprint arXiv:2601.03267 , year=
-
[22]
arXiv preprint arXiv:2507.02592 , year=
Websailor: Navigating super-human reasoning for web agent , author=. arXiv preprint arXiv:2507.02592 , year=
-
[23]
, author =
OpenClaw: Your own personal AI assistant. , author =. 2026 , howpublished =
2026
-
[24]
MaxClaw: Build Your AI Agent with Skills and Subagents , author =
-
[25]
The Eleventh International Conference on Learning Representations , year =
Language models are multilingual chain-of-thought reasoners , author=. The Eleventh International Conference on Learning Representations , year =
-
[26]
arXiv preprint arXiv:2404.04925 , year=
Multilingual large language model: A survey of resources, taxonomy and frontiers , author=. arXiv preprint arXiv:2404.04925 , year=
-
[27]
Revealing the Parallel Multilingual Learning within Large Language Models
Mu, Yongyu and Feng, Peinan and Cao, Zhiquan and Wu, Yuzhang and Li, Bei and Wang, Chenglong and Xiao, Tong and Song, Kai and Liu, Tongran and Zhang, Chunliang and Zhu, JingBo. Revealing the Parallel Multilingual Learning within Large Language Models. Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing. 2024. doi:10.1865...
-
[28]
MAPS : A Multilingual Benchmark for Agent Performance and Security
Hofman, Omer and Brokman, Jonathan and Rachmil, Oren and Bose, Shamik and Pahuja, Vikas and Shimizu, Toshiya and Starostina, Trisha and Marchisio, Kelly and Goldfarb-Tarrant, Seraphina and Vainshtein, Roman. MAPS : A Multilingual Benchmark for Agent Performance and Security. Findings of the A ssociation for C omputational L inguistics: EACL 2026. 2026. do...
-
[29]
arXiv preprint arXiv:2409.00088 , year=
On-device language models: A comprehensive review , author=. arXiv preprint arXiv:2409.00088 , year=
-
[30]
Ge, Yuan and Liu, Yilun and Hu, Chi and Meng, Weibin and Tao, Shimin and Zhao, Xiaofeng and Xia, Mahong and Li, Zhang and Chen, Boxing and Yang, Hao and Li, Bei and Xiao, Tong and Zhu, JingBo. Clustering and Ranking: Diversity-preserved Instruction Selection through Expert-aligned Quality Estimation. Proceedings of the 2024 Conference on Empirical Methods...
-
[31]
PAPILLON: Privacy preservation from Internet-based and local language model ensembles , author=. Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers) , pages=
2025
-
[32]
ICLR 2025 Workshop on Building Trust in Language Models and Applications , year=
On-premises LLM deployment demands a middle path: Preserving privacy without sacrificing model confidentiality , author=. ICLR 2025 Workshop on Building Trust in Language Models and Applications , year=
2025
-
[33]
Advances in neural information processing systems , volume=
Toolformer: Language models can teach themselves to use tools , author=. Advances in neural information processing systems , volume=
-
[34]
Advances in neural information processing systems , volume=
Reflexion: Language agents with verbal reinforcement learning , author=. Advances in neural information processing systems , volume=
-
[35]
Sirui Hong and Mingchen Zhuge and Jonathan Chen and Xiawu Zheng and Yuheng Cheng and Jinlin Wang and Ceyao Zhang and Zili Wang and Steven Ka Shing Yau and Zijuan Lin and Liyang Zhou and Chenyu Ran and Lingfeng Xiao and Chenglin Wu and J. Meta. The Twelfth International Conference on Learning Representations , year=
-
[36]
First conference on language modeling , year=
Autogen: Enabling next-gen LLM applications via multi-agent conversations , author=. First conference on language modeling , year=
-
[37]
Yu, Haofei and Xuan, Keyang and Li, Fenghai and Zhu, Kunlun and Lei, Zijie and Zhang, Jiaxun and Qi, Ziheng and Richardson, Kyle and You, Jiaxuan. T iny S cientist: An Interactive, Extensible, and Controllable Framework for Building Research Agents. Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing: System Demonstratio...
-
[38]
R esear S tudio: A Human-intervenable Framework for Building Controllable Deep Research Agents
Yang, Linyi and Weng, Yixuan. R esear S tudio: A Human-intervenable Framework for Building Controllable Deep Research Agents. Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing: System Demonstrations. 2025. doi:10.18653/v1/2025.emnlp-demos.69
-
[39]
2026 , howpublished =
Anthropic , title =. 2026 , howpublished =
2026
-
[40]
2026 , howpublished =
Zhipu-AI , title =. 2026 , howpublished =
2026
-
[41]
GitHub repository , howpublished =
MiniMax-AI , title =. GitHub repository , howpublished =. 2026 , publisher =
2026
-
[42]
2024 , howpublished =
OpenAI , title =. 2024 , howpublished =
2024
-
[43]
Proceedings of the 29th symposium on operating systems principles , pages=
Efficient memory management for large language model serving with pagedattention , author=. Proceedings of the 29th symposium on operating systems principles , pages=
-
[44]
, author=
Ollama: Ollama is the easiest way to automate your work using open models, while keeping your data safe. , author=. 2024 , howpublished=
2024
-
[45]
2026 , howpublished=
2026
-
[46]
arXiv preprint arXiv:2603.28052 , year=
Meta-harness: End-to-end optimization of model harnesses , author=. arXiv preprint arXiv:2603.28052 , year=
-
[47]
arXiv preprint arXiv:2604.25850 , year=
Agentic harness engineering: Observability-driven automatic evolution of coding-agent harnesses , author=. arXiv preprint arXiv:2604.25850 , year=
-
[48]
arXiv preprint arXiv:2605.30621 , year=
Harness Updating Is Not Harness Benefit: Disentangling Evolution Capabilities in Self-Evolving LLM Agents , author=. arXiv preprint arXiv:2605.30621 , year=
-
[49]
arXiv preprint arXiv:2606.01779 , year=
HarnessForge: Joint Harness and Policy Evolution for Adaptive Agent Systems , author=. arXiv preprint arXiv:2606.01779 , year=
-
[50]
arXiv preprint arXiv:2604.21003 , year=
The Last Harness You'll Ever Build , author=. arXiv preprint arXiv:2604.21003 , year=
-
[51]
Yang, John and Jimenez, Carlos E and Wettig, Alexander and Lieret, Kilian and Yao, Shunyu and Narasimhan, Karthik and Press, Ofir , year=
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.