REVIEW 3 major objections 6 minor 41 references
This paper shows that the programming language an agent works in is a major, model-consistent driver of token cost—OCaml costs 1.28–1.69× more than Python even after controlling for problem difficulty—and traces the extra tokens to compile-
Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →
T0 review · deepseek-v4-flash
2026-08-01 04:35 UTC pith:JJVWL4P7
load-bearing objection Solid empirical study of language-driven token cost, but the tokenmaxxing headline overreaches because the analysis uses medians when the practical claim needs means. the 3 major comments →
The Best Programming Language for Tokenmaxxing: An Investigation of Coding Agent Behavior Across Programming Languages
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
On the paper's own terms, the discovery is that language choice independently drives agent token cost. A mixed-effects model with problem as random effect shows OCaml costs 1.28–1.69× more output tokens than Python (p < 0.001 for all five models), with significant but model-dependent penalties for Java and Rust. At similar accuracy, OCaml is most expensive, Python cheapest. Re-executing every intermediate solution and labeling spans reveals why: agents in OCaml loop on compile errors and revert to byte-identical broken code, while Python solutions break through faster. Agents also polish already-passing solutions and sometimes prototype in Python before translating.
What carries the argument
The central machinery is the solution-snapshot trajectory: every solution file the agent writes is re-executed after each turn, and each consecutive pair of snapshots is classified by how test outcomes change (breakthrough, stuck, regressed, churn, stay, etc.). A 12-category span-label taxonomy—applied by an LLM judge on the work between snapshots—attributes actions like fix bug, debug failure, revert, and cosmetic refactor. This converts raw token counts into a causal story: token cost differences come from different distributions of these actions across languages.
Load-bearing premise
The load-bearing premise for the 'why' story is that the LLM-judge span labels are accurate enough to attribute token consumption to specific behaviors; the paper reports no human agreement or error analysis for these labels, and if they are systematically wrong the behavioral account collapses even though the raw token-cost differences remain.
What would settle it
Take a random sample of 200 snapshot spans, have two human annotators label them with the same 12-category taxonomy, and compare with the LLM judge's labels. If agreement is below substantial (e.g., Cohen's kappa < 0.6) or the error analysis shows systematic confusion between, say, 'fix bug' and 'cosmetic refactor', the behavioral-mechanism conclusions in §4.3–4.5 would not survive.
If this is right
- Token efficiency is language-dependent and should be a standard benchmark metric when evaluating multilingual coding agents.
- Agents working in lower-resource languages (OCaml, Rust) can be made more efficient by improving compile-error recovery and reducing revert loops, not by improving algorithm quality.
- Substantial tokens are wasted after a correct solution exists—Gemma spends hundreds to thousands of extra tokens polishing—so early-stopping or submission strategies can cut cost without hurting accuracy.
- Agents deliberately prototype in Python before translating to unfamiliar target languages, meaning token cost for a non-Python task can include hidden Python work.
- The language cost gap persists across both open-weight and proprietary models, so it is a property of the agent-plus-language combination rather than of one model family.
Where Pith is reading between the lines
- If the LLM span-label taxonomy is unreliable—the paper reports no human validation—the behavioral mechanism story may be overstated even though the raw token-cost result stands; a human-agreement check on a sample of spans would settle it.
- The 'tokenmaxxing' framing is satirical, but the finding has a serious cost-sensitivity reading: for users paying per token, Python is the cheapest target language, and for benchmark designers, per-language token cost should accompany accuracy numbers.
- The observed Python-prototyping behavior suggests a testable optimization: an agent explicitly allowed to reason in Python before translating might reduce total token cost; comparing a translation-prompted agent against direct generation would test this.
- Because token costs are heavy-tailed and the agent runs are capped at 40 turns/600 seconds, real-world long-running agents might show even larger language gaps than reported.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper studies how the choice of programming language affects the token consumption of coding agents. Using a 100-problem subset of LiveCodeBench (MiniLCB) and the mini-swe-agent scaffold, the authors run five models (Gemma, Qwen, GLM, GPT-5.5, Sonnet) in Python, Java, Rust, and OCaml, collecting 2,000 trajectories. They measure total output tokens per trajectory, fit mixed-effects models with problem difficulty as a random effect, and report that OCaml incurs a robust token penalty relative to Python (1.28–1.69×, p < 0.001) across models, with Python cheapest. They then analyze trajectories by re-executing every intermediate solution, abstracting test-outcome transitions, and labeling snapshot spans with a 12-category taxonomy applied by an LLM judge. This yields qualitative findings such as compile-error loops in OCaml, post-pass revision, planning in comments, and Python prototyping before translation. The paper argues that language choice is an important driver of agent process cost and that the mechanism is inefficient iterative correction on less familiar languages.
Significance. If the central claims hold, the paper identifies a practically important and relatively underexplored dimension of coding-agent cost: process cost varies systematically across programming languages even after controlling for problem difficulty. The design has notable strengths: 2,000 trajectories across five models and four languages; a clear external test harness; re-execution of intermediate snapshots; and a mixed-effects analysis that explicitly controls for problem identity. The snapshot-transition graph abstraction is a useful methodological contribution that could be reused by other trajectory studies. However, the headline cost claim is tied to a median/log-scale quantity, while the paper's own tokenmaxxing motivation is about total spend; and the qualitative mechanism story depends entirely on LLM-assigned span labels with no reported validation. Both issues are fixable with additional analyses, and the paper would be substantially stronger if the authors report mean/total token comparisons and validate the span-label taxonomy.
major comments (3)
- [§4.1, Fig. 1, Table 2] The headline claim that OCaml is the most expensive language for tokenmaxxing rests on medians and mixed-effects coefficients that are location shifts on the log scale. The paper itself notes that token costs are heavy-tailed and that 'a few hard problems inflate the mean,' but it never reports arithmetic means or total output tokens per language. A significant median or geometric-mean penalty can coexist with a larger right tail for Python, in which case Python could have the larger total token spend over a set of tasks. Since tokenmaxxing is about total expenditure, the paper should report arithmetic mean/sum token consumption per model–language pair with bootstrap confidence intervals, and report the relevant variance components (e.g., problem-level and language-level tail behavior). If the means support the same ordering, the claim is strengthened; if not, the conclusion should be re
- [§4.2.3, Tables 4–6, §4.3–4.5] All qualitative mechanism conclusions—compile-error loops, revert-dominated stuck states, Python prototyping, cosmetic refactoring—are aggregates of span labels assigned by GPT-5-Mini according to a 12-category taxonomy introduced in this paper. No validation is reported: no human agreement, no error analysis, no random spot-check, no per-language or per-model audit. The taxonomy is an invented coding scheme, so the reliability of the labels is load-bearing for the 'why' story. The authors should validate on a sample of spans (e.g., two human annotators with inter-annotator agreement, or a manual audit of a stratified random sample), report per-label accuracy/confusion, and either correct the label-based claims or qualify them as exploratory. The raw token-count result may survive even if labels are noisy, but the behavioral conclusions as stated require this evidence.
- [§4.1, Contribution (2), Fig. 1] The contribution states that 'at comparable success rates, lower-resource languages cost significantly more than Python,' but the paper does not formally establish comparable success rates. The text itself notes that Gemma performs much worse on OCaml, and Figure 1 suggests a visible accuracy gap for that pair. If success rates differ across languages, token comparisons across all traces mix failure-related spending with success-related spending, which is relevant to the interpretation that the language effect is not merely a success-rate artifact. The authors should report whether the token penalty persists when restricting to successful trajectories, or otherwise formally test and report the accuracy differences across languages. This would make the 'comparable success rates' claim precise.
minor comments (6)
- [Abstract] Typo: 'the cost (in tokens) can very significantly by programming language' should read 'can vary significantly.'
- [§4.1, Fig. 1] Figure 1 reports medians with no uncertainty intervals; adding bootstrap CIs around the medians (and around the mixed-effects contrasts) would help readers assess the stability of the language ordering.
- [Appendix E] The paper measures only output tokens, not input tokens. The justification based on prefix caching is reasonable for some platforms but not all; please state explicitly in the main text that 'cost' means output-token cost, and ideally report a sensitivity analysis with input tokens for the open-weight models where full histories are observable.
- [Table 1] The Rust compile command uses an en dash in '–release'; this should be '--release'.
- [§4.3] Minor typo: 'model-lanugage pair' should be 'model-language pair.'
- [§3.3 / §4] The paper says code and data will be released upon publication. For reproducibility, please make the MiniLCB subset, prompts, and the snapshot-re-execution harness available with the accepted version, not only 'upon publication.'
Circularity Check
No significant circularity: the token-cost result is a direct measurement, not a fitted or self-referential derivation.
full rationale
The paper's central claim—that OCaml traces consume more output tokens than Python traces (Table 2, §4.1)—is a direct measurement from 2,000 collected agent trajectories. The mixed-effects model estimates language coefficients from observed token counts while treating problem identity as a random effect; no parameter is fitted to a subset of the data and then presented as a prediction of a closely related quantity. The language contrasts are not constructed from the span-label taxonomy or from any prior result. The MiniLCB construction relies on the observation that 499 LiveCodeBench problems are language-neutral because they use standard I/O and have no starter code; this is a dataset property that is independently checkable and is not the paper's conclusion. The citations to the authors' own Agnostics and MultiPL-E work appear as background for using per-language compile-and-run configurations, but the load-bearing justification for the token-cost finding is the experimental data itself, not those citations. The span-label taxonomy was developed from trajectories and applied post hoc to explain observed differences; its lack of validation is a measurement-validity concern, not circularity. No equation in the paper defines one result in terms of another by construction, and no fitted input is renamed as a prediction. Statistical concerns about median versus mean effects are robustness issues, not evidence of circularity.
Axiom & Free-Parameter Ledger
free parameters (3)
- Mixed-effect language coefficients =
Gemma: Java 0.85, Rust 1.07, OCaml 1.44; Qwen: 1.21/1.20/1.69; GLM: 1.34/1.57/1.54; GPT-5.5: 1.30/1.36/1.30; Sonnet: 1.1
- Problem random-effect variance (ICC) =
0.73–0.97
- Budget limits =
40 turns, 600 s wall-clock, 6K/10K reasoning-token caps
axioms (6)
- domain assumption The 499 language-agnostic LiveCodeBench tasks are equally hard in all four languages when translated to standard I/O.
- domain assumption Output tokens are the correct cost measure; input tokens are negligible due to prefix caching.
- domain assumption Re-executing intermediate solutions offline with ./test.sh yields the same test outcomes the agent would see live.
- standard math The mixed-effects model's random intercept fully captures problem difficulty, with no language×problem interaction.
- ad hoc to paper GPT-5-Mini assigns span labels according to the 12-category taxonomy with adequate accuracy.
- domain assumption Agents follow the instruction 'These are the only tests I care about' and submit when ./test.sh passes.
invented entities (1)
-
12-category span-label taxonomy
no independent evidence
read the original abstract
Although coding agents are now very effective in a variety of programming languages, this paper first shows that the cost (in tokens) can very significantly by programming language. We evaluate five recent models on programming problems in Python, Java, Rust, and OCaml. We carefully control for problem difficulty, and show that there can be stark variation in token consumption that is consistent across models. To understand why, we analyze both the structure and content of agent trajectories. First, we re-execute every intermediate solution and abstract each trajectory as a sequence of test-outcome vectors, then label the work between successive solutions. This reveals agents repeatedly producing noncompiling solutions in unfamiliar languages and revising solutions that already pass. Second, we analyze trajectory text, finding that agents plan solutions in code comments, distrust the provided tests in favor of inputs they invent, and sidestep unfamiliar target languages by prototyping in Python. Our results show that by-language token efficiency is a metric that should be considered when benchmarking and developing multilingual agents, and, for the tokenmaxxer, a guide to the most expensive language to work in.
Figures
Reference graph
Works this paper leans on
-
[1]
Jensen Huang: Nvidia's Future, Physical AI, Rise of the Agent, Inference Explosion, AI PR Crisis , author =
-
[2]
Gonzalez and Clark Barrett and Ying Sheng , booktitle=
Lianmin Zheng and Liangsheng Yin and Zhiqiang Xie and Chuyue Sun and Jeff Huang and Cody Hao Yu and Shiyi Cao and Christos Kozyrakis and Ion Stoica and Joseph E. Gonzalez and Clark Barrett and Ying Sheng , booktitle=. 2024 , url=
2024
-
[3]
2026 , booktitle=
Agnostics: Learning to Code in Any Programming Language via Reinforcement with a Universal Learning Environment , author=. 2026 , booktitle=
2026
-
[4]
Maria Ivanova and Pavel Zadorozhny and Rodion Levichev and Ivan Petrov and Adamenko Pavel and Ivan Lopatin and Alexey Kutalev and Dmitrii Babaev , booktitle=. Multi-. 2026 , url=
2026
-
[5]
2026 , month = apr, howpublished =
Gemma 4: Byte for byte, the most capable open models , author =. 2026 , month = apr, howpublished =
2026
-
[6]
2026 , month = apr, howpublished =
2026
-
[7]
2026 , month = feb, howpublished =
2026
-
[8]
Kwon, Woosuk and Li, Zhuohan and Zhuang, Siyuan and Sheng, Ying and Zheng, Lianmin and Yu, Cody Hao and Gonzalez, Joseph and Zhang, Hao and Stoica, Ion , title =. 2023 , isbn =. doi:10.1145/3600006.3613165 , booktitle =
arXiv 2023
-
[9]
Lozhkov, Anton and Li, Raymond and Allal, Loubna Ben and Cassano, Federico and. 2024 , month = feb, number =. doi:10.48550/arXiv.2402.19173 , urldate =. arXiv , keywords =:2402.19173 , primaryclass =
-
[10]
European Review , author=
‘Improving ratings’: audit in the British University system , volume=. European Review , author=. 1997 , pages=
1997
-
[11]
arXiv preprint arXiv:2604.22750 , year=
How do AI agents spend your money? Analyzing and predicting token consumption in agentic coding tasks , author=. arXiv preprint arXiv:2604.22750 , year=
-
[12]
Proceedings of the ACM on Software Engineering , volume=
Reducing cost of llm agents with trajectory reduction , author=. Proceedings of the ACM on Software Engineering , volume=
-
[13]
arXiv preprint arXiv:2601.14470 , year=
Tokenomics: Quantifying Where Tokens Are Used in Agentic Software Engineering , author=. arXiv preprint arXiv:2601.14470 , year=
-
[14]
arXiv preprint arXiv:2507.03254 , year=
Codeagents: A token-efficient framework for codified multi-agent reasoning in llms , author=. arXiv preprint arXiv:2507.03254 , year=
-
[15]
Agent Systems with Harness Engineering , author=
-
[16]
arXiv preprint arXiv:2508.00083 , year=
A survey on code generation with llm-based agents , author=. arXiv preprint arXiv:2508.00083 , year=
-
[17]
2023 , note =
Cassano, Federico and Gouwar, John and Nguyen, Daniel and Nguyen, Sydney and Phipps-Costin, Luna and Pinckney, Donald and Yee, Ming-Ho and Zi, Yangtian and Anderson, Carolyn Jane and Feldman, Molly Q and Guha, Arjun and Greenberg, Michael and Jangda, Abhinav , journal =. 2023 , note =
2023
-
[18]
arXiv preprint arXiv:2210.14868 , year=
Multi-lingual evaluation of code generation models , author=. arXiv preprint arXiv:2210.14868 , year=
-
[19]
International Conference on Learning Representations , volume=
Livecodebench: Holistic and contamination free evaluation of large language models for code , author=. International Conference on Learning Representations , volume=
-
[20]
Advances in Neural Information Processing Systems , volume=
Multi-swe-bench: A multilingual benchmark for issue resolving , author=. Advances in Neural Information Processing Systems , volume=
-
[21]
International Conference on Learning Representations , volume=
Swe-bench: Can language models resolve real-world github issues? , author=. International Conference on Learning Representations , volume=
-
[22]
Advances in Neural Information Processing Systems , volume=
Swe-agent: Agent-computer interfaces enable automated software engineering , author=. Advances in Neural Information Processing Systems , volume=
-
[23]
2025 , eprint=
SWE-smith: Scaling Data for Software Engineering Agents , author=. 2025 , eprint=
2025
-
[24]
arXiv preprint arXiv:2601.11868 , year=
Terminal-bench: Benchmarking agents on hard, realistic tasks in command line interfaces , author=. arXiv preprint arXiv:2601.11868 , year=
-
[25]
arXiv preprint arXiv:2602.10975 , year=
Featurebench: Benchmarking agentic coding for complex feature development , author=. arXiv preprint arXiv:2602.10975 , year=
-
[26]
arXiv preprint arXiv:2602.05892 , year=
Contextbench: A benchmark for context retrieval in coding agents , author=. arXiv preprint arXiv:2602.05892 , year=
-
[27]
arXiv preprint arXiv:2606.07297 , year=
SWE-Explore: Benchmarking How Coding Agents Explore Repositories , author=. arXiv preprint arXiv:2606.07297 , year=
-
[28]
Advances in Neural Information Processing Systems , volume=
Effibench-x: A multi-language benchmark for measuring efficiency of llm-generated code , author=. Advances in Neural Information Processing Systems , volume=
-
[29]
arXiv preprint arXiv:2509.09853 , year=
Swe-effi: Re-evaluating software ai agent system effectiveness under resource constraints , author=. arXiv preprint arXiv:2509.09853 , year=
-
[30]
Proceedings of the 48th IEEE/ACM International Conference on Software Engineering , series =
Dangfeng Pan and Zhensu Sun and Cenyuan Zhang and David Lo and Xiaoning Du , title =. Proceedings of the 48th IEEE/ACM International Conference on Software Engineering , series =. 2026 , eprint =
2026
-
[31]
2025 IEEE/ACM 40th International Conference on Automated Software Engineering , series =
Zhensu Sun and Chengran Yang and Xiaoning Du and Zhou Yang and Li Li and David Lo , title =. 2025 IEEE/ACM 40th International Conference on Automated Software Engineering , series =. 2025 , doi =
2025
-
[32]
2026 , eprint =
Priyansh Trivedi and Olivier Schmitt , title =. 2026 , eprint =
2026
-
[33]
2026 , eprint=
Evaluating AGENTS.md: Are Repository-Level Context Files Helpful for Coding Agents? , author=. 2026 , eprint=
2026
-
[34]
2026 , eprint =
Lola Solovyeva and Fernando Castor , title =. 2026 , eprint =
2026
-
[35]
Proceedings of the 48th IEEE/ACM International Conference on Software Engineering , series =
Pengfei Gao and Chao Peng , title =. Proceedings of the 48th IEEE/ACM International Conference on Software Engineering , series =. 2026 , numpages =
2026
-
[36]
Zhang and Mark Harman and Yiling Lou and Yang Liu and Zhenpeng Chen , title =
Yaoqi Guo and Ying Xiao and Jie M. Zhang and Mark Harman and Yiling Lou and Yang Liu and Zhenpeng Chen , title =. Findings of the Association for Computational Linguistics: ACL 2026 , year =. 2601.05777 , archivePrefix =
Pith/arXiv arXiv 2026
-
[37]
2026 , eprint =
Aman Sharma and Sushrut Thorat and Paras Chopra , title =. 2026 , eprint =
2026
-
[38]
2025 IEEE/ACM 40th International Conference on Automated Software Engineering , series =
Islem Bouzenia and Michael Pradel , title =. 2025 IEEE/ACM 40th International Conference on Automated Software Engineering , series =. 2025 , doi =
2025
-
[39]
Proceedings of the ACM on Programming Languages , volume =
Shuyang Liu and Yang Chen and Rahul Krishna and Saurabh Sinha and Jatin Ganhotra and Reyhaneh Jabbarvand , title =. Proceedings of the ACM on Programming Languages , volume =. 2026 , doi =
2026
-
[40]
Advances in neural information processing systems , year=
EffiBench-X: A Multi-Language Benchmark for Measuring Efficiency of LLM-Generated Code , author=. Advances in neural information processing systems , year=
-
[41]
2026 , eprint=
GLM-5: from Vibe Coding to Agentic Engineering , author=. 2026 , eprint=
2026
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.