Pith. sign in

REVIEW 3 major objections 6 minor 41 references

This paper shows that the programming language an agent works in is a major, model-consistent driver of token cost—OCaml costs 1.28–1.69× more than Python even after controlling for problem difficulty—and traces the extra tokens to compile-

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

T0 review · deepseek-v4-flash

2026-08-01 04:35 UTC pith:JJVWL4P7

load-bearing objection Solid empirical study of language-driven token cost, but the tokenmaxxing headline overreaches because the analysis uses medians when the practical claim needs means. the 3 major comments →

arxiv 2607.22807 v1 pith:JJVWL4P7 submitted 2026-07-24 cs.SE cs.CL

The Best Programming Language for Tokenmaxxing: An Investigation of Coding Agent Behavior Across Programming Languages

classification cs.SE cs.CL
keywords token consumptioncoding agentsprogramming languagestrajectory analysismixed-effects modelcompile errorsagent behaviormultilingual code generation
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper establishes that for coding agents, token consumption (and therefore dollar cost) is strongly and consistently affected by the programming language, not just by problem difficulty. Across five models and four languages, OCaml is the most expensive (1.28–1.69× over Python) and Python the cheapest. The authors trace this to concrete agent behaviors: looping on compile errors in less familiar languages, rewriting already-passing solutions, and quietly prototyping in Python before translating. The result matters because it makes by-language token efficiency a measurable, actionable property of agents and a benchmark axis for multilingual agent development.

Core claim

On the paper's own terms, the discovery is that language choice independently drives agent token cost. A mixed-effects model with problem as random effect shows OCaml costs 1.28–1.69× more output tokens than Python (p < 0.001 for all five models), with significant but model-dependent penalties for Java and Rust. At similar accuracy, OCaml is most expensive, Python cheapest. Re-executing every intermediate solution and labeling spans reveals why: agents in OCaml loop on compile errors and revert to byte-identical broken code, while Python solutions break through faster. Agents also polish already-passing solutions and sometimes prototype in Python before translating.

What carries the argument

The central machinery is the solution-snapshot trajectory: every solution file the agent writes is re-executed after each turn, and each consecutive pair of snapshots is classified by how test outcomes change (breakthrough, stuck, regressed, churn, stay, etc.). A 12-category span-label taxonomy—applied by an LLM judge on the work between snapshots—attributes actions like fix bug, debug failure, revert, and cosmetic refactor. This converts raw token counts into a causal story: token cost differences come from different distributions of these actions across languages.

Load-bearing premise

The load-bearing premise for the 'why' story is that the LLM-judge span labels are accurate enough to attribute token consumption to specific behaviors; the paper reports no human agreement or error analysis for these labels, and if they are systematically wrong the behavioral account collapses even though the raw token-cost differences remain.

What would settle it

Take a random sample of 200 snapshot spans, have two human annotators label them with the same 12-category taxonomy, and compare with the LLM judge's labels. If agreement is below substantial (e.g., Cohen's kappa < 0.6) or the error analysis shows systematic confusion between, say, 'fix bug' and 'cosmetic refactor', the behavioral-mechanism conclusions in §4.3–4.5 would not survive.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Token efficiency is language-dependent and should be a standard benchmark metric when evaluating multilingual coding agents.
  • Agents working in lower-resource languages (OCaml, Rust) can be made more efficient by improving compile-error recovery and reducing revert loops, not by improving algorithm quality.
  • Substantial tokens are wasted after a correct solution exists—Gemma spends hundreds to thousands of extra tokens polishing—so early-stopping or submission strategies can cut cost without hurting accuracy.
  • Agents deliberately prototype in Python before translating to unfamiliar target languages, meaning token cost for a non-Python task can include hidden Python work.
  • The language cost gap persists across both open-weight and proprietary models, so it is a property of the agent-plus-language combination rather than of one model family.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the LLM span-label taxonomy is unreliable—the paper reports no human validation—the behavioral mechanism story may be overstated even though the raw token-cost result stands; a human-agreement check on a sample of spans would settle it.
  • The 'tokenmaxxing' framing is satirical, but the finding has a serious cost-sensitivity reading: for users paying per token, Python is the cheapest target language, and for benchmark designers, per-language token cost should accompany accuracy numbers.
  • The observed Python-prototyping behavior suggests a testable optimization: an agent explicitly allowed to reason in Python before translating might reduce total token cost; comparing a translation-prompted agent against direct generation would test this.
  • Because token costs are heavy-tailed and the agent runs are capped at 40 turns/600 seconds, real-world long-running agents might show even larger language gaps than reported.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper studies how the choice of programming language affects the token consumption of coding agents. Using a 100-problem subset of LiveCodeBench (MiniLCB) and the mini-swe-agent scaffold, the authors run five models (Gemma, Qwen, GLM, GPT-5.5, Sonnet) in Python, Java, Rust, and OCaml, collecting 2,000 trajectories. They measure total output tokens per trajectory, fit mixed-effects models with problem difficulty as a random effect, and report that OCaml incurs a robust token penalty relative to Python (1.28–1.69×, p < 0.001) across models, with Python cheapest. They then analyze trajectories by re-executing every intermediate solution, abstracting test-outcome transitions, and labeling snapshot spans with a 12-category taxonomy applied by an LLM judge. This yields qualitative findings such as compile-error loops in OCaml, post-pass revision, planning in comments, and Python prototyping before translation. The paper argues that language choice is an important driver of agent process cost and that the mechanism is inefficient iterative correction on less familiar languages.

Significance. If the central claims hold, the paper identifies a practically important and relatively underexplored dimension of coding-agent cost: process cost varies systematically across programming languages even after controlling for problem difficulty. The design has notable strengths: 2,000 trajectories across five models and four languages; a clear external test harness; re-execution of intermediate snapshots; and a mixed-effects analysis that explicitly controls for problem identity. The snapshot-transition graph abstraction is a useful methodological contribution that could be reused by other trajectory studies. However, the headline cost claim is tied to a median/log-scale quantity, while the paper's own tokenmaxxing motivation is about total spend; and the qualitative mechanism story depends entirely on LLM-assigned span labels with no reported validation. Both issues are fixable with additional analyses, and the paper would be substantially stronger if the authors report mean/total token comparisons and validate the span-label taxonomy.

major comments (3)
  1. [§4.1, Fig. 1, Table 2] The headline claim that OCaml is the most expensive language for tokenmaxxing rests on medians and mixed-effects coefficients that are location shifts on the log scale. The paper itself notes that token costs are heavy-tailed and that 'a few hard problems inflate the mean,' but it never reports arithmetic means or total output tokens per language. A significant median or geometric-mean penalty can coexist with a larger right tail for Python, in which case Python could have the larger total token spend over a set of tasks. Since tokenmaxxing is about total expenditure, the paper should report arithmetic mean/sum token consumption per model–language pair with bootstrap confidence intervals, and report the relevant variance components (e.g., problem-level and language-level tail behavior). If the means support the same ordering, the claim is strengthened; if not, the conclusion should be re
  2. [§4.2.3, Tables 4–6, §4.3–4.5] All qualitative mechanism conclusions—compile-error loops, revert-dominated stuck states, Python prototyping, cosmetic refactoring—are aggregates of span labels assigned by GPT-5-Mini according to a 12-category taxonomy introduced in this paper. No validation is reported: no human agreement, no error analysis, no random spot-check, no per-language or per-model audit. The taxonomy is an invented coding scheme, so the reliability of the labels is load-bearing for the 'why' story. The authors should validate on a sample of spans (e.g., two human annotators with inter-annotator agreement, or a manual audit of a stratified random sample), report per-label accuracy/confusion, and either correct the label-based claims or qualify them as exploratory. The raw token-count result may survive even if labels are noisy, but the behavioral conclusions as stated require this evidence.
  3. [§4.1, Contribution (2), Fig. 1] The contribution states that 'at comparable success rates, lower-resource languages cost significantly more than Python,' but the paper does not formally establish comparable success rates. The text itself notes that Gemma performs much worse on OCaml, and Figure 1 suggests a visible accuracy gap for that pair. If success rates differ across languages, token comparisons across all traces mix failure-related spending with success-related spending, which is relevant to the interpretation that the language effect is not merely a success-rate artifact. The authors should report whether the token penalty persists when restricting to successful trajectories, or otherwise formally test and report the accuracy differences across languages. This would make the 'comparable success rates' claim precise.
minor comments (6)
  1. [Abstract] Typo: 'the cost (in tokens) can very significantly by programming language' should read 'can vary significantly.'
  2. [§4.1, Fig. 1] Figure 1 reports medians with no uncertainty intervals; adding bootstrap CIs around the medians (and around the mixed-effects contrasts) would help readers assess the stability of the language ordering.
  3. [Appendix E] The paper measures only output tokens, not input tokens. The justification based on prefix caching is reasonable for some platforms but not all; please state explicitly in the main text that 'cost' means output-token cost, and ideally report a sensitivity analysis with input tokens for the open-weight models where full histories are observable.
  4. [Table 1] The Rust compile command uses an en dash in '–release'; this should be '--release'.
  5. [§4.3] Minor typo: 'model-lanugage pair' should be 'model-language pair.'
  6. [§3.3 / §4] The paper says code and data will be released upon publication. For reproducibility, please make the MiniLCB subset, prompts, and the snapshot-re-execution harness available with the accepted version, not only 'upon publication.'

Circularity Check

0 steps flagged

No significant circularity: the token-cost result is a direct measurement, not a fitted or self-referential derivation.

full rationale

The paper's central claim—that OCaml traces consume more output tokens than Python traces (Table 2, §4.1)—is a direct measurement from 2,000 collected agent trajectories. The mixed-effects model estimates language coefficients from observed token counts while treating problem identity as a random effect; no parameter is fitted to a subset of the data and then presented as a prediction of a closely related quantity. The language contrasts are not constructed from the span-label taxonomy or from any prior result. The MiniLCB construction relies on the observation that 499 LiveCodeBench problems are language-neutral because they use standard I/O and have no starter code; this is a dataset property that is independently checkable and is not the paper's conclusion. The citations to the authors' own Agnostics and MultiPL-E work appear as background for using per-language compile-and-run configurations, but the load-bearing justification for the token-cost finding is the experimental data itself, not those citations. The span-label taxonomy was developed from trajectories and applied post hoc to explain observed differences; its lack of validation is a measurement-validity concern, not circularity. No equation in the paper defines one result in terms of another by construction, and no fitted input is renamed as a prediction. Statistical concerns about median versus mean effects are robustness issues, not evidence of circularity.

Axiom & Free-Parameter Ledger

3 free parameters · 6 axioms · 1 invented entities

The paper's quantitative core is an observational measurement on agent traces; the only fitted quantities are the mixed-effects coefficients. The behavioral mechanism story adds a human/AI-derived taxonomy that is not externally validated. No new physical entities are introduced.

free parameters (3)
  • Mixed-effect language coefficients = Gemma: Java 0.85, Rust 1.07, OCaml 1.44; Qwen: 1.21/1.20/1.69; GLM: 1.34/1.57/1.54; GPT-5.5: 1.30/1.36/1.30; Sonnet: 1.1
    These fitted coefficients are the paper's central quantitative output; the claim that OCaml is expensive rests entirely on their magnitude and significance.
  • Problem random-effect variance (ICC) = 0.73–0.97
    The mixed-effects model uses problem as a random effect; the reported ICC justifies the claim that problem difficulty dominates and is being controlled.
  • Budget limits = 40 turns, 600 s wall-clock, 6K/10K reasoning-token caps
    Chosen by hand; these caps bound token generation and therefore shape the measured cost ratios and the amount of post-solution fiddling observed.
axioms (6)
  • domain assumption The 499 language-agnostic LiveCodeBench tasks are equally hard in all four languages when translated to standard I/O.
    Invoked in §3.3 to build MiniLCB; if false, language token ratios could reflect task-translation bias. Based on cited Agnostics/MultiLCB.
  • domain assumption Output tokens are the correct cost measure; input tokens are negligible due to prefix caching.
    Stated in §E; if false, total spend ranking could differ, especially for long multi-turn trajectories.
  • domain assumption Re-executing intermediate solutions offline with ./test.sh yields the same test outcomes the agent would see live.
    Used in §4.2.2 to build snapshot graphs; differences in timeouts or recompilation could mislabel states.
  • standard math The mixed-effects model's random intercept fully captures problem difficulty, with no language×problem interaction.
    Standard assumption of the LMM in §4.1; not tested in the paper.
  • ad hoc to paper GPT-5-Mini assigns span labels according to the 12-category taxonomy with adequate accuracy.
    Invoked in §4.2.3; no validation reported, making the behavioral classification unsupported.
  • domain assumption Agents follow the instruction 'These are the only tests I care about' and submit when ./test.sh passes.
    Prompt template in §F; the cost-to-solution and post-solution analysis assumes agents act on this directive.
invented entities (1)
  • 12-category span-label taxonomy no independent evidence
    purpose: Classifies agent actions between solution snapshots (explore, implement, revert, etc.).
    An analytic construct iterated by the authors; no inter-annotator agreement or external validation; behavioral claims in §4.3/§4.4 depend on it.

pith-pipeline@v1.3.0-alltime-deepseek · 14188 in / 13358 out tokens · 136235 ms · 2026-08-01T04:35:31.499840+00:00 · methodology

0 comments
read the original abstract

Although coding agents are now very effective in a variety of programming languages, this paper first shows that the cost (in tokens) can very significantly by programming language. We evaluate five recent models on programming problems in Python, Java, Rust, and OCaml. We carefully control for problem difficulty, and show that there can be stark variation in token consumption that is consistent across models. To understand why, we analyze both the structure and content of agent trajectories. First, we re-execute every intermediate solution and abstract each trajectory as a sequence of test-outcome vectors, then label the work between successive solutions. This reveals agents repeatedly producing noncompiling solutions in unfamiliar languages and revising solutions that already pass. Second, we analyze trajectory text, finding that agents plan solutions in code comments, distrust the provided tests in favor of inputs they invent, and sidestep unfamiliar target languages by prototyping in Python. Our results show that by-language token efficiency is a metric that should be considered when benchmarking and developing multilingual agents, and, for the tokenmaxxer, a guide to the most expensive language to work in.

Figures

Figures reproduced from arXiv: 2607.22807 by Arjun Guha, Carolyn Jane Anderson, Zixuan Wu.

Figure 1
Figure 1. Figure 1: Accuracy versus median output token usage [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: We test every solution that the agent writes to the solution file, which allows us to abstract each trajectory [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Solution-snapshot transition graphs for GPT-5.5, using the same node and edge semantics as Figure [PITH_FULL_IMAGE:figures/full_fig_p012_3.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

41 extracted references · 8 linked inside Pith

  1. [1]

    Jensen Huang: Nvidia's Future, Physical AI, Rise of the Agent, Inference Explosion, AI PR Crisis , author =

  2. [2]

    Gonzalez and Clark Barrett and Ying Sheng , booktitle=

    Lianmin Zheng and Liangsheng Yin and Zhiqiang Xie and Chuyue Sun and Jeff Huang and Cody Hao Yu and Shiyi Cao and Christos Kozyrakis and Ion Stoica and Joseph E. Gonzalez and Clark Barrett and Ying Sheng , booktitle=. 2024 , url=

  3. [3]

    2026 , booktitle=

    Agnostics: Learning to Code in Any Programming Language via Reinforcement with a Universal Learning Environment , author=. 2026 , booktitle=

  4. [4]

    Maria Ivanova and Pavel Zadorozhny and Rodion Levichev and Ivan Petrov and Adamenko Pavel and Ivan Lopatin and Alexey Kutalev and Dmitrii Babaev , booktitle=. Multi-. 2026 , url=

  5. [5]

    2026 , month = apr, howpublished =

    Gemma 4: Byte for byte, the most capable open models , author =. 2026 , month = apr, howpublished =

  6. [6]

    2026 , month = apr, howpublished =

  7. [7]

    2026 , month = feb, howpublished =

  8. [8]

    2023 , isbn =

    Kwon, Woosuk and Li, Zhuohan and Zhuang, Siyuan and Sheng, Ying and Zheng, Lianmin and Yu, Cody Hao and Gonzalez, Joseph and Zhang, Hao and Stoica, Ion , title =. 2023 , isbn =. doi:10.1145/3600006.3613165 , booktitle =

  9. [9]

    2024 , month = feb, number =

    Lozhkov, Anton and Li, Raymond and Allal, Loubna Ben and Cassano, Federico and. 2024 , month = feb, number =. doi:10.48550/arXiv.2402.19173 , urldate =. arXiv , keywords =:2402.19173 , primaryclass =

  10. [10]

    European Review , author=

    ‘Improving ratings’: audit in the British University system , volume=. European Review , author=. 1997 , pages=

  11. [11]

    arXiv preprint arXiv:2604.22750 , year=

    How do AI agents spend your money? Analyzing and predicting token consumption in agentic coding tasks , author=. arXiv preprint arXiv:2604.22750 , year=

  12. [12]

    Proceedings of the ACM on Software Engineering , volume=

    Reducing cost of llm agents with trajectory reduction , author=. Proceedings of the ACM on Software Engineering , volume=

  13. [13]

    arXiv preprint arXiv:2601.14470 , year=

    Tokenomics: Quantifying Where Tokens Are Used in Agentic Software Engineering , author=. arXiv preprint arXiv:2601.14470 , year=

  14. [14]

    arXiv preprint arXiv:2507.03254 , year=

    Codeagents: A token-efficient framework for codified multi-agent reasoning in llms , author=. arXiv preprint arXiv:2507.03254 , year=

  15. [15]

    Agent Systems with Harness Engineering , author=

  16. [16]

    arXiv preprint arXiv:2508.00083 , year=

    A survey on code generation with llm-based agents , author=. arXiv preprint arXiv:2508.00083 , year=

  17. [17]

    2023 , note =

    Cassano, Federico and Gouwar, John and Nguyen, Daniel and Nguyen, Sydney and Phipps-Costin, Luna and Pinckney, Donald and Yee, Ming-Ho and Zi, Yangtian and Anderson, Carolyn Jane and Feldman, Molly Q and Guha, Arjun and Greenberg, Michael and Jangda, Abhinav , journal =. 2023 , note =

  18. [18]

    arXiv preprint arXiv:2210.14868 , year=

    Multi-lingual evaluation of code generation models , author=. arXiv preprint arXiv:2210.14868 , year=

  19. [19]

    International Conference on Learning Representations , volume=

    Livecodebench: Holistic and contamination free evaluation of large language models for code , author=. International Conference on Learning Representations , volume=

  20. [20]

    Advances in Neural Information Processing Systems , volume=

    Multi-swe-bench: A multilingual benchmark for issue resolving , author=. Advances in Neural Information Processing Systems , volume=

  21. [21]

    International Conference on Learning Representations , volume=

    Swe-bench: Can language models resolve real-world github issues? , author=. International Conference on Learning Representations , volume=

  22. [22]

    Advances in Neural Information Processing Systems , volume=

    Swe-agent: Agent-computer interfaces enable automated software engineering , author=. Advances in Neural Information Processing Systems , volume=

  23. [23]

    2025 , eprint=

    SWE-smith: Scaling Data for Software Engineering Agents , author=. 2025 , eprint=

  24. [24]

    arXiv preprint arXiv:2601.11868 , year=

    Terminal-bench: Benchmarking agents on hard, realistic tasks in command line interfaces , author=. arXiv preprint arXiv:2601.11868 , year=

  25. [25]

    arXiv preprint arXiv:2602.10975 , year=

    Featurebench: Benchmarking agentic coding for complex feature development , author=. arXiv preprint arXiv:2602.10975 , year=

  26. [26]

    arXiv preprint arXiv:2602.05892 , year=

    Contextbench: A benchmark for context retrieval in coding agents , author=. arXiv preprint arXiv:2602.05892 , year=

  27. [27]

    arXiv preprint arXiv:2606.07297 , year=

    SWE-Explore: Benchmarking How Coding Agents Explore Repositories , author=. arXiv preprint arXiv:2606.07297 , year=

  28. [28]

    Advances in Neural Information Processing Systems , volume=

    Effibench-x: A multi-language benchmark for measuring efficiency of llm-generated code , author=. Advances in Neural Information Processing Systems , volume=

  29. [29]

    arXiv preprint arXiv:2509.09853 , year=

    Swe-effi: Re-evaluating software ai agent system effectiveness under resource constraints , author=. arXiv preprint arXiv:2509.09853 , year=

  30. [30]

    Proceedings of the 48th IEEE/ACM International Conference on Software Engineering , series =

    Dangfeng Pan and Zhensu Sun and Cenyuan Zhang and David Lo and Xiaoning Du , title =. Proceedings of the 48th IEEE/ACM International Conference on Software Engineering , series =. 2026 , eprint =

  31. [31]

    2025 IEEE/ACM 40th International Conference on Automated Software Engineering , series =

    Zhensu Sun and Chengran Yang and Xiaoning Du and Zhou Yang and Li Li and David Lo , title =. 2025 IEEE/ACM 40th International Conference on Automated Software Engineering , series =. 2025 , doi =

  32. [32]

    2026 , eprint =

    Priyansh Trivedi and Olivier Schmitt , title =. 2026 , eprint =

  33. [33]

    2026 , eprint=

    Evaluating AGENTS.md: Are Repository-Level Context Files Helpful for Coding Agents? , author=. 2026 , eprint=

  34. [34]

    2026 , eprint =

    Lola Solovyeva and Fernando Castor , title =. 2026 , eprint =

  35. [35]

    Proceedings of the 48th IEEE/ACM International Conference on Software Engineering , series =

    Pengfei Gao and Chao Peng , title =. Proceedings of the 48th IEEE/ACM International Conference on Software Engineering , series =. 2026 , numpages =

  36. [36]

    Zhang and Mark Harman and Yiling Lou and Yang Liu and Zhenpeng Chen , title =

    Yaoqi Guo and Ying Xiao and Jie M. Zhang and Mark Harman and Yiling Lou and Yang Liu and Zhenpeng Chen , title =. Findings of the Association for Computational Linguistics: ACL 2026 , year =. 2601.05777 , archivePrefix =

  37. [37]

    2026 , eprint =

    Aman Sharma and Sushrut Thorat and Paras Chopra , title =. 2026 , eprint =

  38. [38]

    2025 IEEE/ACM 40th International Conference on Automated Software Engineering , series =

    Islem Bouzenia and Michael Pradel , title =. 2025 IEEE/ACM 40th International Conference on Automated Software Engineering , series =. 2025 , doi =

  39. [39]

    Proceedings of the ACM on Programming Languages , volume =

    Shuyang Liu and Yang Chen and Rahul Krishna and Saurabh Sinha and Jatin Ganhotra and Reyhaneh Jabbarvand , title =. Proceedings of the ACM on Programming Languages , volume =. 2026 , doi =

  40. [40]

    Advances in neural information processing systems , year=

    EffiBench-X: A Multi-Language Benchmark for Measuring Efficiency of LLM-Generated Code , author=. Advances in neural information processing systems , year=

  41. [41]

    2026 , eprint=

    GLM-5: from Vibe Coding to Agentic Engineering , author=. 2026 , eprint=