Pith. sign in

REVIEW 1 major objections 4 minor 12 references

Same Task, Different Work: Prompt-Induced Waste in Coding Agents

T0 review · 1 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Prompt wording, not task difficulty, dictates how much and what kind of work a coding agent performs, and requests for alternatives or certainty inflate that work without improving success.

desk verdict A careful preregistered study shows prompt wording changes where agent work goes, not just how much; the core result is solid, but the harness comparison has an undisclosed affiliation and the causal reading of token counts needs tempering. read the letter →

arxiv 2608.01347 v3 pith:I2ABPB5Q submitted 2026-08-02 cs.CL

Sarel Weinberger , Amir Hozez This is my paper · ORCID
classification cs.CL
keywords codingagentspromptengineeringreasoningtokenstoolcallsagentcostwastemechanismsharnessesbenchmark
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Two prompts can request the same code change, produce the same correct patch, and still cause an AI coding agent to perform radically different amounts and kinds of work; the paper's thesis is that prompt wording is therefore work design, deciding what the agent thinks through, what it executes, and when it stops. To establish this, the authors ran a preregistered benchmark of 4,644 valid runs on 24 deterministic coding tasks across several reasoning models and two production harnesses. Asking the agent to develop and compare several approaches multiplied provider-reported reasoning tokens by 2.4–7.4× while creating about three elaborated-but-discarded solution branches and no gain in success, whereas demanding certainty beyond doubt launched repeated re-verification, with the most redundant runs costing about 18× the clean-run median at flat success. The two waste mechanisms have different cost carriers — one lives in reasoning text, the other in tool execution — and a 'bounded efficiency' instruction preserved diagnosis and final validation while avoiding both. A sympathetic reader should care because this reframes prompt engineering as the design of the work itself: short phrases can be expensive not because they are long, but because they authorize work the benchmark does not reward.

What carries the argument

The machinery is a preregistered paired-prompt benchmark: 24 small deterministic coding tasks with hidden evaluators and 18 frozen prompt variants per task, constructed so that primary variants preserve the objective, acceptance criteria, and test command verbatim and only the added instruction changes. The primary outcome is the paired reasoning ratio — a run's provider-reported reasoning tokens divided by the median baseline reasoning in the same model–harness–task block — and a variant counts as wasteful only if the median ratio exceeds 1.5, the task-clustered 95% confidence interval lies above 1.1, success does not improve, and the effect appears on multiple tasks. The causal anatomy comes from condition-blind annotation of 2,801 reasoning traces, in which judges never see the prompt and every count must be backed by a verbatim evidence quote, joined with deterministic tool telemetry. The explanatory object that carries the argument is the cost-carrier split: branch tournaments are token-borne (cost rises while tool calls stay flat), while verification loops are tool-borne (cost, calls, and latency rise together), which is why the same instruction can be cheap in one harness and expensive in another.

What would settle it

Serve the same 24 tasks on a locally hosted model where total compute (FLOPs or sampling iterations) can be measured independently of billed reasoning tokens, and check whether the 'multiple approaches' prompt still shows 2.4–7.4× more compute with flat tool calls; if the inflation vanishes under complete accounting, the token-borne waste claim collapses into a billing artifact, and if it persists, the mechanism is confirmed. A second check: instrument the 'maximum certainty' runs with per-call CPU and wall-clock telemetry to see whether the 18× cost tier reflects genuinely more computation and test executions or merely expanded token accounting.

Watch

Extended reading notes

Core claim

The central claim is that, with the task, acceptance criteria, and test command held fixed, instruction wording determines what a coding agent computes before it finishes, so that two prompts can yield the same correct patch at very different cost. Asking the agent to develop several distinct approaches and compare their trade-offs multiplies reasoning tokens by 2.4–7.4× on every one of the six open-weight holdout models, creates roughly three elaborated but discarded solution branches and exactly one implemented solution, and yields no measured success improvement; this waste is token-borne because tool calls stay nearly flat. Asking to be absolutely certain and to re-verify until nothing can be wrong acts through a different pathway: repeated verification propagates into extra test runs, turns, latency, and context growth, and runs at the highest observed verification level cost about 18× the clean-run median with 2.5× the tool calls and 3× the wall-clock, again with flat success. The effects persist across a frozen holdout, paraphrases, a replication on a frugal reasoning model, and a first-party frontier-model study, while a 'bounded efficiency' instruction stays at or below baseline reasoning and preserves diagnosis and final validation.

Load-bearing premise

The load-bearing premise is that each provider's reported reasoning-token count faithfully measures the extra deliberation work a prompt caused, and that premise is fragile because the paper itself notes that provider-exposed reasoning text is an incomplete transcript of model computation.

Editorial extensions

If this is right

  • Instructions that request several approaches should be reserved for cases where the comparison is itself the deliverable; for an ordinary patch such prompts buy discarded deliberation and no success gain.
  • Certainty language should be replaced by an executable stop rule — run the relevant suite after the final edit, rerun only after a relevant change or failure, stop when the acceptance criteria pass — because 'be absolutely certain' has no observable stopping condition.
  • Cost control must target the cost carrier: token-borne tournaments need constraints on elaboration and on what is produced, while tool-borne verification loops need stopping policies and duplicate-action guards.
  • Harness design can dominate the user-prompt effect, with a 12–15× larger system prefix, 2–7× more turns, and 5–30× higher cost per success, so agent benchmarks must measure harness and model behavior separately.
  • Provider-side caching rebates 61–75% of the bill without changing agent behavior, so cost reporting should state both actual billed cost and estimated no-cache cost; the behavioral claims do not depend on which number is quoted.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: the instruction-to-work mapping probably generalizes to non-coding agents, since a request to 'be thorough' or 'consider all options' in document drafting or data analysis authorizes the same kind of elaborated-but-discarded deliberation; a direct test would rerun this paired-prompt design on non-coding agent tasks.
  • Editorial inference: token-borne waste is visible on the invoice while tool-borne waste hides in latency and call counts, so the two mechanisms differ in how quickly a user would notice and fix them — a prediction that could be tested by surveying which waste types practitioners detect.
  • Editorial inference: the authors' small-task ceiling suggests the headline ratios may not transfer to repository-scale or architectural work where alternatives require genuine implementation and evaluation; the natural extension is the same design on larger tasks where exploration is itself valuable.
  • Editorial inference: the frugal-model replication shows that relative inflation (16.6×) can mislead when the baseline is near zero, so the paper's practice of reporting absolute and relative spend together could become a standard convention for agent cost benchmarks.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

1 major / 4 minor

Summary. The paper presents a preregistered, paired benchmark of 24 deterministic coding tasks run under 18 prompt variants across six open-weight reasoning models and two agent harnesses, later extended to Kimi-K3 and claude-sonnet-5, totaling 4,644 valid runs and 2,801 condition-blind annotated reasoning traces. The central claim is that prompt wording changes not just the amount but the kind of work an agent performs: instructions to develop multiple approaches inflate provider-reported reasoning tokens by 2.4–7.4x (16.6x for Kimi-K3) and create discarded solution branches without success gains; certainty language propagates into repeated tool calls and longer wall-clock times; and harness design amplifies both effects. A bounded-efficiency instruction is neutral or better. The authors present evidence from a frozen holdout, paraphrase tests, a post-registration replication, and a first-party Sonnet 5 study.

Significance. If the findings hold, the paper makes a useful contribution by reframing prompt engineering as work design for agents, with a clear distinction between token-borne and tool-borne cost mechanisms. The study has strong internal validity: preregistered hypotheses and classification thresholds, a frozen holdout, condition-blind annotation with mandatory evidence quotes, deterministic hidden-test evaluators, dual-side wire capture for billing validation, and a public repository with SHA-256-verified prompts and run ledgers. The replication across models, harnesses, and prompt families strengthens the scope claims. The main weakness is that the headline reasoning-token ratios rest on provider-reported counts that are not independently validated; the paper is appropriately cautious about this in the limitations, but the quantitative magnitude claims are load-bearing.

major comments (1)
  1. [§3.1, Table 3, Appendix C.4] The paper's most quantitative claims for the multiple-approaches family — the 2.4–7.4x holdout ratio, the 16.6x Kimi-K3 ratio, and the token-borne classification — rely entirely on provider-reported reasoning tokens as the measure of added work. The manuscript acknowledges (Section 8; Appendix D.1) that provider-exposed reasoning is incomplete and not necessarily faithful, and it excludes claude-sonnet-5 from trace-level claims for exactly this reason, but it never provides an independent cross-check for the open models. Because the tool-layer effects for multiple approaches are weak (directional, p≈0.06, zero edit delta), the entire measured cost increase is attributed to reasoning tokens. If condition-specific token inflation is present (e.g., the instruction induces more verbalization per unit of actual deliberation), the effect sizes would overstate the difference in computation. I request a robustness analysis that compares reasoning-token ratios against a deterministic measure such as wall-clock latency, time-to-first-edit, or per-token generation time, or that bounds the plausible stylistic component. At minimum, report latency and no-cache-cost ratios for multiple approaches alongside the token ratios.
minor comments (4)
  1. [Abstract and §3.2] The abstract states 'Runs with high redundant verification cost 18x the clean-run median' without the descriptive caveat that the paper correctly applies in Section 3.2 ('This is a descriptive comparison across observed mechanism levels'). Please add 'in observed runs' or a footnote to the abstract and the first mention to avoid over-reading.
  2. [§5, Paraphrases and repetition] The main text says 'All four paraphrases of multiple approaches, deep thinking, and maximum certainty elevate reasoning on every model tested' without specifying that this is three models; Appendix E.3 provides the details. Please state the model count in the main text for clarity.
  3. [Appendix F.4] The 'missing criteria' variant is listed in the stress family but explicitly documented as 'no executed run of this variant on this task – rendered for documentation.' Since it is not part of the empirical results, consider moving it to a separate 'unexecuted templates' subsection to avoid confusion.
  4. [Acknowledgments] The acknowledgment that Claude operated as an autonomous research assistant is transparent, but for reproducibility it would be helpful to clarify which stages (e.g., annotation, analysis) were AI-assisted versus author-run, since this bears on the auditability of the pipelines.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: the empirical comparisons are anchored to external hidden-test success and deterministic tool telemetry; the main caveat is that provider-reported reasoning tokens are treated as the operational measure of deliberation work, which is a measurement-validity limitation rather than a circular derivation.

full rationale

The paper's central claims are empirical and externally anchored rather than definitionally forced. Prompt conditions are randomized within paired blocks with byte-exact frozen wording; success is measured by hidden deterministic evaluators that never appear in the workspace, so 'no success gain' is an external check rather than a consequence of how waste was classified. The waste-classification thresholds (median ratio >1.5, CI lower bound above 1.1, no material success gain, multiple-task requirement) were preregistered before results were inspected, and the frozen holdout and post-registration replications (Kimi-K3, claude-sonnet-5, paraphrases) provide out-of-sample checks. The token-borne versus tool-borne distinction is derived from jointly observing reasoning-token ratios, tool-call counts, turns, latency, and cost; it is a data-driven categorization, not an equation that reduces to its own input. The branch-tournament result (roughly three elaborated but discarded branches, exactly one implemented) is admittedly close to what the multiple-approaches instruction literally requests, but the paper treats this as an empirical verification of the instruction's effect, and the accompanying 2.4-7.4x reasoning ratios and flat success come from independent measurement, not from the wording alone. The bounded-efficiency result is a design-confirmation (the condition instructs efficient behavior and then is observed to avoid the measured waste mechanisms), but the preservation of diagnosis and final validation is measured under condition-blind annotation and deterministic redundancy metrics, so it is not tautological. The acknowledged limitation that provider-exposed reasoning is not a complete transcript of model computation is stated explicitly, claude-sonnet-5 is excluded from trace-level claims, and the 18x verification comparison is correctly labeled descriptive rather than causal because annotation levels are observed mediators. There is no load-bearing self-citation: reference [7] is an external paper on CoT incompleteness, and the authors' own repository is a data/reproducibility artifact rather than a justification of the claims. The only concern that could approach circularity is treating provider-reported reasoning tokens as a faithful measure of deliberation work; if token inflation were condition-specific verbal style rather than added computation, the headline ratios would overstate the effect.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The central claims rest on four domain assumptions about measurement validity and generalization. No free parameters are fitted, and the paper introduces no new physical or conceptual entities beyond the descriptive labels 'branch tournaments' and 'verification loops', which are not claimed to be independent mechanisms with external handles.

assumptions (4)
  • domain assumption Provider-reported reasoning-token counts are treated as a faithful measure of deliberation work.
    The primary outcome is a paired reasoning ratio; the paper acknowledges that visible reasoning is incomplete and excludes claude-sonnet-5 from trace claims, but the headline 2.4-7.4x ratios rely on these counts.
  • domain assumption The 24 deterministic tasks, each at most four files with high success ceilings, generalize to real coding-agent use.
    The paper itself lists this limitation: 'Multiple approaches may be useful on repository-scale or architectural work.' The claim 'waste' is therefore scoped to these small tasks.
  • domain assumption The condition-blind semantic judge reliably identifies waste mechanisms with acceptable inter-annotator agreement.
    The authors report kappa=0.28 overall but 0.55-0.68 on hypothesis-critical fields, and they rely on deterministic proxies for some counts. This assumption underlies the mechanism-level claims (3.5 branches, redundant verification).
  • domain assumption The two harnesses (PI.DEV and Claude Code) differ only in the measured design factors relevant to the cost gap.
    The 5-30x cost difference is attributed to prefix size, turns, and tool composition without an experimental manipulation; unmeasured defaults or engineering quality could contribute.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Same Task, Different Work: Prompt-Induced Waste in Coding Agents." pith.science (2026). https://pith.science/paper/I2ABPB5Q

@misc{pith2026260801347,
  author       = {Pith},
  title        = {Pith review of: Same Task, Different Work: Prompt-Induced Waste in Coding Agents},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/I2ABPB5Q}},
  note         = {Machine review of arXiv:2608.01347}
}
read the original abstract

Two prompts can request the same code change and produce the same correct patch, yet cause a coding agent to perform radically different kinds and amounts of work. We study this effect in a preregistered benchmark spanning 4,644 valid runs, 24 deterministic coding tasks, seven reasoning models, and two real agent harnesses. The central finding is that prompt wording does not merely scale total effort; it changes where that effort is spent. Multiple approaches and deep thinking primarily inflate reasoning. Multiple approaches increases reasoning by 2.4x to 7.4x across all six open models and creates about three elaborated but discarded solution branches, while still yielding only one implemented solution and no success gain. Maximum certainty activates a different pathway: repeated verification propagates into extra test runs, tool calls, turns, latency, and context growth. Runs with high redundant verification cost 18x the clean-run median, execute 2.5x more tool calls, and take 3x longer, again without a success gradient. These mechanisms therefore have distinct cost carriers: some prompts are reasoning-heavy and token-borne, while others are tool-heavy and system-borne. Harness design amplifies both effects and changes cost per successful task by 5x to 30x in our setting. The findings survive a frozen holdout, paraphrase tests, a Kimi-K3 replication, and a first-party Claude Sonnet 5 study. In contrast, bounded-efficiency wording preserves diagnosis and final validation while avoiding the measured waste mechanisms. Prompt engineering for coding agents is therefore work design: it determines what the agent thinks through, what it executes, and when it stops.

Figures

Figures reproduced from arXiv: 2608.01347 by the authors.

Figure 1
Figure 1. Cost carriers by mechanism (generated from [PITH_FULL_IMAGE:figures/full_fig_p010_1.png] view at source ↗
Figure 1
Figure 1. Different mechanisms have different cost carriers. Left: the first discarded solution branch [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Baseline tool-call composition (generated from [PITH_FULL_IMAGE:figures/full_fig_p011_2.png] view at source ↗
Figures from the paper (1 more)
Figure 2
Figure 2. Figure 2: Baseline tool-call composition. Claude Code is verification-heavy; PI.DEV is inspection [PITH_FULL_IMAGE:figures/full_fig_p005_2.png]

Review history (3 revisions) →

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

12 extracted references · 1 canonical work pages

  1. [1]

    https://github.com/PointFiveLabs/ prompt-efficiency-benchmark, 2026

    PointFive Labs.prompt-efficiency-benchmark: benchmark code, task fixtures, prompt variants, raw ledgers, and analysis. https://github.com/PointFiveLabs/ prompt-efficiency-benchmark, 2026

    2026

  2. [2]

    Earendil Works.Pi: an AI agent toolkit and coding-agent CLI.https://pi.dev, 2026

    2026

  3. [3]

    Anthropic.Claude Code.https://claude.com/claude-code, 2026

    2026

  4. [4]

    Together AI.Inference platform and model catalog.https://www.together.ai, 2026. 7

    work page 2026

  5. [5]

    BerriAI.LiteLLM proxy (v1.93.0).https://github.com/BerriAI/litellm, 2026

    2026

  6. [6]

    Moonshot AI.Kimi K3 technical blog.https://www.kimi.com/blog/kimi-k3, 2026

    2026

  7. [7]

    Baherwani, T

    V. Baherwani, T. Goldstein, and A. Panda.Not All LLM Reasoning Is Visible in the Chain-of- Thought. arXiv:2607.22925, 2026

    arXiv 2026

  8. [8]

    C.-E. Sun, L. Liu, G. Yan, Z. Wang, and T.-W. Weng.LLM Agents Already Know When to Call Tools — Even Without Reasoning(When2Tool). arXiv:2605.09252, 2026

    arXiv 2026

Show all 12 references
  1. [9]

    J. Liu, C. Qian, Z. Su, Q. Zong, S. Huang, B. He, and Y. R. Fung.CostBench: Evaluating Multi- Turn Cost-Optimal Planning and Adaptation in Dynamic Environments for LLM Tool-Use Agents. arXiv:2511.02734, 2025

    2025 arXiv

  2. [10]

    D. Wu, J. Wang, Y. Meng, Y. Zhang, L. Sun, and Z. Wang.CATP-LLM: Empowering Large Language Models for Cost-Aware Tool Planning. arXiv:2411.16313, 2024

    2024 arXiv

  3. [11]

    X. Yang, L. Li, H. Zhou, T. Zhu, X. Qu, Y. Fan, Q. Wei, R. Ye, L. Kang, Y. Qin, D. Liu, Q. Li, N. Ding, S. Chen, and J. Shao.Toward Efficient Agents: Memory, Tool Learning, and Planning. arXiv:2601.14192, 2026

    2026 arXiv

  4. [12]

    Post-success

    M. Zheng, D. O’Brien, S. Cui, P. Pashakhanloo, R. Mukherjee, M. Kim, and S. Kuhar.COR VUS: Context Optimization and Reduction Via Underlying Synchronization for LLM Coding Agents. arXiv:2607.22711, 2026. A Experimental Design and Preregistration A.1 Preregistration Hypotheses ...

    2026 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.