{"id":"40fb19d0-134d-4682-8c89-fce76348db37","arxiv_id":"2607.22585","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Across two coding models and three harnesses, harness choice changed tokens per solved task by up to 40x while pass rates moved only 0-8 percentage points, with harness failure fingerprints stable across models.","lead":"This paper measured how much the harness — the software scaffolding that wraps a coding AI — changes cost and behavior when the AI model itself is held fixed. On a 50-task benchmark, harness choice moved token cost per solved task by up to 40x while pass rates barely moved, arguing that leaders should compare harness-model pairs rather than model names alone.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Table 2 and Table 4 solved counts disagree in 4 of 6 cells, contradicting the abstract's 0–8 pp pass-rate claim","rationale":"The reader's weakest_assumption was token-accounting comparability, which is a legitimate measurement concern. However, the most load-bearing issue is the internal inconsistency between Table 2 and Table 4: it is a concrete, observable discrepancy that undermines the abstract's specific '0–8 pp' claim, rather than a hypothetical accounting difference. The reader did note this inconsistency in the rationale but did not make it the weakest assumption. The cost-side finding (40× token gap) is supported by disjoint bootstrap CIs and the OpenCode turn-budget sensitivity check, so the qualitative conclusion that harness-model pairs matter remains credible. The paper deserves a conditional accept pending correction of the tables; the verdict should remain CONDITIONAL, hence UNCHANGED.","tokens_in":8696,"tokens_out":6108,"duration_ms":60565,"concrete_test":"Download the released raw trial logs from the anonymous repository and count solved=True per (harness, model) cell. Compare these counts to Table 2's percentages and Table 4's Sv column. If Table 4 is correct, the abstract's '0–8 pp' range must be revised (largest gap becomes ~10 pp) and the bootstrap CI statement re-run; if Table 2 is correct, Table 4 must be corrected. This check settles which table is trustworthy and directly tests the headline pass-rate claim.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's own data are internally inconsistent. Using the stated n=50 denominator, Table 2's pass rates imply solved counts of 24, 19, 25, 23, 25, 23 for (Goose-Qwen, Goose-MiniMax, OpenCode-Qwen, OpenCode-MiniMax, OpenHands-Qwen, OpenHands-MiniMax). Table 4 lists 23, 17, 25, 22, 25, 21 respectively. Thus 4 of 6 cells disagree by 1–2 solved tasks (2–4 percentage points). This is not a rounding artifact: 23/50 = 46%, not 48%, and 17/50 = 34%, not 38%. The abstract's '0–8 percentage points' range is contradicted by Table 4, which implies a largest within-model gap of 10 pp (Goose MiniMax 34% vs OpenCode MiniMax 44%). Since the headline claim explicitly includes this quantitative bound, the pass-rate portion of the central claim is not reliably supported by the paper's own evidence. The discrepancy also raises concerns about data-handling reliability more broadly, though the 40× token gap is large and bootstrap CIs are disjoint, so the cost-side finding is less immediately threatened.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper presents a controlled evaluation of two coding models (Qwen 3.6 Plus and MiniMax M2.5) across three open-source harnesses (Goose, OpenCode, OpenHands-SDK) on a stratified 50-task subset of Terminal-Bench Pro. It reports pass rates, tokens per solved task, turn counts, no-action turns, and a six-class failure taxonomy. The central empirical claims are: (i) harness choice induces up to a 40× difference in tokens per solved task while paired within-model pass-rate differences remain 0–8 percentage points; (ii) harness-specific failure fingerprints replicate across models; (iii) no-action turns constitute an overlooked per-task oversight and wait tax. The paper concludes that leaderboards should compare harness–model pairs, not model names alone, and should report token/latency/oversight metrics.","tokens_in":8984,"tokens_out":7584,"duration_ms":79022,"significance":"If the cost-side result holds, this is a genuinely useful contribution to coding-agent evaluation. The paper quantifies an understudied axis of variance—the scaffold—and makes a concrete, actionable recommendation: report harness–model pairs with token and latency budgets. Strengths include the fixed task-selection protocol, 300 trials, paired-task bootstrap CIs, the large and robust 40× token gap with disjoint confidence intervals, and the public release of anonymized configs, raw logs, and analysis scripts. The pass-rate bound, however, is not currently supported by the paper's own tables, and the token-accounting asymmetry deserves a sensitivity check before the quantitative headline can be accepted. The failure-fingerprint interpretation also needs to separate structurally enforced taxonomy differences from genuine behavioral differences.","major_comments":[{"comment":"The paper's headline pass-rate claim is internally inconsistent. Using the stated n=50 per cell, Table 2 pass rates imply solved counts of 24, 19, 25, 23, 25, 23 for (Goose-Qwen, Goose-MiniMax, OpenCode-Qwen, OpenCode-MiniMax, OpenHands-Qwen, OpenHands-MiniMax), while Table 4 lists 23, 17, 25, 22, 25, 21. Thus 4 of 6 cells disagree by 1–2 solved tasks (2–4 pp), and this is not rounding: 23/50 = 46%, not 48%; 17/50 = 34%, not 38%. Under Table 4, the largest within-model gap is 10 pp (Goose vs. OpenCode on MiniMax: 34% vs. 44%), contradicting the abstract's '0–8 percentage points' and the §4.1 statement that pairwise differences are 'at most −8.0 pp.' Since the pass-rate bound is part of the central claim and is used to argue that cost, not accuracy, is the differentiator, this inconsistency must be resolved. Please correct the counts/rates and re-run the paired bootstrap CIs; the released","section":"Tables 2 and 4; Abstract; §4.1"},{"comment":"The token-accounting asymmetry is a load-bearing threat to the 40× claim. OpenCode and OpenHands-SDK report ATIF totals of prompt + completion + cached tokens, while Goose reports a 'total-only' field with no per-direction split and no clarification of whether cached or input tokens are included. If Goose's total excludes certain token classes, part of the gap could be an accounting artifact rather than harness behavior. The paper acknowledges the asymmetry in Limitations, but because the abstract and §4.2 report a precise '40×' factor, the authors should validate comparability: e.g., a provider-billed token audit on a subset of trials, or a sensitivity analysis recomputing the ratio under worst-case assumptions about what Goose's total excludes. Without this, the quantitative magnitude of the central claim rests on an unverified equivalence of two distinct telemetry paths.","section":"§3.5, §6 (Limitations)"},{"comment":"OpenCode is not subject to the 40-turn cap that Goose and OpenHands-SDK receive, and the sensitivity analysis reports only the fraction of OpenCode trials below 40 turns (88% Qwen, 71% MiniMax), not the token contribution of the over-40-turn trials. Because tokens per solved task is amortized with failed-trial tokens in the numerator (§3.5), truncating 12–29% of OpenCode trials—even if they are in the failure tail—could remove a non-negligible share of OpenCode's total token consumption. Please report tokens per solved task for the subset of OpenCode trials with ≤40 turns, or provide a matched comparison in which Goose and OpenHands-SDK are also run without a turn cap. This would separate the effect of turn-budget policy from the per-turn context-growth effect that the paper claims drives the 40× gap.","section":"§3.4, §4.2"},{"comment":"The failure taxonomy is not symmetric across harnesses: the MAX_TURNS category is reachable only when a turn cap is passed, so OpenCode is structurally unable to produce it (Appendix A, rule 3). The paper acknowledges this, but then uses the 0/0 MAX_TURNS count as part of OpenCode's fingerprint and infers that 'runs that would manifest as MAX TURNS under Goose or OpenHands-SDK instead surface here as TIME or HANG.' That inference is untested. For the failure-fingerprint contribution, comparisons should either be restricted to categories that are definitionally available to all three harnesses, or supported by manual reclassification of the over-budget OpenCode trials into the taxonomy.","section":"§4.5, Appendix A"}],"minor_comments":[{"comment":"The sensitivity analysis reports OpenCode trial counts as 42/48 and 34/48; the denominator 48 rather than 50 should be explained (missing turn-count records?).","section":"§4.2"},{"comment":"The SYS category has only 2 tasks, so the '0% across all harnesses' result is based on 4 pooled trials; the text should caution against treating this as a robust failure.","section":"Table 5"},{"comment":"The Pareto plot would benefit from error bars or at least a note that the pass-rate axis spans only 30–50%, making the small pass-rate differences visually compressed.","section":"Figure 1"},{"comment":"The claim that the system prompt template was 'held largely identical' is vague. Since configs are released, please state which template components were identical and which were modified.","section":"§3.4"},{"comment":"The manual review of a 20% random sample is mentioned without detail; a brief statement of agreement or any disagreements would strengthen the reliability of the taxonomy.","section":"Appendix A"}],"recommendation":"major_revision","confidential_remarks":"The discrepancy between Tables 2 and 4 is the most urgent issue; given the released logs, it should be fixable and is not necessarily a fundamental design flaw. The token-accounting asymmetry and the turn-budget asymmetry also need explicit sensitivity analyses before the quantitative '40×' claim can be accepted as stated. If the authors can reconcile the counts and provide the requested robustness checks, the paper would be a useful contribution to coding-agent evaluation."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague — worth a look if you care about coding-agent evaluation. The paper does something simple and useful: fixes the model, varies the harness, and measures cost and failure modes on a stratified 50-task subset of Terminal-Bench Pro. The headline result — up to a 40x difference in tokens per solved task between Goose and OpenCode, with disjoint bootstrap CIs for both models — is real and well-argued. The no-action-turn oversight tax (10x) is a nice mechanistic proxy. Those findings survive scrutiny.\n\nWhat's genuinely new is the controlled design: two models × three harnesses × 50 tasks, with a clear attempt to keep prompts and evaluation consistent. That's more than the earlier Terminal-Bench 2.0 observation of a 65x token spread.\n\nNow the soft spots. The most serious is a hard internal inconsistency: Table 2 and Table 4 disagree on solved counts in 4 of 6 cells. Table 2 gives Goose-MiniMax 38% pass rate; Table 4 shows 17 solved out of 50 = 34%. The abstract's '0–8 pp' bound is contradicted by Table 4, which implies a 10 pp largest gap. This isn't rounding. It doesn't sink the cost-side finding, but it does destabilize the pass-rate claim and casts doubt on the authors' data pipeline. They need to fix the tables and recheck the raw logs.\n\nSecond, token accounting is not fully comparably instrumented: Goose reports a total-only field, while OpenCode and OpenHands-SDK report ATIF totals of prompt+completion+cached. If Goose's total excludes cached tokens or counts differently, a chunk of the 40x gap could be an accounting artifact. The paper flags this in Limitations, but it's central enough that a provider-bill cross-check should be required.\n\nThird, OpenCode runs without a turn cap while the other two have 40-turn caps. The sensitivity analysis (88% of OpenCode trials under 40 turns) is decent, but it would be cleaner to find a way to apply the same budget.\n\nThe fingerprint claim ('largely model-independent') is based on only two models; the authors acknowledge this, so it's a mild overstatement rather than a fatal flaw.\n\nBottom line: the central message — harness-model pairs, not model names, determine real deployment cost and oversight — is well supported. The quantitative details, especially the pass-rate bound, need a revision. I'd send this to peer review, not desk reject. A careful referee should ask for corrected tables and a token-accounting validation.","headline":"A mostly solid controlled study with a real internal table inconsistency; the 40x token-cost result is robust, the pass-rate bound is not.","tokens_in":9575,"tokens_out":2836,"would_cite":true,"duration_ms":26161,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Harness choice, not model choice, drives up to a 40× token-cost difference in coding-agent evaluation, while pass rates barely move.","keywords":["coding agents","harness","scaffold","evaluation methodology","token efficiency","pass rate","failure modes","human oversight"],"falsifier":"Cross-check harness-reported token totals against the API provider's billing records for a handful of trials; if the 40× ratio disappears, the central cost claim is an accounting artifact, while if it persists, the harness-behavior explanation is confirmed.","tokens_in":1223,"feed_emoji":"💸","tokens_out":1634,"duration_ms":58751,"temperature":0.7,"pith_summary":"This paper argues that public coding-agent leaderboards mislead because they rank models while treating the surrounding harness—the software that gives the model tools, manages context, and decides when to stop—as invisible. Holding two recent coding models fixed and varying three open-source harnesses on 50 coding tasks, the authors find that harness choice changes tokens per solved task by up to 40×, while paired pass-rate differences stay within 0–8 percentage points. Failure modes also cluster by harness and replicate across both models, meaning scaffold design, not model intelligence, determines how an agent fails and how much it costs to run. The authors conclude that evaluations should compare harness–model pairs and report token use, latency, and idle-turn counts alongside pass rate.","feed_headline":"Harness choice drives 40x token-cost gap in coding agents","feed_subtitle":"Pass rates barely move, but scaffold choice dominates cost and failure patterns. Model-only leaderboards mislead.","key_machinery":"The load-bearing object is the harness—the wrapper software that issues tools, manages context, and decides when to stop—operationalized by three measurement instruments: tokens per solved task (amortized over failed trials), average no-action turns per task (a proxy for human oversight burden), and a six-class failure taxonomy (REASON, VERIFY, TIME, MAX_TURNS, HANG, ERROR). The failure fingerprint, the per-harness distribution across these categories, is the mechanism that transfers across models and identifies scaffold-level biases.","core_discovery":"The central claim is that when the model is held fixed, the harness—not the model—determines the economically relevant behavior of a coding agent. Across 300 trials, tokens per solved task ranged from roughly 28,000 to 1.55 million, a 40× spread with identical ordering for both models, while pass rates differed by no more than 8 percentage points and most differences were within bootstrap confidence intervals around zero. Each harness exhibited a distinctive failure fingerprint—one stops cleanly when stuck, one persists with plausible-but-wrong solutions or exhausts its turn budget, one spins on idle turns until timeout—and these fingerprints replicated across both models. The authors interp","pith_inferences":["If harness telemetry paths count tokens differently (e.g., one excludes cached tokens or only counts prompt tokens), part of the 40× gap could be accounting rather than behavior; the paper flags this asymmetry in its limitations, and the claim could be checked against provider billing records.","At n=50, pass-rate confidence intervals are wide; a larger task set could reveal smaller or more nuanced pass-rate differences, and the paper's own category table hints that harness advantage may be task-type dependent (e.g., fresh-implementation tasks).","A practical selection protocol follows from the paper's framing: for a given model and budget, run a short stratified pilot across harnesses, measure tokens per solved task and idle-turn rate, and pick the pair on the Pareto frontier.","The failure-fingerprint concept could be repurposed as a regression diagnostic: a shift in a harness's failure mix after a software update would indicate scaffold-level change independent of model behavior."],"forward_implications":["Leaderboards that report model name and pass rate alone conflate two independent sources of variance; they should also report tokens per solved task, latency, no-action turns, and the full harness specification.","A deployer who selects a coding agent by pass rate alone can end up on a Pareto-dominated point, paying up to 40× more in tokens for no measurable accuracy gain.","Harness-specific failure fingerprints imply different oversight expectations: with some harnesses a user can trust the agent's done signal, while with others they must independently verify outputs.","No-action turns act as a per-task wait tax in interactive use, not just a token tax; reducing them improves human-in-the-loop efficiency.","The cost-side effect of harness choice (up to 40×) dwarfs the effect of a model upgrade (1.0–1.3×), so model-only comparisons without harness control are uninterpretable in deployment terms."],"fun_headline_variants":["Harness choice, not model, drives 40x coding-agent cost","Scaffold choice causes 40x token-cost gap in agents","Model leaderboards ignore the 40x harness tax","Pass rates flat, but harness spikes token use 40x","The hidden variable: harness choice in coding-agent tests"],"cache_read_input_tokens":10752,"weakest_assumption_plain":"The 40× gap rests on the assumption that the token totals reported by the three harnesses count the same cost quantity; if one harness's field excludes cached or input tokens while another includes them, part of the gap could be a measurement artifact.","fun_headline_variants_meta":{"raw":{"variants":["Harness choice, not model, drives 40x coding-agent cost","Scaffold choice causes 40x token-cost gap in agents","Model leaderboards ignore the 40x harness tax","Pass rates flat, but harness spikes token use 40x","The hidden variable: harness choice in coding-agent tests"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000747,"raw_usage":{"total_tokens":3197,"prompt_tokens":807,"completion_tokens":2390,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":551,"completion_tokens_details":{"reasoning_tokens":2305}},"tokens_in":551,"tokens_out":2390,"duration_ms":18095,"temperature":1.0,"reasoning_tokens":2305,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-02T11:55:32.760471+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Cross-check harness-reported token totals against the API provider's billing records for a handful of trials; if the 40× ratio disappears, the central cost claim is an accounting artifact, while if it persists, the harness-behavior explanation is confirmed.","supporting_citations":[],"review_version":1}