{"id":"8aa959cb-88ed-4212-9db2-5440ebd3a9a3","arxiv_id":"2608.13547","paper_version":1,"verdict":"ACCEPT","confidence":"MODERATE","novelty_score":8.0,"correctness_risk":"low","formal_verification":"none","parameter_count":1,"one_line_summary":"Matched execution scores can hide the difference between command-generation errors and transport damage, as QuoteBench's fixed-reply replay across eight model configurations demonstrates.","lead":"The paper introduces QuoteBench, a 56-task benchmark that separates errors in a model's generated Bash command from errors introduced when the command is repackaged and reparsed. It shows that a matched execution score can hide large opposing effects, and that the deployment configuration can reorder model rankings.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Validator completeness is the only residual risk, but the large effect sizes and the escaping control make it unlikely to overturn the central claim; verdict unchanged.","rationale":"The reader's weakest assumption — validator completeness — is indeed the most plausible soft spot, aligning with the paper's own stated limitation. I agree with the reader's identification. However, I judge this concern as adequately mitigated by the paper's multi-layered validation: the mutation audit, oracle solvability checks, real-ssh grounding, and the correct-escaping control. The large effect sizes and consistent direction across all eight configurations and both userlands make it very unlikely that a validator blind spot could reverse the central finding that matched scores can hide transport damage and compensation. The paper is careful to describe the compensation as a realized contrast over stored generations, not as a universal causal effect, and the limitations section appropriately scopes the claims. I found no internal inconsistency in the decomposition, controls, or statistical treatment. Therefore the reader's ACCEPT verdict should stand, with the residual validator-completeness risk as a minor caveat rather than a reason to change the verdict.","tokens_in":28943,"tokens_out":10463,"duration_ms":118572,"concrete_test":"Independently re-implement all 56 final-state validators from the task specifications alone, without consulting the released validator code, and re-score all stored replies for the eight same-window configurations (448 raw and 448 disclosed-boundary replies across the four cells). If any model's damage (RN−RR) or compensation (NN−RN) shifts by more than 3 percentage points relative to Table 4, the headline numbers are validator-sensitive and the paper should be revised to report the discrepancy.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim — that matched execution scores can hide a large transport effect — depends on exact final-state scores for all 56 tasks across the four crossed cells. If any validator falsely accepts or falsely rejects on one transport, the damage and compensation numbers would shift. The paper acknowledges this: Section 3.1 says the mutation audit 'covers the enumerated invalid states' and that 'Other validator blind spots may remain,' and Appendix F.1 reiterates that 'Unenumerated false positives remain possible.' This is the weakest load-bearing premise. However, the risk is substantially mitigated: the 197/197 mutation rejections, the oracle solvability controls, the real-ssh grounding, and the escape control in Section 4.2 (correct escaping reproduces the raw-path outcome for all 448 public pairs) together indicate the transport effect is real and not a validator artifact. The effect sizes (55.4–73.2 pp damage) are far larger than any plausible validator error rate given the checks performed. To overturn the qualitative conclusion, a blind spot would need to systematically remove most of the measured damage or compensation across multiple families and configurations — an unlikely scenario. No internal inconsistency or circular step was found.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"QuoteBench is a 56-task, one-shot Bash-command benchmark built from 14 incident-derived operation families, each with one benign control and three hazardous payload variants, scored by exact final-state validators. The paper's central contribution is a crossed design: for each of eight same-window model configurations it separates the generation contract (raw versus a disclosed-boundary contract that tells the model its reply will be interpolated inside double quotes) from the execution transport (raw versus a nested double-quoted parser), and replays stored replies across the four cells without additional model calls. The key empirical finding is that moving a fixed raw reply through the added parser lowers success by 55.4–73.2 percentage points in every configuration, while the disclosed-boundary contract recovers 30.4–60.7 points for six configurations; the matched gap is thus an identity sum of two large opposing components (Eq. (1)), illustrated by GPT-5.6-sol's -3.6 = -64.3 + 60.7. Robustness evidence includes a 197/197 validator mutation audit, oracle and naive-probe checks, a correct-escaping control that reproduces every raw-path outcome for all 448 public replay pairs, real-ssh grounding (seven of eight configurations match to the decimal), leave-one-family-out ranges with Holm-adjusted family-sign tests, three-draw repetitions, and a private 42-payload crossover.","tokens_in":29112,"tokens_out":24217,"duration_ms":215536,"significance":"Assuming the measurements are correct, this is a significant evaluation-methodology result. It provides a rare controlled decomposition of a matched score into transport damage and contract-conditioned compensation, with a fixed-reply design that avoids confounds from regeneration. The finding that raw-path scores are near-saturated at the frontier while nested-path scores separate models, and that one model pair reverses under the two paths, gives concrete evidence that leaderboard comparisons of command-issuing agents are path-dependent. The paper is unusually careful: the decomposition is an identity and is labeled as such; the primary mechanism analysis is restricted to same-window configurations; the 'compensation' is described as a realized contrast over stored generations, not a fitted parameter; and the limitations (purposive family set, observational native-tool campaign, single stored generations on effort ladders, unenumerated validator blind spots) are stated explicitly in Section 3.1, Appendix F.1, and the Limitations section. The release of the harness, exact prompts, SHA-256 manifest, and sanitized rollout archive supports reproducibility.","major_comments":[],"minor_comments":[{"comment":"Please fix the typographical spacing errors, including 'QuoteBenchmeasures' in the abstract and 'writtenpoints' in Section 4.2, and clean up the missing spaces around minus signs and percent symbols throughout (for example, '-64.3damage', '+60 .7compensation', and '-3.6points').","section":"Abstract / Section 4.2"},{"comment":"The sentence 'The realized generation-by-transport interaction ... ranges from -7.1 to +119.6 points across the eight configurations (Table 4)' cites Table 4, but Table 4 does not report the interaction column; either add that column or cite Appendix Table 14 instead.","section":"Section 4.2 / Table 4"},{"comment":"The family-level compensation values quoted in prose (json-write +50.0, sed-replace +46.9, find-glob -12.5, grep-count +15.6, hostile-filenames +18.8) are not tabulated anywhere in the manuscript; please add a family-level compensation table or point to a specific artifact table so the claim that compensation concentrates where the hazard is explicit can be checked.","section":"Section 4.3"},{"comment":"Because the headline damage and compensation numbers depend on exact final-state labels, I suggest adding one sentence stating what a hypothetical validator blind spot would need to look like (systematically transport-correlated and large enough to remove most of the 55.4-73.2 pp damage) to threaten the conclusion; the existing controls already make that scenario implausible, but the reader should not have to assemble the argument from multiple sections.","section":"Section 3.1 / Appendix F.1"},{"comment":"The sentence 'Raw generation itself is close to saturated at the frontier' should be scoped to 'the six frontier configurations', because Table 4's same-window raw scores range from 78.6% to 100% across all eight configurations, and the lower-scoring configurations still show substantial raw-path spread.","section":"Section 4.1"},{"comment":"Appendix A.2 gives verbatim prompts for the raw and disclosed-boundary contracts but not for the native contract; please include the native-tool prompt or explicitly state that it is available in the artifact, for full reproducibility.","section":"Appendix A.2"}],"recommendation":"minor_revision","confidential_remarks":"This is a well-executed measurement paper. The central claim is sound and the residual validator-completeness concern is, in my view, adequately mitigated by the reported controls. I recommend minor revision for the presentation and reproducibility issues listed above; no further scientific review should be needed."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things you should know. First, the central empirical claim survives reading: replaying identical stored replies through one added double-quoted parser destroys 55–73 points of pass rate in every same-window configuration, and the decomposition of a near-zero matched gap into large damage plus large compensation is not a validator artifact, as far as I can tell. Second, the paper does something rare in benchmark work: it fixes the model output and varies a single parser, so the attribution is actually identified rather than assumed.\n\nWhat is new is the fixed-reply two-axis crossover and the decomposition identity YNN - YRR = (YRN - YRR) + (YNN - YRN). The identity is trivial, but the empirical content is not: the two components are large, opposite, and stable across eight configurations, real-ssh grounding, private payloads, repeated draws, and both userlands. The validator audit (197/197 mutations rejected, oracle and benign-probe checks) is thorough. The escaping control, where correct interpolation reproduces every raw-path outcome for all 448 public pairs, is the right falsification probe. The paper also deserves credit for reporting its own limitations: single stored generations on effort ladders, non-comparable compute budgets across providers, and possible validator blind spots.\n\nSoft spots are real but proportionate. Validator completeness is the load-bearing premise; the paper concedes in Section 3.1 and Appendix F.1 that unenumerated false positives remain possible. I do not think that undermines the qualitative conclusion — the effect sizes are far larger than any plausible validator error given the checks — but absolute pass rates and the exact decomposition numbers should be treated as benchmark-specific. The \"one unambiguous reversal plus four single-task margins\" is honestly labeled, but it means the leaderboard reordering is not a headline result on its own. The private payloads are withheld, which is the right call for contamination, but it does put some replication claims out of reach for now.\n\nWho gets value: anyone designing or using agent evaluation harnesses, and people building command-issuing agents where the execution path is not under their control. This is an experimental measurement, not a theory, and the paper is clear about that.\n\nI would send it to peer review. The central claim is well-supported, the method is a genuine contribution, and the weaknesses are disclosed rather than hidden. I would not desk reject.","headline":"The fixed-reply crossover design actually delivers: matched scores here demonstrably hide a large transport effect, and the paper is unusually honest about where it could still be wrong.","tokens_in":29657,"tokens_out":1608,"would_cite":true,"duration_ms":18009,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Matched execution scores can hide command-path failures: replaying the same stored reply through one added parser costs 55.4–73.2 points, and disclosing the boundary recovers 30.4–60.7, so a near-zero matched gap can mask two large…","keywords":["Bash quoting","command-path failures","matched execution scores","transport damage","contract-conditioned compensation","LLM coding agents","final-state validation","nested transport"],"falsifier":"Re-run the 448 public fixed-reply pairs with correct escaping at the interpolation point: the paper's claim predicts that every raw-path success is restored. A single pair in which correct escaping does not restore the raw-path outcome, or a validator false accept or reject found by additional mutation classes, would break the attribution of the loss to the unescaped parser.","tokens_in":28715,"feed_emoji":"💻","tokens_out":8872,"duration_ms":86175,"temperature":0.7,"pith_summary":"QuoteBench isolates a failure mode that ordinary leaderboard scores hide: a command that is correct on the direct shell path can fail when an interface layer wraps, quotes, or re-parses it. Across eight fixed model configurations, replaying the same stored command through one extra double-quoted parser lowers success by 55.4 to 73.2 percentage points, and telling the model about that boundary recovers 30.4 to 60.7 points for six of the eight. Because these two effects are large and opposite, a matched score can look nearly unchanged; GPT-5.6-sol's gap of $-3.6$ points is the sum of $-64.3$ points of transport damage and $+60.7$ points of compensation. If the paper is right, reporting only matched execution scores is insufficient: the command path, generation contract, and validator are part of the measured system, and model rankings can change when the deployment path changes.","feed_headline":"Matched scores hide 55–73-point command-path failures","feed_subtitle":"Replaying the same command through one added quote parser loses up to 73 points, then disclosure recovers up to 61.","key_machinery":"The crossed $2\\times2$ design with cells $RR$, $RN$, $NR$, $NN$, where generation contract precedes transport, together with the identity $Y_{NN}-Y_{RR} = (Y_{RN}-Y_{RR}) + (Y_{NN}-Y_{RN})$. The \"nested transport\" is one deliberately unescaped double-quoted parser added to the command path, standing in for remote, container, or CI wrappers. Fixed-reply replay keeps the stored command constant so any change between $RR$ and $RN$ is attributable to the parser, and exact final-state validators score only the resulting files, argument vectors, JSON, directory state, or Git history. Escaping at the interpolation point reproduces raw-path outcomes, closing the attribution loop by showing the damage is removed when the boundary is handled correctly.","core_discovery":"The central discovery is an empirical decomposition: matched score equals transport damage plus contract-conditioned compensation, where damage is fixed-reply transport loss and compensation is the recovery from telling the model about the boundary. Using 56 tasks and exact final-state validation, the paper crosses generation contract (raw versus disclosed-boundary) with execution transport (raw versus a nested double-quoted parser). Fixed-reply replay, comparing $RN$ against $RR$, shows a loss in every configuration; the disclosed-boundary contract, comparing $NN$ against $RN$, recovers most of it in six configurations. Correct escaping at the interpolation point reproduces the raw-path outcome for all 448 public pairs, so the loss is attributable to the unescaped transform rather than to model error. The paper concludes that matched scores should be reported alongside the full command path, and that raw generation is nearly saturated at the frontier while boundary adaptation is what still separates models.","pith_inferences":["A practical diagnostic follows from the masking threshold: for any model with a small matched gap, report the two off-diagonal cells, since a near-zero gap is exactly the case where damage and compensation may both be large.","The same four-cell decomposition should transfer to any pipeline that transforms generated output before execution, so matched scores in tool-use, structured-action, and serialization benchmarks may carry the same ambiguity even where the paper does not test them.","Because validator completeness is the load-bearing premise, adding new mutation classes or independent final-state checkers on the private payloads is a testable way to check whether the absolute damage and compensation numbers shift while the sign pattern persists.","A further testable extension is to make the frozen 56-task core a regression audit: models that rank high on matched nested scores should also be required to show positive compensation and small transport damage, not just a high diagonal cell."],"forward_implications":["A matched execution score alone cannot tell whether a model generated a wrong command or a correct command that was mangled by the execution path.","Model leaderboards are path-dependent: one unambiguous reversal and four single-task-margin reversals appear when comparing raw and nested orderings, so rankings should be reported together with the deployment configuration.","Effort-ladder gains in matched score can be contract-conditioned compensation rather than improved cross-path portability; Opus-4.8's matched gap moves from $-48.2$ to $-3.6$ points while its nested-replay pass rate stays near its starting level.","Correct escaping at the interpolation point, or a temporary-script transport, restores every raw-path success in the public replays, so the transport damage is removable at the harness side.","Evaluators of command-issuing agents should report the generation contract, execution transport, operating point, and final-state validator, because raw generation is near saturation at the frontier and the remaining signal is boundary adaptation."],"supporting_citations":[{"why":"Defines Bash quoting and parser rules, the ground truth the final-state validators check and the basis for the nested double-quoted transport.","marker":"Free Software Foundation, 2025"},{"why":"Documents hostile-filename and quoting hazards that motivate the 14 operation families and their hazardous payload tiers.","marker":"Wheeler, 2010"},{"why":"The static-analysis baseline that flags only 34.6 percent of nested-only failures, supporting the claim that the fault lives in downstream interpolation rather than command syntax.","marker":"Holen, 2012"},{"why":"The command-generation benchmark tradition that scores emitted programs under fixed transport, which QuoteBench isolates and extends.","marker":"Lin et al., 2018"},{"why":"The competition-style translation benchmark supplying the same fixed-transport scoring baseline for generated commands.","marker":"Agarwal et al., 2021"},{"why":"Shows that the agent-computer interface changes coding performance, the boundary QuoteBench measures directly.","marker":"Yang et al., 2024"},{"why":"Shows prompt-format sensitivity moves scores with a nuisance channel but cannot separate what the channel destroyed from what the model produced; fixed-output replay is the method that makes that separation possible.","marker":"Sclar et al., 2024"},{"why":"Demonstrates that permissive validators can accept incorrect agent patches, motivating QuoteBench's mutation audit of every final-state validator.","marker":"Yu et al., 2025"},{"why":"Concurrent harness-variance work reorders leaderboards by swapping the whole scaffold; QuoteBench fixes the model output and changes one parser to attribute a reversal to the command path.","marker":"Zhang et al., 2026b"}],"fun_headline_variants":["Matched scores hide 55–73 pt loss from one reparse","One added parser costs up to 73 points that matched scores miss","Replay same command: 55–73 pt drop, disclosure recovers up to 61","Matched gap -3.6 hides -64.3 damage and +60.7 recovery","Raw generation nearly saturated; boundary adaptation separates models"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the final-state validators and the execution harness correctly classify success on both the raw and nested paths for all 56 tasks; the paper acknowledges that other validator blind spots may remain, and a wrong accept or reject would shift the damage and compensation numbers.","fun_headline_variants_meta":{"raw":{"variants":["Matched scores hide 55–73 pt loss from one reparse","One added parser costs up to 73 points that matched scores miss","Replay same command: 55–73 pt drop, disclosure recovers up to 61","Matched gap -3.6 hides -64.3 damage and +60.7 recovery","Raw generation nearly saturated; boundary adaptation separates models"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000759,"raw_usage":{"total_tokens":3391,"prompt_tokens":986,"completion_tokens":2405,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":602,"completion_tokens_details":{"reasoning_tokens":2305}},"tokens_in":602,"tokens_out":2405,"duration_ms":17972,"temperature":1.0,"reasoning_tokens":2305,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T04:24:33.553928+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the 448 public fixed-reply pairs with correct escaping at the interpolation point: the paper's claim predicts that every raw-path success is restored. A single pair in which correct escaping does not restore the raw-path outcome, or a validator false accept or reject found by additional mutation classes, would break the attribution of the loss to the unescaped parser.","supporting_citations":[],"review_version":1}