{"id":"5b914093-88bf-44bd-8d7a-dc0578d936b1","arxiv_id":"2606.08529","paper_version":1,"verdict":"UNVERDICTED","confidence":"LOW","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A controlled comparison shows scaffold choice alters GAIA Level 1-2 accuracy by up to 28 points, with effects varying by model family rather than capability tier alone.","lead":"This paper runs a pre-registered controlled test of three different scaffolds on five AI models using the GAIA benchmark. It finds that changing the scaffold can shift accuracy by up to 28 points on the same model and tasks.","discovery_kind":"extension","skeptic_critique":{"model":"grok-4.3","headline":"Scaffold implementations may contain unmeasured biases favoring structured designs","rationale":"The reader's weakest assumption directly identifies the same load-bearing risk; the abstract-only review correctly flags unverifiable implementation fairness as the primary uncertainty preventing a stronger verdict.","tokens_in":1856,"tokens_out":246,"duration_ms":9250,"concrete_test":"Extract the exact prompt templates and decision logic from the methods section for all three scaffolds; count lines of custom recovery code and measure prompt token length; if the multi-agent scaffolds exceed ReAct by >30% on either metric, re-implement ReAct with equivalent engineering effort and recompute the Opus Level-2 robust-slice gap.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim (scaffold choice alone produces up to 28pp gaps) requires that accuracy differences arise from architectural differences rather than from unequal engineering effort across scaffolds. The abstract describes ReAct, Planner-Actor-Rater, and planner-then-executor but supplies no information on whether prompt templates, tool-calling rules, or error-recovery logic were matched for specificity and length; any systematic advantage in the multi-agent versions would inflate the reported elicitation gap.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.3","summary":"The paper reports a pre-registered controlled experiment comparing three scaffolds (ReAct, Planner-Actor-Rater, planner-then-executor) on GAIA validation Levels 1 and 2 across five models. It claims scaffold choice alone produces accuracy differences up to 28 percentage points (Opus, Level 2, robust slice), that the elicitation gap does not reliably shrink with model capability, that model family rather than tier conditions the multi-agent advantage, and that structured scaffolds recover from errors more effectively while using fewer tool calls.","tokens_in":1933,"tokens_out":511,"duration_ms":8897,"significance":"If the controlled measurements hold after addressing implementation-equivalence concerns, the work supplies direct evidence that published agent scores are scaffold-conditional and quantifies the elicitation gap under fixed tasks. The pre-registration and fixed-task design are strengths that reduce post-hoc flexibility.","major_comments":[{"comment":"Abstract and methods: the 28pp claim and the rejection of the pre-registered model-capability hypothesis rest on the assumption that the three scaffolds were implemented with matched prompt specificity, tool-calling rules, and error-recovery logic. No section supplies the prompt templates, length statistics, or decision rules used for each scaffold, leaving open the possibility that engineering effort differences drive the observed gaps rather than architectural structure.","section":"Abstract / Methods"},{"comment":"Results (all slices): the abstract states that scaffold effects vary significantly by model and that tier-scaling holds only at Level 1 under the robust slice, yet supplies no error bars, exact sample sizes per cell, or statistical tests for the 28pp difference or the model-family interaction. These omissions are load-bearing for the central claim that scaffold variation produces gaps of at least 10 points.","section":"Results"},{"comment":"Discussion: the claim that structured scaffolds recover more often from mid-trajectory errors is presented without a breakdown of error types or recovery rates per scaffold-model pair; without this, it is unclear whether the advantage is architectural or an artifact of differing recovery heuristics.","section":"Discussion"}],"minor_comments":[{"comment":"The abstract refers to 'robust slice' without defining the exclusion criteria in the provided text; a brief definition or reference to the pre-registration would improve clarity.","section":"Abstract"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":{"model":"grok-4.3","summary":"We thank the referee for their thorough review and constructive suggestions. We address each of the major comments point-by-point below, indicating the revisions we will make to the manuscript.","responses":[{"response":"We agree that providing the prompt templates, length statistics, and decision rules is essential to substantiate that the observed differences arise from architectural structure rather than implementation details. In the revised manuscript, we will include these in a new appendix, along with a description of how prompt specificity and error-recovery logic were matched across scaffolds per the pre-registration protocol. This will allow readers to assess equivalence directly.","revision_made":"yes","referee_comment":"[Abstract / Methods] Abstract and methods: the 28pp claim and the rejection of the pre-registered model-capability hypothesis rest on the assumption that the three scaffolds were implemented with matched prompt specificity, tool-calling rules, and error-recovery logic. No section supplies the prompt templates, length statistics, or decision rules used for each scaffold, leaving open the possibility that engineering effort differences drive the observed gaps rather than architectural structure."},{"response":"We acknowledge the need for statistical rigor in presenting the results. The sample sizes are determined by the GAIA validation sets (53 questions at Level 1, 86 at Level 2) with three attempts per question. In the revision, we will add error bars to all reported accuracies, specify the exact n per cell, and include appropriate statistical tests (such as pairwise proportion tests or interaction terms in a logistic model) for the 28 percentage point difference and the model-family interactions. These additions will be made to the results section and figures.","revision_made":"yes","referee_comment":"[Results] Results (all slices): the abstract states that scaffold effects vary significantly by model and that tier-scaling holds only at Level 1 under the robust slice, yet supplies no error bars, exact sample sizes per cell, or statistical tests for the 28pp difference or the model-family interaction. These omissions are load-bearing for the central claim that scaffold variation produces gaps of at least 10 points."},{"response":"We will revise the discussion to include a detailed breakdown of error types (categorized as planning, execution, or recovery failures) and the corresponding recovery rates for each scaffold-model combination. This analysis supports the architectural advantage claim and will be presented in a new table to clarify that the differences are not due to heuristic variations alone.","revision_made":"yes","referee_comment":"[Discussion] Discussion: the claim that structured scaffolds recover more often from mid-trajectory errors is presented without a breakdown of error types or recovery rates per scaffold-model pair; without this, it is unclear whether the advantage is architectural or an artifact of differing recovery heuristics."}],"tokens_in":1564,"tokens_out":597,"duration_ms":20654,"standing_objections":[]},"desk_editor":{"model":"grok-4.3","letter":"The one or two things to know: scaffold choice alone moves measured accuracy by as much as 28 percentage points within a single model on GAIA level 2, and the expectation that more capable models would show smaller scaffold effects does not hold.\n\nThe paper runs a pre-registered comparison of ReAct, Planner-Actor-Rater, and planner-then-executor across five models on fixed GAIA validation tasks at levels 1 and 2. It reports concrete gap sizes, rejects the tier-scaling prediction at level 2, shows that model family rather than capability tier conditions the multi-agent advantage, and falsifies the planner-executor edge on file tasks. Structured scaffolds also make fewer tool calls while recovering from errors more often.\n\nThis controlled isolation of the elicitation gap is the main contribution. The design holds tasks and conditions fixed and tests specific predictions ahead of time, which gives the quantitative results more weight than typical single-scaffold reports.\n\nThe soft spot is that the abstract supplies no error bars, statistical tests, or implementation details on prompt templates and error recovery. The stress-test concern about unequal engineering effort across scaffolds therefore stands as a real open question; without those specifics it is hard to know how much of the 28-point gap traces to architecture versus implementation polish.\n\nBenchmark researchers and anyone interpreting agent capability numbers will find the results directly useful. The work deserves a serious referee because the pre-registration and controlled setup make the findings checkable even if the methods section needs expansion.","headline":"Scaffold choice shifts GAIA accuracy by up to 28 points on the same model, and the pre-registered prediction that stronger models are less scaffold-sensitive is rejected.","tokens_in":2418,"tokens_out":383,"would_cite":true,"duration_ms":16373,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.3","headline":"Scaffold choice shifts measured accuracy on GAIA by up to 28 percentage points for the same model.","keywords":["scaffold effects","GAIA benchmark","agent evaluation","model capabilities","controlled comparison","multi-agent systems","ReAct"],"falsifier":"A replication study that implements the identical three scaffolds on the same GAIA tasks and models but measures accuracy gaps below 10 points in every dataset slice.","tokens_in":2735,"feed_emoji":"","tokens_out":719,"duration_ms":23547,"temperature":0.7,"pith_summary":"Published agent scores on benchmarks like GAIA mix what a model can do with what its scaffold lets it do. This study holds tasks and models fixed while testing three scaffolds across five models on GAIA Levels 1 and 2, with three attempts per question. Scaffold variation alone produces accuracy gaps as large as 28 points, confirming the pre-registered hypothesis of at least 10-point differences. Some predictions about model capability and scaffold sensitivity are rejected by the results, including that stronger models would be less affected. The data show that reported capability numbers are conditional on the scaffold and that the elicitation gap does not reliably shrink as models improve.","feed_headline":"Scaffold choice shifts GAIA accuracy by up to 28 points","feed_subtitle":"Controlled tests show agent performance numbers depend heavily on the framework wrapping the model.","key_machinery":"Controlled comparison of three scaffolds (ReAct, Planner-Actor-Rater multi-agent design, and planner-then-executor) across fixed tasks and models on GAIA Levels 1 and 2.","core_discovery":"The paper establishes through a pre-registered controlled comparison that scaffold choice alone moves measured accuracy by as much as 28 percentage points within a single model on GAIA validation Levels 1 and 2. It confirms that scaffold variation produces gaps of at least 10 points but rejects the prediction that more capable models are less scaffold-sensitive; instead, effects vary significantly by model in every slice, with the most capable Anthropic model gaining the most from structured scaffolds at the harder level. The multi-agent advantage over ReAct at Level 2 holds only within the Anthropic family, the predicted planner-executor advantage on file-reading tasks is falsified, and str","pith_inferences":["Agent evaluations may need to report results across multiple scaffolds to isolate model contributions from scaffold contributions.","Benchmarks could incorporate scaffold randomization or standardization to reduce conflation in reported scores.","Error recovery patterns suggest specific scaffold design features worth testing in isolation on other benchmarks."],"forward_implications":["Single-scaffold capability numbers are conditional estimates rather than pure measures of model ability.","The elicitation gap between model potential and observed performance is not guaranteed to shrink as models improve.","Model family rather than capability tier determines sensitivity to structured scaffolds at harder task levels.","Multi-agent scaffolds show an advantage over ReAct at Level 2 only within the Anthropic model family.","Planner-then-executor does not deliver the predicted advantage on file-reading tasks."],"fun_headline_variants":["Scaffolds move GAIA accuracy by 28 points within models","Scaffold variation produces 28-point GAIA accuracy gaps","GAIA scores change 28 points depending on agent scaffold","Scaffold choice causes 28-point shifts in GAIA accuracy"],"cache_read_input_tokens":2112,"weakest_assumption_plain":"The three scaffolds were implemented without systematic bias in tool-use patterns, error recovery logic, or prompt engineering that would favor one scaffold over another when tasks and models are held fixed.","fun_headline_variants_meta":{"raw":{"variants":["Scaffolds move GAIA accuracy by 28 points within models","Scaffold variation produces 28-point GAIA accuracy gaps","GAIA scores change 28 points depending on agent scaffold","Scaffold choice causes 28-point shifts in GAIA accuracy"]},"model":"grok-4.3","cost_usd":0.005204,"raw_usage":{"total_tokens":2610,"prompt_tokens":842,"num_sources_used":0,"completion_tokens":68,"cost_in_usd_ticks":52037000,"prompt_tokens_details":{"text_tokens":842,"audio_tokens":0,"image_tokens":0,"cached_tokens":256},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":1700,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":842,"tokens_out":68,"duration_ms":9796,"temperature":1.0,"reasoning_tokens":1700,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-06-27T18:25:24.604187+00:00","model_set":{"reader":"grok-4.3"},"falsifier":"A replication study that implements the identical three scaffolds on the same GAIA tasks and models but measures accuracy gaps below 10 points in every dataset slice.","supporting_citations":[],"review_version":1}