{"id":"b6342394-ff04-439a-9773-d378b932e922","arxiv_id":"2608.07424","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A compute-balanced routing policy, CoBa-Routed-Strong, reaches the accuracy of best-of-16 majority voting and self-evaluation weighted voting while using roughly 49 to 59 percent fewer parameter-weighted tokens.","lead":"This paper introduces CoBa, a routing policy that decides whether to spend the next unit of test-time compute on generating more candidate answers, verifying them cheaply, or applying a stronger verifier. It reports that this balanced routing matches the accuracy of much more expensive sampling or self-evaluation baselines at substantially lower compute cost on math and symbolic reasoning benchmarks.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"No validation protocol is described for the routing thresholds, and the learned-controller negative result hints that the fixed policy may be tuned to the benchmark suite; the headline cost savings could therefore be in-sample.","rationale":"The reader's weakest assumption concerned transferability of the cheap-evidence stop criterion to a new distribution. I agree that this is a genuine risk, but I see a more immediate and more load-bearing version of the same concern: the paper never reports a validation protocol for the fixed thresholds, so even the in-sample frontier may be optimistic. The shared-pool replay design and paired bootstrap tests are methodologically sound, and the direct comparison with best-of-16 majority voting is a strong, non-circular baseline; those parts support the empirical result. However, the single fixed configuration in Table 1 is presented without any sensitivity analysis or holdout selection. The paper's own learned-controller experiment degenerating to near-greedy under leave-one-dataset-out is an internal signal that robustly choosing thresholds is hard, which strengthens the need for a validation check. Because the reader's CONDITIONAL verdict already flags the absence of a clear validation procedure and no code/data, my read does not change the verdict; it reinforces the same condition. A leave-one-benchmark-out calibration test would settle whether the cost savings transfer.","tokens_in":15941,"tokens_out":9949,"duration_ms":90180,"concrete_test":"Run leave-one-benchmark-out calibration: for each of the five benchmark families, select the Table 1 thresholds (stop share, s1 threshold, Nmax, K, fusion weights) using only the other four families via a simple grid search over validation accuracy at matched cost, then evaluate on the held-out family. Average the held-out accuracy and parameter-token cost across folds and compare against CoBa-Routed-Strong in Table 2. If held-out macro accuracy drops by more than roughly 1 point, or the cost savings shrink below the reported 49%, the headline frontier is threshold-dependent rather than a general property of the routing policy.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central cost-accuracy claim depends on a single fixed hyperparameter configuration (Table 1: warm-up k=2, stop thresholds share>=0.6 and s1>=0.7, Nmax=8, K=4, fusion weights .20/.30/.15/.45), yet the paper never describes how these values were selected or whether a validation split was used. The only reported attempt to learn a controller under leave-one-dataset-out 'degenerated to a near-greedy policy', which is direct evidence that dataset-specific tuning matters. If the Table 1 thresholds were chosen, even informally, on development runs of the same five benchmarks, then the reported 49.1% and 58.9% parameter-token savings are in-sample numbers; a policy tuned per benchmark need not deliver the same frontier on a new distribution or on a properly held-out split of the same benchmarks. The cheap-evidence stop rule in Algorithm 1 (line 11) is the mechanism through which savings are realized, so its thresholds must be calibrated and validated rather than asserted. Without holdout validation, the paper has not established that 'routing where compute is spent'—as opposed to 'thresholds hand-tuned to these benchmarks'—is what produces the Pareto improvement.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper frames test-time reasoning as a compute-allocation problem in which a system chooses, at each step, among sampling another candidate, applying lightweight verification, invoking a stronger verifier, or stopping. It introduces CoBa, a staged routing policy that first collects a warm-up candidate set, scores all candidates with cheap verification, and routes a subset to a strong verifier. On a shared-pool replay setup over 15 dataset–generator pairs (MATH-500, AIME 2024/2025, AMC 2023, and a Reasoning Gym hard subset), CoBa-Routed-Strong reaches 85.13% macro accuracy, statistically matching a self-evaluation weighted-voting proxy at 85.20% while using 49.1% fewer parameter-weighted tokens, and it reaches the same macro-accuracy region as best-of-16 majority voting at 58.9% lower parameter-weighted token cost. The paper reports paired bootstrap tests, per-dataset results, ablations over routing strength, action-distribution evidence, and an explicit pool-oracle upper bound.","tokens_in":16181,"tokens_out":6907,"duration_ms":59485,"significance":"If the headline claims hold, the paper makes a useful contribution: it turns test-time scaling into an explicit allocation question and provides a reproducible, locally deployable policy that approaches the accuracy of much more expensive sampling and self-evaluation baselines. The controlled replay protocol over shared candidate pools is a genuine strength, as it places CoBa and the baselines on identical generated evidence. The paper is also transparent about the remaining best-of-16 paired edge and about the oracle gap, and it reports multiple cost metrics. The main weakness is that the routing hyperparameters are hand-set and no validation protocol is described, which leaves open the possibility that the reported cost–accuracy frontier is partly an artifact of threshold selection on the evaluated benchmarks. This is the load-bearing issue for the paper's central claim and needs to be addressed before the result can be taken as evidence for the general value of compute-balanced routing.","major_comments":[{"comment":"The central accuracy–cost claim rests on a single fixed hyperparameter configuration (fusion weights .20/.30/.15/.45, stop thresholds share≥0.6 and s1≥0.7, warm-up k=2, Nmax=8, K=4), but the paper gives no validation protocol for these values. The statement in §5 that the values were set before final aggregation with test labels reserved does not rule out tuning on development runs: Appendix A reveals that earlier 4,096-token pools were used during development and quality checks. If the thresholds were selected, even informally, on related data from the same five benchmarks, the reported 49.1% and 58.9% parameter-token savings could be in-sample. This is load-bearing because the savings are realized by the cheap-evidence stop rule (Algorithm 1, line 11) and by the strong-verification route count K. The learned MLP controller's degeneration to a near-greedy policy under leave-one-dataset-out evaluation is direct evidence that dataset-specific tuning matters. I request a held-out validation split (or leave-one-dataset-out calibration) for the thresholds, plus a sensitivity analysis over the Table 1 parameters showing that the Pareto improvement is not an artifact of the exact values chosen.","section":"§4, Table 1, Eq. (6), Appendix A"},{"comment":"The stop criterion assumes that answer agreement (share≥0.6) plus a high Qwen3-8B judge score (s1≥0.7) reliably implies that the current top answer is correct. The manuscript never reports the calibration of this rule: among examples that stop at line 11, what fraction end with the correct final answer, and among examples that do not stop, how often would an early stop have been wrong? This matters for transfer because the cost savings are structural but the accuracy preservation depends on the cheap signal being a good proxy for correctness. The large AIME 2025 oracle gap (71.1 vs 83.3 in Table 3) suggests that stable cheap evidence is sometimes wrong or that the pool lacks a correct candidate; a precision/recall analysis of the stopping rule and of the subsequent strong-verification selection would clarify how much of the reported frontier is due to routing rather than threshold luck.","section":"§5, Algorithm 1, line 11"},{"comment":"The abstract states that CoBa-Routed-Strong 'matches best-of-16 majority voting within 0.01 macro-accuracy points,' but the paper's own paired bootstrap finds best-of-16 significantly more accurate (difference −0.70 points, 95% CI [−1.25,−0.16], p=0.004). Macro accuracy can differ from per-example paired accuracy, but the abstract's wording overstates parity and is inconsistent with the paired conclusion reported later. Please qualify the claim, e.g., 'matches best-of-16 on macro accuracy while the paired test retains a small best-of-16 edge at 2.43× the cost,' and ensure the abstract distinguishes macro-level parity from per-example significance.","section":"Abstract; §5, 'Significance and Ablations'"}],"minor_comments":[{"comment":"The state diagram contains a typo: 'scores + uuncertainly' should read 'scores + uncertainty.'","section":"Figure 1"},{"comment":"The objective introduces a cost-sensitivity parameter λ, but the experiments never estimate or report λ; please explain how the fixed replay policies instantiate this objective, or drop λ from the formulation.","section":"§3, Eq. (5)"},{"comment":"Line 17 says candidates are ranked by 'lightweight score and answer frequency,' while Eq. (6) includes the process-verifier score s2 and the strong-verifier score s3; specify the exact score composition used at each stage so it is clear when V2 and V3 evidence enters the ranking.","section":"Algorithm 1, lines 17–19"},{"comment":"The cost columns for the pool oracle are not defined in the offline replay protocol; please state explicitly how oracle cost is computed (for example, as the cost of the prefix up to the first correct candidate) so readers do not mistake it for a deployable method's cost.","section":"Table 2, oracle row"},{"comment":"The appendix refers to 'released experiment artifacts' but no repository URL or artifact identifier is provided; please include a link or DOI for reproducibility.","section":"Appendix D"}],"recommendation":"major_revision","confidential_remarks":"The paper is likely publishable after the validation issue is resolved. The shared-pool replay and paired bootstrap methodology are strengths, and the negative learned-controller result is honestly reported. I would not require an additional benchmark suite, but a held-out validation of the routing thresholds and a sensitivity analysis are necessary to support the headline cost-saving claims. The abstract's 'matches best-of-16' wording should also be corrected to reflect the paired significance result. I saw no citation or novelty-disclosure concerns."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"This paper treats test-time scaling as a routing problem among generation, cheap verification, strong verification, and stopping. On a shared candidate pool, the fixed CoBa-Routed-Strong policy lands at the accuracy of best-of-16 majority voting and a self-evaluation proxy while using 58.9% and 49.1% fewer parameter-weighted tokens, respectively. That is a concrete, useful result for local deployment, and the experimental protocol is the best part: every method replays over the same stored candidate pools, paired bootstrap tests are per-example, and the paper explicitly concedes that best-of-16 keeps a small accuracy edge at 2.43x the cost. The per-dataset analysis separating allocation failures (correct answer in pool but not selected) from generation failures (no correct answer in pool) is genuinely informative.\n\nWhat is actually new: I am not aware of prior work that puts generation, lightweight verification, strong verification, and stopping into one unified action space and evaluates the resulting policy over a shared pool. The staged cheap-then-strong verification design is simple but effective, and the paper does not oversell it. The proxy baselines are clearly labeled, the oracle gap is reported, and the learned-controller negative result is honestly described.\n\nThe soft spots are real, though not fatal. The routing thresholds in Table 1 (warm-up 2, share >= 0.6, s1 >= 0.7, fusion weights, Nmax 8, K 4) are hand-set, and the paper never describes a validation protocol. The values were supposedly fixed before final aggregation with test labels reserved, but that does not rule out tuning on development runs of the same five benchmarks. The appendix even says earlier 4,096-token pools were used during development and quality checks, which suggests the 16,384-token replay configuration was chosen after looking at these datasets. The learned MLP controller degenerating to a near-greedy policy under leave-one-dataset-out is consistent with the fixed policy being tuned to the suite. So the headline savings are quite possibly in-sample. That is not a reason to desk-reject, but it is a reason to require a holdout validation (per-dataset or per-generator) before the numbers are taken at face value.\n\nMinor issues: no code or data are actually available despite the appendix referring to 'released experiment artifacts', and the self-evaluation weighted-voting baseline is a local proxy rather than a published method, so matching it is less informative than matching best-of-16. The parameter-weighted token metric is reasonable and reported faithfully.\n\nBottom line: this is a solid engineering paper with a fair experimental design and one important validity gap. It deserves a serious referee; I would send it to review with the explicit request that the authors add holdout validation for the thresholds and release the replay code and pools. If the results survive holdout, this is a useful addition to the local test-time scaling toolbox.","headline":"A well-executed local routing policy with a real cost-accuracy payoff, but the hand-set thresholds need holdout validation before the headline savings are taken at face value.","tokens_in":16727,"tokens_out":2474,"would_cite":true,"duration_ms":23003,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"CoBa reframes test-time scaling as compute allocation and matches the accuracy of sampling and self-evaluation baselines at roughly half the parameter-weighted tokens.","keywords":["test-time scaling","compute allocation","routing policy","model verification","majority voting","mathematical reasoning","cost-accuracy frontier","LLM inference"],"falsifier":"Run CoBa-Routed-Strong on a new benchmark where the cheap Qwen3-8B judge is deliberately reversed or made miscalibrated while keeping the candidate pools identical; if accuracy drops to the level of the early-stopped prefix, the cheap-evidence triage layer is carrying the result rather than the routing structure itself.","tokens_in":15719,"feed_emoji":"⚡","tokens_out":4222,"duration_ms":35506,"temperature":0.7,"pith_summary":"This paper claims that test-time scaling should be treated as a compute-allocation problem: at each step, a reasoning system must decide whether to sample another candidate, apply cheap verification, invoke a stronger verifier, or stop. It introduces CoBa, a staged routing policy that first gathers a small set of candidates, scores all of them with cheap evidence, and routes only uncertain or high-value candidates to strong verification. On 3,129 example–generator evaluations across five math and symbolic reasoning benchmarks, CoBa-Routed-Strong reaches 85.13% macro accuracy, statistically matching a self-evaluation weighted-voting proxy at 85.20% while using 49.1% fewer parameter-weighted tokens, and matching best-of-16 majority voting within 0.01 accuracy points while using 58.9% fewer parameter-weighted tokens. A sympathetic reader cares because this gives local reasoning systems a concrete way to reach high accuracy without paying the uniform sampling or uniform evaluation cost on every example, and it reframes the field's evaluation from accuracy alone to accuracy at controlled cost.","feed_headline":"CoBa matches best-of-16 voting with 58.9% fewer tokens","feed_subtitle":"A cheap verify-then-escalate policy reaches the accuracy of heavy sampling and self-evaluation on math reasoning tasks.","key_machinery":"The central mechanism is compute-balanced routing with a cheap-evidence triage layer: answer agreement and a lightweight judge score decide whether to stop early, sample more candidates, or escalate selected candidates to a strong deep verifier. The final ranking $R(c) = 0.20 f(c) + 0.30 s_1(c) + 0.15 s_2(c) + 0.45 s_3(c)$ fuses answer frequency, lightweight judge score, optional process-verifier score, and strong-verifier score, with weights renormalized when sparse scores are missing. This staged design separates candidate generation from candidate verification and concentrates expensive compute on the examples and candidates where the final decision can still change.","core_discovery":"On the paper's own terms, CoBa shows that a fixed, transparent routing policy over generation, verification, and stopping can reach the accuracy region of much more expensive test-time scaling baselines. The policy obtains a small warm-up candidate pool, scores all candidates with a rule-based answer frequency signal and a lightweight Qwen3-8B judge, and stops when the top answer is stable (answer share at least 0.6) and the lightweight judge score is high (at least 0.7). Otherwise it samples more candidates and routes the top few to a stronger Qwen3-14B deep verifier, selecting the final answer by a fused rank that renormalizes when sparse process-verifier scores are absent. In paired bootstrap tests, CoBa-Routed-Strong significantly outperforms greedy decoding by +3.74 accuracy points, is statistically indistinguishable from best-of-8 and from the self-evaluation weighted-voting proxy, and trails best-of-16 by only 0.70 points while using 2.43x fewer parameter-weighted tokens.","pith_inferences":["The authors left implicit that the same triage idea could transfer to code generation and natural-science reasoning, where public test execution or external tools serve as an even stronger cheap-verification tier before escalating to expensive model-based judges.","A testable extension is to recalibrate the stop thresholds (0.6 answer share, 0.7 judge score) on each new distribution, since the paper's fixed thresholds were not tuned per dataset and may be a source of the remaining oracle gap.","The learned MLP controller degenerating to a near-greedy policy suggests that offline trajectories from a fixed router may be too narrow to train a robust learned controller, and that denser process signals or online continuation could be needed before learned routing becomes viable."],"forward_implications":["If the central claim holds, test-time scaling comparisons should report action mix and cost–accuracy frontier alongside accuracy, making routing gains falsifiable rather than hidden behind a single score.","Routing helps most when the candidate pool is informative and unevenly difficult: easy examples stop after cheap agreement, while hard contest problems consume additional samples and strong-verifier calls, as shown by the per-dataset action distributions.","Uniform best-of-N and always-on evaluators pay the hard-example budget on every example, whereas routed policies concentrate that budget where decisions can still change, yielding the reported frontier movement.","The remaining oracle gap separates two failure modes: allocation errors, where better routing could select an existing correct candidate, and generation errors, where the pool lacks a correct answer; this distinction turns oracle gaps into design signals for future systems."],"supporting_citations":[{"why":"Provides the self-consistency baseline and the repeated-sampling paradigm that CoBa compares against through best-of-N majority voting.","marker":"Wang et al. 2022"},{"why":"Establishes chain-of-thought prompting, the reasoning pattern that all candidate generation in this paper builds on.","marker":"Wei et al. 2022"},{"why":"Defines the s1 budget-forcing replay baseline, one of the adaptive test-time scaling methods CoBa is compared with.","marker":"Muennighoff et al. 2025"},{"why":"Introduces verifiers for math word problems, the origin of the verification tier that CoBa routes between.","marker":"Cobbe et al. 2021"},{"why":"Provides process supervision and the process-verifier idea that CoBa's optional $s_2$ score is based on.","marker":"Lightman et al. 2024"},{"why":"Supplies the Qwen3 model family used as generators, lightweight judge, and strong verifier in the CoBa pipeline.","marker":"Yang et al. 2025"},{"why":"Supplies Phi-4-reasoning, used as a generator and as the auxiliary process-verifier score source.","marker":"Abdin et al. 2025"},{"why":"Provides the MATH benchmark, including MATH-500, one of the five evaluation sets.","marker":"Hendrycks et al. 2021"},{"why":"Provides the Reasoning Gym hard subset, the procedural symbolic reasoning evaluation set.","marker":"Stojanovski et al. 2026"}],"fun_headline_variants":["CoBa: best-of-16 accuracy with 58.9% fewer tokens","CoBa: compute-balanced routing cuts tokens by 59%","CoBa: match heavy sampling, save 49% tokens","CoBa: test-time scaling with a compute budget","CoBa: 85% accuracy with 59% less compute"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The cheap-evidence stop criterion assumes that a stable answer share and a high lightweight judge score reliably imply that the current top answer is correct, and if these cheap signals are frequently wrong on a new distribution, early stopping will hurt accuracy even though the cost savings are structural.","fun_headline_variants_meta":{"raw":{"variants":["CoBa: best-of-16 accuracy with 58.9% fewer tokens","CoBa: compute-balanced routing cuts tokens by 59%","CoBa: match heavy sampling, save 49% tokens","CoBa: test-time scaling with a compute budget","CoBa: 85% accuracy with 59% less compute"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000446,"raw_usage":{"total_tokens":2289,"prompt_tokens":1017,"completion_tokens":1272,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":633,"completion_tokens_details":{"reasoning_tokens":1183}},"tokens_in":633,"tokens_out":1272,"duration_ms":11039,"temperature":1.0,"reasoning_tokens":1183,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T04:13:01.800120+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run CoBa-Routed-Strong on a new benchmark where the cheap Qwen3-8B judge is deliberately reversed or made miscalibrated while keeping the candidate pools identical; if accuracy drops to the level of the early-stopped prefix, the cheap-evidence triage layer is carrying the result rather than the routing structure itself.","supporting_citations":[],"review_version":2}