{"id":"ebf8cb9a-a709-47c6-a4ea-90920b4466ac","arxiv_id":"2608.08002","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Best-of-K selection against an ensemble of AI judges overstates quality at most like the square root of log K times common-mode error, which disagreement-based audits cannot see.","lead":"This paper proves that averaging scores from several AI judges does not remove errors that all judges share, and that such shared errors can be exploited when a model searches many candidate answers. It gives mathematical bounds on how much best-of-K selection overstates quality, and explains why judge disagreement alone cannot certify safety.","discovery_kind":"first_principles","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The conditional theorems appear sound, but the real-model audits never certify Assumption 5, so the empirical support for the sqrt(log K) guarantee is missing.","rationale":"The reader's weakest_assumption identifies exactly the same load-bearing concern: the finite-search bounds require Assumption 5, and the paper's own audits do not certify a sub-Gaussian proxy on real judges. The reader's verdict of CONDITIONAL is appropriate: the conditional theorems are sound, but the empirical support for the real-model applicability is missing because the real audits rely on covariance plug-ins without certifying the tail condition. The paper explicitly flags this in the Limitations section, so the concern is acknowledged rather than hidden, which further supports a conditional rather than reject verdict. I see no internal inconsistency in the proofs, no omitted step that changes the statements, and no unsupported mathematical claim within the stated assumptions. The reproducibility gaps noted by the reader (missing checkpoint revision hashes, per-prompt records, DPO logs) are real but secondary to the assumption-certification issue. The proposed test directly checks whether real judge residuals satisfy the sub-Gaussian envelope, which is the condition on which the practical guarantee depends.","tokens_in":25658,"tokens_out":7961,"duration_ms":91349,"concrete_test":"On the GSM8K exact-match anchor (where eta_1 is identically zero), compute realized residuals z_k = R(x,a_k) - q(x,a_k) - bbar(x) on the held-out prompts and estimate the empirical MGF Mhat(lambda) = mean exp(lambda z_k) for lambda in {0.25, 0.5, 1, 2}; compare it to the sub-Gaussian envelope exp(lambda^2 vhat / 2) using vhat from the calibration covariance. If Mhat(lambda) exceeds the envelope by more than a bootstrap confidence interval at any lambda, Assumption 5 is rejected for these judges and the real-model sqrt(log K) bound is not certified. A complementary simulation with centered t-distributed errors (df=3) at J=4, K=32 scaled to the same covariance would show whether the bound degrades qualitatively when the sub-Gaussian condition fails.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central mathematical claims are internally consistent: the projector decomposition, Proposition 4, Theorem 6, Corollary 8, Proposition 9, Corollary 10, and Theorem 11 all hold as conditional statements. The load-bearing concern is not a flaw in the proofs but in the transfer of the theoretical guarantee to real evaluators. Assumption 5 (Eq. 17) requires a joint sub-Gaussian proxy matrix, and Corollary 8 requires a history-conditional MGF bound (Eq. 22). The real-model audits use covariance plug-in estimates, which are second-moment quantities and do not imply the required tail condition. The paper itself states this in the Limitations: the archived real-model experiments use covariance plug-ins rather than certified proxy matrices, and real-data nominal coverage of Eq. (19) is unestablished. Therefore the empirical envelope curves in Table 3 and the descriptive pass fractions (0.81-0.99) are consistency checks under a Gaussian working model, not validation of the finite-search guarantee. If real judge errors are heavy-tailed or become conditionally dependent after the search history, the sqrt(log K) bounds can be optimistic even when the covariance is estimated accurately. The paper is honest about this boundary, so the contribution remains a valid conditional theory, but the practical claim that real ensembles achieve the stated search cost is not demonstrated.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper studies reward hacking in ensembles of learned evaluators under finite best-of-K search. It decomposes the joint error covariance into the aggregation direction and the orthogonal projector, showing that cross-judge disagreement is blind to common-mode response-dependent error and that this common-mode component cannot be identified from internal scores alone (Proposition 4). Under a joint sub-Gaussian assumption on centered judge-error vectors (Assumption 5), Theorem 6 bounds selected-response overstatement and target-quality regret by terms of order sqrt(v_x log K), and Corollary 8 extends the bounds to predictably adaptive proposals under a conditional MGF condition. The paper also shows that a noisy quality anchor adds a rank-one common-mode to the estimated covariance (Proposition 9), that two independent anchors remove it (Corollary 10), and it provides a bounded two-anchor Bernstein certificate (Theorem 11). Empirically, the paper reports fixed-seed Gaussian stress tests, a multi-family best-of-K audit, an all-subset aggregation audit, and an exploratory three-seed DPO pilot. The authors are explicit that the real-model audits use plug-in covariances and do not certify the sub-Gaussian proxy.","tokens_in":25797,"tokens_out":12055,"duration_ms":127443,"significance":"The conditional theoretical results are clean, internally consistent, and likely useful: the projector decomposition (Eqs. 11-12) gives a simple audit identity; Proposition 4 formalizes an important non-identifiability boundary; Theorem 6 and Corollary 8 provide finite-search guarantees with explicit dependence on the projected variance; and the anchor-contamination result (Proposition 9) is a valuable caution for proxy-relative auditing. The paper is unusually honest about the limits of its empirical validation: the synthetic stress test is a code-path check, the real-model pass fractions are not coverage, and the real-task Bernstein certificate is uninformative at m=80. The difficulty is that the abstract and introduction present the real-model audits as 'validating the theory,' which overstates what the evidence supports. The central theoretical contribution is sound, but the manuscript needs to align its claims with the evidentiary boundaries it acknowledges.","major_comments":[{"comment":"The abstract states that 'real-model audits validate the theory,' but §5.2 and the Limitations state that the plug-in pass fractions (0.81–0.99) are not coverage and that 'real-data nominal coverage of Eq. (19) is unestablished.' Because Theorem 6 and Corollary 8 are conditional on Assumption 5 (Eq. 17) or the conditional MGF bound (Eq. 22), and because the real-model audits use covariance plug-ins rather than certified proxy matrices, the empirical results are consistency checks under a Gaussian working model rather than validation of the finite-search guarantee. I recommend replacing 'validate' with 'are consistent with' in the Abstract, §1, and §5.2, and explicitly stating in the main text that no real-model experiment certifies the tail condition required by the theorems.","section":"Abstract; §5.2; Limitations"},{"comment":"Theorem 11 is a valid formal certificate, but its real-data demonstration is null: the paper reports that all 855 cells are range-capped because the m=80 estimation correction (≈0.61) dominates bvs, and the 1.0 pass fractions are trivial. The Abstract's phrase 'propose a bounded two-anchor Bernstein certificate' should be labeled as a purely formal proposal, and the conclusion should state clearly that practical use of the certificate at the reported scale is not demonstrated and would require roughly 10^5 certification pairs. The current §5.5 says this, but the Abstract and §1 do not.","section":"§5.5; Theorem 11; Appendix M"},{"comment":"Corollary 8 is presented as an extension 'to predictably adaptive search,' but no experiment tests the history-conditional MGF condition (Eq. 22). The DPO pilot in Appendix G does not exercise Corollary 8 because it does not condition on a search history in the sense of the corollary. To avoid overclaiming, the paper should state in the contributions that Corollary 8 is a conditional theoretical result whose empirical applicability is not examined in this paper.","section":"§4.3, Corollary 8"}],"minor_comments":[{"comment":"The rows for covariance_weighted and minimum are identical in both reported columns; please add a sentence explaining whether this is coincidental or expected for the chosen data and constraints.","section":"Table 1"},{"comment":"The column header 'bench. frac.' is not self-explanatory; consider renaming it 'descriptive pass fraction (not coverage)' to discourage misinterpretation.","section":"Table 2"},{"comment":"The statement about the largest observed-to-bound ratio of 0.787 would benefit from explicitly stating that the comparison is against the Theorem 6 envelope term with v = ρ + (1−ρ)/J, since the bound is not an equality.","section":"§5.1"},{"comment":"The repository names 'weqweasdas/RM-Gemma-2B' and 'weqweasdas/RM-Mistral-7B' look like placeholders; please provide the actual model identifiers or state that they are anonymized.","section":"Appendix M"},{"comment":"The sentence 'one split fits the aggregator, a disjoint certification split estimates its residual variance' could be read as claiming the certification split estimates the variance of Z directly, but Theorem 11 estimates Var(Z) via anchor cross-covariance; clarify the split's role.","section":"§4.5"},{"comment":"The abstract's 'asymptotically tight for Gaussian projected errors' is accurate only for the maximum-error envelope, as noted in §4.3; consider qualifying 'for the envelope' in the abstract.","section":"Abstract"}],"recommendation":"major_revision","confidential_remarks":"The paper is unusually self-aware, and its appendices already flag most limitations; the main revision is to make the abstract, introduction, and conclusion match the evidence. There is no indication of novelty concealment; the related-work positioning is thorough. The empirical contribution is thin relative to the validation language used, but the conditional theory is sound enough to warrant a revised version."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know: this paper is a legitimate conditional theory paper with unusually honest empirics. The main results—best-of-K overstatement and regret bounds of order sqrt(log K) under a joint sub-Gaussian error model, the adaptive version under conditional calibration, the non-identifiability of common-mode error, and the two-anchor Bernstein certificate—are new as far as I can tell, and the proofs check out. The design-effect identity is classical, but the selection-aware guarantees are not in the cited literature.\n\nWhat's good: the projector decomposition cleanly separates what aggregation removes from what search exploits, and Proposition 4 plus Proposition 9 sharpen the common-mode identification problem. The paper is exemplary in stating what it does not claim: it repeatedly flags that covariance does not imply the sub-Gaussian tail, that the real audits use plug-in covariance rather than certified proxies, and that the 0.81–0.99 pass fractions are descriptive, not coverage. The real-task certificate is reported as valid but uninformative at m=80, and the DPO pilot is labeled underpowered. That candor is rare and worth preserving.\n\nSoft spots: the empirical support for the sqrt(log K) guarantee on real evaluators is missing. Assumption 5 (joint sub-Gaussian proxy) is never certified on the real judges; the experiments use second-moment plug-ins, which don't give the tail. The synthetic stress test is generated from the assumed model, so it checks the code path, not the model. The all-subset audit is single-task (GSM8K), single-generator, low-prevalence. Reproducibility is incomplete: no pinned checkpoint revision hashes, missing training and candidate-level logs, and the certificate needs roughly 1e5 pairs to be informative. These are real gaps, but they are acknowledged in the text rather than hidden.\n\nIf I'm being fair, the abstract's \"validate the theory\" overstates what the real-model audits establish—they are consistency checks, not independent validation. But the body and limitations are precise about this boundary. Citation pattern looks solid; the related work is well integrated.\n\nRecommendation: send it to review. The conditional theorems deserve referee time, and the empirical section raises the right questions even where it doesn't answer them. Accept conditionally on tightening the empirical claims to match the limitations, and ideally adding either a certified proxy experiment or an explicit statement that none exists.","headline":"Solid conditional theory with unusually honest empirics; the real-model gap is real but openly disclosed, and the paper deserves a serious referee.","tokens_in":26402,"tokens_out":1895,"would_cite":true,"duration_ms":21324,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Reward hacking in judge ensembles is driven by the error component that averaging cannot remove: the common-mode error along the aggregation direction, which disagreement diagnostics cannot see.","keywords":["reward hacking","evaluator ensembles","covariance geometry","best-of-K search","sub-Gaussian bounds","common-mode error","disagreement diagnostics","Bernstein certificate"],"falsifier":"Generate candidate responses from a heavy-tailed error family with exactly the same covariance as the paper's Gaussian model, for example scaled t-distributed errors with the same v_x, run best-of-K selection with K=32, and check whether the empirical overstatement regularly exceeds sqrt(2 v_x log K); if it does, the sub-Gaussian condition, not the covariance geometry, is the load-bearing premise.","tokens_in":25374,"feed_emoji":"⚖️","tokens_out":7486,"duration_ms":75727,"temperature":0.7,"pith_summary":"The paper tries to establish where ensemble evaluation actually fails under optimization pressure: not in the judge-specific noise that averaging removes, but in the response-dependent error shared by all judges, which lies along the aggregation direction. It proves that cross-judge disagreement measures only the orthogonal component of error, so disagreement can be high while averaging safely suppresses noise, or low while a shared, exploitable error persists. It then shows that under a joint sub-Gaussian error model, best-of-K selection overstates the chosen response's quality by at most sqrt(2 v_x log K) and loses no more than 2 sqrt(2 v_x log K) in target quality, where v_x is the ensemble-projected error variance. The same geometry shows that a noisy quality anchor injects an artificial rank-one common mode that a two-anchor cross-covariance can remove. A sympathetic reader should care because it converts the folklore that ensembles mitigate reward hacking into a precise statement of what ensembling can and cannot do, and it provides finite-search guarantees that scale only logarithmically with the search budget.","feed_headline":"Reward hacking comes from the judge error that disagreement can't see","feed_subtitle":"New covariance bounds show that adding judges helps only when it shrinks the error direction that best-of-K search exploits.","key_machinery":"The load-bearing object is the covariance geometry of the calibrated judge-error vectors e(x,a), split by two complementary projectors: the aggregation direction 11^T/J, along which the ensemble mean lives and whose projected variance v_x = $J^{{-2}}$ 1^T Gamma_x 1 controls what finite search can exploit, and the orthogonal projector P = I - (1/J) 11^T, whose squared norm equals calibrated disagreement. The argument couples this geometry to a sub-Gaussian moment condition on the projected errors, so the maximum over K candidates behaves like a Gaussian maximum with cost $\\sqrt$(2 v_x log K); for independent Gaussian projected errors, extreme-value theory makes this order tight. The two-anchor construction Cov(e~(1), e~(2)) = Sigma_x is the decontamination device that removes the artificial rank-one common mode induced by measuring all judges against one noisy proxy.","core_discovery":"On the paper's own terms, the central discovery is that the error exposed to finite search is exactly the projection of the joint judge error onto the aggregation direction: for a uniform ensemble, the all-ones vector. The ensemble mean retains this common-mode component, while disagreement is the energy in the orthogonal complement, isolated by the projector P = I - (1/J) 11^T. Consequently, response-dependent common-mode error is not identifiable from internal judge scores alone (Proposition 4), and a held-out quality proxy contaminated by its own noise masquerades as common-mode error via a rank-one term $tau^{2}$ 11^T (Proposition 9). Under a joint sub-Gaussian error model with proxy matrix Gamma_x, the paper proves finite-search bounds: the selected response's overstatement is at most b_bar(x) + $\\sqrt$(2 v_x log K) in expectation and the regret against the best searched candidate at most 2 $\\sqrt$(2 v_x log K), with a conditional version for predictably adaptive proposals. It also supplies a bounded two-anchor Bernstein certificate that converts an estimated variance upper bound into a valid search tail without assuming Gaussian errors.","pith_inferences":["A testable consequence the paper leaves implicit: on any task with a verifiable objective, such as exact-match math or unit-tested code, one could measure the real common-mode error directly and compare it with v_x estimated from the two-anchor cross-covariance; agreement would endorse the certificate, disagreement would localize which anchor assumption fails.","The identifiability result suggests that disagreement-based preference data selection, a common practice in alignment pipelines, cannot by itself prevent reward hacking; pairing such selection with a downstream verifier on a held-out task would convert Proposition 4 into an operational screening rule.","The sqrt(log K) guarantee is an average-case statement over a declared candidate distribution; one could stress-test it with adversarial candidate generators that deliberately target the common-mode direction, in which case the sub-Gaussian proxy may fail and the bounded Bernstein certificate would be the safer bound.","By analogy with the design-effect interpretation, the paper's effective-panel-size formula may extend to arbitrary weighted ensembles: the effective number of judges for search is the ratio of mean marginal variance to projected variance, so procuring diverse judges should target the covariance spectrum rather than the nominal panel count."],"forward_implications":["If Theorem 6 is right, ensembling helps precisely to the extent that it shrinks the projected variance v_x; adding correlated judges that barely raise the effective panel size leaves the sqrt(log K) search cost nearly unchanged.","Disagreement-based diagnostics such as active learning, uncertainty sampling, or disagreement filtering cannot certify safety against common-mode failure: Proposition 4 says only an external anchor or additional structural assumptions can do that.","The sqrt(log K) scaling makes search pressure mild in K: increasing the candidate budget by several orders of magnitude adds only a few single-candidate standard deviations to expected overstatement, so large failures at large K point to a large common-mode component rather than to the search itself.","A noisy quality anchor inflates the estimated common-mode variance by its own variance, and the two-anchor cross-covariance removes that inflation only when the anchor errors are conditionally independent of each other and of the judge errors.","The bounded two-anchor Bernstein certificate gives finite-sample, non-Gaussian tail guarantees for selected overstatement and regret, at the price of a linear term that can dominate at modest certification sample sizes."],"supporting_citations":[{"why":"Supplies the design-effect formula that the paper adapts for the variance of the ensemble-mean error, the baseline for effective panel size.","marker":"(Kish, 1965)"},{"why":"Establishes reward-model overoptimization as the target phenomenon and provides the best-of-n scaling context this paper's search bounds build on.","marker":"(Gao et al., 2023)"},{"why":"Documents empirically that shared reward-model errors survive ensembling, the phenomenon the covariance geometry explains.","marker":"(Eisenstein et al., 2023)"},{"why":"Provides the ensemble-mitigation baseline and conservative aggregation comparisons against which this paper positions its mean-aggregation analysis.","marker":"(Coste et al., 2024)"},{"why":"Motivates the disagreement-based active learning question and shows why disagreement alone is a weak signal for common-mode risk.","marker":"(Gleave and Irving, 2022)"},{"why":"Used in the proof of Theorem 11 to turn the bounded two-anchor cross-covariance estimate into a one-sided variance upper bound.","marker":"(Hoeffding, 1963)"},{"why":"Supplies the bounded-variable moment-generating-function bound that converts the variance cap into a finite-search tail for selected overstatement and regret.","marker":"(Bennett, 1962)"},{"why":"Gives the extreme-value result establishing that the sqrt(log K) order is asymptotically tight for Gaussian projected maxima.","marker":"(Leadbetter et al., 2012)"}],"fun_headline_variants":["Disagreement can't see the shared judge error that search exploits","Best-of-K exploits the judge error that disagreement ignores","More judges may not shrink the error that reward hacking uses","Common-mode judge error is invisible to disagreement metrics","Reward hacking: the error that disagreement hides"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The bounds stand on the assumption that the judges' centered errors, after calibration, satisfy a joint sub-Gaussian tail bound with a known or certified proxy matrix, and, for adaptive search, that this tail bound survives conditioning on the search history; if real judge errors are heavy-tailed or the proxy matrix is not certified, the sqrt(log K) guarantees can be optimistic.","fun_headline_variants_meta":{"raw":{"variants":["Disagreement can't see the shared judge error that search exploits","Best-of-K exploits the judge error that disagreement ignores","More judges may not shrink the error that reward hacking uses","Common-mode judge error is invisible to disagreement metrics","Reward hacking: the error that disagreement hides"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000206,"raw_usage":{"total_tokens":1412,"prompt_tokens":979,"completion_tokens":433,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":595,"completion_tokens_details":{"reasoning_tokens":369}},"tokens_in":595,"tokens_out":433,"duration_ms":5450,"temperature":1.0,"reasoning_tokens":369,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T00:35:11.806827+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Generate candidate responses from a heavy-tailed error family with exactly the same covariance as the paper's Gaussian model, for example scaled t-distributed errors with the same v_x, run best-of-K selection with K=32, and check whether the empirical overstatement regularly exceeds sqrt(2 v_x log K); if it does, the sub-Gaussian condition, not the covariance geometry, is the load-bearing premise.","supporting_citations":[],"review_version":1}