{"id":"be3be2c0-3da1-4e8e-8bfb-60bea83ce008","arxiv_id":"2504.21303","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A Bayesian interval-ranking method using anchor models yields probabilistic LLM rankings that the authors show remain discriminative down to roughly 20 questions.","lead":"The paper introduces a Bayesian method that ranks an LLM by testing which interval, defined by a set of anchor models, its hidden capability falls into, using a small question set. It reports that this method separates models better than plain accuracy, Pass@N, or mean plus standard deviation, down to about 20 questions.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Equation 6 is not the marginal likelihood of the paper's stated model: for O>1 it averages endpoint binomials instead of integrating over θ, so all reported Bayes@10 posterior intervals are computed from an unjustified likelihood.","rationale":"The reader's weakest-assumption analysis correctly targets the likelihood construction, but the most load-bearing problem is more specific than Eq. 4's linearity assumption. The paper's own stated model is a continuous latent θ with uniform prior per interval and O independent Bernoulli trials per query; the correct marginal likelihood is an integral over θ of the binomial likelihood. Eq. 6 replaces that integral with an average of endpoint binomial likelihoods, which corresponds to a different, unstated two-point mixture model. This is an internal inconsistency that affects every O=10 result, including the Bayes@10 comparisons in Figure 3 and the M=20 robustness claims. The error is invisible for O=1 but material for O=10; the concrete test above would settle it with a simple analytic computation. I therefore recommend keeping the conditional verdict: the paper should be accepted only after Eq. 6 is corrected or explicitly justified as a mixture model, and the affected posterior intervals and conclusions are recomputed. The reader's concern about Eq. 4 remains valid but is secondary; even a linear p_j(θ) does not make Eq. 6 correct for O>1.","tokens_in":18527,"tokens_out":10281,"duration_ms":114889,"concrete_test":"For a representative adjacent anchor pair and query from Figure 1, compute the exact marginal likelihood L_exact(K) = ∫_0^1 C(10,K) [p_i + t(p_{i+1}-p_i)]^K [1 - p_i - t(p_{i+1}-p_i)]^{10-K} dt and compare it to Eq. 6's average of Binom(K;10,p_i) and Binom(K;10,p_{i+1}) for K=0..10. If the two likelihood vectors differ by more than 5% in total variation distance, Eq. 6 is not the stated model's likelihood and all Bayes@10 posterior intervals in Figures 2 and 3 must be recomputed.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The load-bearing flaw is internal to the multi-trial likelihood, not just the linearity assumption in Eq. 4. The paper states in Section 2.3 that θ is a continuous latent variable with a uniform prior within each interval (Eq. 2), and in Section 3.4 that each query is modeled as O independent Bernoulli trials with success probability p_j(θ) (Eq. 7). Under those assumptions, the marginal likelihood for an interval is the integral of the binomial likelihood over θ, not the endpoint average in Eq. 6. Eq. 6 is instead the likelihood of a two-point mixture in which the test model behaves like L_i for all O trials with probability 1/2 and like L_{i+1} otherwise. That is a different generative model, and it contradicts the uniform-θ/linear-p justification attached to Eq. 4. The discrepancy vanishes for O=1 but is large for O=10. For example, with p_i=0.2, p_{i+1}=0.8, O=2, K=1, Eq. 6 gives 0.32, while the exact uniform-θ integral gives 0.44. Because Figures 2 and 3 and the Bayes@10 comparisons all use O=10, every reported multi-trial posterior is computed from an unstated and internally inconsistent likelihood. This directly undermines the central claim of statistically robust probabilistic rankings under limited sample sizes. The reader's Eq. 4 concern is valid but secondary: even if p_j(θ) were exactly linear, Eq. 6 would still be wrong for O>1.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a Bayesian method for ranking LLMs into capability intervals defined by a set of anchor models. The method treats latent capability as a continuous variable with a uniform prior across anchor-defined intervals, models per-question responses as conditionally independent Bernoulli trials, and claims that the resulting posterior probabilities provide statistically robust rankings even with as few as 20 questions. The authors evaluate the method on five open-weight test models using GPT-series anchors and a curated 50-question set, comparing it with accuracy, Pass@N, and mean-plus-standard-deviation reporting. The central claim is that the Bayesian approach yields enhanced discriminative capability and stable analytical outcomes under limited sample sizes.","tokens_in":18885,"tokens_out":5520,"duration_ms":56813,"significance":"The paper addresses a real and practically important problem: how to make probabilistic, uncertainty-aware claims about LLM rankings when only a small number of evaluation questions can be afforded. The proposed framework is conceptually attractive, and the detailed query set and anchor-model protocol are useful contributions. However, the core multi-trial likelihood (Eq. 6) is not the marginal likelihood of the described model, and because all reported experiments use O=10 trials, the primary empirical claims rest on an internally inconsistent derivation. The paper also does not validate the key linearity assumption in Eq. 4 or the calibration of the reported posterior probabilities. If the likelihood is corrected and the assumptions are validated empirically, the method could be a useful addition to the LLM-evaluation toolkit; as it stands, the central claim is not supported.","major_comments":[{"comment":"Equation (6) is not the marginal likelihood of the model stated in Section 2.3. Under the uniform-θ prior in Eq. (2) and the linear-response justification of Eq. (4), the probability of observing K successes in O trials given that θ lies in an interval should be the integral of the binomial likelihood over θ, not the average of the two endpoint binomial likelihoods. Eq. (6) instead corresponds to a different generative model in which the test model behaves exactly like L_i for all O trials with probability 1/2 and like L_{i+1} otherwise. The discrepancy is not negligible: for p_i=0.2, p_{i+1}=0.8, O=2, K=1, Eq. (6) gives 0.32 while the exact uniform-θ integral gives 0.44. Since Figures 2 and 3 and all Bayes@10 comparisons use O=10, every reported multi-trial posterior is computed from a likelihood that is inconsistent with the paper's own model description. This directly undermines the central claim of statistically robust probabilistic rankings.","section":"Section 3.4, Eq. (6)"},{"comment":"The assertion that the probability of a correct response is linear in θ within each interval is presented as a consequence of maximum entropy, but maximum entropy over θ only justifies the uniform prior in Eq. (2); it does not imply linearity of the item-response function. If the true p_j(θ) is nonlinear, Eq. (4) is biased and the resulting posterior intervals will be miscalibrated. The paper does not provide a derivation of this linearity, nor does it check it against the anchor-model data (e.g., by testing whether success probabilities vary roughly linearly across the anchor points). The limitations section acknowledges only conditional query independence, not this load-bearing assumption, so the assumption's status is unclear.","section":"Section 2.3, Eq. (4)"},{"comment":"The anchor capability values θ_i and the per-question anchor probabilities Pr(Q_j|L_i) are estimated from O=10 trials but are then treated as known constants in the prior (Eq. 2) and in the likelihood (Eqs. 4, 6, 7). This ignores the sampling uncertainty in the anchor responses and overstates the precision of the posterior probabilities reported in Figures 2 and 3. For example, the text states that M=20 retains 'at least 65% confidence' in the most likely interval, but no uncertainty is attached to that figure. A bootstrap or sensitivity analysis around the anchor estimates would be needed to support such quantitative claims.","section":"Sections 2.3 and 3.3"},{"comment":"The claim of 'statistical robustness' at M=20 is supported only by qualitative inspection of Figures 2 and 3. No calibration checks, coverage probabilities, or repeated-sampling simulations are reported, and the query set is selected post hoc based on the authors' knowledge of other model families. The paper would need to demonstrate, at least on synthetic data or via a bootstrap over queries, that the posterior intervals are actually calibrated and that the method's rankings track ground truth. Without this, the conclusion that the method 'maintains statistical robustness' is not established.","section":"Sections 3.2 and 4"}],"minor_comments":[{"comment":"The sentence 'the factorization in Eqs. 3 assumes assumes conditional query independence' contains a duplicated word ('assumes assumes').","section":"Section 5.1"},{"comment":"The text says 'It does not provide any statistica interpretation' — 'statistica' should be 'statistical'.","section":"Section 4, Figure 3 bullet 3"},{"comment":"The exact numerical posterior probabilities are not reported in a table; the text mentions thresholds such as 'at least 65% confidence' but the reader cannot verify these values from the figures. A supplementary table with the full probability mass functions for each test model and each M would improve the paper.","section":"Figures 2 and 3"},{"comment":"The notation 'Prest' is nonstandard and appears without definition. Consider using a hat or a subscript, e.g., \\hat{P} or P_{\\mathrm{est}}, to clarify that these are estimated quantities.","section":"Notation"},{"comment":"Appendix A is referenced in the text as containing the complete success-rate distribution, but the figure or table is not present in the manuscript body. Please include the appendix material or remove the reference.","section":"Appendix A"}],"recommendation":"major_revision","confidential_remarks":"The main issue is the internally inconsistent multi-trial likelihood in Eq. (6). This is fixable by either integrating the binomial likelihood over θ numerically or explicitly changing the model to a two-point mixture, but either way all experimental results need to be recomputed. The authors should also be asked to provide calibration or simulation evidence for the 'statistical robustness' claim. The paper is not ready for acceptance in its current form."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: the paper has a useful idea buried under a formal mistake. The Bayesian interval-ranking setup is genuinely new packaging—anchor models define ordered capability intervals, the prior is proportional to interval width, and the posterior gives the probability that a test model falls in each interval. That is a reasonable tool for practitioners who want more than a scalar accuracy. The authors also do several things right: they include the full 50-question set in the appendix, test sample sizes down to 5, and compare against accuracy, Pass@10, and mean±std.\n\nThe problem is the multi-trial likelihood. Eq. 4, for a single trial, averages the endpoint anchor probabilities. If θ is uniform in the interval and p_j(θ) is linear—their stated justification—that average is the marginal probability, so Eq. 4 is fine for O=1. But Eq. 6, for O>1, averages the endpoint binomial likelihoods. That is not the marginal of the stated model. The correct expression integrates the binomial likelihood over θ; Eq. 6 is the likelihood of a two-point mixture in which the model is either L_i or L_{i+1} with probability 1/2. The discrepancy is large at O=10, and all the Bayes@10 figures use O=10. So the central calibration claim—probabilistic rankings down to 20 questions—is computed from an unstated likelihood. The paper's own limitations section flags conditional query independence but not this.\n\nThe reader's concern about Eq. 4's linearity is valid but secondary. Even with exact linearity, Eq. 6 is wrong for O>1. There are also smaller issues: the query set is selected post hoc from early-2025 models, anchor probabilities are treated as known without uncertainty, and the method comparison is qualitative. None of those are fatal on their own.\n\nWho is this for? Practitioners who want cheap, uncertainty-aware model comparisons could use the single-trial version after a fix. The idea is worth a serious referee, but not as submitted. I would ask for a rewrite that states the generative model honestly, fixes Eq. 6 (or redefines it deliberately as a two-component mixture and says why), reports anchor uncertainty, and releases the response data so the calibration claim can be checked.\n\nRecommendation: send to peer review, but expect major revision. I would not cite the Bayes@10 numbers until the likelihood is corrected.","headline":"A useful Bayesian interval-ranking idea, undermined by an internally inconsistent multi-trial likelihood (Eq. 6) that all the headline Bayes@10 results depend on.","tokens_in":19396,"tokens_out":3417,"would_cite":false,"duration_ms":37642,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that a Bayesian interval posterior over anchor models yields statistically robust probabilistic LLM rankings with as few as 20 curated questions, resolving ambiguities that scalar metrics leave tied.","keywords":["Bayesian evaluation","LLM ranking","limited sample","capability intervals","anchor models","maximum entropy","probabilistic ranking","hypothesis testing"],"falsifier":"For each GPT-series anchor, hold that anchor out as a test model, rebuild the posterior from the remaining five anchors, and check whether the true interval is the modal one; a systematic failure identifies the midpoint assumption as the culprit.","tokens_in":18314,"feed_emoji":"📊","tokens_out":8348,"duration_ms":72086,"temperature":0.7,"pith_summary":"This paper argues that LLM evaluation should be framed as Bayesian inference over capability intervals rather than as a scalar accuracy score. Treating each model's latent capability as an unknown parameter, the method places a test model among a set of anchor models whose per-question success rates are measured in advance. The paper claims that with about 20 curated questions the resulting posterior yields stable, interpretable probability statements—for instance, a model's chance of falling between GPT-4 and o1—and that these rankings separate models that traditional metrics tie, such as QwQ-32B and Llama-4-Maverick. A reader should care because the approach offers calibrated confidence in small-sample, real-world evaluation settings.","feed_headline":"Bayesian ranking keeps LLM comparisons reliable at 20 questions","feed_subtitle":"Separates tied models like QwQ-32B and Llama-4-Maverick with probability statements instead of scalar scores.","key_machinery":"The load-bearing object is the interval posterior $\\Pr(\\theta_i < \\theta_x \\le \\theta_{i+1} \\mid \\mathcal{Q}=\\mathbf{q})$, which assigns the test model a probability of falling into each gap between consecutive anchor capabilities. The key identity is Eq. 4: the likelihood of a single answer given that the test model lies in interval $(\\theta_i,\\theta_{i+1}]$ is taken to be the arithmetic mean of the boundary anchor probabilities, justified by maximum entropy as the most neutral choice when $\\theta$ is assumed uniform inside the interval. The prior on the interval is then proportional to its width $\\theta_{i+1}-\\theta_i$ (Eq. 2), and the multi-trial likelihood (Eqs. 6 and 7) incorporates repeated Bernoulli draws. This machinery converts measured anchor success rates into a calibrated posterior over rankings.","core_discovery":"On the paper's own terms, the central claim is that the Bayesian interval posterior of Eq. 1, coupled with the maximum-entropy likelihood of Eq. 4 and its multi-trial generalization in Eqs. 6 and 7, provides statistically robust probabilistic rankings down to about 20 evaluation questions. The method treats the answer pattern on a curated query set as a binary readout of latent capability, and produces a probability distribution over seven intervals defined by six GPT-series anchors. Empirically, the paper reports that this procedure resolves performance ambiguities that accuracy, Pass@N, and mean-plus-standard-error reporting cannot, including separating QwQ-32B from Llama-4-Maverick despite overlapping mean estimates. It further claims that the posterior remains stable at 20 questions while degrading below 10, and that adding a test model to the anchor set refines the ranking of its neighbors.","pith_inferences":["The paper does not test the linearity of Eq. 4 directly; holding out anchor models and rebuilding the posterior would reveal whether the midpoint likelihood is the cause of misranking arms.","The prior is proportional to interval width, so anchor models clustered in capability space will produce weak priors; anchor selection is therefore part of the statistical model, not just experimental design.","The same binary-readout machinery could apply to other evaluations where anchor performance profiles are measurable, such as code correctness or preference judgments, with the curated query set replaced by the relevant task distribution.","The conditional-independence caveat raised in the limitations section could be checked empirically by grouping the 50 questions by domain and seeing whether posterior intervals shift when a domain is dropped."],"forward_implications":["Probabilistic ranking statements of the form 'the test model has a P% chance of lying between anchor i and i+1' remain decisive at M=20, with at least 65% posterior mass on the most likely interval.","The method separates models that scalar metrics tie, including QwQ-32B from Llama-4-Maverick, which accuracy and mean±std treat as comparable.","Multi-trial Bayesian reports are sharper than single-trial ones and keep the same category boundaries, while Pass@10 inflates accuracy and hides differences like Llama-4's inferiority to QwQ-32B.","Adding a test model to the anchor set can refine the ranking of nearby models, e.g., incorporating QwQ-32B would disambiguate it from DeepSeek-V3-0324.","The approach remains stable at M=20 questions and degrades only below M=10, where the peak interval probability falls below 50%."],"supporting_citations":[{"why":"Supplies the maximum-entropy principle used to justify the uniform prior (Eq. 2) and the midpoint likelihood (Eq. 4).","marker":"[Jaynes, 2003]"},{"why":"Provides the IRT/Rasch baseline whose distributional assumptions the paper's approach is designed to avoid.","marker":"[Rasch, 1993]"},{"why":"Defines the Pass@N baseline whose inflated accuracy the paper contrasts with Bayesian interval precision.","marker":"[Chen et al., 2021]"},{"why":"Represents the mean-plus-standard-error reporting baseline compared in the experiments.","marker":"[Blackwell et al., 2024]"},{"why":"Shows the cost of Elo-based head-to-head evaluation that motivates the anchor-prior alternative.","marker":"[Chiang et al., 2024]"},{"why":"Prior work on limited-sample LLM evaluation that the paper positions against.","marker":"[Polo et al., 2024]"},{"why":"Supplies 30 curated questions from SuperGPQA for the query set.","marker":"[M-A-P Team et al., 2025]"},{"why":"Supplies GPQA-Diamond questions used in the curated query set.","marker":"[Rein et al., 2023]"}],"fun_headline_variants":["Bayesian LLM rankings stay reliable with just 20 questions","20 questions enough for Bayesian LLM comparison","Bayesian probability beats scalar scores in LLM tests","Bayesian method separates tied LLMs on a 20-item quiz","Confident LLM rankings from small-sample Bayesian inference"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"Eq. 4 assumes that inside each capability interval the probability that a test model answers correctly is the simple average of the two neighboring anchor probabilities, which requires that correctness varies linearly with the latent capability within each interval.","fun_headline_variants_meta":{"raw":{"variants":["Bayesian LLM rankings stay reliable with just 20 questions","20 questions enough for Bayesian LLM comparison","Bayesian probability beats scalar scores in LLM tests","Bayesian method separates tied LLMs on a 20-item quiz","Confident LLM rankings from small-sample Bayesian inference"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000209,"raw_usage":{"total_tokens":1364,"prompt_tokens":860,"completion_tokens":504,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":476,"completion_tokens_details":{"reasoning_tokens":425}},"tokens_in":476,"tokens_out":504,"duration_ms":5395,"temperature":1.0,"reasoning_tokens":425,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T05:06:32.411291+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"For each GPT-series anchor, hold that anchor out as a test model, rebuild the posterior from the remaining five anchors, and check whether the true interval is the modal one; a systematic failure identifies the midpoint assumption as the culprit.","supporting_citations":[{"cited_title":"Probability theory: The logic of science","cited_arxiv_id":null,"evidence_quote":"Supplies the maximum-entropy principle used to justify the uniform prior (Eq. 2) and the midpoint likelihood (Eq. 4)."},{"cited_title":"MESA Press, Chicago, 50th anniversary edition, 1993","cited_arxiv_id":null,"evidence_quote":"Provides the IRT/Rasch baseline whose distributional assumptions the paper's approach is designed to avoid."}],"review_version":1}