{"id":"482413ce-46ea-40e5-9a0b-05c728772644","arxiv_id":"2504.12589","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":3,"one_line_summary":"BetaConform estimates LLM ensemble judgment accuracy from few labeled samples via a mixture of Beta-Binomial distributions, conformal-style adaptive stopping, and text-similarity prior transfer, but its theoretical guarantee is not rigorously established.","lead":"The paper proposes BetaConform, a method to estimate how often groups of LLM 'judges' give correct answers using only a handful of labeled examples, mixing statistical models, adaptive stopping, and transfer from similar datasets. A generalist reader might care because accurate LLM evaluation usually requires expensive human labels, and this promises to cut that cost.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Proposition 5.2's error bound does not follow from quantile stability; Eq. (11)'s rate has a sign error, so the 'theoretically guaranteed' claim is unsupported (and the 10-sample transfer headline is outside the conformal guarantee).","rationale":"The reader correctly identifies the large-sample quantile approximation as fragile, and that is part of the problem. But the more load-bearing defect is internal to the proof: Prop. 5.2's relative error bound on the vote-error rate would not follow even if all asymptotic quantile approximations were granted, because the argument silently converts a relative bound on E[S] into a relative bound on a tail probability through monotonicity. This is not a matter of finite-sample corrections; the proof step is invalid in the limit. The same proof also derives Eq. (11) from an Op arithmetic mistake: two Op(1/sqrt(r)) fluctuations do not cancel in a difference, so the claimed stopping-time formula, and the experimental claim that actual sample counts 'match the theoretical bound' (Section 8.3, RQ3), rest on an unsupported rate. The 10-sample transfer result in Table 2 is even more remote from the guarantee: the conformal stopping branch is bypassed entirely in the transfer experiments, and no separate bound is provided for r=10. In good faith, I credit the paper's empirical motivation and the clear demonstration that the vanilla Binomial assumption is inadequate; Figures 3 and 4 do show a two-peaked structure that a mixture of Beta-Binomials can capture. However, the central advertised contribution is the theoretical guarantee, and that guarantee is not established. Because the reader's verdict is already REJECT, my findings do not move the verdict; they add a more fundamental reason for the same rejection.","tokens_in":21103,"tokens_out":6023,"duration_ms":68196,"concrete_test":"Independently re-derive the chain from Eq. (21) to Eq. (41), and specifically test the monotonicity step: search over two-component Beta-Binomial mixtures with k=11 (or k=3) for a family where E[S] changes by about 1% while P(S<ceil(k/2)) changes by substantially more than 1% (e.g., a steep CDF near the threshold). If such a family exists, the step from Eq. (39) to Eq. (41) is invalid and Prop. 5.2 cannot be certified. A second, complementary check would simulate Eq. (10) at r around 40-60 and compare |q^r - q^{r-1}| with tau(1/sqrt(r-1)+1/sqrt(r)); if the empirical increments follow the sum rate rather than the difference rate, Eq. (11)'s sample-size formula is falsified.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that BetaConform delivers a 'theoretically guaranteed distribution estimation' (Abstract), with Eq. (11) determining the sample amount and Prop. 5.2 bounding the majority-voting error rate. Even granting the large-sample quantile approximation, the proof of Prop. 5.2 fails. Appendix A.2 first asserts a relative bound on E[S] (Eq. 39), then claims the same multiplicative bound holds for P(S<ceil(k/2)) solely because the Beta-Binomial CDF is monotone. Monotonicity preserves order, not ratios: a two-component Beta-Binomial whose mass sits near the voting threshold can have a small percentage change in E[S] yet a much larger percentage change in the tail probability. Thus Eq. (12) is not a consequence of the preceding argument, and the advertised guarantee has no quantitative content. Separately, Eq. (24) writes the quantile-increment rate as tau(1/sqrt(r-1)-1/sqrt(r)), but combining the two O_p(1/sqrt(r)) terms from Eq. (21) gives O_p(1/sqrt(r-1)+1/sqrt(r)); the order of a sum of Op terms cannot be reduced by cancellation. Therefore Eq. (11) and the derived r>=56 are also unjustified. Finally, the headline 3.37% error margin with 10 samples (Table 2) comes from the transfer branch, which uses no conformal stopping rule and is not covered by any stated finite-sample bound.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"BetaConform models the number of correct judgments in an LLM ensemble as a two-component Beta-Binomial mixture, fits it with an EM-style estimator, uses a conformal-prediction-inspired quantile-stability rule to decide how many samples to label, and optionally transfers mixture parameters from textually similar source datasets. The experiments compare error margins against Binomial and single Beta-Binomial baselines on ten benchmarks with five LLM families and report lower average margins, including a 10-sample transfer result on TruthfulQA with a 3.37% error margin. The paper's headline contribution is a theoretical guarantee: Eq. (11) determines the required sample amount and Proposition 5.2 bounds the majority-voting error rate under adaptive stopping.","tokens_in":21433,"tokens_out":9786,"duration_ms":98019,"significance":"The problem is practically relevant, and the empirical evaluation is reasonably broad: ten datasets, five LLM families, thirty repetitions, and ablations of the transfer design. The observation that a Binomial model understates the plateau in majority-voting error is well supported by the figures. However, the load-bearing claim is the theoretical guarantee, and that claim is not established. The derivation of Eq. (11) is invalid, the step from a bound on E[S] to a bound on the tail error probability in Proposition 5.2 does not follow, and the EM update in Eq. (7) is internally inconsistent. The 10-sample transfer headline is a favorable single cell that is not covered by any stated guarantee and is contradicted qualitatively by several other cells in Table 2. If the theoretical-claim language were removed, the paper would be a heuristic empirical study; as submitted, the advertised central contribution fails.","major_comments":[{"comment":"The derivation of the sample-size formula is invalid. Combining q^r = q + O_p(1/sqrt(r)) and q^{r-1} = q + O_p(1/sqrt(r-1)) gives q^r - q^{r-1} = O_p(1/sqrt(r) + 1/sqrt(r-1)), whose leading order is O_p(1/sqrt(r-1)); there is no cancellation that yields O_p(1/sqrt(r-1) - 1/sqrt(r)). Moreover, an O_p statement is a stochastic asymptotic order and does not imply a deterministic inequality of the form tau*(1/sqrt(r-1) - 1/sqrt(r)) <= xi with a fixed tau. Consequently Proposition 5.1 and Eq. (11) are not proved, and the claimed r >= 56 (or the r >= 51 stated in Section 8) has no theoretical basis.","section":"Appendix A.1, Eqs. (21)-(25)"},{"comment":"The step from a multiplicative bound on E[S] to the same multiplicative bound on P(S < ceil(k/2)) is invalid. Monotonicity of the Beta-Binomial CDF implies only that probability statements are ordered as the distribution shifts; it does not control the ratio of tail probabilities. A two-component mixture whose mass is concentrated near the voting threshold can undergo a small relative change in its mean while the tail probability changes by a much larger factor, so Eq. (12) does not follow from Eq. (39). The proof also never uses the finite-sample coverage property of conformal prediction, so the phrase 'theoretically guaranteed' in the abstract is unsupported.","section":"Appendix A.2, Eqs. (39)-(41), Proposition 5.2"},{"comment":"The M-step update is inconsistent with the mixture model. Both mixture components are updated using the same responsibilities gamma_i^1, whereas the second component should use gamma_i^2 = 1 - gamma_i^1; as printed, the algorithm cannot maximize the mixture likelihood. In addition, the E-step in Eq. (6) uses the Beta density evaluated at p_i = S_i/k rather than the Beta-Binomial probability of S_i, so it is not the EM for the model in Assumption 4.1. The authors should either correct the updates to a proper Beta-Binomial EM or explicitly state that the procedure is a heuristic approximation.","section":"Section 4.3, Eq. (7)"},{"comment":"The reported thresholds and the implied sample counts are inconsistent. Section 5 states that xi = 0.03 and tau = 25 lead to r >= 56, while Section 8 sets xi = 0.01 and says this requires at least r >= 51 samples. Substituting xi = 0.01 into Eq. (11) gives r approximately 116, not 51. The sample counts in Table 1 (roughly 40 to 61) therefore cannot validate Eq. (11) under the stated experimental configuration, and the text in Section 8.1 describing deviations of 3.14 to 12.86 samples from the 'estimated value 56' mixes two different threshold settings.","section":"Sections 5.1 and 8"},{"comment":"The claim that the transfer mechanism will not cause performance degradation when source and target datasets are distinct is contradicted by the reported numbers. For example, HalluDial with GPT-3.5 degrades from 16.62 to 23.79, JudgeBench with GPT-3.5 from 8.23 to 14.36, RewardBench with Qwen from 9.47 to 12.84, and LLMBar with GPT-4 from 12.90 to 17.70. The abstract's 3.37% result on TruthfulQA is a single favorable cell and is not representative of the average transfer margins in Table 2.","section":"Section 6 and Table 2"}],"minor_comments":[{"comment":"The paper is described as MAP estimation, but no prior distribution is defined; the EM updates in Eq. (7) are maximum-likelihood updates. The MAP interpretation should be made explicit or the terminology should be changed.","section":"Abstract and Section 4.3"},{"comment":"The Binomial rows appear to contain fewer values than the column headers imply, and the sample counts and standard deviations for the Binomial baseline are not reported. Please reformat the table or clarify the reported entries.","section":"Table 1"},{"comment":"The transfer hyperparameters alpha and beta are not specified anywhere in the paper, and no sensitivity analysis is reported for the sigmoid threshold embedded in Eq. (14). These values are needed for reproducibility.","section":"Section 6, Eq. (14)"},{"comment":"The same quantile-stability passage appears twice in Appendix A.2, once around Eqs. (29)-(38) and again around Eqs. (33)-(40); this is an editing error and obscures the absence of a real proof of Proposition 5.2.","section":"Appendix A.2"},{"comment":"The symbol S is used both for the random variable denoting the number of correct judgments and for the set of labeled samples; please disambiguate the notation.","section":"Section 3"}],"recommendation":"reject","confidential_remarks":"The paper has a useful empirical core, but the central advertised contribution is a theoretical guarantee, and that guarantee is not established: Eq. (11) rests on an invalid O_p cancellation, Proposition 5.2's bound does not follow from monotonicity, and the EM update in Eq. (7) is internally inconsistent. These are load-bearing errors rather than presentational issues. The paper could potentially be revised into an empirical study of a heuristic estimator, but that would require removing the 'theoretically guaranteed' framing and reworking the claims and experiments around the inconsistency in the reported thresholds. As submitted, rejection is appropriate."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Punchline: the paper has a genuinely useful empirical finding, but the advertised theoretical guarantee is not proven; the proofs in Appendix A have real holes, and the headline 10-sample transfer result is outside any stated bound. This should go to peer review, but it needs major revision, not acceptance.\n\nThe core observation is solid. Real LLM ensemble judgment distributions are two-peaked, and the vanilla Binomial assumption understates the error rate as the ensemble grows. Fitting a two-component Beta-Binomial mixture and using EM is a natural fix, and the experiments show consistent improvements over the Binomial in most cells. The embedding-similarity transfer is a reasonable heuristic, and the ablations in Table 5 suggest the design choices (log(ri), sigmoid, cosine similarity) actually matter.\n\nThe problems start when the paper claims a theoretical guarantee. Prop 5.2's error bound does not follow from the preceding argument. The proof first bounds E[S] multiplicatively, then asserts the same bound for P(S < ceil(k/2)) because the CDF is monotone. Monotonicity preserves order, not ratios; a mixture whose mass is concentrated near the voting threshold can have a small change in its mean and a much larger change in the tail probability. So Eq. (12) is unsupported. Separately, Eq. (24) treats the difference of two O_p terms as if the orders cancel, but a sum of O_p(1/sqrt(r-1)) and O_p(1/sqrt(r)) cannot be reduced to O_p(1/sqrt(r-1)-1/sqrt(r)). That makes Eq. (11) and the derived r>=56 unjustified. The authors also contradict themselves: Section 5 sets xi=0.03 and gets r>=56; Section 8 sets xi=0.01 and says r>=51, which cannot both be true with tau=25. The 10-sample transfer result (3.37%) comes from the transfer branch, which has no conformal stopping rule and no finite-sample bound at all.\n\nThere are smaller issues. Eq. (7) looks like a typo (the same gamma_i^1 is used for both mixture components). The transfer step degrades results in several cells, and on average it hurts GPT-3.5. No code or data is released, which makes the empirical numbers hard to check.\n\nWho is this for? People working on LLM-as-judge evaluation and anyone interested in efficient estimation of ensemble performance. The empirical finding is worth knowing, but the theoretical packaging should not be trusted. I would send this to peer review, with instructions to scrutinize the proofs, and I would ask the authors to either repair the guarantees or drop the word 'guaranteed' and present the method as a heuristic with supporting evidence. Release of code and data should be part of the revision. This is a revise-and-resubmit, not a desk reject.","headline":"The empirical finding that Binomial is a bad fit for LLM ensemble judgments is solid, but the theoretical guarantee advertised in the abstract is not proven and the supporting math has real holes.","tokens_in":21972,"tokens_out":5433,"would_cite":false,"duration_ms":52040,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A mixture of two Beta-Binomial distributions plus conformal stopping estimates an LLM ensemble judge's error rate from ten samples, with a 3.37 percent margin on TruthfulQA.","keywords":["LLM-as-a-judge","ensemble judgment distribution","Beta-Binomial mixture","conformal prediction","adaptive stopping","prior transfer","sample efficiency","maximum a posteriori estimation"],"falsifier":"Take a fully labeled dataset, fix the ensemble, and run BetaConform's adaptive stopping on many random subsamples; compute the true majority-voting error rate from all labels and check whether the fraction of runs whose estimate falls inside the Proposition 5.2 interval $(1 \\pm \\min(\\xi, \\tau/\\sqrt{r})) \\tilde{P}_{\\mathrm{BB}}$ is at least the nominal conformal confidence $1 - \\epsilon$. If coverage is lower, the claimed guarantee fails at the sample sizes the paper uses.","tokens_in":20904,"feed_emoji":"🎯","tokens_out":9541,"duration_ms":86519,"temperature":0.7,"pith_summary":"This paper aims to show that the reliability of an ensemble of LLM judges can be estimated accurately from very few labeled samples, by replacing the standard Binomial assumption with a more realistic distributional model. The central claim is that a mixture of two Beta-Binomial distributions, one for easy questions and one for hard ones, captures how the number of correct judgments is distributed across an ensemble, and that a conformal-prediction rule can decide when enough samples have been seen. On top of this, a text-embedding similarity mechanism transfers distribution priors from open-source datasets to sharpen estimates when annotations are scarce. If the claim is right, auditing an LLM judge on a new dataset costs tens of labeled examples rather than thousands, and the headline result is an error margin as small as 3.37 percent on TruthfulQA from ten samples.","feed_headline":"Ten samples gauge LLM ensemble judgment to 3.37% error","feed_subtitle":"A Beta-Binomial mixture plus adaptive conformal stopping cuts the annotation cost of auditing LLM judges.","key_machinery":"The load-bearing object is the mixture-of-Beta-Binomials model, where a Beta-Binomial count is a Binomial whose success probability is itself drawn from a Beta distribution, permitting overdispersion beyond what a single Binomial allows; the two components reproduce the empirical two-peak pattern and keep the majority-voting error rate from decaying monotonically with ensemble size. The second mechanism is the conformal-prediction stopping rule, which monitors the empirical $(1-\\epsilon)$-quantile of the nonconformity scores $|S_i - E[S]|$ and halts when the quantile moves by less than $\\xi$; the paper derives from this rule the sample-size formula $\\tau(1/\\sqrt{r-1} - 1/\\sqrt{r}) \\leq \\xi$ and the error-rate interval $(1 \\pm \\min(\\xi, \\tau/\\sqrt{r})) \\tilde{P}_{\\mathrm{BB}}$. Finally, the transfer weight $\\lambda_i = \\log(r_i) \\cdot \\sigma(\\alpha \\cdot (\\mathrm{CosSim}(\\bar{E}_0, \\bar{E}_i) - \\beta))$ mediates how much each source dataset's fitted parameters contribute to the target estimate, suppressing contributions from textually distant datasets.","core_discovery":"Stated on its own terms, the paper's discovery is that the judgment distribution of an LLM ensemble has two peaks, near all wrong and all correct, and is therefore modeled as $S \\sim w \\cdot \\mathrm{BB}(k, \\alpha_1, \\beta_1) + (1 - w) \\cdot \\mathrm{BB}(k, \\alpha_2, \\beta_2)$, a mixture of two Beta-Binomial distributions that stratify questions by difficulty. Instead of assuming a fixed per-judge accuracy, the mixture is fitted to labeled samples by an expectation-maximization algorithm, and the majority-voting error rate is read off the mixture's cumulative distribution. To answer, the paper borrows conformal prediction: with nonconformity scores $s_i = |S_i - E[S]|$, sampling stops when the $(1-\\epsilon)$-quantile changes by at most $\\xi$, which, by the asymptotic rate $O_p(1/\\sqrt{r})$ of quantile concentration, yields a required sample count $r$ satisfying $\\tau(1/\\sqrt{r-1} - 1/\\sqrt{r}) \\leq \\xi$ and an error-rate bound $(1 \\pm \\min(\\xi, \\tau/\\sqrt{r})) \\tilde{P}_{\\mathrm{BB}}$. When only ten samples are available, the estimate is improved by transferring distribution parameters from source datasets, weighted by $\\lambda_i = \\log(r_i) \\cdot \\sigma(\\alpha \\cdot (\\mathrm{CosSim}(\\bar{E}_0, \\bar{E}_i) - \\beta))$, so that textually similar datasets contribute more. The empirical demonstration includes a 3.37 percent error margin for a Llama ensemble on TruthfulQA with ten samples.","pith_inferences":["Not drawn in the paper: the same conformal stopping logic should apply to other ensemble statistics besides the count of correct judgments, since its nonconformity score only requires a stable expectation-like quantity.","The sigmoid transfer weight implies a natural rejection rule: below a similarity threshold the prior contributes almost nothing, so practitioners could decide a priori whether transfer will help for their target dataset.","The 3.37 percent figure is an empirical error margin, not a demonstrated consequence of the asymptotic bound, because the proofs rely on quantile concentration at sample sizes where the asymptotics may be unreliable; treating the guarantee as literal would require a finite-sample check.","A testable extension: datasets with more than two difficulty strata might be better fit by three or more Beta-Binomial components, and model-selection criteria such as BIC could decide the component count per dataset."],"forward_implications":["An audit of an LLM judge ensemble needs only tens of labeled examples, around 56 for the thresholds $\\xi = 0.01$, $\\tau = 25$, to reach the paper's stopping criterion, instead of full-dataset annotation.","The two-peak mixture predicts that adding more judges to an ensemble does not keep lowering the majority-voting error rate; the rate plateaus, contradicting the Condorcet-style conclusion drawn from a Binomial model.","The conformal stopping rule provides a concrete, threshold-driven answer to how many samples are enough, with sample counts scaling roughly as $\\tau(1/\\sqrt{r-1} - 1/\\sqrt{r}) \\leq \\xi$.","Prior transfer helps only when source and target datasets are textually similar; the sigmoid on embedding cosine similarity prevents distinct datasets from degrading the estimate.","Under the stopping threshold, the estimated majority-voting error rate stays within a relative factor $(1 \\pm \\min(\\xi, \\tau/\\sqrt{r}))$ of the true mixture error rate."],"supporting_citations":[{"why":"Foundational tutorial on conformal prediction that supplies the quantile-based, distribution-free logic behind the adaptive stopping rule.","marker":"Shafer & V ovk, 2008"},{"why":"Unified review of conformal prediction used to justify estimating sampling deviation without distributional assumptions.","marker":"Fontana et al., 2023"},{"why":"Provides the Condorcet jury-theorem reasoning that the paper argues is unrealistic for LLM ensembles, motivating the mixture model.","marker":"De Condorcet et al., 2014"},{"why":"HaluEval dataset used both for the two-peak distribution examination and as a central experimental benchmark.","marker":"Li et al., 2023a"},{"why":"TruthfulQA dataset behind the headline ten-sample, 3.37 percent error-margin result and the transfer experiments.","marker":"Lin et al., 2021"},{"why":"JudgeBench dataset used to demonstrate the Binomial model's failure and to evaluate the proposed estimator.","marker":"Tan et al., 2024"},{"why":"NV-Embed-V2 embedding model used to compute text-similarity transfer weights between source and target datasets.","marker":"Lee et al., 2025"}],"fun_headline_variants":["Two-peak Beta mixture cuts judge audit to 10 samples","Adaptive stopping reduces labels for LLM judge evaluation","Prior transfer sharpens LLM judge metrics from scarce labels","BetaConform: 10 samples, 3.37% error for LLM judge","Two-peak Beta-Binomial models LLM ensemble judgment"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the empirical quantile of the nonconformity scores converges to its population value at the $1/\\sqrt{r}$ rate promised by large-sample theory, even when only tens of samples, sometimes ten, are used; if convergence is slower in that regime, the stopping rule, the sample counts, and the error-rate bound all lose their footing.","fun_headline_variants_meta":{"raw":{"variants":["Two-peak Beta mixture cuts judge audit to 10 samples","Adaptive stopping reduces labels for LLM judge evaluation","Prior transfer sharpens LLM judge metrics from scarce labels","BetaConform: 10 samples, 3.37% error for LLM judge","Two-peak Beta-Binomial models LLM ensemble judgment"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000574,"raw_usage":{"total_tokens":2788,"prompt_tokens":1103,"completion_tokens":1685,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":719,"completion_tokens_details":{"reasoning_tokens":1597}},"tokens_in":719,"tokens_out":1685,"duration_ms":12489,"temperature":1.0,"reasoning_tokens":1597,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T12:27:41.716344+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a fully labeled dataset, fix the ensemble, and run BetaConform's adaptive stopping on many random subsamples; compute the true majority-voting error rate from all labels and check whether the fraction of runs whose estimate falls inside the Proposition 5.2 interval $(1 \\pm \\min(\\xi, \\tau/\\sqrt{r})) \\tilde{P}_{\\mathrm{BB}}$ is at least the nominal conformal confidence $1 - \\epsilon$. If coverage is lower, the claimed guarantee fails at the sample sizes the paper uses.","supporting_citations":[{"cited_title":"Throughout all the experiments, the sampling temperature of all LLMs is set to","cited_arxiv_id":null,"evidence_quote":"JudgeBench dataset used to demonstrate the Binomial model's failure and to evaluate the proposed estimator."}],"review_version":1}