{"id":"bd72ff70-e45d-4644-beb4-2120b93981e3","arxiv_id":"2501.03991","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"Multi-LLM response agreement plus focal loss in an auxiliary BERT confidence model improves LLM calibration across 12 models and four prompt styles.","lead":"A team trains a small BERT model to estimate how confident large language models should be, using answers from several LLMs and different prompt styles. They report that combining multi-model agreement with focal loss improves calibration, and that few-shot prompts work best.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Calibration scores are computed against binary labels from an unvalidated judge model; if those labels are noisy or biased, the claimed superiority of Calib-* over baselines may be an artifact of fitting the judge.","rationale":"The reader's weakest_assumption identifies the same load-bearing concern I would flag: the judge model's binary correctness labels are the foundation of every reported calibration score. If the judge is inaccurate or systematically biased, the ECE/Brier/AUC numbers do not measure calibration to true answer correctness, and the method rankings could be artifacts of fitting the judge rather than genuine calibration improvements. The paper cites Prometheus's general correlation with human evaluators, but that does not establish validity for this specific QA setting, judge prompt, and answer distribution. The self-reported limitation in Section 7 confirms that this risk is acknowledged but not addressed. I also note two secondary concerns that reinforce the need for a conditional verdict: the win-count aggregation lacks error bars or significance testing, and the Calib-n versus Calib-1 comparison is confounded by training-data volume since Calib-n sees n times more training answers. However, the judge-label validity issue is the most fundamental because it threatens all comparisons equally and is the easiest to underweight. A human-annotation study on a stratified subsample would settle whether the reported win counts survive a change of correctness labels. Since the reader already issued CONDITIONAL, my read does not alter that verdict.","tokens_in":61832,"tokens_out":6817,"duration_ms":72070,"concrete_test":"Draw a stratified random sample of about 50 test answers per dataset, prompt style, and model family (roughly 800–1000 cases), and have 2–3 human annotators apply the same semantic-equivalence rubric used in the judge prompt. Recompute all four evaluation metrics and the win-count aggregations using the human-majority correctness labels instead of the Prometheus labels. If the top method in Fig. 3c changes, or if any Calib-* method's lead over baselines shrinks to a margin that a binomial test would attribute to chance, the central claim is not supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3.2 defines ci = J(ai ==semantic y | q, y, ai) using Prometheus-8x7b-v2.0 (Eq. 1). Every ECE, ECE-t, Brier, and AUC value in Tables 1–8 and all win-count aggregations in Figs. 2–3 are computed against these ci. The paper's own Section 7 concedes that the optimal correctness-assessment method is underexplored, and no validation of Prometheus on this four-dataset QA setup is provided. This is load-bearing because Calib-* and APRICOT are trained to predict exactly these ci, while LLM Prob. and Verbalized % are not; calibration quality is therefore measured as agreement with the judge rather than with ground-truth semantic equivalence. If Prometheus is noisy or biased by answer length, prompt style, or model family, the auxiliary methods' advantage may be inflated, and the win-count ranking that supports the headline claims could change materially.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies factors influencing LLM calibration: inter-model response agreement, loss functions (BCE, focal, AUC surrogate), and prompt styles (Verbalized, Zero-shot, CoT, Few-shot). It proposes Calib-n, an auxiliary BERT-based model that takes joint question-answer strings from n target LLMs as input and is trained on judge-provided binary correctness labels, with Calib-1 as a single-LLM variant. Experiments cover 12 LLMs (2B-72B) and four open-ended QA datasets, comparing against LLM probabilities, Platt scaling, verbalized confidence, and APRICOT across ECE, ECE-t, Brier, and AUC. The headline claims are that Calib-n outperforms state-of-the-art baselines, that focal loss improves calibration and (FL)Calib-1 is best overall, that few-shot prompts are most effective for auxiliary methods, and that auxiliary models are robust to accuracy variations.","tokens_in":62029,"tokens_out":3978,"duration_ms":39261,"significance":"If the central claims hold, the paper makes a useful empirical contribution by broadening calibration evaluation to 12 LLMs and four prompt styles and by demonstrating a practical auxiliary-model recipe (response agreement plus focal loss plus few-shot prompts). The experimental surface is large: the paper ships extensive tables, covers five model families, and explicitly compares against a recent state-of-the-art method. However, the headline conclusions rest on win-count aggregations without statistical support, and the correctness labels that ground all metrics come from an unvalidated judge model. The claims are therefore not yet established at the level of certainty the paper suggests, though they are plausible and testable.","major_comments":[{"comment":"All calibration metrics in Tables 1–8 and the win counts in Figs. 2–6 are computed against binary correctness labels c_i produced by Prometheus-8x7b-v2.0. The paper provides no validation of this judge on the four datasets or against human semantic-equivalence judgments, and Section 7 states that the optimal correctness-assessment method is underexplored. If the judge is noisy or biased (e.g., by answer length, prompt style, or model family), then the apparent superiority of Calib-* methods—which are trained to predict these very labels—over LLM probabilities and verbalized confidence could be inflated. Please add a human-annotated validation sample with judge agreement, and ideally a robustness check using exact-match or ROUGE-based labels for the calibration metrics.","section":"§3.2, Eq. (1), §7"},{"comment":"The headline conclusions ('Calib-n outperforms the state-of-the-art baselines', '(FL)Calib-1 exhibits the best overall performance') are based entirely on win-count aggregation without error bars, confidence intervals, or significance tests. Since the underlying ECE/Brier values differ by small margins in many cells (e.g., Table 1, Gemma2-27b few-shot: (BCE)Calib-n ECE 0.088 vs (FL)Calib-n ECE 0.085), a single metric or split can change the count. Please report paired significance tests or bootstrap CIs over the 12 models × 4 prompts × 4 datasets, and make clear which aggregation choices drive each claim.","section":"§4.3, §5.1, Figs. 2–3"},{"comment":"The comparison between Calib-n and Calib-1 is confounded by training-set size. Calib-n is trained on n joint strings per question (one per target LLM), giving n·k training instances, whereas Calib-1 is trained on k instances for a single LLM. Any improvement from Calib-n could therefore reflect more training data rather than inter-model response agreement per se. Please include a controlled comparison in which Calib-1 is trained on the same number of instances (e.g., by replicating or augmenting single-LLM examples), or otherwise show that the agreement signal, not data volume, drives the result.","section":"§3.3, §4.2, §5.1"},{"comment":"The definition of ECE is non-standard and potentially inconsistent with the reported values: the text says ECE is a 'weighted ... average of the squared difference' between bin accuracy and confidence, whereas the standard ECE (Guo et al., 2017b) uses the absolute difference. If the squared variant was actually used, the ECE values are not comparable to prior work and the ranking across methods could differ. Please provide the exact binning formula and confirm whether squared or absolute differences were used, ideally with a reference to the implementation.","section":"§4.3"}],"minor_comments":[{"comment":"The phrase 'four open-ended quenstion-answering datasets' contains a typo; it should be 'question-answering'.","section":"§4.1"},{"comment":"The sentence starting 'Givek questions, we minimize' appears to be missing a space and a word; it should read 'Given k questions, we minimize'.","section":"§3.3"},{"comment":"The Brier score entry for Verbalized % under the Verb. prompt is reported as 113.606, which is outside the valid [0,1] range for a Brier score and appears to be a data-entry error.","section":"Table 2, Phi3-7b row"},{"comment":"The caption contains the typo 'response aggrement' and should read 'response agreement'.","section":"Fig. 1 caption"},{"comment":"Two sub-figure rows are both described as 'The second row of Fig. 6'; the dataset-specific panel is actually the third row.","section":"Appendix A.4"},{"comment":"The footnote 'Code and data will be released upon acceptance' prevents independent verification in the meantime; please provide an anonymous code repository or supplementary material link for review.","section":"Abstract and §1"}],"recommendation":"major_revision","confidential_remarks":"The stress-test concern about circularity is not supported by my reading: Prometheus is an external judge, and its judgments are not used to train the calibration model in a way that would make the calibration metrics circular. However, the judge-label validity concern is real and load-bearing. The paper is a broad empirical study, and the missing statistical validation and the Calib-n versus Calib-1 data-size confound are fixable within the manuscript's scope; hence major revision rather than rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First thing to know: this is a genuinely broad empirical study—12 LLMs from 2B to 72B, four prompt styles, four datasets—and the Calib-n idea of training a BERT auxiliary head on multiple LLMs' responses plus focal loss is a reasonable combination that others will want to try. The paper does useful work: it shows few-shot prompts help auxiliary-model calibration, that auxiliary methods are more stable than internal probabilities when accuracy varies, and that verbalized confidence is weak. The comparison against APRICOT is fair as far as it goes.\n\nThe soft spots are real but not fatal on their own. The largest is the correctness label. Every ECE, Brier, and AUC number is computed against binary judgments from Prometheus-8x7b, and there is no validation of that judge on these datasets. Calib-* and APRICOT are trained to predict those same labels, so their advantage over methods that never saw the judge could be inflated if the judge is biased by answer length, style, or model family. The authors concede in Section 7 that the optimal correctness-assessment method is underexplored, which is honest but doesn't fix the concern.\n\nSecond, the Calib-n versus Calib-1 comparison is confounded: Calib-n sees n times more training examples (one per answer per question), so the benefit attributed to response agreement could partly be a data-volume effect. A matched-data control would settle this.\n\nThird, the headline conclusions rest on win-count aggregation without error bars or significance tests. Many wins are marginal; a few flips could change the ordering. The 12-model sweep is impressive, but the summary statistics are fragile.\n\nNone of this is a kill shot. The paper is a solid empirical mapping of the calibration landscape, the writing is clear, and the limitations are stated. Code and data will presumably make it reproducible. For someone working on calibration or uncertainty, this is worth reading as a broad comparison, and the few-shot finding is practically useful. It deserves a serious referee: the issues are fixable with a judge validation study and a data-matched Calib-1 control. I'd send it out.","headline":"A broad, useful empirical sweep of calibration factors with a plausible but not fully proven central claim; the unvalidated judge model and a data-volume confound weaken the headline.","tokens_in":62558,"tokens_out":2452,"would_cite":true,"duration_ms":24738,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Confidence in an LLM answer is best obtained not from the LLM itself but from a small auxiliary model that sees several LLMs' responses, is trained on judge-scored correctness, and is optimized with focal loss.","keywords":["LLM calibration","confidence estimation","response agreement","focal loss","prompt styles","auxiliary model","expected calibration error","verbalized confidence"],"falsifier":"Have human annotators provide correctness labels on a sample of the test questions and recompute all method rankings on those labels; if focal Calib-1 and Calib-n no longer beat Platt-scaled LLM probabilities and APRICOT, or if the ordering changes when a different judge model is used, the central claim does not hold.","tokens_in":61643,"feed_emoji":"🎯","tokens_out":8361,"duration_ms":74683,"temperature":0.7,"pith_summary":"This paper tries to establish that a calibrated confidence score for an LLM's answer can be extracted more reliably by a text-only auxiliary model than by the LLM's own probabilities, its verbalized confidence, or existing post-hoc scaling, and it identifies which ingredients matter. The proposed Calib-n framework feeds a small transformer the question paired with answers from one or several LLMs and trains it to predict a judge model's binary correctness verdict; pooling several answers lets the estimator see inter-model agreement. Across 12 LLMs, 4 QA datasets, and 4 prompt styles, the paper reports that focal loss and response agreement improve calibration over BCE and AUC losses and over baselines including verbalized confidence, Platt-scaled probabilities, and APRICOT, with the focal Calib-1 variant winning most comparisons overall. It also finds that few-shot prompts help most, and that auxiliary-model confidence stays stable across accuracy levels while LLM internal probabilities and verbalized confidence become overconfident as accuracy falls. If correct, this gives a black-box-friendly route to calibrated confidence for LLM-based systems.","feed_headline":"Focal loss and shared answers sharpen LLM confidence","feed_subtitle":"A trained auxiliary model beats the LLMs' own confidence scores across 12 models and 4 prompt styles.","key_machinery":"The load-bearing object is the Calib-n auxiliary estimator: a small transformer (BERT-base-sized) with a 768-to-n classification head and sigmoid that maps n joint strings of the question plus each answer to per-answer probabilities, trained to match judge-provided binary correctness. Response agreement enters by giving the model all n answers at once, so it can learn that disagreement signals uncertainty. Three training losses are compared: binary cross-entropy, focal loss (which up-weights hard, high-loss examples, with the paper's default focal parameters), and an AUC surrogate loss that maximizes separation between correct and incorrect answers. The paper's conclusions rest on aggregate win counts across four calibration metrics and many model-dataset-prompt combinations.","core_discovery":"The central discovery is that calibration of LLM answers can be improved by an auxiliary confidence model trained on response agreement: concatenate the question with each candidate answer, run a BERT-base-sized transformer with a sigmoid head, and train it to predict whether a judge model, Prometheus-8x7b-v2.0, marked the answer correct. When responses from several LLMs are pooled, the model can detect low consensus and assign lower confidence; when the training loss is focal loss, the single-model variant (FL)Calib-1 is the best overall method by aggregate win counts across ECE, temperature-scaled ECE, Brier score, and selective-accuracy AUC. The paper also establishes that prompt style matters: few-shot prompts give the largest calibration gains for auxiliary models, and that the auxiliary methods' calibration is approximately flat across accuracy levels, unlike LLM probabilities and verbalized confidence, whose errors rise sharply as accuracy drops.","pith_inferences":["Because the correctness labels come from a judge model, part of Calib-n's edge may be learning the judge's notion of correctness rather than ground truth; a human-labeled subset would separate these.","The accuracy-dependent winner pattern suggests an adaptive system could estimate its own accuracy and switch between FL Calib-1, Calib-n with Platt scaling, and Platt-scaled LLM probabilities, a direct testable extension.","Since the auxiliary model consumes only text, the same Calib-n design should transfer to API-only or proprietary LLMs and to heterogeneous model mixes, and to open-ended tasks beyond the four QA datasets.","If few-shot prompts help mostly by increasing response agreement, one could ablate agreement directly by feeding identical answers versus varied answers to verify the proposed mechanism."],"forward_implications":["Confidence can be obtained for black-box LLMs with no access to weights or logits: only the text answers and a judge are needed.","Focal loss should be the default for training auxiliary confidence estimators; BCE and AUC losses trail it, and Platt scaling does not generalize to improve Calib-n.","Calibration studies should vary prompt style, because few-shot prompts are materially best for auxiliary methods and fixed-prompt conclusions can mislead.","A single static confidence method is not optimal across accuracy ranges: different methods win in low, mid, and high accuracy regimes.","Auxiliary confidence estimators stay calibrated when accuracy changes, making them safer for deployment on hard or out-of-distribution inputs."],"supporting_citations":[{"why":"Introduces expected calibration error, the primary evaluation metric used throughout the paper.","marker":"Guo et al. (2017a)"},{"why":"Provides the verbalized-confidence baseline and the temperature-scaled ECE-t metric used for comparison.","marker":"Tian et al. (2023)"},{"why":"Supplies APRICOT, the auxiliary-model baseline that Calib-* methods are compared against.","marker":"Ulmer et al. (2024)"},{"why":"Supplies Prometheus-8x7b-v2.0, the judge whose binary correctness labels define the training target.","marker":"Kim et al. (2024)"},{"why":"Motivates combining multiple model predictions to reduce overconfidence, the basis for response agreement.","marker":"Kim et al. (2023)"},{"why":"Defines focal loss, the loss function found to improve calibration in this study.","marker":"Lin et al. (2020)"},{"why":"Shows focal loss improves neural-network calibration, grounding its use here.","marker":"Mukhoti et al. (2020)"},{"why":"Provides the AUC surrogate loss used as the third training objective.","marker":"Yuan et al. (2021)"},{"why":"Supplies the BERT-base transformer backbone of the auxiliary model.","marker":"Devlin et al. (2018)"},{"why":"Defines Platt scaling, a baseline method and the post-hoc rescaling applied to Calib-n.","marker":"Platt (1999)"}],"fun_headline_variants":["Shared answers and focal loss refine LLM confidence","Consensus among models plus focal loss sharpens calibration","Auxiliary confidence model beats LLM's own scores","Few-shot prompts give auxiliary calibration its edge","Response agreement and focal loss boost calibration"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the judge model's binary semantic-equivalence labels are correct; if that judge is noisy or biased, every ECE, Brier, and AUC comparison is miscalibrated, and the paper's Limitations section concedes the optimal correctness-assessment method remains underexplored.","fun_headline_variants_meta":{"raw":{"variants":["Shared answers and focal loss refine LLM confidence","Consensus among models plus focal loss sharpens calibration","Auxiliary confidence model beats LLM's own scores","Few-shot prompts give auxiliary calibration its edge","Response agreement and focal loss boost calibration"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00018,"raw_usage":{"total_tokens":1294,"prompt_tokens":929,"completion_tokens":365,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":545,"completion_tokens_details":{"reasoning_tokens":294}},"tokens_in":545,"tokens_out":365,"duration_ms":4010,"temperature":1.0,"reasoning_tokens":294,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T21:41:51.949224+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Have human annotators provide correctness labels on a sample of the test questions and recompute all method rankings on those labels; if focal Calib-1 and Calib-n no longer beat Platt-scaled LLM probabilities and APRICOT, or if the ordering changes when a different judge model is used, the central claim does not hold.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Motivates combining multiple model predictions to reduce overconfidence, the basis for response agreement."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines Platt scaling, a baseline method and the post-hoc rescaling applied to Calib-n."}],"review_version":1}