{"id":"3799c240-9c5d-4532-b554-047ff1e0ee6b","arxiv_id":"2608.07419","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":7,"one_line_summary":"CALM uses bilevel optimization to tune per-vocabulary temperature-like logit adjustments during LLM fine-tuning, and reports improved out-of-domain calibration for aligned language models.","lead":"The authors propose CALM, a training-time calibration method that tunes per-vocabulary logit adjustments with a bilevel objective: the model is trained on its task loss while a held-out entropy objective discourages overconfidence. The paper reports lower expected calibration error across four aligned LLMs, with the strongest gains in out-of-domain question answering.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"OOD MCQA results may be inflated by target-domain hyperparameter selection; the paper never states whether Appendix C.3's conf-ECE-based grid search used the MCQA validation split.","rationale":"The reader's weakest_assumption lists two fragile premises: the full-vocabulary-to-four-option transfer and the hyperparameter-selection protocol. I agree with both but regard the second as more load-bearing because it directly threatens the 'particular advantages in out-of-domain generalization' claim. The BOME-inspired first-order scheme is mathematically coherent and the multi-seed disclosure is a point in the paper's favor; I do not see a fatal internal inconsistency in the method itself. However, if the OOD grid search used MCQA validation, then CALM's reported superiority over Regularization and CFT could reflect a larger hyperparameter search on the target benchmark rather than the bilevel entropy objective. The paper's phrasing in Appendix C.3 ('selected based on the lowest conf-ECE') makes this reading natural, and no alternative source-domain selection criterion is described. Because the concern is verifiable from the released code, a conditional verdict with a specific check is the appropriate outcome; if the check confirms leakage, the OOD headline claim would need substantial revision or re-analysis.","tokens_in":28353,"tokens_out":9362,"duration_ms":90667,"concrete_test":"Inspect the released repository's grid-search code to identify the split used for the OOD rows of Table 6. If it is the MCQA calibration split, rerun the OOD comparison with hyperparameters fixed by Alpaca-validation perplexity (or a single default setting shared by all methods) and recompute Table 1. If CALM's conf-ECE advantage over Regularization and CFT persists, the OOD claim survives; if not, the reported advantage is selection leakage.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central OOD claim rests on a model-selection protocol that may silently use the evaluation distribution. Appendix C.3 says hyperparameters are selected by lowest conf-ECE, and Table 6 reports OOD configurations with conf-ECE values; for the OOD MCQA setting, the only validation set on which conf-ECE is well-defined is the MCQA calibration split (the Alpaca split used for S_V has no four-option labels). If the lower/upper learning-rate grids were scored on that split, then CALM and its comparison methods are being tuned to the MCQA benchmark family before the 'zero-shot' test, and CALM's larger grid (12 configurations) versus Regularization (4) and fixed-alpha CFT gives it more chances to fit the target domain. The claimed advantage 'does not fit a particular test distribution' would then be an artifact of target-domain model selection. The paper does not state which split was used, so the central OOD result is not verifiable as written. A related but secondary gap is the missing explicit description of how per-vocabulary d_y,l_y are applied to the A/B/C/D subset; that affects reproducibility, while the selection-leakage threat directly undermines the empirical claim.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes CALM, a training-time calibration method for aligned LLMs. The method is formulated as a bilevel optimization problem: the lower level fine-tunes the model under a cross-entropy loss with per-vocabulary logit adjustments (multiplicative d_y and additive l_y, generalizing temperature scaling), while the upper level maximizes the entropy of the predictive distribution on held-out inputs. A first-order approximation inspired by BOME is used to avoid Hessian computation. The empirical section evaluates CALM against post-hoc temperature scaling, label smoothing, CFT, and two ablations (Regularization and Iterate) on four aligned LLMs, under multiple-choice QA in both in-domain and out-of-domain settings and under open-ended generative QA under domain shift, reporting conf-ECE, cw-ECE, and Sem-ECE. The central claim is that CALM produces well-calibrated LLMs with particular advantages in out-of-domain generalization, at a small accuracy cost and without test-time overhead.","tokens_in":28574,"tokens_out":6534,"duration_ms":55595,"significance":"If the empirical claims hold, the paper makes a useful contribution: it offers a training-time alternative to post-hoc temperature scaling, with an entropy-based upper-level objective that does not require labels, and a computationally tractable first-order bilevel scheme. The study is reasonably broad (four models, two task families, two calibration metrics plus a semantic ECE), and it explicitly reports seed variance, including honest acknowledgment that Llama-3.1 and Mistral-7B results are less stable. The paper also makes a concrete design claim that the bilevel coupling rather than entropy alone is responsible for the gains, which is testable. However, the central out-of-domain claims currently rest on two underspecified protocol details: the split used for hyperparameter selection, and the mapping from full-vocabulary per-token adjustments to the four-option MCQA logits. These issues are load-bearing for the paper's main narrative that calibration transfers without fitting the target distribution.","major_comments":[{"comment":"The manuscript never states which split is used for hyperparameter selection in the OOD MCQA setting. Appendix C.3 says the best configuration per model is selected by lowest conf-ECE while maintaining accuracy above 80% of baseline, but the only split in the OOD setup on which conf-ECE is well-defined for four-option MCQA is the MCQA calibration/validation split: the Alpaca S_V split described in Appendix C.7 has no A/B/C/D labels, so conf-ECE cannot be computed there. If the MCQA validation split was used for model selection, then the 'zero-shot' OOD claim in Section 5.2 is partly a target-domain model-selection outcome, and CALM's larger grid (12 configurations, Table 5) has more opportunities to fit that target than Regularization (4) or CFT, whose MCQA alpha is fixed. The authors must state exactly which split was used; if MCQA validation was used, they must re-run the comparison with hyperparameters selected on a split that does not overlap the OOD evaluation distribution and report whether the Table 1 advantages survive.","section":"Appendix C.3 and C.7"},{"comment":"The paper defines d_y and l_y as per-vocabulary logit adjustments over the full token vocabulary (e.g., 32,000 or 128,256 tokens), and the upper-level entropy objective is computed on the full next-token predictive distribution. The manuscript does not specify how these learned adjustments are mapped to the four extracted A/B/C/D logits in the MCQA evaluation: whether d_y and l_y are indexed by the token IDs of the option letters, whether the option logits are taken from vocabulary positions of 'A', 'B', 'C', 'D' with or without leading spaces, or whether a reduced class set is used. Without this mapping, Equation (2) cannot be instantiated on the MCQA setup, and the claimed transfer from full-vocabulary entropy training to four-way softmax calibration is not reproducible.","section":"Section 4, Eq. (2); Section 5.1"},{"comment":"The text claims that Regularization is 'exactly CALM's entropy objective as a grid-tuned single-level penalty' and that its failure at 'any grid point' demonstrates that the bilevel coupling is necessary. This is overstated: Regularization runs for 30 epochs in the OOD setting versus 5 for CALM, uses only a 2x2 grid ({1e-3, 1e-4} x {1e-5, 1e-6}), and applies the entropy penalty to the training set, whereas CALM's upper level evaluates entropy on held-out validation inputs. The comparison therefore does not hold constant the data split, epoch budget, or search density, and the quoted sentence overinterprets the ablation as evidence for the necessity of the coupling.","section":"Section 5.2; Appendix C.2"},{"comment":"For the generative OOD experiments in Table 3, the paper does not report which split was used to select CALM's hyperparameters or the CFT alpha sweep. Appendix C.3 describes selection by lowest conf-ECE, but conf-ECE is not defined for free-form generation, and Appendix C.7 states that Sem-ECE is computed on 300 evaluation questions with 20 samples each, without mentioning a validation partition. If the hyperparameters were chosen using Sem-ECE on the target-domain evaluation questions, the Table 3 comparison would again be affected by selection on the evaluation distribution. The authors should specify the validation protocol for the generative setting and confirm whether the reported CALM improvements persist under a fixed, non-overlapping selection procedure.","section":"Section 5.3; Appendix C.7"}],"minor_comments":[{"comment":"In the CALM configuration, 'Inner optimization steps: 20 batches per epoch for the lower-level surrogate ˆx' uses an undefined symbol \\hat{x}; this should refer to the surrogate model \\tilde{\\theta} used in Appendix A.","section":"Appendix C.2"},{"comment":"In the CFT paragraph, 'no Logit 2levelwrapper' appears to be a formatting artifact; it should read 'no logit-level wrapper'.","section":"Appendix C.2"},{"comment":"The label-smoothed (LS) row reports Sem-ECE values near 0.9-1.0. The caption calls this 'LS (collapsed)', but the main text does not explain what 'collapsed' means operationally; please clarify how LS is applied in the generative setting and why it yields near-maximal Sem-ECE.","section":"Table 3"},{"comment":"The scalar-versus-vector ablation is reported for a single seed on one model. Given the seed variance documented in Table 11 for Mistral-7B, the 0.0822 versus 0.3140 conf-ECE gap should ideally be accompanied by a multi-seed check, or the text should state that this is a point estimate.","section":"Appendix D.2, Table 12"}],"recommendation":"major_revision","confidential_remarks":"The deciding issue for me is the selection-split question in Appendix C.3/C.7: as written, the natural reading is that OOD MCQA hyperparameters were selected on the MCQA validation split, which would make the headline OOD advantage partly a model-selection artifact. If the authors can clarify that a non-overlapping split was used, or re-run the comparison under strictly held-out selection, the paper's central claim becomes defensible. The missing A/B/C/D mapping is also easily fixed with a precise description. Please weight the first issue heavily in the decision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: CALM is a plausible training-time calibration method, and the bilevel formulation is a genuinely useful generalization of temperature scaling. But the headline OOD MCQA result is not verifiable as written, because the paper never says which validation set is used for hyperparameter selection. If it is the MCQA calibration split, the 'zero-shot' claim partly evaporates.\n\nWhat is genuinely good: the entropy-maximizing upper level over a task-loss lower level, made tractable with a BOME-style first-order residual approximation, is a real methodological step and not just post-hoc TS with extra parameters. The ablations are well chosen: Regularization failing at every grid point supports the claim that the bilevel coupling, not entropy alone, does the work. They also release code and honestly report three-seed variance, showing Vicuna and OLMo stable but Llama and Mistral with large std—so the main-table margins for those two should be read cautiously, and the authors mostly acknowledge this.\n\nThe soft spots are proportionate but real. First and load-bearing: Appendix C.3 says hyperparameters are selected by lowest conf-ECE. In the OOD MCQA setup, training is on Alpaca, which has no four-option labels, so the only sensible place to compute conf-ECE is the MCQA validation split. If that is where the learning rates were chosen, then CALM, Regularization, and CFT are all being tuned on the target benchmark family before the 'zero-shot' test—and CALM's twelve grid configurations give it more chances to fit that family than Regularization's four. The paper never states which split was used, so the headline OOD generalization claim cannot be fully verified. The stress-test note is correct on this.\n\nSecond, the paper never explicitly specifies how the full-vocabulary d_y, l_y adjustments are applied to the A/B/C/D logits. That is a reproducibility gap, though likely minor if the code is clear.\n\nThird, the open-ended evaluation uses 300 questions and a self-cited Sem-ECE metric; worth a footnote but not a dealbreaker.\n\nWho this is for: people working on LLM calibration or training-time uncertainty methods. They should read it carefully, with an eye on the selection protocol. My recommendation: send to peer review—the core idea deserves referee time—but the revision must state the validation split explicitly, ideally report results with model selection done on Alpaca alone, and specify the logit mapping. As written, I would be skeptical of the OOD numbers until those details are pinned down.","headline":"A sensible training-time calibration method whose OOD headline numbers may reflect target-domain hyperparameter selection; worth serious review and a revision, but the zero-shot generality claim is overreaching as written.","tokens_in":29155,"tokens_out":2798,"would_cite":false,"duration_ms":24223,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that entropy-based bilevel fine-tuning, not post-hoc temperature scaling, keeps aligned LLMs calibrated when the domain shifts.","keywords":["LLM calibration","bilevel optimization","temperature scaling","entropy maximization","out-of-domain generalization","expected calibration error","preference alignment","overconfidence"],"falsifier":"Take the released code and one model from Table 1, record the learned per-vocabulary adjustments $(d_y, l_y)$, apply them only to the logits of the four answer tokens A, B, C, and D, and recompute confidence ECE on the MCQA test set. If the reported out-of-domain improvement (for example, Llama-3.1 from $0.1784$ to $0.1050$) does not reproduce under that restriction, then the claimed transfer from full-vocabulary entropy maximization to four-option calibration does not follow from the proposed mechanism.","tokens_in":28113,"feed_emoji":"🎯","tokens_out":18610,"duration_ms":130694,"temperature":0.7,"pith_summary":"Preference-aligned LLMs tend to be overconfident, and the standard remedy — fitting a temperature on a validation set after training — transfers poorly across domains. This paper argues the fix belongs in the training loop: fine-tune the model so that a temperature-like adjustment generalizes, by formulating a bilevel problem in which the lower level minimizes the ordinary next-token cross-entropy while the upper level tunes per-token logit adjustments to maximize the entropy of the predictive distribution on held-out inputs. The claim is that this entropy target, kept in check by the task-optimality constraint, moves overconfident aligned models toward confidence that matches accuracy — and does so where a single fitted temperature cannot, with the largest gains in out-of-domain multiple-choice and generative question answering. If the paper is right, calibration can be built into alignment-time fine-tuning instead of bolted on per dataset, at the price of a few hours of extra training and no inference-time overhead.","feed_headline":"Training-time entropy maximization beats post-hoc temperature scaling","feed_subtitle":"A bilevel loop tunes per-word logit adjustments so aligned LLMs stay calibrated under domain shift.","key_machinery":"The load-bearing object is a bilevel optimization whose upper objective is entropy maximization applied to a temperature-parameterized predictive distribution. The distribution uses per-vocabulary adjustments $(d_y, l_y)$ — a multiplicative rescaling $d_y := \\tau_y^{-1}$ of each token logit plus an additive shift $l_y$ — which generalize the single global temperature of temperature scaling and can redistribute confidence across answer options rather than only sharpen or flatten the whole distribution. The upper level maximizes predictive entropy on held-out inputs; the lower level requires the model parameters to remain optimal for the ordinary cross-entropy task loss. Second-order hypergradients are avoided by a first-order, value-function-based scheme that enforces lower-level optimality through a plug-in residual $\\hat{q} = L_{\\mathrm{tr}}(\\theta;\\alpha) - L_{\\mathrm{tr}}(\\tilde{\\theta};\\alpha)$ built from a short inner optimization, with the joint update direction $d = g_{\\mathrm{ent}} + \\lambda g_q$ combining the entropy gradient with an adaptive multiple of the residual gradient. The ablations identify that coupling term — not the entropy penalty alone — as what the calibration gains require.","core_discovery":"The paper's central claim is that maximizing the Shannon entropy of the predictive distribution, as the objective of an upper-level optimization, is the right training-time objective for calibrating aligned LLMs. Post-hoc temperature scaling is reframed as a special two-stage procedure — fit the model, then rescale logits — and generalized into a bilevel problem: the lower level trains the model under a parametric cross-entropy loss, and the upper level selects temperature-like hyperparameters $\\alpha = (l,\\tau)$, consisting of additive logit shifts and per-class temperatures, to maximize entropy on held-out inputs subject to the model remaining optimal for the task loss. Because miscalibration in aligned models is predominantly overconfidence, raising entropy moves the model from an overconcentrated regime toward a point where confidence matches accuracy, while the lower-level constraint stops the entropy objective from eroding discrimination. The reported experiments, on four 7-8B DPO- or RLHF-aligned models, show that this procedure (CALM) lowers confidence ECE and classwise ECE relative to the uncalibrated baseline and to training-time baselines including calibration-aware fine-tuning and label smoothing, with the clearest margins in out-of-domain multiple-choice QA and in cross-domain generative QA measured by semantic ECE, while preserving most of the task accuracy.","pith_inferences":["Because the upper-level entropy objective needs no ground-truth labels, a natural extension is to run CALM before preference alignment, on the base model with unlabeled text, to see whether it preempts the overconfidence that DPO and RLHF later induce; the paper only tests already-aligned models.","The large gap between scalar and per-vocabulary adjustments (confidence ECE 0.3140 for the scalar variant versus 0.0822 for CALM on Mistral-7B) suggests that aligned-model miscalibration is asymmetric across tokens, which invites testing whether the same vectorization helps other calibration objectives, such as classwise temperature scaling or binning-based losses, not just entropy maximization.","A stress test of the method's own domain-dependence would be to select CALM's hyperparameters on a validation split drawn from a different domain than the test set; if the out-of-domain gains shrink when validation and test domains are separated, some of the reported advantage may reflect tuning on the evaluation distribution rather than genuine transfer."],"forward_implications":["In the out-of-domain multiple-choice setting, CALM attains the best confidence ECE on three of the four models, cutting confidence ECE substantially relative to the uncalibrated baseline (Llama-3.1: $0.1784 \\to 0.1050$, about a 41% reduction) while keeping accuracy within about two percentage points.","Post-hoc temperature scaling is inconsistent under domain shift: on Llama-3.1 it raises confidence ECE from 0.1784 to 0.2513, whereas the training-time adjustments learned by CALM remain effective on unseen domains.","The ablations imply that the bilevel constraint itself does the work: a flat single-level entropy penalty (Regularization), even with its weight grid-tuned, and an alternating scheme without the coupling term (Iterate) both fail to match CALM, and Iterate can collapse accuracy.","In cross-domain free-form generation (trained on PopQA, tested on TriviaQA), CALM achieves the lowest mean semantic ECE ($0.0671$ versus $0.0846$ for the uncalibrated baseline), while label smoothing collapses under the shift and a tuning-budget-matched calibration-aware fine-tuning baseline still trails.","CALM preserves general capability: its out-of-domain-to-in-domain accuracy gap on five standard benchmarks is the smallest among training-based methods, and because all added machinery lives in training, inference cost is unchanged."],"supporting_citations":[{"why":"Supplies the first-order bilevel approximation scheme that CALM adapts to avoid second-order Hessian-inverse hypergradients at LLM scale.","marker":"(Liu et al., 2022)"},{"why":"Defines temperature scaling and the confidence-ECE metric; the post-hoc method CALM generalizes and the canonical baseline it must beat.","marker":"(Guo et al., 2017)"},{"why":"The calibration-aware fine-tuning (CFT) baseline, the closest training-time competitor, which CALM must outperform under a matched hyperparameter budget.","marker":"(Xiao et al., 2025a)"},{"why":"Supplies the semantic-sampling protocol and Sem-ECE metric used to measure calibration of free-form generative answers.","marker":"(Wang et al., 2026)"},{"why":"DPO, the alignment method whose overconfident outputs three of the four test models exhibit and that CALM is designed to recalibrate.","marker":"(Rafailov et al., 2023)"},{"why":"RLHF, the alignment method used for the fourth test model (Vicuna-7B) and the source of the overconfidence the method targets.","marker":"(Ouyang et al., 2022)"},{"why":"Supplies the Alpaca instruction dataset used for out-of-domain training in the primary multiple-choice evaluation.","marker":"(Taori et al., 2023)"},{"why":"Label smoothing, the training-time calibration baseline that CALM compares against and that collapses under generative domain shift.","marker":"(Müller et al., 2019)"}],"fun_headline_variants":["Bilevel entropy tuning beats post-hoc temperature scaling","Training-time entropy max beats post-hoc temperature scaling","LLM calibration: bilevel entropy beats post-hoc scaling","Entropy-maximizing bilevel loop calibrates aligned LLMs","Domain-robust calibration via bilevel entropy maximization"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The central premise is that teaching an already-aligned model on general instruction data to spread its next-word predictions more evenly will also make its four-option multiple-choice answers better calibrated, yet the paper never specifies exactly how the learned per-word adjustments are applied to the A/B/C/D options or why full-vocabulary entropy should transfer to a four-way softmax.","fun_headline_variants_meta":{"raw":{"variants":["Bilevel entropy tuning beats post-hoc temperature scaling","Training-time entropy max beats post-hoc temperature scaling","LLM calibration: bilevel entropy beats post-hoc scaling","Entropy-maximizing bilevel loop calibrates aligned LLMs","Domain-robust calibration via bilevel entropy maximization"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000933,"raw_usage":{"total_tokens":3997,"prompt_tokens":952,"completion_tokens":3045,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":568,"completion_tokens_details":{"reasoning_tokens":2976}},"tokens_in":568,"tokens_out":3045,"duration_ms":19259,"temperature":1.0,"reasoning_tokens":2976,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T14:27:17.302008+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take the released code and one model from Table 1, record the learned per-vocabulary adjustments $(d_y, l_y)$, apply them only to the logits of the four answer tokens A, B, C, and D, and recompute confidence ECE on the MCQA test set. If the reported out-of-domain improvement (for example, Llama-3.1 from $0.1784$ to $0.1050$) does not reproduce under that restriction, then the claimed transfer from full-vocabulary entropy maximization to four-option calibration does not follow from the proposed mechanism.","supporting_citations":[],"review_version":2}