{"id":"a9f85424-af74-4c60-8c23-3161e8ce26cc","arxiv_id":"2506.23492","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"SMART learns a per-sample temperature from the gap between the top two logits using a soft-bin ECE objective, achieving strong calibration with as few as 50 validation samples.","lead":"What if a neural network is too sure? This paper proposes SMART, a lightweight recalibration method that uses the margin between the top two class scores to adjust per-sample confidence, and reports strong calibration with only 50 validation samples.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Prop. 3.2 does not establish that the logit gap is sufficient: the bound in Eq. (19) depends on S, which encodes all other logits, so the claimed theoretical foundation for h_phi(g) is incomplete.","rationale":"The reader's weakest_assumption identifies exactly the same load-bearing concern: the method assumes the optimal per-sample temperature is approximately a function of the logit gap, but the paper's Proposition 3.2 only proves a bound that depends on the full logit vector through S. My stress-test sharpens this by showing the bound's endpoints are functions of S, not of g alone, and by noting that the proof fixes a single target confidence while the training objective is a binned SoftECE over all samples. This is a correctness risk in the theoretical narrative, not an internal inconsistency in the experiments. The empirical evidence in Tables 1, 5, and the data-efficiency curves is favorable, and the method is simple and reproducible in principle, so the paper remains CONDITIONAL pending either a corrected proof, a clear statement that the method is justified empirically, or an additional sufficiency test. Since the reader already assigned CONDITIONAL, I do not move the verdict; I recommend no change beyond the conditions the reader stated.","tokens_in":26913,"tokens_out":4732,"duration_ms":55367,"concrete_test":"Construct a synthetic check: fix p = 0.8 and g = 1 with K = 10. Use two logit vectors, both having zmax = 0 and z2nd = -1, but set the remaining 8 logits to -100 in case A and to -1.01 in case B. Numerically solve Eq. (11) for T in each case; if the two T values differ substantially, g alone is not sufficient and Eq. (19) is driven by S rather than by g. Then extend to a scatter: sample random K = 10 logits, compute the implied T range for each g, and report the conditional variance Var(T | g). If this variance is non-negligible, the 'tightly bounds' claim should be weakened or the paper should demonstrate empirically that SMART's MLP compensates for the missing S dependence. Finally, re-run the Table 5 logit-gap-vs-all-logits ablation under ImageNet-C with 50 validation samples to test sufficiency under shift.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central empirical claim is that a 49-parameter MLP mapping the top-2 logit gap g to a per-sample temperature achieves SOTA calibration. The theoretical support in Propositions 3.1 and 3.2 does not establish sufficiency of g. Eq. (11) asks for T satisfying exp(zmax/T)/sum_j exp(zj/T) = p. In Eq. (18), S = 1/p - 1 = sum_{j != M} exp((zj - zM)/T), and Eq. (19) gives -g/log(S) < T <= -g/log(S/(K-1)). The endpoints depend on S, which depends on every logit through the exponentials. Two samples with the same g can require very different T when the remaining K-2 logits differ, because S changes and the interval shifts. Proposition 3.1 actually shows T is unbounded with zmax fixed; Proposition 3.2 only constrains T to an interval whose location is controlled by the full logit vector. Thus g is not shown to be a sufficient statistic, and h_phi(g) is not theoretically guaranteed to recover the optimal per-sample temperature. The proof also fixes a single target confidence p, whereas the SoftECE objective in Eq. (10) sums over all bins and samples; the link from Eq. (11) to the actual training loss is not supplied. This does not disprove the empirical results, but it means the SOTA claim rests on the Table 1 numbers and the Table 5 input ablation, which is ImageNet-only and does not test whether residual dependence on other logits is negligible under distribution shift or 50-sample validation regimes.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes SMART, a post-hoc calibration method that predicts a per-sample temperature from the gap between the top-two logits. A small MLP maps the normalized gap to a positive temperature, and the model is trained by minimizing a soft-binned ECE loss on validation logits. The authors claim that SMART achieves state-of-the-art calibration on CIFAR-10/100, ImageNet-1K, and corrupted, long-tailed, and sketch-shifted variants, while using only 49 parameters and as few as 50 calibration samples. Theoretical support is offered by Propositions 3.1 and 3.2, which aim to show that the optimal temperature for reaching a target confidence is tightly bounded by the logit gap. The paper also reports ablations on the input signal, loss function, validation size, bin count, runtime, and additional metrics such as AdaECE and CECE.","tokens_in":27284,"tokens_out":6517,"duration_ms":65347,"significance":"The logit-gap design is a genuinely simple and appealing idea: it is scalar, prediction-preserving, and cheap, and the soft-bin objective is well matched to small validation sets. The empirical study is broad, covering multiple architectures, several distribution-shift scenarios, multiple metrics, seed-averaged results, and ablation tables. If the main-table rankings survive standard-size validation sets, this would be a practically useful method. However, the theoretical contribution does not establish that the gap is a sufficient statistic for optimal temperatures, and the headline claim of consistent superiority is contradicted by several Spline entries in Table 1. Credit is due for reporting parameter counts, runtime, bin sensitivity, and validation-size curves, which help the reader judge the method's trade-offs.","major_comments":[{"comment":"Proposition 3.2 does not establish sufficiency of g for T. The interval in Eq. (19) is -g/log S < T <= -g/log(S/(K-1)) with S = sum_{j != M} exp((z_j - z_M)/T), so the endpoints depend on the full logit vector and on T itself; two samples with the same gap can require very different temperatures. The paper should either prove a bound in terms of g alone, quantify the residual dependence on the remaining logits, or explicitly present the result as an empirical motivation rather than a theoretical foundation.","section":"§3.4, Eq. (19)"},{"comment":"The statement that SMART 'consistently outperforms' the baselines is not supported by Table 1: Spline beats SMART on ImageNet-1K ViT-B-32 (0.81 vs 0.87), on ImageNet-LT ViT-B-32 (0.72 vs 0.79), and on ImageNet-LT DenseNet-121 (0.79 vs 0.81). The abstract and Section 4.2 need qualification, and these counterexamples should be analyzed, for example by checking whether they correspond to particular gap regimes or class counts.","section":"Table 1 and §4.2"},{"comment":"The main ImageNet comparisons use a 50-sample validation set, which is 0.1% of the original 50,000-sample validation split. This protocol strongly favors the 49-parameter SMART over PTS, CTS, and Spline, whose parameter counts grow with the number of classes. The paper should add at least one main-table comparison with a conventional validation size, such as 2,000 or 5,000 samples, and should report how baseline hyperparameters were selected under the 50-sample budget. Until then, the SOTA claim is mostly a claim about the ultra-low-data regime.","section":"§4.1, Table 1"},{"comment":"The theoretical analysis solves a per-sample equation with a fixed target confidence p-hat, while the training objective SoftECE in Eq. (10) is a binned, sample-weighted average over all samples and bins. The paper does not show that a temperature satisfying Eq. (11) minimizes Eq. (10), nor how the target p-hat is derived during training. This is a missing link between the proof and the actual objective; at minimum, the theory section should be labeled as motivation.","section":"§3.3–3.4, Eq. (10) and Eq. (11)"}],"minor_comments":[{"comment":"Equation (19) uses n-1 while the class count is K elsewhere; the notation should be unified.","section":"Eq. (19)"},{"comment":"Section 3.3 uses b both as a bin index and as a bin width in the phrase 'setting b -> 0', which makes the limiting statement confusing.","section":"§3.3"},{"comment":"Section 4.1 says the validation set is '0.001' of the original validation set, but Figure 4 labels the axis '0.001%'; since 50 samples is 0.1% of 50,000, the text and the figure should be reconciled.","section":"§4.1 and Fig. 4"},{"comment":"In Table 6, the Spline accuracy for DenseNet-121 on ImageNet-C is 40.83%, identical to the Swin-B value and 19.6 points above the Vanilla baseline; this looks like a copy error and should be checked.","section":"Table 6"},{"comment":"Figure 2's axis labels appear as raw Unicode glyphs in the submitted PDF; these need to be regenerated for readability.","section":"Fig. 2"},{"comment":"References [4] and [36] both list a 'Frenkel' author, but the two entries appear to be different papers; the citations in Sections 2 and 4.2 should be checked for consistency.","section":"References [4] and [36]"}],"recommendation":"major_revision","confidential_remarks":"The current version is not ready for acceptance; the central SOTA claim is both contradicted by Table 1 and tested under a protocol that favors the method. I would encourage a revision that reframes the contribution as a data-efficient calibrator rather than a general SOTA method."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know about this one. The core idea is genuinely simple and empirically promising: feed the top-2 logit gap into a 49-parameter MLP that outputs a per-sample temperature, trained on soft-binned ECE. The ablation in Table 5 shows the gap is a better scalar input than entropy, max logit, or confidence, and the method holds up across CIFAR and ImageNet variants. That is real work and worth a look. But the theoretical section is a mess, and the paper overstates both its novelty and its consistency.\n\nWhat is new: as far as I can tell, no prior work uses the top-2 margin as the sole input for per-sample temperature. That is a legitimate contribution. The SoftECE objective is not new — it is Karandikar et al. (NeurIPS 2021) — and the abstract calling it “novel” is inaccurate. The “consistently outperforms” claim is also contradicted by a few Spline wins in Table 1 (e.g., ImageNet ViT-B-32, ImageNet-LT DenseNet-121). Those are minor overclaims, easy to fix.\n\nThe bigger problem is Section 3.4. Proposition 3.2 derives a bound on T for a fixed target confidence p, but as written the inequality is reversed. For p > 0.5, the case the paper says matters, the stated interval is empty: the left endpoint is larger than the right endpoint. The correct bound from the same algebra is -g/log(S/(K-1)) ≤ T < -g/log(S). That is a fixable typo, but it matters because the paper uses this proposition to claim the logit gap “tightly bounds” the optimal temperature. Even with the correct direction, the bound only gives a range whose width grows with K; it does not show T is a function of g alone. The sufficiency argument is therefore not established. The empirical ablation carries the paper, not the theory.\n\nOther concerns: the ImageNet experiments use 50 validation samples, which naturally favors a 49-parameter model over PTS or Spline. The data-efficiency story is plausible, but it should be discussed more carefully. The code link is anonymous, so I could not verify the experiments, and the hyperparameters are not fully specified.\n\nWho should read this: people working on post-hoc calibration will find the logit-gap idea useful and the empirical comparison informative. It deserves serious peer review, but the authors need to fix the sign error, drop the “novel SoftECE” claim, temper the “consistently outperforms” language, and ideally release code. I would send it back for major revision rather than reject.","headline":"A useful empirical calibration method with a broken proof; the logit-gap idea stands on its own, but the theory section needs a fix.","tokens_in":27831,"tokens_out":7637,"would_cite":true,"duration_ms":71489,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A 49-parameter calibrator keys off the top-two logit gap to beat temperature scaling.","keywords":["model calibration","temperature scaling","expected calibration error","logit gap","post-hoc calibration","sample-wise temperature","soft-binned ECE","data efficiency"],"falsifier":"Take a validation set and, for each sample, compute the oracle temperature that best recalibrates that sample by grid search. Split samples into groups with equal logit gap $g$ but different values of the sum of the remaining logits. If, within a fixed-$g$ group, the oracle temperature varies substantially across subgroups and a method that additionally uses the sum or $z_{\\max}$ achieves lower ECE than SMART, then $g$ alone is not sufficient and the central claim fails.","tokens_in":26705,"feed_emoji":"🌡️","tokens_out":6935,"duration_ms":69953,"temperature":0.7,"pith_summary":"SMART is a post-hoc calibration method: after a classifier is trained, it changes only the scale of the logits, never the predictions. Standard temperature scaling divides every sample's logits by one global temperature; SMART instead learns a small network that reads the gap between the largest and second-largest logit and outputs a per-sample temperature. The paper argues that this scalar gap is a denoised decision-boundary signal and proves that the temperature needed to reach a target confidence is tightly bounded by the gap, so the gap alone can drive recalibration. With a soft-binned ECE loss and 49 parameters, SMART reports lower calibration error than temperature scaling, class-wise scaling, parametric temperature scaling, and spline calibration across CIFAR-10/100, ImageNet-1K, and shifted variants, remaining stable with as few as 50 validation samples.","feed_headline":"One logit gap beats temperature scaling for calibration","feed_subtitle":"SMART's 49-parameter per-sample temperatures cut calibration error across CIFAR, ImageNet, and shifted sets.","key_machinery":"The load-bearing object is the logit gap $g_i = z_{i,\\max} - z_{i,2\\mathrm{nd}}$, a scalar derived from the top two logits, combined with the scalar-to-scalar regression $T = h_\\phi(g_i)$ implemented as a one-hidden-layer MLP with a $\\mathrm{softplus}$ output. A second component is the SoftECE objective, which replaces hard bin membership with soft weights $w_{i,b} \\propto \\exp(-\\alpha(\\hat p_i - c_b)^2)$; as $\\alpha \\to 0$ it becomes global accuracy and as bin width shrinks it approaches pointwise accuracy, so $\\alpha$ and bin count act as explicit bias-variance knobs. The theoretical carrier is Proposition 3.2, which bounds the temperature $T$ in terms of $g$, the target confidence $\\hat p$, and the class count $K$, justifying the use of $g$ as a control signal for recalibration.","core_discovery":"On the paper's own terms, the central discovery is that the margin between the largest and second-largest logit, $g_i = z_{i,\\max} - z_{i,2\\mathrm{nd}}$, is the right scalar input for sample-wise temperature scaling. The authors prove that the implicit temperature $T$ that moves the softmax probability to a target confidence $\\hat p$ is unbounded when only $z_{\\max}$ is fixed (Proposition 3.1), but is sandwiched by $g$: $-g/\\log(S) < T \\le -g/\\log(S/(K-1))$, where $S = 1/\\hat p - 1$ (Proposition 3.2). A one-hidden-layer network $T = h_\\phi(g_i)$ trained with the soft-binned SoftECE objective then achieves the lowest ECE and AdaECE in nearly all reported comparisons, with 49 parameters, exact accuracy preservation, and stable behavior down to 50 validation samples, including on ImageNet corruption, long-tailed, and sketch variants.","pith_inferences":["If the logit gap is as informative as claimed, the same scalar could be reused outside calibration, for example as a feature for selective prediction or out-of-distribution detection, since it already encodes decision-boundary proximity.","The paper's bound is an interval, not an equality, so the residual information in the other logits is not proven irrelevant; conditioning the temperature map on $z_{\\max}$ or the sum of the remaining logits may push ECE lower than the reported 49-parameter version.","The appendix's finding that high-gap samples are under-confident suggests SMART is effectively learning a margin-to-confidence correction; if that correction is largely architecture-independent, the same learned map might transfer across models without retraining.","Because SoftECE's weighting parameter $\\alpha$ and bin count are the explicit bias-variance knobs, adapting them to the validation-set size could extend the data-efficiency results beyond the fixed settings reported."],"forward_implications":["A calibrator whose parameter count does not grow with the number of classes is enough for large-scale tasks: 49 parameters serve ImageNet's 1,000 classes, whereas class-wise and spline calibrators scale linearly with class count.","Data-efficient recalibration becomes practical: the method reports stable calibration error with validation sets around 50 samples, a regime where full-logit parametric methods show high variance.","Calibration under distribution shift is attainable with a scalar margin signal: SMART stays stable on corrupted, long-tailed, and sketch inputs where spline calibration degrades sharply.","Accuracy is preserved by construction while confidence is repaired, because temperature scaling does not change the argmax; the reported tables confirm zero accuracy loss across all combinations.","SMART composes with training-time calibration losses: applying it on top of Brier, MMCE, label smoothing, or focal-loss training reduces ECE and AdaECE in nearly all table entries."],"supporting_citations":[{"why":"Defines temperature scaling and expected calibration error, the global baseline SMART is designed to beat.","marker":"[1]"},{"why":"Parameterized temperature scaling supplies the full-logit parametric baseline and the 91-parameter comparison.","marker":"[6]"},{"why":"Soft calibration objectives provide the SoftECE training loss and its bias-variance properties.","marker":"[8]"},{"why":"Spline calibration is the non-parametric baseline SMART compares against and that collapses under distribution shift.","marker":"[10]"},{"why":"Class-based temperature scaling is the per-class baseline whose C-parameter cost SMART avoids.","marker":"[36]"},{"why":"Reformulates calibration as regression of accuracy on confidence, the framing SMART builds its SoftECE derivation on.","marker":"[23]"},{"why":"Documents bin-size bias in ECE, which motivates SMART's bin-sensitivity analysis.","marker":"[37]"}],"fun_headline_variants":["Logit gap unlocks per-sample temperature scaling","SMART recalibrates with 49 parameters, no accuracy loss","Margin-aware temperature scaling beats uniform shifts","Single scalar logit gap gives robust calibration","Per-sample temperatures from top-two logit margin"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method collapses if the gap between the top two logits does not carry enough information to determine the right temperature change, because the network never sees the rest of the logit vector.","fun_headline_variants_meta":{"raw":{"variants":["Logit gap unlocks per-sample temperature scaling","SMART recalibrates with 49 parameters, no accuracy loss","Margin-aware temperature scaling beats uniform shifts","Single scalar logit gap gives robust calibration","Per-sample temperatures from top-two logit margin"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00066,"raw_usage":{"total_tokens":3053,"prompt_tokens":1017,"completion_tokens":2036,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":633,"completion_tokens_details":{"reasoning_tokens":1965}},"tokens_in":633,"tokens_out":2036,"duration_ms":16240,"temperature":1.0,"reasoning_tokens":1965,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T21:41:29.539939+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a validation set and, for each sample, compute the oracle temperature that best recalibrates that sample by grid search. Split samples into groups with equal logit gap $g$ but different values of the sum of the remaining logits. If, within a fixed-$g$ group, the oracle temperature varies substantially across subgroups and a method that additionally uses the sum or $z_{\\max}$ achieves lower ECE than SMART, then $g$ alone is not sufficient and the central claim fails.","supporting_citations":[{"cited_title":"On calibration of modern neural networks","cited_arxiv_id":null,"evidence_quote":"Defines temperature scaling and expected calibration error, the global baseline SMART is designed to beat."},{"cited_title":"Parameterized temperature scaling for boosting the expressive power in post-hoc uncertainty calibration","cited_arxiv_id":null,"evidence_quote":"Parameterized temperature scaling supplies the full-logit parametric baseline and the 91-parameter comparison."},{"cited_title":"Soft calibration objectives for neural networks","cited_arxiv_id":null,"evidence_quote":"Soft calibration objectives provide the SoftECE training loss and its bias-variance properties."},{"cited_title":"Calibration of neural networks using splines","cited_arxiv_id":null,"evidence_quote":"Spline calibration is the non-parametric baseline SMART compares against and that collapses under distribution shift."},{"cited_title":"Network calibration by class-based temperature scaling","cited_arxiv_id":null,"evidence_quote":"Class-based temperature scaling is the per-class baseline whose C-parameter cost SMART avoids."},{"cited_title":"the increased dimensionality introduces substantial noise for precise temperature parameterization,","cited_arxiv_id":null,"evidence_quote":"Documents bin-size bias in ECE, which motivates SMART's bin-sensitivity analysis."}],"review_version":1}