Combining Priors with Experience: Confidence Calibration Based on Binomial Process Modeling

Dong Pan; Haoyang Yu; Jinzong Dong; Zhaohui Jiang

REVIEW 5 major objections 5 minor 54 references

Combining Priors with Experience: Confidence Calibration Based on Binomial Process Modeling

T0 review · 5 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A three-parameter beta-derived curve can replace histogram binning for confidence calibration.

desk verdict Useful empirical recipe, but the headline guarantees (3/B sample efficiency, consistency) don't survive contact with the proofs or with the paper's own simulations. read the letter →

arxiv 2412.10658 v3 pith:A2ZICU5B submitted 2024-12-14 stat.ME cs.AIcs.LG

Jinzong Dong , Zhaohui Jiang , Dong Pan , Haoyang Yu This is my paper · ORCID

classification stat.MEcs.AIcs.LG MSC 62F1562G0562G20

keywords confidencecalibrationbinomialprocessbetadistributioncurvetrueerrorsampleefficiencymaximumlikelihoodestimationconsistentmeasure

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper proposes a confidence-calibration method that treats the sampling of correctness labels as a binomial process and estimates a continuous calibration curve by maximizing the binomial-process likelihood. The curve is drawn from a three-parameter function family derived from the assumption that confidence scores for correct and incorrect predictions follow $\beta$ distributions, so prior structure and empirical data enter through one optimization. The central claim is that this curve can be recovered from far fewer samples than histogram binning needs: roughly $\frac{3}{B}$ of the binning sample size, where $B$ is the number of bins. The same fitted curve yields a new calibration-error metric, TCE$_{\rm bpm}$, which the paper proves to be a consistent calibration measure. A sympathetic reader would care because this promises stable calibration curves and reliable error estimates in small-sample and low-density regions where binning becomes noisy or non-monotonic.

What carries the argument

The load-bearing object is the three-parameter function family $g(S;\alpha,\beta,c)=\frac{1}{1+S^{-\alpha}(1-S)^{\beta}e^c}$, obtained by applying Bayes' rule to $\beta$-distributed confidence scores for wrong and right predictions. The family is the hypothesis space whose closure contains the true calibration curve under the $\beta$ assumption; its three parameters make three observations sufficient to determine it via linear equations in $\log(1/g-1)$; and its monotone shape repairs the non-monotonic spikes that histogram binning produces in low-density regions. Around this family, the paper builds two devices: an equivalent convex objective, $\sum_b P(b)\exp\left((g(\hat{S}_b;\theta)-\frac{N^{\rm pos}_{\hat{S}_b}}{N_{\hat{S}_b}})^2\right)$, that replaces the non-convex binomial likelihood, and a moment-estimated $\beta$ density for confidence scores that converts the fitted curve into the integral metric TCE$_{\rm bpm}$.

What would settle it

Simulate data from the paper's own non-beta true distributions D2 or D3, which use log and log-flip transforms, and run Algorithm 1 at sample sizes satisfying Theorem 4; if the mean absolute error between the fitted curve and the true curve exceeds the theorem's bound, the beta-family premise fails. Alternatively, on a real logit dataset, compare the fitted curve against a high-resolution nonparametric estimate such as isotonic regression or many-bin histogram to see whether deviations beyond the claimed error tolerance appear.

Watch

Extended reading notes

Core claim

The paper's central discovery is that a calibration curve of the form $$g(S; \$\alpha$, \$\beta$, c) = \frac{1}{1 + $S^{{-\alpha}}$(1-S)^{\$\beta$} e^c}$$ — exactly the posterior $P(H=1|S)$ that results when $P(S|H=0)$ and $P(S|H=1)$ are $\beta$ distributions — can be estimated by maximizing the binomial-process likelihood, and that three distinct observations suffice to identify its three parameters. Under the $\beta$ assumption, the estimated curve is Lipschitz continuous with respect to the data distribution in Wasserstein distance, and Theorem 4 states the sample bound $N \geq \frac{3\ln(1/\delta)}{2\varepsilon^2}$ for expected absolute error at most $\varepsilon$ with probability $1-\delta$. Because the bound does not contain the bin count $B$, the advantage over histogram binning, whose bound is $\frac{B\ln(1/\delta)}{2\varepsilon^2}$, grows as $B$ grows. The associated metric TCE$_{\rm bpm}$, defined as $\int_0^1 |g(S;\theta_D)-S|\,\xi_D(S)\,dS$ with $\xi_D$ a moment-estimated $\beta$ density, is shown to satisfy robust completeness and soundness, i.e., it is a consistent calibration measure whenever the true curve lies in the family.

Load-bearing premise

The guarantees hold only when the true calibration curve has exactly the three-parameter beta-derived form, which is equivalent to requiring that confidence scores for correct and incorrect predictions both follow beta distributions.

Editorial extensions

If this is right

Calibration curves can be estimated with sample budgets that do not grow with the number of bins, making low-density confidence regions usable without binning them away.
TCE$_{\rm bpm}$ is a consistent calibration measure: it tends to zero as a distribution approaches perfect calibration and stays large when the distribution is far from calibration, in the Wasserstein metric.
The sample-size requirement is $\frac{3}{B}$ of histogram binning's requirement, so the advantage widens as the number of bins increases.
Binomial-process simulation provides ground-truth calibration datasets, enabling direct comparison of any calibration metric against the true calibration error.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Because Theorem 4's guarantee depends on the beta-family assumption, the practical value of the method on real logit distributions will hinge on how close those distributions are to beta; the paper does not test this directly, so a natural next experiment compares the fitted curve with a nonparametric oracle on real data.
Since three representative observations determine the curve, an active or quantile-based rule for picking three bins could reduce the computational cost of the Bayesian binning average, a step the paper explicitly leaves open.
The same binomial-process simulator could benchmark calibration methods as well as metrics, separating model bias from estimation variance under a known ground truth.
The metric's integral form suggests a family of binning-free calibration measures obtained by varying the confidence-density estimator, with TCE$_{\rm bpm}$ as one instance.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Short version: the method itself is usable — it is basically beta calibration with a reweighted exponential-squared loss and binning-scheme averaging, plus a clean way to simulate calibration benchmarks. The real-data plots and simulated benchmarks suggest it behaves reasonably. But the theory sections are not sound, and the claims advertised in the abstract do not follow from the proofs.

What is genuinely new: the simulation generator (Algorithm 3) is a nice idea, and the TCE_bpm integral estimator is a natural use of the fitted curve. The binning-average objective is a small but real variation on existing fitting methods. The paper correctly points out that beta calibration is the closest competitor, but then omits it from Table 1, which makes the empirical advantage unquantified — that is a concrete omission.

Soft spots, in order:

1. The MLE/exp-squared equivalence (Appendix A.1) is not established. Maximizing a binomial likelihood and minimizing a squared exponential difference are not equivalent objectives; the appendix only differentiates a single term and asserts the rest.

2. Theorem 4's sample-efficiency proof has a missing Lipschitz constant. Lemma 1 is fine — three points determine three parameters — but the Hoeffding uniform bound gives E|g - ĝ| ≤ L·ε, not ε, and the union bound over locations is not handled. The stated 3 ln(1/δ)/(2ε²) bound is not proven.

3. The consistency result (Corollary 1) is conditional on the true curve lying in the three-parameter family, which is exactly assuming P(S|H=0) and P(S|H=1) are beta. The paper's own D2 and D3 benchmarks use curves outside this family (logflip and log forms), so the guarantees do not apply to the validation. Section 3.2 itself warns about wrong families, and Section 6 admits the 'three most representative bins' implementation is future work — so Theorem 4 is not even connected to Algorithm 1.

4. Theorem 3's robust soundness proof constructs a 'perfectly calibrated' D' by replacing S with P(H=1|S); that new distribution is not generally calibrated, so the argument does not go through.

None of these are typo-level issues; they are load-bearing for the advertised contributions. The empirical work might survive without the theory, and the paper could be revised into a useful applied calibration paper if the theory is redone or removed. But as it stands, the abstract and conclusion overclaim.

Who this is for: anyone working on calibration might read the empirical sections for the benchmark generator, but not for the theoretical guarantees. I'd send it to a serious referee — the empirical recipe is worth engaging with — but I would expect major revision, with the theory either fixed or cut.

Referee Report

5 major / 5 minor

Summary. The paper proposes a post-hoc confidence calibration method that models the sampling of calibration data as a binomial process and estimates a continuous calibration curve in the three-parameter family g(S; alpha, beta, c) = 1/(1 + S^{-alpha}(1-S)^beta e^c). The authors claim that this estimator is Lipschitz continuous with respect to the data distribution, that it requires only 3/B of the sample size needed for histogram binning, and that the induced metric TCE_bpm is a consistent calibration measure in the sense of Blasiok et al. They also propose a simulation scheme (Algorithm 3) for generating calibration data from a prespecified true curve and confidence distribution, and they report experiments on ten real logit datasets and five simulated distributions.

Significance. If the theoretical guarantees were valid, the paper would make a useful contribution: a continuous calibration curve with a small parametric family, a sample-efficiency improvement over binning, and a calibration metric with consistency guarantees. The simulation benchmark is a sensible idea and the code is made available. However, the central theoretical claims are not established. The equivalence between the stated maximum-likelihood objective and the actually optimized objective is unproven, the sample-efficiency proof has gaps, and the consistency theorem relies on a beta-distribution assumption that the paper's own simulated benchmarks violate. The omission of beta calibration, the closest existing functional form, from the empirical comparison further weakens the evaluation. The method may have heuristic value, but the advertised guarantees are not supported by the manuscript.

major comments (5)

[Section 3.1 and Appendix A.1] The claim that Eq. 9 is equivalent to Eq. 8 is not proven. Appendix A.1 differentiates a single binomial probability term (Eq. 22) and notes that its maximum over g occurs at the empirical frequency, but this does not establish that the argmax over theta of the averaged binomial likelihood in Eq. 8 coincides with the argmin over theta of the weighted exponential-squared objective in Eq. 9. These are different functionals of theta, and no argument is given that their global optimizers match. Since Algorithm 1 actually minimizes Eq. 9, the abstract's statement that the method maximizes a binomial-process likelihood is unsupported.
[Appendix A.6, Theorem 4] The sample-efficiency bound is not proven as stated. The proof applies per-location Hoeffding bounds (Eq. 48) and then writes N = N' * N_hatS, but the theorem statement claims a total-sample bound N >= 3 ln(1/delta)/(2 epsilon^2) with no dependence on the number of locations N' and no mechanism for selecting the three representative locations. Equation (50) also drops the Lipschitz constant L in passing from L * E|...| to L * epsilon, and the theorem statement then asserts E|g - P| <= epsilon, which would require L = 1, never established. Moreover, Section 6 admits that selecting the three most representative bins is future work, so the guarantee does not apply to the implemented Algorithm 1, which uses all bins. The headline '3/B' claim is therefore not supported.
[Corollary 1, Section 4, and Appendix B.1] The consistency and sample-efficiency guarantees require P(S|H=0) and P(S|H=1) to be beta distributions, which is equivalent to assuming the true calibration curve lies exactly in the family Eq. 12. The empirical evaluation violates this condition: the simulated curves D2 (1 - e^{-0.12}(1-S)^{0.58}) and D3 (e^{-0.03} S^{1.27}) are not of the form in Eq. 12, and Algorithm 3, which samples S from a beta and H from Bernoulli(g_true(S)), generically induces non-beta class-conditional distributions. Thus the theoretical guarantees do not cover the validation experiments, and the model bias is unquantified. Section 3.2 itself warns that a wrongly selected family puts the truth outside the feasible region, but no real-data test of the beta-family assumption is reported.
[Appendix A.4, Theorem 3] The proof of robust completeness contains unjustified steps. The constructed dataset D* uses non-integer counts N_hatS * (P(H=1|S) + epsilon/2), which is not a valid realization of a binomial process. The Hoeffding inequality in Eq. 36 gives per-location concentration, but the subsequent bound on the maximum over all S in Eq. 38 is not derived from a valid union bound, and the Lipschitz constant is applied without specifying a coupling. Because Theorem 3 underlies Corollary 1, the consistency claim is not proven.
[Section 5.3, Table 1] The most directly relevant baseline, beta calibration (Kull et al. 2017a; 2017b), is cited in the related work but is omitted from Table 1 and from the metric comparisons in Section 5.2. Since the functional form in Eq. 12 is the beta-calibration map, the empirical advantage over this close competitor is unquantified, and the claim of a novel prior-integration approach is weakened without this comparison.

minor comments (5)

[Throughout] The metric name is typeset inconsistently as 'T CEbpm' and 'TCE_bpm'; please unify the notation.
[Appendix A.2 and A.3] There are repeated typos: 'Lipchitz' should be 'Lipschitz'. The appendix headings 'The Proof of Theorem 1' and 'The Proof of Theorem 2' do not match the theorem numbering in the main text, which labels the proofs as A.2 and A.3.
[Appendix A.6, Lemma 1, Eq. (45)] In the linear system for log(1/g - 1), the coefficient of beta should multiply log(1 - S*_1), not log(S*_1) as written; the displayed equation incorrectly has beta·log(S*_1) twice.
[Table 3] The entry 'ECE debaised' should read 'ECE debiased'; also the reference for ECE sweep is listed as 'AISTA TS' but the cited work is from AISTATS.
[Algorithm 1] The pseudocode is ambiguous: it computes P(D|g) over binnings but does not specify how the argmin over alpha, beta, c is updated within the loop; the actual optimization (Nelder-Mead) is only described in Appendix B.2.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the sample-efficiency and consistency results are explicit conditional theorems under the stated beta class-conditional model, not conclusions assumed in their premises; the self-citations are not load-bearing.

full rationale

The paper's advertised guarantees are conditional in the body. Theorem 4 begins "For the function family in Eq. 12, if P(Ŝ|H=0) and P(Ŝ|H=1) follow beta distribution..." and Corollary 1 uses exactly that premise to show that the hypothesis class G contains the true curve. This is a legitimate theorem with an explicit assumption, not a hidden identification of output with input: the proof supplies a parameter-counting Lemma 1 and a concentration argument rather than merely restating the conclusion. Theorem 3's first condition, "The hypothesis set G includes the true calibration curve," is likewise an explicit premise, and Corollary 1 derives it from Eq. 10. The paper even warns in Section 3.2 that if the prior family is not correctly selected, the true calibration curve will not be in the feasible region, which confirms the conditional status of the guarantees. The D2 and D3 benchmark curves that fall outside Eq. 12 are a model-misspecification or robustness issue for the empirical demonstration, not circularity, because the theoretical claims are not asserted to hold for curves outside the stated family. Section 6's admission that selecting the three representative bins of Theorem 4 is future work is a proof-implementation gap, not a circularity. Self-citations (Dong et al. 2024; Jiang et al. 2023) appear only as related-work context and do not carry any derivation. The Eq. 12 family coincides with beta calibration from Kull et al. 2017b, which the paper cites for the beta-distribution premise; reusing a cited external functional form is a novelty-attribution concern, not a self-referential derivation. No step in the claimed derivation chain reduces to its own input by construction.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

The central theoretical claims rest on a three-parameter family that is fitted to the data, plus beta-distribution and monotonicity assumptions that are not verified. The TCEbpm metric adds two more moment-fitted parameters. No new physical or conceptual entities are introduced.

free parameters (5)

alpha = fitted per dataset
Shape parameter in g(S; alpha, beta, c) controlling the power of S; fitted by minimizing Eq. 9.
beta = fitted per dataset
Shape parameter in g(S; alpha, beta, c) controlling the power of 1-S; fitted by minimizing Eq. 9.
c = fitted per dataset
Log-odds offset in g(S; alpha, beta, c); fitted by minimizing Eq. 9.
Beta density parameters a1, a2 = moment estimates from data
Used to approximate the confidence score density xi(S) in TCEbpm via moment matching in Algorithm 2.
Number of bins B = range [N/100, N/20]
Chosen by hand as equal-mass binning scheme; affects the objective in Eq. 8 and Eq. 9.

assumptions (5)

domain assumption P(S|H=0) and P(S|H=1) follow beta distributions.
Used to derive the parametric family Eq. 12 via Bayes' theorem and explicitly assumed in Corollary 1 and Theorem 4.
domain assumption The true calibration curve belongs to the function family G defined in Eq. 12.
Required by Theorem 3's first condition and by Lemma 1; the paper does not establish it for real data, and simulated distributions D2 and D3 may violate it.
domain assumption The calibration curve is monotonically increasing, enforced by alpha >= 0 and beta >= 0.
Section 3.2 invokes monotonicity from prior literature; Algorithm 1 optimizes alpha, beta, c without describing constrained optimization, so the assumption may not be enforced.
ad hoc to paper The argmin in Eq. 9 is unique and the map from data distributions to theta is Lipschitz.
Theorem 1's proof assumes such regularity without proving it; this premise is needed for all downstream continuity and consistency claims.
standard math Standard Hoeffding bounds apply to binned hit counts and the loss integrates over the fitted curve.
Used in the proofs of Theorems 3 and 4 to control sampling error.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Combining Priors with Experience: Confidence Calibration Based on Binomial Process Modeling." pith.science (2026). https://pith.science/paper/A2ZICU5B

@misc{pith2026241210658,
  author       = {Pith},
  title        = {Pith review of: Combining Priors with Experience: Confidence Calibration Based on Binomial Process Modeling},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/A2ZICU5B}},
  note         = {Machine review of arXiv:2412.10658}
}

abstract

Confidence calibration of classification models is a technique to estimate the true posterior probability of the predicted class, which is critical for ensuring reliable decision-making in practical applications. Existing confidence calibration methods mostly use statistical techniques to estimate the calibration curve from data or fit a user-defined calibration function, but often overlook fully mining and utilizing the prior distribution behind the calibration curve. However, a well-informed prior distribution can provide valuable insights beyond the empirical data under the limited data or low-density regions of confidence scores. To fill this gap, this paper proposes a new method that integrates the prior distribution behind the calibration curve with empirical data to estimate a continuous calibration curve, which is realized by modeling the sampling process of calibration data as a binomial process and maximizing the likelihood function of the binomial process. We prove that the calibration curve estimating method is Lipschitz continuous with respect to data distribution and requires a sample size of $3/B$ of that required for histogram binning, where $B$ represents the number of bins. Also, a new calibration metric ($TCE_{bpm}$), which leverages the estimated calibration curve to estimate the true calibration error (TCE), is designed. $TCE_{bpm}$ is proven to be a consistent calibration measure. Furthermore, realistic calibration datasets can be generated by the binomial process modeling from a preset true calibration curve and confidence score distribution, which can serve as a benchmark to measure and compare the discrepancy between existing calibration metrics and the true calibration error. The effectiveness of our calibration method and metric are verified in real-world and simulated data.

Figures

Figures reproduced from arXiv: 2412.10658 by the authors.

**Figure 2.** Visualization of the selected true calibration cu [PITH_FULL_IMAGE:figures/full_fig_p014_2.png] view at source ↗

**Figure 3.** Visualization of the estimated calibration curve [PITH_FULL_IMAGE:figures/full_fig_p017_3.png] view at source ↗

Figures from the paper (2 more)

**Figure 4.** Figure 4: Visualization of estimated results of calibratio [PITH_FULL_IMAGE:figures/full_fig_p018_4.png]

**Figure 5.** Figure 5: Calibration metrics comparison in all confidence s [PITH_FULL_IMAGE:figures/full_fig_p019_5.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

54 extracted references · 35 canonical work pages

[1]

, " * write output.state after.block = add.period write newline

ENTRY address archivePrefix author booktitle chapter edition editor eid eprint howpublished institution isbn journal key month note number organization pages publisher school series title type volume year label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block FUNCTION init.state.consts #0 'before.a...
[2]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...
[3]

W.; Pfau, D.; Schaul, T.; Shillingford, B.; and De Freitas, N

Andrychowicz, M.; Denil, M.; Gomez, S.; Hoffman, M. W.; Pfau, D.; Schaul, T.; Shillingford, B.; and De Freitas, N. 2016. Learning to learn by gradient descent by gradient descent. Advances in neural information processing systems, 29

2016
[4]

B asiok, J.; Gopalan, P.; Hu, L.; and Nakkiran, P. 2023. A unifying theory of distance from calibration. In Proceedings of the 55th Annual ACM Symposium on Theory of Computing, 1727--1740

work page 2023
[5]

Blasiok, J.; and Nakkiran, P. 2023. Smooth ECE: Principled Reliability Diagrams via Kernel Smoothing. In The Twelfth International Conference on Learning Representations

work page 2023
[6]

Br \"o cker, J. 2012. Estimating reliability and resolution of probability forecasts through decomposition of the empirical score. Climate dynamics, 39: 655--667

work page 2012
[7]

Chidambaram, M.; Lee, H.; McSwiggen, C.; and Rezchikov, S. 2024. How Flawed Is ECE? An Analysis via Logit Smoothing. In Forty-first International Conference on Machine Learning

work page 2024
[8]

Deng, J.; Dong, W.; Socher, R.; Li, L.-J.; Li, K.; and Fei-Fei, L. 2009. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, 248--255. Ieee

2009

Show all 54 references

[9]

Dong, J.; Jiang, Z.; Pan, D.; Chen, Z.; Guan, Q.; Zhang, H.; Gui, G.; and Gui, W. 2024. A survey on confidence calibration of deep learning under class imbalance data. Authorea Preprints

2024
[10]

Fernando, K. R. M.; and Tsokos, C. P. 2021. Dynamically weighted balanced loss: class imbalanced learning and confidence calibration of deep neural networks. IEEE Transactions on Neural Networks and Learning Systems, 33(7): 2940--2951

2021
[11]

A.; and Fricker, T

Ferro, C. A.; and Fricker, T. E. 2012. A bias-corrected decomposition of the Brier score. Quarterly Journal of the Royal Meteorological Society, 138(668): 1954--1960

2012
[12]

Geng, J.; Cai, F.; Wang, Y.; Koeppl, H.; Nakov, P.; and Gurevych, I. 2024. A Survey of Confidence Estimation and Calibration in Large Language Models. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human La...

2024
[13]

Grimmett, G.; and Stirzaker, D. 2020. Probability and random processes. Oxford university press

2020
[14]

Guo, C.; Pleiss, G.; Sun, Y.; and Weinberger, K. Q. 2017. On calibration of modern neural networks. In International conference on machine learning, 1321--1330. PMLR

2017
[15]

Gupta, K.; Rahimi, A.; Ajanthan, T.; Mensink, T.; Sminchisescu, C.; and Hartley, R. 2020. Calibration of neural networks using splines. arXiv preprint arXiv:2006.12800

2020 arXiv
[16]

Han, Y.; Liu, D.; Shang, J.; Zheng, L.; Zhong, J.; Cao, W.; Sun, H.; and Xie, W. 2024. BALQUE: Batch active learning by querying unstable examples with calibrated confidence. Pattern Recognition, 110385

2024
[17]

He, K.; Zhang, X.; Ren, S.; and Sun, J. 2016. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, 770--778

2016
[18]

Huang, G.; Liu, Z.; Van Der Maaten, L.; and Weinberger, K. Q. 2017. Densely connected convolutional networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, 4700--4708

2017
[19]

Huang, G.; Sun, Y.; Liu, Z.; Sedra, D.; and Weinberger, K. Q. 2016. Deep networks with stochastic depth. In Computer Vision--ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11--14, 2016, Proceedings, Part IV 14, 646--661. Springer

2016
[20]

Huang, L.; Zhao, J.; Zhu, B.; Chen, H.; and Broucke, S. V. 2020. An experimental investigation of calibration techniques for imbalanced data. Ieee Access, 8: 127343--127352

2020
[21]

Jiang, X.; Osl, M.; Kim, J.; and Ohno-Machado, L. 2012. Calibrating predictive model estimates to support personalized medicine. Journal of the American Medical Informatics Association, 19(2): 263--274

2012
[22]

Jiang, Z.; Dong, J.; Pan, D.; Wang, T.; and Gui, W. 2023. A novel intelligent monitoring method for the closing time of the taphole of blast furnace based on two-stage classification. Engineering Applications of Artificial Intelligence, 120: 105849

2023
[23]

H.; and Dokania, P

Joy, T.; Pinto, F.; Lim, S.-N.; Torr, P. H.; and Dokania, P. K. 2023. Sample-dependent adaptive temperature scaling for improved calibration. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, 14919--14926

2023
[24]

Krizhevsky, A.; Hinton, G.; et al. 2009. Learning multiple layers of features from tiny images

2009
[25]

Kull, M.; Perello Nieto, M.; K \"a ngsepp, M.; Silva Filho, T.; Song, H.; and Flach, P. 2019 a . Beyond temperature scaling: Obtaining well-calibrated multi-class probabilities with dirichlet calibration. Advances in neural information processing systems, 32

2019
[26]

Kull, M.; Perello-Nieto, M.; K \"a ngsepp, M.; Song, H.; Flach, P.; et al. 2019 b . Beyond temperature scaling: Obtaining well-calibrated multiclass probabilities with Dirichlet calibration. arXiv preprint arXiv:1910.12656

2019 arXiv
[27]

Kull, M.; Silva Filho, T.; and Flach, P. 2017 a . Beta calibration: a well-founded and easily implemented improvement on logistic calibration for binary classifiers. In Artificial intelligence and statistics, 623--631. PMLR

2017
[28]

M.; and Flach, P

Kull, M.; Silva Filho, T. M.; and Flach, P. 2017 b . Beyond sigmoids: How to obtain well-calibrated probabilities from binary classifiers with beta calibration. Electronic Journal of Statistics, 11: 5052--5080

2017
[29]

S.; and Ma, T

Kumar, A.; Liang, P. S.; and Ma, T. 2019. Verified uncertainty calibration. Advances in Neural Information Processing Systems, 32

2019
[30]

Lavine, M. 1991. Sensitivity in Bayesian statistics: the prior and the likelihood. Journal of the American Statistical Association, 86(414): 396--399

1991
[31]

Li, Y.; and Caragea, C. 2023. Distilling calibrated knowledge for stance detection. In Findings of the Association for Computational Linguistics: ACL 2023, 6316--6329

2023
[32]

Liu, B.; Rony, J.; Galdran, A.; Dolz, J.; and Ben Ayed, I. 2023. Class Adaptive Network Calibration. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 16070--16079

2023
[33]

Lu, Z.; Lin, R.; and Hu, H. 2024. Disentangling Modality and Posture Factors: Memory-Attention and Orthogonal Decomposition for Visible-Infrared Person Re-Identification. IEEE Transactions on Neural Networks and Learning Systems

2024
[34]

Luo, X.; Wu, J.; Yang, J.; Chen, H.; Li, Z.; Peng, H.; and Zhou, C. 2024. Knowledge Distillation Guided Interpretable Brain Subgraph Neural Networks for Brain Disorder Exploration. IEEE Transactions on Neural Networks and Learning Systems

2024
[35]

Mokhtari, A.; and Ribeiro, A. 2020. Stochastic quasi-newton methods. Proceedings of the IEEE, 108(11): 1906--1922

2020
[36]

M \"u ller, R.; Kornblith, S.; and Hinton, G. E. 2019. When does label smoothing help? Advances in neural information processing systems, 32

2019
[37]

A.; Khan, S

Munir, M. A.; Khan, S. H.; Khan, M. H.; Ali, M.; and Shahbaz Khan, F. 2024. Cal-DETR: Calibrated Detection Transformer. Advances in Neural Information Processing Systems, 36

2024
[38]

P.; Cooper, G.; and Hauskrecht, M

Naeini, M. P.; Cooper, G.; and Hauskrecht, M. 2015. Obtaining well calibrated probabilities using bayesian binning. In Proceedings of the AAAI conference on artificial intelligence, volume 29

2015
[39]

A.; and Mead, R

Nelder, J. A.; and Mead, R. 1965. A simplex method for function minimization. The computer journal, 7(4): 308--313

1965
[40]

W.; Zhang, L.; Jerfel, G.; and Tran, D

Nixon, J.; Dusenberry, M. W.; Zhang, L.; Jerfel, G.; and Tran, D. 2019. Measuring Calibration in Deep Learning. In CVPR workshops, volume 2

2019
[41]

H.; Yang, B.; Pfeiffer, M.; and Zhang, D

Patel, K.; Beluch, W. H.; Yang, B.; Pfeiffer, M.; and Zhang, D. 2020. Multi-Class Uncertainty Calibration via Mutual Information Maximization-based Binning. In International Conference on Learning Representations

2020
[42]

Penso, C.; Frenkel, L.; and Goldberger, J. 2024. Confidence Calibration of a Medical Imaging Classification System that is Robust to Label Noise. IEEE Transactions on Medical Imaging

2024
[43]

Platt, J.; et al. 1999. Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods. Advances in large margin classifiers, 10(3): 61--74

1999
[44]

Rahimi, A.; Shaban, A.; Cheng, C.-A.; Hartley, R.; and Boots, B. 2020. Intra order-preserving functions for calibration of multi-class neural networks. Advances in Neural Information Processing Systems, 33: 13456--13467

2020
[45]

Roelofs, R.; Cain, N.; Shlens, J.; and Mozer, M. C. 2022. Mitigating bias in calibration error estimation. In International Conference on Artificial Intelligence and Statistics, 4036--4054. PMLR

2022
[46]

Silva Filho, T.; Song, H.; Perello-Nieto, M.; Santos-Rodriguez, R.; Kull, M.; and Flach, P. 2023. Classifier calibration: a survey on how to assess and improve predicted class probabilities. Machine Learning, 112(9): 3211--3260

2023
[47]

Wang, M.; Yang, H.; Huang, J.; and Cheng, Q. 2024. Moderate Message Passing Improves Calibration: A Universal Way to Mitigate Confidence Bias in Graph Neural Networks. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, 21681--21689

2024
[48]

Wang, X.; and McCallum, A. 2006. Topics over time: a non-markov continuous-time model of topical trends. In Proceedings of the 12th ACM SIGKDD international conference on Knowledge discovery and data mining, 424--433

2006
[49]

Zadrozny, B.; and Elkan, C. 2001. Obtaining calibrated probability estimates from decision trees and naive bayesian classifiers. In Icml, volume 1, 609--616

2001
[50]

Zadrozny, B.; and Elkan, C. 2002. Transforming classifier scores into accurate multiclass probability estimates. In Proceedings of the eighth ACM SIGKDD international conference on Knowledge discovery and data mining, 694--699

2002
[51]

Zagoruyko, S.; and Komodakis, N. 2016. Wide residual networks. arXiv preprint arXiv:1605.07146

2016 arXiv
[52]

Zellner, A. 1996. Models, prior information, and Bayesian analysis. Journal of Econometrics, 75(1): 51--68

1996
[53]

Zhang, J.; Kailkhura, B.; and Han, T. Y.-J. 2020. Mix-n-match: Ensemble and compositional methods for uncertainty calibration in deep learning. In International conference on machine learning, 11117--11128. PMLR

2020
[54]

Zhang, X.-Y.; Xie, G.-S.; Li, X.; Mei, T.; and Liu, C.-L. 2023. A survey on learning to reject. Proceedings of the IEEE, 111(2): 185--215

2023

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.

[1] [1]

, " * write output.state after.block = add.period write newline

ENTRY address archivePrefix author booktitle chapter edition editor eid eprint howpublished institution isbn journal key month note number organization pages publisher school series title type volume year label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block FUNCTION init.state.consts #0 'before.a...

[2] [2]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

[3] [3]

W.; Pfau, D.; Schaul, T.; Shillingford, B.; and De Freitas, N

Andrychowicz, M.; Denil, M.; Gomez, S.; Hoffman, M. W.; Pfau, D.; Schaul, T.; Shillingford, B.; and De Freitas, N. 2016. Learning to learn by gradient descent by gradient descent. Advances in neural information processing systems, 29

2016

[4] [4]

B asiok, J.; Gopalan, P.; Hu, L.; and Nakkiran, P. 2023. A unifying theory of distance from calibration. In Proceedings of the 55th Annual ACM Symposium on Theory of Computing, 1727--1740

work page 2023

[5] [5]

Blasiok, J.; and Nakkiran, P. 2023. Smooth ECE: Principled Reliability Diagrams via Kernel Smoothing. In The Twelfth International Conference on Learning Representations

work page 2023

[6] [6]

Br \"o cker, J. 2012. Estimating reliability and resolution of probability forecasts through decomposition of the empirical score. Climate dynamics, 39: 655--667

work page 2012

[7] [7]

Chidambaram, M.; Lee, H.; McSwiggen, C.; and Rezchikov, S. 2024. How Flawed Is ECE? An Analysis via Logit Smoothing. In Forty-first International Conference on Machine Learning

work page 2024

[8] [8]

Deng, J.; Dong, W.; Socher, R.; Li, L.-J.; Li, K.; and Fei-Fei, L. 2009. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, 248--255. Ieee

2009