Bin-wise Temperature Scaling (BTS): Improvement in Confidence Calibration Performance through Simple Scaling Techniques

Byeongmoon Ji; Hyemin Jung; Jihyeun Yoon; Kyungyul Kim; Younghak Shin

REVIEW 3 major objections 5 minor 18 references

Bin-wise Temperature Scaling (BTS): Improvement in Confidence Calibration Performance through Simple Scaling Techniques

T0 review · 3 major / 5 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Replacing a single global temperature with per-bin temperatures, plus augmenting validation samples, lowers expected calibration error below temperature scaling across all tested models and datasets.

desk verdict Bin-wise temperature scaling with augmented validation bins is a plausible, clearly described extension of temperature scaling, but the all-12 superiority claim rests on an unverified distributional assumption. read the letter →

arxiv 1908.11528 v2 pith:YYBQGNF3 submitted 2019-08-30 cs.CV

Byeongmoon Ji , Hyemin Jung , Jihyeun Yoon , Kyungyul Kim , Younghak Shin This is my paper · ORCID

classification cs.CV

keywords confidencecalibrationtemperaturescalingbin-wiseexpectederrordataaugmentationpost-hocdeepneuralnetworksreliabilitydiagrams

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Modern image classifiers tend to be overconfident, and temperature scaling fixes this with a single global temperature fitted on a validation set. This paper argues that one temperature is too coarse: because most predictions have high confidence, the fitted temperature is biased toward high-confidence samples and leaves low-confidence samples miscalibrated. The authors propose bin-wise temperature scaling (BTS), which splits validation samples into confidence bins and fits a separate temperature for each bin, and augmentation-based BTS (ABTS), which adds transformed copies of validation images to low-confidence bins so those temperatures are stable. Across CIFAR-10/100, Caltech Birds, and Stanford Cars, with ResNet, DenseNet, VGG, and Wide ResNet models, ABTS with number-of-samples binning reports lower expected calibration error than temperature scaling in all twelve experimental settings. The calibration gain comes without retraining or changing test error, since all scaling happens after the model produces its logits.

What carries the argument

The central object is the bin-wise temperature vector $\{t_j\}$, one scalar per confidence bin, applied as in Eq. (3) to soften the softmax of each test sample according to the bin its confidence falls in. Two details carry the argument: the number-of-samples binning rule, which avoids tiny low-confidence bins by collecting equal counts and uses a 0.999 threshold to keep extremely high-confidence samples in one bin, and the augmentation step, which doubles the validation samples in the eight bins below confidence 0.8 using simple transformations (shifting, brightness, contrast, blur). The bin-wise temperatures are what let low-confidence samples be treated differently from high-confidence ones; the augmentation is what keeps the low-confidence temperature estimates from being unstable.

What would settle it

Train a classifier on one of the paper's datasets, hold out a separate split of the test set, fit ABTS temperatures on augmented validation data, and compute ECE on the held-out split; if the ECE is not lower than temperature scaling's, or drops sharply as augmentation strength increases, the transfer assumption fails. A more direct check is to compare the confidence histograms of the augmented validation set and the test set and look for a shift in the low-confidence bins.

Watch

Extended reading notes

Core claim

The central claim is that the calibration quality of a deep classifier can be improved by replacing temperature scaling's single scalar temperature with a set of bin-specific temperatures, and that fitting those temperatures on an augmented validation set makes them stable enough to beat the baseline consistently. Formally, the calibrated confidence for a test sample in bin $j$ is $\hat{q}_i = \max_c \sigma_{SM}(z^j_i / t_j)_c$, where each $t_j$ is optimized on validation samples whose confidence falls in bin $j$. The paper evaluates two binning schemes: equal-width confidence intervals and equal-count bins with a 0.999 threshold that collects all near-certain samples into one bin. It then augments the eight bins below confidence 0.8 by image shifting, brightness, contrast, or blur, doubling the number of validation samples per augmentation. Using expected calibration error (ECE) as the metric, the experiments report that ABTS with number-of-samples binning beats temperature scaling in every case in Table 1, and BTS beats temperature scaling in 10 or 11 of 12 cases depending on the binning scheme.

Load-bearing premise

The augmented validation set, including the synthetically transformed low-confidence samples, has a confidence distribution close enough to the test set that temperatures fitted on it transfer to test samples.

Editorial extensions

If this is right

Calibration of deployed classifiers can be improved by a post-processing step that requires no retraining and leaves test error unchanged.
ABTS with number-of-samples binning matches or beats TS in every configuration reported in Table 1, and BTS beats TS in most cases.
The choice among the four tested augmentation types makes only a marginal difference; all beat TS on CIFAR-100.
The method is not sensitive to the number of bins between 5 and 50, so it does not require fine per-model tuning.
Low-confidence samples, which TS tends to push further under-confident, become better calibrated under ABTS as shown in the reliability diagrams.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

One consequence the paper leaves implicit is that the improvement should transfer to any post-hoc calibration pipeline where the validation set can be augmented; on small validation sets the augmentation step may matter more.
A testable extension is to replace the fixed augmentation of the eight bins below 0.8 with a learned or per-dataset selection of which bins and which augmentation strengths, which might further improve low-confidence calibration.
Since the bin-wise temperatures are fitted independently, a natural next step is to smooth the temperature curve across bins, retaining the gain while removing discontinuities at bin boundaries.
The paper compares only against temperature scaling; comparing BTS and ABTS against histogram binning or isotonic regression would isolate whether the benefit comes from per-bin scaling or from binning itself.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Quick take: this is a legitimate, low-cost extension of temperature scaling. The specific binning rule (equal-count bins with a 0.999 high-confidence cap) and the augmentation of low-confidence validation bins are new relative to Guo et al., and the empirical claim that ABTS with number-of-samples binning beats TS in all twelve settings is directly visible in Table 1. The authors also show robustness to the choice of augmentation (Table 2) and to bin count (Table 3), which is more than many calibration papers do.

Where it gets soft: the comparison is only against TS. No histogram binning, no isotonic regression, no Brier or NLL reported. ECE is reported without error bars or significance tests, and the evaluation bin count isn't specified, so we don't know how sensitive the numbers are to that choice. The abstract says the proposed methods 'consistently improve' calibration, but Table 1 shows BTS loses to TS in a couple of cases, so that wording overclaims.

More importantly, the stress-test concern is real. ABTS fits temperatures on shifted or brightened validation images, then applies them to untouched test images. That only works if the augmented validation confidence distribution is close to the test distribution. The authors acknowledge the risk in Section 4 but never check it. A shift of 4-8 pixels on 32x32 CIFAR is not a small perturbation, and the gains are concentrated in low-confidence bins with few test samples, so small-sample noise or a distribution mismatch could flip some of the twelve results. The paper needs at least a histogram comparison of confidence before and after augmentation and ideally bootstrap confidence intervals on the ECE differences.

That said, the core idea is reasonable and clearly described. The claimed advantage is modest, roughly 0.3 to 2.7 points ECE, and the experiments are easy to reproduce. I would send it to review: an editor should ask for the missing baselines and the distributional check, but the work is coherent and honest. I would cite it as a representative bin-wise scaling variant in a calibration paper. It is a workshop-level contribution, but a legitimate one.

Referee Report

3 major / 5 minor

Summary. The paper proposes two post-hoc confidence calibration methods that generalize temperature scaling (TS): bin-wise temperature scaling (BTS), which partitions validation samples into bins and fits one temperature per bin, and augmentation-based BTS (ABTS), which augments low-confidence validation bins before fitting the temperatures. Two binning schemes are considered: equal-width confidence intervals and equal sample counts with a fixed high-confidence threshold of 0.999. The methods are evaluated on CIFAR-10/100, Caltech-UCSD Birds, and Stanford Cars with several CNN architectures, using expected calibration error (ECE). The central empirical claim is that ABTS with sample-count binning outperforms TS in all 12 dataset/model configurations in Table 1.

Significance. If the results are reproducible, the contribution is practically useful: it shows that a simple bin-wise extension of TS, with a modest number of augmented low-confidence samples, can reduce ECE substantially on common benchmarks while preserving test error rates. The evaluation covers a reasonable spread of datasets and architectures, and Tables 2 and 3 provide useful sensitivity analyses for augmentation type and the number of bins. The paper is self-contained and follows the standard validation-set fitting / test-set evaluation protocol for post-hoc calibration. Its main limitations are the absence of uncertainty quantification and the unexamined distribution shift introduced by validation augmentation.

major comments (3)

[3.3, Eq. (4)] The number of bins N used to compute ECE is never stated. ECE values depend strongly on the evaluation binning; Table 3 demonstrates that similar bin-count choices affect the reported ECE of the proposed methods. Without specifying the evaluation bin count, the reported numbers cannot be reproduced or compared with published results, and the all-12 claim in Section 4 is not fully checkable. Please state the evaluation bin count and report ECE for at least two evaluation bin counts.
[Table 1 and Section 4] The headline claim that ABTS with number-of-samples binning outperforms TS in all 12 cases is based on single point estimates of ECE. No error bars, bootstrap confidence intervals, or significance tests are reported. The margin is small in some rows (e.g., Birds DenseNet: 1.44 vs 1.40), so the claim of consistent improvement is not statistically established. Please provide repeated runs or bootstrap confidence intervals over test samples, and consider a paired comparison across configurations.
[2.3 and Section 4] The ABTS mechanism assumes that temperatures fitted on augmented validation images transfer to unshifted test images. The paper acknowledges that strong augmentation can change the validation/test distribution, but it provides no check that this is not happening here. The x-axis shifts of 4–8 pixels on 32×32 images are a substantial geometric perturbation, and the benefit of ABTS is concentrated in low-confidence bins where test samples are few. An empirical check is needed, e.g., comparing confidence histograms of the augmented validation set and the test set, or fitting on augmented data and evaluating on a clean held-out validation split; without such a check, the improvement of ABTS over BTS could be an artifact of the distribution shift rather than a genuine calibration improvement.

minor comments (5)

[Section 2.2] The paper discusses how low-confidence bins contain very few validation samples but does not report the actual bin counts or confidence ranges. Reporting these for at least one representative model would help assess the stability of the per-bin temperature estimates.
[Section 3.1] The random split of CIFAR training data into training and validation sets is not seeded. Please provide the random seed or release the exact split so that the experiments can be reproduced.
[Section 2.2 and Table 3] The effective number of bins under the 0.999 threshold can be smaller than the nominal 50, but the actual number of bins used in Tables 1–3 is not reported. This would matter for interpreting the claimed insensitivity to bin count.
[Section 4] The sentence in Section 4 that ABTS outperforms TS in all cases is true only for the number-of-samples variant; ABTS with confidence-interval binning does not beat TS on Birds DenseNet (1.70 vs 1.44). The text is mostly clear on this point, but a brief clarification would avoid over-reading the table.
[Section 4 and Table 1 caption] The augmentation parameters (shift ranges, brightness range, contrast strength, blur sigma) are described in prose and the table caption. Releasing the exact code or configuration used with the imgaug library would remove ambiguity about the implementation.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the method is an empirical post-hoc calibration procedure evaluated on a held-out test set, with no prediction that reduces to its fitted inputs.

full rationale

The paper makes no formal derivation claim; it proposes a post-hoc calibration method and evaluates it empirically. BTS fits one temperature per confidence bin on a validation set, and ABTS augments low-confidence validation bins before fitting. Test-set ECE is then measured. This is the standard protocol for post-hoc calibration: the per-bin temperatures are fitted parameters, and the reported ECE is an out-of-sample evaluation, not a quantity that equals the training objective by construction. No equation in the paper defines the test ECE in terms of the fitted validation loss, and the fitted temperatures are applied to disjoint test samples. The admitted limitation that strong data augmentation can change the distribution between validation and test sets only underscores that the result is an empirical transfer claim rather than a tautology. The citations, such as Guo et al. for temperature scaling, are standard external references and are not self-citations; no load-bearing claim is justified by a self-citation chain. The concern about augmentation-induced distribution shift is a correctness risk, not a circularity of the derivation, so the circularity score is 0.

Assumptions & free parameters 4 free parameters · 3 assumptions · 0 invented entities

The central claim rests on standard calibration assumptions (representative validation set, transferable temperatures) and on hand-chosen hyperparameters. No new entities or physical quantities are introduced.

free parameters (4)

Number of bins = 50
Hand-chosen for all experiments; Table 3 shows sensitivity, so it is not fitted to the test set, but it is a free hyperparameter.
High-confidence bin threshold = 0.999
Hand-chosen cutoff at which all remaining validation samples are collected into a single bin (Section 2.2).
Augmented low-confidence bin count = 8 bins with confidence below 0.8
Hand-chosen, fixed across datasets and models; no sensitivity analysis is provided (Section 2.3).
Augmentation parameters = shift [-4, -8], brightness [-150, 150], contrast strength 0.5, Gaussian blur sigma [0, 1]
Hand-chosen augmentation ranges and techniques applied to validation samples in ABTS (Section 4, Table 2).

assumptions (3)

domain assumption Validation set is representative of the test set's confidence distribution
Required for any post-hoc calibration; temperatures fitted on validation are applied to test samples (Sections 2.1-2.3).
domain assumption Augmented validation samples preserve the semantics of the confidence scores
ABTS creates synthetic validation samples via image transformations and fits temperatures on them; if transforms distort the confidence distribution, the fitted temperatures mislead. The authors acknowledge this risk in Section 4.
domain assumption A single temperature per bin can correct miscalibration within that bin
The method assumes that scaling logits by a bin-specific constant suffices to make accuracy match confidence inside each bin (Equation 3).

how reviews work

0 comments

Cite this review

Pith. "Pith review of Bin-wise Temperature Scaling (BTS): Improvement in Confidence Calibration Performance through Simple Scaling Techniques." pith.science (2026). https://pith.science/paper/YYBQGNF3

@misc{pith2026190811528,
  author       = {Pith},
  title        = {Pith review of: Bin-wise Temperature Scaling (BTS): Improvement in Confidence Calibration Performance through Simple Scaling Techniques},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/YYBQGNF3}},
  note         = {Machine review of arXiv:1908.11528}
}

read the original abstract

The prediction reliability of neural networks is important in many applications. Specifically, in safety-critical domains, such as cancer prediction or autonomous driving, a reliable confidence of model's prediction is critical for the interpretation of the results. Modern deep neural networks have achieved a significant improvement in performance for many different image classification tasks. However, these networks tend to be poorly calibrated in terms of output confidence. Temperature scaling is an efficient post-processing-based calibration scheme and obtains well calibrated results. In this study, we leverage the concept of temperature scaling to build a sophisticated bin-wise scaling. Furthermore, we adopt augmentation of validation samples for elaborated scaling. The proposed methods consistently improve calibration performance with various datasets and deep convolutional neural network models.

Figures

Figures reproduced from arXiv: 1908.11528 by the authors.

**Figure 1.** Concepts of TS, BTS, and ABTS using reliability diagrams with the number of samples for each bin. (Left) TS finds a scalar [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗

**Figure 2.** Reliability diagrams and ECE (%) of (left) baseline uncalibrated model, (center) TS, and (right) proposed ABTS (based on the [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗

**Figure 3.** Example pictures from CIFAR-10 test set with actual [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

18 extracted references · 10 canonical work pages

[1]

M. H. DeGroot and S. E. Fienberg. The comparison and evaluation of forecasters. The Statistician: Journal of the Institute of Statisticians, 32:12–22, 1983

work page 1983
[2]

Golan and R

I. Golan and R. El-Yaniv. Deep anomaly detection using geometric transformations. CoRR, abs/1805.10917, 2018

arXiv 2018
[3]

C. Guo, G. Pleiss, Y . Sun, and K. Q. Weinberger. On cali- bration of modern neural networks. CoRR, abs/1706.04599, 2017

arXiv 2017
[4]

K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. CoRR, abs/1512.03385, 2015

arXiv 2015
[5]

Huang, Z

G. Huang, Z. Liu, and K. Q. Weinberger. Densely connected convolutional networks. CoRR, abs/1608.06993, 2016

arXiv 2016
[6]

A. B. Jung. imgaug. https://github.com/aleju/ imgaug, 2018. [Online; accessed 30-Oct-2018]

work page 2018
[7]

Krause, M

J. Krause, M. Stark, J. Deng, and L. Fei-Fei. 3d object rep- resentations for ﬁne-grained categorization. In 4th Interna- tional IEEE Workshop on 3D Representation and Recogni- tion (3dRR-13), Sydney, Australia, 2013

work page 2013
[8]

Krizhevsky, V

A. Krizhevsky, V . Nair, and G. Hinton. Cifar-100 (canadian institute for advanced research)

work page

Show all 18 references

[9]

Kumar, S

A. Kumar, S. Sarawagi, and U. Jain. Trainable calibra- tion measures for neural networks from kernel mean em- beddings. In J. Dy and A. Krause, editors, Proceedings of the 35th International Conference on Machine Learning , volume 80 of Proceedings of Machine Learning Research...

2018
[10]

M. P. Naeini, G. F. Cooper, and M. Hauskrecht. Obtain- ing well calibrated probabilities using bayesian binning. In Proceedings of the Twenty-Ninth AAAI Conference on Artiﬁ- cial Intelligence, AAAI’15, pages 2901–2907. AAAI Press, 2015

2015
[11]

Niculescu-Mizil and R

A. Niculescu-Mizil and R. Caruana. Predicting good proba- bilities with supervised learning. In Proceedings of the 22Nd International Conference on Machine Learning , ICML ’05, pages 625–632, New York, NY , USA, 2005. ACM. 6

2005
[12]

Pereyra, G

G. Pereyra, G. Tucker, J. Chorowski, L. Kaiser, and G. E. Hinton. Regularizing neural networks by penalizing conﬁ- dent output distributions. CoRR, abs/1701.06548, 2017

2017 arXiv
[13]

Russakovsky, J

O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein, A. C. Berg, and L. Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision (IJCV), 115(3):211–252, 2015

2015
[14]

S. Seo, P. H. Seo, and B. Han. Conﬁdence calibration in deep neural networks through stochastic inferences. CoRR, abs/1809.10877, 2018

2018 arXiv
[15]

Simonyan and A

K. Simonyan and A. Zisserman. Very deep convolutional networks for large-scale image recognition. In International Conference on Learning Representations, 2015

2015
[16]

Timofte, R

R. Timofte, R. Rothe, and L. V . Gool. Seven ways to im- prove example-based single image super resolution. CoRR, abs/1511.02228, 2015

2015 arXiv
[17]

C. Wah, S. Branson, P. Welinder, P. Perona, and S. Belongie. The Caltech-UCSD Birds-200-2011 Dataset. Technical Re- port CNS-TR-2011-001, California Institute of Technology, 2011

2011
[18]

Zagoruyko and N

S. Zagoruyko and N. Komodakis. Wide residual networks. CoRR, abs/1605.07146, 2016. 7

2016 arXiv

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.

[1] [1]

M. H. DeGroot and S. E. Fienberg. The comparison and evaluation of forecasters. The Statistician: Journal of the Institute of Statisticians, 32:12–22, 1983

work page 1983

[2] [2]

Golan and R

I. Golan and R. El-Yaniv. Deep anomaly detection using geometric transformations. CoRR, abs/1805.10917, 2018

arXiv 2018

[3] [3]

C. Guo, G. Pleiss, Y . Sun, and K. Q. Weinberger. On cali- bration of modern neural networks. CoRR, abs/1706.04599, 2017

arXiv 2017

[4] [4]

K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. CoRR, abs/1512.03385, 2015

arXiv 2015

[5] [5]

Huang, Z

G. Huang, Z. Liu, and K. Q. Weinberger. Densely connected convolutional networks. CoRR, abs/1608.06993, 2016

arXiv 2016

[6] [6]

A. B. Jung. imgaug. https://github.com/aleju/ imgaug, 2018. [Online; accessed 30-Oct-2018]

work page 2018

[7] [7]

Krause, M

J. Krause, M. Stark, J. Deng, and L. Fei-Fei. 3d object rep- resentations for ﬁne-grained categorization. In 4th Interna- tional IEEE Workshop on 3D Representation and Recogni- tion (3dRR-13), Sydney, Australia, 2013

work page 2013

[8] [8]

Krizhevsky, V

A. Krizhevsky, V . Nair, and G. Hinton. Cifar-100 (canadian institute for advanced research)

work page