Pith. sign in

REVIEW 2 major objections 33 references

Assessing model calibration with boosting trees

T0 review · 2 major / 0 minor · reviewed 2026-06-27 · grok-4.3

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Boosting trees can test necessary conditions for calibration and auto-calibration in regression models.

desk verdict Boosting trees test for calibration conditions looks like a useful applied extension, but needs full method details to assess validity. read the letter →

arxiv 2606.08084 v1 pith:SSFIASYN submitted 2026-06-06 math.ST stat.APstat.MLstat.TH

Selim Gatti This is my paper · ORCID
classification math.STstat.APstat.MLstat.TH
keywords calibrationauto-calibrationboostingtreesregressionmodelinginsurancepricingstatisticaltestingconditionalexpectation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper shows that boosting trees can check necessary conditions for a regression model to be calibrated, so that predicted means match true conditional means for almost all feature sets. It also covers the weaker auto-calibration property, under which observations sharing the same predicted mean have an expectation that equals that prediction. This matters for applications such as insurance pricing, where auto-calibration prevents cross-subsidization across price groups even when full calibration remains out of reach with finite noisy data. The approach is backed by a numerical study on a large insurance dataset in which the tests demonstrate high power to detect violations.

What carries the argument

Boosting trees applied to test necessary conditions for calibration and auto-calibration of a regression function.

What would settle it

Generate data from a regression model known to violate calibration or auto-calibration, apply the boosting-tree tests, and check whether they fail to reject the null hypothesis of no violation.

Watch

Extended reading notes

Core claim

Boosting trees can be used to test necessary conditions for calibration and auto-calibration, respectively. The practical relevance of our approach is supported by a numerical example, in which the proposed tests prove to be very powerful on a large insurance dataset.

Load-bearing premise

The boosting-tree procedure itself does not introduce bias or dependence that would invalidate the test when applied to finite noisy samples.

Editorial extensions

If this is right

  • Passing the tests confirms that the model meets necessary conditions for matching predicted and true conditional means.
  • The tests can be used to verify auto-calibration and thereby rule out cross-subsidization between price cohorts in insurance applications.
  • The method remains applicable even when perfect calibration cannot be achieved because of finite samples and noise.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The same boosting-tree tests could be applied to compare calibration properties across different regression fitting procedures on the same dataset.
  • Repeated application of the tests during model development might identify feature transformations that improve satisfaction of the necessary conditions.
  • If the tests reject on a given model, retraining with added constraints that enforce the tested identities could be explored as a corrective step.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 0 minor

Summary. The paper claims that boosting trees can be used to test necessary conditions for calibration (matching true conditional means) and auto-calibration (matching conditional expectations given the predicted mean) in regression models. Auto-calibration is highlighted for its relevance in insurance pricing to avoid cross-subsidization. The approach is supported by a numerical example on a large insurance dataset in which the proposed tests are reported to be very powerful.

Significance. If the boosting-tree tests are shown to be valid without introducing bias or dependence on finite samples, the method would supply a practical, tree-based diagnostic for necessary conditions of calibration that is directly applicable to insurance and similar pricing contexts. The numerical example on real data is a strength if accompanied by proper controls and baselines, but the overall significance hinges on resolving the validity questions for the test construction itself.

major comments (2)
  1. [Abstract / numerical example] Abstract and numerical-example section: the claim that the tests 'prove to be very powerful' rests on a single insurance dataset example, yet no power calculations, error bars, baseline comparisons, or controls for the boosting procedure are described. This prevents verification that the observed power is not an artifact of the example design.
  2. [Method description (auto-calibration test)] Method for auto-calibration test: the boosting-tree procedure used to test the necessary condition for auto-calibration must be shown not to introduce bias or dependence on finite noisy samples. The manuscript should supply either a theoretical argument for validity under the null or simulation evidence that the test size is controlled.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the constructive comments on our manuscript. We address each major comment below and indicate the revisions we will make.

read point-by-point responses

  1. Referee: [Abstract / numerical example] Abstract and numerical-example section: the claim that the tests 'prove to be very powerful' rests on a single insurance dataset example, yet no power calculations, error bars, baseline comparisons, or controls for the boosting procedure are described. This prevents verification that the observed power is not an artifact of the example design.

    Authors: We agree that the numerical example is limited to a single large insurance dataset and does not include power calculations, error bars, baseline comparisons, or explicit controls for the boosting procedure. While the dataset size supports the observed results, this is a valid concern. In the revision we will add a simulation study with known calibrated and miscalibrated models to quantify power, include bootstrap-based error bars on the insurance results, and provide comparisons against binning and isotonic regression baselines. We will also revise the abstract language from 'prove to be very powerful' to 'demonstrate strong performance'. revision: yes

  2. Referee: [Method description (auto-calibration test)] Method for auto-calibration test: the boosting-tree procedure used to test the necessary condition for auto-calibration must be shown not to introduce bias or dependence on finite noisy samples. The manuscript should supply either a theoretical argument for validity under the null or simulation evidence that the test size is controlled.

    Authors: We acknowledge that the manuscript currently provides neither a theoretical argument nor simulation evidence that the boosting-tree test controls size under the null of auto-calibration. Finite-sample bias in tree construction is a legitimate issue. We will add a simulation study in the revised version that generates data under the null for a range of sample sizes and noise levels, applies the test, and reports empirical rejection rates to confirm size control at the nominal level. revision: yes

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity detected

full rationale

The paper proposes a method using boosting trees to test necessary conditions for calibration and auto-calibration, supported by a numerical example on insurance data. The abstract and available description present this as an independent diagnostic approach without any equations, derivations, or claims that reduce by construction to fitted parameters, self-definitions, or self-citation chains. No load-bearing steps matching the enumerated circularity patterns are identifiable from the provided content, and the central claim does not rely on renaming known results or importing uniqueness via prior author work. The derivation chain is self-contained as a proposed testing procedure.

Assumptions & free parameters 0 free parameters · 0 assumptions · 0 invented entities

No free parameters, axioms, or invented entities are described in the abstract. The approach appears to rely on standard properties of boosting and conditional expectation, but these are not itemized.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Assessing model calibration with boosting trees." pith.science (2026). https://pith.science/paper/SSFIASYN

@misc{pith2026260608084,
  author       = {Pith},
  title        = {Pith review of: Assessing model calibration with boosting trees},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/SSFIASYN}},
  note         = {Machine review of arXiv:2606.08084}
}
read the original abstract

The main goal in regression modelling consists in approximating the conditional mean of a response given a set of features. A regression function is said to be calibrated if the resulting mean estimates match the true conditional means for almost every set of features. Aiming for calibration seems not achievable in practice as one typically deals with finite samples of noisy observations. A weaker notion of calibration is auto-calibration, and it means that the expectation of responses being given the same mean estimate matches this estimate. This notion is important, e.g., in insurance pricing as it ensures no cross-subsidization between different price cohorts. In this paper, we show that boosting trees can be used to test necessary conditions for calibration and auto-calibration, respectively. The practical relevance of our approach is supported by a numerical example, in which the proposed tests prove to be very powerful on a large insurance dataset.

Figures

Figures reproduced from arXiv: 2606.08084 by the authors.

Figure 1
Figure 1. (a) The large rectangle represents the full dataset [PITH_FULL_IMAGE:figures/full_fig_p007_1.png] view at source ↗
Figure 2
Figure 2. Histogram of the true annual frequency of claims for each policy. The [PITH_FULL_IMAGE:figures/full_fig_p014_2.png] view at source ↗
Figure 3
Figure 3. Plots of the mean estimates against the true conditional means on the test set [PITH_FULL_IMAGE:figures/full_fig_p018_3.png] view at source ↗
Figures from the paper (7 more)
Figure 4
Figure 4. Figure 4: Plots of the mean estimates against the true conditional means on the test set [PITH_FULL_IMAGE:figures/full_fig_p022_4.png]
Figure 5
Figure 5. Figure 5: Plots of binned averages of responses (actuals) against binned averages of mean [PITH_FULL_IMAGE:figures/full_fig_p024_5.png]
Figure 6
Figure 6. Figure 6: Plots of binned averages of responses (actuals) against binned averages of mean [PITH_FULL_IMAGE:figures/full_fig_p026_6.png]
Figure 7
Figure 7. Figure 7: The empirical density of the test statistics [PITH_FULL_IMAGE:figures/full_fig_p028_7.png]
Figure 8
Figure 8. Figure 8: The empirical density of the test statistics [PITH_FULL_IMAGE:figures/full_fig_p028_8.png]
Figure 9
Figure 9. Figure 9: Number of rejections of the null-hypothesis of calibration and auto-calibration, re [PITH_FULL_IMAGE:figures/full_fig_p030_9.png]
Figure 10
Figure 10. Figure 10: Calibration plots of two different models. Only the case where the learning set [PITH_FULL_IMAGE:figures/full_fig_p035_10.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

33 extracted references · 7 canonical work pages

  1. [1]

    Bierens, H. (1990). A consistent conditional moment test of functional form.Econometrica58, 1443–1458

    1990

  2. [2]

    Bross, I. D. J. (1953).Design for decision. New York: Macmillan

    1953

  3. [3]

    Delong, L., Gatti, S., W¨ uthrich, M. V. (2026). Calibration bands for mean estimates within the exponential dispersion family.Statistical Theory and Related Fields, 1-43

    2026

  4. [4]

    Delong, L., W¨ uthrich, M. V. (2025). Isotonic regression for variance estimation and its role in mean estimation and model validation.North American Actuarial Journal29/3, 563-591

    2025

  5. [5]

    Delong, L., W¨ uthrich, M. V. (2025). Universal inference for testing calibration of mean estimates within the exponential dispersion family.arXiv: 2510.23821

    work page arXiv 2025

  6. [6]

    Denuit, M., Charpentier, A., Trufin, J. (2021). Autocalibration and Tweedie-dominance for insur- ance pricing with machine learning.Insurance: Mathematics and Economics101/B, 485-497

    2021

  7. [7]

    Denuit, M., Huyghe, J., Trufin, J., Verdebout, T. (2024). Testing for auto-calibration with Lorenz and concentration curves.Insurance: Mathematics and Economics117, 130-139

    2024

  8. [8]

    (2018).CASdatasetsRpackage vignette.Reference manual

    Dutang, C., Charpentier, A. (2018).CASdatasetsRpackage vignette.Reference manual. Version 1.0-8, packaged 2018-05-20. 31

    2018

Show all 33 references
  1. [9]

    Fissler, T., Lorentzen, C., Mayer, M. (2022). Model comparison and calibration assessment: User guide for consistent scoring functions in machine learning and actuarial practice.arXiv: 2202.12780

    2022

  2. [10]

    Freund, Y., Schapire, R. E. (1997). A decision-theoretic generalization of online learning and an application to boosting.Journal of Computer and System Sciences55/1, 119-139

    1997

  3. [11]

    Friedman, J. H. (2001). Greedy function approximation: a gradient boosting machine.Annals of Statistics29/5, 1189-1232

    2001

  4. [12]

    Gneiting, T. (2011). Making and evaluating point forecasts.Journal of the American Statistical Association106, 746-762

    2011

  5. [13]

    Gneiting, T., Raftery, A. E. (2007). Strictly proper scoring rules, prediction, and estimation.Journal of the American Statistical Association102/477, 359-378

    2007

  6. [14]

    Gneiting, T., Resin, J. (2023). Regression diagnostics meets forecast evaluation: Conditional cali- bration, reliability diagrams, and coefficient of determination.Electronic Journal of Statistics17/2, 3226-3286

    2023

  7. [15]

    H¨ ardle, W., Mammen, E. (1993). Comparing nonparametric versus parametric regression fits.The Annals of Statistics21, 1926-1947

    1993

  8. [16]

    (2009).The elements of statistical learning

    Hastie, T., Tibshirani, R., Friedman, J. (2009).The elements of statistical learning. Data min- ing, inference, and prediction.2nd ed. New York: Springer Series in Statistics.https://link. springer.com/book/10.1007/978-0-387-84858-7

    2009 doi

  9. [17]

    Henzi, A., Puke, M., Dimitriadis, T., Ziegel, J. (2022). A safe Hosmer–Lemeshow test.Journal of Statistics in Data Science2/2, 175-189

    2022

  10. [18]

    W., Lemeshow, S

    Hosmer, D. W., Lemeshow, S. (1980). Goodness of fit tests for the multiple logistic regression model.Communications in Statistics - Theory and Methods9, 1043-1069

    1980

  11. [19]

    (2004).Probability essentials.2nd ed

    Jacod, J., Protter, P. (2004).Probability essentials.2nd ed. Berlin: Springer.https://link. springer.com/book/10.1007/978-3-642-55682-1

    2004 doi

  12. [20]

    Kearns, M., Valiant, L.G. (1988). Learning boolean formulae or finite automata is hard as factoring. Technical Report TR-14-88. Harvard University Aiken Computation Laboratory

    1988

  13. [21]

    (2020).Probability Theory

    Klenke, A. (2020).Probability Theory. A comprehensive course.3rd ed. Cham: Springer.https: //link.springer.com/book/10.1007/978-3-030-56402-5

    2020 doi

  14. [22]

    Kr¨ uger, F., Ziegel, J. F. (2021). Generic conditions for forecast dominance.Journal of Business & Economic Statistics39/4, 972-983

    2021

  15. [23]

    H., Winkler, R

    Murphy, A. H., Winkler, R. L. (1977). Reliability of subjective probability forecasts of precipitation and temperature.Applied Statistics26, 41-47

    1977

  16. [24]

    Pohle, M. O. (2020). The Murphy decomposition and the calibration-resolution principle: A new perspective on forecast evaluation.arXiv: 2005.01835

    2020

  17. [25]

    R Core Team (2021).R: A language and environment for statistical computing.RFoundation for Statistical Computing, Vienna, Austria.https://www.R-project.org/

    2021

  18. [26]

    Sanders, F. (1963). On subjective probability forecasting.Journal of Applied Meteorology and Cli- matology2, 191-201

    1963

  19. [27]

    Valiant, L.G. (1984). A theory of learnable.Communications of the Association for Computing Machinery ACM27/11, 1134-1142

    1984

  20. [28]

    W¨ uthrich, M.V. (2025). Auto-calibration tests for discrete finite regression functions.European Actuarial Journal15, 335-341. 32

    2025

  21. [29]

    W¨ uthrich, M.V., Buser, C. (2016). Data analytics for non-life insurance pricing.Swiss Finance Institute Research Paper16/68, Available at SSRN:https://ssrn.com/abstract=2870308

    2016

  22. [30]

    (2023).Statistical foundations of actuarial learning and its applications

    W¨ uthrich, M.V., Merz, M. (2023).Statistical foundations of actuarial learning and its applications. Cham: Springer.https://link.springer.com/book/10.1007/978-3-031-12409-9

    2023 doi

  23. [31]

    W¨ uthrich, M.V., Richman, R., Avanzi, B., Lindholm, M., Maggi, M., Mayer, M., Schelldorfer, J, Scognamiglio, S. (2026). AI tools for actuaries. Available at SSRN:https://ssrn.com/abstract= 5162304

    2026

  24. [32]

    W¨ uthrich, M.V., Ziegel, J. (2024). Isotonic recalibration under a low signal-to-noise ratio.Scandi- navian Actuarial Journal2024/3, 279-299. 33 A Proof of Proposition 3.1 The proof of Proposition 3.1 is given in this appendix. As the proof of Proposition 3.4 is perfectly sim...

    2024

  25. [33]

    This can be explained by two different factors

    There, we observe that the constructed calibration bands are only able to detect violations of calibration for the overfitting models DNN2 and GBM2 in the case where the learning set corresponds to 60% of the datasetD. This can be explained by two different factors. On the one...

Pith tools

Reviewed June 27, 2026 · model on record in the stance chip above.