{"id":"5ea272ba-a3ec-45d5-aeda-e5cff1369934","arxiv_id":"2510.19328","paper_version":2,"verdict":"REJECT","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":5,"one_line_summary":"Clustered Calibration groups samples by learned representations and calibrates each cluster separately, with a new cluster-binned ECE claimed to rank models by both calibration and AUC.","lead":"This paper proposes calibrating classifiers by first clustering samples in a learned representation space (SHAP values, leaf indices, or activations) and then fitting a separate calibration curve per cluster. It also introduces a cluster-based calibration error metric that it claims selects the model with the best calibration and AUC.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Lemma 2.4's inference from lower CECE to higher AUC is invalid; Theorem 2.5 fails even under the paper's ECE-minimization idealization.","rationale":"The reader's verdict is REJECT and I reach the same conclusion, so the recommendation is UNCHANGED relative to their verdict. I only partially endorse their stated weakest_assumption: the ECE-minimization versus likelihood-fitting mismatch is real and affects Lemmas 2.1/2.3, but the decisive flaw is the CECE-to-AUC implication in Lemma 2.4. That implication is false as a matter of decision theory: ECE/CECE is a bin-averaged calibration loss, not a measure of rank ordering. The proof's step (a)(ii) asserts that reducing ECE improves threshold accuracy and TPR/FPR, but a calibration map can lower ECE by flattening scores, which typically harms separation. The paper's own Table 2 even shows cases where CECE improves while AUC ties (e.g., House, Jannis), which is consistent with monotone transformations preserving AUC, but Theorem 2.5 claims strict AUC improvement from strict CECE improvement without proof. Since the central novelty is the model-selection guarantee, this unsupported implication justifies rejection unless it is replaced with a correct proof or a clearly stated empirical conjecture. I did not rely on the lack of code or error bars in my decision; the theoretical counterexample is sufficient.","tokens_in":13495,"tokens_out":13188,"duration_ms":105850,"concrete_test":"Synthetic check: generate two equal-size clusters with contrasting base rates and score assignments (e.g., cluster 1: positives at 0.9/negatives at 0.1; cluster 2: positives at 0.4/negatives at 0.6). On a calibration split, fit each of the paper's parametric calibrators (Platt, TS, Beta, Dirichlet) both globally and per cluster; on a held-out test split compute CECE and ROC-AUC for each model. Repeat over 100 random variations of base rates, cluster sizes, and score gaps. If any run yields CECE(CCL) < CECE(global) while AUC(CCL) < AUC(global), Theorem 2.5 is disproved in the exact setting claimed. The same script should also test the hand-built logistic counterexample described in the review.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central contribution is Theorem 2.5: lower CECE implies higher ROC-AUC. The reader's concern about the §2.5 note (calibrators are ECE-optimal) is legitimate, but the load-bearing break is in Lemma 2.4. Its proof claims that after a downward/upward correction of a cluster-bin's average probability, 'for a given acceptance threshold th, the accuracy of the accepted samples ... will increase' and hence TPR/FPR and AUC increase. This conflates calibration with ranking. CECE is an average absolute deviation between mean predicted probability and observed frequency within fixed cluster bins; it imposes no constraint on the relative ordering of individual scores that ROC-AUC measures. A cluster-wise calibrator can drive CECE to zero by mapping every score in a cluster to that cluster's base rate, while destroying all score separation and lowering AUC to 0.5. In a two-cluster example, per-cluster monotone logistic maps can eliminate cluster-level calibration error while reordering clusters so that positives fall below negatives, reducing AUC. Granting the §2.5 ECE-minimization assumption would only justify Lemmas 2.1 and 2.3; it cannot repair Lemma 2.4. Thus the model-selection guarantee is unsupported.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Clustered Calibration (CCL), a method that clusters samples by a learned representation (SHAP values, leaf indices, etc.) and fits a separate parametric calibrator per cluster, with hierarchical shrinkage toward a global mapping. It also introduces Clustered ECE (CECE), a calibration metric that bins samples by the learned clusters. The paper claims theoretical guarantees: (i) per-cluster ECE of cluster-specific calibrators is upper-bounded by global calibration, (ii) CECE of the ensemble is upper-bounded by unified CECE, and (iii) lower CECE implies higher ROC-AUC (Theorem 2.5), making CECE a model-selection metric that optimizes both calibration and discrimination. Experiments on tabular datasets and image/text benchmarks are presented as supporting these claims.","tokens_in":13932,"tokens_out":3944,"duration_ms":33770,"significance":"If the theoretical guarantees were sound, the paper would make a substantial contribution: a calibration metric that provably selects models with better discrimination as well as calibration is genuinely useful, and the CCL framework is generic and empirically broad. The paper also makes a sensible point that fixed-bin ECE can mis-rank certain calibrators. However, the theoretical backbone is not sound: the main implication from CECE to AUC is invalid, and the auxiliary assumption used in the proofs is not satisfied by the implemented calibrators. The empirical results may suggest CCL is a useful heuristic, but the central claimed guarantee is unsupported.","major_comments":[{"comment":"The proof that lower CECE implies higher AUC is invalid. CECE is a cluster-bin average absolute deviation between mean predicted probability and observed frequency; it imposes no constraint on the relative ordering of individual scores, which is what ROC-AUC measures. A per-cluster calibrator can drive CECE to zero while destroying within-cluster score separation, e.g., by mapping all scores in a cluster to the cluster base rate, making AUC approach 0.5. Step (a)(ii) states 'for a given acceptance threshold th, the accuracy of the accepted samples ... will increase ... Therefore, a higher TPR/FPR and thus a higher AUC.' Accuracy of accepted samples does not imply higher TPR and FPR at every threshold, and higher TPR/FPR pointwise does not imply higher AUC. This flaw is independent of the ECE-optimality assumption in §2.5; even granting that assumption, the implication does not follow. Th","section":"Lemma 2.4 / Theorem 2.5"},{"comment":"All theoretical results rely on the assumption that the trained parametric calibration method c denotes c*, the model minimizing ECE(c,X). The implemented calibrators—Platt, temperature scaling, beta, and Dirichlet—are fitted by likelihood/log-loss minimization, not by minimizing ECE. Consequently, Lemma 2.1's contradiction argument fails: a global calibrator can have smaller ECE on a cluster than the cluster-specific calibrator, because the cluster-specific calibrator is not optimizing that objective. The upper bounds in Lemmas 2.1 and 2.3 are therefore not consequences of the training procedure. Table 2 illustrates the mismatch: on WiDS, CCL-Platt has CECE 0.0087 but ECE 0.0096, while unified Platt has CECE 0.0117 but ECE 0.0051—the unified version has lower standard ECE. The assumption is not a harmless idealization; it is contradicted by the method's own implementation.","section":"§2.5 Note"},{"comment":"The claimed upper bound CECE(C,CL,X) ≤ CECE(c,X) is essentially a restatement of the ECE-optimality assumption rather than a proven result. Step 4 of the proof says 'Should there be a better set of parameters, they would have been learned when training c on B_i,' which is precisely the c=c* assumption. Since the cluster-specific calibrators are not ECE-optimal in practice, a global calibrator can outperform the cluster-specific one on a given cluster, and the inequality can fail. The paper does not provide any independent argument for this bound, so it cannot support the subsequent AUC claims.","section":"Lemma 2.3"},{"comment":"The number of clusters k is selected by the elbow method, and the paper tests partitions of 5 to 100 clusters, but no sensitivity analysis is reported: the chosen k per dataset, representation, and clustering method are not given, nor is stability of the results across k. CECE and the CCL predictions depend directly on the partition, so it is unclear whether the claimed improvements and model-selection rankings in Tables 2–4 are robust or an artifact of a particular elbow choice. This is a load-bearing empirical gap for the model-selection claim.","section":"§3.1 Experiment Setup"}],"minor_comments":[{"comment":"The caption says 'Claustered Calibration'—typo for 'Clustered Calibration.'","section":"Figure 2"},{"comment":"The text states a paired t-test was performed for 'the three parametric calibration methods (Platt, Beta, TS)' and later 'for the four parametric calibration methods (Platt, Beta, TS, Dirichlet).' This inconsistency should be corrected.","section":"§3.2.2"},{"comment":"The row for threshold 0 is confusing: with no rejection, the error rate should be the overall error, but values differ from the full evaluation (e.g., 0.019 vs 0.002 for the base classifier). The threshold semantics need clarification.","section":"Table 4"},{"comment":"The abstract says 'six tabular datasets,' but the experiments section lists ten data sources. Please align the counts.","section":"Abstract and §3.1"},{"comment":"No code or data availability statement is provided. Given the empirical nature of the paper, releasing code would aid reproducibility.","section":"General"}],"recommendation":"reject","confidential_remarks":"The central contribution is the claim that CECE is the only metric that guarantees model selection for both discrimination and calibration. That claim rests on Lemma 2.4 and Theorem 2.5, which are invalid even under the paper's own idealizing assumption. The assumption itself is not satisfied by the implemented calibrators. These are load-bearing errors that cannot be fixed locally; the theoretical contribution would need to be substantially reworked, and the empirical claims reinterpreted as heuristic observations. I therefore recommend rejection, while acknowledging that the empirical results may still be of interest if reframed."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The core idea—cluster a learned representation (SHAP values or leaf indices) and fit separate calibrators per cluster—is reasonable and the experiments are suggestive. Across several tabular datasets, the per-cluster calibrators often improve ECE and log-loss relative to global calibrators, and the t-SNE visuals make the case that SHAP-based clusters do find heterogeneity in label frequencies. That part deserves credit: it is a natural synthesis of bin-wise/local calibration with clustering, and I don't think the exact combination has appeared before.\n\nThe problem is the theory, and it is not a minor gap. Lemma 2.4 claims that lower cluster-binned calibration error (CECE) implies higher ROC-AUC. That is false. CECE is an average deviation between mean predicted probability and observed frequency within fixed clusters; it imposes no constraint on the relative ordering of individual scores. A per-cluster calibrator could drive CECE to zero by mapping every score in a cluster to that cluster's base rate, which would destroy score separation and send AUC to 0.5. The proof conflates “the average probability in a bin became more accurate” with “the ranking improved.” That is a load-bearing error: Theorem 2.5, the claimed model-selection guarantee, collapses. Granting the §2.5 note that calibrators minimize ECE would only justify Lemmas 2.1 and 2.3; it cannot repair Lemma 2.4. And in any case, the implemented calibrators (Platt, TS, Beta, Dirichlet) are trained by log-loss, not ECE, so the note is not satisfied by the actual experiments.\n\nThere are also softer issues. The abstract describes a “soft mixture with hierarchical shrinkage,” but the full text uses hard clustering (K-means or agglomerative) and independent per-cluster calibrators; those are different methods. The elbow choice of k has no sensitivity analysis, and the tables report t-test p-values without standard deviations or error bars. The CECE circularity concern is minor—the clustering is independent of the calibrator—but the metric is defined on the same clusters used to evaluate it, so it is not a neutral external benchmark.\n\nWho is this for?Someone interested in practical calibration for tabular models might find the empirical recipe useful, but the theoretical claims should not be trusted. A careful referee would need the authors to either fix the proof (I doubt it can be fixed) or reframe the paper as an empirical method paper without the AUC guarantee. I would not cite it in my own work while the central theorem stands.\n\nRecommendation: send it to peer review, because the empirical idea is promising and the flaw in Lemma 2.4 is instructive—but the editor should expect major revision or a reframing before publication.","headline":"The empirical idea is worth a look, but the headline guarantee—lower CECE implies higher AUC—is false, and the paper should not be accepted on that claim.","tokens_in":14298,"tokens_out":2590,"would_cite":false,"duration_ms":24050,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper argues that cluster-specific calibration beats a single global calibration curve and that its cluster-based error metric gives a model-selection rule that optimizes both calibration and discrimination.","keywords":["probability calibration","expected calibration error","clustered calibration","model selection","ROC-AUC","learned representations","subpopulations","uncertainty quantification"],"falsifier":"Take one dataset and one cluster partition; fit both a global calibrator and cluster-specific calibrators of the same family, minimizing ECE directly on the calibration split. If any cluster's global-calibrator ECE is below the cluster-specific calibrator's ECE, Lemma 2.1 is contradicted; a synthetic mixture with two well-separated Gaussian components and different label rates should be the easiest place to search.","tokens_in":13447,"feed_emoji":"🎯","tokens_out":4898,"duration_ms":45887,"temperature":0.7,"pith_summary":"The paper argues that a classifier's predicted probabilities can be made more reliable by first clustering the data in a learned representation of the inputs, so that each cluster approximates a subpopulation the model treats differently, and then fitting a separate calibration curve to each cluster instead of one global curve. It introduces CECE, a calibration metric whose bins are those learned clusters, and claims that CECE is the only common metric that lets model selection optimize both calibration and discrimination: a lower CECE provably implies a higher ROC-AUC, a standard measure of ranking ability. If this holds, calibration and ranking stop being competing objectives: improving calibration on the right subpopulations also improves the model's ability to separate positive from negative cases.","feed_headline":"Cluster-based calibration promises accuracy plus discrimination","feed_subtitle":"A cluster-aware error metric could replace ECE for model selection when fit and ranking both matter.","key_machinery":"The central object is CECE, an expected-calibration-error variant computed over cluster bins defined by a learned representation rather than over confidence intervals. Because the bin assignment depends only on the sample's embedded representation, not on the calibrated output, the partition is invariant to the calibrator; that invariance is what lets a CECE comparison between calibrators translate into a ranking of their ROC-AUC. The companion CCL procedure is the ensemble of cluster-specific calibrators that this metric evaluates, and the paper shows that such clusters can reveal meaningful subpopulations that are not defined in advance.","core_discovery":"On the paper's own terms, the central discovery is that shifting the binning of calibration error from predicted-confidence intervals to learned subpopulations changes what the metric can certify. CCL trains an ensemble of cluster-specific calibrators, and CECE evaluates them on the same clusters. The authors prove that, under an optimal-fit assumption for the calibrators, CECE bounds the error of any global calibrator from above and that a strict CECE improvement guarantees a strict ROC-AUC improvement, so the cluster-based metric can be used for model selection where standard ECE cannot, because standard ECE's bins shift when the calibrator changes.","pith_inferences":["The proof chain assumes that each fitted calibrator is the ECE-optimal member of its parametric family for the cluster it is fit to, but the experiments fit calibrators by log-loss; a direct test would be to fit by ECE and see whether the inequalities survive.","The abstract describes a soft mixture of calibrators with hierarchical shrinkage, but the body's algorithm and theory use a hard cluster assignment; the published guarantees should be read as applying to the hard-partition version.","If the optimal-fit assumption is relaxed in favor of realistic distributional assumptions, such as cluster-specific label noise, the AUC implication could fail; constructing a small counterexample would settle the true scope of the theorem.","The cluster-binning idea could naturally extend to multiclass problems by computing cluster-wise calibration error over class probabilities, and to regression by binning residuals, though the paper does not develop those extensions."],"forward_implications":["If CECE is a valid model-selection criterion, practitioners can tune calibration methods without a separate discrimination check, since lower CECE would certify better ranking as well.","Cluster-specific calibration can identify unknown subpopulations, such as rare clinical subgroups, and correct their calibration errors, adding interpretability to the calibration step.","CCL is a template method that can be applied on top of any base classifier, any learned representation, any clustering algorithm, and any parametric calibrator, giving it broad applicability.","In rejection and abstention settings, choosing the CCL variant with the best CECE should produce the lowest error on accepted samples, supporting safer deferral to human experts.","The paper's argument that fixed-bin ECE can mis-rank soft, region-aware calibrators even when proper scoring rules improve suggests that log-loss and Brier score are more reliable bases for model selection in such settings."],"fun_headline_variants":["Cluster-based calibration error beats ECE for model selection","Learned subpopulations make calibration metrics honest","CECE: a calibration error that doesn't misrank soft calibrators","Ditch ECE bins: cluster-aware calibration improves ranking","Calibrate per cluster, not per confidence interval"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The chain of guarantees rests on the assumption that the fitted calibrator for each cluster is the one that truly minimizes calibration error on that cluster; the implementations reported in the paper fit their parameters by log-loss, not by calibration error, so a global calibrator can in principle beat a cluster-specific one on that cluster's data.","fun_headline_variants_meta":{"raw":{"variants":["Cluster-based calibration error beats ECE for model selection","Learned subpopulations make calibration metrics honest","CECE: a calibration error that doesn't misrank soft calibrators","Ditch ECE bins: cluster-aware calibration improves ranking","Calibrate per cluster, not per confidence interval"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.0002,"raw_usage":{"total_tokens":1193,"prompt_tokens":710,"completion_tokens":483,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":454,"completion_tokens_details":{"reasoning_tokens":403}},"tokens_in":454,"tokens_out":483,"duration_ms":5056,"temperature":1.0,"reasoning_tokens":403,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-04T08:41:45.690410+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take one dataset and one cluster partition; fit both a global calibrator and cluster-specific calibrators of the same family, minimizing ECE directly on the calibration split. If any cluster's global-calibrator ECE is below the cluster-specific calibrator's ECE, Lemma 2.1 is contradicted; a synthetic mixture with two well-separated Gaussian components and different label rates should be the easiest place to search.","supporting_citations":[],"review_version":1}