Pith. sign in

REVIEW 3 major objections 5 minor 7 references

Hierarchical Group-Conditional Conformal Risk Control for Selective Prediction in Language Models

T0 review · 3 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read HG-CRC, a hierarchical group-conditional calibration method, achieves an empirical zero violation rate and worst-group excess risk 0 on ARC Challenge for Qwen3-4B and Llama-3.1-8B.

desk verdict A clearly specified hierarchical group-conditional CRC with a sound split-variant guarantee, an honest but deflating empirical story, and a real gap between the headline zeros and the formally guaranteed procedure. read the letter →

arxiv 2607.24562 v1 pith:Y3E52UQT submitted 2026-07-27 cs.AI

Murilo Salem , Luísa Böhm , Daniel Pontes , Anderson Ferrugem This is my paper · ORCID
classification cs.AI
keywords selectivepredictionconformalriskcontrolgroup-conditionalguaranteeshierarchicalcalibrationBonferronicorrectionabstentionlargelanguagemodelsworst-groupexcess
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that the standard marginal guarantee of conformal risk control is the wrong target for selective prediction in language models: a model can meet its overall error budget while a subgroup that becomes over-represented at test time absorbs most of the errors. HG-CRC fixes this by assigning one calibrated abstention threshold to every node of a user-defined group hierarchy, applying a Bonferroni correction so all nodes are certified simultaneously, and routing each example to the deepest certified node that will answer it. The method needs only a held-out calibration set and no retraining, and it is tested on three LLMs and two benchmarks across eight distribution-shift and robustness settings. The headline result is that on ARC Challenge, for models with sufficiently high base accuracy, HG-CRC attains an empirical violation rate of 0% and worst-group excess risk 0, at a participation cost of 22 to 37 percentage points relative to global CRC; the paper is explicit that these zeros are empirical upper bounds of about 0.6% at 500 bootstrap trials, and that the result does not carry over to MMLU-Pro.

What carries the argument

The load-bearing object is a rooted group hierarchy $\mathcal{H}$ of nested membership masks, combined with a leaf-first selective predictor $v^*(x)=\arg\max_{v\in\Pi(x)}\{\mathrm{depth}(v): v \text{ certified}, s(x)\le \tau_v\}$, which sends each test point to the deepest certified ancestor whose threshold it satisfies. Each node's threshold is chosen from a Clopper-Pearson upper bound $U_v(\tau)$ at per-node level $\hat\delta=\delta/|\mathcal{H}|$, and the key twist is residual calibration: a parent node is calibrated on exactly the examples that reach it after deeper certified nodes claim their own, via the residual map $\rho_v(x)=m_v(x)\prod_{c\in\mathrm{desc}(v)}(1-m_c(x)g_{\tau_c}(x))$. This aligns each node's calibration population with its deployment population. The guarantees form a cascade: split calibration gives an unconditional simultaneous bound (Proposition 1), in-sample calibration with test-point augmentation is exact (Proposition 2), and the deployed plain in-sample procedure is valid up to a measured jackknife slack $\varepsilon_n$ (Lemma 1) under a threshold-regularity assumption.

What would settle it

Run the split variant of HG-CRC on ARC Challenge with exogenous human difficulty labels and 500 bootstrap trials, and count trials where any node's realized risk exceeds $\alpha$; a violation rate meaningfully above $\delta$ would refute the simultaneous claim. Alternatively, construct a synthetic score distribution with an atom at the selected grid threshold so that Assumption 1 fails, and check whether the deployed in-sample violation rate exceeds $\delta+|\mathcal{H}|\varepsilon_n$.

Watch

Extended reading notes

Core claim

The paper's central claim is that the marginal risk guarantee of conformal risk control is not enough for selective prediction in language models, because a population-level budget can be met while subgroup risk exceeds it under shifts in group composition. HG-CRC replaces the single threshold with one per node of a user-defined hierarchy, calibrates each node on its residual population after deeper certified nodes claim their own examples, and answers a test point if its deepest certified ancestor's threshold is satisfied. The formal machinery gives a simultaneous guarantee: with probability at least $1-\delta$, every node's deployed risk is at most $\alpha$ in the split variant, with the in-sample variant holding the same bound up to a measured slack $\varepsilon_n$. On ARC Challenge the procedure reaches an empirical violation rate of $0$ and worst-group excess risk $0$ for Qwen3-4B and Llama-3.1-8B at 500 bootstrap trials, which the paper treats as an upper bound of about $0.6\%$; on MMLU-Pro it either abstains entirely or retains a worst-group excess risk of $0.014$.

Load-bearing premise

The deployed in-sample guarantee relies on the threshold-regularity assumption that each calibration curve crosses the risk budget cleanly at the chosen threshold, so a one-point change in calibration data never flips the selected threshold; on the small ARC pool the measured worst-case slack is about 0.34, and if that assumption fails, the simultaneous guarantee degrades.

Editorial extensions

If this is right

  • On ARC Challenge, HG-CRC achieves an empirical violation rate of 0 and WGER 0 for Qwen3-4B and Llama-3.1-8B across 500 bootstrap trials, which the paper reads as an upper bound of about 0.6% rather than a certified zero.
  • Under a mixture shift that over-represents hard examples, global CRC violates the risk budget in 47% of trials for Qwen and 32% for Llama, while groupwise thresholds hold it; HG-CRC keeps the simultaneous version of that protection.
  • Label noise makes the method abstain more rather than violate: at 10% label noise participation drops to 8.5% with a 0% violation rate.
  • The difficulty level of the hierarchy is what clears the budget on ARC: without it, violations return to the 11% level of global CRC.
  • Bonferroni correction is required for the formal simultaneous guarantee, and its empirical effect appears only when many nodes are tested, as on MMLU-Pro where removing it raises the violation rate from 0.6% to 14.0%.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The equal-participation comparison suggests that an oracle global threshold matched to HG-CRC's coverage would protect per-group risk about as well in most settings, so the method's practical value lies in automatically finding a safe per-group operating point and in holding a simultaneous certificate, not in a large per-group safety margin.
  • A direct test of the endogenous-difficulty concern would use human grade-level difficulty labels from the original ARC release; the paper's own exogenous-model check suggests the mixture-shift failure would persist.
  • Because participation collapses for models whose accuracy is near or below $1-\alpha$, HG-CRC can serve as a deployability diagnostic: near-zero participation under the hierarchy is evidence that no threshold can meet the budget for that model and task.
  • The measured jackknife slack $\varepsilon_n$ could be reported alongside empirical violation rates in production, turning the in-sample guarantee into a transparent worst-case bound.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes HG-CRC, a conformal risk control method that calibrates thresholds for every node of a user-specified group hierarchy and applies a Bonferroni correction to provide simultaneous per-node risk guarantees in selective prediction with abstention. A leaf-first policy routes each example to the deepest certified ancestor whose score threshold it satisfies. Three theoretical results are offered: Proposition 1 gives an unconditional simultaneous bound for a split-calibration variant, Proposition 2 gives an exact bound for an in-sample variant with test-point augmentation, and Lemma 1 covers the deployed plain in-sample variant up to an additive slack ε_n. The empirical study covers three LLMs and two benchmarks, reporting a 0% empirical violation rate and WGER=0 on ARC Challenge for Qwen3-4B and Llama-3.1-8B, with participation cost of 22–37 pp relative to global CRC, plus ablations, robustness checks, and an equal-participation comparison that shows a coverage-matched global threshold controls per-group risk about as well as HG-CRC in most settings.

Significance. If the formal guarantees and empirical results hold, HG-CRC is a useful post-hoc calibration framework for group-conditional selective prediction in LLMs, requiring no retraining and providing simultaneous certificates over hierarchical groups. The paper is unusually transparent: it explicitly caveats that the headline zeros are empirical upper bounds at 500 bootstrap trials, that difficulty bins are endogenous to the deployed score, that worst-case slack for the deployed guarantee is ε_n≈0.34 on the small ARC pool, and that at equal participation a single global threshold matches HG-CRC on per-group safety in most measured settings. The split-variant guarantee (Proposition 1) is the paper's main formal contribution and its proof is coherent. The study also ships a 500-trial evaluation with rule-of-three error bars, an exogenous-difficulty robustness check, and an honest equal-participation analysis, which are strengths.

major comments (3)
  1. [Section 3.5, Lemma 1] The stated bound "P(R_v(τ*_v)>α | ρ_v(x)=1) ≤ δ̂+ε_n" is not what the proof establishes. The proof defines ε_n as the marginal disagreement probability P(â(x)≠â+(x)), but then derives a conditional slack of ε_n/π_v, where π_v=P(g_τ(x)=1,ρ_v(x)=1), and the text immediately after the proof concedes that "the conditional slack the lemma adds to δ̂ is ε_n/π_v". For sparse nodes (small π_v, e.g., Gemma's leaf cells), ε_n/π_v can greatly exceed ε_n. The simultaneous bound should therefore be δ + |H|·sup_v(ε_n/π_v) (or an analogous expression), not δ + |H|·ε_n. With the measured worst-case marginal slack ε_n≈0.34 on the small ARC pool (Section 8.3), and |H|=5, the bound δ+|H|ε_n is already vacuous; using the correct conditional slack makes the deployed variant's worst-case guarantee even weaker. This is a load-bearing error because Lemma 1 is the only formal support for the deployed default procedure used in all experiments. The lemma must be restated with the correct conditional slack, and the numerical discussion in Section 8.3 must be revised to report conditional worst-case slack at the nodes that actually answer.
  2. [Section 3.5, Remark 2 and Section 8.3] The formal simultaneous guarantee (Proposition 1) holds only for the split variant, yet the split variant is never evaluated at the paper's headline 500-trial resolution. Remark 2 reports a 200-resample comparison of participation costs and a McNemar test on node-violation indicators, but it does not report the split variant's WGER or violation rate at 500 trials, nor does it provide the split-variant analog of the Table 2 headline numbers. Since the abstract and Section 5.2 present the 0% violation and WGER=0 as the main empirical result for the deployed in-sample variant, and since the in-sample variant's guarantee is only approximate (Lemma 1, with the issue above), the paper should report the split variant's WGER and violation rate at the same 500-trial resolution, ideally as a primary result or as a clearly labeled alternative. Without this, the reader cannot verify that the variant carrying the rigorous guarantee actually realizes the claimed empirical behavior in the regimes of interest.
  3. [Section 8.2, Table 11] The equal-participation analysis shows that, at matched coverage, a single global threshold controls per-group risk about as well as HG-CRC in five of six cells, with the exception of Gemma under mixture shift. This is an honest and important qualification of the empirical contribution, but it should be reflected in the abstract and introduction: the headline "0% violation and WGER=0" could be read as evidence that group-conditional calibration inherently protects subgroups better than any single threshold, whereas the paper's own data indicate the effect is mostly a participation effect. The authors do state this in Section 8.2, but the framing in the Abstract and Section 1 ("enforcing simultaneous risk guarantees") should explicitly acknowledge that the empirical per-group safety margin over a coverage-matched global threshold is small, and that the value of HG-CRC lies in the automatic selection of a safe operating point and in the formal simultaneous certificate of the split variant.
minor comments (5)
  1. [Section 3.2] The sentence "Nodes below this threshold are pruned from H, reducing |H| and accordingly tightening the per-node budget δ̂" is backwards: reducing |H| increases δ̂=δ/|H|, i.e., loosens the per-node budget. The sentence should say "increasing" or "loosening".
  2. [Section 4.2 and Algorithm 1] It is unclear whether the threshold grid T in split mode is computed on the full calibration set, on the node's own fold, or on the residual cell. The proof of Proposition 1 assumes T is either fixed a priori or a permutation-invariant function of the node's calibration fold F_ℓ, but Section 4.2 says the grid consists of 100 quantiles of the NLL scores in the calibration set. Please clarify the exact computation of T in split mode and reconcile it with the proof.
  3. [Section 3.5, Lemma 1 proof] In the sentence "since the disagreement set has probability at most ε_n and enters both the numerator and the conditioning event of R_v = E[ℓ|g_τ(x)=1,ρ_v(x)=1], the conditional risks differ by |R_deployed−R_aug| ≤ ε_n/π_v", the variable π_v is defined only in the preceding sentence. Please ensure the notation is introduced before use, and consider defining the conditional disagreement probability explicitly.
  4. [Section 5.3, Table 4] For Gemma, the exogenous-difficulty row reports a groupwise violation of 0.07, which is above δ=0.05, while the endogenous row reports 0.11. The text says the effect "persists" under exogenous difficulty, but the exogenous groupwise violation for Gemma is actually lower and still above the target; please state this explicitly so the reader can see the quantitative contrast.
  5. [Section 8.3] The limitation paragraph on the in-sample guarantee says the worst-case slack "is materially weaker than the headline numbers suggest"; given that δ+|H|ε_n≈1.75 on the ARC pool, the worst-case guarantee is vacuous (the probability bound exceeds 1). The wording should say "vacuous" or "has no meaningful worst-case content" for that configuration, rather than only "materially weaker".

Circularity Check

1 steps flagged · score 3.0 of 10

The ARC difficulty grouping is defined by the deployed NLL score, making difficulty-shift results partially self-definitional; the paper explicitly acknowledges and mitigates this with an exogenous difficulty check, so circularity is partial.

  1. self definitional [Section 4.3 (Group Hierarchy); Section 7.5 (Difficulty Shift); Section 8.3 (Limitations)]
    "Difficulty nodes (level 3). Within each domain, we further partition examples into three difficulty bins (easy, medium, hard) based on NLL score percentiles: the k-th calibration example is assigned to hard if its NLL exceeds the 67th percentile ... Note on endogeneity. Because difficulty bins are derived from NLL percentiles—the same score used for thresholding—the 'hard' bin by construction contains high-NLL examples, creating an alignment between group membership and calibrated threshold values. ..."

    On ARC the only non-trivial grouping is difficulty, and difficulty is assigned by thresholding the deployed uncertainty score s(x) = -log p'_k. Since HG-CRC's decisions also threshold that same score, the 'hard' group is by definition the set of high-NLL examples. A 'shift toward hard' is then a shift toward high NLL, which is precisely the perturbation that an NLL threshold is least likely to survive. The difficulty-shift result '0% participation / risk 0.000' in the cal-easy/test-hard direction is conceded by the paper to hold 'almost by construction.' This is a genuine self-definitional step for the ARC difficulty and mixture-shift experiments.

full rationale

The core methodological contribution — simultaneous group-conditional risk control via Bonferroni-corrected per-node Clopper-Pearson thresholds with residual calibration — is not circular. Proposition 1 is an application of standard conformal risk control to disjoint calibration folds, with the residual partition made data-independent by construction; it does not rely on any assumption equivalent to its conclusion, and it is not justified by self-citation. The in-sample variant's Lemma 1 introduces an explicit slack epsilon_n and is honestly reported as weaker, which is a validity concern but not a circularity: the bound is not obtained by assuming what it proves. The acknowledged endogeneity of the ARC difficulty bins is the main circular step. Section 4.3 defines 'hard' as the top NLL tercile of the deployed model, so group membership and the thresholded score are defined in terms of each other. The paper itself labels this 'load-bearing' in Section 8.3 for the 47% mixture-shift failure, and Section 7.5 concedes the easy-to-hard transfer result is '0.000 almost by construction.' This self-definitional step is material to the ARC difficulty experiments, but it is partially repaired by the exogenous-difficulty control in Table 4, which keeps the qualitative conclusion (global CRC fails under difficulty shift) intact. The equal-participation analysis in Section 8.2 and the WGER-rewards-abstention caveat in Section 8.3 further reduce the strength of the empirical claims without constituting circularity. No other load-bearing self-citations or fitted-input-as-prediction patterns were found: the baselines and benchmarks are external, the formal guarantees follow from exchangeability and union bounds, and the empirical zeros are explicitly presented as non-certified upper bounds. Overall, the self-definitional difficulty grouping is real and affects the ARC headline narrative, but the paper names the issue, measures its effect, and provides an independent exogenous check. The score of 3 reflects one partially load-bearing self-definitional grouping whose impact is explicitly bounded by the paper's own controls.

Assumptions & free parameters 4 free parameters · 3 assumptions · 0 invented entities

The method's guarantees rest on the hand-chosen hierarchy (difficulty bins from the deployed score), the minimum group size, the finite threshold grid, and Assumption 1 for the deployed in-sample variant. These are the main elements the reader is asked to accept rather than derive.

free parameters (4)
  • Nmin = 30
    Minimum node size below which a group is pruned from the hierarchy; chosen by hand in all experiments (Section 4.3). Appendix D shows it has a threshold effect on participation and violations.
  • Difficulty bin thresholds = 33rd and 67th percentiles of calibration NLL scores
    Defines easy/medium/hard groups from the deployed uncertainty score (Section 4.3). This hand-chosen grouping is the load-bearing non-trivial level of the ARC hierarchy.
  • Threshold grid size = 100 quantiles
    Finite grid of candidate thresholds selected as empirical score quantiles of the calibration set (Section 4.2).
  • Risk and confidence levels = alpha=0.10, delta=0.05
    User-specified risk budget and confidence level; standard inputs to CRC, not fitted to data.
assumptions (3)
  • domain assumption Calibration and test points are i.i.d. from the same distribution P, and group masks are fixed and nested (Definition 1).
    Invoked in Proposition 1 for exchangeability; the paper notes the in-sample variant only approaches this.
  • standard math Clopper-Pearson upper bound controls the one-sided error of the binomial rate for exchangeable calibration and test points.
    Foundational result from Angelopoulos et al. 2023 used for every node's certification.
  • ad hoc to paper Assumption 1 (threshold regularity): the CP curve crosses alpha transversally at the selected grid threshold with a margin gamma >> 1/nc outside a boundary set of probability O(epsilon_n).
    Needed for Lemma 1 to give the deployed in-sample variant a guarantee up to epsilon_n; the measured worst-case epsilon_n is 0.34 on the small ARC pool (Section 8.3).

how reviews work

0 comments
Cite this review

Pith. "Pith review of Hierarchical Group-Conditional Conformal Risk Control for Selective Prediction in Language Models." pith.science (2026). https://pith.science/paper/Y3E52UQT

@misc{pith2026260724562,
  author       = {Pith},
  title        = {Pith review of: Hierarchical Group-Conditional Conformal Risk Control for Selective Prediction in Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/Y3E52UQT}},
  note         = {Machine review of arXiv:2607.24562}
}
read the original abstract

Large language models serve heterogeneous populations structured by domain, topic difficulty, and linguistic style. Conformal risk control (CRC) gives rigorous marginal risk guarantees for selective prediction with abstention, but marginal guarantees do not imply per-group ones: a model can meet the population budget while systematically over-exposing subgroups to errors. Under mild shift in group composition, standard CRC violates the budget in up to 47% of trials. We propose HG-CRC (Hierarchical Group-Conditional CRC), a post-hoc calibration framework enforcing simultaneous risk guarantees across all nodes of a user-defined group hierarchy. It applies a Bonferroni correction over nodes and a leaf-first policy that uses the most specific applicable threshold, falling back to coarser nodes when a finer one is uncertified or rejects the example. It needs only a held-out calibration set, with no retraining. We evaluate on three models (Qwen3-4B, Llama-3.1-8B-Instruct, Gemma-3-4B) and two benchmarks (ARC Challenge, MMLU-Pro) across eight configurations probing IID generalization, heterogeneity, mixture/domain/prompt/difficulty shift, label noise, and quantization. Main result: HG-CRC reaches an empirical 0% violation rate and WGER=0 on ARC Challenge for high-accuracy models (Qwen3-4B, Llama-3.1-8B). At 500 bootstrap trials these zeros are empirical upper bounds (true rate up to 0.6%), not certified. Results are benchmark-specific: on MMLU-Pro these models abstain entirely or (Llama) retain WGER=0.014. Gemma-3-4B, poorly calibrated here, degrades gracefully by abstaining. Participation cost vs. global CRC is 22 to 37 points. Ablations show hierarchical depth clears the budget: removing difficulty level returns violations to about 11%. Bonferroni is needed for the theoretical guarantee, though its empirical effect matters only with many nodes.

Figures

Figures reproduced from arXiv: 2607.24562 by the authors.

Figure 1
Figure 1. Mixture shift violation rates (ARC Challenge). Global CRC (blue) violates the risk budget in [PITH_FULL_IMAGE:figures/full_fig_p013_1.png] view at source ↗
Figure 2
Figure 2. Label noise robustness (ARC Challenge). Risk (solid lines, left axis) and participation (dashed [PITH_FULL_IMAGE:figures/full_fig_p018_2.png] view at source ↗
Figure 3
Figure 3. Quantization robustness (ARC Challenge). Accuracy (left), participation (center), and risk (right) [PITH_FULL_IMAGE:figures/full_fig_p019_3.png] view at source ↗

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

7 extracted references · 2 canonical work pages

  1. [3]

    Think you have solved question answering? Try ARC, the AI2 reasoning challenge.arXiv preprint arXiv:1803.05457,

    Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? Try ARC, the AI2 reasoning challenge.arXiv preprint arXiv:1803.05457,

    arXiv

  2. [6]

    Yaniv Romano, Rina Foygel Barber, Chiara Sabbati, and Emmanuel J. Candès. With malice toward none: Assessing uncertainty via equalized coverage.Harvard Data Science Review, 2020a. Yaniv Romano, Matteo Sesia, and Emmanuel J. Candès. Classification with valid and adaptive coverage. In Advances in Neural Information Processing Systems, 2020b. Gemma Team. Gem...

    arXiv

  3. [2008]

    A Extended Results: MMLU-Pro MMLU-Pro results are qualitatively different from ARC Challenge due to structural model limitations. Qwen3- 4B and Gemma-3-4B have base error rates above 56% on MMLU-Pro, meaning that even the most conservative threshold cannot certify a non-trivial fraction of examples underα = 0.10. As a result, participation is essentially ...

    work page

  4. [2017]

    Uncertainty estimation in autoregressive structured prediction.arXiv preprint arXiv:2002.07650,

    Andrey Malinin and Mark Gales. Uncertainty estimation in autoregressive structured prediction.arXiv preprint arXiv:2002.07650,

    arXiv 2002

  5. [2018]

    The Llama 3 herd of models.arXiv preprint arXiv:2407.21783,

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Lesecq, et al. The Llama 3 herd of models.arXiv preprint arXiv:2407.21783,

    arXiv

  6. [2021]

    A Positive/Unlabeled Approach for the Segmentation of Medical Sequences using Point-Wise Supervision

    Henrik Boström, Ulf Johansson, and Tuwe Löfström. Mondrian conformal prediction.arXiv preprint arXiv:2107.08394,

    work page Pith review arXiv

  7. [2023]

    Rina Foygel Barber, Emmanuel J

    URLhttps://arxiv.org/abs/ 2208.02814. Rina Foygel Barber, Emmanuel J. Candès, Aaditya Ramdas, and Ryan J. Tibshirani. The limits of distribution-free conditional predictive inference. volume 10, pp. 455–482,

    arXiv

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.