{"id":"25729f19-9a23-4513-8a97-298de2282fac","arxiv_id":"2501.02767","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"RCP-GNN couples a rank-based conformal score with a differentiable conformal training loss to produce smaller prediction sets at target empirical coverage for node classification.","lead":"This paper trains graph neural networks with a differentiable rank-based conformal prediction loss so their prediction sets shrink while still hitting a target coverage rate. A smart generalist might care because it addresses a practical bottleneck: making GNN uncertainty estimates smaller and more useful in high-stakes decisions.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Coverage guarantee is not established: Eq. 13's rank score lacks a valid conformal argument, and the test-set split for conformal training may leak labels into the final evaluation.","rationale":"The reader correctly identifies the weakest assumption: the validity of the rank-based conformity score and the exchangeability/independence of the calibration set. My concern is the same, and it is load-bearing because the paper's contribution is a conformal guarantee. Eq. 13 is not a standard split-conformal score; its second term depends on the calibration set, and no proof is given that the resulting scores are exchangeable. The text also gives an alternative prediction-set construction (Eq. 12) that is not equivalent to thresholding the score, leaving the coverage argument unsupported. Independently, the conformal training stage explicitly uses a 50/50 split of the test set for calibration and prediction-set construction, and the final 'standard rank-based CP' does not specify a calibration set that is untouched by training. If the final evaluation uses nodes that appeared in conformal training, the empirical coverage and inefficiency numbers are inflated. These issues are testable: a synthetic exchangeability check would validate or refute the score, and code inspection would clarify the split. Until both are addressed, the central claim cannot be trusted, matching the reader's CONDITIONAL verdict.","tokens_in":11026,"tokens_out":7815,"duration_ms":69382,"concrete_test":"1) On synthetic i.i.d. data with no graph, apply RANK exactly as in Eq. 13 for alpha in {0.05, 0.1, 0.2} over 1000 repetitions; if empirical coverage is consistently below 1-alpha, the score is not validly conformal. 2) Inspect the GitHub code to trace the exact data flow: identify the withheld fraction, and verify that the final calibration set used for prediction sets is disjoint from the nodes whose labels were used in Eq. 15 and Eq. 18 during conformal training. Re-run the experiments with a calibration set that is untouched by training; if coverage or inefficiency changes materially, the original results reflect leakage.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The paper's central claim is that RCP-GNN achieves a pre-defined marginal coverage with reduced inefficiency. This requires a valid conformal score and an exchangeable, model-independent calibration set. Neither is established.\n\nFirst, Eq. 13 defines V(xi,yi) = [rank of mu_yi(xi) among classes] - 1 + (1/n)[rank of mu_yi(xi) among calibration nodes]. The second term makes the score a function of the entire calibration set, so the calibration scores are not computed with a fixed score function; the paper asserts that thresholding this score yields coverage, but offers no exchangeability proof. Furthermore, the prediction set in Eq. 12 is defined as a top-k rule based on r*_alpha and mu*, not as {k: V(x,k) <= Q}, so the claimed equivalence between the two constructions is unexplained.\n\nSecond, conformal training uses test data: 'we randomly split the test data set Dte into folds with 50%/50% as \\hat{D}_calib/\\hat{D}_te' and trains GNN_cor using labels from both (via the calibration threshold and Lcp in Eq. 18). The withheld fraction mentioned before the split is never specified. After training, 'standard rank-based CP are conduct on \\tilde{\\mu}(X)' but it is not stated which calibration set is used and whether it is independent of the trained model and exchangeable with the evaluation nodes. If the test nodes used for evaluation overlap with those used in conformal training, Tables 3-4 are not a valid measurement of generalization. The paper therefore provides no sound argument for the advertised coverage guarantee.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes RCP-GNN, a conformal prediction method for GNN node classification built on a rank-based conformity score. It introduces a differentiable surrogate for rank-based scores, a conformal training loss L_cp that is optimized jointly with the prediction loss on a correction GNN, and an empirical evaluation on eight graph datasets. The central claim is that RCP-GNN achieves any predefined target marginal coverage while producing smaller prediction sets than existing baselines.","tokens_in":11382,"tokens_out":6474,"duration_ms":63438,"significance":"If the validity and efficiency claims are established, this would be a useful contribution: the code is released, the experiments cover eight datasets, ablation studies isolate the main components, and the differentiable rank score is a natural idea. The paper also reports coverage standard deviations over repeated calibration/test splits. However, the paper currently does not provide a validity proof for the rank-based score, and the experimental protocol for splitting the test data during conformal training is ambiguous; these issues bear directly on the central coverage and inefficiency claims.","major_comments":[{"comment":"The rank-based conformity score in Eq. (13) is asserted, not proved. The second term, (1/n) times the rank of the predicted probability for the true class among the calibration nodes, makes the score a function of the entire calibration set, so the calibration scores are not values of a fixed score function evaluated at each node. The paper does not provide the exchangeability or quantile argument that would yield the claimed 1-alpha marginal coverage. In addition, the prediction set in Eq. (12) is defined by a top-r*_alpha rule and a threshold mu*, while the text says this is equivalent to thresholding the Eq. (13) score by Q; the equivalence is not demonstrated. A formal theorem is needed, including tie handling and the finite-sample quantile correction.","section":"RANK, Eq. (13)"},{"comment":"The protocol for splitting the test data is ambiguous and potentially leaks labels into evaluation. The text says Dte is split 50/50 into Dhat_calib and Dhat_te and that 'a fraction' of test data is withheld before splitting, but the size of that fraction and the membership of the final evaluation set are not specified. If L_cp in Eq. (18) is computed on Dhat_te labels and GNN_cor is trained on them, and if the same nodes subsequently appear in Tables 3 and 4, the reported coverage is not a valid out-of-sample measurement. Please specify the exact split, which nodes are used for training GNN_cor, which are used for the final calibration, and which are used for evaluation, and ensure the final calibration set is exchangeable with and independent of the trained model.","section":"Conformal Training (test-data split)"},{"comment":"Table 4 reports inefficiency point estimates without standard deviations, although Table 3 reports standard deviations for coverage; without error bars, the claimed reductions (up to 11.28%) cannot be distinguished from sampling noise. The hyperparameters lambda, tau, and kappa are chosen by random search, but the paper does not state the selection criterion or whether selection is performed on the test set. If test-set coverage or inefficiency is used for selection, the empirical coverage is partly a product of model selection. Please state the selection protocol and add error bars to the inefficiency results.","section":"Table 4 and Implementation"},{"comment":"The abstract claims 'any pre-defined target marginal coverage', but the experiments evaluate only alpha = 0.05 in Table 3 and alpha in {0.1, 0.2, 0.3} in Figure 2. Unless a general proof is supplied, the claim should be restricted to the tested values or accompanied by evidence for a wider range of alpha.","section":"Abstract and Experiments"}],"minor_comments":[{"comment":"There are repeated typos: 'Temperate scaling' should be 'Temperature scaling', 'withhold' should be 'withheld', and 'converge' should be 'coverage' in several places.","section":"Throughout"},{"comment":"The cross-entropy loss in Eq. (20) is written in a binary form with log(1 - mu), which is not appropriate for multi-class node classification; a standard softmax cross-entropy loss should be used.","section":"Eq. (20)"},{"comment":"Equation (17) uses Vtilde(x_i, k) for arbitrary class k, but Eq. (16) defines Vtilde only for the true label y_i; the score should be defined for all candidate labels.","section":"Eqs. (16)-(17)"},{"comment":"The annotation 'Marked: Best' is not supported by the Cora row, where RCP-GNN has size 1.92 versus RCP-THR's 1.65; the criterion for 'best' should be clarified.","section":"Table 5"},{"comment":"The differentiable APS score in Eq. (23) mixes a sigmoid term and the original mu~_k(x_i), but the derivation from Eq. (9) is not explained; please provide the approximation used.","section":"Eq. (23)"}],"recommendation":"major_revision","confidential_remarks":"The central methodological ingredient, Eq. (13), comes from the authors' self-cited prior work (Luo and Zhou 2024). The editor may wish to verify that the prior work contains a complete proof of the rank-based conformal guarantee and that the current paper's contribution is sufficiently distinct from an application of that score to GNNs."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The thing to know about this paper: the headline claim is a conformal coverage guarantee, but the paper never proves one, and the experimental protocol appears to train on test labels. The empirical numbers are plausible, but as written they do not support “any pre-defined target marginal coverage.”\n\nWhat is actually new: applying a rank-based conformity score, borrowed from the authors' own earlier work, to GNNs inside a differentiable conformal training objective. The soft-rank approximation in Eq. 14 and the smooth quantile computation are a reasonable engineering extension of CF-GNN, and the ablation against THR and APS variants shows the rank score matters. Running on eight datasets with coverage and inefficiency reported is decent effort, and the code link is a plus.\n\nThe soft spots are serious, though. Eq. 13 defines V(xi,yi) using the rank of mu_yi(xi) both within the class scores and across the calibration set. That second term makes the score a function of the whole calibration set, so the calibration scores are not computed with a fixed score function. No exchangeability argument is given, and the prediction set in Eq. 12 is a top-k rule based on r*_alpha and mu*, not the set {k: V(x,k) <= Q}. The claimed equivalence is unexplained. Without that, the “rank-based conformal prediction” is not a conformal method in the usual sense.\n\nThen there is the data protocol. The paper says the test set is randomly split 50/50 into calibration and test for conformal training, and that “a fraction” is withheld before that split. That fraction is never specified. The conformal loss in Eq. 18 uses labels from the withheld test part, so labels from the test distribution enter training. After training, standard rank-based CP is run on tilde-mu(X), but it is not stated which calibration set is used and whether it is independent and exchangeable with the evaluation nodes. If the evaluation nodes overlap with those used in conformal training, Tables 3 and 4 are not a valid measure of generalization. This is a load-bearing problem, not a stylistic one. A smaller issue: Table 4 reports inefficiency without error bars, even though coverage in Table 3 has them.\n\nThe hyperparameter tuning via random search over lambda, tau, kappa, and model size further weakens the coverage claim: hitting the target coverage on a benchmark after tuning is not evidence of a distribution-free guarantee.\n\nWho would get value from this: researchers working on conformal training for graphs might use the empirical recipe as a baseline or a cautionary example. But the method, as presented, should not be relied upon for uncertainty quantification in high-stakes settings.\n\nRecommendation: a serious editor should send this to peer review, because the idea is relevant and the gap is potentially fixable, but the reviewers should require a valid conformal argument and a clean, non-leaky evaluation protocol before publication.","headline":"Rank-based conformal training for GNNs is a plausible empirical recipe, but the paper does not establish the advertised coverage guarantee and its evaluation may leak test labels.","tokens_in":11844,"tokens_out":2709,"would_cite":false,"duration_ms":26043,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Rank-based conformal training enables GNN prediction sets to hit any predefined target coverage while reducing average set size by up to 11.28% on eight node-classification benchmarks.","keywords":["graph neural networks","conformal prediction","node classification","uncertainty quantification","rank-based conformity score","conformal training","prediction set efficiency","marginal coverage"],"falsifier":"Take one benchmark graph, train RCP-GNN using the paper's 50/50 test split for conformal training, then run the final rank-based conformal prediction on a completely separate calibration split that was never used in any training loss or threshold calculation, across many random splits at $\\alpha = 0.1$; if average coverage falls below 0.9 by more than sampling error, the validity claim collapses.","tokens_in":10871,"feed_emoji":"🎯","tokens_out":5944,"duration_ms":52863,"temperature":0.7,"pith_summary":"Graph neural networks used for node classification rarely come with guarantees that the set of plausible labels is right-sized. The paper proposes RCP-GNN, which folds conformal prediction into GNN training: a rank-based conformity score determines how many top classes to include, and a differentiable version of it is used as a training loss so the prediction sets themselves are optimized. The paper reports that on eight benchmark graphs the method reaches any pre-specified marginal coverage, usually closer to the target than competing calibration methods, while producing smaller average prediction sets, up to 11.28% smaller than the best baseline on Cora. If the guarantees hold, GNNs can be deployed in high-stakes settings with uncertainty sets that are both statistically valid and practically small.","feed_headline":"Rank-based conformal training shrinks GNN prediction sets","feed_subtitle":"On eight node-classification benchmarks, the method hits any target coverage while cutting prediction-set size by up to 11.28%.","key_machinery":"The load-bearing object is the rank-based conformity score of Eq. 13: $V(x_i,y_i) = [\\text{rank of } \\hat{\\mu}_{y_i}(x_i) \\text{ among class probabilities}] - 1 + \\frac{1}{n}[\\text{rank of } \\hat{\\mu}_{y_i}(x_i) \\text{ among the calibration values for that class}]$. The first term decides how many top classes enter the prediction set; the second breaks ties and lets the calibration quantile set the threshold. For training, this hard ranking is replaced by a differentiable approximation $\\tilde{V}(x_i,k) = \\sum_j \\operatorname{sigmoid}((\\tilde{\\mu}_j(x_i) - \\tilde{\\mu}_k(x_i))/\\tau)$, and the quantile computation is smoothed using differentiable sorting; the resulting conformity loss $L_{\\text{cp}}$ is minimized together with the cross-entropy loss on a topology-aware correction model. This mechanism is what lets prediction sets be adjusted during training rather than fixed after it.","core_discovery":"The central discovery is that the rank of the true class's predicted probability, relative to all class probabilities at a node, can serve as a conformity score for GNNs and can be made differentiable so that conformal prediction is no longer a post-training afterthought. The paper defines $V(x_i,y_i)$ as the rank of the true class's probability among all classes plus a small tie-breaking term based on the rank of that probability within the calibration set; thresholds derived from these scores yield top-$k$ prediction sets with nominal $1-\\alpha$ coverage. RCP-GNN then trains a correction GNN with a sigmoid-smoothed version of this ranking as a conformity loss, jointly with the prediction loss, so the learned probabilities are shaped to make prediction sets compact. On Cora, DBLP, CiteSeer, PubMed, Amazon-Computers, Amazon-Photo, Coauthor-CS, and Coauthor-Physics, the trained model reaches target coverage on all datasets and consistently reduces inefficiency, with up to 11.28% smaller prediction sets than the best baseline.","pith_inferences":["Editorial inference: if the rank score is as miscalibration-robust as claimed, a natural extension is to regression or link prediction, where the analogue of 'rank among class probabilities' would be rank of residuals; the paper lists link prediction as future work.","Editorial inference: the 50/50 split of test data into calibration and evaluation halves during conformal training suggests the final marginal coverage may depend on how the conformal-training calibration fold relates to the final conformal-prediction calibration set; an independent test would use a completely held-out calibration set untouched by training.","Editorial inference: because the method reduces prediction-set size most on Cora and least on DBLP, the gain likely depends on graph homophily or class balance; comparing datasets with controlled label noise could reveal where ranking helps.","Editorial inference: the smooth-sorting temperature and loss weight are dataset-tuned, so transferring to a new domain probably requires re-tuning; the reported insensitivity to temperature is only over the tested range."],"forward_implications":["Any predefined target marginal coverage, not just $\\alpha = 0.05$ or $0.1$, should be reachable by the same training and calibration procedure, since the rank score and quantile adapt to $\\alpha$.","Deploying the trained correction model with standard rank-based conformal prediction at test time gives prediction sets that are valid and smaller on average than post-hoc calibration baselines across eight graphs.","Topology-aware correction during conformal training is responsible for the efficiency gain: without the conformity loss, prediction sets are larger.","The rank-based score is performance-critical: replacing it with threshold or adaptive conformity scores within the same training scheme degrades efficiency or coverage.","Because the differentiable rank score is smooth in model parameters, conformal training scales to standard SGD mini-batching rather than requiring full retraining."],"supporting_citations":[{"why":"Defines the underlying rank-based conformity score that RCP-GNN inherits, adapts to GNNs, and differentiably approximates.","marker":"(Luo and Zhou 2024)"},{"why":"Supplies the fast differentiable sorting and ranking operators used to smooth the hard rank computation in conformal training.","marker":"(Blondel et al. 2020)"},{"why":"Provides the smooth quantile estimation used by the differentiable calibration step.","marker":"(Chernozhukov, Fernández-Val, and Galichon 2007)"},{"why":"Establishes the split-conformal framework whose train/calibration/test splitting the method follows.","marker":"(Lei, Rinaldo, and Wasserman 2013)"},{"why":"Gives the main state-of-the-art baseline CF-GNN and justifies exchangeability for transductive node classification, which RCP-GNN relies on.","marker":"(Huang et al. 2023)"},{"why":"Introduces conformal training with soft assignment of classes to prediction sets, adapted here for the conformity loss.","marker":"(Stutz et al. 2022)"},{"why":"Defines APS, the adaptive conformity score baseline that RCP-GNN compares against and generalizes with ranking.","marker":"(Romano, Sesia, and Candes 2020)"},{"why":"Defines THR, the threshold-based prediction-set baseline, and the quantile calibration used for coverage.","marker":"(Sadinle, Lei, and Wasserman 2019)"}],"fun_headline_variants":["Rank-based conformal training tightens GNN prediction sets","Conformal rank training shrinks GNN sets, keeps coverage","GNNs hit target coverage with smaller rank-conformal sets","Training GNNs with rank conformity for compact predictions"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The guarantee only holds if the calibration nodes used for the final threshold are interchangeable with the test nodes and were not touched while the correction model learned to shrink prediction sets; the paper does not prove this separation.","fun_headline_variants_meta":{"raw":{"variants":["Rank-based conformal training tightens GNN prediction sets","Conformal rank training shrinks GNN sets, keeps coverage","GNNs hit target coverage with smaller rank-conformal sets","Training GNNs with rank conformity for compact predictions"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00028,"raw_usage":{"total_tokens":1670,"prompt_tokens":963,"completion_tokens":707,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":579,"completion_tokens_details":{"reasoning_tokens":639}},"tokens_in":579,"tokens_out":707,"duration_ms":6406,"temperature":1.0,"reasoning_tokens":639,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T22:04:23.509060+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take one benchmark graph, train RCP-GNN using the paper's 50/50 test split for conformal training, then run the final rank-based conformal prediction on a completely separate calibration split that was never used in any training loss or threshold calculation, across many random splits at $\\alpha = 0.1$; if average coverage falls below 0.9 by more than sampling error, the validity claim collapses.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the fast differentiable sorting and ranking operators used to smooth the hard rank computation in conformal training."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the smooth quantile estimation used by the differentiable calibration step."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Establishes the split-conformal framework whose train/calibration/test splitting the method follows."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Gives the main state-of-the-art baseline CF-GNN and justifies exchangeability for transductive node classification, which RCP-GNN relies on."},{"cited_title":"D.; Cemgil, A","cited_arxiv_id":null,"evidence_quote":"Introduces conformal training with soft assignment of classes to prediction sets, adapted here for the conformity loss."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines APS, the adaptive conformity score baseline that RCP-GNN compares against and generalizes with ranking."}],"review_version":1}