{"id":"1f0316c2-2169-4775-b930-4a5aa452e5d7","arxiv_id":"2505.16466","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"A post-hoc rating compression plus a high-confidence negative-sample loss reduces overconfidence and slightly improves recommendation accuracy in GNN-based recommender systems.","lead":"This paper says that GNN-based recommender systems are overconfident: they give high confidence scores to predictions that are often wrong. The authors propose a scoring adjustment and a training loss that bring predicted confidence closer to actual accuracy, and report small accuracy gains on three public datasets.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Whether Conf-GNNRec is a post-hoc calibrator or a training regularizer is never pinned down, and the calibration gain is never quantified; either gap alone would block the central claim.","rationale":"I read the paper as a heuristic confidence-calibration companion method: it documents overconfidence in LightGCN/KGCL, proposes a nonlinear rating compression and a penalty on high-confidence negative predictions, and reports consistent small gains on three datasets. The public code, standard datasets, and multiple baselines are real independent evidence, and the method is simple enough to reproduce. The load-bearing weak point is not the heuristic nature of the loss but the fact that the intervention is not pinned down: Eq. (5) is post-hoc-capable, Eq. (6) is not, and no experiment separates them. This ambiguity directly controls whether the empirical gains support 'calibration' or merely 'retraining with an extra regularizer.' I therefore agree with the reader's weakest-assumption identification. I would add that the absence of any scalar calibration metric (ECE/ACE) makes the visual reliability-diagram claim hard to audit; this is secondary but reinforces the same condition. With the proposed ablation and ECE reporting, the claim could be settled; without them, the paper should remain conditional. No issue with the authors' credibility is implied; the concern is entirely about experimental identification and reporting.","tokens_in":7204,"tokens_out":5278,"duration_ms":38408,"concrete_test":"Inspect the released code (github.com/FairyMeng/Conf-GNNRec) to determine exactly where Eq. (6) enters the pipeline. Then run three variants on Gowalla with LightGCN: (a) frozen baseline + Eq. (5) post hoc; (b) baseline retrained with Eq. (6) added to the original loss; (c) baseline retrained with an auxiliary L2 regularizer chosen to match the magnitude of Eq. (6). For all variants report ECE/ACE computed from the reliability bins of Section 3.2.2 together with Precision@20 and Accuracy@20. If (a) alone does not reduce ECE while (b) does, the calibration effect is a training effect, not post-hoc calibration; if (b) and (c) improve similarly, the gains are regularization artifacts. This settles whether the central claim holds.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that Conf-GNNRec calibrates GNN-based recommender confidence, as defined by Eq. (3), and that applying it improves confidence-accuracy alignment and Precision@20/Accuracy@20. For that claim to hold, 'applying Conf-GNNRec' must be a well-defined, fairly compared intervention. The paper does not establish this. The Introduction calls the approach a 'post-calibration strategy without other additional inputs,' which matches Eq. (5) applied to a frozen model's scores. But Section 2.3 introduces Eq. (6) as a confidence loss, which requires gradients and therefore changes the trained model. No experiment states whether the '+Conf-GNNRec' rows in Table 2 come from (a) retraining each baseline with Eq. (6) added to the original objective, (b) applying Eq. (5) post hoc to frozen baselines, or (c) both. If (a) or (c) is used, the comparison is between two differently trained models, so the Precision/Accuracy improvements and the reliability-diagram changes in Figures 2-3 can be caused by generic regularization or training effects rather than by a confidence-calibration mechanism. Section 3.2.1 only reports 'application of Conf-GNNRec' without describing the protocol, and Section 3.2.2 provides visual reliability diagrams but no scalar calibration error such as ECE/ACE, so the headline calibration improvement is not quantified. There is also a notation problem in Eq. (6): p_hat_{u,i} was defined as max_i z_{u,i}, which is item-independent, so the loss does not literally implement the described per-item penalty on high-confidence negative samples.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper argues that GNN-based recommender systems output overconfident predictions, defines a perfect-calibration condition in Eq. (3), and proposes Conf-GNNRec to quantify and calibrate confidence via a rating-compression step (Eq. (5)) and a confidence-aware regularization loss (Eq. (6)). It reports that applying Conf-GNNRec to LightGCN, KGAT, MVIN, and KGCL improves Precision@20 and Accuracy@20 on Gowalla, Yelp2018, and Amazon-Book, and presents reliability diagrams before and after the intervention. The central claims are that existing methods violate the calibration condition and that Conf-GNNRec brings confidence closer to accuracy while improving ranking accuracy.","tokens_in":7555,"tokens_out":6555,"duration_ms":56810,"significance":"If established, the paper would address a genuinely underexplored issue: confidence calibration for GNN-based recommendation rather than node classification. The choice of representative baselines, public datasets, and the public code link are strengths, and the reported ranking improvements are consistent across all model-dataset combinations. However, the current manuscript does not contain a quantitative calibration metric, does not specify the experimental protocol that defines '+Conf-GNNRec,' and contains a rating-calibration formula whose monotonicity is questionable. These issues are load-bearing because the paper's headline claim is about calibration, not only ranking accuracy.","major_comments":[{"comment":"The rating calibration function as written is not monotone and therefore cannot preserve ranking information. For any r_{u,i} just above the threshold, r_c(r_{u,i}) = \\bar{r}_u + \\tau \\log(r_{u,i} - \\bar{r}_u) tends to -\\infty as r_{u,i} approaches \\bar{r}_u from above, while r_{u,i} = \\bar{r}_u is left unchanged. This creates a discontinuity and inverts the relative order of items near the mean, contradicting the stated design goals of retaining relative ranking and maintaining recommendation performance. The formula needs to be corrected or accompanied by an explicit admissible range for the scores and a reported value of \\tau, and the experiments need to be re-run under the corrected transformation.","section":"Sec. 2.3, Eq. (5)"},{"comment":"The paper never states whether the '+Conf-GNNRec' rows in Table 2 are obtained by (a) applying Eq. (5) post hoc to frozen baseline models, (b) retraining each baseline with Eq. (6) added to the original objective, or (c) both. This distinction is essential: Eq. (6) requires gradients and therefore changes the trained model, so any observed improvement in Precision/Accuracy and any change in reliability diagrams could be a generic regularization or training effect rather than a property of the calibration method. The Introduction calls the approach a post-calibration strategy, but Section 2.3 introduces a training loss. Please specify the exact intervention for each experiment and, if retraining is used, include a control baseline trained with an additional regularizer of comparable strength.","section":"Sec. 2.3 and Sec. 3.2.1, Eq. (6) and Table 2"},{"comment":"The confidence loss does not implement the described penalty. In Section 2.2, \\hat{p}_{u,i} is defined as max_i z_{u,i}, which is a user-level quantity and is identical for every item i of user u. As a result, Eq. (6) sums (1 - p_{u,i}) multiplied by a user-level constant and cannot specifically penalize 'wrongly predicted items with high confidence.' Moreover, p_{u,i} in Eq. (6) is not defined in the paper; presumably it should be the item-specific probability \\sigma(r_{u,i}). An item-specific confidence and an explicit indicator for incorrect prediction are needed for the loss to match the stated purpose.","section":"Sec. 2.3, Eq. (6)"},{"comment":"The central calibration claim is supported only by visual comparison of reliability diagrams. The paper should report a scalar calibration error, such as ECE or ACE computed with the same ten-bin protocol, for every model and dataset before and after applying Conf-GNNRec. Without a quantitative metric, the statement that Conf-GNNRec 'significantly reduces the error between confidence and accuracy' is not verifiable from the reported evidence. The value of the temperature factor \\tau used in the experiments and its sensitivity should also be reported.","section":"Sec. 3.2.2, Figs. 2 and 3"},{"comment":"The calibration condition in Eq. (3) is stated for a single prediction \\hat{y}_{u,i}, but the experimental evaluation is for top-k recommendation. The text says 'For the top-k task, we use the mean value instead,' but it is not explained how each user's multiple test items are mapped into the probability P(\\hat{y} = y | \\hat{p} = p). Please define precisely how the confidence bins and bin-wise accuracy are computed for the reliability diagrams, including how ties among multiple positive test items are handled.","section":"Sec. 2.2 and Sec. 3.1.3, Eq. (3)"}],"minor_comments":[{"comment":"The text defines \\bar{z}_u as the mean of all elements in z_u, but Eq. (5) uses \\bar{r}_u. These notations should be reconciled, and the variable being averaged should be stated unambiguously.","section":"Sec. 2.3, after Eq. (5)"},{"comment":"The reported Precision@20 and Accuracy@20 improvements are small (roughly 1.5% to 6.7%) and are presented without error bars or significance tests. Reporting variance over multiple random seeds would strengthen the claim that Conf-GNNRec improves all models and datasets.","section":"Table 2"},{"comment":"The KGCL experiments use a knowledge-graph-augmented dataset that is not the one described in Table 1. The paper should state explicitly how the knowledge graph changes the data split and whether the comparison with other baselines is affected by this difference.","section":"Sec. 3.1.2"},{"comment":"The caption of Figure 3 should state whether the diagrams correspond to post-hoc Eq. (5), retraining with Eq. (6), or both, so that the reader can interpret the comparison with Figure 2.","section":"Sec. 3.2.2"}],"recommendation":"major_revision","confidential_remarks":"The paper addresses a timely and relevant problem, and the overall direction is worth publishing if the technical issues are resolved. In my view, the most serious problems are the non-monotonicity of Eq. (5), the unclear experimental protocol for what '+Conf-GNNRec' means, and the absence of any quantitative calibration metric. I would also encourage the authors to compare against standard calibration baselines such as temperature scaling or the calibration method of [16], since the current evaluation only compares each baseline with itself. The 5-page format may have contributed to the missing details, but the central claims require them."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper does something useful and modest: it measures overconfidence in GNN recommenders, then applies a two-part heuristic—a per-user log compression of high ratings (Eq 5) and a confidence-weighted penalty on high-confidence negative samples (Eq 6)—and shows consistent small gains in Precision@20 and Accuracy@20 across four baselines and three datasets. That is a real empirical pattern, and the code is public. The reliability diagrams before/after do show the bars moving toward the diagonal, so the qualitative direction is believable. I agree with the reader that the first-contribution claim overstates novelty: confidence calibration for GNNs is established by ref [16], and this is an adaptation rather than a first. But the specific combination is new enough to be publishable as a short workshop-style contribution if the protocol were pinned down. The soft spots are real and load-bearing. Most importantly, the paper never states whether '+Conf-GNNRec' means retraining with the Eq (6) loss, applying Eq (5) post hoc to frozen models, or both. The Introduction calls it a post-calibration strategy, but Eq (6) requires gradients. If retraining is involved, then the reported Precision/Accuracy gains and reliability improvements can be a generic training effect, not a calibration mechanism. That ambiguity sits right on the central claim and appears in Sections 2.3 and 3.2.1. Second, the calibration improvement is only shown visually; no ECE/ACE or any scalar calibration error is reported, so the headline 'confidence matches accuracy' claim is not quantified. Third, Eq (6) has a notation problem: p_hat was defined as max over items, so the per-item penalty (1-p_u,i) hat-p_u,i does not literally penalize each high-confidence negative item as described. There's also a typo in Eq (5) ('bar z' vs 'bar r'), and tau is not reported. The reader's circularity concern I would downgrade: the evaluation is on held-out test data, so there is no direct fitting of the calibration target. But the tau and loss-weight are free parameters tuned on the same data used for the diagrams, so some optimism bias is possible. Bottom line: the work is honest, the heuristics are simple and cheap, and the gains are consistent but modest. As is, the paper is not fully supported. But it deserves a serious referee: the ambiguity is fixable with one clear protocol sentence, ECE numbers are easy to add, and the notation can be corrected. I would engage with it if the authors clarify the protocol and quantify the calibration error. For peer review: yes, send it out—a competent reviewer can handle the fixes in one round.","headline":"A plausible but under-specified calibration heuristic for GNN recommenders; the central claim is blocked by an unresolved training-vs-post-hoc ambiguity and missing quantitative calibration metrics.","tokens_in":8063,"tokens_out":664,"would_cite":false,"duration_ms":6991,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"GNN-based recommenders overstate confidence, and a two-part calibration scheme realigns confidence with accuracy while lifting top-20 precision.","keywords":["recommendation","confidence calibration","graph neural networks","overconfidence","rating calibration","reliability diagram","trustworthy recommendation"],"falsifier":"On a held-out test set, apply only the rating calibration of Eq. (5) to the frozen outputs of a trained model and rebuild the reliability diagram; if the bins stay as far from the diagonal as before, then the calibration claim fails and the reported improvement would have to come entirely from the loss term during training.","tokens_in":7014,"feed_emoji":"🎯","tokens_out":3232,"duration_ms":23817,"temperature":0.7,"pith_summary":"The paper tries to establish that existing GNN-based recommender models are overconfident: their predicted confidence is systematically higher than actual accuracy. It proposes Conf-GNNRec, a two-part scheme that compresses excessively high ratings with a user-specific logarithmic transform and adds a confidence loss that penalizes high-confidence wrong predictions. The authors report that applying Conf-GNNRec improves Precision@20 and Accuracy@20 on all tested baseline models and datasets, while shifting reliability diagrams closer to the diagonal. The underlying goal is to make recommendation confidence trustworthy enough for high-stakes use.","feed_headline":"GNN recommender overconfidence gets a calibration fix","feed_subtitle":"A rating compression plus a confidence loss aligns predicted confidence with true accuracy on three public datasets.","key_machinery":"The load-bearing pieces are (i) the perfect-calibration condition of Eq. (3), which serves as the target; (ii) the rating recalibration formula of Eq. (5), which leaves ratings at or below the user's mean untouched and nonlinearly compresses ratings above it with a temperature-controlled logarithm; and (iii) the confidence loss of Eq. (6), which multiplies $(1 - p_{u,i})$ by a confidence-scaled log term to penalize high-confidence mistakes on negative samples. The reliability diagram, using ten equal bins, is the measuring device that both exposes the overconfidence and demonstrates the fix.","core_discovery":"The central discovery is that the confidence output of GNN-based recommenders does not match accuracy, violating the ideal calibration condition $P(\\hat{y}_{u,i} = y_{u,i} \\mid \\hat{p}_{u,i} = p) = p$ for all $p \\in [0,1]$. The paper quantifies this with reliability diagrams on LightGCN and KGCL, showing confidence bars far above the diagonal, and then shows that its two-part scheme—user-specific log compression of ratings above the user's mean, plus a confidence loss that grows nonlinearly with confidence on wrong predictions—brings confidence closer to accuracy while improving ranking metrics.","pith_inferences":["The paper does not pin down whether the confidence loss is applied during training or only after; if it is a training loss, the reported gains mix calibration with representation learning, and a pure post-hoc test of Eq. (5) alone would be needed to isolate the calibration effect.","The method's reliance on a per-user mean rating may behave differently for cold-start users with few interactions; this is a natural failure mode worth testing.","Because the overconfidence stems from the final normalization layer's relative magnitudes, similar miscalibration is plausible in non-GNN recommenders that share that output structure; Conf-GNNRec's components might transfer there."],"forward_implications":["If the central claim is right, any GNN-based recommender that outputs probabilities through a normalization layer is likely overconfident, so calibration should become a standard evaluation step.","Conf-GNNRec acts as a drop-in correction that improves top-20 precision and accuracy across LightGCN, KGAT, MVIN, and KGCL.","The improvement is larger on denser datasets and on more complex models such as KGCL, suggesting that models with more information aggregation accumulate more overconfidence that the fix can correct.","Reliability diagrams become closer to the diagonal after the fix, meaning that high-confidence predictions are genuinely more reliable, which matters for high-stakes recommendation domains."],"supporting_citations":[{"why":"LightGCN is the primary baseline used to demonstrate overconfidence and serves as a testbed for Conf-GNNRec.","marker":"[6]"},{"why":"KGCL provides the second reliability-diagram demonstration and is one of the four evaluated baseline methods.","marker":"[7]"},{"why":"KGAT is used as a baseline in the before/after performance comparison.","marker":"[8]"},{"why":"MVIN is used as a baseline in the before/after performance comparison.","marker":"[9]"},{"why":"Supplies the equal-bin reliability-diagram protocol used to quantify miscalibration and evaluate the proposed fix.","marker":"[16]"}],"fun_headline_variants":["Calibrating overconfident GNN recommenders with Conf-GNNRec","GNN recommender confidence gets a reality check","Conf-GNNRec: Compress ratings to fix overconfidence","Overconfident GNN recommenders? A calibration recipe","Aligning GNN recommender confidence with true accuracy"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper never states clearly whether Conf-GNNRec is applied only to a frozen model's ratings or used as an additional training loss, and the empirical claims depend on which one it is.","fun_headline_variants_meta":{"raw":{"variants":["Calibrating overconfident GNN recommenders with Conf-GNNRec","GNN recommender confidence gets a reality check","Conf-GNNRec: Compress ratings to fix overconfidence","Overconfident GNN recommenders? A calibration recipe","Aligning GNN recommender confidence with true accuracy"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000241,"raw_usage":{"total_tokens":1487,"prompt_tokens":875,"completion_tokens":612,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":491,"completion_tokens_details":{"reasoning_tokens":529}},"tokens_in":491,"tokens_out":612,"duration_ms":4606,"temperature":1.0,"reasoning_tokens":529,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T15:00:21.968062+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"On a held-out test set, apply only the rating calibration of Eq. (5) to the frozen outputs of a trained model and rebuild the reliability diagram; if the bins stay as far from the diagonal as before, then the calibration claim fails and the reported improvement would have to come entirely from the loss term during training.","supporting_citations":[{"cited_title":"Kgat: Knowledge graph at- tention network for recommendation,","cited_arxiv_id":null,"evidence_quote":"KGAT is used as a baseline in the before/after performance comparison."},{"cited_title":"Mvin: Learning multiview items for recommendation,","cited_arxiv_id":null,"evidence_quote":"MVIN is used as a baseline in the before/after performance comparison."},{"cited_title":"Lightgcn: Simplifying and powering graph convolution network for recommendation,","cited_arxiv_id":null,"evidence_quote":"LightGCN is the primary baseline used to demonstrate overconfidence and serves as a testbed for Conf-GNNRec."},{"cited_title":"Kgcl: A knowledge-enhanced graph contrastive learning framework for session-based recommendation,","cited_arxiv_id":null,"evidence_quote":"KGCL provides the second reliability-diagram demonstration and is one of the four evaluated baseline methods."},{"cited_title":"Be confident! towards trustworthy graph neural networks via confidence calibration,","cited_arxiv_id":null,"evidence_quote":"Supplies the equal-bin reliability-diagram protocol used to quantify miscalibration and evaluate the proposed fix."}],"review_version":1}