{"id":"cef086c2-63ad-4dbe-8d54-0ea1403d624b","arxiv_id":"2505.03774","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"OODHG detects out-of-distribution nodes in heterogeneous graphs by propagating energy scores along meta-paths and classifying the remaining in-distribution nodes.","lead":"This paper proposes a method, OODHG, that detects out-of-distribution nodes in heterogeneous graphs by computing energy scores and propagating them along meta-paths, then classifying the in-distribution nodes. A generalist reader might care because real-world graphs such as social, citation, and recommendation networks contain multiple node and edge types, and detecting unseen or anomalous node types in them is an open practical problem.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Test-set threshold fitting inflates Tables 2-3; the claimed gains over Energy may not survive a valid held-out threshold protocol.","rationale":"In good faith, the paper does two things: it proposes a plausible meta-path energy propagation mechanism and it evaluates it. The mechanism is coherent: Eq. 6 is a standard lazy random-walk smoothing restricted to target-type nodes through the meta-path adjacency matrix of Eq. 7, and the energy loss of Eq. 11 is a squared hinge that matches the negative-energy decision rule. The ablation study in Sec. 5.3 suggests both EP and L_E contribute, and the visualizations in Figs. 4-5 corroborate that propagation separates ID/OOD energy distributions. Thus the reader's weakest_assumption, about local correlation of OOD status along meta-paths, is not the weakest link: the paper's own ablations and visualizations provide indirect evidence for it, and even if the mechanism is imperfect, its failure would degrade the method rather than invalidate the contribution. The more load-bearing issue is the evaluation protocol. Section 5.1 explicitly says thresholds were explored on the test set and the best performance for each method was reported. This is a correctness risk for every reported comparison in Tables 2 and 3, and it directly underlies the headline claim of superiority over Energy and other baselines. The reader flagged this as a secondary concern, so we partially agree, but I elevate it to the primary concern. I keep the verdict CONDITIONAL because the concern is addressable by a rerun with a validation-based threshold protocol, and the method's internal ablations suggest the core mechanism is real. If the mechanism lacked independent support, I would have voted REJECT or UNVERDICTED, but the ablations and visualizations provide sufficient support to justify a conditional accept pending the threshold re-evaluation.","tokens_in":19633,"tokens_out":1669,"duration_ms":15864,"concrete_test":"Re-run the full Table 2 and Table 3 comparisons with thresholds fixed using only the training/validation split (e.g., energy threshold chosen to maximize validation F1, or using the standard 95% TPR operating point computed on validation), then evaluate all methods on the held-out test set at their fixed thresholds. Report the deltas; if OODHG's AUROC/FPR@95 margins over Energy shrink by more than roughly 30% or change sign, the central claim of superiority needs to be reframed as conditional on test-set threshold tuning.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The paper's central claim is that OODHG outperforms baselines on OOD detection and ID classification. The experiments are weakened by test-set threshold fitting: Section 5.1 states that the energy threshold was explored between 1 and 2 with step 0.05 and the softmax threshold between 0.5 and 0.9, reporting 'the corresponding best performance for each case.' On the test set, choosing each method's threshold to maximize its reported AUROC, AUPR, FPR@95, and F1 makes the comparisons in Tables 2 and 3 partially in-sample. A detector that is actually worse can appear better if its threshold is chosen more favorably, and the reported margins (e.g., DBLP FPR@95 improves from 16.20 to 5.99) may partly reflect threshold optimism rather than the energy propagation mechanism. The paper also mixes threshold selection with the energy-loss margin m_in in the same paragraph, making it unclear whether m_in was also tuned on the test set. The ablation study in Figure 3 does show EP and L_E help under a consistent protocol, but its reported numbers use the same test-set-selected thresholds. The propagation assumption in Eqs. 6-7 is plausible and is supported by the ablations, so the evaluation protocol is the load-bearing weakness.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes OODHG, a method for out-of-distribution detection in heterogeneous graphs. It uses an HGNN encoder (SeHGNN) to produce logits, computes per-node energy scores (Eq. 3), propagates these scores along user-selected meta-paths (Eqs. 6-7), and averages the propagated scores across paths (Eq. 8). Training combines a classification loss with a squared hinge energy loss that penalizes ID nodes whose energy exceeds a margin (Eqs. 9-11). Experiments on DBLP, ACM, and IMDB compare OODHG with graph-specific baselines (OpenWGL, OODGAT, GNNSafe) and classical energy/softmax detectors (MSP, ODIN, Energy), reporting AUROC, AUPR, FPR@95, and K+1 Micro/Macro-F1. The paper claims consistent superiority in both OOD detection and ID node classification, with ablations suggesting that both energy propagation and the energy loss contribute to the reported gains.","tokens_in":19964,"tokens_out":7787,"duration_ms":80329,"significance":"If the results survive a stricter evaluation protocol, OODHG is a solid and timely contribution: node-level OOD detection in heterogeneous graphs is indeed underexplored, and the proposed mechanism is simple, interpretable, and naturally extends energy-based detection to heterogeneous structure. The use of meta-path-based propagation is coherent with the SeHGNN backbone, and the ablation study in Figure 3 at least isolates the two proposed components. The main weakness is that the quantitative claims rest on an evaluation protocol in which decision thresholds and possibly hyperparameters are selected using test-set labels. The threshold-free AUROC/AUPR results and the qualitative visualizations are encouraging, but they do not by themselves validate the K+1 classification gains or the FPR@95 comparisons. The paper should be judged after a clean held-out evaluation.","major_comments":[{"comment":"The threshold selection procedure invalidates the K+1 classification comparisons in Table 3 and potentially affects the FPR@95 comparisons in Table 2. The text states that the energy threshold was explored between 1 and 2 with step 0.05, the softmax threshold between 0.5 and 0.9, and that 'the corresponding best performance for each case' was reported. This selects the decision threshold using test-set labels, so the Micro-F1 and Macro-F1 numbers in Table 3 (e.g., DBLP Micro-F1 91.37 vs 88.97 for Energy) are in-sample estimates of the best achievable threshold for each method, not evaluations of a fixed decision rule. The issue is material on IMDB, where the reported gains are small (e.g., Micro-F1 51.94 vs 50.52). Please recompute all threshold-dependent metrics using thresholds selected on a validation split, or report threshold-free metrics (AUROC, AUPR) as the primary comparison. If FPR@95 is computed by the standard procedure of fixing TPR at 95% from the score distribution, state this explicitly; otherwise it inherits the same test-set fitting problem.","section":"Section 5.1"},{"comment":"The selection of hyperparameters alpha, gamma, K, and especially the margin m_in is not described with a clear protocol. Section 5.1 reports dataset-specific values of m_in (-3, -2, -3), and Section 5.5 presents a parameter sensitivity analysis only on DBLP. If any of these hyperparameters were chosen by test-set performance, then even the threshold-independent AUROC/AUPR entries in Table 2 are partly in-sample, because a model trained with test-tuned m_in is not a model selected by a valid model-selection procedure. Please specify for each hyperparameter whether it is a fixed default, selected on a validation split, or selected by sensitivity analysis, and provide evidence that the reported rankings are stable under reasonable variations of these hyperparameters.","section":"Section 5.1 and Section 5.5"},{"comment":"For the graph-based baselines OpenWGL, OODGAT, and GNNSafe, the text says heterogeneous graphs are converted into homogeneous views using different meta-paths and that 'the best performance' is reported. If that best is selected on the test set, the baselines also benefit from in-sample meta-path selection, making the comparison asymmetric: OODHG is evaluated as a single configuration while each baseline is allowed to pick its best meta-path view. Please report the protocol explicitly, e.g., validation-based meta-path selection, averaging over all meta-paths, or a fixed meta-path set shared by all methods.","section":"Section 5.1, Baseline Methods"}],"minor_comments":[{"comment":"The decision rule is given as '-E_i <= tau' in Algorithm 1 but as '-E_i < tau' in Figure 2 and in Eq. (4); make the inequality consistent.","section":"Algorithm 1 and Figure 2"},{"comment":"Equation (8) mixes vector and scalar notation: E^(k,P_n)_i is used for the propagated energy of node i under path P_n, while E_i is defined as the final scalar energy. Please define all symbols explicitly before Eq. (6).","section":"Eq. (8)"},{"comment":"The text refers to 'Marginal Hyperparameters m_in' and contains the typo 'm_in m' in the margin hyperparameter paragraph; Figure 6(d) also uses inconsistent formatting for m_in. Please correct these typos.","section":"Section 5.5 and Figure 6(d)"},{"comment":"Reference [22] appears to be a duplicate of reference [13]: both are titled 'Supervised algorithmic fairness in distribution shifts: A survey' with the same arXiv identifier. Please merge or correct.","section":"References"},{"comment":"The IMDB multi-label conversion, retaining only the first two labels and binarizing them, is a large simplification that is acknowledged in the text; please state explicitly that the same conversion is applied to all compared methods and discuss whether this preprocessing favors any particular detector.","section":"Section 5.1, IMDB preprocessing"},{"comment":"The visualization analysis in Figures 4 and 5 is qualitative only; since the paper already computes quantitative metrics, a small table reporting overlap or separation before and after energy propagation would make the claim more precise.","section":"Section 5.4"}],"recommendation":"major_revision","confidential_remarks":"The manuscript addresses a relevant gap and the proposed mechanism is plausible, but the central quantitative claims are undermined by test-set threshold selection and unclear hyperparameter selection. This is a localized experimental-protocol problem rather than a fundamental flaw in the method, so I recommend major revision rather than rejection. Please ask the authors to rerun the comparisons with a validation-based threshold protocol, clarify the FPR@95 computation, and disclose the selection procedure for all hyperparameters and for the baseline meta-path views. The novelty claim itself is not the main concern."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's the short version: this is a clean, incremental extension of energy-based OOD detection to heterogeneous graphs, and the core mechanism—meta-path-constrained energy propagation plus an energy hinge loss—is sensible and well motivated. The ablations show both components help. The main experimental weakness is the threshold protocol: Section 5.1 says they scanned thresholds and reported the best result for each case. That makes the K+1 classification numbers in Table 3 in-sample and inflated. The stress-test note overreaches a bit in saying this also undermines AUROC/AUPR/FPR@95; those are ranking metrics and don't depend on the reported threshold, so Table 2 is largely clean. But Table 3 is a real problem, and the margin m_in and hyperparameters alpha, gamma, K are not disclosed for the main runs, with no code released. These are fixable, not fatal.\n\nWhat's new: applying energy propagation along meta-paths whose endpoints are the target node type, with row-normalized adjacency products and averaging over paths. That is a straightforward adaptation of prior graph-energy propagation work [36, 39], but it's the right simple recipe for heterogeneous graphs. The paper is clearly written, the equations are correct, and the design choices are explained.\n\nWhat's soft, in proportion: the threshold fitting is the load-bearing issue for the classification claim. The baselines got the same treatment, so the comparison is not unfair in protocol, but the absolute numbers are optimistic and the claimed margins over Energy could shrink under a proper validation-set threshold. The propagation assumption—that averaging with meta-path neighbors improves the OOD signal—is plausible and the ablations support it, but there's no formal or isolated test of when it fails. The parameter sensitivity analysis on DBLP is nice, but it doesn't give the exact settings used for the main tables.\n\nWho it's for: anyone working on OOD detection in heterogeneous graphs will find a useful baseline and a clear starting point. It deserves a serious referee. I'd send it to review with a request for a validation-based threshold selection protocol, full hyperparameter disclosure, and code. My verdict would be conditional accept: the idea is worth publishing, but the evaluation needs to be re-run honestly.","headline":"Useful heterogeneous-graph extension of energy-based OOD detection, but the reported K+1 gains are partly artifacts of test-set threshold selection.","tokens_in":20410,"tokens_out":3296,"would_cite":true,"duration_ms":32939,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that propagating energy scores along meta-paths lets a heterogeneous-graph model reliably flag out-of-distribution nodes, reporting gains over seven baselines on three real-world networks.","keywords":["heterogeneous graph","out-of-distribution detection","energy score","meta-path","energy propagation","graph neural networks","node classification","energy loss"],"falsifier":"Construct a heterogeneous graph in which OOD nodes are deliberately placed so that none of their meta-path neighbors are OOD; if OODHG's detection AUROC on that graph is no better than plain per-node energy scoring, the propagation step fails for non-local OOD patterns.","tokens_in":19479,"feed_emoji":"🔍","tokens_out":10692,"duration_ms":100184,"temperature":0.7,"pith_summary":"Out-of-distribution detection is well studied for images and homogeneous graphs, but heterogeneous graphs — where nodes and edges come in several types, as in citation, social, and recommender networks — have received little attention. The paper tries to close that gap with OODHG, a method that computes an energy score for each target-type node from the logits of a heterogeneous graph neural network, then refines that score by propagating it along meta-paths that connect same-type nodes. The training objective adds an energy constraint that pushes in-distribution node energies low, making the OOD threshold easier to set. On the DBLP, ACM, and IMDB datasets, the authors report that OODHG outperforms homogeneous-graph OOD detectors and classical softmax- and energy-based baselines on both OOD detection and in-distribution node classification. The takeaway claim is that structural propagation, not just better node features, is what makes the energy signal work in heterogeneous graphs.","feed_headline":"Energy propagation along meta-paths sharpens OOD detection","feed_subtitle":"On three heterogeneous graph datasets, the method beats softmax and energy baselines on detection and classification.","key_machinery":"The central object is the meta-path energy-propagation operator. For a meta-path $P_n$ whose first and last node types are the target type, the paper composes row-normalized adjacency matrices $\\hat A_{t_i,t_{i+1}}$ between consecutive node types to obtain $\\hat A$, a row-normalized adjacency between target-type nodes, and then iterates $\\mathbf{E}^{(k,P_n)} = \\gamma \\mathbf{E}^{(k-1,P_n)} + (1-\\gamma)\\hat A \\mathbf{E}^{(k-1,P_n)}$; the final score for a node is the average over all chosen meta-paths. This operator injects heterogeneous structure into the OOD signal: it lets a target-type node inherit the energy of semantically related same-type neighbors, so the decision rule $-\\hat E_i \\le \\tau$ sees smoothed, structurally aware scores instead of raw per-node logit energies. The squared-hinge energy loss is the second piece of machinery; it directly regularizes the propagated energies during training, pulling in-distribution energies below a margin while leaving OOD energies comparatively high.","core_discovery":"On its own terms, the paper's central claim is that energy scores computed from heterogeneous-graph node logits become a reliable OOD detector when they are propagated along meta-paths whose endpoints are the target node type. The proposed OODHG pipeline first learns node embeddings with a heterogeneous graph neural network, computes per-node energy $E_i = -\\log \\sum_{c=1}^{K} e^{h_i[c]}$, then refines these energies by iteratively mixing with the energies of meta-path neighbors, averages over meta-paths, and thresholds negative energy to separate OOD from ID nodes. A squared-hinge energy loss $\\mathcal{L}_E = \\frac{1}{N}\\sum_i [\\max(0, E_i - m_\\mathrm{in})]^2$ is added to the cross-entropy classification loss so that ID nodes are pulled below a margin, widening the gap to OOD nodes. The reported experiments on DBLP, ACM, and IMDB show consistent gains over homogeneous-graph OOD detectors and classical softmax/energy baselines, with ablations and density plots indicating that both the propagation step and the energy loss reduce the overlap between ID and OOD energy distributions.","pith_inferences":["Editorial extension: if OOD status is not locally correlated along the chosen meta-paths — say, OOD nodes are isolated singletons surrounded by in-distribution nodes — the propagation operator would average the signal away; testing the method on graphs with deliberately scattered OOD nodes would expose this dependence.","Editorial extension: the paper tunes the propagation weight, step count, loss weight, margin, and decision threshold per dataset; a learnable or automatically adjusted version of these parameters is an obvious next step that the current results do not cover.","Editorial extension: because the energy loss regularizes only the target node type, the same propagated-energy score could serve as a plug-in OOD signal for other target-type prediction tasks such as link prediction or anomaly detection on heterogeneous graphs, though the paper does not test this.","Editorial extension: the decision threshold is chosen by grid search and best performance is reported; a threshold-free evaluation would clarify how much of the reported gain is intrinsic to the energy score rather than to threshold selection."],"forward_implications":["On DBLP, ACM, and IMDB, the method with the strongest tested backbone raises OOD detection AUROC by about 2.6, 6.4, and 2.5 percentage points over the best baseline and cuts FPR@95 by 10.2, 32.8, and 3.6 percentage points respectively, according to the paper's tables.","Removing either energy propagation or the energy loss degrades detection performance in the paper's ablations, and the largest gains appear when both components act together.","Detection quality tracks representation quality: the paper compares three heterogeneous graph neural network backbones and reports that the strongest backbone yields the best OOD detection and classification results.","The reported comparisons favor energy-based scoring over softmax confidence in heterogeneous graphs, with the energy baseline already outperforming MSP and ODIN before propagation is added.","The extra cost is modest: with the meta-path adjacency precomputed, propagation contributes $O(k|\\Phi|N_\\mathrm{target}^2)$ per epoch, so the method does not trade away practical training time for its detection gains."],"supporting_citations":[{"why":"Supplies the energy score as an OOD signal and the energy-loss view tied to cross-entropy.","marker":"[15]"},{"why":"Provides the heterogeneous GNN backbone and the meta-path filtering and semantic-fusion setup the method builds on.","marker":"[40]"},{"why":"Shows that propagating entropy/uncertainty over graph structure improves open-set recognition, motivating the energy propagation step.","marker":"[39]"},{"why":"Defines meta-path-based neighbor semantics and the heterogeneous aggregation used to adapt baselines.","marker":"[34]"},{"why":"Introduces the DBLP, ACM, and IMDB datasets and a strong heterogeneous GNN variant used as a backbone comparison.","marker":"[18]"},{"why":"Establishes energy-based OOD detection on graphs, the direct predecessor and strongest non-propagation baseline.","marker":"[36]"},{"why":"Introduces open-world graph learning with class uncertainty, providing a key comparison baseline.","marker":"[35]"}],"fun_headline_variants":["Meta-path energy propagation sharpens OOD detection","Energy flow through meta-paths flags OOD nodes","Propagate energy to spot OOD nodes in heterogeneous graphs","Heterogeneous OOD detection boosted by energy propagation","Energy messages along meta-paths improve OOD detection"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing assumption is that out-of-distribution status is locally correlated along the chosen meta-paths: averaging a node's energy with its meta-path neighbors must make OOD nodes stand out rather than blend in.","fun_headline_variants_meta":{"raw":{"variants":["Meta-path energy propagation sharpens OOD detection","Energy flow through meta-paths flags OOD nodes","Propagate energy to spot OOD nodes in heterogeneous graphs","Heterogeneous OOD detection boosted by energy propagation","Energy messages along meta-paths improve OOD detection"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000277,"raw_usage":{"total_tokens":1695,"prompt_tokens":1037,"completion_tokens":658,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":653,"completion_tokens_details":{"reasoning_tokens":581}},"tokens_in":653,"tokens_out":658,"duration_ms":6315,"temperature":1.0,"reasoning_tokens":581,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T05:09:55.599160+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Construct a heterogeneous graph in which OOD nodes are deliberately placed so that none of their meta-path neighbors are OOD; if OODHG's detection AUROC on that graph is no better than plain per-node energy scoring, the propagation step fails for non-local OOD patterns.","supporting_citations":[{"cited_title":"Energy- based out-of-distribution detection","cited_arxiv_id":null,"evidence_quote":"Supplies the energy score as an OOD signal and the energy-loss view tied to cross-entropy."},{"cited_title":"Graph open-set recognition via entropymessage passing,in: 2023IEEE InternationalConference on DataMining(ICDM),IEEE.pp.1469–1474","cited_arxiv_id":null,"evidence_quote":"Shows that propagating entropy/uncertainty over graph structure improves open-set recognition, motivating the energy propagation step."},{"cited_title":"Energy-based out-of- distribution detection for graph neural networks, in: The Eleventh International Conference on Learning Representations","cited_arxiv_id":null,"evidence_quote":"Establishes energy-based OOD detection on graphs, the direct predecessor and strongest non-propagation baseline."}],"review_version":1}