{"id":"751007ef-5a42-42bc-92ca-762aeb89b55f","arxiv_id":"2411.14035","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"HG2M and HG2M+ distill heterogeneous graph neural networks into feature-only MLPs, achieving near-teacher accuracy with up to 379x faster inference.","lead":"This paper teaches a plain MLP to imitate a heterogeneous graph neural network, cutting inference time by up to 379x while keeping most of the accuracy. It then adds a meta-path-based distillation step that sometimes beats the teacher HGNN.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Reported gains over MLPs may largely reflect transductive soft-label access to validation/test nodes; purely inductive results show much smaller gains and large gaps to HGNNs.","rationale":"The reader's weakest assumption identifies the transductive evaluation asymmetry, and my reading converges on the same point as the most load-bearing issue for the central claim. The core mechanism of the paper is legitimate and the efficiency results are credible, since removing structure-dependent neighbor fetching will naturally reduce latency. The accuracy claim, however, is where the protocol matters most: training the student on teacher soft labels for validation and test nodes gives it a form of access that the vanilla MLP baseline does not receive, so the reported margin over MLPs conflates distillation with transductive pseudo-label access. The inductive results in Table IV already suggest that without this access the student is far less competitive with the teacher on several datasets, which is a concrete, checkable prediction of my concern. The proposed test directly isolates the effect by removing the asymmetric soft-label access while keeping the method otherwise unchanged. I agree with the reader's conditional verdict: the paper should be accepted only after the claims are conditioned on the evaluation protocol and the comparison with MLP baselines is clarified. I do not see a basis for rejection, because the inductive setting still shows positive gains over MLPs and the speedup evidence is strong.","tokens_in":21792,"tokens_out":9215,"duration_ms":101200,"concrete_test":"Re-run the Table III/IV comparison with HG2M and HG2M+ trained using soft-label distillation only over the labeled training nodes V_L^t, with RND and RMPD restricted to labeled nodes as well, so the student never sees teacher soft labels for validation or test nodes. Keep all other hyperparameters fixed and compare against the same MLP and RSAGE baselines. If the transductive/prod gains over MLP collapse toward the inductive numbers in Table IV, the headline 'significantly outperform vanilla MLPs' is substantially an artifact of test-node soft-label access; if the gains largely persist, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section V-A5 states that in the transductive setting the entire graph, including validation and test nodes, is used to generate teacher soft labels, and Eq. (2) is trained with lambda = 0 on all target-type nodes, so the student is directly trained on teacher predictions for the very nodes on which it is later evaluated. The vanilla MLP baseline receives no such soft-label supervision. This makes the headline claim 'significantly outperform vanilla MLPs' an apples-to-oranges comparison: the student has effectively been given a teacher-generated training target for each test node, while the MLP has only the labeled training nodes. The inductive rows of Table IV are the natural control: when soft labels for the evaluation nodes are withheld, HG2M+ beats MLP by only about 0.8-4.3 points and is 2.3-11.6 points below RSAGE on TMDB, CroVal, and ArXiv, whereas the transductive/prod numbers show much larger apparent gains. The concern is not that using unlabeled test features is illegitimate in transductive learning; it is that the abstract and RQ1 conclusions state the result without conditioning on this asymmetric access, so the central accuracy claim as written is likely overstated.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper proposes HG2M and HG2M+, two knowledge-distillation methods for compressing heterogeneous graph neural networks (HGNNs) into feature-only MLP students. HG2M trains an MLP on node features using soft labels from a pretrained HGNN teacher, while HG2M+ additionally filters teacher predictions by confidence/uncertainty (reliable node distillation) and injects meta-path-based relational supervision through a learned intra-class neighbor selector (reliable meta-path distillation). The authors evaluate on six heterogeneous graph datasets under transductive, inductive, and combined production protocols, reporting accuracy competitive with or better than the RSAGE teacher and inference speedups of 39.81x to 379.24x over the teacher. The paper also includes ablations, alternative teacher architectures, feature-noise robustness, and hyperparameter sensitivity studies.","tokens_in":21983,"tokens_out":11465,"duration_ms":115549,"significance":"If the reported accuracy held under a fair comparison, the paper would provide a practical recipe for latency-sensitive deployment of heterogeneous graph models, and the breadth of experiments (six datasets, including IGB-3M-19; multiple teacher architectures; ablations; noise robustness) is a genuine strength. The public code release is also commendable. However, the central accuracy claim is currently supported mainly by a transductive protocol in which the student receives teacher soft labels for the evaluation nodes, while the vanilla MLP and HGNN baselines do not. The inductive rows of Table IV, which are the cleanest control, show much smaller gains over MLPs and clear degradation below the teacher. The efficiency result is robust by construction, since the student is an MLP; the accuracy contribution needs re-scoping and re-evaluation before the paper can be accepted.","major_comments":[{"comment":"The transductive evaluation is asymmetric. The paper states in Section V-A5 that in the tran setting the entire graph, including validation and test nodes, is used to generate soft labels, and Eq. (2) is trained with lambda = 0, so the student is trained exclusively on teacher predictions for every target node, including the nodes on which it is later evaluated. The vanilla MLP and RSAGE baselines receive no such supervision, so Table III and the tran/prod rows of Table IV do not provide a fair comparison for the headline claim of significantly outperforming vanilla MLPs. Additionally, model selection is performed on validation accuracy (Section V-A3), but validation nodes are themselves used as training targets in this protocol, which makes the reported numbers optimistic. Please rerun the transductive experiments with soft labels generated only for labeled and observed nodes (as in the ind protocol), or add baselines that receive exactly the same teacher soft-label supervision, and condition all abstract and RQ1 claims on the resulting protocol.","section":"Section V-A5, Eq. (2), Section V-A3"},{"comment":"The inductive rows are the natural control and contradict the generality of the claim. In Table IV, on the ind rows HG2M+ improves over MLP by only 1.40 to 4.81 points, whereas the tran rows show gains of 6.10 to 15.98 points; and on the ind rows HG2M+ is below RSAGE by 11.60 points on ArXiv, 7.85 points on TMDB, and 2.30 points on CroVal. Since the inductive setting is the deployment-relevant one (teacher soft labels for new nodes are not available at inference), the statement in the abstract and Section V-B that HG2Ms achieve competitive or even better performance than HGNNs should be restricted to the transductive case or substantially tempered.","section":"Section V-B, Table IV"},{"comment":"No significance tests are reported. With five seeds and overlapping standard deviations (e.g., Table III, IGB-549K-19: HG2M+ 59.62 +/- 0.49 vs. RSAGE 58.53 +/- 0.55; Table VIII, IGB-549K-19: w/RND 59.24 +/- 0.47 vs. HG2M+ 59.62 +/- 0.49), the word 'significant' is not statistically supported. Paired significance tests across the five seeds would strengthen the main comparison.","section":"Section V-A3, Tables III and IV"}],"minor_comments":[{"comment":"The intra-class classifier in RMPD uses teacher soft-label similarity f3_uv = z_u dot z_v as a feature to decide which teacher soft labels z_u will be distilled; this is a mild circularity because the same teacher outputs are used both to select and to supervise the selected pairs. Please discuss this limitation and, ideally, ablate the f3 feature.","section":"Section IV-B2"},{"comment":"The information-theoretic analysis is informal: Eq. (7) is a mutual-information decomposition, but no estimates of I(E;y) or I(X;y|E) are provided, so the paragraph should be framed as intuition rather than a proof.","section":"Section IV-C"},{"comment":"The contribution bullet claiming to be the first to integrate HGNNs with MLPs through knowledge distillation is difficult to reconcile with the inclusion of HIRE [45] as a heterogeneous knowledge distillation baseline; please clarify the specific distinction or soften the novelty claim.","section":"Section I and Section V-C"},{"comment":"The hyperparameter table is ambiguous: the # layers and hidden dim rows appear to have an extra column relative to the listed model names. Please reformat so each value is clearly aligned with its model.","section":"Table II"},{"comment":"Inference times are reported on only five randomly chosen nodes without variance across node choices; please add standard deviations or describe the sampling procedure more carefully.","section":"Table VI"},{"comment":"The row labeled 'HG2M/GLNN' is unclear because GLNN is a homogeneous-graph method; please clarify whether this entry is HG2M with a GLNN-style objective or a separate baseline.","section":"Table V"},{"comment":"The limitations paragraph discusses RMPD feature generalization and task scope, but does not mention the transductive soft-label access issue raised above; adding this as a limitation would improve transparency.","section":"Limitations and Future Work"}],"recommendation":"major_revision","confidential_remarks":"The main risk is the transductive protocol asymmetry: the paper's headline gains over MLPs are driven by training the student on teacher soft labels for the exact evaluation nodes, while baselines do not receive this signal. The inductive rows of Table IV show a much smaller and less favorable picture, so the central claim needs to be re-scoped or re-generated. The novelty claim about being the first to distill HGNNs into MLPs also deserves scrutiny given that HIRE [45] is listed as a heterogeneous KD baseline. If the authors fix the evaluation protocol and revise the conclusions accordingly, the paper could become acceptable."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's my take on 2411.14035. The paper is a solid, if incremental, extension of GNN-to-MLP distillation to heterogeneous graphs, with a useful practical payoff: a 379x inference speedup on IGB-3M-19 at near-teacher accuracy. The two additions—reliable node distillation (RND) and reliable meta-path distillation (RMPD)—are reasonable variations on known pseudo-labeling and meta-path ideas, and the experimental breadth is real: six datasets, transductive and inductive splits, multiple teacher architectures, ablations, noise robustness, and released code.\n\nThe main flaw is the comparison against vanilla MLPs. In the transductive protocol (Section V-A5), soft labels are generated for all target-type nodes, including validation and test, and the student is trained on them with lambda = 0. The MLP baseline gets no such extra supervision. So the headline 'significantly outperform vanilla MLPs' is partly an artifact of the student seeing teacher predictions on the very nodes it is later evaluated on. The inductive rows of Table IV are the honest control: there, HG2M+ beats MLP by 1.4-4.8 points and trails RSAGE by up to 11.6 points on TMDB, CroVal, and ArXiv. That is still a positive result, but it is much weaker than the transductive numbers suggest. The abstract and RQ1 conclusion should condition on this.\n\nThe novelty claim also needs correcting. The paper says 'distilling HGNNs into MLPs for heterogeneous graphs has not been explored,' yet it later compares against HIRE (2022) and TeKAP (ICLR 2025), both of which do heterogeneous graph distillation. The 'first' statement doesn't survive contact with its own baseline table.\n\nMinor concerns: no significance tests beyond overlapping stds from five runs, and the information-theoretic justification in Section IV-C is loose—the student maximizes I(X;y), not I(X;y|E), so the decomposition doesn't quite support the argument.\n\nOverall, the empirical contribution is useful and the speedups are credible. The paper deserves a serious referee, but it needs a revised novelty statement, an explicit discussion of the transductive asymmetry, and ideally a significance analysis. I'd send it out.","headline":"Useful empirical extension of GNN-to-MLP distillation to heterogeneous graphs, with real speedups, but the headline MLP comparison is inflated by transductive soft-label access and the novelty claim ignores existing heterogeneous distillation baselines.","tokens_in":22535,"tokens_out":4236,"would_cite":true,"duration_ms":38657,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Distilled feature-only MLPs can match or beat heterogeneous GNN teachers on node classification while running inference up to 379× faster, the authors argue.","keywords":["heterogeneous graph neural networks","knowledge distillation","MLP student","meta-path","inference acceleration","node classification","reliable pseudo-labeling","transductive learning"],"falsifier":"Re-run the transductive experiments on one small dataset (e.g., TMDB) with soft labels generated only from training-time nodes, so validation and test nodes are unseen by the teacher, and check whether HG2M and HG2M+ still outperform vanilla MLPs and stay within about 1% of the teacher. If the margin over MLPs shrinks or disappears, the reported gains depend on the asymmetric access; if it holds, the distillation claim is robust.","tokens_in":1717,"feed_emoji":"⚡","tokens_out":2462,"duration_ms":69535,"temperature":0.7,"pith_summary":"This paper tries to show that the accuracy of heterogeneous graph neural networks (HGNNs) can be transplanted into plain MLPs that never look at the graph at inference time. The authors propose two students, HG2M and HG2M+, trained on the soft labels of a teacher HGNN; the plus version also distills reliable meta-path-based neighbor knowledge so that heterogeneous semantics survive without structure. On six datasets, the distilled MLPs match or beat the teacher HGNN on most benchmarks, improve on vanilla MLPs by about 6–8% on average, and infer 39.81× to 379.24× faster. The payoff, if correct, is that latency-constrained deployments can use heavy relational models only offline, during teacher training, and serve predictions with lightweight feature-only MLPs.","feed_headline":"Feature-only MLPs match hetero GNNs at 379x speed","feed_subtitle":"Trained on an HGNN teacher's soft labels, the MLP keeps near-teacher accuracy with no graph access at inference.","key_machinery":"The load-bearing machinery is teacher-student knowledge distillation instantiated on heterogeneous graphs. A teacher HGNN (the heterogeneous GraphSAGE variant RSAGE by default) produces soft labels $z_v$ for every target-type node; the student MLP minimizes $\\mathcal{L} = \\lambda \\mathcal{L}_{CE} + (1-\\lambda)\\mathcal{L}_{KL}$ in HG2M, and in HG2M+ a combined loss of reliable node distillation and reliable meta-path distillation (Eq. 6), where only high-confidence, low-entropy nodes $R$ and reliable intra-class meta-path neighbor pairs $M_P$ contribute. The meta-path distillation term uses a logistic-regression scorer on attribute similarity, connection strength, and soft-label similarity to estimate whether a neighbor shares the anchor's class, so structural semantics are injected without requiring graph access at inference. The efficiency claim follows because the deployed model is an MLP: no neighbor fetching, no relation-specific aggregation, only a forward pass on the node's own features.","core_discovery":"On the paper's own terms, the central discovery is that an MLP trained solely on node features can reproduce the node-classification behavior of a heterogeneous GNN teacher when the supervision consists of the teacher's soft labels extended with reliable and intra-class meta-path-based neighbors. HG2M uses only the standard distillation objective, a weighted combination of cross-entropy against true labels and KL divergence against teacher logits. HG2M+ adds reliable node distillation, which discards low-confidence, high-entropy teacher outputs, and reliable meta-path distillation, which trains the student to match the teacher's prediction for an anchor node's selected meta-path neighbors. Across TMDB, CroVal, ArXiv, IGB-549K-19, IGB-549K-2K, and IGB-3M-19, HG2M+ ranks first on 5 of 6 transductive benchmarks and in the production (transductive-plus-inductive) setting beats the teacher by 0.34% on average, while inference on five nodes takes 0.23–0.34 ms versus 10.75–125.15 ms for the teacher.","pith_inferences":["Editorial inference: because the transductive protocol uses the full graph to generate teacher soft labels for validation and test nodes, while vanilla MLP and HGNN baselines receive no such extra supervision, the reported margin over MLPs may partly reflect asymmetric access rather than pure distillation.","Editorial inference: the same recipe could plausibly be carried to link prediction or graph classification by choosing task-appropriate distillation targets, though the meta-path machinery would need adaptation beyond node classification.","Editorial inference: the reliable meta-path distillation relies on a logistic-regression classifier trained on labeled and reliable unlabeled pairs, so a natural stress test is whether the gains persist on graphs with weak homophily or semantically meaningless meta-paths, a limitation the paper itself flags."],"forward_implications":["If the central claim holds, latency-sensitive systems can serve node classifiers from MLPs while retaining HGNN-level accuracy, since graph-dependent computation is confined to offline teacher training.","The large-scale result (379× speedup and 28× lower inference memory on IGB-3M-19) implies that structure-free inference can be practical for large heterogeneous graphs under tight latency budgets.","Because HG2M+ improves over HG2M, the paper implies that explicitly distilling relational semantics through reliable meta-path neighbors closes most of the residual gap left by logit-only distillation.","The insensitivity of performance to the trade-off weight $\\lambda$ (setting it to 0 works best) suggests that filtered teacher soft labels alone carry enough supervision for the student, without direct label loss.","The noisy-feature experiments suggest that distilled MLPs inherit some of the teacher's structure-awareness even when input features degrade, as long as reliable nodes and meta-path pairs are used."],"supporting_citations":[{"why":"Supplies the GNN-to-MLP distillation baseline and the transductive/inductive protocol that HG2M extends to heterogeneous graphs.","marker":"[17]"},{"why":"Provides the soft-label KL-divergence knowledge distillation objective used to train the student MLPs.","marker":"[21]"},{"why":"Defines meta-paths and heterogeneous semantic attention, the basis for the meta-path distillation in HG2M+.","marker":"[11]"},{"why":"Provides the large-scale IGB datasets used for the 379× speedup and memory-efficiency evaluations.","marker":"[16]"},{"why":"Motivates filtering unreliable teacher predictions, which reliable node distillation adapts for heterogeneous graphs.","marker":"[19]"},{"why":"RSAGE, the teacher architecture, is a heterogeneous extension of GraphSAGE, so this work defines the efficiency baseline being compared.","marker":"[41]"}],"fun_headline_variants":["MLPs match hetero GNNs at 379x speed without graph access","No graph needed: MLPs rival hetero GNNs at 379x speed","Hetero GNN knowledge squeezed into MLPs for 379x faster inference","Distill hetero GNN into MLP: 379x faster inference, no graph needed","No graph? No problem: MLPs match hetero GNNs at 379x speed"],"cache_read_input_tokens":24704,"weakest_assumption_plain":"The load-bearing premise is that giving the student MLP teacher soft labels for validation and test nodes, computed from the whole graph, while giving vanilla MLP and HGNN baselines no such extra supervision still counts as a fair accuracy comparison.","fun_headline_variants_meta":{"raw":{"variants":["MLPs match hetero GNNs at 379x speed without graph access","No graph needed: MLPs rival hetero GNNs at 379x speed","Hetero GNN knowledge squeezed into MLPs for 379x faster inference","Distill hetero GNN into MLP: 379x faster inference, no graph needed","No graph? No problem: MLPs match hetero GNNs at 379x speed"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001636,"raw_usage":{"total_tokens":6527,"prompt_tokens":994,"completion_tokens":5533,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":610,"completion_tokens_details":{"reasoning_tokens":5420}},"tokens_in":610,"tokens_out":5533,"duration_ms":34197,"temperature":1.0,"reasoning_tokens":5420,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T15:35:27.819881+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the transductive experiments on one small dataset (e.g., TMDB) with soft labels generated only from training-time nodes, so validation and test nodes are unseen by the teacher, and check whether HG2M and HG2M+ still outperform vanilla MLPs and stay within about 1% of the teacher. If the margin over MLPs shrinks or disappears, the reported gains depend on the asymmetric access; if it holds, the distillation claim is robust.","supporting_citations":[{"cited_title":"Graph-less neural networks: Teaching old MLPs new tricks via distillation,","cited_arxiv_id":null,"evidence_quote":"Supplies the GNN-to-MLP distillation baseline and the transductive/inductive protocol that HG2M extends to heterogeneous graphs."},{"cited_title":"Heterogeneous graph attention network,","cited_arxiv_id":null,"evidence_quote":"Defines meta-paths and heterogeneous semantic attention, the basis for the meta-path distillation in HG2M+."},{"cited_title":"Igb: Addressing the gaps in labeling, features, heterogeneity, and size of public graph datasets for deep learning research,","cited_arxiv_id":null,"evidence_quote":"Provides the large-scale IGB datasets used for the 379× speedup and memory-efficiency evaluations."},{"cited_title":"Quantifying the knowledge in gnns for reliable distillation into mlps,","cited_arxiv_id":null,"evidence_quote":"Motivates filtering unreliable teacher predictions, which reliable node distillation adapts for heterogeneous graphs."},{"cited_title":"Inductive representation learn- ing on large graphs,","cited_arxiv_id":null,"evidence_quote":"RSAGE, the teacher architecture, is a heterogeneous extension of GraphSAGE, so this work defines the efficiency baseline being compared."}],"review_version":1}