{"id":"fa6c1c05-b347-4034-90bf-25b3e3518106","arxiv_id":"2502.06784","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"RelGNN passes messages along schema-derived atomic routes through junction tables and improves over heterogeneous GNN baselines on most RelBench tasks, with the largest gain on one trial-database regression task.","lead":"RelGNN is a graph neural network for relational databases that replaces multi-hop message passing with composite single-step messages along atomic routes derived from the database schema. On the 30-task RelBench benchmark it beats a standard heterogeneous GNN on most tasks, with the largest reported gain a 25% reduction in MAE on one regression task.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The SOTA claim is not yet supported: only two task-level baselines and no uncertainty estimates, so many reported gains may be noise.","rationale":"The reader's weakest_assumption focuses on the router premise, which is a design assumption that could limit generality. However, the paper's central claim is explicitly about RelBench performance, and the most direct threat to that claim is the quality of the empirical comparison: only one GNN baseline, no error bars, and no significance tests. Even if the router premise were false in some unseen database, RelGNN could still dominate on the evaluated RelBench tasks; conversely, if the gains are within noise or disappear against stronger baselines, the stated claim fails regardless of the mechanism's validity. The reader's rationale already mentions the weak baselines and missing variance, so there is partial agreement, but the chosen weakest_assumption is not the most load-bearing for the headline claim. I therefore recommend keeping the verdict CONDITIONAL, since the concern points to additional evidence needed rather than a fundamental flaw.","tokens_in":18612,"tokens_out":11030,"duration_ms":100705,"concrete_test":"Run RelGNN against the official RelBench leaderboard and at least three recent published RDL baselines (e.g., ContextGNN for recommendation tasks, and any GNN-based RDL model from the last year) using their public code and identical data pipeline; also rerun all 30 tasks with 10 seeds and report mean±std with paired bootstrap significance tests. If RelGNN wins a statistically significant majority (e.g., >20/30 with p<0.05) and retains the 25% outlier with a confidence interval excluding zero, the SOTA claim stands; otherwise it should be softened to 'competitive with strong baselines on RelBench'.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim ('outperforms state-of-the-art baselines on the vast majority of tasks') is load-bearing on the comparison set. The only RDL baseline is a heterogeneous GraphSAGE from the RelBench pipeline, plus LightGBM as a non-RDL method; GAT and GIN appear only as backbones in the ablation, not as tuned SOTA baselines. No standard deviations or significance tests are reported despite averaging over only five seeds, and many relative gains are 0–4% (e.g., rel-amazon user-churn +1%, item-churn 0%, rel-stack tasks 0%, rel-hm user-item-purchase 0%). A 0.5–2% gain in ROC-AUC over five seeds can easily fall within run-to-run variance, especially on tasks with large test sets. Without comparison to the broader set of published RelBench entries (e.g., ContextGNN, LLM-based RDL methods, or the official leaderboard) and without error bars, the 'vast majority' and 'up to 25%' statements are not established. This is load-bearing because if the gains are within noise or vanish against stronger baselines, the paper's headline claim fails even if the atomic-route mechanism is sound.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes RELGNN, a GNN architecture for relational entity graphs built from databases. Its core idea is to detect 'bridge' and 'hub' node types (tables with two or more foreign keys), derive 'atomic routes' from the schema, and replace the usual two-hop message passing through these nodes with a single composite aggregation using attention. The method is evaluated on all 30 tasks of the RelBench benchmark, claiming state-of-the-art performance on 27 of 30 tasks with up to 25% improvement over the standard heterogeneous GraphSAGE baseline.","tokens_in":18776,"tokens_out":7393,"duration_ms":59841,"significance":"If the empirical claims were fully supported, RELGNN would be a valuable architectural default for relational deep learning: it is schema-automatic, model-agnostic, and the ablation in Appendix C credibly isolates the benefit of atomic routes from the attention mechanism by using a GraphSAGE-style aggregation variant. The paper also releases code. However, the current experimental evidence only weakly supports the headline claim: results are reported as five-seed means without variance or significance testing, and the comparison set for the 'state-of-the-art' claim is very small (LightGBM plus one heterogeneous GraphSAGE instantiation; GAT and GIN appear only in the ablation). The significance of the contribution will therefore depend on whether these empirical gaps are closed.","major_comments":[{"comment":"All performance comparisons are reported only as point estimates averaged over five seeds, with no standard deviations, confidence intervals, or significance tests. Several of the claimed gains are small in absolute terms (e.g., Table 1: rel-amazon user-churn +1%, item-churn 0%, rel-stack user-engagement 0%, user-badge 0%; Table 2: rel-stack post-votes 0%; Table 3: rel-hm user-item-purchase 0%). Without uncertainty estimates, these gains cannot be distinguished from run-to-run noise, and the central claim that RELGNN 'outperforms state-of-the-art baselines on the vast majority of tasks' is not established. The authors should report variance, ideally perform paired significance tests across the five seeds, and state how many of the 30 improvements are significant at a conventional level.","section":"Sections 5.1-5.3, Tables 1-3"},{"comment":"The 'state-of-the-art' claim is not supported by the chosen baselines. Tables 1-3 compare RELGNN only against LightGBM and a single heterogeneous GraphSAGE implementation from the RelBench pipeline; GAT and GIN are shown only in the ablation tables (Tables 5-7). The related work (Section 6) discusses more recent RDL methods such as ContextGNN and LLM-based approaches, but no experimental comparison is made. To justify 'state-of-the-art,' the paper should compare against at least the current leading RelBench entries (including any published leaderboard results) or explicitly limit the claim to the RelBench reference baseline.","section":"Abstract, Section 5, Section 6"},{"comment":"The design premise that bridge and hub nodes 'function mostly as routers' and contribute little semantic content is asserted without empirical support. The composite message passing in Eq. (4) fuses the intermediate node's embedding with the source embedding via a learnable linear combination, but the model is not evaluated on any schema where junction-table attributes carry strong predictive signal, nor is a variant tested that retains the intermediate node's features separately. The authors should either provide evidence for the router assumption or demonstrate robustness to feature-rich connector tables; otherwise the generality of the approach beyond the specific RelBench schemas is uncertain.","section":"Section 3.1, Eq. (4)"}],"minor_comments":[{"comment":"Atomic routes for multiple foreign keys are described as 'hyperedges' (Definition 4.1, case 2), but they are implemented as ordinary aggregated paths; the terminology is confusing and should be clarified or removed.","section":"Section 4.1"},{"comment":"Equation (5) includes a projection term W_proj h_dst in addition to the attention aggregation, which acts as a residual connection; this design choice is not discussed.","section":"Eq. (5)"},{"comment":"The relative gain reported in Tables 1-3 is never defined; the authors should state how it is computed, especially for MAE where lower is better.","section":"Tables 1-3"},{"comment":"The paper does not report the values of K for the MAP@K recommendation metrics, nor the hyperparameter settings (hidden dimension, number of layers, fanout) used for the reported results; this information is needed for reproducibility.","section":"Section 5"},{"comment":"Several typos: 'Figure 1 and 3' should be 'Figures 1 and 3' (Section 4.1); Table 2 header contains an extra parenthesis '(RELGNN (ours))'; Appendix D says 'of d is the dimension' and should be 'and d is the dimension'; the dataset name appears inconsistently as 'RELBENCH' and 'RelBench' throughout.","section":"Throughout"},{"comment":"RelGNN without attention occasionally outperforms the full RelGNN with attention (e.g., Table 5 item-churn 82.94 vs 82.64; Table 6 driver-position 3.792 vs 3.798; Table 7 user-item-purchase 0.90 vs 0.77). This is not commented on; the authors should acknowledge that the attention mechanism does not always help, which is consistent with their claim that atomic routes are the main contributor.","section":"Tables 5-7"}],"recommendation":"major_revision","confidential_remarks":"The paper is submitted to ICML by the same group that released RelBench; the evaluation is on a benchmark from the same group. This is not itself a problem, but the choice of baselines (only the reference implementation) and the SOTA claim should be scrutinized. The authors should also be encouraged to compare with the official RelBench leaderboard and to supply uncertainty estimates, since the current reporting format makes it difficult for readers to assess whether the gains are real."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: the paper is worth reading for the atomic-route idea, but the \"state-of-the-art\" framing is not yet established. The mechanism is clearly defined: atomic routes derived automatically from foreign-key structure, collapsing two-hop paths through bridge/hub nodes into one composite attention step. The ablations do a good job isolating the effect of atomic routes from attention; RELGNN without attention still beats the GraphSAGE baseline on most tasks, so the core mechanism seems to carry weight.\n\nWhat's genuinely new is the automatic, schema-level construction of routes, which contrasts with hand-designed meta-paths. The implementation is careful: same pipeline, same heads, same sampling as the baseline, and code is available. The limitation discussions are candid, especially about the self-loop tables and the recommendation head.\n\nThe soft spots are real but not fatal. The headline claim of outperforming \"state-of-the-art baselines\" is based on only two tuned baselines—the heterogeneous GraphSAGE from RelBench and LightGBM—plus GAT/GIN backbones that appear only in the ablation, not as serious contenders. All numbers are five-seed averages with no standard deviations or significance tests, and several tasks show 0% gain or tiny deltas. Relative gains of 1-4% on ROC-AUC over five seeds can easily be noise, especially on large test sets. The \"up to 25%\" figure is real but it's one regression task. So the broad SOTA claim is not yet supported; what is supported is that atomic routes help on many tasks relative to this particular GraphSAGE baseline.\n\nA second concern is the assumption that bridge and hub nodes act mostly as routers. If junction tables carry predictive attributes of their own, the composite step might discard signal. The paper doesn't test that case, so the generality is uncertain.\n\nWho is this for? RDL researchers and anyone building GNNs on relational schemas. It deserves a serious referee: the idea is clear, the experiments are extensive, and the limitations are honestly scoped. The right outcome is a major revision that adds variance estimates, compares against a broader set of published RelBench entries, and tones down the SOTA language or backs it up.","headline":"RelGNN's atomic-route message passing is a plausible architectural contribution, but the SOTA claim rests on a thin baseline set and no error bars; worth a careful revision, not a desk reject.","tokens_in":19362,"tokens_out":2122,"would_cite":true,"duration_ms":18269,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"RelGNN contends that collapsing two-hop junction-table paths into single atomic routes makes relational GNNs more accurate, winning 27 of 30 benchmark tasks.","keywords":["relational deep learning","graph neural networks","atomic routes","composite message passing","many-to-many relationships","junction tables","graph attention","relational databases"],"falsifier":"Train RelGNN and a standard two-hop heterogeneous GNN on a database whose junction tables are given predictive attributes, such as making transaction amount the sole signal for a churn outcome, and compare test accuracy; if the composite one-hop model does not at least match the two-hop baseline, the router assumption that connector tables carry little semantic content is falsified.","tokens_in":18351,"feed_emoji":"🕸️","tokens_out":13269,"duration_ms":99573,"temperature":0.7,"pith_summary":"RelGNN argues that graphs built from relational databases have a structure standard heterogeneous GNNs mishandle: many-to-many relationships are materialized as junction tables, so information between two entity types travels through an intermediate table in two hops, with the intermediate's own signal re-aggregated along the way. The paper introduces atomic routes—the minimal path between source and destination node types—and a composite message-passing scheme that completes that exchange in a single hop, fusing the intermediate and source representations before attention-weighted aggregation. On the 30 RelBench tasks, the resulting model beats the standard heterogeneous GraphSAGE baseline on 27 tasks and matches it on the rest, with gains up to 25 percent. The load-bearing structural premise is that connector tables function mostly as routers, so their semantic content can be down-weighted. If that premise holds, RelGNN offers a default GNN design for relational entity graphs that removes redundant aggregation and scales with schema size rather than row count.","feed_headline":"Atomic one-hop routes outdo standard GNNs on 27 of 30 tasks","feed_subtitle":"Collapsing junction-table two-hop paths into one composite step cuts redundancy and lifts accuracy by up to 25 percent.","key_machinery":"The central object is the atomic route: a simple path between node types, derived automatically from the schema, that supports a single-hop interaction between source and destination nodes—an edge when the source table has one foreign key, and a path $(source \\rightarrow intermediate \\rightarrow destination)$ when a table has multiple foreign keys. It carries the argument by collapsing two-hop message passing into one composite message, $m^{(l+1)}_{(dst,mid,src)} = \\mathrm{AGGR}(h^{(l)}_{dst}, \\{\\mathrm{FUSE}(h^{(l)}_{mid}, h^{(l)}_{src})\\})$, with FUSE a learned linear mix of intermediate and source embeddings and AGGR a multi-head attention that uses the destination as query. Because routes are computed at the schema-graph level, where nodes are table types rather than rows, the machinery is independent of database size, and per-route weight matrices keep different routes from entangling their signals during the final summation.","core_discovery":"The paper's claim is that the two-step message flow induced by primary–foreign key links can be reorganized into one-step routes. For a table with one foreign key, an atomic route is the edge itself; for a table with multiple foreign keys, each pair of referenced tables forms a route (source → intermediate → destination). RelGNN replaces the two-hop aggregation with a single composite step $m^{(l+1)}_{(dst,mid,src)} = \\mathrm{AGGR}(h^{(l)}_{dst}, \\{\\mathrm{FUSE}(h^{(l)}_{mid}, h^{(l)}_{src})\\})$, then sums over all routes ending at the destination. FUSE is a learned linear combination of intermediate and source embeddings, and AGGR is multi-head attention with the destination as query; distinct weights are learned for every atomic route. The authors report that this architecture outperforms the standard heterogeneous GNN on 27 of 30 RelBench tasks and achieves parity on the remaining three, and their ablations attribute the gain primarily to the atomic-route formulation rather than to attention.","pith_inferences":["An untested boundary of the router assumption: if a junction table carries predictive columns of its own, such as a transaction amount or a review text, treating it as a low-content router and folding it into a composite route could discard signal; a benchmark with informative junction attributes would settle this.","The atomic-route principle generalizes beyond RelBench: any schema-level star motif around a multi-foreign-key table can be collapsed into pairwise composite routes, so the same argument should transfer to temporal event graphs, supply-chain networks, and medical claims data where junction tables are ubiquitous.","A testable extension is to learn per-table weights that decide how much of the intermediate node's own representation survives the FUSE step, letting the model adapt to connector tables that are semantically rich instead of assuming they are pure routers."],"forward_implications":["RelGNN surpasses the standard heterogeneous GNN on 27 of 30 RelBench tasks and matches it on the other three, with the largest relative gains on complex primary–foreign key schemas such as rel-f1 and rel-trial.","The ablation shows the improvement persists when attention is removed, indicating that the atomic-route composite message passing itself, not the attention mechanism, is the source of most of the gain.","On schemas with little junction-table structure, such as rel-amazon and rel-hm, the relative gains shrink to 0–4 percent, which is consistent with the claim that the value comes from fixing bridge and hub message passing.","Because atomic routes are derived from table types rather than rows, the approach stays efficient as databases grow; the active connection count is bounded by the neighborhood sampling configuration.","Self-loop tables, as in rel-stack, are an orthogonal difficulty; adding a schema-level relative positional encoding yields roughly a 2 percent improvement on the two rel-stack classification tasks."],"supporting_citations":[{"why":"Defines Relational Deep Learning and the relational entity graph construction—tables as node types, primary–foreign key links as edges—that atomic routes operate on, and supplies the temporal neighbor sampling used in experiments.","marker":"Fey et al. (2024)"},{"why":"Provides RelBench, the benchmark with 30 tasks over 7 relational databases that supplies every dataset, baseline pipeline, prediction head, and evaluation metric in the experiments.","marker":"Robinson et al. (2024)"},{"why":"Gives the GraphSAGE aggregation used in the FUSE operation and the heterogeneous GraphSAGE baseline that RelGNN is compared against, plus the temporal neighbor sampling scheme.","marker":"Hamilton et al. (2017)"},{"why":"Represents the standard heterogeneous GNN message passing over edge semantics whose two-hop aggregation through bridge and hub nodes RelGNN identifies as inefficient.","marker":"Schlichtkrull et al. (2018)"},{"why":"A standard heterogeneous graph transformer also cited as the style of model RelGNN improves on; its edge-semantic message passing does not exploit the primary–foreign key router structure.","marker":"Hu et al. (2020)"},{"why":"Supplies the multi-head attention with queries, keys, and values used to instantiate the AGGR operation in the composite message passing.","marker":"Vaswani et al. (2017)"}],"fun_headline_variants":["Composite one-hop routes beat standard GNNs on 27/30 tasks","Atomic routes collapse two-hop joins, lifting RelBench scores","RelGNN: single-hop composite messages slash redundancy, boost accuracy","Replacing two-hop SQL-style joins with atomic routes lifts GNNs","One-step atomic routes outdo multi-hop GNNs on 27 RelBench tasks"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The claim rests on connector tables with two or more foreign keys acting mainly as routers whose own semantic content can be down-weighted; if a junction table's attributes carry strong predictive signal, collapsing its two-hop path into one composite route could discard that signal.","fun_headline_variants_meta":{"raw":{"variants":["Composite one-hop routes beat standard GNNs on 27/30 tasks","Atomic routes collapse two-hop joins, lifting RelBench scores","RelGNN: single-hop composite messages slash redundancy, boost accuracy","Replacing two-hop SQL-style joins with atomic routes lifts GNNs","One-step atomic routes outdo multi-hop GNNs on 27 RelBench tasks"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000681,"raw_usage":{"total_tokens":3113,"prompt_tokens":988,"completion_tokens":2125,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":604,"completion_tokens_details":{"reasoning_tokens":2039}},"tokens_in":604,"tokens_out":2125,"duration_ms":12837,"temperature":1.0,"reasoning_tokens":2039,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-08T14:20:00.492850+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train RelGNN and a standard two-hop heterogeneous GNN on a database whose junction tables are given predictive attributes, such as making transaction amount the sole signal for a churn outcome, and compare test accuracy; if the composite one-hop model does not at least match the two-hop baseline, the router assumption that connector tables carry little semantic content is falsified.","supporting_citations":[{"cited_title":"E., Ranjan, R., Robinson, J., Ying, R., You, J., and Leskovec, J","cited_arxiv_id":null,"evidence_quote":"Defines Relational Deep Learning and the relational entity graph construction—tables as node types, primary–foreign key links as edges—that atomic routes operate on, and supplies the temporal neighbor sampling used in experiments."},{"cited_title":"E., Yuan, Y., Zhang, Z., He, X., and Leskovec, J","cited_arxiv_id":null,"evidence_quote":"Provides RelBench, the benchmark with 30 tasks over 7 relational databases that supplies every dataset, baseline pipeline, prediction head, and evaluation metric in the experiments."},{"cited_title":"Inductive representation learning on large graphs","cited_arxiv_id":null,"evidence_quote":"Gives the GraphSAGE aggregation used in the FUSE operation and the heterogeneous GraphSAGE baseline that RelGNN is compared against, plus the temporal neighbor sampling scheme."},{"cited_title":"N., Bloem, P., van den Berg, R., Titov, I., and Welling, M","cited_arxiv_id":null,"evidence_quote":"Represents the standard heterogeneous GNN message passing over edge semantics whose two-hop aggregation through bridge and hub nodes RelGNN identifies as inefficient."},{"cited_title":"N., Kaiser, L., and Polosukhin, I","cited_arxiv_id":null,"evidence_quote":"Supplies the multi-head attention with queries, keys, and values used to instantiate the AGGR operation in the composite message passing."}],"review_version":1}