Pith. sign in

REVIEW 3 major objections 4 minor 147 references

Exploring and Improving Initialization for Deep Graph Neural Networks: A Signal Propagation Perspective

T0 review · 3 major / 4 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Searching for layer-wise weight variances that keep forward, backward, and graph-embedding signals stable yields an initialization, SPoGInit, that turns depth in GCNs from a liability into an asset.

desk verdict A useful three-metric signal-propagation framework and a practical initialization search, but the central causal claim is confounded by a label-dependent backward term that the paper never discloses. read the letter →

arxiv 2506.16790 v2 pith:7EHDOLYW submitted 2025-06-20 cs.LG

Senmiao Wang , Yupeng Chen , Yushun Zhang , Ruoyu Sun , Tian Ding This is my paper · ORCID
classification cs.LG
keywords graphneuralnetworksinitializationsignalpropagationover-smoothingembeddingvariationdeepGCNNNGPcorrespondencevariancesearch
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that performance degradation in deep graph convolutional networks is best understood as a joint failure of three signal-propagation mechanisms, and that fixing all three at initialization is enough to let depth help rather than hurt. It defines three metrics: forward signal propagation, backward signal propagation, and graph embedding variation, a normalized Dirichlet-energy quantity tied to over-smoothing. It proves that standard initializations, including Conventional, Kaiming, LeCun, and Xavier, cannot keep all three metrics stable simultaneously in vanilla GCNs or residual GCNs. The proposed remedy, SPoGInit, searches per-layer weight variances to stabilize the three metrics, and the paper reports that deep GCNs across several architectures improve with depth or degrade far less than baselines. A sympathetic reader would care because this makes initialization alone, without changing the architecture or message-passing mechanism, a viable and transferable fix for the depth problem.

What carries the argument

The machinery is a three-metric signal-propagation analysis paired with a variance-search algorithm. The forward metric $M^{(L)}_{\mathrm{FSP}}$ is the expected output-input Frobenius norm ratio, the backward metric $M^{(L)}_{\mathrm{BSP}}$ is the expected squared gradient norm of the first-layer weights at initialization, and the graph embedding variation metric $M^{(L)}_{\mathrm{GEV}}$ is the expected normalized Dirichlet energy $\mathbb{E}[\mathrm{Dir}(H^{(L)})/\|H^{(L)}\|_F^2]$. SPoGInit solves $\min_{\{\sigma_{w,l}\}} w_1 V_{\mathrm{FSP}} + w_2 V_{\mathrm{BSP}} - w_3 M^{(L)}_{\mathrm{GEV}}$ using projected gradient descent on layer-wise variance scaling factors, with the NNGP covariance recursion $\Sigma^{(l+1)} = \sigma_w^2 \hat{A} G(\Sigma^{(l)}) \hat{A}$ as the theoretical lens. The graph embedding variation term is what connects initialization directly to over-smoothing, and the search is what lets the same framework adapt to vanilla GCNs, residual GCNs, MixHop, GAT, and bipartite GCNs.

What would settle it

Train a 64-layer GCN with SPoGInit's searched variance profile but at width 16 and width 512 on the same datasets: if the accuracy gains over baselines shrink or vanish at either width, or if the three evaluated metrics no longer track training stability, the infinite-width premise is not faithful. A second decisive check is to run SPoGInit without the label-dependent backward term: if the accuracy gains disappear, the source of the improvement is gradient geometry informed by training labels, not generic signal stability.

Watch

Extended reading notes

Core claim

The central claim is that over-smoothing and gradient pathology in deep GCNs are two faces of one signal-propagation problem, and that a variance search over three metrics can resolve both. Under the infinite-width neural-network Gaussian process approximation, the paper shows that ReLU-activated vanilla GCNs driven by standard initializations either lose forward signal or lose graph embedding variation, and that the variation metric is actually independent of weight variance for ReLU, so no scalar rescaling can fix over-smoothing there. For residual GCNs, it shows that any nonzero initialization variance that satisfies the natural residual condition makes forward signal explode while graph embedding variation vanishes. The paper then proposes SPoGInit, which minimizes a weighted objective combining forward stability, backward stability, and graph embedding variation by adjusting layer-wise initialization variances, and demonstrates that the resulting initialization stabilizes all three metrics and yields accuracy gains as depth grows, including on long-range tasks and combinatorial optimization problems.

Load-bearing premise

The theoretical results live in the infinite-width Gaussian-process limit, and the experiments use width-64 networks, so the entire argument stands on the assumption that this limit faithfully predicts signal propagation and training behavior at finite width.

Editorial extensions

If this is right

  • Widely used initializations provably fail to control all three signal-propagation metrics at once, so the depth problem in GCNs is not only an architectural or normalization issue but also an initialization issue.
  • For ReLU-activated vanilla GCNs, graph embedding variation is independent of weight variance, meaning over-smoothing cannot be cured by scalar rescaling; a per-layer search or a different activation is required.
  • SPoGInit stabilizes forward, backward, and graph embedding variation across GCN, ResGCN, gatResGCN, MixHop, and GAT, with 64-layer ResGCN and gatResGCN models gaining accuracy over their 4-layer counterparts.
  • On long-range tasks such as missing-feature node classification and MILP-based combinatorial optimization, SPoGInit shifts the optimal depth toward larger networks and improves the accuracy of 16-layer bipartite GCNs.
  • SPoGInit is transferable across starting initializations: starting from Conventional, Xavier, VirgoFor, or VirgoBack, the variance search improves deep-network accuracy relative to the starting point.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A label-agnostic variant of SPoGInit that drops the backward signal term, or computes it from random labels, would isolate whether the gains come from label-dependent gradient geometry or from generic signal stability; the paper's BSP term uses training labels, which makes the searched initialization label-dependent while all baselines are label-agnostic.
  • If the mechanism is right, the same three-metric objective could serve as a training-free diagnostic for whether a proposed GCN architecture will tolerate depth, predicting the depth at which degradation begins before any training is run.
  • The variance-search idea may transfer to other message-passing layers and graph transformers, since the metrics only require forward and backward passes plus a normalized Dirichlet energy computation over the final embeddings.
  • Because SPoGInit uses training labels at initialization, it is effectively a hybrid between an initialization and a warm start; future comparisons should state this asymmetry explicitly when interpreting accuracy differences.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. This paper argues that depth-related performance degradation in graph convolutional networks can be understood as a signal-propagation failure involving three metrics: forward signal propagation (FSP), backward signal propagation (BSP), and graph embedding variation (GEV), the last being a graph-specific normalized Dirichlet energy. Under an NNGP infinite-width approximation, it proves that for ReLU vanilla GCNs and linear ResGCNs standard variance choices drive FSP and/or GEV to degenerate limits, and it proposes SPoGInit, an initialization search that minimizes a weighted combination of the three metrics by rescaling per-layer weight variances. Experiments on Cora, PubMed, OGBN-Arxiv, Arxiv-year, additional homophily/heterophily datasets, missing-feature settings, and MILP show that SPoGInit stabilizes the SP metrics and reduces accuracy degradation in deep GCN, ResGCN, MixHop, and GAT models compared with Xavier, Conventional, VirgoFor, VirgoBack, and G-Init initializations.

Significance. If the causal claim were established, the paper would make a useful contribution: it offers a compact diagnostic (FSP/BSP/GEV), gives careful NNGP derivations for the forward and GEV recurrences, and shows that depth degradation can be attacked at initialization, a route that is orthogonal to normalization, dropout, and architectural rewiring. The empirical scope is broad, the SP plots in Figures 3-5 directly illustrate the proposed diagnostic, and the integration experiments with DGN, CO-GNN, and G-Init strengthen the practical claims. The main weakness is that the paper does not yet isolate the mechanism: SPoGInit's backward term is label-dependent while all baselines are label-agnostic, and the theoretical results cover only a subset of the stated claims. These gaps are fixable with targeted controls and rewording.

major comments (3)
  1. [Section 4, Algorithm 1 (Appendix E), Eq. (49)] The BSP term in SPoGInit's objective is computed from the gradient of the training loss on the true training labels (Algorithm 1, line 5; Eq. (49), with g^(l)=∂ℓ/∂W^(l)), so the searched initialization is label-dependent. Every baseline initialization (Conventional, Xavier, VirgoFor, VirgoBack, G-Init) is label-agnostic, and Appendix G.2's ablation does not control for this asymmetry because removing the BSP term removes both gradient stabilization and label information. The experiments therefore do not isolate whether SPoGInit's gains come from stabilizing backward signal propagation or from exploiting label information at initialization; this is a comparison-integrity and interpretability problem rather than leakage, since labels are legitimately available at initialization. A decisive control is to run the same search with random labels (or a label-agnostic BSP proxy) and compare accuracy; if the gains persist, the causal interpretation is supported, and if not, the central claim needs substantial qualification.
  2. [Section 3 and Abstract] The abstract and Section 3 state that standard initializations are theoretically proven to fail to control all three SP metrics simultaneously. However, Theorem 3.1 (vanilla GCN) and Theorem 3.3 (ResGCN) establish results only for FSP and GEV; no theorem anywhere characterizes BSP, and the sentence following Theorem 3.3 asserts a shrinking BSP that is not a stated or proved consequence of the theorem. In addition, Theorem 3.1 is proved for ReLU while the main vanilla GCN experiments use tanh (Section 5.1), and Theorem 3.3 is proved for linear ResGCN while the ResGCN experiments use ReLU. The theoretical support for the headline claim is therefore narrower than presented; the authors should either supply the missing BSP and nonlinearity results or reword the abstract and Section 3 to match what is proved.
  3. [Section 6, Infinite-width-limit regime, and Section 5.1] All experiments use width 64, while Propositions C.1 and D.1 are sequential infinite-width limits. The only direct validation is the Kolmogorov-Smirnov test in the Infinite-width-limit regime paragraph of Section 6, applied to three node embeddings from the final layer of a 4-layer network; this does not establish that the NNGP recursions are faithful at L=64 and width 64 for all three metrics. Since the negative theorems are stated in the NNGP limit, the transfer of those conclusions to the finite-width models in Figures 3-5 and Tables 2-5 needs additional support, for example a width-ablation showing that the SP metric curves or the searched variances stabilize as width grows. I treat this as a correctness-risk concern rather than an observed contradiction.
minor comments (4)
  1. [Appendix E.2] The text introducing the w2 analysis says Table 7 reports the values, but the displayed table is numbered Table 6; fix this cross-reference.
  2. [Table 2] On OGBN-Arxiv, SPoGInit's vanilla GCN degradation from 4 to 64 layers is 21.4 points, substantially worse than VirgoFor (8.5) and VirgoBack (8.8); the text in Section 5.2 says SPoGInit significantly reduces performance degradation and should be qualified to most settings, with a discussion of this exception.
  3. [Section 5.1] The paper reports that all results are averaged over at least three runs, but Tables 4, 5, and 9-11 report single numbers without standard deviations; for differences of one to two accuracy points this is insufficient evidence, and the authors should report error bars or significance tests.
  4. [Appendix G.1] Please state whether the code for SPoGInit will be released; the search algorithm is intricate, and reproducibility would benefit from code or detailed pseudo-code in addition to the current description.

Circularity Check

1 steps flagged · score 3.0 of 10

SPoGInit's signal-propagation gains largely restate the optimizer's own objective; the accuracy claims remain independent, so circularity is partial and mild.

  1. fitted input called prediction [Section 4 Eq. (2); Section 5.2 Q1 (Figures 3-5); Appendix E Eq. (49) and Algorithm 1]
    "SPoGInit solves the following optimization problem: minimize {σw,l}L l=1 w1VFSP +w2VBSP−w3M(L) GEV, (2) ... In contrast, SPoGInit is highly effective in stabilizing all three SP metrics."

    The layer-wise variances returned by SPoGInit are obtained by gradient descent on an objective whose three terms are exactly the FSP, BSP, and GEV metrics (Eq. 2; Appendix E uses point estimates of these same quantities in Eq. 49). Therefore the Section 5.2 answer to Q1, that SPoGInit stabilizes all three SP metrics, is essentially a report on the success of the minimization rather than an independent validation that the proposed SP framework improves signal propagation. The independent content of the paper lies in the test-accuracy comparisons (Tables 2-5, 9-11, 17-18) and the ablation in Table 20, which test an outcome not contained in the objective; those results keep the central performance claim from being purely definitional.

full rationale

The negative theorems about Conventional, Kaiming, LeCun, and Xavier initializations are derived from the stated NNGP correspondence (Proposition C.1) and standard spectral properties of the normalized adjacency matrix; they are not fitted to the experimental outcomes, and the numerical SP plots serve as validation of the bounds. The main circularity is limited to the Q1 demonstration: SPoGInit's variances are chosen by minimizing an objective whose three terms are exactly the FSP, BSP, and GEV metrics (Eq. 2; Appendix E Eq. 49), so Figures 3-5 showing that SPoGInit 'stabilizes all three SP metrics' largely report the result of the minimization rather than an independent test of the SP framework. This is a fitted-input-called-prediction step if the SP stabilization is treated as the paper's evidence for the framework. However, the accuracy improvements on Cora, PubMed, OGBN-Arxiv, Arxiv-year, the missing-feature and MILP tasks, and the Table 20 ablation test an independent outcome (test accuracy), so the central claim that SPoGInit mitigates depth degradation does not reduce to the objective by definition. One non-circular but important confound is that the BSP term uses gradients on the true training labels y(t) (Algorithm 1 line 5, Appendix E), making the searched initialization label-dependent while all baselines are label-agnostic; this threatens the causal interpretation of the accuracy gains but is not a definitional reduction. Self-citations (e.g., Han et al. 2022 for the MILP setting) are not load-bearing, and no uniqueness theorem or ansatz is imported from the authors' prior work. Overall the circularity is partial and mild: score 3.0.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

The central claim inherits three classes of assumptions: the infinite-width NNGP correspondence that all theorems rely on, the identification of over-smoothing with initialization-time normalized Dirichlet energy, and an implicit transfer claim that stabilizing the three metrics at initialization causes the observed accuracy gains. SPoGInit adds hand-chosen objective weights and a label-dependent search for layer-wise variances, which are the true tunable quantities of the method.

free parameters (3)
  • layer-wise weight variances sigma_w,l = Optimized per dataset and architecture by SPoGInit (Figure 6 shows an example 64-layer pattern)
    These are the quantities SPoGInit searches for; they are data-dependent and are the method's core output, so the central claim depends on their fitted values.
  • SPoGInit objective weights w1, w2, w3 = (1, 10, 1) for vanilla GCN; (1, 1, 1) for other architectures
    Hand-selected; Appendix E.2 tunes w2 on Cora and justifies the default value of 10, but there is no systematic sensitivity study.
  • SPoGInit search hyperparameters (learning rate, early-stopping delta, iterations) = Learning rates 0.02 to 0.2, delta 10 or 20, up to 40 to 100 iterations
    Chosen per dataset and model family in Appendix G.1; no sensitivity analysis is reported for these choices.
assumptions (3)
  • domain assumption NNGP correspondence (Proposition C.1): as hidden widths go sequentially to infinity, pre-activation channels are i.i.d. Gaussian with covariance recursion Sigma^(l+1) = sigma_w^2 * A_hat * G(Sigma^(l)) * A_hat.
    All theoretical bounds (Theorems 3.1, 3.2, 3.3, C.7, C.13, C.14) hold only under this infinite-width approximation. The paper acknowledges the Gaussian property is not guaranteed at finite width and supports it with a KS test on only three embeddings.
  • domain assumption Initialization-time normalized Dirichlet energy (GEV) is a valid proxy for the over-smoothing that degrades accuracy after training.
    The link between the GEV metric and training-time over-smoothing is asserted and validated only through the success of the method; it is not proven independently.
  • domain assumption The BSP metric, defined as the expected norm of the first-layer loss gradient at initialization computed on training labels, predicts gradient stability during training.
    No theorem covers BSP in GCNs; Figures 1, 3, 4, and 5 show it only empirically. The metric intrinsically requires training labels, as seen in Algorithm 1.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Exploring and Improving Initialization for Deep Graph Neural Networks: A Signal Propagation Perspective." pith.science (2026). https://pith.science/paper/7EHDOLYW

@misc{pith2026250616790,
  author       = {Pith},
  title        = {Pith review of: Exploring and Improving Initialization for Deep Graph Neural Networks: A Signal Propagation Perspective},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/7EHDOLYW}},
  note         = {Machine review of arXiv:2506.16790}
}
read the original abstract

Graph Neural Networks (GNNs) often suffer from performance degradation as the network depth increases. This paper addresses this issue by introducing initialization methods that enhance signal propagation (SP) within GNNs. We propose three key metrics for effective SP in GNNs: forward propagation, backward propagation, and graph embedding variation (GEV). While the first two metrics derive from classical SP theory, the third is specifically designed for GNNs. We theoretically demonstrate that a broad range of commonly used initialization methods for GNNs, which exhibit performance degradation with increasing depth, fail to control these three metrics simultaneously. To deal with this limitation, a direct exploitation of the SP analysis--searching for weight initialization variances that optimize the three metrics--is shown to significantly enhance the SP in deep GCNs. This approach is called Signal Propagation on Graph-guided Initialization (SPoGInit). Our experiments demonstrate that SPoGInit outperforms commonly used initialization methods on various tasks and architectures. Notably, SPoGInit enables performance improvements as GNNs deepen, which represents a significant advancement in addressing depth-related challenges and highlights the validity and effectiveness of the SP analysis framework.

Figures

Figures reproduced from arXiv: 2506.16790 by the authors.

Figure 1
Figure 1. Plots of (a,d) forward metrics, (b,e) backward metrics, and (c,f) graph embedding variation metrics [PITH_FULL_IMAGE:figures/full_fig_p006_1.png] view at source ↗
Figure 2
Figure 2. (a) The forward metrics and (b) the graph embedding variation metrics of ReLU-activated deep [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 3
Figure 3. Plots of (a) forward metrics, (b) backward metrics, and (c) graph embedding variation metrics [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗
Figures from the paper (8 more)
Figure 4
Figure 4. Figure 4: Plots of (a) forward metrics, (b) backward metrics, and (c) graph embedding variation metrics of [PITH_FULL_IMAGE:figures/full_fig_p009_4.png]
Figure 5
Figure 5. Figure 5: Plots of (a) forward metrics, (b) backward metrics, and (c) graph embedding variation metrics of [PITH_FULL_IMAGE:figures/full_fig_p010_5.png]
Figure 6
Figure 6. Figure 6: Layer-wise σw,i of a 64-layer tanh-activated GCN using different initialization methods on the Cora datasets. In SPoGInit, we set the learning rate as 0.02, the early stop step δ as 10, and the number of total step is set as 40. 48 [PITH_FULL_IMAGE:figures/full_fig_p0…
Figure 7
Figure 7. Figure 7: The heatmap of the activation and gradient matrix of a 256-layer tanh-activated GCN with [PITH_FULL_IMAGE:figures/full_fig_p054_7.png]
Figure 8
Figure 8. Figure 8: The heatmap of the activation and gradient matrix of a 256-layer tanh-activated GCN with Xavier [PITH_FULL_IMAGE:figures/full_fig_p055_8.png]
Figure 9
Figure 9. Figure 9: The heatmap of the activation and gradient matrix of a 256-layer tanh-activated GCN with VirgoFor [PITH_FULL_IMAGE:figures/full_fig_p055_9.png]
Figure 10
Figure 10. Figure 10: The heatmap of the activation and gradient matrix of a 256-layer tanh-activated GCN with [PITH_FULL_IMAGE:figures/full_fig_p056_10.png]
Figure 11
Figure 11. Figure 11: The heatmap of the activation and gradient matrix of a 256-layer tanh-activated GCN with [PITH_FULL_IMAGE:figures/full_fig_p056_11.png]

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

147 extracted references · 54 canonical work pages

  1. [1]

    Mixhop : Higher-order graph convolutional architectures via sparsified neighborhood mixing

    Sami Abu-El-Haija, Bryan Perozzi, Amol Kapoor, Nazanin Alipourfard, Kristina Lerman, Hrayr Harutyunyan, Greg Ver Steeg, and Aram Galstyan. Mixhop : Higher-order graph convolutional architectures via sparsified neighborhood mixing. In International Conference on Machine Learning, pp.\ 21--29. PMLR, 2019

    2019

  2. [2]

    On the bottleneck of graph neural networks and its practical implications

    Uri Alon and Eran Yahav. On the bottleneck of graph neural networks and its practical implications. In International Conference on Learning Representations, 2021

    2021

  3. [3]

    A machine learning-based approximation of strong branching

    Alejandro Marcos Alvarez, Quentin Louveaux, and Louis Wehenkel. A machine learning-based approximation of strong branching. INFORMS Journal on Computing, 29 0 (1): 0 185--195, 2017

    2017

  4. [4]

    Label propagation across graphs: Node classification using graph neural tangent kernels

    Artun Bayer, Arindam Chowdhury, and Santiago Segarra. Label propagation across graphs: Node classification using graph neural tangent kernels. In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp.\ 5483--5487. IEEE, 2022

    2022

  5. [5]

    Machine learning for combinatorial optimization: a methodological tour d’horizon

    Yoshua Bengio, Andrea Lodi, and Antoine Prouvost. Machine learning for combinatorial optimization: a methodological tour d’horizon. European Journal of Operational Research, 290 0 (2): 0 405--421, 2021

    2021

  6. [6]

    Decision diagrams for optimization, volume 1

    David Bergman, Andre A Cire, Willem-Jan Van Hoeve, and John Hooker. Decision diagrams for optimization, volume 1. Springer, 2016

    2016

  7. [7]

    The maximum clique problem

    Immanuel M Bomze, Marco Budinich, Panos M Pardalos, and Marcello Pelillo. The maximum clique problem. Handbook of Combinatorial Optimization: Supplement Volume A, pp.\ 1--74, 1999

    1999

  8. [8]

    Can graph neural networks go deeper without over-smoothing? yes, with a randomized path exploration! In IEEE Transactions on Emerging Topics in Computational Intelligence

    Kushal Bose and Swagatam Das. Can graph neural networks go deeper without over-smoothing? yes, with a randomized path exploration! In IEEE Transactions on Emerging Topics in Computational Intelligence. IEEE, 2023

    2023

Show all 147 references
  1. [9]

    Reconnaissance de la parole par reseaux connexionnistes

    L \'e on Bottou. Reconnaissance de la parole par reseaux connexionnistes. In Proceedings of neuro Nimes, volume 88, pp.\ 197--218, 1988

    1988

  2. [10]

    A survey on optimization metaheuristics

    Ilhem Boussa \" d, Julien Lepagnot, and Patrick Siarry. A survey on optimization metaheuristics. Information sciences, 237: 0 82--117, 2013

    2013

  3. [11]

    A note on over-smoothing for graph neural networks

    Chen Cai and Yusu Wang. A note on over-smoothing for graph neural networks. In International Conference on Machine Learning. PMLR, 2020

    2020

  4. [12]

    Measuring and relieving the over-smoothing problem for graph neural networks from the topological view

    Deli Chen, Yankai Lin, Wei Li, Peng Li, Jie Zhou, and Xu Sun. Measuring and relieving the over-smoothing problem for graph neural networks from the topological view. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pp.\ 3438--3445, 2020 a

    2020

  5. [13]

    Dirichlet energy enhancement of graph neural networks by framelet augmentation

    Jialin Chen, Yuelin Wang, Cristian Bodnar, Rex Ying, Pietro Li \`o , and Yu Guang Wang. Dirichlet energy enhancement of graph neural networks by framelet augmentation. 2022 a

    2022

  6. [14]

    Universal deep gnns: Rethinking residual connection in gnns from a path decomposition perspective for preventing the over-smoothing

    Jie Chen, Weiqi Liu, Zhizhong Huang, Junbin Gao, Junping Zhang, and Jian Pu. Universal deep gnns: Rethinking residual connection in gnns from a path decomposition perspective for preventing the over-smoothing. arXiv preprint arXiv:2205.15127, 2022 b

    2022 arXiv

  7. [15]

    Simple and deep graph convolutional networks

    Ming Chen, Zhewei Wei, Zengfeng Huang, Bolin Ding, and Yaliang Li. Simple and deep graph convolutional networks. In International Conference on Machine Learning, pp.\ 1725--1735. PMLR, 2020 b

    2020

  8. [16]

    Dynamical isometry and a mean field theory of rnns: Gating enables signal propagation in recurrent neural networks

    Minmin Chen, Jeffrey Pennington, and Samuel Schoenholz. Dynamical isometry and a mean field theory of rnns: Gating enables signal propagation in recurrent neural networks. In International Conference on Machine Learning, pp.\ 873--882. PMLR, 2018

    2018

  9. [17]

    Bag of tricks for training deeper graph neural networks: A comprehensive benchmark study

    Tianlong Chen, Kaixiong Zhou, Keyu Duan, Wenqing Zheng, Peihao Wang, Xia Hu, and Zhangyang Wang. Bag of tricks for training deeper graph neural networks: A comprehensive benchmark study. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022 c

    2022

  10. [18]

    On representing linear programs by graph neural networks

    Ziang Chen, Jialin Liu, Xinshang Wang, and Wotao Yin. On representing linear programs by graph neural networks. In The Eleventh International Conference on Learning Representations, 2023

    2023

  11. [19]

    Expressive power of graph neural networks for (mixed-integer) quadratic programs

    Ziang Chen, Xiaohan Chen, Jialin Liu, Xinshang Wang, and Wotao Yin. Expressive power of graph neural networks for (mixed-integer) quadratic programs. arXiv preprint arXiv:2406.05938, 2024

    2024

  12. [20]

    Adaptive universal generalized pagerank graph neural network

    Eli Chien, Jianhao Peng, Pan Li, and Olgica Milenkovic. Adaptive universal generalized pagerank graph neural network. In International Conference on Learning Representations, 2021

    2021

  13. [21]

    Better not to propagate: Understanding edge uncertainty and over-smoothing in signed graph neural networks

    Yoonhyuk Choi, Jiho Choi, Taewook Ko, and Chong-Kwon Kim. Better not to propagate: Understanding edge uncertainty and over-smoothing in signed graph neural networks. arXiv preprint arXiv:2408.04895, 2024

    2024 arXiv

  14. [22]

    Approximation algorithms for bin-packing—an updated survey

    Edward G Coffman Jr, Michael R Garey, and David S Johnson. Approximation algorithms for bin-packing—an updated survey. In Algorithm design for computer system design, pp.\ 49--106. Springer, 1984

    1984

  15. [23]

    On provable benefits of depth in training graph convolutional networks

    Weilin Cong, Morteza Ramezani, and Mehrdad Mahdavi. On provable benefits of depth in training graph convolutional networks. Advances in Neural Information Processing Systems, 34: 0 9936--9949, 2021

    2021

  16. [24]

    Metainit: Initializing learning by learning to initialize

    Yann N Dauphin and Samuel S Schoenholz. Metainit: Initializing learning by learning to initialize. Advances in Neural Information Processing Systems, 32, 2019

    2019

  17. [25]

    Graph neural networks as gradient flows: Understanding graph convolutions via energy

    Francesco Di Giovanni, James Rowbottom, Benjamin P Chamberlain, Thomas Markovich, and Michael M Bronstein. Graph neural networks as gradient flows: Understanding graph convolutions via energy. arXiv preprint arXiv:2206.10991, 2022

    2022 arXiv

  18. [26]

    Adagnn: Graph neural networks with adaptive frequency response filter

    Yushun Dong, Kaize Ding, Brian Jalaian, Shuiwang Ji, and Jundong Li. Adagnn: Graph neural networks with adaptive frequency response filter. In Proceedings of the 30th ACM international conference on information & knowledge management, pp.\ 392--401, 2021

    2021

  19. [27]

    Graph neural tangent kernel: Fusing graph neural networks with graph kernels

    Simon S Du, Kangcheng Hou, Russ R Salakhutdinov, Barnabas Poczos, Ruosong Wang, and Keyulu Xu. Graph neural tangent kernel: Fusing graph neural networks with graph kernels. Advances in Neural Information Processing Systems, 32, 2019

    2019

  20. [28]

    Learning from the dark: boosting graph convolutional neural networks with diverse negative samples

    Wei Duan, Junyu Xuan, Maoying Qiao, and Jie Lu. Learning from the dark: boosting graph convolutional neural networks with diverse negative samples. In Proceedings of the AAAI conference on artificial intelligence, volume 36, pp.\ 6550--6558, 2022

    2022

  21. [29]

    Graph convolutional neural networks with diverse negative samples via decomposed determinant point processes

    Wei Duan, Junyu Xuan, Maoying Qiao, and Jie Lu. Graph convolutional neural networks with diverse negative samples via decomposed determinant point processes. IEEE Transactions on Neural Networks and Learning Systems, 2023

    2023

  22. [30]

    Layer-diverse negative sampling for graph neural networks

    Wei Duan, Jie Lu, Yu Guang Wang, and Junyu Xuan. Layer-diverse negative sampling for graph neural networks. Transactions on Machine Learning Research, 2024

    2024

  23. [31]

    The power of depth for feedforward neural networks

    Ronen Eldan and Ohad Shamir. The power of depth for feedforward neural networks. In Conference on learning theory, pp.\ 907--940. PMLR, 2016

    2016

  24. [32]

    DropMessage : Unifying random dropping for graph neural networks

    Taoran Fang, Zhiqing Xiao, Chunping Wang, Jiarong Xu, Xuan Yang, and Yang Yang. DropMessage : Unifying random dropping for graph neural networks. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pp.\ 4267--4275, 2023

    2023

  25. [33]

    Grato: Graph neural network framework tackling over-smoothing with neural architecture search

    Xinshun Feng, Herun Wan, Shangbin Feng, Hongrui Wang, Qinghua Zheng, Jun Zhou, and Minnan Luo. Grato: Graph neural network framework tackling over-smoothing with neural architecture search. In Proceedings of the 31st ACM International Conference on Information & Knowledge Mana...

    2022

  26. [34]

    Matthias Fey and Jan E. Lenssen. Fast graph representation learning with PyTorch Geometric . In ICLR Workshop on Representation Learning on Graphs and Manifolds, 2019

    2019

  27. [35]

    Cooperative graph neural networks

    Ben Finkelshtein, Xingyue Huang, Michael Bronstein, and Ismail Ilkan Ceylan. Cooperative graph neural networks. arXiv preprint arXiv:2310.01267, 2023

    2023 arXiv

  28. [36]

    Exact combinatorial optimization with graph convolutional neural networks

    Maxime Gasse, Didier Ch \'e telat, Nicola Ferroni, Laurent Charlin, and Andrea Lodi. Exact combinatorial optimization with graph convolutional neural networks. Advances in Neural Information Processing Systems, 32, 2019

    2019

  29. [37]

    Predict then propagate: Graph neural networks meet personalized pagerank

    Johannes Gasteiger, Aleksandar Bojchevski, and Stephan G \"u nnemann. Predict then propagate: Graph neural networks meet personalized pagerank. In International Conference on Learning Representations, 2019

    2019

  30. [38]

    The travelling salesman problem and related problems

    Bezalel Gavish and Stephen C Graves. The travelling salesman problem and related problems. 1978

    1978

  31. [39]

    Graph convolutional networks from the perspective of sheaves and the neural tangent kernel

    Thomas Gebhart. Graph convolutional networks from the perspective of sheaves and the neural tangent kernel. In Topological, Algebraic and Geometric Learning Workshops 2022, pp.\ 124--132. PMLR, 2022

    2022

  32. [40]

    Dynamical isometry and a mean field theory of lstms and grus

    Dar Gilboa, Bo Chang, Minmin Chen, Greg Yang, Samuel S Schoenholz, Ed H Chi, and Jeffrey Pennington. Dynamical isometry and a mean field theory of lstms and grus. arXiv preprint arXiv:1901.08987, 2019

    1901 arXiv

  33. [41]

    Understanding the difficulty of training deep feedforward neural networks

    Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedforward neural networks. In International Conference on Artificial Intelligence and Statistics, pp.\ 249--256. JMLR Workshop and Conference Proceedings, 2010

    2010

  34. [42]

    Orthogonal graph neural networks

    Kai Guo, Kaixiong Zhou, Xia Hu, Yu Li, Yi Chang, and Xin Wang. Orthogonal graph neural networks. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pp.\ 3996--4004, 2022

    2022

  35. [43]

    Contranorm: A contrastive learning perspective on oversmoothing and beyond

    Xiaojun Guo, Yifei Wang, Tianqi Du, and Yisen Wang. Contranorm: A contrastive learning perspective on oversmoothing and beyond. In International Conference on Learning Representations, 2023

    2023

  36. [44]

    Structure-aware dropedge toward deep graph convolutional networks

    Jiaqi Han, Wenbing Huang, Yu Rong, Tingyang Xu, Fuchun Sun, and Junzhou Huang. Structure-aware dropedge toward deep graph convolutional networks. IEEE Transactions on Neural Networks and Learning Systems, 2023 a

    2023

  37. [45]

    A gnn-guided predict-and-search framework for mixed-integer linear programming

    Qingyu Han, Linxin Yang, Qian Chen, Xiang Zhou, Dong Zhang, Akang Wang, Ruoyu Sun, and Xiaodong Luo. A gnn-guided predict-and-search framework for mixed-integer linear programming. In The Eleventh International Conference on Learning Representations, 2022

    2022

  38. [46]

    Mlpinit: Embarrassingly simple gnn training acceleration with mlp initialization

    Xiaotian Han, Tong Zhao, Yozen Liu, Xia Hu, and Neil Shah. Mlpinit: Embarrassingly simple gnn training acceleration with mlp initialization. 2023 b

    2023

  39. [47]

    Which neural net architectures give rise to exploding and vanishing gradients? Advances in Neural Information Processing Systems, 31, 2018

    Boris Hanin. Which neural net architectures give rise to exploding and vanishing gradients? Advances in Neural Information Processing Systems, 31, 2018

    2018

  40. [48]

    Inequalities

    Godfrey Harold Hardy, John Edensor Littlewood, George P \'o lya, Gy \"o rgy P \'o lya, et al. Inequalities. Cambridge university press, 1952

    1952

  41. [49]

    On the impact of the activation function on deep neural networks training

    Soufiane Hayou, Arnaud Doucet, and Judith Rousseau. On the impact of the activation function on deep neural networks training. In International Conference on Machine Learning, pp.\ 2672--2680. PMLR, 2019

    2019

  42. [50]

    The curse of depth in kernel regime

    Soufiane Hayou, Arnaud Doucet, and Judith Rousseau. The curse of depth in kernel regime. In I (Still) Can't Believe It's Not Better! Workshop at NeurIPS 2021, pp.\ 41--47. PMLR, 2022

    2021

  43. [51]

    Delving deep into rectifiers: Surpassing human-level performance on imagenet classification

    Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In Proceedings of the IEEE International Conference on Computer Vision, pp.\ 1026--1034, 2015

    2015

  44. [52]

    Deep residual learning for image recognition

    Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp.\ 770--778, 2016

    2016

  45. [53]

    Open graph benchmark: Datasets for machine learning on graphs

    Weihua Hu, Matthias Fey, Marinka Zitnik, Yuxiao Dong, Hongyu Ren, Bowen Liu, Michele Catasta, and Jure Leskovec. Open graph benchmark: Datasets for machine learning on graphs. Advances in Neural Information Processing Systems, 33: 0 22118--22133, 2020

    2020

  46. [54]

    Towards deepening graph neural networks: A gntk-based optimization perspective

    Wei Huang, Yayong Li, Weitao Du, Jie Yin, Richard Yi Da Xu, Ling Chen, and Miao Zhang. Towards deepening graph neural networks: A gntk-based optimization perspective. International Conference on Learning Representations, 2022

    2022

  47. [55]

    Tackling over-smoothing for general graph convolutional networks

    Wenbing Huang, Yu Rong, Tingyang Xu, Fuchun Sun, and Junzhou Huang. Tackling over-smoothing for general graph convolutional networks. arXiv preprint arXiv:2008.09864, 2020

    2008 arXiv

  48. [56]

    Batch normalization: Accelerating deep network training by reducing internal covariate shift

    Sergey Ioffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. In International Conference on Machine Learning, pp.\ 448--456. PMLR, 2015

    2015

  49. [57]

    Old can be gold: Better gradient flow can make vanilla-gcns great again

    Ajay Jaiswal, Peihao Wang, Tianlong Chen, Justin Rousseau, Ying Ding, and Zhangyang Wang. Old can be gold: Better gradient flow can make vanilla-gcns great again. Advances in Neural Information Processing Systems, 35: 0 7561--7574, 2022

    2022

  50. [58]

    Fast graph neural tangent kernel via kronecker sketching

    Shunhua Jiang, Yunze Man, Zhao Song, Zheng Yu, and Danyang Zhuo. Fast graph neural tangent kernel via kronecker sketching. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pp.\ 7033--7041, 2022

    2022

  51. [59]

    Towards feature overcorrelation in deeper graph neural networks

    Wei Jin, Xiaorui Liu, Yao Ma, Charu Aggarwal, and Jiliang Tang. Towards feature overcorrelation in deeper graph neural networks. 2022

    2022

  52. [60]

    Reducing oversmoothing in graph neural networks by changing the activation function

    Dimitrios Kelesis, Dimitrios Vogiatzis, Georgios Katsimpras, Dimitris Fotakis, and Georgios Paliouras. Reducing oversmoothing in graph neural networks by changing the activation function. In ECAI 2023 - 26th European Conference on Artificial Intelligence, September 30 - Octobe...

    2023

  53. [61]

    Reducing oversmoothing through informed weight initialization in graph neural networks

    Dimitrios Kelesis, Dimitris Fotakis, and Georgios Paliouras. Reducing oversmoothing through informed weight initialization in graph neural networks. arXiv preprint arXiv:2410.23830, 2024

    2024 arXiv

  54. [62]

    Not too little, not too much: a theoretical analysis of graph (over)smoothing

    Nicolas Keriven. Not too little, not too much: a theoretical analysis of graph (over)smoothing. In Advances in Neural Information Processing Systems, 2022

    2022

  55. [63]

    Learning to branch in mixed integer programming

    Elias Khalil, Pierre Le Bodic, Le Song, George Nemhauser, and Bistra Dilkina. Learning to branch in mixed integer programming. In Proceedings of the AAAI conference on artificial intelligence, volume 30, 2016

    2016

  56. [64]

    Learning to run heuristics in tree search

    Elias B Khalil, Bistra Dilkina, George L Nemhauser, Shabbir Ahmed, and Yufen Shao. Learning to run heuristics in tree search. In Ijcai, pp.\ 659--666, 2017

    2017

  57. [65]

    Semi-supervised classification with graph convolutional networks

    Thomas N Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. International Conference on Learning Representations, 2017

    2017

  58. [66]

    Goat: A global transformer on large-scale graphs

    Kezhi Kong, Jiuhai Chen, John Kirchenbauer, Renkun Ni, C Bayan Bruss, and Tom Goldstein. Goat: A global transformer on large-scale graphs. In International Conference on Machine Learning, pp.\ 17375--17390. PMLR, 2023

    2023

  59. [67]

    Graph neural tangent kernel: Convergence on large graphs

    Sanjukta Krishnagopal and Luana Ruiz. Graph neural tangent kernel: Convergence on large graphs. arXiv preprint arXiv:2301.10808, 2023

    2023 arXiv

  60. [68]

    Imagenet classification with deep convolutional neural networks

    Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. Advances in Neural Information Processing Systems, 25 0 (1106-1114): 0 1, 2012

    2012

  61. [69]

    Efficient backprop

    Yann LeCun, L \'e on Bottou, Genevieve B Orr, and Klaus-Robert M \"u ller. Efficient backprop. In Neural networks: Tricks of the trade, pp.\ 9--50. Springer, 2002

    2002

  62. [70]

    Deep neural networks as gaussian processes

    Jaehoon Lee, Yasaman Bahri, Roman Novak, Samuel S Schoenholz, Jeffrey Pennington, and Jascha Sohl-Dickstein. Deep neural networks as gaussian processes. In International Conference on Learning Representations, 2018

    2018

  63. [71]

    Pdhg-unrolled learning-to-optimize method for large-scale linear programming

    Bingheng Li, Linxin Yang, Yupeng Chen, Senmiao Wang, Qian Chen, Haitao Mao, Yao Ma, Akang Wang, Tian Ding, Jiliang Tang, et al. Pdhg-unrolled learning-to-optimize method for large-scale linear programming. arXiv preprint arXiv:2406.01908, 2024 a

    2024 arXiv

  64. [72]

    Deepgcns: Can gcns go as deep as cnns? In Proceedings of the IEEE/CVF International Conference on Computer Vision, 2019

    Guohao Li, Matthias M \"u ller, Ali Thabet, and Bernard Ghanem. Deepgcns: Can gcns go as deep as cnns? In Proceedings of the IEEE/CVF International Conference on Computer Vision, 2019

    2019

  65. [73]

    Deepergcn: All you need to train deeper gcns, 2020

    Guohao Li, Chenxin Xiong, Ali Thabet, and Bernard Ghanem. Deepergcn: All you need to train deeper gcns, 2020

    2020

  66. [74]

    Training graph neural networks with 1000 layers

    Guohao Li, Matthias M \"u ller, Bernard Ghanem, and Vladlen Koltun. Training graph neural networks with 1000 layers. In International Conference on Machine Learning, 2021

    2021

  67. [75]

    On the initialization of graph neural networks

    Jiahang Li, Yakun Song, Xiang Song, and David Wipf. On the initialization of graph neural networks. In International Conference on Machine Learning, pp.\ 19911--19931. PMLR, 2023

    2023

  68. [76]

    On random deep weight-tied autoencoders: Exact asymptotic analysis, phase transitions, and implications to training

    Ping Li and Phan-Minh Nguyen. On random deep weight-tied autoencoders: Exact asymptotic analysis, phase transitions, and implications to training. In International Conference on Learning Representations, 2019

    2019

  69. [77]

    On the power of small-size graph neural networks for linear programming

    Qian Li, Tian Ding, Linxin Yang, Minghui Ouyang, Qingjiang Shi, and Ruoyu Sun. On the power of small-size graph neural networks for linear programming. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024 b

    2024

  70. [78]

    Deeper insights into graph convolutional networks for semi-supervised learning

    Qimai Li, Zhichao Han, and Xiao-Ming Wu. Deeper insights into graph convolutional networks for semi-supervised learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018

    2018

  71. [79]

    Why deep neural networks for function approximation? In 5th International Conference on Learning Representations, ICLR 2017, 2017

    Shiyu Liang and R Srikant. Why deep neural networks for function approximation? In 5th International Conference on Learning Representations, ICLR 2017, 2017

    2017

  72. [80]

    Large scale learning on non-homophilous graphs: New benchmarks and strong simple methods

    Derek Lim, Felix Hohne, Xiuyu Li, Sijia Linda Huang, Vaishnavi Gupta, Omkar Bhalerao, and Ser Nam Lim. Large scale learning on non-homophilous graphs: New benchmarks and strong simple methods. In Advances in Neural Information Processing Systems, volume 34, pp.\ 20887--20902, 2021

    2021

  73. [81]

    Towards deeper graph neural networks

    Meng Liu, Hongyang Gao, and Shuiwang Ji. Towards deeper graph neural networks. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp.\ 338--348, 2020

    2020

  74. [82]

    Skipnode: On alleviating over-smoothing for deep graph convolutional networks

    Weigang Lu, Yibing Zhan, Ziyu Guan, Liu Liu, Baosheng Yu, Wei Zhao, Yaming Yang, and Dacheng Tao. Skipnode: On alleviating over-smoothing for deep graph convolutional networks. arXiv preprint arXiv:2112.11628, 2021

    2021 arXiv

  75. [83]

    Break the ceiling: Stronger multi-scale deep graph convolutional networks

    Sitao Luan, Mingde Zhao, Xiao-Wen Chang, and Doina Precup. Break the ceiling: Stronger multi-scale deep graph convolutional networks. In Advances in Neural Information Processing Systems, volume 32, 2019

    2019

  76. [84]

    Training matters: Unlocking potentials of deeper graph convolutional neural networks

    Sitao Luan, Mingde Zhao, Xiao-Wen Chang, and Doina Precup. Training matters: Unlocking potentials of deeper graph convolutional neural networks. arXiv preprint arXiv:2008.08838, 2020

    2008 arXiv

  77. [85]

    Classic gnns are strong baselines: Reassessing gnns for node classification

    Yuankai Luo, Lei Shi, and Xiao-Ming Wu. Classic gnns are strong baselines: Reassessing gnns for node classification. arXiv preprint arXiv:2406.08993, 2024

    2024 arXiv

  78. [86]

    A fractional graph laplacian approach to oversmoothing

    Sohir Maskey, Raffaele Paolino, Aras Bacho, and Gitta Kutyniok. A fractional graph laplacian approach to oversmoothing. Advances in Neural Information Processing Systems, 36, 2024

    2024

  79. [87]

    Gaussian process behaviour in wide deep neural networks

    Alexander G de G Matthews, Mark Rowland, Jiri Hron, Richard E Turner, and Zoubin Ghahramani. Gaussian process behaviour in wide deep neural networks. In International Conference on Learning Representations, 2018

    2018

  80. [88]

    Scattering gcn: Overcoming oversmoothness in graph convolutional networks

    Yimeng Min, Frederik Wenkel, and Guy Wolf. Scattering gcn: Overcoming oversmoothness in graph convolutional networks. Advances in Neural Information Processing Systems, 33: 0 14498--14508, 2020

    2020

  81. [89]

    Solving mixed integer programs using neural networks

    Vinod Nair, Sergey Bartunov, Felix Gimeno, Ingrid Von Glehn, Pawel Lichocki, Ivan Lobov, Brendan O'Donoghue, Nicolas Sonnerat, Christian Tjandraatmadja, Pengming Wang, et al. Solving mixed integer programs using neural networks. arXiv preprint arXiv:2012.13349, 2020

    2012 arXiv

  82. [90]

    Bayesian learning for neural networks, volume 118

    Radford M Neal. Bayesian learning for neural networks, volume 118. Springer-Verlag, 1996

    1996

  83. [91]

    Random gradient-free minimization of convex functions

    Yurii Nesterov and Vladimir Spokoiny. Random gradient-free minimization of convex functions. Foundations of Computational Mathematics, 17 0 (2): 0 527--566, 2017

    2017

  84. [92]

    Revisiting over-smoothing and over-squashing using ollivier-ricci curvature

    Khang Nguyen, Nong Minh Hieu, Vinh Duc Nguyen, Nhat Ho, Stanley Osher, and Tan Minh Nguyen. Revisiting over-smoothing and over-squashing using ollivier-ricci curvature. In International Conference on Machine Learning, pp.\ 25956--25979. PMLR, 2023

    2023

  85. [93]

    Graph neural networks exponentially lose expressive power for node classification

    Kenta Oono and Taiji Suzuki. Graph neural networks exponentially lose expressive power for node classification. In International Conference on Learning Representations, 2019

    2019

  86. [94]

    Applications of combinatorial optimization

    Vangelis Th Paschos. Applications of combinatorial optimization. John Wiley & Sons, 2014

    2014

  87. [95]

    Resurrecting the sigmoid in deep learning through dynamical isometry: Theory and practice

    Jeffrey Pennington, Samuel Schoenholz, and Surya Ganguli. Resurrecting the sigmoid in deep learning through dynamical isometry: Theory and practice. Advances in Neural Information Processing Systems, 30, 2017

    2017

  88. [96]

    The emergence of spectral universality in deep networks

    Jeffrey Pennington, Samuel Schoenholz, and Surya Ganguli. The emergence of spectral universality in deep networks. In International Conference on Artificial Intelligence and Statistics, pp.\ 1924--1932. PMLR, 2018

    1924

  89. [97]

    A critical look at the evaluation of gnns under heterophily: Are we really making progress? arXiv preprint arXiv:2302.11640, 2023

    Oleg Platonov, Denis Kuznedelev, Michael Diskin, Artem Babenko, and Liudmila Prokhorenkova. A critical look at the evaluation of gnns under heterophily: Are we really making progress? arXiv preprint arXiv:2302.11640, 2023

    2023 arXiv

  90. [98]

    Exponential expressivity in deep neural networks through transient chaos

    Ben Poole, Subhaneil Lahiri, Maithra Raghu, Jascha Sohl-Dickstein, and Surya Ganguli. Exponential expressivity in deep neural networks through transient chaos. Advances in Neural Information Processing Systems, 29, 2016

    2016

  91. [99]

    Exploring the power of graph neural networks in solving linear optimization problems

    Chendi Qian, Didier Ch \'e telat, and Christopher Morris. Exploring the power of graph neural networks in solving linear optimization problems. In International Conference on Artificial Intelligence and Statistics, pp.\ 1432--1440. PMLR, 2024

    2024

  92. [100]

    Dropedge: Towards deep graph convolutional networks on node classification

    Yu Rong, Wenbing Huang, Tingyang Xu, and Junzhou Huang. Dropedge: Towards deep graph convolutional networks on node classification. In International Conference on Learning Representations, 2020

    2020

  93. [101]

    Simplifying the theory on over-smoothing

    Andreas Roth. Simplifying the theory on over-smoothing. arXiv preprint arXiv:2407.11876, 2024

    2024 arXiv

  94. [102]

    Rank collapse causes over-smoothing and over-correlation in graph neural networks

    Andreas Roth and Thomas Liebig. Rank collapse causes over-smoothing and over-correlation in graph neural networks. In Learning on Graphs Conference, pp.\ 35--1. PMLR, 2024

    2024

  95. [103]

    A survey on oversmoothing in graph neural networks

    T Konstantin Rusch, Michael M Bronstein, and Siddhartha Mishra. A survey on oversmoothing in graph neural networks. arXiv preprint arXiv:2303.10993, 2023 a

    2023 arXiv

  96. [104]

    Gradient gating for deep multi-rate learning on graphs

    T Konstantin Rusch, Benjamin P Chamberlain, Michael W Mahoney, Michael M Bronstein, and Siddhartha Mishra. Gradient gating for deep multi-rate learning on graphs. In International Conference on Learning Representations, 2023 b

    2023

  97. [105]

    New insights into graph convolutional networks using neural tangent kernels

    Mahalakshmi Sabanayagam, Pascal Esser, and Debarghya Ghoshdastidar. New insights into graph convolutional networks using neural tangent kernels. arXiv preprint arXiv:2110.04060, 2021

    2021 arXiv

  98. [106]

    Representation power of graph convolutions: Neural tangent kernel analysis

    Mahalakshmi Sabanayagam, Pascal Esser, and Debarghya Ghoshdastidar. Representation power of graph convolutions: Neural tangent kernel analysis. arXiv preprint arXiv:2210.09809, 2022

    2022 arXiv

  99. [107]

    Exact solutions to the nonlinear dynamics of learning in deep linear neural networks

    Andrew M Saxe, James L McClelland, and Surya Ganguli. Exact solutions to the nonlinear dynamics of learning in deep linear neural networks. In International Conference on Learning Represenatations, 2014

    2014

  100. [108]

    Deep information propagation

    Samuel S Schoenholz, Justin Gilmer, Surya Ganguli, and Jascha Sohl-Dickstein. Deep information propagation. In International Conference on Learning Representations, 2017

    2017

  101. [109]

    Residual connections and normalization can provably prevent oversmoothing in gnns

    Michael Scholkemper, Xinyi Wu, Ali Jadbabaie, and Michael T Schaub. Residual connections and normalization can provably prevent oversmoothing in gnns. arXiv preprint arXiv:2406.02997, 2024

    2024

  102. [110]

    Collective classification in network data

    Prithviraj Sen, Galileo Namata, Mustafa Bilgic, Lise Getoor, Brian Galligher, and Tina Eliassi-Rad. Collective classification in network data. AI magazine, 29 0 (3): 0 93--93, 2008

    2008

  103. [111]

    Pitfalls of graph neural network evaluation

    Oleksandr Shchur, Maximilian Mumme, Aleksandar Bojchevski, and Stephan G \"u nnemann. Pitfalls of graph neural network evaluation. arXiv preprint arXiv:1811.05868, 2018

    2018 arXiv

  104. [112]

    Graph transformers: A survey

    Ahsan Shehzad, Feng Xia, Shagufta Abid, Ciyuan Peng, Shuo Yu, Dongyu Zhang, and Karin Verspoor. Graph transformers: A survey. arXiv preprint arXiv:2407.09777, 2024

    2024

  105. [113]

    Very deep convolutional networks for large-scale image recognition

    Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. In International Conference on Learning Representations, 2015

    2015

  106. [114]

    On the infinite width limit of neural networks with a standard parameterization

    Jascha Sohl-Dickstein, Roman Novak, Samuel S Schoenholz, and Jaehoon Lee. On the infinite width limit of neural networks with a standard parameterization. arXiv preprint arXiv:2001.07301, 2020

    2001 arXiv

  107. [115]

    Dropout: A simple way to prevent neural networks from overfitting

    Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: A simple way to prevent neural networks from overfitting. In Journal of Machine Learning Research, volume 15, pp.\ 1929--1958, 2014

    1929

  108. [116]

    Representation benefits of deep feedforward networks

    Matus Telgarsky. Representation benefits of deep feedforward networks. arXiv preprint arXiv:1509.08101, 2015

    2015 arXiv

  109. [117]

    Graph attention networks

    Petar Veli c kovi \'c , Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Li \`o , and Yoshua Bengio. Graph attention networks. 2017

    2017

  110. [118]

    Bag of tricks for node classification with graph neural networks

    Yangkun Wang, Jiarui Jin, Weinan Zhang, Yong Yu, Zheng Zhang, and David Wipf. Bag of tricks for node classification with graph neural networks. arXiv preprint arXiv:2103.13355, 2021

    2021 arXiv

  111. [119]

    Graph Neural Networks: Foundations, Frontiers, and Applications

    Lingfei Wu, Peng Cui, Jian Pei, and Liang Zhao. Graph Neural Networks: Foundations, Frontiers, and Applications. Springer Nature, 2022 a

    2022

  112. [120]

    Nodeformer: A scalable graph structure learning transformer for node classification

    Qitian Wu, Wentao Zhao, Zenan Li, David P Wipf, and Junchi Yan. Nodeformer: A scalable graph structure learning transformer for node classification. Advances in Neural Information Processing Systems, 35: 0 27387--27401, 2022 b

    2022

  113. [121]

    A non-asymptotic analysis of oversmoothing in graph neural networks

    Xinyi Wu, Zhengdao Chen, William Wang, and Ali Jadbabaie. A non-asymptotic analysis of oversmoothing in graph neural networks. International Conference on Learning Representations, 2023

    2023

  114. [122]

    Demystifying oversmoothing in attention-based graph neural networks

    Xinyi Wu, Amir Ajorlou, Zihui Wu, and Ali Jadbabaie. Demystifying oversmoothing in attention-based graph neural networks. Advances in Neural Information Processing Systems, 36, 2024

    2024

  115. [123]

    A comprehensive survey on graph neural networks

    Zonghan Wu, Shirui Pan, Fengwen Chen, Guodong Long, Chengqi Zhang, and S Yu Philip. A comprehensive survey on graph neural networks. In IEEE Transactions on Neural Networks and Learning Systems, volume 32, pp.\ 4--24. IEEE, 2020

    2020

  116. [124]

    Dynamical isometry and a mean field theory of CNNs : How to train 10,000-layer vanilla convolutional neural networks

    Lechao Xiao, Yasaman Bahri, Jascha Sohl-Dickstein, Samuel Schoenholz, and Jeffrey Pennington. Dynamical isometry and a mean field theory of CNNs : How to train 10,000-layer vanilla convolutional neural networks. In International Conference on Machine Learning, pp.\ 5393--5402....

    2018

  117. [125]

    Representation learning on graphs with jumping knowledge networks

    Keyulu Xu, Chengtao Li, Yonglong Tian, Tomohiro Sonobe, Ken-ichi Kawarabayashi, and Stefanie Jegelka. Representation learning on graphs with jumping knowledge networks. In International Conference on Machine Learning, pp.\ 5453--5462. PMLR, 2018

    2018

  118. [126]

    Optimization of graph neural networks: Implicit acceleration by skip connections and more depth

    Keyulu Xu, Mozhi Zhang, Stefanie Jegelka, and Kenji Kawaguchi. Optimization of graph neural networks: Implicit acceleration by skip connections and more depth. In International Conference on Machine Learning, pp.\ 11592--11602. PMLR, 2021

    2021

  119. [127]

    Two sides of the same coin: Heterophily and oversmoothing in graph convolutional neural networks

    Yujun Yan, Milad Hashemi, Kevin Swersky, Yaoqing Yang, and Danai Koutra. Two sides of the same coin: Heterophily and oversmoothing in graph convolutional neural networks. In 2022 IEEE International Conference on Data Mining (ICDM). IEEE, 2022

    2022

  120. [128]

    Revisiting over-smoothing in deep gcns

    Chaoqi Yang, Ruijie Wang, Shuochao Yao, Shengzhong Liu, and Tarek Abdelzaher. Revisiting over-smoothing in deep gcns. arXiv preprint arXiv:2003.13663, 2020

    2003 arXiv

  121. [129]

    Graph neural networks are inherently good generalizers: Insights by bridging gnns and mlps

    Chenxiao Yang, Qitian Wu, Jiahua Wang, and Junchi Yan. Graph neural networks are inherently good generalizers: Insights by bridging gnns and mlps. In International Conference on Learning Representations, 2023 a

    2023

  122. [130]

    Mean field residual networks: On the edge of chaos

    Ge Yang and Samuel S Schoenholz. Mean field residual networks: On the edge of chaos. Advances in Neural Information Processing Systems, 30, 2017

    2017

  123. [131]

    Bridging smoothness and approximation: Theoretical insights into over-smoothing in graph neural networks

    Guangrui Yang, Jianfei Li, Ming Li, Han Feng, and Ding-Xuan Zhou. Bridging smoothness and approximation: Theoretical insights into over-smoothing in graph neural networks. arXiv preprint arXiv:2407.01281, 2024

    2024 arXiv

  124. [132]

    Tackling over-smoothing in graph convolutional networks with em-based joint topology optimization and node classification

    Rui Yang, Wenrui Dai, Chenglin Li, Junni Zou, and Hongkai Xiong. Tackling over-smoothing in graph convolutional networks with em-based joint topology optimization and node classification. In IEEE Transactions on Signal and Information Processing over Networks, volume 9, pp.\ 1...

    2023

  125. [133]

    Revisiting semi-supervised learning with graph embeddings

    Zhilin Yang, William Cohen, and Ruslan Salakhudinov. Revisiting semi-supervised learning with graph embeddings. In International Conference on Machine Learning, pp.\ 40--48. PMLR, 2016

    2016

  126. [134]

    Revisiting graph convolutional network on semi-supervised node classification from an optimization perspective

    Hongwei Zhang, Tijin Yan, Zenjun Xie, Yuanqing Xia, and Yuan Zhang. Revisiting graph convolutional network on semi-supervised node classification from an optimization perspective. arXiv preprint arXiv:2009.11469, 2020

    2009 arXiv

  127. [135]

    Model degradation hinders deep graph neural networks

    Wentao Zhang, Zeang Sheng, Ziqi Yin, Yuezihan Jiang, Yikuan Xia, Jun Gao, Zhi Yang, and Bin Cui. Model degradation hinders deep graph neural networks. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pp.\ 2493--2503, 2022 a

    2022

  128. [136]

    Graph-adaptive rectified linear unit for graph neural networks

    Yifei Zhang, Hao Zhu, Ziqiao Meng, Piotr Koniusz, and Irwin King. Graph-adaptive rectified linear unit for graph neural networks. In Proceedings of the ACM web conference 2022, pp.\ 1331--1339, 2022 b

    2022

  129. [137]

    Pairnorm: Tackling oversmoothing in gnns

    Lingxiao Zhao and Leman Akoglu. Pairnorm: Tackling oversmoothing in gnns. In International Conference on Learning Representations, 2020

    2020

  130. [138]

    Tackling oversmoothing of gnns with contrastive learning

    Lecheng Zheng, Dongqi Fu, and Jingrui He. Tackling oversmoothing of gnns with contrastive learning. 2021

    2021

  131. [139]

    Graph neural networks: A review of methods and applications

    Jie Zhou, Ganqu Cui, Shengding Hu, Zhengyan Zhang, Cheng Yang, Zhiyuan Liu, Lifeng Wang, Changcheng Li, and Maosong Sun. Graph neural networks: A review of methods and applications. AI open, 1: 0 57--81, 2020 a

    2020

  132. [140]

    Towards deeper graph neural networks with differentiable group normalization

    Kaixiong Zhou, Xiao Huang, Yuening Li, Daochen Zha, Rui Chen, and Xia Hu. Towards deeper graph neural networks with differentiable group normalization. In Advances in Neural Information Processing Systems, volume 33, pp.\ 4917--4928, 2020 b

    2020

  133. [141]

    Dirichlet energy constrained learning for deep graph neural networks

    Kaixiong Zhou, Xiao Huang, Daochen Zha, Rui Chen, Li Li, Soo-Hyun Choi, and Xia Hu. Dirichlet energy constrained learning for deep graph neural networks. In Advances in Neural Information Processing Systems, volume 34, pp.\ 21834--21846, 2021 a

    2021

  134. [142]

    Understanding and resolving performance degradation in deep graph convolutional networks

    Kuangqi Zhou, Yanfei Dong, Kaixin Wang, Wee Sun Lee, Bryan Hooi, Huan Xu, and Jiashi Feng. Understanding and resolving performance degradation in deep graph convolutional networks. In Proceedings of the 30th ACM International Conference on Information and Knowledge Management,...

    2021

  135. [143]

    On the explainability of graph convolutional network with gcn tangent kernel

    Xianchen Zhou and Hongxia Wang. On the explainability of graph convolutional network with gcn tangent kernel. Neural Computation, 35 0 (1): 0 1--26, 2022

    2022

  136. [144]

    Gradinit: Learning to initialize neural networks for stable and efficient training

    Chen Zhu, Renkun Ni, Zheng Xu, Kezhi Kong, W Ronny Huang, and Tom Goldstein. Gradinit: Learning to initialize neural networks for stable and efficient training. Advances in Neural Information Processing Systems, 34: 0 16410--16422, 2021

    2021

  137. [145]

    Beyond homophily in graph neural networks: Current limitations and effective designs

    Jiong Zhu, Yujun Yan, Lingxiao Zhao, Mark Heimann, Leman Akoglu, and Danai Koutra. Beyond homophily in graph neural networks: Current limitations and effective designs. Advances in Neural Information Processing Systems, 33: 0 7793--7804, 2020

    2020

  138. [146]

    Layer-dependent importance sampling for training deep and large graph convolutional networks

    Difan Zou, Ziniu Hu, Yewen Wang, Song Jiang, Yizhou Sun, and Quanquan Gu. Layer-dependent importance sampling for training deep and large graph convolutional networks. In Advances in Neural Information Processing Systems, volume 32, 2019

    2019

  139. [147]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.