Pith. sign in

REVIEW 3 major objections 5 minor 27 references

Variational Graph Recurrent Neural Networks

T0 review · 3 major / 5 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A graph recurrent network that embeds nodes as random latent variables consistently beats deterministic dynamic-embedding baselines in link prediction, with the largest margins on sparse graphs.

desk verdict A solid variational dynamic graph model, but the 'state-of-the-art' claim outruns the baseline selection. read the letter →

arxiv 1908.09710 v3 pith:HE5CTW23 submitted 2019-08-26 cs.LG stat.ML

Ehsan Hajiramezanali , Arman Hasanzadeh , Nick Duffield , Krishna R Narayanan , Mingyuan Zhou , Xiaoning Qian This is my paper · ORCID
classification cs.LGstat.ML
keywords variationalgraphrecurrentneuralnetworkdynamicembeddingsemi-implicitinferencelinkpredictionnodeuncertaintyconvolutionalsparsegraphs
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to show that stochastic node embeddings fix a weakness of deterministic dynamic graph embeddings. Existing recurrent graph models compress all temporal variability into a deterministic hidden state, which tends to overfit high-variation sequences; VGRNN instead treats each node's latent representation at each time step as a random vector whose prior is learned from the previous hidden state, and SI-VGRNN makes that posterior non-Gaussian through semi-implicit variational inference. If the claims hold, dynamic link prediction on evolving graphs is improved simply by making the latent state random and hierarchical, with no need for smoothness assumptions. The gain is reported to be largest on sparse graphs with few clusters, where deterministic baselines overfit the small number of observed links.

What carries the argument

The machinery is a hierarchical latent variable $Z^{(t)}$ inserted between the adjacency matrix and the recurrent state of a graph recurrent network. At each snapshot the prior parameters come from a network $\phi_{\text{prior}}(h_{t-1})$, the decoder reconstructs $A^{(t)}$ from $Z^{(t)}$ by an inner product, and the recurrent update $h_t = f(A^{(t)}, \phi_x(X^{(t)}), \phi_z(Z^{(t)}), h_{t-1})$ carries information forward. The ELBO sums over snapshots a reconstruction term plus a KL divergence between the graph-convolutional posterior and the conditional prior. SI-VGRNN adds stochastic graph-convolutional layers that transform injected noise into the posterior parameters, making the marginal posterior implicit while keeping $q(Z^{(t)} \mid \psi_t)$ Gaussian conditionally.

What would settle it

Ablate the stochastic latent variable by decoding from its posterior mean or from $h_t$ alone with matched parameter count on the same six datasets; if the deterministic version reaches the same AUC/AP on future-edge prediction, the central claim fails. A second check: if setting the posterior equal to the learned prior (zero KL) leaves link prediction unchanged, the latent hierarchy is not carrying the signal.

Watch

Extended reading notes

Core claim

The central claim is that graph recurrent networks become substantially better predictors of dynamic graph evolution when the node embedding at each time step is a random variable drawn from a variational posterior whose parameters are graph-convolutional functions of current topology, node attributes, and the previous recurrent hidden state. The model replaces the static Gaussian prior of a vanilla variational graph autoencoder with a learned conditional prior $p(Z^{(t)} \mid h_{t-1})$, so future snapshots can be generated without assuming smoothness. SI-VGRNN extends the posterior by drawing its parameters from an implicit mixing distribution, yielding flexible non-Gaussian node representations. Reported results show higher AUC and AP than deterministic baselines on dynamic link detection, link prediction, and new-link prediction, with the largest margins on very sparse graphs; the paper also shows that the variance of a node's latent representation rises when the node moves between communities.

Load-bearing premise

Everything rests on the assumption that one compressed hidden vector at the previous time step carries all the information the next latent state needs; if that vector loses details, especially for newly added nodes that start with a zero hidden state, the learned prior and all future predictions are biased.

Editorial extensions

If this is right

  • VGRNN can predict edges in future snapshots because the prior at time $t$ is a function of $h_{t-1}$; deterministic recurrent autoencoders cannot do this without retraining.
  • On link detection across all six datasets, both VGRNN and SI-VGRNN beat VGAE, DynAE, DynRNN, and DynAERNN in AUC and AP, with the largest margins on sparse graphs such as Facebook.
  • For dynamic link prediction and new-link prediction, the proposed models outperform baselines on Enron, COLAB, Facebook, and Social Evolution; new-link gains are larger, indicating the learned prior captures temporal trends, not just current topology.
  • Latent variance is informative: it rises for a node switching communities and stays stable for stationary nodes, so uncertainty encodes abrupt topological change.
  • SI-VGRNN's advantage over VGRNN is small in future-edge prediction because the prior remains Gaussian; making the prior implicit is named in the paper as a natural next step.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A testable extension would replace the Gaussian conditional prior with a semi-implicit prior; if the paper's diagnosis is right, future-edge prediction should improve more than VGRNN's current gains.
  • The variance spike for community-switching nodes suggests node-level predictive uncertainty could serve as an unsupervised changepoint or anomaly signal in dynamic graphs, a task the paper does not evaluate.
  • The treatment of new nodes—zero hidden state and unit Gaussian prior—is the regime most likely to limit inductive performance; conditioning the prior on node attributes instead of hidden state alone could be tested against the reported new-link results.
  • If stochastic latents help mainly by preventing overfitting on sparse snapshots, the same hierarchical conditioning should transfer to other sparse dynamic graph tasks such as temporal community detection and link ranking under missing data.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper introduces VGRNN and SI-VGRNN, variational graph recurrent neural networks that maintain stochastic latent node representations over time. A GRNN backbone with graph-convolutional recurrent updates produces a hidden state h_{t-1}, which conditions both the prior for the latent variables Z(t) at each snapshot and the variational posterior used for inference. The generative model factorizes across snapshots and reconstructs the observed adjacency matrix via an inner-product decoder. SI-VGRNN applies semi-implicit variational inference to learn a more flexible, non-Gaussian posterior by injecting random noise through additional GCN layers. The authors evaluate the methods on dynamic link detection, dynamic link prediction, and dynamic new link prediction across six datasets, reporting AUC/AP improvements over DynAE, DynRNN, DynAERNN, and VGAE, and they provide a qualitative latent-space analysis on a simulated graph.

Significance. If the performance claims withstand closer comparison, the paper makes a useful contribution to dynamic graph representation learning: it is one of the first to combine stochastic latent variables with graph recurrent architectures and to show that a semi-implicit posterior can improve dynamic link prediction. The manuscript also provides a public implementation, reports standard errors over ten runs, and discusses when the gains are largest (sparse, low-clustering graphs). The methodological core, including the ELBO objective and the SIVI lower bound, is broadly sound and follows established template models such as VRNN and VGAE. The main weakness is empirical support: the claimed 'state-of-the-art' comparison omits several strong dynamic embedding baselines that the paper itself cites, and there is a potential input-feature confound for attribute-free datasets. These issues are addressable and do not appear to require changes to the model formulation itself.

major comments (3)
  1. [Section 4, 'Competing methods'; Tables 2-4] The abstract and Section 4.1 claim that VGRNN and SI-VGRNN 'consistently outperform the existing baseline and state-of-the-art methods by a significant margin,' but the comparison in Tables 2-4 contains only DynAE, DynRNN, DynAERNN, and VGAE. The paper itself cites DyREP [24], DynGEM [10], and CTDNE-based dynamic embedding [26], none of which is evaluated. DynAERNN is not the strongest published dynamic embedding method for this task, so the 'state-of-the-art' portion of the claim is unsupported by the evidence. Please add the strongest published baselines (at minimum DyREP and DynGEM) or explicitly re-scope the claim to 'the baselines considered in this paper.'
  2. [Section 4, 'Experimental setups'] For datasets without node attributes, VGRNN and SI-VGRNN receive an N_t-by-N_t identity matrix as node attributes at each time t, but the manuscript does not state what inputs DynAE, DynRNN, and DynAERNN receive. If those baselines consume only adjacency information while the proposed models receive identity-based node features, the gap in Tables 2-4 could be attributable to richer input features rather than to stochastic latent dynamics or the hierarchical prior. This is a load-bearing confound for the central claim. Please report the exact input to each method and include an ablation in which baselines receive the same node-feature inputs, or an ablation in which VGRNN receives only adjacency information.
  3. [Section 3.3, Eqs. (9)-(11)] The SI-VGRNN encoder is described by the recursion ell_j^{(t)} = GNN_j(A^{(t)}, CONCAT(h_{t-1}, epsilon_j^{(t)}, ell_{j-1}^{(t)})), but the functions GNN_j and the initialization ell_0^{(t)} = phi_x^tau(X^{(t)}) are not specified precisely; the superscript on phi_x appears to be a typo. In addition, Eq. (11) is asserted as a lower bound on the SI-VGRNN ELBO without a derivation in the main text, and the referenced supplement is not part of the arXiv version. Since the semi-implicit objective is central to the SI-VGRNN results, please provide a complete derivation or a precise citation to the derivation, and define the encoder architecture, including how L, the noise dimensions, and the deterministic covariance Sigma are combined.
minor comments (5)
  1. [Section 3.2, Eq. (2)] The notation p(Z(t)) in Eq. (2) omits the conditioning on h_{t-1}; the later text clarifies that this is a conditional prior, but the notation should be made consistent (e.g., p(Z(t) | h_{t-1})) to avoid confusion with an unconditional prior.
  2. [Section 3.2 and Figure 1 caption] The sentence 'operations (a)-(d) correspond to equations (2)-(4), and (3.2)' and the Figure 1 caption refer to '(3.2)' as if it were an equation; this should be Eq. (6), the posterior definition.
  3. [Section 4, 'Experimental setups'] The statement 'the number of parameters are the same for the competing methods' is vague; please specify how parameters were matched across architectures with different structures (e.g., which layer widths were adjusted) and report the resulting parameter counts.
  4. [Section 4, Tables 2-4] Tables 2-4 combine AUC and AP blocks in a way that is dense and hard to scan; adding explicit row subheadings or separating the two metrics into distinct panels would improve readability.
  5. [Section 4.2] The qualitative latent-space analysis would be clearer if the red and green node contours in Figure 4 were explicitly identified in the caption and if the variance values were reported numerically as a function of time.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the VGRNN/SI-VGRNN derivation is self-contained and predictions are evaluated on held-out future snapshots.

full rationale

The derivation chain is not circular. The generative model in Eqs. (2)-(5) defines a hierarchical prior p(Z(t)|A(<t),X(<t),Z(<t)) via the recurrent state h_{t-1}, and h_{t-1} is itself updated causally from earlier A, X, and Z through Eq. (4); the ELBO in Eq. (7) is the standard variational bound for that factorization. For SI-VGRNN, Eq. (11) is a Jensen lower bound on Eq. (10), namely KL(E_q q(Z|psi)||p) <= E_q KL(q(Z|psi)||p), so it is not a restatement of the model's outputs. The predictive claims in Tables 3-4 are evaluated on future snapshots that were not used to fit the model, so no fitted input is relabeled as a prediction. The only author-overlap citation is SIVI [25], used as an off-the-shelf inference method; no uniqueness theorem or self-citation chain is invoked to force the architecture. The abstract's 'state-of-the-art' wording is under-supported because strong dynamic baselines such as DyREP and DynGEM are absent from the comparison, but that is an empirical-support gap, not a circular reduction. Under the hard rules, no circular step can be exhibited from the paper's own equations, so the circularity score is 0.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

The central derivation relies on standard variational inference and reparameterization, plus domain assumptions about how dynamic graphs and node identities behave. No new physical entities or ad hoc numeric constants are introduced; the listed hyperparameters are standard model choices.

free parameters (5)
  • Hidden state dimension = 32
    Chosen for the GRU hidden layer and all phi layers; the paper states this setup was used for all datasets but does not report sensitivity analysis.
  • Learning rate = 0.01
    Fixed across experiments; no schedule or sensitivity analysis reported.
  • Number of training epochs = 1500
    Fixed with early stopping; no ablation on epoch count.
  • Number of GCN layers in encoder = 2
    Two-layer GCN with sizes [32,16] used for mu and sigma in VGRNN.
  • Noise dimension in SI-VGRNN = 16
    Dimension of injected Gaussian noise epsilon in the semi-implicit encoder.
assumptions (5)
  • standard math The ELBO lower bound is a valid optimization objective for the generative model in Equation (5).
    Standard variational inference result, assumed without proof.
  • domain assumption The prior and posterior factorize across nodes (mean-field approximation).
    Equations (2) and (6) assume independent node latents; correlations between nodes are only captured through the GNN feature extractors.
  • ad hoc to paper The recurrent hidden state ht-1 is a sufficient statistic of the past for predicting Z(t).
    The model conditions the prior solely on ht-1 (Equation (2)), which is a strong Markov assumption for dynamic graphs with variable node sets.
  • domain assumption New nodes entering at time t receive hidden state 0 and prior N(0,I).
    Stated in Section 3.2; this assumption may misrepresent nodes that enter with a known structural role.
  • standard math The reparameterization trick can be applied to sample Z(t) and the SIVI noise.
    Required for gradient-based ELBO optimization; relies on continuous latent variables.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Variational Graph Recurrent Neural Networks." pith.science (2026). https://pith.science/paper/HE5CTW23

@misc{pith2026190809710,
  author       = {Pith},
  title        = {Pith review of: Variational Graph Recurrent Neural Networks},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/HE5CTW23}},
  note         = {Machine review of arXiv:1908.09710}
}
read the original abstract

Representation learning over graph structured data has been mostly studied in static graph settings while efforts for modeling dynamic graphs are still scant. In this paper, we develop a novel hierarchical variational model that introduces additional latent random variables to jointly model the hidden states of a graph recurrent neural network (GRNN) to capture both topology and node attribute changes in dynamic graphs. We argue that the use of high-level latent random variables in this variational GRNN (VGRNN) can better capture potential variability observed in dynamic graphs as well as the uncertainty of node latent representation. With semi-implicit variational inference developed for this new VGRNN architecture (SI-VGRNN), we show that flexible non-Gaussian latent representations can further help dynamic graph analytic tasks. Our experiments with multiple real-world dynamic graph datasets demonstrate that SI-VGRNN and VGRNN consistently outperform the existing baseline and state-of-the-art methods by a significant margin in dynamic link prediction.

Figures

Figures reproduced from arXiv: 1908.09710 by the authors.

Figure 1
Figure 1. Graphical illustrations of each operation of VGRNN; (a) computing the conditional prior [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Evolution of graph statistics through time. that our proposed models have better generalization, which is the result of including random latent variables in our model. We note that our proposed methods improve new link prediction more substantially which shows that they can capture temporal trends better than the competing methods. Comparing VGRNN with SI-VGRNN shows that the prediction results are almost the same f… view at source ↗
Figure 3
Figure 3. Evolution of simulated graph topology through time [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Latent representations of the simulated graph in different time steps in 2-d space using VGRNN. 4.2 Interpretable latent representations To show that VGRNN learns more interpretable latent representations, we simulated a dynamic graph with three communities in which a …

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

27 extracted references · 19 canonical work pages

  1. [24]

    Dyrep: Learning representations over dynamic graphs

    Rakshit Trivedi, Mehrdad Farajtabar, Prasenjeet Biswal, and Hongyuan Zha. Dyrep: Learning representations over dynamic graphs. In International Conference on Learning Representations, 2019

    work page 2019

  2. [10]

    Dyngem: Deep embedding method for dynamic graphs

    Palash Goyal, Nitin Kamra, Xinran He, and Yan Liu. Dyngem: Deep embedding method for dynamic graphs. arXiv preprint arXiv:1805.11273, 2018

    arXiv 2018

  3. [26]

    Dynamic network embedding by modeling triadic closure process

    Lekui Zhou, Yang Yang, Xiang Ren, Fei Wu, and Yueting Zhuang. Dynamic network embedding by modeling triadic closure process. InThirty-Second AAAI Conference on Artificial Intelligence, 2018. 11

    work page 2018

  4. [1]

    Robust negative sam- pling for network embedding

    Mohammadreza Armandpour, Patrick Ding, Jianhua Huang, and Xia Hu. Robust negative sam- pling for network embedding. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 3191–3198. AAAI, 2019

    work page 2019

  5. [2]

    Deep gaussian embedding of graphs: Unsuper- vised inductive learning via ranking

    Aleksandar Bojchevski and Stephan Günnemann. Deep gaussian embedding of graphs: Unsuper- vised inductive learning via ranking. In International Conference on Learning Representations,

    work page

  6. [3]

    A recurrent latent variable model for sequential data

    Junyoung Chung, Kyle Kastner, Laurent Dinh, Kratarth Goel, Aaron C Courville, and Yoshua Bengio. A recurrent latent variable model for sequential data. InAdvances in neural information processing systems, pages 2980–2988, 2015

    work page 2015

  7. [4]

    Convolutional neural networks on graphs with fast localized spectral filtering

    Michaël Defferrard, Xavier Bresson, and Pierre Vandergheynst. Convolutional neural networks on graphs with fast localized spectral filtering. In Advances in Neural Information Processing Systems, pages 3844–3852, 2016

    2016

  8. [5]

    Learning structural node embeddings via diffusion wavelets

    Claire Donnat, Marinka Zitnik, David Hallac, and Jure Leskovec. Learning structural node embeddings via diffusion wavelets. In International ACM Conference on Knowledge Discovery and Data Mining (KDD), volume 24, 2018

    work page 2018

Show all 27 references
  1. [6]

    Sequential neural models with stochastic layers

    Marco Fraccaro, Søren Kaae Sø nderby, Ulrich Paquet, and Ole Winther. Sequential neural models with stochastic layers. In D. D. Lee, M. Sugiyama, U. V . Luxburg, I. Guyon, and R. Garnett, editors, Advances in Neural Information Processing Systems 29 , pages 2199–2207. Curran A...

    2016

  2. [7]

    Sequential neural models with stochastic layers

    Marco Fraccaro, Søren Kaae Sønderby, Ulrich Paquet, and Ole Winther. Sequential neural models with stochastic layers. In Advances in neural information processing systems, pages 2199–2207, 2016

    2016

  3. [8]

    Z-forcing: Training stochastic recurrent networks

    Alias Parth Goyal, Anirudh Goyal, Alessandro Sordoni, Marc-Alexandre Côté, Nan Rose- mary Ke, and Yoshua Bengio. Z-forcing: Training stochastic recurrent networks. In I. Guyon, U. V . Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances...

    2017

  4. [9]

    Z-forcing: Training stochastic recurrent networks

    Anirudh Goyal, Alessandro Sordoni, Marc-Alexandre Côté, Nan Rosemary Ke, and Yoshua Bengio. Z-forcing: Training stochastic recurrent networks. In Advances in neural information processing systems, pages 6713–6723, 2017

    2017

  5. [11]

    dyngraph2vec: Capturing network dynamics using dynamic graph representation learning

    Palash Goyal, Sujit Rokka Chhetri, and Arquimedes Canedo. dyngraph2vec: Capturing network dynamics using dynamic graph representation learning. Knowledge-Based Systems, 2019

    2019

  6. [12]

    node2vec: Scalable feature learning for networks

    Aditya Grover and Jure Leskovec. node2vec: Scalable feature learning for networks. In Proceedings of the 22nd ACM SIGKDD international conference on Knowledge discovery and data mining, pages 855–864. ACM, 2016

    2016

  7. [13]

    Inductive representation learning on large graphs

    Will Hamilton, Zhitao Ying, and Jure Leskovec. Inductive representation learning on large graphs. In Advances in Neural Information Processing Systems, pages 1024–1034, 2017

    2017

  8. [14]

    Variational graph auto-encoders

    Thomas N Kipf and Max Welling. Variational graph auto-encoders. arXiv preprint arXiv:1611.07308, 2016

    2016 arXiv

  9. [15]

    Semi-supervised classification with graph convolutional networks

    Thomas N Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. In International Conference on Learning Representations, 2017

    2017

  10. [16]

    Attributed network embedding for learning in a dynamic environment

    Jundong Li, Harsh Dani, Xia Hu, Jiliang Tang, Yi Chang, and Huan Liu. Attributed network embedding for learning in a dynamic environment. In Proceedings of the 2017 ACM on Conference on Information and Knowledge Management, pages 387–396. ACM, 2017. 10

    2017

  11. [17]

    Doubly semi- implicit variational inference

    Dmitry Molchanov, Valery Kharitonov, Artem Sobolev, and Dmitry Vetrov. Doubly semi- implicit variational inference. arXiv preprint arXiv:1810.02789, 2018

    2018 arXiv

  12. [18]

    Automatic differentiation in pytorch

    Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. In NIPS-W, 2017

    2017

  13. [19]

    Deepwalk: Online learning of social repre- sentations

    Bryan Perozzi, Rami Al-Rfou, and Steven Skiena. Deepwalk: Online learning of social repre- sentations. In Proceedings of the 20th ACM SIGKDD international conference on Knowledge discovery and data mining, pages 701–710. ACM, 2014

  14. [20]

    struc2vec: Learning node representations from structural identity

    Leonardo FR Ribeiro, Pedro HP Saverese, and Daniel R Figueiredo. struc2vec: Learning node representations from structural identity. In Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pages 385–394. ACM, 2017

    2017

  15. [21]

    Structured sequence modeling with graph convolutional recurrent networks

    Youngjoo Seo, Michaël Defferrard, Pierre Vandergheynst, and Xavier Bresson. Structured sequence modeling with graph convolutional recurrent networks. In International Conference on Neural Information Processing, pages 362–373. Springer, 2018

    2018

  16. [22]

    Variational bi-lstms

    Samira Shabanian, Devansh Arpit, Adam Trischler, and Yoshua Bengio. Variational bi-lstms. arXiv preprint arXiv:1711.05717, 2017

    2017 arXiv

  17. [23]

    Line: Large- scale information network embedding

    Jian Tang, Meng Qu, Mingzhe Wang, Ming Zhang, Jun Yan, and Qiaozhu Mei. Line: Large- scale information network embedding. In Proceedings of the 24th International Conference on World Wide Web, pages 1067–1077. International World Wide Web Conferences Steering Committee, 2015

    2015

  18. [25]

    Semi-implicit variational inference

    Mingzhang Yin and Mingyuan Zhou. Semi-implicit variational inference. In International Conference on Machine Learning, pages 5660–5669, 2018

    2018

  19. [2018]

    URL https://openreview.net/forum?id=r1ZdKJ-0W

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.