{"id":"646b4387-3398-46f7-b284-c5af71d478a7","arxiv_id":"2412.11293","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Mamba-based dynamic graph embedding models achieve comparable or better link prediction than transformer-based models on five benchmarks, with linear instead of quadratic scaling.","lead":"This paper tests whether Mamba, a fast state-space model, can replace transformers for embedding dynamic graphs that change over time. The Mamba-based models match or beat the transformer versions on several link prediction benchmarks while scaling linearly with history length.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The headline Mamba-vs-Transformer advantage may be an artifact of evaluation leakage: Section 4.3 trains the logistic-regression classifier on pairs \"from each graph snapshot\" without limiting them to training timestamps, so test edges may enter classifier training.","rationale":"The reader identified the same weakest assumption, and I agree it is the load-bearing point. I considered alternatives—missing baselines (GraphSSM/TGN/TGAT), lack of code, per-dataset hyperparameter search—but these affect generalizability or interpretability rather than the internal validity of the central comparison. The classifier-split ambiguity is different: it threatens the meaning of every reported number. A wrong split would make the central claim false rather than merely incomplete. The paper is otherwise a straightforward architecture comparison; the Mamba linear-complexity argument is supported by Fig 7 (FLOPs/MACs) and is not in dispute. The correct remedy is to make the evaluation protocol explicit and reproducible, which is exactly a conditional-acceptance condition. The reader's CONDITIONAL verdict remains appropriate; no change is needed beyond enforcing the requested clarifications and code release.","tokens_in":18223,"tokens_out":4508,"duration_ms":43178,"concrete_test":"Obtain the code or re-run the protocol with an explicit temporal split for the classifier: (i) fit logistic regression only on positive/negative node pairs sampled from timestamps 1..T_train (the first 70%), using the same 1:10 ratio; (ii) evaluate MAP/MRR only on snapshots in the held-out 20%; (iii) ensure early stopping/validation of the classifier uses only the 10% validation timestamps. Compare the recomputed Tables 5–10 with the published values. If performance drops substantially or the Mamba-vs-transformer ordering changes on UCI/Bitcoin/Reality Mining, the leakage concern is confirmed; if the numbers are unchanged, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central empirical claim—that Mamba variants outperform transformers on temporally dynamic benchmarks—rests on the MAP/MRR numbers in Tables 5–10. Those numbers are produced by a logistic-regression classifier whose training set is described in §4.3 only as node pairs \"sampled as positive (existing edges) and negative (non-existing edges) from each graph snapshot.\" The text never states that these samples are restricted to the 70% training timestamps, or that validation/test snapshots are excluded from classifier fitting. Since §4.2 explicitly says the trained encoder is used to \"predict and save the embeddings of the nodes for all timestamps,\" the embeddings for test snapshots are available at classifier fit time. If the classifier is fit on positive/negative pairs drawn from all snapshots—including test snapshots—then the reported MAP/MRR reflect the classifier's ability to memorize or rank edges it has already seen, not temporal link prediction on held-out snapshots. This would directly invalidate the abstract's \"comparable or superior performance\" claim. No code is released, so the ambiguity cannot be resolved by inspection; no limitation statement in the paper acknowledges or clarifies the split. This is the single most load-bearing unresolved point because every table and the headline comparison inherit it.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper presents a comparative empirical study of dynamic graph embedding methods based on transformer and Mamba (state-space) architectures. The authors propose three models: ST-TransformerG2G, which augments a prior TransformerG2G model with graph convolutional layers; DG-Mamba, which replaces attention with a selective state-space model; and GDG-Mamba, which adds GINE edge convolutions before the Mamba layer. The models embed each node as a multivariate Gaussian and are trained with a triplet-based contrastive loss. The experiments evaluate temporal link prediction on five datasets (Reality Mining, UCI, SBM, Bitcoin-OTC, Slashdot) using MAP and MRR, with results reported across lookback values and against several baselines. The central claim is that Mamba-based models achieve comparable or superior link prediction performance to transformers while being computationally more efficient, especially on datasets with high temporal variability.","tokens_in":18645,"tokens_out":3694,"duration_ms":34581,"significance":"If the empirical claims hold, the paper would provide a useful data point for the dynamic-graph-embedding community: it shows that selective state-space models can replace attention in a probabilistic temporal graph encoder without losing accuracy, and it analyzes the learned state matrices to motivate this finding. The paper's strengths include the breadth of datasets, per-lookback reporting, multiple random initializations with standard deviations, and a direct FLOPs/MACs comparison. The main contribution is comparative rather than theoretical; its value depends on the correctness of the evaluation protocol, which is currently not fully specified. The absence of code, the ambiguous classifier split, and the omission of a cited SSM baseline (GraphSSM) are the main factors limiting the paper's current conclusiveness.","major_comments":[{"comment":"The temporal split for the logistic regression link-prediction classifier is unspecified. Section 4.2 states that after training, the model is used to 'predict and save the embeddings of the nodes for all timestamps,' and Section 4.3 describes the classifier as trained on positive and negative pairs 'sampled ... from each graph snapshot.' This wording does not restrict classifier training to the 70% training timestamps or to the validation timestamps. If test-snapshot edges enter classifier fitting, the MAP/MRR values in Tables 5-10 reflect memorization rather than temporal link prediction on held-out snapshots, which would invalidate the paper's central performance claim. The authors should state explicitly that classifier training pairs come only from training/validation timestamps, or, if they do not, revise the protocol and all reported numbers accordingly.","section":"Section 4.3"},{"comment":"The hyperparameter and lookback selection procedure is not clearly separated from the test set. Tables 2-4 report per-dataset optimal lookbacks and Optuna-tuned hyperparameters, and Table 10 selects the best-performing lookback for each model. If these values were chosen by maximizing MAP on the test split, the reported results are optimistically biased and the comparison is not a fair evaluation of model selection. The paper should state whether the Optuna objective and lookback choices used the validation split only, and should provide validation-selected results or a nested selection procedure.","section":"Tables 2-4 and Table 10"},{"comment":"GraphSSM [27] is cited as an existing state-space model for temporal graphs, yet it is not included as a baseline in Table 10 or anywhere in the experiments. For a paper whose title and abstract promise a comparative study of Mamba-based and transformer-based dynamic graph embedding, omitting the most relevant SSM baseline weakens the comparison. The authors should either add GraphSSM to the experiments or justify its exclusion, for example by explaining that it targets continuous-time event data or a different link prediction protocol.","section":"Section 2.2 and Section 4.3"},{"comment":"The baseline comparison is incomplete and inconsistently reported. The prose in Section 4.3 lists ROLAND as one of the six baselines, but Table 10 contains results for only DynGEM, dyngraph2vecAE, dyngraph2vecAERNN, EvolveGCN, DynG2G, and the proposed/prior transformer models; ROLAND results are absent. Additionally, several baseline rows report no standard deviations, making it unclear whether those numbers were obtained under the same five-seed protocol and the same evaluation split as the proposed models. The authors should provide complete baseline results under a shared protocol or explicitly state which numbers were taken from prior publications.","section":"Table 10 and Section 4.3"},{"comment":"The DG-Mamba results on Slashdot contain a striking anomaly: for lookback l=5 the MAP is 0.0006 +/- 0.0003, while at l=3 it is 0.0676 +/- 0.0009. This sudden collapse is not explained in the text and may indicate a training failure, a degenerate embedding, or a numerical issue in that configuration. Because Table 10 uses the best lookback for DG-Mamba on Slashdot, this anomaly does not directly affect the headline comparison, but it should be investigated and either explained or removed, and the reported best value should be verified.","section":"Table 9"}],"minor_comments":[{"comment":"The Slashdot dataset description is inconsistent: Table 1 lists 50,825 nodes while the prose says 50,824 nodes; the two should be reconciled.","section":"Section 4.1"},{"comment":"The abstract refers to 'Bitcoin' while the experiments use the Bitcoin-OTC dataset; use the full name consistently throughout the paper.","section":"Abstract and Table 1"},{"comment":"The classifier is said to use 'Mean Average Precision (MAP) loss,' which is not a standard loss for logistic regression; the authors should define the exact objective or clarify that they mean a ranking-style loss.","section":"Section 4.3"},{"comment":"Several hyperparameter values, such as dmodel = 1899 for UCI in Table 3 and dmodel = 1000 for SBM in Table 4, appear to equal the number of nodes in the dataset; this is surprising for a model dimension and should be explained in the text.","section":"Tables 3 and 4"},{"comment":"The computational efficiency comparison reports FLOPs and MACs as a function of lookback, but it does not report wall-clock training or inference times; a brief runtime comparison would strengthen the efficiency claim.","section":"Section 4.5"},{"comment":"The visual analysis of state and attention matrices is qualitative and uses only two datasets; the authors should state whether the figures are representative or provide a quantitative summary of the claimed difference in temporal focus.","section":"Section 4.4"}],"recommendation":"major_revision","confidential_remarks":"The paper is within scope for a graph representation learning or temporal graph mining venue. The main issue is the unresolved evaluation protocol: the classifier split ambiguity in Section 4.3 is load-bearing, and the hyperparameter selection procedure could bias the reported numbers. The missing GraphSSM baseline is also problematic given the paper's framing. I would support a major-revision decision rather than rejection because the ambiguity could plausibly be fixed with a clarified protocol, code release, and additional experiments; the proposed models themselves are clearly described and the central comparison is potentially useful."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nRead the Mamba-vs-Transformer dynamic graph embedding paper. My take: it's a genuinely useful empirical comparison, but the central numerical claim is currently underdetermined by an ambiguous evaluation split, and the authors cite the one baseline they should have compared against without actually comparing against it.\n\nWhat's new: three model combinations (ST-TransformerG2G, DG-Mamba, GDG-Mamba) built from known components, with careful tables across five datasets, all lookbacks reported, and a nice FLOPs/MACs measurement showing linear scaling for the Mamba variants. The state-matrix/attention analysis is a nice qualitative touch. That part is solid.\n\nThe soft spots, in decreasing order of weight. First, Section 4.3 describes the link-prediction classifier as trained on pairs 'sampled from each graph snapshot' with no explicit statement that test snapshots are excluded. Since the embeddings are generated for all timestamps up front, the reported MAP/MRR could be optimistic if the classifier sees test edges. The paper never clarifies this. That's a load-bearing ambiguity, not a nitpick. Second, the authors cite GraphSSM (Li et al.) as an SSM-based temporal graph framework but never run it as a baseline. That's the most directly relevant method to compare against, and its absence is hard to justify. Third, the baseline set is old: DynGEM, dyngraph2vec, EvolveGCN, plus their own prior work. TGN and TGAT are cited but not in the tables. The comparison would be stronger with at least one modern continuous-time method.\n\nMinor: per-dataset Optuna tuning is fine since all lookback results are shown, but a fixed hyperparameter budget across models would make the comparison more convincing. No code is released, which makes the split ambiguity unresolvable.\n\nIf the classifier split is actually clean, the Mamba models' wins on UCI, Bitcoin, and Reality Mining are a solid empirical finding, not a theoretical breakthrough. If the split leaks, the paper's headline collapses. So I'd send it to peer review but require a hard clarification of the classifier protocol, code or at least a precise description of train/val/test timestamps for classifier fitting, and a GraphSSM comparison. This is a paper that needs a serious referee, not a desk reject, because the experimental scope and efficiency measurements are worth preserving.\n\nRecommendation: engage with it, but treat the numeric tables as provisional until the split is clarified.","headline":"Useful empirical comparison, but the headline Mamba-vs-Transformer claim hinges on an ambiguous classifier split and a missing GraphSSM baseline.","tokens_in":19053,"tokens_out":2507,"would_cite":false,"duration_ms":22291,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Mamba-based models match or beat transformers on dynamic graph link prediction.","keywords":["dynamic graph embedding","state-space models","Mamba","transformers","link prediction","temporal graphs","graph neural networks"],"falsifier":"If the classifier were trained on node pairs sampled from test-snapshot edges, the reported numbers would not measure temporal generalization. A direct check would be to re-run the link prediction evaluation with a strict temporal split where the classifier sees no edges from the test interval, then compare MAP and MRR; if the performance drops substantially, the paper's central claim would be weakened.","tokens_in":1154,"feed_emoji":"📈","tokens_out":1764,"duration_ms":50523,"temperature":0.7,"pith_summary":"This paper tries to establish that selective state-space models (Mamba) can substitute for attention-based transformers in dynamic graph embedding, yielding comparable or better temporal link prediction on volatile real-world benchmarks while cutting the computational cost of long histories from quadratic to linear. The authors introduce three models, ST-TransformerG2G, DG-Mamba, and GDG-Mamba, and evaluate them on five datasets, reporting that Mamba variants outperform transformer-based models on datasets with high temporal variability. If correct, the finding matters because attention's quadratic cost is the main barrier to modeling long graph histories, so linear-complexity SSMs would let temporal graph models scale to longer and larger sequences.","feed_headline":"Mamba rivals transformers on dynamic graph prediction","feed_subtitle":"Selective state-space models match or beat attention on volatile networks at linear cost, the paper reports.","key_machinery":"The carrying object is the Mamba selective state-space layer, a linear-time sequence model whose state update uses input-dependent matrices B, C, and time step delta to selectively compress history. In DG-Mamba and GDG-Mamba, this layer processes the per-node sequence of graph-snapshot features, optionally preprocessed by GINE convolutions that fold node and edge attributes into node features, followed by mean pooling, a linear-tanh projection, and two heads emitting Gaussian embedding means and variances. The Mamba layer replaces the transformer's pairwise attention so that temporal mixing cost is linear in lookback length, and the hidden-attention identity is used to interpret the learned state matrix as cumulative attention-like weights between timestamps.","core_discovery":"The central claim is that Mamba-based models achieve comparable or superior performance to transformer-based approaches in link prediction tasks while offering significant computational efficiency gains on longer sequences. On the paper's own terms, DG-Mamba and GDG-Mamba beat TransformerG2G and ST-TransformerG2G in MAP and MRR on UCI, Bitcoin-OTC, and Reality Mining, match them on the stable SBM benchmark, and fall short only on Slashdot, which has just 12 timestamps. The paper also claims that analysis of the learned state transition matrix A reveals an attention-like mechanism, so Mamba is not merely an efficient approximation but captures focused long-range temporal dependencies, and that measured FLOPs and MACs scale linearly with lookback while the transformer baselines grow quadratically.","pith_inferences":["If the hidden-attention view of Mamba is faithful, one could distill trained Mamba graph models into sparse or structured attention transformers that run fast on existing hardware, or vice versa.","The same architecture could be extended to continuous-time temporal graphs by replacing fixed snapshots with event-driven state updates, where the linear scan maps naturally to event sequences.","The reported efficiency suggests a testable extension: stress-test lookback length on large graphs to see where transformer memory fails and Mamba plateaus in accuracy.","The probabilistic Gaussian output already gives per-node uncertainty; a natural next step is to calibrate those uncertainties on future-edge prediction and use them for anomaly detection, not just ranking."],"forward_implications":["Dynamic graph embedding can use longer lookback windows without the quadratic blowup of attention, so models can attend to entire histories rather than a few recent snapshots.","Adding edge features through GINE convolutions helps most on transient datasets, suggesting that spatial edge information is key where links churn quickly.","On datasets with very few timestamps, the extra complexity of SSMs and GINE can hurt, so transformer with GCN remains the better choice for short histories.","The attention-like interpretation of the Mamba state matrix gives a way to inspect what temporal dependencies a trained model has learned, similar to attention weights.","Linear complexity makes these models candidates for streaming or online link prediction on long-running dynamic graphs."],"supporting_citations":[{"why":"Supplies the Mamba selective state-space architecture that DG-Mamba and GDG-Mamba are built on.","marker":"[25]"},{"why":"Provides the probabilistic Gaussian embedding framework and the TransformerG2G baseline that the new models extend and compare against.","marker":"[16]"},{"why":"Defines the attention mechanism and positional encoding reused in ST-TransformerG2G and the quadratic-complexity baseline.","marker":"[15]"},{"why":"Gives the structured SSM background and HiPPO initialization that Mamba extends.","marker":"[24]"},{"why":"Supplies the hidden-attention identity used to analyze the learned state matrix A.","marker":"[38]"},{"why":"Provides the triplet-based contrastive loss used to train all models.","marker":"[35]"},{"why":"Cited as the source of the GINE convolution used in GDG-Mamba for spatial edge-feature encoding.","marker":"[34]"}],"fun_headline_variants":["Mamba matches transformers on dynamic graphs at linear cost","Mamba outperforms transformers on volatile dynamic graph data","State-space model beats attention for dynamic graph embedding with linear scaling","Mamba's linear complexity matches transformer accuracy on temporal graphs"],"cache_read_input_tokens":21120,"weakest_assumption_plain":"The load-bearing premise is that the logistic regression link prediction classifier is trained only on edges from training timestamps, so that the reported MAP and MRR measure prediction of future graph snapshots rather than reconstruction of already-seen edges.","fun_headline_variants_meta":{"raw":{"variants":["Mamba matches transformers on dynamic graphs at linear cost","Mamba outperforms transformers on volatile dynamic graph data","State-space model beats attention for dynamic graph embedding with linear scaling","Mamba's linear complexity matches transformer accuracy on temporal graphs"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000585,"raw_usage":{"total_tokens":2760,"prompt_tokens":966,"completion_tokens":1794,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":582,"completion_tokens_details":{"reasoning_tokens":1728}},"tokens_in":582,"tokens_out":1794,"duration_ms":12659,"temperature":1.0,"reasoning_tokens":1728,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T15:04:47.666378+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"If the classifier were trained on node pairs sampled from test-snapshot edges, the reported numbers would not measure temporal generalization. A direct check would be to re-run the link prediction evaluation with a strict temporal split where the classifier sees no edges from the test interval, then compare MAP and MRR; if the performance drops substantially, the paper's central claim would be weakened.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the probabilistic Gaussian embedding framework and the TransformerG2G baseline that the new models extend and compare against."},{"cited_title":"Bojchevski, S","cited_arxiv_id":null,"evidence_quote":"Provides the triplet-based contrastive loss used to train all models."}],"review_version":1}