pith. sign in

arxiv: 2604.24078 · v1 · submitted 2026-04-27 · 💻 cs.LG

Explaining Temporal Graph Predictions With Shapley Values

Lea-Marie Sussek , Stefan Heindorf This is my paper

Pith reviewed 2026-05-08 04:18 UTC · model grok-4.3

classification 💻 cs.LG
keywords temporal graph neural networksShapley valuesmodel interpretabilityOwen valuesKernelSHAPexplanation methodsTGNN
0
0 comments X

The pith

Two Shapley-based explainers interpret how temporal graph neural networks combine events and features to make predictions.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper presents two new methods to explain the predictions of temporal graph neural networks in a model-agnostic way. The first uses KernelSHAP to score the contribution of each temporal event, like an edge in the graph at a specific time. The second breaks those scores down further into the roles of individual features within each event using Owen values. A reader would care because these networks are used for forecasting on time-evolving data, yet their internal logic has been opaque, risking undetected biases or errors.

Core claim

The authors introduce an event-level explainer that estimates Shapley values for individual temporal events in TGNNs via the KernelSHAP algorithm, and a feature-level explainer that decomposes these into Owen values to reveal feature dependencies. These methods outperform existing state-of-the-art explainers across metrics and datasets, and the feature explainer identifies a faulty timestamp extraction in a standard TGAT implementation, which explains performance issues with sparse explanations.

What carries the argument

The central mechanism is the application of Shapley values at the event level using KernelSHAP, extended by Owen value decomposition to attribute importance to features within temporal events, enabling hierarchical explanations of TGNN predictions.

If this is right

  • Any TGNN can be explained without retraining or internal access.
  • Explanations can diagnose implementation bugs in common libraries, such as incorrect timestamp handling.
  • Improved interpretability supports safer deployment in domains relying on temporal graph data.
  • The methods highlight why performance drops occur on very sparse explanations.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • This approach could be adapted to explain other sequence or time-series models that use graph structures.
  • Explanation techniques like this might be used proactively during model development to enforce correct use of temporal information.
  • Users of TGAT and similar models should verify their timestamp processing if relying on these explanations for trust.

Load-bearing premise

KernelSHAP approximations and Owen value decompositions accurately represent the true contributions of temporal events and features in TGNNs without major bias from the model-agnostic setting or intricate temporal interactions.

What would settle it

Compute exact Shapley values for a toy TGNN on a small temporal graph dataset using exhaustive enumeration, then check if the proposed KernelSHAP-based estimates match closely; large discrepancies would disprove the reliability of the approximations.

Figures

Figures reproduced from arXiv: 2604.24078 by Lea-Marie Sussek, Stefan Heindorf.

Figure 1
Figure 1. Figure 1: Comparison of the explanations generated by the different explainers. The connection between view at source ↗
Figure 3
Figure 3. Figure 3: Complete Sparsity-vs-Fidelity curve on the generated view at source ↗
Figure 4
Figure 4. Figure 4: Example of a graph structure within the artificial dataset. view at source ↗
read the original abstract

Temporal Graph Neural Networks (TGNNs) have become increasingly popular in recent years due to their superior predictive performance by combining both spatial and temporal information. However, how these models utilize the information to make predictions is rather unexplored, leading to potentially faulty or biased models. This work introduces two novel model-agnostic explainers for local explanations of TGNNs based on Shapley and Owen values. The first method, an event-level (edge-level) Shapley explainer, applies the KernelSHAP algorithm to estimate contribution scores for individual temporal events, providing interpretable descriptions for model behavior. The second, a feature-level Shapley explainer, extends this framework by decomposing event-level Shapley values into Owen values, and thereby uncovers hierarchical dependencies of the event and its features. The explainers outperform SOTA explainers on different metrics and datasets. Additionally, the Feature Explainer reveals a faulty extraction of actual timestamps of a commonly used TGAT implementation, helping to further understand performance drops on very sparse explanations.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

2 major / 2 minor

Summary. The manuscript introduces two model-agnostic explainers for Temporal Graph Neural Networks (TGNNs) based on Shapley and Owen values. The first is an event-level (edge-level) explainer that applies the KernelSHAP algorithm to estimate contribution scores for individual temporal events. The second is a feature-level explainer that decomposes the event-level Shapley values into Owen values to uncover hierarchical dependencies between events and their features. The authors claim that these explainers outperform existing state-of-the-art methods on multiple metrics and datasets, and that the feature explainer additionally reveals a faulty timestamp extraction bug in a commonly used TGAT implementation.

Significance. If the approximations are shown to be faithful, the work would provide a useful addition to the interpretability toolkit for TGNNs, which are increasingly deployed in domains requiring both spatial and temporal reasoning. The hierarchical decomposition via Owen values is a natural extension that could help practitioners debug models and understand performance drops on sparse explanations. The reported discovery of a timestamp extraction issue in TGAT illustrates how explanation methods can surface implementation problems.

major comments (2)
  1. [Experiments (likely §4–5) and Method (KernelSHAP / Owen decomposition description)] The central empirical claims (outperformance over SOTA and the TGAT timestamp bug diagnosis) rest on the fidelity of the KernelSHAP-based event contributions and their Owen decomposition. No section reports a controlled validation comparing the approximate scores against exact Shapley values computed by enumeration on synthetic temporal graphs with known ground-truth contributions. Without such a check, it remains possible that the reported superiority and the bug diagnosis are influenced by sampling bias arising from temporal ordering and message-passing dependencies.
  2. [Method and Experiments sections] The weakest assumption—that KernelSHAP coalitions sampled from a background distribution faithfully capture non-additive temporal interactions—is not stress-tested. The manuscript should include at least one synthetic experiment where exact vs. approximate values can be compared directly, especially for the sparse-explanation regime highlighted in the abstract.
minor comments (2)
  1. [Abstract] The abstract refers to outperformance on “different metrics and datasets” without naming them; the introduction or results section should list the concrete metrics (e.g., fidelity, sparsity) and datasets used.
  2. [Throughout] Notation for temporal events, features, and the background distribution used in KernelSHAP should be introduced once and used consistently to aid readability.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for their insightful comments, which help improve the rigor of our work. We address the concerns about validating the approximations of our explainers point by point below. We agree that additional synthetic validation experiments would enhance the manuscript and plan to incorporate them in the revised version.

read point-by-point responses

  1. Referee: [Experiments (likely §4–5) and Method (KernelSHAP / Owen decomposition description)] The central empirical claims (outperformance over SOTA and the TGAT timestamp bug diagnosis) rest on the fidelity of the KernelSHAP-based event contributions and their Owen decomposition. No section reports a controlled validation comparing the approximate scores against exact Shapley values computed by enumeration on synthetic temporal graphs with known ground-truth contributions. Without such a check, it remains possible that the reported superiority and the bug diagnosis are influenced by sampling bias arising from temporal ordering and message-passing dependencies.

    Authors: We acknowledge the importance of verifying the fidelity of our KernelSHAP approximations against exact Shapley values. Computing exact values is computationally intractable for graphs with more than a small number of events due to the 2^n complexity. However, to address this, we will add a controlled experiment on small synthetic temporal graphs where exact enumeration is feasible. These graphs will have known ground-truth contributions, allowing direct comparison of our approximate event-level Shapley values and the Owen decomposition to the exact ones. This will specifically test for any sampling bias from temporal dependencies and include the sparse-explanation setting. We expect this to confirm the reliability of our reported results and the bug diagnosis. revision: yes

  2. Referee: [Method and Experiments sections] The weakest assumption—that KernelSHAP coalitions sampled from a background distribution faithfully capture non-additive temporal interactions—is not stress-tested. The manuscript should include at least one synthetic experiment where exact vs. approximate values can be compared directly, especially for the sparse-explanation regime highlighted in the abstract.

    Authors: We agree that stress-testing the core assumption of KernelSHAP in the context of temporal interactions is necessary. In the revision, we will include at least one synthetic experiment on small temporal graphs enabling exact Shapley computation. This experiment will compare exact and approximate values, focusing on non-additive interactions and the sparse regime. Such validation will demonstrate that the sampled coalitions effectively capture the relevant dependencies, thereby supporting the outperformance claims and the practical utility of the feature-level explainer. revision: yes

Circularity Check

0 steps flagged

No circularity: standard Shapley application to new domain

full rationale

The paper defines event-level and feature-level explainers by directly applying the existing KernelSHAP algorithm and Owen-value decomposition to TGNN predictions. No equation or claim reduces a result to its own fitted inputs or self-citations; performance metrics and the TGAT timestamp observation are empirical outcomes from running the off-the-shelf methods on public datasets. The derivation chain consists of standard references to Shapley (1953), Owen (1977), and Lundberg et al. (2017) without load-bearing self-citation or ansatz smuggling. The central claims therefore remain independent of the paper's own outputs.

Axiom & Free-Parameter Ledger

0 free parameters · 0 axioms · 0 invented entities

Abstract-only review provides no explicit free parameters, axioms, or invented entities; the approach relies on standard Shapley value theory and KernelSHAP approximation without additional postulates described.

pith-pipeline@v0.9.0 · 5467 in / 1158 out tokens · 39936 ms · 2026-05-08T04:18:12.001679+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

26 extracted references · 1 canonical work pages · 1 internal anchor

  1. [1]

    Evaluating explainability for graph neural networks.Scientific Data, 10(1):144,

    [Agarwalet al., 2023 ] Chirag Agarwal, Owen Queen, Himabindu Lakkaraju, and Marinka Zitnik. Evaluating explainability for graph neural networks.Scientific Data, 10(1):144,

    2023

  2. [2]

    Gnnshap: Scalable and accurate GNN explanation using shapley values

    [Akkas and Azad, 2024] Selahattin Akkas and Ariful Azad. Gnnshap: Scalable and accurate GNN explanation using shapley values. InWWW, pages 827–838. ACM,

    2024

  3. [3]

    Tempme: Towards the explainability of temporal graph neural networks via motif discovery

    [Chen and Ying, 2023] Jialin Chen and Rex Ying. Tempme: Towards the explainability of temporal graph neural networks via motif discovery. InNeurIPS,

    2023

  4. [4]

    Malliaros

    [Duval and Malliaros, 2021] Alexandre Duval and Fragkiskos D. Malliaros. Graphsvx: Shapley value explanations for graph neural networks. InECML/PKDD, pages 302–318. Springer,

    2021

  5. [5]

    [re] reproducibility study of ”explaining temporal graph models through an explorer-navigator framework”.Trans

    [Ghasemiet al., 2024 ] Helia Ghasemi, Christina Isaicu, Jesse Wonnink, and Andreas Berentzen. [re] reproducibility study of ”explaining temporal graph models through an explorer-navigator framework”.Trans. Mach. Learn. Res., 2024,

    2024

  6. [6]

    Vu, Zhe Jiang, and My T

    [Heet al., 2022 ] Wenchong He, Minh N. Vu, Zhe Jiang, and My T. Thai. An explainer for temporal graph neural networks. InGLOBECOM, pages 6384–6389. IEEE,

    2022

  7. [7]

    Representation learning for dynamic graphs: A survey.J

    [Kazemiet al., 2020 ] Seyed Mehran Kazemi, Rishab Goel, Kshitij Jain, Ivan Kobyzev, Akshay Sethi, Peter Forsyth, and Pascal Poupart. Representation learning for dynamic graphs: A survey.J. Mach. Learn. Res., 21:70:1–70:73,

    2020

  8. [8]

    Lundberg and Su-In Lee

    [Lundberg and Lee, 2017] Scott M. Lundberg and Su-In Lee. A unified approach to interpreting model predictions. In NIPS, pages 4765–4774,

    2017

  9. [9]

    CARE: modeling interacting dynamics under temporal environmental variation

    [Luoet al., 2023 ] Xiao Luo, Haixin Wang, Zijie Huang, Huiyu Jiang, Abhijeet Gangan, Song Jiang, and Yizhou Sun. CARE: modeling interacting dynamics under temporal environmental variation. InNeurIPS,

    2023

  10. [10]

    2 edition,

    [Molnar, 2022] Christoph Molnar.Interpretable Machine Learning. 2 edition,

    2022

  11. [11]

    Values of games with a priori unions

    [Owen, 1977] Guilliermo Owen. Values of games with a priori unions. InMathematical economics and game theory: Essays in honor of Oskar Morgenstern, pages 76–88. Springer,

    1977

  12. [12]

    Academic Press, San Diego, 3rd edition,

    [Owen, 1995] Guillermo Owen.Game Theory. Academic Press, San Diego, 3rd edition,

    1995

  13. [13]

    Linguistic inquiry and word count: Liwc 2001.Mahway: Lawrence Erlbaum Associates, 71(2001):2001,

    [Pennebakeret al., 2001 ] James W Pennebaker, Martha E Francis, and Roger J Booth. Linguistic inquiry and word count: Liwc 2001.Mahway: Lawrence Erlbaum Associates, 71(2001):2001,

    2001

  14. [14]

    Towards better evaluation for dynamic link prediction

    [Poursafaeiet al., 2022 ] Farimah Poursafaei, Shenyang Huang, Kellin Pelrine, and Reihaneh Rabbany. Towards better evaluation for dynamic link prediction. InNeurIPS,

    2022

  15. [15]

    Cody: Counterfactual explainers for dynamic graphs

    [Quet al., 2025 ] Zhan Qu, Daniel Gomm, and Michael F¨arber. Cody: Counterfactual explainers for dynamic graphs. InICML. OpenReview.net,

    2025

  16. [16]

    Temporal Graph Networks for Deep Learning on Dynamic Graphs

    [Rossiet al., 2020 ] Emanuele Rossi, Ben Chamberlain, Fabrizio Frasca, Davide Eynard, Federico Monti, and Michael M. Bronstein. Temporal graph networks for deep learning on dynamic graphs.CoRR, abs/2006.10637,

    work page internal anchor Pith review arXiv 2020

  17. [17]

    A value for n-person games

    [Shapley, 1953] Lloyd S Shapley. A value for n-person games. InContributions to the Theory of Games II, pages 307–317

    1953

  18. [18]

    Traffic prediction in optical networks using graph convolutional generative adversarial networks

    [Vinchoffet al., 2020 ] Connor Vinchoff, Nathan Chung, Tyler Gordon, Liam Lyford, and Michal Aibin. Traffic prediction in optical networks using graph convolutional generative adversarial networks. InICTON, pages 1–4. IEEE,

    2020

  19. [19]

    Vu and My T

    [Vu and Thai, 2020] Minh N. Vu and My T. Thai. Pgm-explainer: Probabilistic graphical model explanations for graph neural networks. InNeurIPS,

    2020

  20. [20]

    Link prediction in social networks: the state-of-the-art.Sci

    [Wanget al., 2015 ] Peng Wang, Baowen Xu, Yurong Wu, and Xiaoyu Zhou. Link prediction in social networks: the state-of-the-art.Sci. China Inf. Sci., 58(1):1–38,

    2015

  21. [21]

    Inductive representation learning in temporal networks via causal anonymous walks

    [Wanget al., 2021 ] Yanbang Wang, Yen-Yu Chang, Yunyu Liu, Jure Leskovec, and Pan Li. Inductive representation learning in temporal networks via causal anonymous walks. InICLR. OpenReview.net,

    2021

  22. [22]

    Explaining temporal graph models through an explorer-navigator framework

    [Xiaet al., 2023 ] Wenwen Xia, Mincai Lai, Caihua Shan, Yao Zhang, Xinnan Dai, Xiang Li, and Dongsheng Li. Explaining temporal graph models through an explorer-navigator framework. InICLR. OpenReview.net,

    2023

  23. [23]

    Inductive representation learning on temporal graphs

    [Xuet al., 2020 ] Da Xu, Chuanwei Ruan, Evren K¨orpeoglu, Sushant Kumar, and Kannan Achan. Inductive representation learning on temporal graphs. InICLR. OpenReview.net,

    2020

  24. [24]

    Gnnexplainer: Generating explanations for graph neural networks

    [Yinget al., 2019 ] Zhitao Ying, Dylan Bourgeois, Jiaxuan You, Marinka Zitnik, and Jure Leskovec. Gnnexplainer: Generating explanations for graph neural networks. In NeurIPS, pages 9240–9251,

    2019

  25. [25]

    Towards better dynamic graph learning: New architecture and unified library

    [Yuet al., 2023 ] Le Yu, Leilei Sun, Bowen Du, and Weifeng Lv. Towards better dynamic graph learning: New architecture and unified library. InNeurIPS,

    2023

  26. [26]

    On explainability of graph neural networks via subgraph explorations

    [Yuanet al., 2021 ] Hao Yuan, Haiyang Yu, Jie Wang, Kang Li, and Shuiwang Ji. On explainability of graph neural networks via subgraph explorations. InICML, volume 139 ofProceedings of Machine Learning Research, pages 12241–12252. PMLR, 2021

    2021