REVIEW 4 major objections 5 minor 28 references
Are GNNs Actually Effective for Multimodal Fault Diagnosis in Microservice Systems?
T0 review · 4 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash
Pith's one-line read Plain MLP matches or beats GNNs in microservice fault diagnosis
desk verdict Good question, sensible baseline, but the ablation isn't controlled enough to back the headline claim. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is DiagMLP, an ablation baseline defined by two single-layer MLPs joined by concatenation. It takes per-node multimodal embeddings for metrics, logs, and traces, appends a learnable position embedding per node, fuses the modalities node-by-node, then concatenates all nodes and fuses them into a system-level representation for detection, localization, and classification. Its role is to replace a GNN module inside an existing pipeline with a topology-agnostic module that retains all multimodal fusion, so any performance difference can be attributed to graph modeling alone. Learnable position embeddings give nodes distinct identities without an explicit dependency graph, and the single-layer design keeps the baseline deliberately minimal, so parity with GNNs is evidence against the value of topology.
What would settle it
Re-run the original GNN frameworks (Eadro, TVDiag, DeepHunt) under the paper's corrected protocol—fixed window splitting and an added validation set—and compare against DiagMLP on the same five datasets; if any baseline then beats DiagMLP by a clear margin, the parity result is an artifact of protocol fixes rather than evidence against graph modeling. A complementary test on a synthetic dataset with faults injected along known dependency paths would show whether topology ever matters.
Extended reading notes
Core claim
On the paper's own terms, the discovery is that graph structure is not the source of performance in current GNN-based fault diagnosis pipelines. DiagMLP is inserted into Eadro, TVDiag, and DeepHunt in place of their GNN modules—graph attention, GraphSAGE, and graph autoencoder backbones—while every other component stays fixed. The resulting model performs at parity or better: fault detection F1 of 96.7 percent versus 92.1 percent on one dataset and 90.8 percent versus 90.7 percent on another; classification within one standard deviation on a third; and localization Top-1 accuracy that is substantially higher on the two largest comparisons. UMAP visualizations show that preprocessed multimodal features already separate fault classes by root cause before any graph modeling, and that neither MLP nor GNN embeddings add much separation. The authors conclude that preprocessing pipelines already encode critical dependency information, making explicit graph modeling at best redundant and at worst a source of instability.
Load-bearing premise
The conclusion rests on the fairness of the ablation: DiagMLP and each GNN baseline must differ only in the graph module, with identical preprocessing, embeddings, data splits, and evaluation, so that observed parity reflects the redundancy of graph modeling rather than protocol fixes.
Editorial extensions
If this is right
- If the central claim is right, reported accuracy gains in recent GNN-based fault diagnosis papers should be re-assigned to preprocessing and multimodal fusion, not to graph architecture.
- Simple MLP baselines like DiagMLP should be added to every future fault-diagnosis evaluation before a GNN is claimed to help.
- Standardized evaluation protocols—including fixed window splitting, validation sets, and re-running baselines under identical conditions—are needed to prevent preprocessing artifacts from masquerading as model gains.
- Larger datasets with realistic dependency-driven fault propagation are needed to test whether any architecture can exploit topology; current datasets with dozens of service instances may simply be too small for graphs to matter.
Reading between the lines
- A direct testable extension would be to construct synthetic microservice data where faults are injected along known dependency edges; if DiagMLP still matches GNNs there, the case for graph modeling shrinks further, and if it fails there, that defines the regime where GNNs earn their complexity.
- The parity result, if it generalizes, suggests that many multimodal deep-learning pipelines may be carrying their performance in feature preprocessing and fusion, a hypothesis worth probing in adjacent domains such as cloud infrastructure monitoring and network failure diagnosis.
- The learnable position embeddings are a quiet part of the result: they give the MLP a way to distinguish nodes without topology, so a follow-up could test whether those embeddings implicitly recover service-dependency structure by comparing them to the true dependency graph.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes DiagMLP, a deliberately topology-agnostic MLP baseline for multimodal fault diagnosis in microservice systems, and evaluates whether GNN modules in existing frameworks add value beyond preprocessing and multimodal fusion. The authors replace the GNN components in Eadro, TVDiag, and DeepHunt with DiagMLP while purportedly keeping all other pipeline elements, and compare performance on five datasets (SN, TT, GAIA, D1, D2) across fault detection, localization, and classification. They report that DiagMLP matches or exceeds the GNN-based baselines, concluding that graph structures contribute marginally to diagnostic performance. UMAP visualizations are used both as motivation and as supporting evidence.
Significance. If substantiated, this is a valuable negative result for the microservice fault diagnosis community. It challenges the prevailing assumption that explicit service-dependency graphs are necessary, and it advocates for standardized baselines and protocols. The paper is well-motivated and the DiagMLP design is admirably simple. However, the current experimental evidence does not support the central claim because the ablation is not fully controlled: protocol changes (window-splitting correction, validation set, reimplemented preprocessing) are conflated with the removal of the GNN module. The significance of the paper depends on whether the authors can provide a rigorous controlled comparison.
major comments (4)
- [Section IV.A, Notes (3) and Table IV footnote] The central ablation is not controlled. The paper states that it 'rectified issues in prior frameworks, such as the window-splitting method in Eadro, which risked data leakage, and incorporated a validation set,' while Table IV's footnote says that results without standard deviations are 'directly extracted from the cited papers.' This means several baseline numbers (e.g., DiagFusion, CHASE, and possibly parts of DeepHunt and TVDiag) were produced under the original, potentially flawed protocol, whereas DiagMLP was evaluated under the corrected protocol. The large SN Top-1 localization gap (80.2±3.1% vs. 41.8±14.5%) is exactly the kind of difference that fixing a data-leaking window split could produce. To support the claim that GNN modules contribute marginally, all baselines must be re-run under the identical corrected protocol (same window splitting, same validation set) or the claims must be explicitly restricted to the re-run baselines.
- [Section IV.A(b)] For SN and TT, the paper says it 'reimplemented preprocessing code due to missing original scripts.' This implies the input features for DiagMLP and for the Eadro baseline may differ from those in the original Eadro paper. If the Eadro results in Tables II and IV are taken from the original publication, then the comparison conflates preprocessing differences with architecture differences, contradicting the paper's statement in Section IV that 'we preserve all other pipeline elements.' The authors should either demonstrate that their reimplemented preprocessing reproduces the original Eadro results (a reproducibility check) or re-run Eadro using the reimplemented preprocessing under the corrected protocol.
- [Tables II and III] The detection and classification results are reported without standard deviations, yet the text in Section IV.B claims 'differences within standard deviations' for GAIA classification and treats the SN/TT detection improvements as meaningful. A parity claim is a negative result and requires statistical evidence that the observed differences are not significant. Please provide standard deviations, confidence intervals, or significance tests for all key comparisons, especially the detection results in Table II where DiagMLP appears to outperform Eadro by a large margin (F1 96.7 vs. 92.1 on SN).
- [Section III.B and Section IV.C] The UMAP visualizations in Figs. 3 and 5 are used both to motivate the DiagMLP design and to explain the observed parity (see 'Key Insight 2'). This is circular: the same plots that justify the baseline are offered as evidence that GNNs do not improve separability. Moreover, visual inspection is not a quantitative evaluation. Please report objective separability metrics (e.g., silhouette score, k-nearest-neighbor accuracy) computed on the preprocessed input features, MLP embeddings, and GNN embeddings to support the claim that preprocessing already encodes dependency information.
minor comments (5)
- [Section I] In the sentence 'we introduce DiagMLP, a ablation experimental baseline,' the article should be 'an ablation experimental baseline.'
- [Section III.B] The sentence 'we project Xt from from R3dN to R2' contains a duplicated 'from.'
- [References] Reference [13] has a typo: 'F ound.' should be 'Found.'
- [Table II] The detection results in Table II lack standard deviations; if available from the re-run experiments, please include them to allow readers to assess variability.
- [Figure 5] The three panels in Figure 5 are not labeled with (a), (b), (c) in the caption or on the figures themselves; please add labels for clarity.
Circularity Check
No circular derivation; the parity claim is an empirical ablation result, and the protocol concerns are experimental confounds rather than circularity.
full rationale
The paper's central claim, that DiagMLP matches GNN-based performance, is an empirical claim established by running a new model on published datasets. No equation in the paper defines DiagMLP's output in terms of the GNN baselines' outputs; Eq. (2) and Eq. (3) simply define an MLP. There is no fitted parameter that is then renamed as a prediction, no uniqueness theorem imported from the authors' prior work, and no ansatz smuggled in via citation. The UMAP visualizations (Fig. 3 and Fig. 5) are used both to motivate and to explain the result, but they are exploratory evidence rather than a derivation, so this is consistency, not circularity. The paper does cite prior work by overlapping authors, but none of these citations carries the central argument; the parity claim is supported by the experimental tables. The concern raised in the review about changed preprocessing and validation protocol (Section IV.A note (3) and the Table IV footnote) is a threat to the validity of the comparison, not a circularity: it concerns whether GNN baselines were evaluated under the same protocol, not whether the conclusion is presupposed by its inputs. Therefore no circular step can be identified under the required standard.
Assumptions & free parameters
free parameters (2)
- Per-node learnable position embedding p_i =
learned during training; dimension equals embedding dimension d
- Fusion MLP hidden dimension, dropout rate, and training hyperparameters =
not reported in the paper
assumptions (3)
- domain assumption The published baseline numbers for Eadro, TVDiag, and DeepHunt are directly comparable to the re-run DiagMLP results despite protocol changes (window splitting, validation set).
- domain assumption UMAP visualizations are a valid indicator of class separability and hence of whether topology modeling adds information.
- domain assumption The five datasets (SN, TT, GAIA, D1, D2) are representative enough to support a general claim about GNN utility in microservice fault diagnosis.
Cite this review
Pith. "Pith review of Are GNNs Actually Effective for Multimodal Fault Diagnosis in Microservice Systems?." pith.science (2026). https://pith.science/paper/2IT7EJTS
@misc{pith2026250102766,
author = {Pith},
title = {Pith review of: Are GNNs Actually Effective for Multimodal Fault Diagnosis in Microservice Systems?},
year = {2026},
howpublished = {\url{https://pith.science/paper/2IT7EJTS}},
note = {Machine review of arXiv:2501.02766}
}
read the original abstract
Graph Neural Networks (GNNs) are widely adopted for fault diagnosis in microservice systems, premised on their ability to model service dependencies. However, the necessity of explicit graph structures remains underexamined, as existing evaluations conflate preprocessing with architectural contributions. To isolate the true value of GNNs, we propose DiagMLP, a deliberately minimal, topology-agnostic baseline that retains multimodal fusion capabilities while excluding graph modeling. Through ablation experiments across five datasets, DiagMLP achieves performance parity with state-of-the-art GNN-based methods in fault detection, localization, and classification. These findings challenge the prevailing assumption that graph structures are indispensable, revealing that: (i) preprocessing pipelines already encode critical dependency information, and (ii) GNN modules contribute marginally beyond multimodality fusion. Our work advocates for systematic re-evaluation of architectural complexity and highlights the need for standardized baseline protocols to validate model innovations.
Figures
Figures from the paper (1 more)
Reference graph
Works this paper leans on
-
[1]
R. Xin, P. Chen, and Z. Zhao, ‘Causalrca: Causal inference based pre- cise fine-grained root cause localization for microservice applications’, Journal of Systems and Software , vol. 203, p. 111724, 2023
work page 2023
-
[2]
Eadro: An End-to-End Troubleshooting Framework for Microservices on Multi-Source Data,
C. Lee, T. Yang, Z. Chen, Y . Su, and M. R. Lyu, "Eadro: An End-to-End Troubleshooting Framework for Microservices on Multi-Source Data," in Proc. 45th Int. Conf. Softw. Eng. , 2023, pp. 1750–1762
work page 2023
-
[3]
Interpretable Failure Localization for Microservice Sys- tems Based on Graph Autoencoder,
Y . Sun et al., "Interpretable Failure Localization for Microservice Sys- tems Based on Graph Autoencoder," ACM Trans. Softw. Eng. Methodol., 2024
work page 2024
-
[4]
Fault-Aware Service Scheduling Optimization Frame- work in Edge Data Center,
Y . Zhang et al., "Fault-Aware Service Scheduling Optimization Frame- work in Edge Data Center," in Proc. 2024 IEEE 21st Int. Conf. Mobile Ad-Hoc Smart Syst. (MASS) , 2024, pp. 514–519
work page 2024
-
[5]
TVDiag: A Task-oriented and View-invariant Fail- ure Diagnosis Framework with Multimodal Data,
S. Xie et al., "TVDiag: A Task-oriented and View-invariant Fail- ure Diagnosis Framework with Multimodal Data," arXiv preprint arXiv:2407.19711, 2024
arXiv 2024
-
[6]
Robust Failure Diagnosis of Microservice System Through Multimodal Data,
S. Zhang et al., "Robust Failure Diagnosis of Microservice System Through Multimodal Data," IEEE Trans. Serv. Comput.(TSC) , vol. 16, no. 6, pp. 3851–3864, 2023
work page 2023
-
[7]
Z. Zhao et al., "CHASE: A Causal Heterogeneous Graph based Frame- work for Root Cause Analysis in Multimodal Microservice Systems," arXiv preprint arXiv:2406.19711 , 2024
arXiv 2024
-
[8]
Graph neural networks: A review of methods and applications,
J. Zhou et al., "Graph neural networks: A review of methods and applications," AI Open , vol. 1, pp. 57–81, 2020
work page 2020
Show all 28 references
-
[9]
DeepTraLog: Trace-Log Combined Microservice Anomaly Detection through Graph-based Deep Learning,
C. Zhang et al., "DeepTraLog: Trace-Log Combined Microservice Anomaly Detection through Graph-based Deep Learning," in Proc. 2022 IEEE/ACM 44th Int. Conf. Softw. Eng. (ICSE) , 2022, pp. 623–634
2022
-
[10]
Twin Graph-Based Anomaly Detection via Attentive Multi-Modal Learning for Microser- vice System,
J. Huang, Y . Yang, H. Yu, J. Li, and X. Zheng, "Twin Graph-Based Anomaly Detection via Attentive Multi-Modal Learning for Microser- vice System," in Proc. 2023 38th IEEE/ACM Int. Conf. Autom. Softw. Eng. (ASE) , 2023, pp. 66–78
2023
-
[11]
Drain: An Online Log Parsing Approach with Fixed Depth Tree,
P. He, J. Zhu, Z. Zheng, and M. R. Lyu, "Drain: An Online Log Parsing Approach with Fixed Depth Tree," in Proc. 2017 IEEE Int. Conf. Web Services (ICWS) , 2017, pp. 33–40
2017
-
[12]
MULAN: Multi-modal Causal Structure Learning and Root Cause Analysis for Microservice Systems,
L. Zheng, Z. Chen, J. He, and H. Chen, "MULAN: Multi-modal Causal Structure Learning and Root Cause Analysis for Microservice Systems," in Proc. ACM Web Conf. 2024 , 2024, pp. 4107–4116
2024
-
[13]
Nezha: Interpretable Fine-Grained Root Causes Analysis for Microservices on Multi-modal Observability Data,
G. Yu, P. Chen, Y . Li, H. Chen, X. Li, and Z. Zheng, "Nezha: Interpretable Fine-Grained Root Causes Analysis for Microservices on Multi-modal Observability Data," in Proc. 31st ACM Joint Eur . Softw. Eng. Conf. Symp. F ound. Softw. Eng. , 2023, pp. 553–565
2023
-
[14]
An empirical evaluation of generic convolutional and recurrent networks for sequence modeling,
S. Bai, J. Z. Kolter, and V . Koltun, "An empirical evaluation of generic convolutional and recurrent networks for sequence modeling," arXiv preprint arXiv:1803.01271, 2018
2018 arXiv
-
[15]
Attention is All you Need,
A. Vaswani et al., "Attention is All you Need," in Proc. Neural Inf. Process. Syst. (NeurIPS) , 2017
2017
-
[16]
Enriching Word Vectors with Subword Information,
P. Bojanowski, E. Grave, A. Joulin, and T. Mikolov, "Enriching Word Vectors with Subword Information," Trans. Assoc. Comput. Linguistics , vol. 5, pp. 135–146, 2017
2017
-
[17]
Glove: Global vectors for word representation,
J. Pennington, R. Socher, and C. D. Manning, "Glove: Global vectors for word representation," in Proc. 2014 Conf. Empir . Methods Nat. Lang. Process. (EMNLP), 2014, pp. 1532–1543
2014
-
[18]
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding,
J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, "BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding," in Proc. 2019 Conf. North Amer . Chapter Assoc. Comput. Linguistics: Human Lang. Technol. , V ol. 1, 2019, pp. 4171–4186
2019
-
[19]
DGERCL: A Dynamic Graph Embedding Approach for Root Cause Localization in Microser- vice Systems,
H. Cheng, Q. Li, B. Liu, S. Liu, and L. Pan, "DGERCL: A Dynamic Graph Embedding Approach for Root Cause Localization in Microser- vice Systems," IEEE Trans. Serv. Comput. (TSC) , vol. 17, no. 1, pp. 1–12, 2024
2024
-
[20]
Deep sets,
M. Zaheer, S. Kottur, S. Ravanbakhsh, B. Poczos, R. R. Salakhutdinov, and A. J. Smola, "Deep sets," Adv. Neural Inf. Process. Syst. (NeurIPS) , vol. 30, 2017
2017
-
[21]
Set transformer: A framework for attention-based permutation-invariant neu- ral networks,
J. Lee, Y . Lee, J. Kim, A. Kosiorek, S. Choi, and Y . W. Teh, "Set transformer: A framework for attention-based permutation-invariant neu- ral networks," in Proc. Int. Conf. Mach. Learn. (ICML) , 2019, pp. 3744–3753
2019
-
[22]
Characterizing Microservice Dependency and Perfor- mance: Alibaba Trace Analysis,
S. Luo et al., "Characterizing Microservice Dependency and Perfor- mance: Alibaba Trace Analysis," in Proc. ACM Symp. Cloud Comput. , 2021, pp. 412–426
2021
-
[23]
CloudRCA: A Root Cause Analysis Framework for Cloud Computing Platforms,
Y . Zhang et al., "CloudRCA: A Root Cause Analysis Framework for Cloud Computing Platforms," in Proc. 30th ACM Int. Conf. Inf. Knowl. Manage. (CIKM), 2021, pp. 4373–4382
2021
-
[24]
Failure Diagnosis in Microservice Systems: A Compre- hensive Survey and Analysis,
S. Zhang et al., "Failure Diagnosis in Microservice Systems: A Compre- hensive Survey and Analysis," arXiv preprint arXiv:2407.01710 , 2024
2024 arXiv
-
[25]
Graph Neural Networks with Learnable Structural and Positional Representa- tions,
V . P. Dwivedi, A. T. Luu, T. Laurent, Y . Bengio, and X. Bresson, "Graph Neural Networks with Learnable Structural and Positional Representa- tions," in Proc. Int. Conf. Learn. Represent. (ICLR) , 2022
2022
-
[26]
Inductive Representation Learning on Large Graphs,
W. Hamilton, Z. Ying, and J. Leskovec, "Inductive Representation Learning on Large Graphs," in Adv. Neural Inf. Process. Syst. (NeurIPS) , vol. 30, 2017
2017
-
[27]
Graph Attention Networks,
P. Veli ˇckovi´c, G. Cucurull, A. Casanova, A. Romero, P. Liò, and Y . Bengio, "Graph Attention Networks," in Proc. Int. Conf. Learn. Represent. (ICLR), 2018
2018
-
[28]
UMAP: Uniform Manifold Approximation and Projection,
L. McInnes, J. Healy, N. Saul, and L. Großberger, "UMAP: Uniform Manifold Approximation and Projection," J. Open Source Softw. , vol. 3, no. 29, p. 861, 2018
2018
Reviewed August 10, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.