{"id":"575bac3e-c693-4f03-bed3-27d29a272af9","arxiv_id":"2501.02766","paper_version":2,"verdict":"REJECT","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":2,"one_line_summary":"Replacing GNN components with a topology-agnostic MLP maintains or improves performance across five microservice fault diagnosis datasets.","lead":"This paper proposes DiagMLP, a simple MLP that replaces graph neural network modules in three microservice fault diagnosis systems, and reports that it matches or beats them on five datasets. If the comparison holds, graph structure is not what makes these systems work; preprocessing and multimodal fusion carry most of the signal.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The ablation is uncontrolled: DiagMLP is compared against GNN baselines extracted from the original papers while the authors changed preprocessing and evaluation protocol (new window splitting, added validation set), so the observed parity may be an artifact of protocol fixes rather than evidence…","rationale":"I read the paper's stated goal as isolating the contribution of GNN modules to microservice fault diagnosis. DiagMLP is a sensible minimal baseline, and the UMAP-driven motivating analysis is supportive but not decisive. The load-bearing condition for the paper's conclusion is that the GNN-based and MLP-based pipelines differ only in the modeling module. That condition is not met, and the manuscript itself flags the protocol changes: reimplemented preprocessing for SN/TT, a corrected window-splitting method for Eadro, and an added validation set. Because several baseline numbers in Table IV are directly extracted from the original papers, the comparison conflates architectural choice with evaluation-protocol changes. The observed large localization improvements of DiagMLP over Eadro are precisely the kind of gains one would expect from removing data leakage, so the reported parity does not establish that GNNs contribute marginally. This is an empirical, checkable concern, not a matter of philosophical disagreement with the GNN-skeptic position. I therefore agree with the reader's REJECT verdict: the evidence as submitted does not support the strong claim in the Abstract. A revised version that reruns every baseline under one identical protocol, releases code and hyperparameters, and softens the generalization claims could change this assessment, but as presented the central argument has a real soft spot.","tokens_in":8530,"tokens_out":3940,"duration_ms":38779,"concrete_test":"Re-run the Eadro GAT pipeline under the exact protocol used for DiagMLP, using the same reimplemented SN/TT preprocessing, the same corrected window splitting, and the same validation set for early stopping and model selection, then report SN/TT localization Top-1/3/5 over at least 10 seeds. If corrected Eadro's SN Top-1 rises from 41.8% toward DiagMLP's 80% range, the central claim collapses; if it stays near 41.8%, the MLP parity result is robust. Additionally, release the code and hyperparameters for all reruns so protocol identity can be verified.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that 'GNN modules contribute marginally beyond multimodality fusion' (Abstract), supported by DiagMLP parity with state-of-the-art GNN methods. This requires an ablation in which the GNN module is the only component changed. The paper does not deliver that. Section IV.A states that for SN and TT the authors 'reimplemented preprocessing code due to missing original scripts,' and note (3) says they 'rectified issues in prior frameworks, such as the window-splitting method in Eadro,' which 'risked data leakage, and incorporated a validation set.' Table IV's footnote says results without standard deviations are 'directly extracted from the cited papers'; only part of Table IV was rerun. Thus the DiagMLP numbers are produced under a different data split, preprocessing, and model-selection protocol than several baselines. The massive SN Top-1 localization gap (80.2 +/- 3.1 vs 41.8 +/- 14.5) is exactly the kind of difference that correcting a data-leaking window-splitting procedure would produce. If Eadro's original numbers come from the flawed split, the comparison does not isolate graph modeling; it compares a cleanly evaluated MLP to a leaky GNN. The parity claim for detection/classification is weaker but still not controlled. The paper's motivating UMAP and the observation that preprocessed features are separable are legitimate evidence, but they do not establish that GNNs add nothing when evaluated fairly.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes DiagMLP, a deliberately topology-agnostic MLP baseline for multimodal fault diagnosis in microservice systems, and evaluates whether GNN modules in existing frameworks add value beyond preprocessing and multimodal fusion. The authors replace the GNN components in Eadro, TVDiag, and DeepHunt with DiagMLP while purportedly keeping all other pipeline elements, and compare performance on five datasets (SN, TT, GAIA, D1, D2) across fault detection, localization, and classification. They report that DiagMLP matches or exceeds the GNN-based baselines, concluding that graph structures contribute marginally to diagnostic performance. UMAP visualizations are used both as motivation and as supporting evidence.","tokens_in":8769,"tokens_out":7713,"duration_ms":72053,"significance":"If substantiated, this is a valuable negative result for the microservice fault diagnosis community. It challenges the prevailing assumption that explicit service-dependency graphs are necessary, and it advocates for standardized baselines and protocols. The paper is well-motivated and the DiagMLP design is admirably simple. However, the current experimental evidence does not support the central claim because the ablation is not fully controlled: protocol changes (window-splitting correction, validation set, reimplemented preprocessing) are conflated with the removal of the GNN module. The significance of the paper depends on whether the authors can provide a rigorous controlled comparison.","major_comments":[{"comment":"The central ablation is not controlled. The paper states that it 'rectified issues in prior frameworks, such as the window-splitting method in Eadro, which risked data leakage, and incorporated a validation set,' while Table IV's footnote says that results without standard deviations are 'directly extracted from the cited papers.' This means several baseline numbers (e.g., DiagFusion, CHASE, and possibly parts of DeepHunt and TVDiag) were produced under the original, potentially flawed protocol, whereas DiagMLP was evaluated under the corrected protocol. The large SN Top-1 localization gap (80.2±3.1% vs. 41.8±14.5%) is exactly the kind of difference that fixing a data-leaking window split could produce. To support the claim that GNN modules contribute marginally, all baselines must be re-run under the identical corrected protocol (same window splitting, same validation set) or the claims must be explicitly restricted to the re-run baselines.","section":"Section IV.A, Notes (3) and Table IV footnote"},{"comment":"For SN and TT, the paper says it 'reimplemented preprocessing code due to missing original scripts.' This implies the input features for DiagMLP and for the Eadro baseline may differ from those in the original Eadro paper. If the Eadro results in Tables II and IV are taken from the original publication, then the comparison conflates preprocessing differences with architecture differences, contradicting the paper's statement in Section IV that 'we preserve all other pipeline elements.' The authors should either demonstrate that their reimplemented preprocessing reproduces the original Eadro results (a reproducibility check) or re-run Eadro using the reimplemented preprocessing under the corrected protocol.","section":"Section IV.A(b)"},{"comment":"The detection and classification results are reported without standard deviations, yet the text in Section IV.B claims 'differences within standard deviations' for GAIA classification and treats the SN/TT detection improvements as meaningful. A parity claim is a negative result and requires statistical evidence that the observed differences are not significant. Please provide standard deviations, confidence intervals, or significance tests for all key comparisons, especially the detection results in Table II where DiagMLP appears to outperform Eadro by a large margin (F1 96.7 vs. 92.1 on SN).","section":"Tables II and III"},{"comment":"The UMAP visualizations in Figs. 3 and 5 are used both to motivate the DiagMLP design and to explain the observed parity (see 'Key Insight 2'). This is circular: the same plots that justify the baseline are offered as evidence that GNNs do not improve separability. Moreover, visual inspection is not a quantitative evaluation. Please report objective separability metrics (e.g., silhouette score, k-nearest-neighbor accuracy) computed on the preprocessed input features, MLP embeddings, and GNN embeddings to support the claim that preprocessing already encodes dependency information.","section":"Section III.B and Section IV.C"}],"minor_comments":[{"comment":"In the sentence 'we introduce DiagMLP, a ablation experimental baseline,' the article should be 'an ablation experimental baseline.'","section":"Section I"},{"comment":"The sentence 'we project Xt from from R3dN to R2' contains a duplicated 'from.'","section":"Section III.B"},{"comment":"Reference [13] has a typo: 'F ound.' should be 'Found.'","section":"References"},{"comment":"The detection results in Table II lack standard deviations; if available from the re-run experiments, please include them to allow readers to assess variability.","section":"Table II"},{"comment":"The three panels in Figure 5 are not labeled with (a), (b), (c) in the caption or on the figures themselves; please add labels for clarity.","section":"Figure 5"}],"recommendation":"major_revision","confidential_remarks":"The paper addresses a timely and important question, and the proposed DiagMLP baseline is a useful contribution to the community. However, the uncontrolled comparison is a serious flaw that undermines the central claim. The revision path is clear: re-run all baselines under the corrected protocol, or restrict the claims to the set of baselines that were properly re-run. If the authors cannot obtain or reimplement the original code for some baselines, they should remove those comparisons and temper the abstract accordingly. I recommend major revision rather than reject because the issue is experimental fixability, not a fundamental error in the paper's logic."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper asks a question worth asking: do GNNs actually earn their keep in microservice fault diagnosis, or are they riding on better preprocessing? DiagMLP is a sensible minimal baseline, and the paper is clearly written and honest about its own steps. That honesty is a real strength, and it makes the paper useful as a starting point for discussion.\n\nWhat's genuinely new here is the attempt to isolate the modeling stage by swapping GNN modules for an MLP across five datasets. The UMAP motivating example is nice, and the authors are transparent that they reimplemented preprocessing for SN and TT and fixed Eadro's window-splitting, which risked data leakage. They also added a validation set.\n\nBut that transparency also exposes the central problem. For SN and TT, the Eadro numbers in Tables II and IV come from the original papers, which used the old, leaky protocol. DiagMLP is evaluated under corrected splits and with a validation set. So the comparison does not isolate graph modeling; it compares a cleanly evaluated MLP to a GNN evaluated under a worse protocol. The large localization gains — 80.2 vs 41.8 Top-1 on SN — are exactly what you'd expect from fixing data leakage. For GAIA, D1, and D2, some baseline numbers carry standard deviations (presumably rerun) but others don't, so the same confound partially applies. As reported, the ablation is not controlled at the level the abstract claims.\n\nThere's also a minor circularity in using UMAP separability both to motivate the baseline and to explain its success. That's consistent evidence, not independent confirmation.\n\nThe underlying question is legitimate and the field does need standardized baselines. I would send this to peer review, because a serious referee can push for the obvious fix: re-run every GNN baseline under the exact same preprocessing, splits, and model selection, and release code and data. If the results hold up, this becomes a valuable paper. As submitted, the evidence is suggestive but not conclusive.","headline":"Good question, sensible baseline, but the ablation isn't controlled enough to back the headline claim.","tokens_in":9356,"tokens_out":2298,"would_cite":false,"duration_ms":23150,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Plain MLP matches or beats GNNs in microservice fault diagnosis","keywords":["microservice fault diagnosis","graph neural networks","multimodal telemetry","multilayer perceptron baseline","ablation study","fault localization","dependency graph","preprocessing"],"falsifier":"Re-run the original GNN frameworks (Eadro, TVDiag, DeepHunt) under the paper's corrected protocol—fixed window splitting and an added validation set—and compare against DiagMLP on the same five datasets; if any baseline then beats DiagMLP by a clear margin, the parity result is an artifact of protocol fixes rather than evidence against graph modeling. A complementary test on a synthetic dataset with faults injected along known dependency paths would show whether topology ever matters.","tokens_in":8265,"feed_emoji":"⚙️","tokens_out":7684,"duration_ms":70486,"temperature":0.7,"pith_summary":"This paper asks whether graph neural networks actually earn their place in multimodal fault diagnosis for microservice systems. The authors build DiagMLP, a deliberately minimal, topology-agnostic baseline that keeps the same preprocessing, embedding, and task modules as existing GNN-based pipelines but replaces the graph module with two concatenation-based MLPs. Across five public datasets covering fault detection, localization, and classification, DiagMLP matches or outperforms state-of-the-art GNN-based methods. The paper's claim is that the reported gains of GNN frameworks come mostly from preprocessing and multimodal fusion, not from modeling the service dependency graph. If correct, this means simple MLP baselines and standardized evaluation protocols should be standard before any graph-based innovation is claimed.","feed_headline":"Plain MLP matches or beats GNNs in fault diagnosis","feed_subtitle":"Gains from graph modules may be preprocessing in disguise; a simple two-stage MLP holds its own on five datasets.","key_machinery":"The central object is DiagMLP, an ablation baseline defined by two single-layer MLPs joined by concatenation. It takes per-node multimodal embeddings for metrics, logs, and traces, appends a learnable position embedding per node, fuses the modalities node-by-node, then concatenates all nodes and fuses them into a system-level representation for detection, localization, and classification. Its role is to replace a GNN module inside an existing pipeline with a topology-agnostic module that retains all multimodal fusion, so any performance difference can be attributed to graph modeling alone. Learnable position embeddings give nodes distinct identities without an explicit dependency graph, and the single-layer design keeps the baseline deliberately minimal, so parity with GNNs is evidence against the value of topology.","core_discovery":"On the paper's own terms, the discovery is that graph structure is not the source of performance in current GNN-based fault diagnosis pipelines. DiagMLP is inserted into Eadro, TVDiag, and DeepHunt in place of their GNN modules—graph attention, GraphSAGE, and graph autoencoder backbones—while every other component stays fixed. The resulting model performs at parity or better: fault detection F1 of 96.7 percent versus 92.1 percent on one dataset and 90.8 percent versus 90.7 percent on another; classification within one standard deviation on a third; and localization Top-1 accuracy that is substantially higher on the two largest comparisons. UMAP visualizations show that preprocessed multimodal features already separate fault classes by root cause before any graph modeling, and that neither MLP nor GNN embeddings add much separation. The authors conclude that preprocessing pipelines already encode critical dependency information, making explicit graph modeling at best redundant and at worst a source of instability.","pith_inferences":["A direct testable extension would be to construct synthetic microservice data where faults are injected along known dependency edges; if DiagMLP still matches GNNs there, the case for graph modeling shrinks further, and if it fails there, that defines the regime where GNNs earn their complexity.","The parity result, if it generalizes, suggests that many multimodal deep-learning pipelines may be carrying their performance in feature preprocessing and fusion, a hypothesis worth probing in adjacent domains such as cloud infrastructure monitoring and network failure diagnosis.","The learnable position embeddings are a quiet part of the result: they give the MLP a way to distinguish nodes without topology, so a follow-up could test whether those embeddings implicitly recover service-dependency structure by comparing them to the true dependency graph."],"forward_implications":["If the central claim is right, reported accuracy gains in recent GNN-based fault diagnosis papers should be re-assigned to preprocessing and multimodal fusion, not to graph architecture.","Simple MLP baselines like DiagMLP should be added to every future fault-diagnosis evaluation before a GNN is claimed to help.","Standardized evaluation protocols—including fixed window splitting, validation sets, and re-running baselines under identical conditions—are needed to prevent preprocessing artifacts from masquerading as model gains.","Larger datasets with realistic dependency-driven fault propagation are needed to test whether any architecture can exploit topology; current datasets with dozens of service instances may simply be too small for graphs to matter."],"supporting_citations":[{"why":"Supplies the SN and TT datasets and the end-to-end GAT-based baseline whose GNN module DiagMLP replaces for detection and localization.","marker":"[2]"},{"why":"Supplies the D1 and D2 datasets and the graph-autoencoder baseline whose GNN module DiagMLP replaces for localization.","marker":"[3]"},{"why":"Supplies the GAIA dataset and the GraphSAGE-based baseline used for classification and localization comparison.","marker":"[5]"},{"why":"Provides a comparison baseline and motivates the observation that trace-based preprocessing may already encode dependency graphs.","marker":"[6]"},{"why":"Comparison baseline for localization based on a causal heterogeneous graph, used to frame the gap in controlled GNN ablations.","marker":"[7]"},{"why":"Shows how trace and log data are turned into graphs, supporting the claim that preprocessing embeds topology before GNNs run.","marker":"[9]"},{"why":"Supplies the learnable positional embedding idea that lets DiagMLP distinguish nodes without an explicit dependency graph.","marker":"[25]"},{"why":"Supplies the UMAP projection used to visualize that preprocessed features already separate fault windows by root cause.","marker":"[28]"}],"fun_headline_variants":["GNNs for fault diagnosis? A plain MLP matches them","Graphs don't help GNNs in fault diagnosis, study finds","MLP beats graph neural networks for microservice faults","Simple MLP outperforms GNNs on multimodal fault data","Graph structure in GNNs is redundant for fault diagnosis"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The conclusion rests on the fairness of the ablation: DiagMLP and each GNN baseline must differ only in the graph module, with identical preprocessing, embeddings, data splits, and evaluation, so that observed parity reflects the redundancy of graph modeling rather than protocol fixes.","fun_headline_variants_meta":{"raw":{"variants":["GNNs for fault diagnosis? A plain MLP matches them","Graphs don't help GNNs in fault diagnosis, study finds","MLP beats graph neural networks for microservice faults","Simple MLP outperforms GNNs on multimodal fault data","Graph structure in GNNs is redundant for fault diagnosis"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000174,"raw_usage":{"total_tokens":1265,"prompt_tokens":909,"completion_tokens":356,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":525,"completion_tokens_details":{"reasoning_tokens":270}},"tokens_in":525,"tokens_out":356,"duration_ms":4203,"temperature":1.0,"reasoning_tokens":270,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T22:05:12.834625+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the original GNN frameworks (Eadro, TVDiag, DeepHunt) under the paper's corrected protocol—fixed window splitting and an added validation set—and compare against DiagMLP on the same five datasets; if any baseline then beats DiagMLP by a clear margin, the parity result is an artifact of protocol fixes rather than evidence against graph modeling. A complementary test on a synthetic dataset with faults injected along known dependency paths would show whether topology ever matters.","supporting_citations":[{"cited_title":"Eadro: An End-to-End Troubleshooting Framework for Microservices on Multi-Source Data,","cited_arxiv_id":null,"evidence_quote":"Supplies the SN and TT datasets and the end-to-end GAT-based baseline whose GNN module DiagMLP replaces for detection and localization."},{"cited_title":"Interpretable Failure Localization for Microservice Sys- tems Based on Graph Autoencoder,","cited_arxiv_id":null,"evidence_quote":"Supplies the D1 and D2 datasets and the graph-autoencoder baseline whose GNN module DiagMLP replaces for localization."},{"cited_title":"Robust Failure Diagnosis of Microservice System Through Multimodal Data,","cited_arxiv_id":null,"evidence_quote":"Provides a comparison baseline and motivates the observation that trace-based preprocessing may already encode dependency graphs."},{"cited_title":"DeepTraLog: Trace-Log Combined Microservice Anomaly Detection through Graph-based Deep Learning,","cited_arxiv_id":null,"evidence_quote":"Shows how trace and log data are turned into graphs, supporting the claim that preprocessing embeds topology before GNNs run."},{"cited_title":"Graph Neural Networks with Learnable Structural and Positional Representa- tions,","cited_arxiv_id":null,"evidence_quote":"Supplies the learnable positional embedding idea that lets DiagMLP distinguish nodes without an explicit dependency graph."},{"cited_title":"UMAP: Uniform Manifold Approximation and Projection,","cited_arxiv_id":null,"evidence_quote":"Supplies the UMAP projection used to visualize that preprocessed features already separate fault windows by root cause."}],"review_version":1}