{"id":"5fd14a96-2ff5-4a05-a17a-d822ebf89c6c","arxiv_id":"2411.08172","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"FL4Deep localizes faults across the full deep learning pipeline using a knowledge graph built from static and dynamic code information, outperforming four prior tools on three of six fault types.","lead":"FL4Deep is a new system-level fault localization tool for deep learning software that builds a knowledge graph from static and dynamic information about the whole pipeline, from data splitting to deployment. In tests on 100 faulty scripts, it found more faults than four existing tools for data, library mismatch, and loss function errors.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Validation scripts appear to be mutated descendants of training scripts, so learned-component gains may reflect data leakage rather than true localization skill.","rationale":"The paper's central claim is that FL4Deep outperforms four prior techniques on 100 faulty scripts, with the strongest results in data, library mismatch, and loss-function faults, and the best precision/recall balance in five of six categories. The most load-bearing condition for that claim is that the learned components are tested on examples that are not near-duplicates of their training data. The validation set consists of 20 seed posts plus mutants; the training set is drawn from the same prior-study pools [23,107] and Defect4ML. The paper does not establish disjointness, and the described construction makes overlap not merely possible but likely. If overlap exists, the dynamic classifiers and NodePiece would perform well for reasons unrelated to fault-localization quality, and the precision/recall advantage over DeepFD and other baselines would be an artifact. This is exactly the reader's weakest assumption, so I agree. A second concern, that comparing deployment mismatches with tools never designed to detect them is unfair, is real but secondary: it affects mainly the 20 library-mismatch samples and does not bear on the learned components. I am not rejecting the paper: the replication package is promised and the concrete test above can settle the matter. If the check shows disjointness, the evaluation's core numbers should be taken at face value modulo the baseline-comparison issue. If it shows overlap, the conditional verdict should be strengthened and the RQ1 numbers recomputed on a truly disjoint validation set. For now, the appropriate disposition is unchanged from the reader's conditional verdict.","tokens_in":27879,"tokens_out":5179,"duration_ms":57312,"concrete_test":"Download the replication package [63]; extract the Stack Overflow post IDs and GitHub URLs for all 75 training and 100 validation scripts. Compute (a) exact set intersection and (b) token or AST similarity between each validation script and every training script to detect mutated descendants. If any validation script shares a seed post with or has high similarity to a training script, re-run RQ1 on a disjoint split in which all mutants derived from training posts are moved to training or removed, and report the dynamic-classifier and NodePiece results separately. If no overlap exists, the concern is resolved and the reported numbers can stand.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The evaluation's core comparison depends on the 100 validation scripts being independent of the 75 training scripts, because the dynamic-fault classifiers (RF/DT/KNN, Section 4.2) and the NodePiece link predictor (Section 4.4) are fit on the training set and then scored on the validation set. Section 4.1 says validation was built from '20 SO posts gathered from prior studies [23,107]' plus mutation operators applied to those posts. The training set was itself drawn from '17 buggy codes ... sourced from ... Cao et al. [23]' and 30 from Defect4ML [64], with 12 reported by both sources. If the validation SO posts are selected from the same [23]/[107] corpora, then every mutated validation script shares a seed post and much of its code and KG structure with a training script. For the learned classifiers, training and validation dynamic traces would come from the same architectures and similar training runs, producing near-duplicate feature vectors and inflated precision/recall. For NodePiece, a mutated sibling's KG is nearly isomorphic to a training graph, making inductive link prediction artificially easy. The paper never asserts disjointness, and the construction described makes overlap likely. Because three of the six headline fault categories and most precision/recall wins involve these learned components, the claimed superiority over DeepFD, DeepLocalize, AutoTrainer, and UMLAUT depends on this independence.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes FL4Deep, a knowledge-graph-based fault localization technique for deep learning software. FL4Deep extracts static information about the dataset, model, training environment, and deployment environment, plus dynamic information from training logs, and uses this to build a knowledge graph. Faults are identified by a set of KG rules, by random forest/decision tree/KNN classifiers trained on statistics of dynamic traces, and by a NodePiece link-prediction model that completes missing KG relationships; the final output is a ranked list of root causes. The approach is evaluated on a validation set of 100 buggy DL scripts (with six fault-type counts summing to 106) and compared with DeepFD, DeepLocalize, AutoTrainer, and UMLAUT. The authors report best or near-best performance on data, library mismatch, loss function, insufficient iteration, and activation function faults, and an ablation study showing static information is the most important component.","tokens_in":28139,"tokens_out":11199,"duration_ms":102386,"significance":"If the evaluation is sound, the paper makes a useful contribution to fault localization for DL-based systems. Its strengths include the system-level scope beyond the trained model, the combination of static rules with learned dynamic classifiers and graph link prediction, the use of a real-world bug dataset with a publicly available replication package, and the comparison with four existing tools using their own replication packages. The ablation study and the use of Fisher's exact test are also positive features. The central caveat is that the headline comparison depends on the independence of the 100 validation scripts from the 75 training scripts used to fit the learned components; the manuscript does not currently establish this, and there are also internal numerical inconsistencies in the reported results that must be resolved before the claims can be taken at face value.","major_comments":[{"comment":"Section 4.1 states that the 75-script training set includes 17 buggy codes from Cao et al. [23] and 30 from Defect4ML [64], with 12 reported by both sources, while the 100-script validation set was built from 20 SO posts gathered from prior studies [23, 107] plus mutation operators applied to those posts. Because [23] is a common source, and Section 6.1 confirms that validation scripts are \"buggy DL scripts previously employed by other studies\" and then mutated, the paper never establishes that the validation scripts (or their mutated descendants) are disjoint from the training scripts. The dynamic fault classifiers (RF/DT/KNN, Section 4.2) and the NodePiece link predictor (Section 4.4) are fit on the 75 training samples and then scored on the 100 validation samples; any overlap or near-duplicate mutation between the two sets will inflate the reported accuracy for loss, activation, insufficient iteration, and optimization faults, and will make inductive link prediction artificially easy because validation KGs would be nearly isomorphic to training KGs. This independence is load-bearing for the central claim of outperforming DeepFD, DeepLocalize, AutoTrainer, and UMLAUT. The authors should provide explicit evidence of disjointness (e.g., a full list of training and validation sample IDs and mutation lineages, or a deduplication analysis) or re-run the evaluation on a properly separated hold-out set.","section":"Sections 4.1, 5.2, 6.1"},{"comment":"The six issue-type sample counts in Table 5 and Table 7 sum to 106 (19+20+16+15+10+26), not the 100 scripts stated in the abstract and in Section 5.2. If scripts may contain multiple simultaneous faults, this should be stated explicitly and the per-fault-type evaluation protocol defined. Independently, specific rows are internally inconsistent: for UMLAUT on Data, Table 5 reports 8 detected faults while Table 7 reports FP=27, FN=10 with 19 samples, implying TP=9 and yielding precision 0.25 and recall 0.47, neither matching the printed PR=0.23/RC=0.44; for FL4Deep on Insufficient iteration, Table 5 reports 8 detected faults while Table 7 reports FP=1, FN=5, RC=0.62, which with 15 positive samples implies either TP=8/FN=7 (recall 0.53) or TP=10/FN=5 (recall 0.67). The paper must reconcile these numbers and specify how a predicted fault is counted as \"identified\" (e.g., top-1, top-k, or any rank in the output list); without that, the headline accuracy and precision/recall comparisons are not verifiable.","section":"Section 5.2/5.4, Tables 5 and 7"},{"comment":"The sensitivity analysis opens with \"Using the same 20 samples we used to compare approaches,\" but Section 5.2 reports the comparison on 100 scripts. If the ablation was actually run on only 20 scripts, the conclusions of RQ2 (e.g., static information removal causes a 67% drop with p=0.04) do not apply to the 100-script evaluation and should be redone; if it is a typo, it must be corrected. In addition, the Fisher's exact test should report the contingency tables for each ablation comparison and should account for the paired nature of the 100-sample evaluation and the multiple comparisons across the six fault types.","section":"Section 5.3"}],"minor_comments":[{"comment":"Finding 4 says UMLAUT's precision on activation function faults is \"0.26%\" but the correct value is 26% (0.26 as a proportion); please fix.","section":"Section 5.4, Finding 4"},{"comment":"The phrase \"unseen samples\" in Section 6.1 is ambiguous: clarify whether \"unseen\" refers to the baselines' prior evaluations or to FL4Deep's training set, and if the latter, explain how disjointness was ensured.","section":"Section 6.1"},{"comment":"Table 4 is hard to read because each tool's output is a variable-length list (e.g., UMLAUT's warnings and FL4Deep's ranked root causes) with inconsistent formatting; a clearer layout or legend mapping each tool's output to the six fault categories would help readers interpret the comparison.","section":"Table 4"},{"comment":"The claim \"For the first time\" is strong given that UMLAUT and DeepDiagnosis already analyze multiple stages of the DL pipeline; consider softening the novelty claim to \"system-level\" or \"full-pipeline\" framing.","section":"Abstract and Section 1"},{"comment":"The ranking of root causes uses priors from Humbatova et al. without a sensitivity analysis; a short discussion of how the ranking would change under alternative priors would strengthen the paper.","section":"Section 4.5"}],"recommendation":"major_revision","confidential_remarks":"The central risk is the unstated dependence between the 75 training samples and the 100 validation samples; if the authors can demonstrate disjointness (e.g., via sample IDs in the replication package), the paper could become acceptable after revisions. The internal inconsistencies in Tables 5 and 7 are also concerning and need to be resolved before publication. The topic fits the journal's scope, and the replication package and comparison with four baselines are positive aspects worth preserving."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"FL4Deep is the first serious attempt I've seen to localize faults across the whole DL pipeline—data prep, model training, deployment—and the KG-based reasoning with static rules is a real new combination. The paper ships a replication package, uses four baselines, and has an ablation study. Those are real strengths.\n\nThat said, the evaluation has a load-bearing problem. The training set (75 scripts) is drawn from Cao et al., Defect4ML, and SO posts; the validation set (100 scripts) is built from 20 SO posts from Cao et al. and Wardat et al., mutated into new samples. The classifiers and NodePiece link predictor are fit on the training set and scored on validation. Mutated siblings of training posts share code and KG structure with training graphs, so their traces are near-duplicates. The paper never asserts the two sets are disjoint, and the described construction makes overlap likely. That means the precision/recall wins on data, loss function, and deployment faults—most of which come from learned components—could be inflated by data leakage. This is not a nit; it's the difference between a tool that genuinely localizes faults and one that recognizes mutated versions of training examples.\n\nTwo smaller issues. First, the paper says in the intro and Section 3.3 that it excludes data and framework faults, then happily evaluates and reports results for data faults and deployment library mismatches. That scope contradiction needs to be fixed. Second, the RQ3 section has a duplicated paragraph—copy-paste error, easy to fix. Also, the deployment comparison is against tools never designed to catch deployment faults, so 100% on library mismatch is less impressive than it looks, though the static rules are legitimate.\n\nThe central idea is sound and the system is useful. But the headline numbers cannot be trusted until disjointness is demonstrated. I'd tell the authors to re-run validation on a clearly disjoint set, or at minimum report the overlap and adjust conclusions. If they do that, this is a solid TOSEM-level paper. As is, I'd accept it for peer review with a request for major revision.","headline":"A genuinely novel system-level fault localization idea, but the headline numbers rest on a training/validation overlap the paper never rules out.","tokens_in":28647,"tokens_out":2436,"would_cite":true,"duration_ms":25592,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"FL4Deep claims that system-level fault localization for deep learning software—covering data, model construction, and deployment—beats model-centric tools, with best accuracy on data, library-mismatch, and loss-function faults in a…","keywords":["fault localization","deep learning software","knowledge graph","system-level debugging","software testing","data pipeline faults","deployment mismatch","static and dynamic analysis"],"falsifier":"Check every script in the 100-script validation set for exact or near-duplicate overlap with the 75-script training set, especially the 20 SO posts reused from prior studies and any mutants derived from them; if overlap exists, recompute precision and recall on a strictly disjoint hold-out set and see whether the reported data-fault accuracy stays near 84%.","tokens_in":27671,"feed_emoji":"🐛","tokens_out":6468,"duration_ms":60881,"temperature":0.7,"pith_summary":"FL4Deep's central claim is that fault localization for deep learning software should treat the whole development pipeline—data preparation, model construction and training, and deployment—as the search space, not just the neural network model. The paper argues that model-centric tools miss whole fault classes, and that a knowledge graph built from static facts plus dynamic training logs can localize a wider range of faults. On 100 faulty DL scripts, the approach reports the best accuracy for data faults (84%), training/deployment library mismatches (100%), and loss-function faults (69%), and the most balanced precision and recall across five of six fault categories. The authors' intended contribution is the first system-level fault localizer for DL software, with static information carrying most of the performance.","feed_headline":"System-level fault localizer catches data and deployment bugs others miss","feed_subtitle":"In 100 faulty scripts it hits 84% on data faults and 100% on library mismatches.","key_machinery":"The load-bearing object is the system-level knowledge graph: a labeled directed graph that encodes facts about the dataset, model structure and hyperparameters, training environment, deployment environment, and per-epoch training logs as RDF triples. Faults are represented as inference rules in Notation3, and a reasoning engine derives fault-related facts and links them to the system parts where the root cause lives. Dynamic training traces are compressed with eight statistical operators and fed to Random Forest, Decision Tree, and KNN classifiers whose majority vote predicts training-phase faults; NodePiece, an anchor-based inductive link predictor, completes missing KG relationships; and the final ranked list orders root causes by how often each fault type appears in prior DL-bug studies.","core_discovery":"On the authors' own terms, the discovery is that a knowledge graph spanning the entire DL pipeline can serve as the backbone of fault localization: static and dynamic information is extracted from the system, encoded as RDF triples, enriched by rule-based reasoning and inductive link prediction, and converted into a ranked list of root causes. This lets FL4Deep identify faults that originate outside the model—such as a train/deploy library version mismatch—which prior techniques that analyze only model training cannot see. In the reported evaluation, FL4Deep wins three of six fault categories on accuracy, and on precision/recall it is the best-balanced method for data, library mismatch, loss function, insufficient iteration, and activation-function faults, with precision/recall of 1.0/0.84, 1.0/1.0, 0.85/0.69, 0.89/0.62, and 0.89/0.92 respectively.","pith_inferences":["If the validation scripts are not strictly disjoint from the 75 training scripts, the reported dynamic-fault numbers are an upper bound; the paper does not state disjointness, so a leakage check is the first thing an independent evaluator should run.","The ranking step uses global fault frequencies from prior studies; an adaptive ranker that uses knowledge-graph confidence scores could improve recall for rare faults like optimizer issues, where FL4Deep lags DeepFD.","The same knowledge-graph-plus-rules recipe should transfer to classical ML pipelines and to data-centric debugging, since the data and deployment rules are not specific to neural architectures.","A cross-framework test on PyTorch would reveal how much of the reported accuracy comes from the static extraction rules versus the learned components, since only the former should port without retraining."],"forward_implications":["Faults that live outside the model—such as mismatched library versions between training and deployment—are localizable at 100% accuracy, a fault class the four compared approaches miss entirely.","A single execution of the buggy script suffices for the method, avoiding the ten-run training strategy that DeepFD uses, while still matching or beating it on most fault categories.","Static information is the highest-leverage component: removing it drops performance by 67% with a p-value of 0.04, so future DL fault localizers should invest in pipeline-wide static facts.","The knowledge-graph design lets the method rank multiple simultaneous root causes by prior fault frequency, giving developers an ordered debugging checklist rather than a single suspect line.","Because the approach targets the whole pipeline, it can be extended to DL frameworks beyond Keras and TensorFlow and to deployment faults that only appear after model export."],"supporting_citations":[{"why":"DeepFD is a comparison baseline and also supplies 17 training buggy samples; FL4Deep is measured against its fault-diagnosis performance.","marker":"[23]"},{"why":"DeepLocalize is a comparison baseline and a source of validation SO posts; it represents the model-centric approach FL4Deep aims to surpass.","marker":"[107]"},{"why":"UMLAUT is a comparison baseline that detects activation and data issues; its tendency to over-report activation faults is contrasted with FL4Deep's balance.","marker":"[86]"},{"why":"AutoTrainer is a comparison baseline focused on training-phase faults; FL4Deep is compared against its detection of activation and loss issues.","marker":"[122]"},{"why":"The taxonomy of real DL faults provides the fault types FL4Deep targets and the relative frequencies used to rank root causes.","marker":"[46]"},{"why":"The Defect4ML benchmark supplies 30 training samples and defines deployment-mismatch fault types such as library and environment mismatches.","marker":"[64]"},{"why":"The study of DL deployment challenges motivates and defines deployment fault rules for OS, architecture, and library mismatches.","marker":"[26]"},{"why":"NodePiece is the inductive link prediction algorithm used to complete missing relationships in the generated knowledge graph.","marker":"[34]"},{"why":"The replication package contains the datasets and source code, grounding the reproducibility of the reported evaluation.","marker":"[63]"}],"fun_headline_variants":["Whole-pipeline bug finder beats model-only tools on 3 of 6 fault types","System-level FL4Deep hits 100% on library mismatches, 84% on data faults","Knowledge-graph fault localizer catches bugs outside the model, wins 3 categories","Beyond the model: new tool localizes faults across entire DL pipeline","FL4Deep: system-level fault localization with top precision on 5 bug types"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The 100 validation scripts must be independent of the 75 training scripts; if the same Stack Overflow posts or mutated versions of them appear in both sets, the measured precision and recall for dynamic faults are inflated.","fun_headline_variants_meta":{"raw":{"variants":["Whole-pipeline bug finder beats model-only tools on 3 of 6 fault types","System-level FL4Deep hits 100% on library mismatches, 84% on data faults","Knowledge-graph fault localizer catches bugs outside the model, wins 3 categories","Beyond the model: new tool localizes faults across entire DL pipeline","FL4Deep: system-level fault localization with top precision on 5 bug types"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000316,"raw_usage":{"total_tokens":1789,"prompt_tokens":948,"completion_tokens":841,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":564,"completion_tokens_details":{"reasoning_tokens":732}},"tokens_in":564,"tokens_out":841,"duration_ms":8221,"temperature":1.0,"reasoning_tokens":732,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T21:53:03.626609+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Check every script in the 100-script validation set for exact or near-duplicate overlap with the 75-script training set, especially the 20 SO posts reused from prior studies and any mutants derived from them; if overlap exists, recompute precision and recall on a strictly disjoint hold-out set and see whether the reported data-fault accuracy stays near 84%.","supporting_citations":[{"cited_title":"DeepFD: Automated Fault Diagnosis and Localization for Deep Learning Programs","cited_arxiv_id":"2205.01938","evidence_quote":"DeepFD is a comparison baseline and also supplies 17 training buggy samples; FL4Deep is measured against its fault-diagnosis performance."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"DeepLocalize is a comparison baseline and a source of validation SO posts; it represents the model-centric approach FL4Deep aims to surpass."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"UMLAUT is a comparison baseline that detects activation and data issues; its tendency to over-report activation faults is contrasted with FL4Deep's balance."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"AutoTrainer is a comparison baseline focused on training-phase faults; FL4Deep is compared against its detection of activation and loss issues."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The Defect4ML benchmark supplies 30 training samples and defines deployment-mismatch fault types such as library and environment mismatches."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The replication package contains the datasets and source code, grounding the reproducibility of the reported evaluation."}],"review_version":1}