{"id":"068b02af-c778-4791-9c87-711e29103899","arxiv_id":"2501.00790","paper_version":2,"verdict":"REJECT","confidence":"MODERATE","novelty_score":3.0,"correctness_risk":"high","formal_verification":"none","parameter_count":6,"one_line_summary":"LENS-XAI reports 95.34%, 99.92%, 98.42%, and 99.34% accuracy on Edge-IIoTset, UKM-IDS20, CTU-13, and NSL-KDD using VAE, knowledge distillation, and attribution-based explainability, but the claimed 10% training protocol is contradicted by its own algorithm.","lead":"This paper describes LENS-XAI, an intrusion detection framework that combines variational autoencoders, knowledge distillation, and feature-attribution explanations, and reports high accuracies on four network security benchmarks. A generalist might read it because lightweight, explainable intrusion detection matters for IoT and edge devices, but the paper's training protocol is internally inconsistent and the reported results are not backed by code or reproducible artifacts.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Algorithm 2 trains VAE, teacher, and student on the full dataset before the 10/90 split, so the reported accuracies do not support the 10%-data training claim.","rationale":"The reader's weakest_assumption correctly identifies the load-bearing vulnerability. The strongest claim—10% training data yields 95.34%, 99.92%, 98.42%, and 99.34% accuracy—requires that all learned components be fit exclusively on the 10% training partition. The paper's own Algorithm 2 invalidates this: preprocessing, VAE training, teacher training, and student distillation all happen on the full merged dataset D before the 10/90 split in Step 5. Section 3.4.1 reinforces this by stating the teacher is trained on the full dataset. This is not a disagreement about evaluation conventions; it is an internal inconsistency that directly inflates the reported numbers. The lack of open code, seeds, or architecture details is secondary but compounds the problem: there is no way to check whether the implemented pipeline actually followed the claimed 10% protocol. A re-run with the split moved to the start is the minimal, decisive test. If the re-run reproduces the reported accuracies, the central claim survives; if not, the paper's headline result is invalid. Since this is the same concern the reader identified, the verdict remains REJECT / UNCHANGED.","tokens_in":24595,"tokens_out":3021,"duration_ms":29539,"concrete_test":"Re-run the full LENS-XAI pipeline on all four datasets with the partition moved to the very beginning: first create a stratified 10% training / 90% test split (for NSL-KDD, use the standard KDDTrain+ / KDDTest+ split, training on only 10% of KDDTrain+), then fit preprocessing statistics, the VAE, the teacher, and the student exclusively on the training partition, and finally evaluate the student on the untouched test partition. Report per-class accuracy, macro-F1, and confusion matrices. If accuracies fall materially below 95.34% (Edge-IIoTset), 99.92% (UKM20), 98.42% (CTU-13), and 99.34% (NSL-KDD), the 10%-data claim is refuted; if they match, the leakage concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—competitive accuracy from only 10% of the data—requires that every learned component be fit on the 10% training partition before the 90% test partition is touched. Algorithm 2 violates this: Step 1 preprocesses the full merged dataset D, Step 2 trains the VAE on mini-batches from D, Step 3 trains the teacher on all latent embeddings {z} from D and then trains the student on the same full-D latent embeddings, and only Step 5 partitions data into 10% train / 90% test. Section 3.4.1 explicitly states 'Teacher Model: Trained on the full dataset.' Thus test samples contribute to the VAE's latent space, the teacher's soft targets, and the student's distillation objective before evaluation. The standard scaler and encoders are also fit on D before the split, adding a second leakage path. This is an internal inconsistency with Section 3.6's 'stringent partition' description, not a mere ambiguity. The reported accuracies are therefore upper bounds on what a genuinely 10%-trained system would achieve, and the paper's headline result is unsupported as written.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes LENS-XAI, a framework that combines variational autoencoders, knowledge distillation, and variable-attribution-based explainability for network intrusion detection in IIoT and edge environments. The central claim is that the framework achieves state-of-the-art performance (95.34% on Edge-IIoTset, 99.92% on UKM20, 98.42% on CTU-13, and 99.34% on NSL-KDD) while training on only 10% of the available data. The methodology describes preprocessing, VAE latent representation learning, teacher-student distillation with a 10/90 train/test split, and attribution-based explanations. The evaluation compares the teacher and student models against several baselines across four datasets and reports complexity, inference time, and explainability results.","tokens_in":24786,"tokens_out":4488,"duration_ms":41036,"significance":"If the results were valid, the paper would offer a practically relevant lightweight IDS with a clear integration of VAE-based representation learning, knowledge distillation, and XAI, and the 10%-data training claim would be noteworthy for resource-constrained deployment. The paper also provides an explicit pipeline in Algorithm 2, which is helpful for reproducibility assessment, and it reports a full ablation-free comparison across several public datasets. However, the central empirical claim is not supported by the experimental protocol as described: the pipeline appears to train the VAE, teacher, and student on the full dataset before the 10/90 split, which makes the reported accuracies upper bounds on a genuinely 10%-trained system. Because the main contribution of the paper is empirical, this flaw undermines the significance of the reported results.","major_comments":[{"comment":"The reported results do not support the claim of training on 10% of the data. Algorithm 2 trains the VAE on mini-batches sampled from the full merged dataset D in Step 2, trains the teacher on all latent embeddings {z} from D and trains the student on the same full-D latent embeddings in Step 3, and only in Step 5 partitions the data into 10% for training and 90% for testing. Section 3.4.1 explicitly states that the teacher is 'Trained on the full dataset.' This means test instances contribute to the VAE's latent space, the teacher's soft targets, and the student's distillation objective before any evaluation is performed. Section 3.6's description of a 'stringent partition' is therefore internally inconsistent with the pipeline in Algorithm 2, and the accuracies in Tables 3-9 cannot be interpreted as the performance of a model trained on 10% of the data.","section":"Algorithm 2, Steps 2-3 and 5; Section 3.4.1"},{"comment":"The evaluation on NSL-KDD uses a random 10/90 split of the full dataset, whereas the standard NSL-KDD protocol uses the fixed KDDTrain+ and KDDTest+ subsets, as the paper itself notes in Section 4.2. Comparisons with prior results trained and tested on the official split (RNN-IDS, MCNN-DFS, etc. in Table 8) are not valid under a different, nonstandard split. The paper does not report the class distribution of the 10% training partition, so it is unclear whether rare classes like U2R are present at all in training; the very low U2R accuracy (37.14% for the teacher, 23.80% for the student in Table 9) is consistent with a training partition that largely omits or underrepresents that class. The protocol needs to be clarified and, if a nonstandard split is used, justified with the class breakdown.","section":"Section 4.3.4 and Section 4.2 (NSL-KDD)"},{"comment":"The headline accuracy figures mix teacher and student results without attribution. The 95.34% on Edge-IIoTset, 99.92% on UKM20, and 98.42% on CTU-13 are teacher-model results (Tables 3, 5, and 7), while the 99.34% on NSL-KDD is the student-model result (Table 8; the teacher achieves 98.66%). The abstract and conclusion present these four numbers as a single claim for the framework, which obscures which model is being claimed as the lightweight deployable system. Since the paper argues that the student model is the efficient solution, the student's per-dataset accuracy should be reported consistently and separately.","section":"Abstract and Conclusion vs. Tables 3, 5, 7, 8"},{"comment":"The framework's central efficiency and performance claims are not supported by any sensitivity analysis or hyperparameter selection procedure. The hyperparameters beta, latent dimension k, temperature T, distillation weight alpha, training epochs, and train_fraction are set to single values without a validation-based criterion or an ablation study. In particular, the observation that the student occasionally outperforms the teacher (e.g., Edge-IIoTset F1 in Table 3 and NSL-KDD accuracy in Table 8) is unexpected under standard knowledge distillation and is not explained. The paper should provide a parameter study or, at minimum, state how these values were chosen and whether performance is stable under small perturbations.","section":"Section 4.1, Eq. (2) and Eq. (3)"}],"minor_comments":[{"comment":"The order of the four datasets in the abstract and conclusion ('Edge-IIoTset, UKM20, CTU-13, NSL-KDD') matches the results order, but the reader must reconstruct which number belongs to which model; please state explicitly, e.g., 'teacher model on Edge-IIoTset, student model on NSL-KDD.'","section":"Abstract and Section 4.3"},{"comment":"The text contains typographical and formatting issues, including 'artifical intelligence' in the introduction and 'V AEs' with a space throughout; these should be corrected for consistency.","section":"Section 1 and throughout"},{"comment":"The table caption and the surrounding text use both 'LENS-XAI' and 'LENSXAI' inconsistently; the same applies to 'Teacher' vs. 'T eacher' in Table 4.","section":"Section 4.3.4, Table 8"},{"comment":"The limitations section discusses dependence on labeled data and explainability overhead, but it does not acknowledge the apparent leakage of test data into the training pipeline described in Algorithm 2, nor does it discuss the discrepancy between the 10%-data claim and the full-data teacher training described in Section 3.4.1.","section":"Section 5.3 (Limitations)"},{"comment":"The variable attribution algorithm defines conditional expectations over partially specified feature subsets, but the practical implementation of these expectations (e.g., using background data, a surrogate model, or approximations) is not described; please specify how Equation (6) is computed in the experiments.","section":"Algorithm 1"}],"recommendation":"reject","confidential_remarks":"The paper is closely related to the authors' prior KD-XVAE work [23], but the novelty distinction is not clearly delineated; this is compounded by the experimental protocol issue, which invalidates the headline quantitative claims. The flaw in Algorithm 2 is not a matter of presentation, so a major revision would require re-running all experiments under a correctly partitioned protocol, which goes beyond a standard revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: don't trust the 10%-data headline. Algorithm 2 trains the VAE on the full merged dataset, trains teacher and student on the resulting latent embeddings, and only then splits into 10% train / 90% test. Section 3.4.1 says the teacher is trained on the full dataset. That means the test partition is visible before evaluation, so the reported accuracies are upper bounds for a genuinely 10%-trained system. This is the paper's central claim, and it collapses as written.\n\nWhat's genuinely useful: the paper is clearly written, evaluates on four public datasets, gives per-attack breakdowns, and reports parameter counts and inference times. The variable-attribution explainability section shows concrete feature-level contributions for teacher and student, which is a nice practical touch. The limitations paragraph is honest about labeled-data dependence and weak rare-class detection. The method itself is an incremental extension of the authors' own KD-XVAE — VAE, knowledge distillation, attribution — and Section 2.4 says as much. That's not a crime, but it means the data-integrity problem is the main story.\n\nOther soft spots: no code, no seeds, no error bars. The improvements over baselines are often within what you'd expect from run-to-run variation. The comparisons don't always align in task setup (NSL-KDD's standard test split vs. random split), so the 'outperforms SOTA' framing is overstated. The 'merge preprocessed datasets into a unified dataset D' line in Algorithm 2 is also confusing and, taken literally, would mix datasets before training.\n\nI don't see fabricated results or a circularity problem — the comparisons are external and the explanation procedure is standard. The flaw is a correctable methodological one, but as submitted the headline result is unsupported. I would reject, and if the authors redo the experiments with a true 10/90 split before any model is fit, release code and seeds, and tone down the SOTA claims, it could be a competent applied paper worth a second look.","headline":"The 10%-data training claim is contradicted by the paper's own Algorithm 2, so the headline accuracies are unsupported as written, despite solid organization and useful engineering detail.","tokens_in":25371,"tokens_out":5220,"would_cite":false,"duration_ms":49819,"reading_group":"maybe","serious_thinker":"no","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"LENS-XAI claims that a VAE-based teacher–student pipeline with attribution explainability reaches state-of-the-art intrusion detection accuracy on four benchmarks while training on 10% of the data.","keywords":["intrusion detection","knowledge distillation","variational autoencoder","explainable AI","Industrial Internet of Things","network security","teacher-student model","feature attribution"],"falsifier":"Train the VAE, teacher, and student strictly on the 10% partition described in Section 3.6, keep the remaining 90% completely untouched during training, and re-report the four accuracies; if the numbers fall materially below 95.34%, 99.92%, 98.42%, and 99.34%, the central claim is not supported.","tokens_in":24342,"feed_emoji":"🛡️","tokens_out":4413,"duration_ms":38734,"temperature":0.7,"pith_summary":"The paper proposes LENS-XAI, a lightweight intrusion detection pipeline that combines a variational autoencoder (VAE) with knowledge distillation and an attribution-based explainability step. The central claim is that this pipeline reaches state-of-the-art detection accuracy on four benchmark network datasets while being trained on only 10% of the available labeled data and producing feature-level explanations for each prediction. If the claim holds, resource-constrained IoT and edge deployments could get accurate, interpretable detectors without large labeled training sets. The reported accuracies are 95.34% on Edge-IIoTset, 99.92% on UKM20, 98.42% on CTU-13, and 99.34% on NSL-KDD, with the distilled student model staying close to or above the teacher while using fewer parameters and less inference time.","feed_headline":"Intrusion detector claims 99.3% accuracy trained on 10% of data","feed_subtitle":"A VAE-plus-distillation pipeline keeps detection accurate, fast, and explainable for resource-constrained networks.","key_machinery":"The load-bearing mechanism is a three-stage pipeline: a VAE encoder compresses each input into a low-dimensional latent representation; a high-capacity teacher model is trained on those embeddings; and a compact student model learns to match the teacher's temperature-softened output distribution (Equation 3) while also fitting hard labels. The distillation temperature $T=2$ and mixing weight $\\alpha=0.5$ control how much soft versus hard signal the student receives. Explainability comes from a variable-attribution decomposition (Equations 4–8) that expresses a prediction as a baseline plus sequential marginal contributions of features, with a two-step heuristic ordering to reduce order sensitivity. This attribution step is what turns the detector's decisions into per-feature explanations for analysts.","core_discovery":"On its own terms, LENS-XAI is a claim that one architecture can jointly deliver data efficiency, lightweight operation, and explainability for intrusion detection. The teacher model trains on VAE latent embeddings, and the student is distilled from it using temperature-scaled soft targets; variable attribution then breaks each prediction into baseline plus per-feature contributions. Across the four datasets, the framework reports the four accuracies above, with the student model cutting parameter counts substantially and reducing per-batch inference time while keeping accuracy within a fraction of a point (and on NSL-KDD exceeding the teacher). The paper presents this as outperforming the state-of-the-art baselines it compares against on each dataset.","pith_inferences":["A strict reading of Algorithm 2 suggests the VAE and distillation steps occur before the 10/90 split, so the data-efficiency advantage may be smaller than advertised; rerunning with a genuine 10% training partition would settle this.","The attribution method's dependence on feature ordering means explanations are heuristic; different orderings could change which features appear responsible for a prediction.","Because the reported performance on rare attack classes such as SQL Injection and U2R is much lower than the aggregate accuracy, the headline numbers mainly reflect frequent classes; a deployment-focused evaluation should weight rare attacks separately."],"forward_implications":["If the 10%-training result is real, intrusion detection systems could be deployed in settings with very few labeled attack samples.","The distilled student model's parameter and inference-time reductions make the approach feasible on edge hardware without sacrificing much accuracy.","Feature-level attributions allow security analysts to inspect why a flow was flagged, addressing the black-box objection to deep-learning IDS.","The same VAE-plus-distillation pattern could be transferred to other anomaly detection domains where labeled data is scarce."],"supporting_citations":[{"why":"Supplies the variational autoencoder objective whose latent embeddings drive anomaly detection.","marker":"[31]"},{"why":"Defines the temperature-scaled distillation loss that transfers teacher knowledge to the compact student.","marker":"[32]"},{"why":"Prior VAE-plus-knowledge-distillation system that LENS-XAI extends with attribution-based explainability.","marker":"[23]"},{"why":"Provides the Edge-IIoTset state-of-the-art baselines the framework is compared against.","marker":"[38]"},{"why":"Provides the CTU-13 machine-learning baselines the framework's accuracy is measured against.","marker":"[39]"},{"why":"Provides the NSL-KDD deep-learning baselines the framework's accuracy is measured against.","marker":"[51]"},{"why":"Supplies the break-down attribution methodology used to decompose predictions into per-feature contributions.","marker":"[36]"},{"why":"Supplies the order-sensitive attribution algorithm that the explainability module adapts for variable contributions.","marker":"[33]"}],"fun_headline_variants":["LENS-XAI: detect intrusions with 99.9% accuracy, 10% data","Explainable AI intrusion detection: 99.9% accuracy, minimal data","Lightweight IDS: 99.9% accurate, trained on 10% of data","Knowledge distillation + VAE for fast, accurate, explainable IDS"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The 10%-data claim collapses if the model components were trained on the full dataset before the 90%/10% split, and the paper's own Algorithm 2 plus Section 3.4.1 both suggest they were.","fun_headline_variants_meta":{"raw":{"variants":["LENS-XAI: detect intrusions with 99.9% accuracy, 10% data","Explainable AI intrusion detection: 99.9% accuracy, minimal data","Lightweight IDS: 99.9% accurate, trained on 10% of data","Knowledge distillation + VAE for fast, accurate, explainable IDS"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001435,"raw_usage":{"total_tokens":5815,"prompt_tokens":1005,"completion_tokens":4810,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":621,"completion_tokens_details":{"reasoning_tokens":4719}},"tokens_in":621,"tokens_out":4810,"duration_ms":31188,"temperature":1.0,"reasoning_tokens":4719,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T22:42:13.221942+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train the VAE, teacher, and student strictly on the 10% partition described in Section 3.6, keep the remaining 90% completely untouched during training, and re-report the four accuracies; if the numbers fall materially below 95.34%, 99.92%, 98.42%, and 99.34%, the central claim is not supported.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Prior VAE-plus-knowledge-distillation system that LENS-XAI extends with attribution-based explainability."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the Edge-IIoTset state-of-the-art baselines the framework is compared against."},{"cited_title":"Al-Turaiki, N","cited_arxiv_id":null,"evidence_quote":"Provides the NSL-KDD deep-learning baselines the framework's accuracy is measured against."},{"cited_title":"Biecek, T","cited_arxiv_id":null,"evidence_quote":"Supplies the break-down attribution methodology used to decompose predictions into per-feature contributions."}],"review_version":1}