{"id":"371b0ead-3dce-491a-9f00-2ea104fba702","arxiv_id":"2507.13870","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"Label unification across four cybersecurity NER datasets does not improve cross-dataset generalization, and the LST-NER graph matching model offers no gain over BERT-base-NER.","lead":"The paper tests whether merging the labels of four cybersecurity datasets lets a single model recognize entities across all of them. It finds that simple label merging hurts performance and that a transfer learning model built on BERT does not beat a standard pretrained one.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Load-bearing concern: the paper's label mapping is a hand-crafted, non-expert artifact that is explicitly admitted to be 'likely suboptimal,' and no alternative mapping is tested; the central negative conclusion about label merging is therefore contingent on an unvalidated artifact.","rationale":"The reader's conditional verdict identified the hand-crafted, non-expert label mapping as the weakest assumption, and that is exactly the load-bearing concern I find. The paper's central negative conclusion—that simple label merging is insufficient for robust cross-dataset generalization—depends on treating the specific mapping in Appendix Table 8 as representative of label unification. The authors explicitly disclaim expertise and call the mapping 'likely suboptimal,' while also discarding a substantial number of fine-grained labels, which Appendix B shows inflates false negatives. Without testing alternative mappings, the observed failures cannot be cleanly attributed to label merging as a strategy rather than to this particular instantiation of it.\n\nI do not see this as an internal inconsistency: the reported matrices and tables support the narrower empirical claim that their unified models underperform on several datasets. The concern is about external validity and the strength of the generalization made in the conclusion. That is a sufficiency-of-evidence problem, not a soundness problem, which is consistent with the reader's CONDITIONAL verdict rather than a rejection.\n\nThe LST-NER reimplementation issue is real but secondary to the central claim about label merging. Even if the graph model were faithfully reproduced, the label-mapping concern would remain. Conversely, if an expert mapping changed the unified-model results, the central claim would fail regardless of the graph model's performance. Therefore I focus the test on expert-designed mappings.\n\nThe proposed test—having domain experts independently construct mappings and rerunning the same BiLSTM experiments with variance across seeds—would directly determine whether the negative conclusion is robust to mapping quality. Until such a check is run, the conditional verdict is appropriate, and no change to the reader's verdict is needed.","tokens_in":8885,"tokens_out":3222,"duration_ms":41235,"concrete_test":"Recruit at least two independent cybersecurity NER domain experts (or use a validated external taxonomy plus expert review) to each design a label-unification mapping from the four original schemas to a common coarse-grained schema, with explicit entity definitions. Then rerun the BiLSTM unified-model experiments from Tables 1 and 3 using each expert mapping, with five random seeds per condition, reporting mean and standard deviation of span-F1. Pre-specify a decision rule: if any expert-designed mapping yields a combined-model span-F1 on CYNER and APTNER at least as high as the single-dataset baselines (0.40 and 0.41, respectively) and raises the mean off-diagonal cross-dataset span-F1 by at least 0.05 absolute, then the conclusion 'simple label merging is not sufficient' would not hold for expert-quality unification.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that 'simple label merging is not sufficient for robust generalization' in cybersecurity NER. This claim rests on showing that the particular unification implemented here fails to improve cross-dataset generalization. The mapping in Appendix Table 8 is the sole representative of 'simple label merging' in the experiments, but it is not established as a fair or typical instance of label unification. The authors state in Appendix A: 'our unification of labels is likely suboptimal. We are not cybersecurity experts and some of the label-unification-related issues may be lessened if cybersecurity experts conducted the unification.'\n\nThe mapping has potentially consequential choices: ATTACKER's 'THREAT_ACTOR' and 'GENERAL_IDENTITY' are mapped to 'Organization'; CYNER's 'Indicator' is discarded entirely; DNRTI's 'Tool' maps to 'System' while APTNER's operating-system tokens also map to 'System'; and many fine-grained labels are dropped to O. Appendix B confirms that the unified models have a 5-15% higher false-negative ratio than reference models, indicating that dropped labels systematically bias predictions toward O. This means the observed underperformance of unified models (Table 3) and poor cross-dataset scores (Table 1) could be driven by an arbitrary or poorly chosen taxonomy rather than by an inherent limitation of label merging.\n\nIf the paper claimed only 'our mapping does not work,' the argument would be internally secure. But the conclusion generalizes to 'simple label merging is not sufficient,' which requires that the mapping be representative. Because no expert-designed mapping, no automated semantic mapping, and no ablation varying the target label set is tested, the external validity of the negative conclusion is the weakest load-bearing point.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper investigates whether unifying NER label sets across four cybersecurity datasets (APTNER, CYNER, DNRTI, ATTACKER) enables cross-dataset generalization. The authors map all original labels to four coarse-grained types, train BiLSTM taggers, and report a 4x4 span-F1 matrix of pairwise cross-dataset evaluations; they find that off-diagonal scores are far below same-dataset scores. They then analyze prediction errors qualitatively, propose a multi-head model that preserves original label sets, and reimplement the LST-NER graph-matching transfer model. Reported results show that unified training underperforms single-dataset baselines on some datasets, that multi-head weight sharing gives only marginal gains, and that LST-NER does not improve over BERT-base-NER. The paper concludes that 'simple label merging is not sufficient for robust generalization'.","tokens_in":9207,"tokens_out":6891,"duration_ms":76353,"significance":"The study has clear strengths: the pairwise evaluation matrix in Table 1 is a useful systematic comparison, the duplicate-sentence leakage check is careful, the JS-div analysis adds quantitative context, and the authors are transparent about the ChatGPT-assisted reimplementation and about their own limitations in designing the label mapping. I also agree with the reader's assessment that there is no circular reasoning: the mapping is a documented design choice and the comparisons are against external baselines. However, the paper's central negative conclusion is broader than the evidence. All results are single runs without variance, statistical-significance language is used without tests, and the label unification is represented by exactly one hand-crafted mapping that the authors themselves call 'likely suboptimal.' If the conclusion is reframed as a claim about this particular unification, the paper is a solid empirical contribution; in its current form, the general claim is not yet established.","major_comments":[{"comment":"The central conclusion in Section 6 that 'simple label merging is not sufficient for robust generalization' is supported by only one hand-crafted label mapping, which Appendix A itself calls 'likely suboptimal' and attributes to non-experts. Because the mapping discards CYNER's Indicator label, maps ATTACKER's THREAT_ACTOR and GENERAL_IDENTITY to Organization, and maps DNRTI's Tool but not APTNER's operating-system labels to System, the measured underperformance of unified models in Tables 1 and 3 could be an artifact of this particular taxonomy rather than a property of label merging in general. Without at least one alternative mapping or an expert-designed mapping, the evidence supports only a claim about this unification, not the general conclusion.","section":"2.2, Table 8, Appendix A"},{"comment":"The phrases 'performed significantly better' (Section 5.1) and 'no significant performance gains' (Abstract and Section 5.2) are not supported by any statistical test or repeated runs; all tables report single-run span-F1 values. In particular, Table 5 reports differences of 0.01–0.02 between BERT-base-NER and LST-NER, which are described as not significant, but with a single run there is no basis for deciding whether the differences are noise or real. The same issue affects the '5–15% lower ratio of FNs' in Section 4.5 and Appendix B.","section":"5.1, 5.2, Abstract"},{"comment":"The LST-NER result is based on a from-scratch, ChatGPT-assisted reimplementation of Zheng et al. (2024), since the original paper provides no public code. The manuscript does not provide enough detail about the graph construction and Gromov-Wasserstein matching procedure to allow a reader to verify that the reimplementation faithfully follows the original method. Consequently, the 'no significant gains over BERT-base-NER' result is evidence about this reimplementation, not necessarily about LST-NER as proposed by Zheng et al., and it should be framed as such.","section":"5.2, Disclosure of Chatbot Use"},{"comment":"The row 'Combined 0.38 -' in Table 3 is undefined: it is not clear what development set the 0.38 score is computed on, since there is no combined development dataset. The comparison between the combined model and the 'Original' column also mixes training-data size with label unification, so the claim that performance on CYNER and APTNER is 'hindered by the combined dataset' is not isolated from the effect of simply adding more data with different label distributions.","section":"Table 3"}],"minor_comments":[{"comment":"The heading 'Langauge Metrics' should be 'Language Metrics', and 'dissapointing' in Section 5.1 should be 'disappointing'.","section":"4.6, 5.1"},{"comment":"Section 4 refers to 'section 3.4' when describing the selection of off-diagonal entries, but the setup is described in Section 2.4.","section":"4"},{"comment":"The header 'val\\train Combined Original' and the inconsistent capitalization 'Aptner/Cyner' versus 'APTNER/CYNER' should be fixed for readability.","section":"Table 3"},{"comment":"The sentence 'This applies to all models discussed in the paper' follows the description of removing overlapping sentences for each train-test pairing; it should be clarified how this applies to the multi-head and LST-NER experiments, which are not simple pairwise train-test setups.","section":"2.2"},{"comment":"The BiLSTM baseline uses cross-entropy loss without a CRF output layer; since CRFs are standard for NER, this choice should be justified or acknowledged as a limitation, as it may affect absolute scores.","section":"2.3"},{"comment":"In Table 8, the label name 'MALW ARE' appears to be a typo for 'MALWARE'.","section":"Table 8"},{"comment":"The correlations in Table 6 are computed over only 12 off-diagonal cells; reporting p-values or confidence intervals would help assess whether the negative correlations are meaningful.","section":"Table 6"}],"recommendation":"major_revision","confidential_remarks":"To the editor: The paper is honest and the negative result is potentially valuable, but the authors should be asked to either narrow the conclusion to the specific mapping tested or provide evidence that the mapping is representative. The explicit admission in Appendix A that the mapping is 'likely suboptimal' makes the over-generalized conclusion a load-bearing issue. The absence of significance testing is also a barrier, but it is fixable by adding multiple seeds and confidence intervals or significance tests. I would not treat the ChatGPT disclosure as a misconduct concern; it is adequate and the authors state they reviewed the code."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nQuick take: this is a useful, honest negative-result paper. It documents that four popular cybersecurity NER datasets do not productively merge under a single coarse label set, and it does so with a clean pairwise evaluation and some careful qualitative analysis. If you work on dataset unification or domain adaptation for NER, worth reading.\n\nWhat's genuinely new: this is the first pairwise cross-dataset evaluation of APTNER, CYNER, DNRTI, ATTACKER under one label set, and the first attempt to apply LST-NER in this domain. The negative result is valuable—future researchers won't repeat this failure. The authors also did some things right: they remove duplicate sentences across datasets, they report JS-div correlations to explain performance, they run an O-class imbalance experiment to quantify the effect of dropping labels, and they openly admit their label mapping is likely suboptimal because they're not cybersecurity experts. They also disclose ChatGPT-assisted implementation.\n\nSoft spots, in order of importance. First, the central conclusion is stated more broadly than the evidence. The paper says \"simple label merging is not sufficient for robust generalization,\" but only one non-expert, hand-crafted mapping is tested. The stress-test concern is fair: no alternative mapping, no expert-designed mapping, no ablation over the target label set. That said, the qualitative analysis shows genuine annotation disagreements—\"sample\" as Malware vs O, \"Linux\" as System vs O—that wouldn't necessarily disappear with a better mapping. So the core observation is probably robust, but the external validity is limited. Second, all claims of significance (\"significantly better,\" \"no significant gains\") are made without statistical tests, and tables report single runs without variance. That's a real weakness for a paper with \"significant\" in the text. Third, the LST-NER null result rests on a from-scratch implementation without the original code; the disclosure is honest, but a flawed reimplementation could give a false negative for the method itself.\n\nWho this is for: researchers working on combining NER datasets, particularly in cybersecurity, and anyone studying label schema mismatches. It deserves a serious referee; I'd send it to review, but I'd ask the authors to either temper the generalization or add an expert-mapped or automatically-mapped alternative, and to provide variance or statistical tests for the \"significant\" language.\n\nRecommendation: engage with it as a conference/workshop submission, not a journal-shattering claim. With modest revisions, it's a solid empirical contribution.","headline":"Honest negative-result paper on label unification for cybersecurity NER; the core finding is credible, but the broad conclusion about label merging needs tempering.","tokens_in":9758,"tokens_out":2985,"would_cite":true,"duration_ms":35157,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Merging labels across cybersecurity NER datasets does not make models generalize better between datasets, and neither preserving per-dataset labels with shared weights nor graph-based transfer closes the gap.","keywords":["named entity recognition","cybersecurity","label unification","cross-dataset generalization","BiLSTM","multi-head model","graph-based transfer","span-F1"],"falsifier":"Ask cybersecurity experts to design a revised label mapping in place of the appendix's Table 8, then repeat the unified and combined training experiments with all other settings unchanged; if the expert mapping lifts cross-dataset span-F1 on APTNER and CYNER to at least the single-dataset baselines, the claim that simple label merging is not enough is falsified for practical purposes. A second check is to run LST-NER with the original method's released code; if it then clearly beats BERT-base-NER on these datasets, the reported 'no significant gain' would not be a property of the method.","tokens_in":8696,"feed_emoji":"🛡️","tokens_out":7134,"duration_ms":82131,"temperature":0.7,"pith_summary":"The paper asks whether combining four cybersecurity named-entity recognition datasets under a common coarse label set helps models transfer across datasets. It finds that it does not: models trained on unified data consistently lose to models trained and tested on the same dataset, and training on all four datasets at once lowers performance on APTNER and CYNER. The authors trace the failures to label-definition differences, annotation-span differences, and an increased bias toward predicting non-entities. Two alternatives that avoid unification, a multihead model with shared weights and a graph-matching transfer model, provide only slight or statistically insignificant gains. The paper concludes that simple label merging is not sufficient and that more targeted domain adaptation is needed.","feed_headline":"Label merging fails to boost cross-dataset NER","feed_subtitle":"Unified models trail same-dataset baselines on four cybersecurity corpora; graph transfer adds no significant gain.","key_machinery":"The load-bearing object is the hand-crafted label unification schema: every original label from APTNER, DNRTI, ATTACKER, and CYNER is mapped onto four coarse categories taken from CYNER, namely Organization, System, Vulnerability, and Malware, with labels that do not fit discarded as O. This schema creates the common label space in which the pairwise BiLSTM evaluations and the combined-training experiments are run, and it defines what counts as a cross-dataset error. The span-F1 matrix and Jensen-Shannon divergence metrics are then used to separate schema-induced errors from domain shift. The multihead and LST-NER models are designed to bypass the schema by preserving original label sets; LST-NER does this by representing label relationships as graphs and matching them with Gromov-Wasserstein distance, so that structural knowledge can transfer without label correspondence.","core_discovery":"The paper's central claim is that simple label merging is not sufficient for robust cross-dataset generalization in cybersecurity NER. In pairwise evaluations, no cross-dataset row outperformed the same-dataset diagonal, and a model trained on all four unified datasets scored below a model trained only on APTNER and only on CYNER. Qualitative analysis attributes these failures to discrepancies in what tokens and labels mean across datasets, to disagreement over span boundaries, and to a measurable increase in predicting non-entities, all of which the coarse unification amplifies. The paper then checks two model-based alternatives: a multihead model that keeps each dataset's original labels while sharing encoder weights, and the LST-NER graph-matching transfer model built on BERT-base-NER. Both leave the central picture unchanged, leading the authors to reject label-schema merging as a sufficient strategy.","pith_inferences":["Whether a better label mapping would change the conclusion is left open; the authors' own admission that their mapping is likely suboptimal suggests an expert-designed schema is the first decisive test.","The systematic precision-over-recall gap points to a testable fix the paper does not try: rebalancing toward entity tokens or adjusting the threshold for predicting non-entities could recover recall lost during unification.","Because the LST-NER model was reimplemented from scratch without the original code, the fair reading is that this implementation showed no gain; an official-code replication is the follow-up that would decide whether graph matching itself adds nothing.","The error trends could seed a diagnostic resource: a corpus of cross-dataset predictions annotated by failure type, which would let future domain-adaptation work target the specific bottleneck."],"forward_implications":["Combining datasets through a shared coarse label set will not, by itself, improve cybersecurity NER; on APTNER and CYNER the combined model scored below a model trained only on the same dataset.","Cross-dataset transfer in this domain is highly uneven: APTNER and DNRTI transfer to each other reasonably well, while CYNER and ATTACKER transfer poorly, partly due to smaller training sets.","A model's apparent cross-dataset errors can reflect label-definition differences rather than failures of entity recognition, so evaluation should distinguish definition mismatch from misidentification.","Preserving each dataset's original label set with shared encoder weights can match unified-label performance while keeping label specificity.","Graph-based cross-domain transfer on top of a general NER model offers no measurable advantage over the same model without graphs in this setting."],"supporting_citations":[{"why":"Supplies the APTNER dataset, one of the four corpora, with 21 fine-grained entity types.","marker":"(Wang et al., 2022)"},{"why":"Supplies the CYNER dataset and the four coarse label names used as the unification schema.","marker":"(Alam et al., 2022)"},{"why":"Supplies the DNRTI dataset whose labels are mapped into the unified schema.","marker":"(Wang et al., 2020)"},{"why":"Supplies the ATTACKER dataset and its fine-grained labels, which drive several unification problems.","marker":"(Deka et al., 2024)"},{"why":"Defines the LST-NER graph-matching method the paper reimplements and compares against BERT-base-NER.","marker":"(Zheng et al., 2024)"},{"why":"Identifies the lack of standard labels as a core problem motivating dataset combination.","marker":"(Gao et al., 2021a)"}],"fun_headline_variants":["Label unification can't fix cross-dataset NER","Merged labels hurt cyber NER cross-dataset gains","Unified labels fail to generalize in cyber NER","No gain from label merging in cross-dataset NER","Cross-dataset cyber NER: label merging not enough"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper's negative conclusion rests on one hand-crafted mapping of each dataset's labels onto four coarse categories, and the authors state in the appendix that this mapping is likely suboptimal because they are not cybersecurity experts; if an expert-designed mapping produced different results, the conclusion that simple merging is insufficient would not hold as stated.","fun_headline_variants_meta":{"raw":{"variants":["Label unification can't fix cross-dataset NER","Merged labels hurt cyber NER cross-dataset gains","Unified labels fail to generalize in cyber NER","No gain from label merging in cross-dataset NER","Cross-dataset cyber NER: label merging not enough"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000191,"raw_usage":{"total_tokens":1280,"prompt_tokens":819,"completion_tokens":461,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":435,"completion_tokens_details":{"reasoning_tokens":381}},"tokens_in":435,"tokens_out":461,"duration_ms":4819,"temperature":1.0,"reasoning_tokens":381,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T16:14:16.222821+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Ask cybersecurity experts to design a revised label mapping in place of the appendix's Table 8, then repeat the unified and combined training experiments with all other settings unchanged; if the expert mapping lifts cross-dataset span-F1 on APTNER and CYNER to at least the single-dataset baselines, the claim that simple label merging is not enough is falsified for practical purposes. A second check is to run LST-NER with the original method's released code; if it then clearly beats BERT-base-NER on these datasets, the reported 'no significant gain' would not be a property of the method.","supporting_citations":[{"cited_title":"CyNER: A Python Library for Cybersecurity Named Entity Recognition","cited_arxiv_id":"2204.05754","evidence_quote":"Supplies the CYNER dataset and the four coarse label names used as the unification schema."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the DNRTI dataset whose labels are mapped into the unified schema."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the ATTACKER dataset and its fine-grained labels, which drive several unification problems."},{"cited_title":"Cross-domain Named Entity Recognition via Graph Matching","cited_arxiv_id":"2408.00981","evidence_quote":"Defines the LST-NER graph-matching method the paper reimplements and compares against BERT-base-NER."}],"review_version":1}