{"id":"c3bea509-bc81-442a-89c4-abe8d0a585dc","arxiv_id":"2505.12454","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"The paper introduces UES and NPE, a framework that treats unlabeled-entity and noisy-entity problems separately in distantly supervised NER, and reports average F1 gains over prior baselines.","lead":"Distant supervision labels training data for name recognition using dictionaries or large language models, but the labels contain two different kinds of noise. This paper proposes a two-stage method that handles each noise type separately and reports improved F1 scores on most of nine automatically annotated NER settings.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The full NPE+UES model is not the best row in the paper's own tables (mit-movie, BC5CDR/Dict, UniNER block) and no significance tests are reported, so the central 'significant improvements' claim is unsupported.","rationale":"The paper makes two kinds of claims: a theoretical guarantee about reliable negatives (Theorem 1) and an empirical claim of consistent SOTA F1. I agree with the reader that the Theorem 1 assumption (observed positives contain no false positives and no overlapping true entities) is fragile under NEP and is quantified only on CoNLL03 (0.1%, Section 6.4). But that concern, while valid, is not the most load-bearing: if the overlap were exactly zero, the central claim would still fail because the reported tables already show the full model is not uniformly best. The claim that NPE+UES is superior is the headline result and the basis for the 'state-of-the-art' conclusion; it is contradicted by the paper's own numbers on mit-movie and BC5CDR/Dict, and the UniNER-block average favors NPE-only. The absence of significance testing matters because Table 7's run-to-run variance is comparable to the reported margins; a difference of 0.07-0.6 F1 cannot support 'significant improvements.' The omission of SANTA, a directly related baseline that also separates incomplete and inaccurate noise, further weakens the SOTA comparison. Credit is due for releasing code, using a span-based formulation, and making the UES/NPE decomposition explicit; these make the required check straightforward. My recommendation matches the reader's CONDITIONAL verdict: the paper should require significance testing, a per-setting audit, inclusion of SANTA, and a tempered abstract, but the approach is not so flawed that outright rejection is warranted. Hence I leave the verdict unchanged.","tokens_in":20386,"tokens_out":9425,"duration_ms":90006,"concrete_test":"Using the released code, re-run NPE+UES, NPE-only, UES-only, TopNeg, and Neg. Sampling+ on every dataset/annotator setting in Tables 4 and 5 with at least 10 random seeds, and compute paired bootstrap 95% confidence intervals for F1 differences. Then audit per-setting wins: if NPE+UES is not significantly better than the best baseline in a clear majority of the settings, or if it is significantly worse than NPE-only or UES-only in any setting, the headline should be narrowed from 'significant improvements on eight datasets' to a setting-specific claim.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The decisive weakness is empirical, not the theorem's overlap condition. The paper's own Tables 4 and 5 contradict the Section 6.2 statement that 'our method achieves the best F1 performance' across settings. On mit-movie (Table 5), NPE+UES scores 56.07, below DS (57.67), Neg. Sampling (57.78), Neg. Sampling+ (57.80), TopNeg (57.72), and its own NPE-only component (57.94). On BC5CDR/Dict (Table 4), NPE+UES scores 76.61, below TopNeg (77.02) and UES-only (77.21). On CoNLL03/UniNER, NPE-only (52.34) beats NPE+UES (52.00), and the UniNER-block average is led by NPE-only (51.53), not NPE+UES (51.15). Several margins are <1 F1 (e.g., +0.07 on mit-restaurant, +0.60 on CoNLL03/KB), but no paired significance test or confidence interval is reported; Table 7's variances (0.3-1.9) are the same order as these differences. The closely related SANTA baseline [28]—which also separates inaccurate and incomplete annotation noise—is cited but omitted from comparisons. Even if the Theorem 1 overlap concern were fully resolved, the central empirical claim would remain unsupported.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper formalizes two sources of noise in distantly supervised named entity recognition—the unlabeled-entity problem (UEP) and the noisy-entity problem (NEP)—and proposes a two-stage span-selection framework. In the first stage, the model is warmed up using cross-entity negatives Nce, which Theorem 1 shows are free of false negatives under a pure span-based UEP with non-overlapping true entities. In the second stage, training uses confident negatives selected by the model and confident positives selected by a class-confidence threshold (NPE). Experiments compare the method with several baselines on CoNLL03, Webpage, BC5CDR, mit-movie, and mit-restaurant annotated by KB-matching, dictionary matching, ChatGPT, and UniNER. The paper additionally contributes five new LLM-annotated datasets and an analysis of noise transition matrices across annotation methods.","tokens_in":20718,"tokens_out":9037,"duration_ms":76199,"significance":"The paper makes several worthwhile contributions: it provides a clean separation of UEP and NEP that is well motivated by the noise transition matrices in Fig. 2; Theorem 1 is valid under its stated assumptions; the synthetic UEP experiments (Table 8) show large gains at high mask ratios; NPE is hyperparameter-free by design; and the authors release code and data. If the full NPE+UES method were consistently superior to state-of-the-art baselines, the paper would be a strong contribution to DS-NER. However, the empirical evidence as reported does not sustain the central claim: the full method is not the best row in several settings in the paper's own tables, and no significance tests are provided. The contribution is therefore real but the paper's headline claim needs substantial revision and additional evidence.","major_comments":[{"comment":"The sentence 'On all the datasets under various settings, our method achieves the best F1 performance among all baselines' is contradicted by the reported numbers. NPE+UES is not the best F1 in Webpage/KB (68.56 versus UES 70.01), BC5CDR/Dict (76.61 versus UES 77.21 and TopNeg 77.02), CoNLL03/UniNER (52.00 versus NPE 52.34), or mit-movie (56.07 versus DS 57.67, Neg. Sampling+ 57.80, TopNeg 57.72, and NPE 57.94). The abstract's claim of significant improvements on eight real-world distant supervision datasets is therefore not supported by the tables; at most four of the eight settings favor the full method. In addition, no paired significance tests or confidence intervals are reported, and the variances in Table 7 (0.3–1.9) are the same order as several reported gains (e.g., +0.60 on CoNLL03/KB), so the 'significant improvements' wording is not justified.","section":"Section 6.2, Tables 4 and 5"},{"comment":"Theorem 1 establishes Nce∩Nfal=∅ under the assumptions of pure UEP and non-overlapping true entities. These assumptions are violated in the real datasets on which the method is evaluated: ChatGPT and UniNER annotations exhibit substantial NEP (Fig. 2), and under NEP a noisy positive span can overlap a false-negative span and put that false negative into Nce. Section 6.4 acknowledges the resulting 'semi-labeled' cases and reports that they constitute only 0.1% of negatives in CoNLL03, but no analogous measurement is given for Webpage, BC5CDR, mit-movie, or mit-restaurant. Since the UES warm-up (Section 5.1.1, Eq. 17) relies on Nce being reliable, the theoretical guarantee does not currently cover the actual deployment conditions, and the empirical support for the assumption is limited to one dataset.","section":"Theorem 1 and Section 6.4"},{"comment":"The closely related SANTA method is discussed in Section 2.1 as a prior approach that also addresses inaccurate and incomplete annotation noise with separate strategies, which is conceptually the closest competitor to the proposed UEP/NEP decomposition. However, SANTA is absent from the baseline list in Section 6.1.3 and from Tables 4 and 5. Omitting this comparison makes the claim of superiority over current state-of-the-art methods incomplete.","section":"Section 2.1 and Section 6.1.3"}],"minor_comments":[{"comment":"The equations use 'NER' and 'NEP' as abbreviations for recall and precision of noisy entities, which collides with the paper's established abbreviations for named entity recognition and noisy-entity problem. Please rename to avoid confusion.","section":"Section 6.3.2, Eq. (18)"},{"comment":"'Unlabled' should be 'Unlabeled'.","section":"Section 6, first sentence"},{"comment":"The caption says 'in two datasets' but the table appears to report only CoNLL03 classes; please clarify which datasets are included and whether the optimal-threshold analysis was run on other datasets.","section":"Table 6 caption"},{"comment":"Calling Nce an 'unbiased negative set' is potentially misleading. Theorem 1 shows it is free of false negatives, but it is not a representative sample of all true negatives; Section 4.2.2 itself notes that training only on Nce hurts F1 because the sampled distribution is incomplete. A term like 'false-negative-free' would be more precise.","section":"Section 5.1.1"},{"comment":"The section says all hyper-parameters are tuned on the development set, but it is not stated whether the baselines were retuned under the same protocol; please specify this for a fair comparison.","section":"Section 6.1.5"}],"recommendation":"major_revision","confidential_remarks":"I would advise the editor that the empirical overclaim in Section 6.2 is the main obstacle. The paper has a solid framework and useful datasets, but the headline needs to be reframed, the missing SANTA baseline should be added, and significance testing should be provided. The 0.1% semi-labeled measurement should also be extended to all datasets before the UES theoretical guarantee can be seen as supporting the real-world results."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Punchline: the UEP/NEP framing and the five new LLM-supervised datasets are genuinely useful, but the paper's central empirical claim is not supported by its own results.\n\nWhat is new: the paper separates unlabeled-entity noise from noisy-entity noise and gives each a targeted mechanism. The reliable-negative warm-up is justified by a clean theorem: under span-based UEP and non-overlapping entities, cross-entity negatives contain no false positives. The two-stage design—warm-up on reliable negatives, then confident-negative sampling plus confidence-based pruning of noisy positives—is sensible, and the method is not circular: thresholds are computed from the model's own predictions, not tuned on test labels. The noise-transition analysis across KB, dictionary, ChatGPT, and UniNER annotations is informative, and the new datasets are a real asset. The synthetic masking experiments with FNR/FNP give a useful diagnostic view of how the model discriminates false negatives.\n\nSoft spots: the empirical evidence does not match the abstract. On mit-movie, NPE+UES scores 56.07, below the direct DS prediction (57.67), below several baselines, and below its own NPE-only component (57.94). On BC5CDR/Dict, UES-only (77.21) and TopNeg (77.02) both beat NPE+UES (76.61). In the UniNER block, the average F1 of NPE-only (51.53) is higher than that of NPE+UES (51.15). The paper reports no significance tests, and the variances in Table 7 (0.3–1.9) are the same order of magnitude as these differences. It also cites SANTA—a closely related method that separates inaccurate and incomplete annotation noise—but never compares against it. The theorem's overlap assumption is measured only on CoNLL03 (0.1%); for the other datasets it is unquantified, though this is secondary. And calling eight dataset-annotator pairs 'eight datasets' is a bit generous.\n\nWho this is for: researchers working on DS-NER, particularly with LLM-generated labels. The datasets and the UEP/NEP decomposition are worth engaging with, and the code and data are shared. But the current write-up oversells the results.\n\nRecommendation: I would send this to peer review, not desk reject. The ideas and datasets justify referee time. The referee should require significance testing, a SANTA comparison, and a rewrite of the claims so the abstract matches the tables.","headline":"Useful framework and reusable LLM-supervised datasets, but the headline empirical claim is undercut by the paper's own tables and missing significance tests.","tokens_in":21260,"tokens_out":5096,"would_cite":true,"duration_ms":45530,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper argues that distant-supervision noise in named entity recognition splits into two problems with different remedies, and that solving each separately beats state-of-the-art baselines on eight datasets.","keywords":["distantly supervised named entity recognition","unlabeled entity problem","noisy entity problem","span-based negative sampling","confident learning","large language model annotation","noise transition matrix","positive-unlabeled learning"],"falsifier":"Count the fraction of semi-unlabeled spans—true entities that overlap but are only partially covered by an observed positive span—in the Webpage, BC5CDR, mit-movie, and mit-restaurant distant annotations. If that fraction is far above the 0.1% reported for the news dataset, the Theorem 1 guarantee that cross-entity negatives contain no false negatives would fail precisely in the regimes where the method claims generality.","tokens_in":20203,"feed_emoji":"🏷️","tokens_out":6983,"duration_ms":60312,"temperature":0.7,"pith_summary":"Distantly supervised named entity recognition builds training data by matching text against dictionaries or by asking large language models to label text, and both routes inject label noise. The paper's central claim is that this noise is not a single problem: entities that are missed entirely (the unlabeled-entity problem) differ in origin and cure from entities that are tagged with the wrong type (the noisy-entity problem), and the mix differs between rule-based and LLM annotators. It proposes a two-stage span-selection framework that warms up the model on a provably clean set of negative spans and then filters both negatives and positives by the model's own confidence, with no extra hyper-parameters. On eight real-world datasets spanning three data sources and four annotation techniques, the method reports higher F1 than current baselines. If the claim holds, DS-NER can be made substantially more robust regardless of whether the distant labels come from dictionaries or from large language models.","feed_headline":"Two cures for noisy distant annotations lift NER on eight datasets","feed_subtitle":"Provably clean warm-up negatives plus confidence pruning beat baselines from dictionary and LLM annotators.","key_machinery":"Two constructions carry the argument. The first is the cross-entity negative set $N_{ce}$, the set of negative spans that overlap some observed entity span; Theorem 1 states that under the span-based unlabeled-entity setting $N_{ce} \\cap N_{fal} = \\emptyset$, i.e., the warm-up negatives are free of false negatives. The second is the class-wise self-confidence threshold $t_l$, the average predicted probability of the observed class over all positive spans of that class; it prunes noisy positives without introducing hyper-parameters, and the paper shows its estimate stays within 0.05 of the optimal threshold on the news dataset.","core_discovery":"The central discovery is that the two noise types can be separated and handled by different mechanisms within one training loop. Under span-based settings, the paper proves (Theorem 1) that cross-entity negatives—spans that overlap an observed entity span—are guaranteed to contain no false negatives, because a false negative is itself a true entity and true entities are assumed not to overlap. That makes the cross-entity set a safe warm-up set. After the warm-up, the model's own predictions define a confident-negative set, and a class-wise average self-confidence threshold defines a confident-positive set; noisy positives are pruned whenever their predicted probability for the observed class falls below the class average. The paper reports that this combined procedure outperforms existing baselines on all tested settings, and that the two components are flexible enough to be used selectively when the dominant noise type is known.","pith_inferences":["A direct extension would measure the semi-unlabeled overlap rate on every dataset used here; the paper reports 0.1% on one dataset, and the UES guarantee would need revision if the rate is materially higher elsewhere.","The same UEP/NEP decomposition could be applied to other span-level information extraction tasks under distant supervision, such as relation or event argument extraction, where dictionary and LLM annotators also produce both missed mentions and wrong-type mentions.","The noise transition matrices in the paper suggest a practical decision rule: inspect the matrix to estimate which noise type dominates, then switch on UES, NPE, or both, converting the framework into a configurable denoising pipeline."],"forward_implications":["Because the noisy-positive threshold is computed from the model's own predictions, the framework can be applied to rule-based and LLM-generated annotations without per-dataset threshold tuning.","Synthetic experiments masking up to 90% of entities show the warm-up on clean negatives keeps F1 high where ordinary training collapses, so the UES component addresses severe unlabeled-entity noise directly.","The two components can be used independently: the paper observes that on dictionary-annotated biomedical data, where wrong-type noise is almost absent, applying noisy-positive elimination can hurt, so a practitioner can deploy UES alone when NEP is known to be rare.","Training time per epoch remains modest compared with heavier denoising baselines, since only one model is trained and negative sampling is capped."],"supporting_citations":[{"why":"Supplies the dictionary-based distant annotation setup and the main robust-training baseline the paper must beat.","marker":"[8]"},{"why":"Provides the span-based negative sampling formulation and the unlabeled-entity analysis the paper builds on.","marker":"[15]"},{"why":"Introduces weighted negative sampling, the improved baseline for unlabeled-entity handling.","marker":"[16]"},{"why":"Proposes top-negative selection, the strongest negative-sampling baseline in the comparison.","marker":"[17]"},{"why":"Provides the count-rank-prune confident-learning procedure that the noisy-positive elimination adapts.","marker":"[43]"},{"why":"Defines the noise transition matrix used to measure UEP and NEP distributions across annotation methods.","marker":"[11]"},{"why":"Introduces the task-specific large language model used as one of the four distant annotators.","marker":"[6]"},{"why":"Provides the domain-dictionary annotation method used for the biomedical dataset.","marker":"[10]"}],"fun_headline_variants":["Two noise cures for distant NER: safe negatives plus pruning","Provably clean warm-up and confidence pruning fix noisy NER","Separating two noise types lifts DS-NER on eight datasets","Safe cross-entity negatives plus self-confidence pruning win"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The reliable warm-up negatives are truly negative only if every observed positive span is a genuine entity and no genuine entity overlaps another; when an annotator labels only part of an entity, a false negative can slip into the reliable set, and the paper measures that slippage on only one dataset.","fun_headline_variants_meta":{"raw":{"variants":["Two noise cures for distant NER: safe negatives plus pruning","Provably clean warm-up and confidence pruning fix noisy NER","Separating two noise types lifts DS-NER on eight datasets","Safe cross-entity negatives plus self-confidence pruning win"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000516,"raw_usage":{"total_tokens":2473,"prompt_tokens":881,"completion_tokens":1592,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":497,"completion_tokens_details":{"reasoning_tokens":1522}},"tokens_in":497,"tokens_out":1592,"duration_ms":12376,"temperature":1.0,"reasoning_tokens":1522,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T20:33:15.071362+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Count the fraction of semi-unlabeled spans—true entities that overlap but are only partially covered by an observed positive span—in the Webpage, BC5CDR, mit-movie, and mit-restaurant distant annotations. If that fraction is far above the 0.1% reported for the news dataset, the Theorem 1 guarantee that cross-entity negatives contain no false negatives would fail precisely in the regimes where the method claims generality.","supporting_citations":[{"cited_title":"Bond: Bert-assisted open-domain named entity recognition with distant supervision,","cited_arxiv_id":null,"evidence_quote":"Supplies the dictionary-based distant annotation setup and the main robust-training baseline the paper must beat."},{"cited_title":"Empirical Analysis of Unlabeled Entity Problem in Named Entity Recognition","cited_arxiv_id":"2012.05426","evidence_quote":"Provides the span-based negative sampling formulation and the unlabeled-entity analysis the paper builds on."},{"cited_title":"Rethinking negative sampling for handling missing entity annotations,","cited_arxiv_id":null,"evidence_quote":"Introduces weighted negative sampling, the improved baseline for unlabeled-entity handling."},{"cited_title":"Better sampling of negatives for distantly supervised named entity recognition","cited_arxiv_id":null,"evidence_quote":"Proposes top-negative selection, the strongest negative-sampling baseline in the comparison."},{"cited_title":"Confident learning: Esti- mating uncertainty in dataset labels,","cited_arxiv_id":null,"evidence_quote":"Provides the count-rank-prune confident-learning procedure that the noisy-positive elimination adapts."},{"cited_title":"Dual t: Reducing estimation error for transition matrix in label- noise learning,","cited_arxiv_id":null,"evidence_quote":"Defines the noise transition matrix used to measure UEP and NEP distributions across annotation methods."},{"cited_title":"Universalner: Targeted distillation from large language models for open named entity recognition,","cited_arxiv_id":null,"evidence_quote":"Introduces the task-specific large language model used as one of the four distant annotators."},{"cited_title":"Learning Named Entity Tagger using Domain-Specific Dictionary","cited_arxiv_id":"1809.03599","evidence_quote":"Provides the domain-dictionary annotation method used for the biomedical dataset."}],"review_version":1}