{"id":"ad9ece4f-5efa-43d2-a944-710665e7dae7","arxiv_id":"2506.05949","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"NameTag 3, a 355M-parameter multilingual fine-tuned model with per-tagset heads and a nested entity decoder, reports state-of-the-art or competitive NER F1 on 21 datasets in 15 languages.","lead":"NameTag 3 is an open-source tool and web service for recognizing named entities like people, places, and organizations in 15+ languages, and it reports top results on many standard test sets. A practical reason to read it: it shows a compact 355M-parameter model can beat much larger LLMs on this task while running thousands of times faster.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Central SOTA claim is internally inconsistent: Table 3 shows NameTag 3 below the cited SOTA on 7 of the 21 in-domain datasets, so 'SOTA on 21 datasets' is unsupported.","rationale":"The most load-bearing concern is not the robustness of the training schedule but the veracity of the headline claim. The abstract says NameTag 3 'achieves state-of-the-art results on 21 test datasets in 15 languages', but Table 3 shows the 355M multilingual model below the cited SOTA on 7 of the 21 in-domain datasets. Even a charitable reading (counting datasets with no previous SOTA as SOTA, and including Table 7 wins) gives at most 20 datasets, not 21. The language list is also inconsistent: English and Tagalog contain both wins and losses, yet are presented as SOTA languages. This is an internal inconsistency, not a matter of statistical significance or choice of sampling temperature. The reader's weakest_assumption focused on single-run variance and the sqrt-sampling choice; those are secondary and do not affect the core discrepancy. Additionally, the reader's concern about a 'partially undisclosed pretrained backbone' is already resolved: Table 2 names XLM-R Large for the flat model and RoBERTa Large/RobeCzech Base for nested models. The SOTA-count discrepancy, however, directly falsifies the paper's primary quantitative claim. It can be fixed by rewording the claim to specify exactly which datasets and languages achieve SOTA, or by adding the missing comparisons; therefore a conditional acceptance requiring that correction is appropriate. If the authors cannot produce 21 datasets with Multi F1 above prior SOTA, the claim must be downgraded to 'competitive on most, SOTA on N.'","tokens_in":10083,"tokens_out":9991,"duration_ms":83918,"concrete_test":"Construct a reconciliation table with one row per test set (21 from Table 3 plus 7 from Table 7) containing: NameTag 3 Multi F1, cited SOTA F1, and a boolean 'beats SOTA'. Count true beats and unique languages with at least one beat. If the count is not 21 or the language set differs from the 15 listed, the abstract and Section 1 must be revised to state the actual number of datasets where NameTag 3 exceeds prior SOTA, or to use 'competitive' on the remaining datasets.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The paper's central claim—'state-of-the-art results on 21 test datasets in 15 languages' (Abstract, Introduction, Conclusion)—is contradicted by its own Table 3. For the multilingual 355M model (Multi), the cited SOTA baseline is higher on 7 of the 21 in-domain rows: Arabic OntoNotes v5 (74.20 vs 76.40), Dutch CoNLL-2002 (94.93 vs 95.70), English OntoNotes v5 (90.19 vs 92.07), English CoNLL-2003 (94.09 vs 94.60), German CoNLL-2003 (87.48 vs 88.38), Maghrebi Arabizi (84.49 vs 86.20), and Spanish CoNLL-2002 (90.29 vs 90.40). Only 11 rows exceed the cited SOTA, and 3 have no prior SOTA (Czech CNEC 2.0 CoNLL, Norwegian Bokmål, Norwegian Nynorsk). Adding the 7 out-of-domain rows in Table 7 yields at most 20 datasets with Multi F1 above (or not below) the cited SOTA, not 21. The language list is also inaccurate: English and Tagalog are listed as SOTA languages although NameTag 3 is below SOTA on English OntoNotes/CoNLL-2003 and Tagalog Ugnayan. This is not a statistical sensitivity issue; it is a direct mismatch between the reported numbers and the headline. The claim is load-bearing because the paper's contribution is framed entirely around this SOTA result.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"NameTag 3 is an open-source named entity recognition tool and web service supporting flat and nested NER. The paper describes a single 355M-parameter multilingual XLM-R model with separate per-tagset classification heads for flat NER, trained jointly on 21 corpora in three tagsets, and a 126M seq2seq model for Czech nested NER. It evaluates the system on 21 in-domain flat NER datasets, 7 out-of-domain datasets, and 3 nested NER datasets, comparing against Stanza, SpaCy, leaderboard SOTA, and DeepSeek-R1/ChatGPT models, and reports throughput measurements. The central claim is that the system achieves state-of-the-art results on 21 test datasets in 15 languages.","tokens_in":10367,"tokens_out":9767,"duration_ms":87655,"significance":"If the empirical claims were fully supported, this would be a practically significant contribution: the code is open-source (MPL 2.0), models and a running web service are released, the single-model design supports multiple tagsets and cross-lingual transfer, and the LLM comparison with throughput measurements is a useful resource trade-off demonstration. The breadth of evaluation—28 flat datasets plus 3 nested datasets—is a strength, as is the explicit reproduction script for the LLM baselines. However, the headline SOTA count is not supported by the paper's own tables, and the absence of variance or significance information makes several sub-1-point margins unreliable. The tool itself is valuable, but the paper needs an accurate and precisely qualified statement of where it exceeds, matches, or trails published results.","major_comments":[{"comment":"The claim that NameTag 3 'achieves state-of-the-art results on 21 test datasets in 15 languages' is directly contradicted by Table 3. For the Multi model, the cited SOTA baseline is higher on seven of the 21 in-domain rows: Arabic OntoNotes v5 (74.20 vs. 76.40), Dutch CoNLL-2002 (94.93 vs. 95.70), English OntoNotes v5 (90.19 vs. 92.07), English CoNLL-2003 (94.09 vs. 94.60), German CoNLL-2003 (87.48 vs. 88.38), Maghrebi UNER Arabizi (84.49 vs. 86.20), and Spanish CoNLL-2002 (90.29 vs. 90.40). Only eleven rows exceed the cited SOTA, and three rows have no prior SOTA, so at most fourteen of the 21 in-domain datasets are 'not below SOTA'. Adding the seven out-of-domain rows in Table 7, where Tagalog Ugnayan is also below SOTA, still gives only seventeen datasets above the cited baseline, not twenty-one. The language list is likewise inaccurate: English is listed as a SOTA language while both English datasets in Table 3 are below SOTA, and Tagalog is listed while Ugnayan is below SOTA. The headline must be corrected to report the actual number of datasets on which the model exceeds the cited baseline, and the criterion for calling a language a 'SOTA language' should be defined explicitly.","section":"Abstract, §1, §5.1, Table 3"},{"comment":"The SOTA comparisons mix heterogeneous baselines and are partly circular. In Table 3, the Ukrainian baseline is 'NameTag 2' (110M), the authors' own previous system, which is not an independent state-of-the-art result. Other SOTA numbers come from different sources with very different parameter scales (e.g., 1117M–1853M for the Wang et al. rows vs. 355M for Mayhew et al.), making head-to-head SOTA comparisons informative only if the training data, tagsets, and evaluation splits are shown to be equivalent. For the Mayhew et al. baseline, the table caption says the better of the 'in-language' and 'all' models is reported, but the table does not state which setting corresponds to each row. The authors should either report a single consistent baseline regime or clearly tag each baseline's source, parameter count, and training setup.","section":"§5.1, Tables 3 and 7"},{"comment":"No variance or significance information is reported. The tables give a single F1 value per system, but several SOTA margins are close to or below 1 F1 point: Multi is 0.11 below SOTA on Spanish CoNLL-2002, 0.51 below on English CoNLL-2003, 0.49 above on Portuguese UNER Bosque, and 1.10 below SOTA on Tagalog Ugnayan. Without multiple seeds, confidence intervals, or a significance test, the exact list of 'SOTA datasets' is not robust. At minimum, the paper should state whether results are single runs or averaged, and it should soften the SOTA language to 'competitive with published SOTA' where margins are within a plausible run-to-run variation.","section":"§5.1, Tables 3 and 7"}],"minor_comments":[{"comment":"The text says the training objective is a 'macro span-based F1 score with uniform weighting', while Table 3 reports span-based micro F1. Please clarify whether the macro objective is a multi-dataset average and how it relates to the micro F1 numbers reported in the results.","section":"§4.1"},{"comment":"In the paragraph following Table 4, '20 percent points higher' should be '20 percentage points higher'.","section":"§5.1"},{"comment":"The list of datasets mixes UNER, CoNLL, OntoNotes, CNEC, and Lang-uk corpora; a small table or column indicating the source benchmark and tagset for each row would improve readability and reduce ambiguity about the harmonization procedure.","section":"§3.1"},{"comment":"The Limitations section states that joint multilingual training makes NameTag 3 'less prone to biases' that affect monolingual models, but no bias evaluation is presented. This should be framed as a hope or design rationale, not as an empirical finding.","section":"Limitations"},{"comment":"The caption of Table 6 would benefit from stating explicitly that the CNEC 2.0 row uses the original 46-type/4-container annotation, whereas the Czech row in Table 3 uses the harmonized 4-label CoNLL tagset; otherwise the two Czech results can be confused.","section":"§5.2, Table 6"}],"recommendation":"major_revision","confidential_remarks":"This is a solid systems paper with a genuinely useful released tool, but the headline SOTA claim is overstated in a way that could mislead users. The authors should be asked to correct the SOTA count, specify the 'SOTA language' criterion, and add uncertainty information or soften close comparisons. With those changes, the paper would be acceptable as a demonstration of a practical multilingual NER service."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's the one thing you should know: the tool is genuinely useful, but the central SOTA claim doesn't survive contact with the paper's own Table 3, and that needs to be fixed before this is published as-is. The stress-test note is correct. For the multilingual 355M model, Table 3 shows the cited SOTA is higher on 7 of the 21 in-domain rows (Arabic OntoNotes, Dutch CoNLL-2002, English OntoNotes and CoNLL-2003, German CoNLL-2003, Maghrebi Arabizi, Spanish CoNLL-2002). Even the monolingual models don't beat SOTA on those. Add the out-of-domain rows and you get at most 20 datasets where Multi is above or not below the cited SOTA, not 21. And English and Tagalog are in the SOTA language list despite being below SOTA on some datasets. So the headline is internally inconsistent.\n\nWhat's actually new and good: the combination of per-tagset heads, a seq2seq nested decoder, and joint training over 21 datasets in 17 languages, all in one 355M model. The cross-lingual transfer numbers on unseen languages (Cebuano, Tagalog, Russian PUD, etc.) are strong, often 10+ F1 over the Mayhew et al. baselines. The LLM comparison is a nice practical result: NameTag 3 beats a 70B DeepSeek-R1 by 20 F1 on CoNLL-2003 while being thousands of times faster. The code and models are released.\n\nSoft spots, in rough order of importance. First, the SOTA claim problem above. Second, there are no standard deviations or significance tests; many of the margins over SOTA are 1-3 F1 points, and the paper treats single runs as definitive. Third, the Czech CNEC 2.0 flat evaluation is on a simplified 4-label harmonization of the corpus, so the 'no prior SOTA' there is a bit of a freebie. Fourth, the Ukrainian SOTA baseline is their own NameTag 2, which is a low bar. The square-root sampling sensitivity is a minor issue given how broad the evaluation is.\n\nThis paper should go to peer review - it's a solid systems contribution with broad evaluation and a useful artifact. But the authors need to correct the SOTA counting and language list, and ideally add some variance estimates.\n\nWho's it for: practitioners who want a single multilingual NER model with an API, and researchers working on cross-lingual transfer. I'd bring it to a reading group.","headline":"Solid systems paper with a genuinely useful tool, but the SOTA headline doesn't match its own Table 3 and needs correcting before publication.","tokens_in":62,"tokens_out":8069,"would_cite":true,"duration_ms":104832,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"One 355M model beats state of the art on 21 NER datasets","keywords":["named entity recognition","multilingual NLP","multitagset learning","nested named entities","fine-tuned language models","cross-lingual transfer","REST web service","state-of-the-art evaluation"],"falsifier":"Retrain the 355M multilingual flat model with the same data but with uniform sentence sampling (no square-root temperature) and with per-dataset micro $F_1$ weighting, then compare all 21 test datasets against the reported baselines; if several of the narrow 1–3 point wins become losses, the central claim depends on the sampling choice and would need qualification.","tokens_in":9840,"feed_emoji":"🏷️","tokens_out":7830,"duration_ms":67915,"temperature":0.7,"pith_summary":"NameTag 3 is an open-source named entity recognition tool and cloud service built around a single fine-tuned 355M-parameter language model trained jointly on 21 corpora in 17 languages. The paper's central claim is that this one modestly sized model reaches state-of-the-art span-based $F_1$ on 21 test datasets in 15 languages and stays competitive on five more, outperforming widely used toolkits on every dataset where direct comparison is possible. The same design supports flat NER with three different entity tagsets and nested NER through a separate seq2seq head, with a 126M Czech model for nested entities. The paper also claims that this fine-tuned approach beats a 70B-parameter LLM by about 20 $F_1$ points on English CoNLL-2003 while running roughly 10,000 times faster, so the contribution is both an accuracy result and a practical service.","feed_headline":"One 355M model beats state of the art on 21 NER datasets","feed_subtitle":"A fine-tuned multilingual model also beats 70B LLMs at named entity recognition and runs 10,000x faster.","key_machinery":"The central machinery is multitagset fine-tuning: one shared Transformer encoder (XLM-R Large, 355M) with a distinct softmax classification head per tagset, so the encoder and all heads are trained jointly but only the head for the requested tagset runs at inference. Training batches are formed by square-root temperature sampling, where corpora are sampled proportionally to the square root of their sentence counts, and the optimization target is a macro span-based $F_1$ averaged uniformly across datasets; that choice is what lets small corpora contribute enough to keep 21 datasets balanced. For nested NER the softmax head is replaced by a seq2seq decoder (Straková et al., 2019) that generates linearized nested labels with hard attention on the current token, and a few pre-training epochs freeze the encoder so the decoder adapts before joint fine-tuning.","core_discovery":"The core claim is that one joint multilingual model can replace a collection of per-language, per-tagset systems without sacrificing accuracy: NameTag 3's 355M flat model, with a separate softmax classification head for each of the CoNLL, UNER, and OntoNotes tagsets sharing one encoder, sets new state-of-the-art scores on datasets including Croatian UNER SET (95.55), Serbian UNER SET (97.10), Ukrainian Lang-uk (92.88), and Chinese UNER GSD (91.53), and remains within about two points of the best published results on Arabic, Dutch, German, Maghrebi, and Spanish. For nested NER, a 126M Czech model achieves 86.39 on CNEC 2.0, above the prior 83.44, while English ACE-2004 and ACE-2005 come within roughly half an $F_1$ point of the state of the art. The paper presents this as evidence that fine-tuning smaller open models is still the most resource-efficient route to accurate NER whenever training data exist.","pith_inferences":["A natural testable extension is to apply the same shared-encoder, per-task-head design to other sequence labeling tasks, such as part-of-speech tagging or chunking, where multilingual joint training could show similar gains.","The reported margins over prior state of the art are often only 1 to 3 $F_1$ points, so an ablation varying the square-root sampling temperature and the evaluation weighting would show whether the SOTA claim is robust or partly an artifact of the training schedule.","The strong zero-shot results on languages unseen during training suggest the model learns transferable span representations; probing those representations with more low-resource languages would clarify how far the transfer extends.","The nested seq2seq head is trained only on English and Czech; combining it with the multilingual flat encoder could provide nested NER for other languages as soon as nested corpora exist."],"forward_implications":["One 355M model can serve 17 languages and three tagsets from a single checkpoint, so users no longer need per-language models for these languages.","The same jointly trained model outperforms popular toolkits on every dataset where both have comparable models, which makes it a drop-in replacement for those pipelines.","When training data are available, fine-tuning a small model is claimed to beat a 70B LLM by about 20 $F_1$ points while being more than 10,000 times faster, shifting the practical default away from prompting.","Nested NER for Czech reaches state of the art with a 126M model, and English nested results are near state of the art with a 355M model, so nested annotation can be served without a larger pipeline.","Cross-lingual transfer to languages not seen in training (for example Cebuano at 96.97 and Tagalog TRG at 97.78) suggests the model generalizes beyond its training languages."],"supporting_citations":[{"why":"Supplies the Universal NER v1 corpora, the train/dev/test splits, and the previous state-of-the-art baseline that NameTag 3 must beat on most flat datasets.","marker":"Mayhew et al. (2024)"},{"why":"Provides the square-root sentence sampling strategy used to construct training batches across corpora.","marker":"van der Goot et al. (2021)"},{"why":"Provides the seq2seq linearization head for nested NER that NameTag 3 fine-tunes.","marker":"Straková et al. (2019)"},{"why":"Stanza is the popular NER toolkit whose per-language F1 scores serve as the main toolkit baseline.","marker":"Qi et al. (2020)"},{"why":"SpaCy is the other popular NER toolkit used as a direct comparison baseline.","marker":"Honnibal and Montani (2017)"},{"why":"Provides several of the leaderboard state-of-the-art baselines on Dutch, English CoNLL-2003, German, and Spanish that NameTag 3 is compared against.","marker":"Wang et al. (2021)"},{"why":"Defines the CoNLL-2012 splits used for the OntoNotes 5.0 corpora in the training and evaluation sets.","marker":"Pradhan et al. (2012)"},{"why":"Supplies the Czech Named Entity Corpus 2.0, the nested corpus for which NameTag 3 reports state-of-the-art results.","marker":"Ševčíková et al. (2007)"}],"fun_headline_variants":["One model, 21 datasets, new SOTA in NER","355M model outdoes 70B LLMs on NER","SOTA NER in 15 languages with a single small model","NameTag 3: one model to rule all NER tagsets"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The claim that one model is state of the art on 21 datasets rests on the particular square-root batch-sampling scheme and the uniformly weighted macro $F_1$ objective; if a different sampling or weighting moved several of the 1–3 point margins, the state-of-the-art statement would weaken.","fun_headline_variants_meta":{"raw":{"variants":["One model, 21 datasets, new SOTA in NER","355M model outdoes 70B LLMs on NER","SOTA NER in 15 languages with a single small model","NameTag 3: one model to rule all NER tagsets"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000455,"raw_usage":{"total_tokens":2329,"prompt_tokens":1034,"completion_tokens":1295,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":650,"completion_tokens_details":{"reasoning_tokens":1219}},"tokens_in":650,"tokens_out":1295,"duration_ms":9971,"temperature":1.0,"reasoning_tokens":1219,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T10:11:41.484391+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Retrain the 355M multilingual flat model with the same data but with uniform sentence sampling (no square-root temperature) and with per-dataset micro $F_1$ weighting, then compare all 21 test datasets against the reported baselines; if several of the narrow 1–3 point wins become losses, the central claim depends on the sampling choice and would need qualification.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the square-root sentence sampling strategy used to construct training batches across corpora."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the CoNLL-2012 splits used for the OntoNotes 5.0 corpora in the training and evaluation sets."}],"review_version":1}