{"id":"639f01e8-aeff-494c-b483-440d050b7cc9","arxiv_id":"2412.02056","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A new manually annotated English-Tamil-Sinhala parallel NER corpus of 3,835 sentences per language, with benchmarks showing XLM-R outperforms monolingual and Indic models, and a case study where NER output improves English-Sinhala NMT BLEU by about 9 points.","lead":"Researchers built a manually annotated English, Tamil, and Sinhala sentence corpus with named entities marked, the first multi-way parallel NER dataset for these languages. They show multilingual XLM-R performs best on Sinhala and Tamil NER and that feeding their NER output into an English-Sinhala translation system improves BLEU from 11.9 to 21.1.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Unverified parallel alignment and cross-language annotation consistency undermine the corpus's central novelty and the cross-lingual NER conclusions.","rationale":"The reader's weakest assumption—that the sentence pairs are faithful translations and that entity labels are comparable across languages—is precisely the load-bearing point. The paper's novel contribution is the multi-way parallel dataset, and the NER benchmark and NMT utility demonstration both inherit its validity. Section 3.3 provides no alignment validation, and Section 3.5 openly documents cross-language annotation discrepancies and mistakes. The reader's conditional verdict is appropriate because these issues are addressable through additional validation and adjudication, but they must be resolved before the dataset can be used as a trusted benchmark. I agree with the reader; no change to the verdict is needed.","tokens_in":13274,"tokens_out":4190,"duration_ms":42150,"concrete_test":"Draw a random sample of 100 sentence triples from the released corpus. Have two independent bilingual speakers (Sinhala-English and Sinhala-Tamil) rate whether the English and Tamil sentences are faithful translations of the Sinhala source, using a 3-point scale (faithful / partial / incorrect). Also have two fresh annotators re-annotate the same triples in all three languages using the published guidelines, then measure token-level label agreement (same entity type for aligned spans) against the released labels. If more than 10% of triples are judged 'incorrect' translations, or if cross-language entity-type agreement for aligned spans is below 80%, the multi-way parallel and cross-lingual comparability claims are not established.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The corpus's central claim is that it is the first multi-way parallel NE annotated corpus, and this property underwrites both the cross-lingual NER analysis and the NMT case study. Section 3.3 states that 3835 Sinhala sentences were manually filtered and the corresponding Tamil and English sentences were extracted from Fernando et al.'s parallel corpus, but no validation of sentence alignment or translation fidelity is reported. Section 3.5 then shows that entity counts do not tally across languages, with reasons including translation discrepancies (one NE in English vs. two in Sinhala) and admitted human annotation mistakes. If the extracted sentences are not faithful translations, or if annotation guidelines were applied inconsistently across languages, then: (1) the 'multi-way parallel' novelty claim is unsupported; (2) the Section 6 variance analysis attributing NER performance differences to language representation or complexity is confounded by annotation inconsistency; (3) the code-switched data used for DEEP+NER in Section 7 may contain incorrect entity spans, compromising the NMT comparison. The paper reports inter-annotator agreement on only 500 tokens per language (<0.5% of the corpus) and does not describe adjudication for the full dataset, so the reliability of the released labels is not established. This is load-bearing because the resource contribution and the benchmark results both depend on the parallel annotations being trustworthy.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript introduces and publicly releases a three-way parallel English-Tamil-Sinhala corpus of 3,835 sentences per language, annotated with CoNLL03 named entities in BIO format. It reports NER experiments with BiLSTM-CRF and several pre-trained language models, including mBERT, XLM-R, IndicBERT, and SinBERT, and claims new benchmark macro-F1 results for Sinhala and Tamil. It then presents a DEEP-style English-to-Sinhala NMT case study in which the proposed NER system is used to generate code-switched data, with a claimed BLEU improvement from 11.9 to 21.12 over a baseline. The central contributions are the resource itself, new NER benchmarks for two low-resource languages, and evidence that language-specific NER can replace an unsupported entity linker in low-resource NMT.","tokens_in":13546,"tokens_out":5395,"duration_ms":51210,"significance":"If validated, the corpus is a valuable resource for low-resource NER and cross-lingual transfer, and the paper's experimental coverage across multiple LM types is commendable. The dataset is publicly released, the annotation uses a standard tag set and scheme, IAA values are reported, and the NMT case study addresses a practically important problem. The paper also gives a credible explanation for XLM-R outperforming the language-specific SinBERT, consistent with prior observations. However, the strongest claims depend on two properties that are not yet fully established: the parallelism and translation fidelity of the extracted sentences, and the reliability of the full annotation. In addition, the NMT attribution is confounded by the simultaneous replacement of the entity linker. These issues are load-bearing but appear addressable with additional analyses, so the work is best viewed as a major-revision candidate rather than a rejection.","major_comments":[{"comment":"The multi-way parallel property is the paper's central novelty, but the only evidence for sentence alignment is the statement that corresponding sentences were extracted from Fernando et al.'s raw parallel corpus. No alignment confidence, manual validation, or error rate is reported. Table 1 then shows substantial cross-lingual discrepancies in B-tag counts, attributed to translation syntax, acronym handling, and human annotation mistakes. Because these discrepancies are not reconciled, the cross-lingual NER variance analysis in Section 6, which attributes performance differences to language representation and complexity, is confounded by annotation inconsistency. Please provide an explicit validation of alignment fidelity and a per-language adjudication summary, or substantially relax the multi-way parallel claim.","section":"Sections 3.3, 3.5, and 6"},{"comment":"The annotation procedure for the full corpus is incomplete. Two annotators per language created the labels, but no adjudication procedure is described. The IAA values (0.83, 0.89, 0.88) are computed on only about 500 tokens per language, which is under 0.5% of the corpus, so the reliability of the released labels is not established. Please describe how disagreements were resolved for the 3,835 sentences per language, and report agreement or post-adjudication correction statistics on a larger sample.","section":"Section 3.4"},{"comment":"The NMT case study is presented as evidence that the proposed NER system improves translation. However, the comparison of DEEP+SLING with DEEP+NER changes both the NER component and the entity-linking component (SLING vs. Pywikibot), so the 9.53 BLEU gain cannot be attributed to NER alone. The entity translation accuracy metric in Table 7 is also not precisely defined, including the exact matching criterion. Please add an ablation that fixes the linking method, or otherwise disentangle the two changes, and define the entity translation accuracy measure.","section":"Section 7, Table 7"},{"comment":"The best-result claims would be strengthened by reporting variance. Section 5 states that each experiment was run with three seeds and averaged, but no standard deviations or statistical significance tests are reported. Several key differences, such as XLM-R versus mXLM-R for Sinhala (87.71 vs. 88.33) and the BLEU improvement in Table 7, are reported as single point estimates. Please report per-seed results or error bars and indicate whether the main comparisons are statistically significant.","section":"Sections 5 and 6, Tables 5-7"}],"minor_comments":[{"comment":"The sentence 'the creators of XLM-R claim that it is better tuned than XLM-R' appears to contain a typo; likely it should read 'better tuned than mBERT'.","section":"Section 4.2"},{"comment":"There are several typos, including 'automtatically', 'pararell', and 'ConLL', which should be corrected in a final pass.","section":"Section 1 and throughout"},{"comment":"The term 'mXLM-R' is used without definition; please define it as the XLM-R model fine-tuned on the concatenated trilingual data.","section":"Table 5"},{"comment":"The text says results for languages not included in a pLM are grayed out, but Table 5 as printed shows numeric values for all models and languages; clarify the visual encoding or adjust the statement.","section":"Section 6, Table 5"},{"comment":"The row label 'DEEP+NER+Wiki data linking' uses inconsistent terminology; use a uniform naming scheme for the baseline, DEEP+SLING, and DEEP+NER systems.","section":"Table 7"},{"comment":"Please state the exact token counts and sentence counts for the train, validation, and test splits per language, and clarify whether the same parallel sentences appear in all three language test sets.","section":"Sections 5 and 6"}],"recommendation":"major_revision","confidential_remarks":"The paper is within scope for a resource-focused venue, and the reader's circularity concern is not, on my reading, a reason to reject: the NER benchmarks are evaluated on externally annotated data and are not forced by the paper's own equations. The main risks are the unvalidated parallel alignment and the missing adjudication details, both of which are fixable with additional analysis, and the confounded NMT comparison, which is fixable with an ablation. The self-citation pattern is transparent given that the raw corpus and SinBERT come from the same group."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the corpus is the real contribution; treat the benchmark and NMT numbers as suggestive rather than proven. The paper is worth engaging with, but the parallel alignment and annotation-adjudication gaps need to be addressed before the cross-lingual claims land.\n\nWhat's new: a manually annotated trilingual (English, Tamil, Sinhala) NER corpus in CONLL03 BIO, publicly released. That genuinely fills a gap: Sinhala public NER data wasn't BIO, Tamil relied mostly on FIRE, and nothing before was multi-way parallel. The IAA numbers (0.83–0.89) are reasonable, and the tag distributions look sensible for government documents. The benchmark sweep across mBERT, XLM-R, IndicBERT, SinBERT is standard but fine as a first pass.\n\nThe soft spots are in the downstream claims. First, the multi-way parallel property is asserted but not validated: Section 3.3 says sentences were extracted from Fernando et al.'s parallel corpus with manual filtering of Sinhala only, but no check of translation fidelity is reported for the Tamil and English side. Section 3.5 then shows entity counts don't tally across languages, with reasons ranging from legitimate syntax differences to admitted human annotation mistakes. That doesn't destroy the corpus, but it does mean the cross-lingual variance discussion in Section 6 is built on sand. Second, IAA is reported on 500 tokens per language; no adjudication is described for the full dataset. For a resource paper, that's a notable gap. Third, the NMT case study uses a weak from-scratch Transformer as baseline, reports no significance tests or standard deviations, and the entity accuracy metric counts NEs identified by their own system on the target side, which is circular. The 11.9 to 21.12 BLEU jump is eye-catching but needs a stronger baseline and error bars to be convincing.\n\nNone of this is fatal for the corpus itself. The resource is public, the annotation is plausible, and the gaps are addressable in revision. The novelty claim of being the first multi-way parallel NER corpus is plausible; the paper just needs to document the parallel alignment and adjudication more carefully.\n\nRecommendation: send it to review. The corpus deserves to be in the ecosystem. The authors should be pushed to validate the parallel alignment, report IAA over a larger sample or full adjudication, and either strengthen the NMT baseline or downplay that section.","headline":"The corpus is the real contribution; the benchmark and NMT numbers are suggestive but not yet proven.","tokens_in":14101,"tokens_out":2059,"would_cite":true,"duration_ms":18934,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper releases the first multi-way parallel English–Tamil–Sinhala NE-annotated corpus, shows an XLM-R model setting Sinhala/Tamil NER benchmarks, and uses its NER output to lift En–Si BLEU from 11.9 to 21.12.","keywords":["Named Entity Recognition","Sinhala","Tamil","multilingual language models","XLM-R","low-resource NLP","parallel corpus","neural machine translation"],"falsifier":"An independent human review of a random sample of the parallel corpus—checking that each Sinhala, Tamil, and English sentence pair really is a translation and that the annotated entity spans align—would settle whether the multi-way parallel claim holds; a second check would re-run the English–Sinhala NMT experiment with the NER-derived entity tags removed or shuffled and see whether the BLEU gain from 11.9 to 21.12 disappears.","tokens_in":13123,"feed_emoji":"🏷️","tokens_out":7171,"duration_ms":55684,"temperature":0.7,"pith_summary":"This paper argues that a single trilingual resource can reset the baseline for named entity recognition in two low-resource languages. It releases the first multi-way parallel English–Tamil–Sinhala corpus with manual NE annotations in the standard CONLL03 tag set and BIO format, then shows that fine-tuning a multilingual XLM-R model on all three languages together yields the best Sinhala and Tamil NER performance they report (macro F1 88.33 and 80.23). The same NER output, plugged into a DEEP-style English–Sinhala neural machine translation pipeline, raises BLEU from 11.9 to 21.12 where the original SLING entity linker failed. If the corpus is sound, the field gains a reusable benchmark for cross-lingual NER and a concrete demonstration that language-specific NER can replace unsupported entity linkers in low-resource NMT.","feed_headline":"First trilingual NER corpus lifts Sinhala-Tamil tagging above 80 F1","feed_subtitle":"The released dataset powers best-in-class NER for both low-resource languages and lifts En-Si BLEU from 11.9 to 21.12.","key_machinery":"The load-bearing object is the released corpus: 3,835 parallel sentences per language drawn from Fernando et al.'s English–Tamil–Sinhala government-document corpus, manually cleaned and annotated by two annotators per language with the four CONLL03 tags (PER, LOC, ORG, MISC) in BIO format. On top of it, the argument runs through fine-tuned multilingual language models, primarily XLM-R, whose cross-lingual representations let a single model trained on all three languages transfer knowledge across them. For the translation case study, the mechanism is the DEEP denoising entity pre-training procedure: named entities in Sinhala Wikidata are detected by the trained NER model, linked to English equivalents via Pywikibot, and used to create code-switched noised sentences that pre-train a Transformer before NMT fine-tuning.","core_discovery":"The paper's central claim is that a carefully filtered and manually annotated multi-way parallel corpus—the same 3,835 sentences translated across English, Tamil, and Sinhala—is sufficient to train a single multilingual NER model that outperforms language-specific models for the two low-resource languages. Fine-tuning XLM-R on the combined corpus gives macro F1 of 88.33 for Sinhala, 80.23 for Tamil, and 89.59 for English, beating a BiLSTM-CRF baseline (65.66 and 47.19 for Sinhala and Tamil) as well as mBERT, IndicBERT, and the Sinhala-only SinBERT. The authors further claim that using this NER system to identify and link entities in Sinhala Wikidata, in place of the SLING linker that does not support Sinhala, turns the DEEP NMT pre-training method from a failure (BLEU 11.59, below baseline 11.9) into a success (BLEU 21.12, entity translation accuracy 62.75% vs 49%).","pith_inferences":["If the corpus is extended to more low-resource languages with the same sentence-aligned design, the multi-way parallel property could make it a standard evaluation set for measuring how much NER quality varies with script, morphology, and language-family representation in multilingual models.","The large gap between Tamil and Sinhala NER performance (80.23 vs 88.33) despite Tamil having more pretraining data suggests that morphological complexity, not corpus size, may be the binding constraint; a controlled morphologically annotated subset could test this.","The NMT result implies that for languages without an entity linker, training a small NER model on a few thousand labeled sentences may be a cheaper route to entity translation than building knowledge-base linking infrastructure."],"forward_implications":["A single XLM-R model fine-tuned on the trilingual corpus can serve as a drop-in NER system for Sinhala and Tamil, outperforming language-specific pLMs for Sinhala.","The released corpus gives researchers a controlled test bed for cross-lingual NER, since the same sentences are annotated in three languages.","Language-specific pLMs trained on modest monolingual data (like SinBERT, 15.7M sentences) can beat traditional BiLSTM-CRF for low-resource NER.","NER output can replace a missing entity linker in DEEP-style NMT pre-training, yielding a 9+ BLEU gain over a strong baseline in English–Sinhala translation.","Entity translation accuracy improves with NER-based linking, from 49% to 62.75%."],"supporting_citations":[{"why":"Supplies the raw English–Tamil–Sinhala parallel corpus of government documents that the new dataset filters and annotates.","marker":"[57]"},{"why":"Defines the DEEP denoising entity pre-training method that the NMT case study re-implements with NER-based entity linking.","marker":"[2]"},{"why":"Provides the XLM-R multilingual language model that yields the best NER results and the multilingual fine-tuned variant.","marker":"[15]"},{"why":"Provides mBERT, the other multilingual baseline compared for NER.","marker":"[14]"},{"why":"Provides the BiLSTM-CRF architecture with affix features used as the traditional deep-learning baseline.","marker":"[16]"},{"why":"Defines the CONLL03 tag set and BIO annotation scheme that the corpus adopts.","marker":"[13]"},{"why":"The only previously public manually annotated Sinhala NE dataset, used to motivate the new resource.","marker":"[10]"},{"why":"Provides IndicBERT, the language-family-specific multilingual baseline.","marker":"[60]"},{"why":"Provides SinBERT, the language-specific Sinhala pLM baseline.","marker":"[61]"}],"fun_headline_variants":["Trilingual NER corpus: Sinhala F1 88, Tamil F1 80","Single NER model beats language-specific on 3-language corpus","NER corpus lifts Sinhala-English BLEU from 11.9 to 21.1","Public parallel NER corpus powers Sinhala, Tamil NER"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The multi-way parallel property rests on the unverified assumption that the English, Tamil, and Sinhala sentences extracted from Fernando et al.'s corpus are faithful translations of each other and that the separately annotated entity labels are comparable across the three languages.","fun_headline_variants_meta":{"raw":{"variants":["Trilingual NER corpus: Sinhala F1 88, Tamil F1 80","Single NER model beats language-specific on 3-language corpus","NER corpus lifts Sinhala-English BLEU from 11.9 to 21.1","Public parallel NER corpus powers Sinhala, Tamil NER"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00029,"raw_usage":{"total_tokens":1667,"prompt_tokens":886,"completion_tokens":781,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":502,"completion_tokens_details":{"reasoning_tokens":706}},"tokens_in":502,"tokens_out":781,"duration_ms":6559,"temperature":1.0,"reasoning_tokens":706,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T23:53:04.373393+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"An independent human review of a random sample of the parallel corpus—checking that each Sinhala, Tamil, and English sentence pair really is a translation and that the annotated entity spans align—would settle whether the multi-way parallel claim holds; a second check would re-run the English–Sinhala NMT experiment with the NER-derived entity tags removed or shuffled and see whether the BLEU gain from 11.9 to 21.12 disappears.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the DEEP denoising entity pre-training method that the NMT case study re-implements with NER-based entity linking."},{"cited_title":"Yadav, R","cited_arxiv_id":null,"evidence_quote":"Provides the BiLSTM-CRF architecture with affix features used as the traditional deep-learning baseline."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the CONLL03 tag set and BIO annotation scheme that the corpus adopts."},{"cited_title":"Manamini, A","cited_arxiv_id":null,"evidence_quote":"The only previously public manually annotated Sinhala NE dataset, used to motivate the new resource."},{"cited_title":"Kakwani, A","cited_arxiv_id":null,"evidence_quote":"Provides IndicBERT, the language-family-specific multilingual baseline."},{"cited_title":"Dhananjaya, P","cited_arxiv_id":null,"evidence_quote":"Provides SinBERT, the language-specific Sinhala pLM baseline."}],"review_version":1}