{"id":"836d1616-9ffc-4261-ad99-8f8ab4761ec8","arxiv_id":"1908.05828","paper_version":1,"verdict":"REJECT","confidence":"HIGH","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":5,"one_line_summary":"The authors claim the first neural NER for Nepali with a grapheme-level BiLSTM-CNN reaching 86.7 F1, but the evaluation uses a simplified IO tag scheme and the stated gains do not match the reported tables.","lead":"This paper builds a neural named entity recognition system for Nepali using standard BiLSTM and CNN components, and reports a new annotated news dataset. The headline performance gains are not supported by the table numbers, and the code and dataset are not yet available.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Headline relative-improvement figures contradict Table 6; no row pair yields 33–50% over the SVM baseline, and the neural gains are ~1%, not the advertised ~10%.","rationale":"Good-faith reading: the paper introduces a Nepali NER dataset and several BiLSTM/CNN variants and reports large gains over SVM and prior neural models. For that central claim to hold, the reported gain sizes must appear in the experiments. They do not. Table 6 gives a 30.9% relative gain over the SVM on OurNepali and 79.6% on ILPRL, so the '33% to 50%' range is unsupported; the claimed 'up to 10%' improvement over state-of-the-art neural models has no corresponding 10% row. I therefore select the internal inconsistency as the single most load-bearing concern. The IO tagging (Section 5.1) and word-embedding training on a corpus mixed with dataset text (Section 3.2.1) are additional, independent threats to comparability: discarding B tags merges adjacent same-type entities and makes scores non-comparable with standard BIO NER, and test-set leakage inflates neural results. Those are secondary choices that could in principle be repaired or argued; the missing numerical support for the abstract cannot be repaired by reinterpretation of the existing tables. Because the proposed check is arithmetic, it settles the issue directly. If the numbers were corrected or additional experiments supplied, the paper would still need to address IO and data leakage before its claims could be accepted; hence the reader's REJECT verdict stands.","tokens_in":8885,"tokens_out":12507,"duration_ms":126385,"concrete_test":"Recompute every relative F1 improvement from Table 6: for each dataset and each baseline, compute (best_model_F1 - baseline_F1)/baseline_F1. If no pair yields a value in [0.33, 0.50] for the SVM/feature comparison, and if no labeled 'state-of-the-art neural' comparison shows a relative gain near 10%, then the abstract's headline claim is unsupported as written and requires either corrected text or the missing experiments.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is the abstract's quantitative comparison. The paper's own Table 6 fails to support the 'relative improvement of 33% to 50%' over feature-based SVM. On OurNepali, best F1 = 86.71 versus SVM 66.26 gives (86.71-66.26)/66.26 ≈ 30.9%; on ILPRL, best F1 = 83.08 versus SVM 46.26 gives ≈ 79.6%. Neither falls in the claimed 33–50% interval. The 'up to 10% improvement over state-of-the-art neural' claim is also unmatched in the table: the relative gains over the closest neural baselines (Lample et al.) are about 1% on either dataset, and no explicitly labeled 'state-of-the-art' comparison shows a 10% gain. Thus the headline result is not reproducible from the reported evidence. This is a load-bearing internal inconsistency, independent of the IO-scheme issue (Section 5.1) and the test-contaminated word embeddings (Section 3.2.1), both of which independently undermine comparability but do not replace the need for the claimed percentages to exist somewhere in the paper.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents the first neural named entity recognition (NER) system for Nepali, built on BiLSTM architectures with CNN-based character- and grapheme-level representations, optionally augmented with POS one-hot vectors. The authors introduce a new Nepali NER dataset (OurNepali), report results on this dataset and on an ILPRL dataset, and compare against a feature-based SVM baseline and published neural baselines. The abstract claims a relative improvement of 33% to 50% over the SVM model and up to 10% over state-of-the-art neural models. The paper also reports that word embeddings were trained on a corpus mixed with the NER dataset texts and that evaluation was performed using an IO tagging scheme without B tags.","tokens_in":9195,"tokens_out":5210,"duration_ms":47548,"significance":"If the claimed results were reproducible, the paper would provide a useful low-resource benchmark: a new Nepali NER dataset, a first neural baseline, and a comparison of embedding and representation choices. The entity-wise F1 tables, the dropout analysis, and the comparison of word2vec/GloVe/FastText variants are potentially informative. However, the central performance claim is internally contradicted by the paper's own tables, and two methodological choices (IO-only evaluation and embedding training mixed with test data) undermine the validity of the reported scores. The dataset contribution is further weakened because the promised repository link is a placeholder.","major_comments":[{"comment":"The headline claim of 'relative improvement of 33% to 50% compared to feature based SVM model' is not supported by the paper's own numbers. On OurNepali, the best model achieves 86.71 versus the SVM baseline at 66.26, a relative gain of (86.71-66.26)/66.26 = 30.9%. On ILPRL, the best model achieves 83.08 versus 46.26, a relative gain of 79.6%. Neither value falls in the advertised 33-50% interval. The 'up to 10% improvement over state-of-the-art neural based model' is also unmatched: relative gains over Lample et al. with fastText are about 1.1% on OurNepali and about 1.0% on ILPRL, and the gain over Lample et al. with word2vec on OurNepali is about 0.25%. The abstract must be corrected or the experiments must be redone.","section":"Abstract; Table 6"},{"comment":"Evaluating with an IO tag scheme without B tags is not standard NER evaluation. The CoNLL-2003 evaluation script, which the paper says it uses, is designed for B/I-tagged data and identifies entities as maximal B-I sequences; with only I and O tags, adjacent entities of the same type are merged into one entity, and the model is never penalized for failing to label the beginning of an entity. This inflates F1 scores and makes the comparison with published BIO-labeled baselines (Bam et al., Ma and Hovy, Lample et al.) invalid. The ILPRL dataset reportedly follows the IOB format, so converting it to IO discards exactly the boundary information needed for a fair NER comparison.","section":"Section 5.1; Table 6"},{"comment":"The word embeddings were trained on a corpus that was 'mixed with the texts from the dataset' before training. Because the dataset includes the test split, the unsupervised embedding training has access to the test sentences, and those sentences also influence the CBOW and skip-gram representations used as model inputs. This is a test-data leakage that inflates the reported F1 scores and breaks comparability with systems using pre-trained embeddings trained on external corpora. The generalization claim of the paper is therefore not supported by the experiments as described.","section":"Section 3.2.1; Tables 3 and 6"},{"comment":"The abstract claims the proposed model 'doesn't require any hand-crafted features and no data pre-processing,' but Section 4.1 describes lemmatizing 299 post-positions with a hand-curated list and training a separate BiLSTM POS tagger to label the dataset. Table 3 shows that lemmatizing post-positions 'boosted the F1 score by almost 10%.' This is a substantial, manually supplied preprocessing step, and its effect is not separated from the neural model's contribution. The contradiction between the abstract's claim and the experimental setup should be resolved.","section":"Section 4.1; Abstract"},{"comment":"The paper promises to release its dataset and code, but the link given is 'github.com/link-yet-to-be-updated'. Since a new dataset is one of the paper's stated contributions, the absence of an actual release makes the results non-reproducible. The authors should either provide a working URL or state clearly that the resource is not yet available.","section":"Section 1; Section 4.1"}],"minor_comments":[{"comment":"There is a spelling error: 'embbeddings' should be 'embeddings'.","section":"Section 2"},{"comment":"The learning rate range is written as '(0,1, 0.01, 0.001, 0.0001)'; this appears to be missing a delimiter and should be '0.1, 0.01, 0.001, 0.0001'.","section":"Section 5.3"},{"comment":"'Since, we there was no publicly available standard Nepali NER dataset' is ungrammatical; consider 'Since there was no publicly available standard Nepali NER dataset'.","section":"Section 4.1"},{"comment":"The text says 'shown in 4' when referring to the PCA visualization; it should say 'Figure 4'.","section":"Section 3.2.1"},{"comment":"The reference to 'table 3' in Section 3.2.3 for a sample sentence conflicts with Table 3, which is a results table; the sample sentence appears to be in Figure 3.","section":"Section 3.2.3 and Section 4.1"},{"comment":"'highly inflectional like Nepali' should be 'highly inflectional languages like Nepali'.","section":"Section 7"}],"recommendation":"reject","confidential_remarks":"The core numerical claims are contradicted by the paper's own tables, and the evaluation methodology problems (IO-only scoring and embedding training mixed with the evaluation data) require a full re-run of the experiments rather than local edits. The dataset, if actually released, could be a useful community resource, but in its current form the manuscript's central claims are unsupported. A fresh submission with corrected evaluation and honest numbers would be more appropriate for this venue."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this paper is worth knowing for its dataset, not for its numbers. The authors built the first neural NER system for Nepali and a new annotated corpus, and that is a real service to low-resource NLP. But the abstract's headline improvements don't match Table 6, the evaluation uses IO tags rather than BIO, and the word embeddings were trained on a corpus that includes the test sentences. Those three problems make the reported F1 scores unreliable.\n\nWhat is genuinely new: a Nepali NER dataset with roughly 3,600 sentences, and a systematic comparison of embeddings, character vs. grapheme representations, POS one-hot vectors, and post-position lemmatization. That is useful empirical work. The finding that fastText skip-gram beats CBOW and word2vec for this language is plausible and consistent with the morphology.\n\nThe soft spots are serious. First, the abstract claims a relative improvement of 33% to 50% over the SVM baseline. Table 6 shows 86.71 vs. 66.26 on OurNepali, which is about 31% relative, and 83.08 vs. 46.26 on ILPRL, which is about 80%. Neither sits in the claimed interval. The \"up to 10%\" neural gain is also absent: the closest comparisons to Lample et al. are under 1% relative on both datasets. Second, Section 5.1 says they train on IO format, dropping B tags. Without a B tag, a model that misses the start of an entity can still score well, and published baselines were evaluated on BIO, so the comparison is apples-to-oranges. Section 4.2 says ILPRL is IOB, so the authors apparently converted it without explaining the effect. Third, Section 3.2.1 says the embedding corpus was \"mixed with the texts from the dataset\" before training. That leaks test-sentence vocabulary into representation learning and inflates the results. There is also a smaller inconsistency: Section 4.1 says lemmatizing post-positions boosted F1 by \"almost 10%,\" but Table 3's biggest raw-to-lemmatized gap is about 4 points absolute.\n\nWho this is for: anyone working on Nepali or Devanagari NER will want the dataset and the empirical map of what works. But they should not quote the F1 numbers until a revised version fixes the evaluation.\n\nRecommendation: I would not desk-reject this. The dataset is scarce and the flaws are fixable in revision. I would send it to referees with a clear instruction to require a corrected evaluation — real BIO tagging, embedding training without test data, and an abstract that matches the tables. If a revision cannot fix the numbers, then reject.","headline":"Useful dataset and first neural NER baseline for Nepali, but the headline numbers don't match the paper's own tables, and the IO tagging plus test-set embedding leakage make the evaluation unreliable.","tokens_in":9662,"tokens_out":3604,"would_cite":false,"duration_ms":34244,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"First neural named-entity recognizer for Nepali, built from grapheme-level CNN and bidirectional LSTM, reports test F1 of 86.71 and claims relative gains of 33–50% over a feature-based SVM and up to 10% over prior neural baselines.","keywords":["Nepali NER","named entity recognition","low-resource NLP","grapheme embeddings","BiLSTM-CNN","Devanagari script","fastText word embeddings","dataset creation"],"falsifier":"Re-run the experiments on both datasets using the standard BIO/IOB annotation scheme and score all models with that same scheme; if the F1 advantage over the SVM and the neural baselines falls below the claimed 33–50% and 10%, the central claim is contradicted.","tokens_in":8692,"feed_emoji":"🇳🇵","tokens_out":9487,"duration_ms":76745,"temperature":0.7,"pith_summary":"The paper claims to build the first neural named-entity recognizer for Nepali, combining a bidirectional LSTM with a CNN that operates on graphemes rather than full characters, and adding one-hot part-of-speech vectors. It also contributes a new labeled news dataset for Nepali and reports test F1 of 86.71 on that dataset and 83.08 on a second annotated dataset. Against the earlier feature-based SVM system for Nepali, the claimed relative improvement is 33–50%, and against the strongest neural models developed for other languages, up to 10%. The authors argue that grapheme-level representations and mild lemmatization of post-positions are particularly effective for a morphologically rich, low-resource language such as Nepali.","feed_headline":"Neural NER for Nepali outperforms SVM by up to 50 percent","feed_subtitle":"A grapheme-aware BiLSTM-CNN plus a new news corpus delivers 86.71 F1 for Nepali NER.","key_machinery":"The central object is the grapheme-level CNN feeding a bidirectional LSTM. Graphemes are the atomic units of the Devanagari writing system; for example, a consonant plus a dependent vowel sign forms a single grapheme. The network learns an embedding for each grapheme, applies convolution filters of sizes 3, 4, and 5, max-pools the results, and concatenates this representation with a pre-trained word embedding and a one-hot part-of-speech vector before the BiLSTM layer. This sub-word pathway is what enables the model to capture Nepali's rich morphology without manual feature engineering.","core_discovery":"The central claim is that a BiLSTM-CNN architecture reading words as sequences of graphemes—the atomic writing units of Devanagari—reaches state-of-the-art NER performance for Nepali without hand-crafted features or preprocessing. Using fastText skip-gram embeddings trained on a large Nepali corpus, the model achieves 86.71 F1 on the authors' new OurNepali dataset and 83.08 F1 on a second dataset, beating the strongest neural baselines developed for other languages by up to 10 percent and the earlier SVM-based Nepali NER by 33–50 percent. The paper attributes the gains to two design choices: sub-word awareness via grapheme CNNs, which captures inflectional morphology, and lemmatizing Nepali post-positions, which alone is reported to raise F1 by about 10 points. The authors release the dataset and code to support further work on Nepali and other Devanagari-script languages.","pith_inferences":["Re-evaluating with the standard BIO/IOB tagging scheme (which includes Beginning tags) is likely to lower the reported F1 scores; the IO scheme used in the paper can credit a system for tagging the inside of an entity without finding its start, so the exact margins over BIO-trained baselines remain uncertain.","The 33–50% improvement over the SVM baseline may shrink if the SVM is supplied with context features; the paper's SVM baseline does not use surrounding words during training.","Testing the grapheme-level representation on other scripts, such as Bengali or Gujarati, would clarify whether the gain comes from the writing system or from the particular morphology of Nepali."],"forward_implications":["The released OurNepali dataset provides a standard benchmark for future Nepali NER research, since no public neural-ready dataset existed before.","The grapheme-level CNN can be transferred to other Devanagari-script languages, such as Hindi and Bengali, which share similar writing-system morphology.","Lemmatizing post-positions is a cheap, language-specific preprocessing step that the paper shows can improve NER F1 by approximately 10 points in an inflectional language.","The BiLSTM + CNN + POS architecture is a plug-and-play recipe that can be applied to other low-resource languages with small annotated corpora."],"supporting_citations":[{"why":"Supplies the feature-based SVM baseline for Nepali NER that the paper claims to beat by 33–50%.","marker":"(Bam and Shahi, 2014)"},{"why":"Defines the BiLSTM-CRF architecture used as a strong neural baseline and re-implemented with different embeddings.","marker":"(Lample et al., 2016)"},{"why":"Defines the BiLSTM-CNN-CRF architecture used as another state-of-the-art neural baseline.","marker":"(Ma and Hovy, 2016)"},{"why":"Introduces the character-level CNN plus BiLSTM design that the paper adapts from characters to graphemes.","marker":"(Chiu and Nichols, 2015)"},{"why":"Provides the grapheme-level awareness idea for word embeddings in morphologically rich languages.","marker":"(Park and Shin, 2018)"},{"why":"Defines the CoNLL-2003 evaluation script and the IO/IOB tagging conventions used to score all models.","marker":"(Tjong Kim Sang and De Meulder, 2003)"},{"why":"Supplies the fastText sub-word embedding method that the paper trains on Nepali and shows to work best.","marker":"(Bojanowski et al., 2016)"}],"fun_headline_variants":["Grapheme-based NER for Nepali beats SVM by 50%","Nepali NER: No features, 50% SVM gain","Grapheme CNN lifts Nepali NER 50% over SVM","Nepali NER: subword graphemes top SVM by 50%","From graphemes to names: Nepali NER up 50%"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The reported gains depend on the IO tagging scheme, which omits Beginning tags; because a system that only marks the inside of an entity can still be scored correct, the comparison to BIO-based baselines may overstate the true improvements.","fun_headline_variants_meta":{"raw":{"variants":["Grapheme-based NER for Nepali beats SVM by 50%","Nepali NER: No features, 50% SVM gain","Grapheme CNN lifts Nepali NER 50% over SVM","Nepali NER: subword graphemes top SVM by 50%","From graphemes to names: Nepali NER up 50%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000266,"raw_usage":{"total_tokens":1553,"prompt_tokens":832,"completion_tokens":721,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":448,"completion_tokens_details":{"reasoning_tokens":616}},"tokens_in":448,"tokens_out":721,"duration_ms":6216,"temperature":1.0,"reasoning_tokens":616,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T13:03:06.739376+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the experiments on both datasets using the standard BIO/IOB annotation scheme and score all models with that same scheme; if the F1 advantage over the SVM and the neural baselines falls below the claimed 33–50% and 10%, the central claim is contradicted.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the feature-based SVM baseline for Nepali NER that the paper claims to beat by 33–50%."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the grapheme-level awareness idea for word embeddings in morphologically rich languages."}],"review_version":1}