{"id":"08c8f80f-7c1f-42cc-a71b-e3ca0b0ef70e","arxiv_id":"2411.15734","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"Nepali-language BERT, RoBERTa, and GPT-2 models trained on a new 27.5 GB corpus score 95.60 on Nep-gLUE, beating the prior best by about two points.","lead":"Researchers built a 27.5 GB Nepali text corpus and used it to train BERT, RoBERTa, and GPT-2 models for Nepali, beating earlier benchmarks. This matters because Nepali, spoken by over 32 million people, has few dedicated language models.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Test-set hyperparameter selection (Sec. 6.1) makes the 95.60 Nep-gLUE score an inflated maximum, invalidating the SOTA comparison.","rationale":"I focused on the evaluation protocol because the central empirical claim is a state-of-the-art number (95.60) and a 2.05-point advantage over the prior best. The paper's own Section 6.1 explicitly chooses the best model on the test set, which is a well-known source of optimistic bias in fine-tuning; with 5 learning rates and up to 8 epoch choices per task, the expected maximum over these runs exceeds the true performance. Since Table 1 compares against published scores that did not use test-set selection, the comparison is not apples-to-apples. This concern does not require speculative assumptions about data overlap, although the corpus/benchmark overlap risk described by the reader would make the inflation worse. The check I propose would settle whether the margin survives a clean validation-based protocol. Because the issue is real but potentially fixable, my read does not change the reader's CONDITIONAL verdict: the paper should be accepted only if the authors re-run the evaluation cleanly and release the models and corpus for verification.","tokens_in":9011,"tokens_out":10511,"duration_ms":93588,"concrete_test":"Re-run Nep-gLUE fine-tuning for BERT and RoBERTa with the same hyperparameter grid (5e-5, 4e-5, 3e-5, 2e-5, 1e-5; 3–10 epochs) but select the best checkpoint on a held-out validation split of each task; then evaluate that single checkpoint on the official test set and average over 3 seeds. If the resulting average score is not above NepBERTa's 93.55 by more than the seed variance, the claimed margin is an artifact of test-set selection.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 6.1 states: 'For each task, we selected the best-performing model on the test set.' The reported Nep-gLUE score of 95.60 is therefore the maximum over fine-tuning runs with five learning rates (5e-5 to 1e-5) and 3–10 epochs, where each run is scored directly on the test set. Selecting the best model on the test set biases each of the four task scores upward, so the composite is not an unbiased estimator of model quality; the baseline scores in Table 1 (NepBERTa 93.55, XLM-R 92.11) were not produced under this protocol and likely used a single run or validation-based selection. The 2.05-point margin over NepBERTa could thus be largely or entirely selection bias. The decontamination concern raised by the reader compounds this: the pretraining corpus and Nep-gLUE both draw on Nepali news sources, and no overlap check is reported (Section 3.1 only excludes other corpora, not the benchmark), so memorization may further inflate the score. But the test-set selection is already sufficient to undermine the central claim of a fair state-of-the-art comparison.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper collects a new 27.5 GB Nepali news corpus, pre-trains BERT, RoBERTa, and GPT-2 models for Nepali, performs instruction tuning on the GPT-2 model, and evaluates the resulting systems on the Nep-gLUE benchmark and an abstractive summarization task. The authors report that their RoBERTa model achieves 95.60 on Nep-gLUE, surpassing existing models, and that their GPT-2-Instruct model obtains the highest ROUGE scores in summarization. The main contributions claimed are the largest monolingual Nepali corpus to date, new monolingual encoder and decoder models, and an initial exploration of instruction tuning for Nepali.","tokens_in":9215,"tokens_out":2343,"duration_ms":20593,"significance":"If the reported results are reliable, the work is useful for Nepali NLP: the corpus is a substantial public resource, the encoder models improve on an established benchmark, and the decoder and instruction-tuned variants address an underserved area. The paper also contributes the first Nepali GPT-2-style model and a concrete recipe for pretraining with TPU resources. The central state-of-the-art claims, however, rest on an evaluation protocol that selects models directly on the test set and on a very small generation baseline, so the magnitude of the claimed improvements is not yet established. The resource and models would still be valuable even if the exact scores are revised after a more rigorous evaluation.","major_comments":[{"comment":"The evaluation protocol selects, for each Nep-gLUE task, the best-performing fine-tuned model directly on the test set (\"For each task, we selected the best-performing model on the test set\"). Since the search spans five learning rates and 3–10 epochs, the reported 95.60 is the maximum over many runs on the test data, which inflates the estimate. The baseline scores in Table 1 (e.g., NepBERTa 93.55) were not produced under this protocol, so the 2-point margin over the previous state of the art cannot be interpreted as a fair comparison. The authors should instead select hyperparameters on a validation split and report test performance for the chosen configuration, together with variance across seeds.","section":"Section 6.1"},{"comment":"No decontamination check is reported between the pretraining corpus, which was scraped from the top 99 Nepali news websites, and the Nep-gLUE tasks, which are also derived from Nepali news text. Section 3.1 only excludes other existing corpora to avoid duplication; it does not verify that Nep-gLUE instances are absent from the pretraining data. If even a fraction of the benchmark appears in the 27.5 GB corpus, the 95.60 score would be inflated by memorization. The authors should report an n-gram overlap analysis or another explicit decontamination test between the pretraining corpus and the benchmark.","section":"Section 3.1 / Section 6.1"},{"comment":"The summarization comparison uses a single baseline, distilgpt-nepali, which has 88.2M parameters while the proposed GPT-2 models have 124M. A capacity difference of this size can by itself explain the ROUGE gains, so the conclusion that the models \"outperform existing models on text generation tasks\" is not yet supported. The evaluation also appears to use a single 80/20 split with no multiple runs or statistical significance testing. A comparison against a same-size or multilingual GPT-2 baseline, and ideally multiple seeds with standard deviations, would be needed to substantiate the generation claim.","section":"Section 6.2 / Table 2"}],"minor_comments":[{"comment":"There is a typo in the sentence \"For our study. we pretrained a single BERT...\" — the period should be a comma.","section":"Section 4"},{"comment":"Table 1 lists \"NepBERT (Pudasaini et al., 2023)\" and \"NepaliBERT (Rajan, 2021)\", but Section 2.2 attributes NepaliBERT to Pudasaini et al. (2023). The naming is inconsistent and should be corrected to avoid confusion about which model is being compared.","section":"Section 7"},{"comment":"The text says \"resulting in average ROGUE scores\" — the metric is ROUGE, not ROGUE. The same section also cites Lin (2004) and Lin and Och (2004) for ROUGE, which appears to be a duplicate citation.","section":"Section 6.2"},{"comment":"There is an extra closing parenthesis in \"(Kafley, 2024))\" in the instruction-tuning dataset description.","section":"Section 3.1"},{"comment":"The table does not define the abbreviations CC and CPS for the task columns, which makes it harder to interpret the per-task scores.","section":"Table 1"}],"recommendation":"major_revision","confidential_remarks":"The core resource contribution is solid and the models are likely useful to the Nepali NLP community, but the headline SOTA claims are not supported by the current evaluation protocol. The test-set hyperparameter selection is the most serious issue and should be fixed in revision; the decontamination and baseline-size concerns also need to be addressed before the paper can be recommended for acceptance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague—\n\nTwo things up front. The resource work here is real: a 27.5 GB Nepali corpus, the largest reported, plus BERT, RoBERTa, and the first Nepali GPT-2, including an instruction-tuned variant. That matters for anyone doing low-resource Indic or Nepali NLP. But the headline result—95.60 on Nep-gLUE, 2 points over the previous best—does not survive reading Section 6.1.\n\nThe paper is mostly a corpus-and-models paper, and that part is solid. They scraped 99 news sites, deduplicated, cleaned, and deliberately excluded existing corpora to avoid overlap. Tokenizers are trained from scratch. Pretraining follows standard recipes, and the training curves look plausible. They also report the hallucination-on-long-contexts behavior honestly, which is more than many papers do.\n\nThe soft spot is the evaluation. Section 6.1 states they selected the best-performing model on the test set for each task, sweeping five learning rates and 3–10 epochs. That turns the reported score into a maximum over the test set, not an estimate of model quality. The baselines in Table 1 were not produced under that protocol, so the 2-point margin is not a fair comparison. No error bars or seed variance either. And because the pretraining corpus and Nep-gLUE both draw on Nepali news sources, the lack of any decontamination check could further inflate the score—though the test-set selection alone is enough to disqualify the SOTA claim.\n\nThe generation evaluation is weaker still: one baseline, DistilGPT2, which is smaller (88.2M vs 124M), and no error bars. The ROUGE improvements are directionally plausible but far from conclusive. Also, the paper does not say whether the corpus or model weights are released. For a resource paper, that is a big omission.\n\nThere are also minor presentation issues—typos, inconsistent citations, \"ROGUE\" for ROUGE—that an editor would clean up.\n\nFor whom: researchers working on Nepali or low-resource language models, who could use the corpus or weights if they appear. Not a methodological advance; it is a new application plus a new resource.\n\nRecommendation: yes, send it to peer review, but with a major-revision request. Reviewers should insist on validation-based model selection or error bars, a decontamination check against the benchmark, and an explicit statement about artifact release. If the authors do that, the resource claim stands and the paper earns its place.","headline":"Real corpus and models for Nepali, but the SOTA claim dissolves under test-set hyperparameter selection.","tokens_in":9717,"tokens_out":3339,"would_cite":false,"duration_ms":30050,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A new 27.5 GB Nepali corpus and three pre-trained transformers beat all prior Nepali models, scoring 95.60 on Nep-gLUE.","keywords":["Nepali language models","pre-trained transformers","Nep-gLUE","low-resource NLP","BERT","RoBERTa","GPT-2","instruction tuning"],"falsifier":"A decontamination scan comparing Nep-gLUE instances against the 27.5 GB corpus, or a re-evaluation with hyperparameters chosen on a held-out validation split rather than the test set, would determine whether the 95.60 and ROUGE gains are genuine.","tokens_in":8814,"feed_emoji":"🇳🇵","tokens_out":4693,"duration_ms":37186,"temperature":0.7,"pith_summary":"This paper sets out to close the resource gap for Nepali, a language of roughly 32 million speakers, by building the largest monolingual corpus to date — 27.5 GB scraped from 99 news websites — and pre-training three Transformer models on it: BERT, RoBERTa, and GPT-2, plus an instruction-tuned GPT-2 variant. On the Nep-gLUE benchmark the authors report that their RoBERTa reaches 95.60, two points above the prior best model, and that their GPT-2-Instruct reaches 20.42 ROUGE-1 on abstractive summarization. The claim is that scale and language-specific tokenization, rather than architectural novelty, are what let monolingual Nepali models surpass both multilingual and earlier monolingual baselines. If correct, this gives Nepali NLP a much stronger pretraining foundation and its first competitive decoder-based generation baselines.","feed_headline":"New Nepali models top 95.60 on Nep-gLUE benchmark","feed_subtitle":"Three pre-trained transformers built from a 27.5 GB corpus beat every prior Nepali model on understanding and generation.","key_machinery":"The load-bearing object is a 27.5 GB monolingual Nepali corpus collected by scraping 99 news websites and deduplicated, cleaned, and normalized down from 30.4 GB, with two BPE tokenizers of 30,522 and 50,256 subwords. On top sit three pretraining setups: BERT and RoBERTa (110M parameters, masked language modeling, 400k steps) and GPT-2 (124M, causal language modeling, 500k steps), followed by supervised instruction tuning of GPT-2 on a cleaned 40k-row Nepali Alpaca set. The corpus feeds both the tokenizers and the pretraining; the instruction set feeds the Instruct variant; the evaluation harness is Nep-gLUE for encoders and a 7,258-pair summarization set for decoders.","core_discovery":"The paper's central claim is that a 2.4x larger Nepali corpus plus architecture-appropriate tokenizers yields state-of-the-art understanding and generation for Nepali. Concretely, the authors' RoBERTa scores 95.60 on Nep-gLUE (NER 93.74, POS 97.52, text classification 94.68, pair similarity 96.49), beating the previous best NepBERTa by 2.05 points, and both BERT (95.18) and RoBERTa outperform every multilingual and monolingual comparator on each task. On generation, their 124M GPT-2 and GPT-2-Instruct models beat the existing distilgpt-nepali on all three ROUGE metrics, with GPT-2-Instruct reaching 20.42 ROUGE-1, 15.89 ROUGE-2, and 17.76 ROUGE-L. The authors present the corpus itself — deliberately scraped from scratch to avoid overlap with prior news-based datasets — and the instruction-tuned variant as resources for future Nepali NLP.","pith_inferences":["If the corpus and benchmark are confirmed disjoint, the results imply data scale is the dominant factor: the 2.05-point jump over NepBERTa came without architectural changes, only more data and a different tokenizer.","A natural next test is whether further scaling the corpus or instruction set continues to improve Nepali generation, and whether longer training sequences fix the hallucination the authors observed at inputs over 400 tokens.","The same corpus-building recipe could transfer to other under-resourced SOV languages, though the paper does not test this.","Reporting per-task test-set variance and contamination ratios would let the community assess whether the 95.60 margin is robust across seeds."],"forward_implications":["Nepali NLP now has a monolingual corpus roughly 2.4x larger than any prior one, giving future work a stronger pretraining base.","Encoder-based models for Nepali set a new state-of-the-art of 95.60 on Nep-gLUE, surpassing both multilingual models and all earlier monolingual models.","A Nepali GPT-2 and its instruction-tuned variant provide the first strong decoder baselines for Nepali text generation, nearly doubling ROUGE-1 over the existing Nepali GPT baseline.","Instruction tuning on a cleaned 40k-row Nepali instruction set improves summarization over the base GPT-2, suggesting monolingual instruction tuning works for Nepali.","The gap between multilingual and language-specific models on Nep-gLUE narrows further, supporting the value of dedicated monolingual modeling for low-resource languages."],"supporting_citations":[{"why":"Supplies the Nep-gLUE benchmark, the prior best NepBERTa baseline, and the corpus size comparison.","marker":"Timilsina et al. 2022"},{"why":"Supplies the BERT architecture and the MLM/NSP pretraining objectives used for the encoder models.","marker":"Devlin et al. 2019"},{"why":"Supplies the RoBERTa setup, including the MLM-only objective and training configuration.","marker":"Liu et al. 2019"},{"why":"Supplies the GPT-2 124M architecture and causal language modeling objective for the decoder.","marker":"Radford et al. 2019"},{"why":"Supplies the publicly available Nepali Alpaca instruction dataset used for instruction tuning.","marker":"Kafley 2024"},{"why":"Supplies the abstractive summarization dataset used to evaluate the GPT-2 models.","marker":"Bhandari 2024"},{"why":"Supplies the BPE subword tokenization method used for the Nepali tokenizers.","marker":"Sennrich et al. 2016"},{"why":"Supplies the NepaliBERT monolingual model used as a baseline comparator.","marker":"Pudasaini et al. 2023"},{"why":"Supplies the existing distilgpt-nepali model used as the generation baseline.","marker":"Maskey 2022"},{"why":"Supplies the XLM-R multilingual model used as a baseline comparator.","marker":"Conneau et al. 2020"}],"fun_headline_variants":["Nepali models hit 95.60 on Nep-gLUE with 27.5GB corpus","2.4x bigger Nepali corpus powers SOTA transformer trio","GPT-2 and RoBERTa beat all prior Nepali NLP models","New Nepali transformers: 95.60 Nep-gLUE, better ROUGE"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The reported scores assume the pretraining corpus is disjoint from the Nep-gLUE benchmark, and that choosing the best checkpoint directly on the test set does not inflate results.","fun_headline_variants_meta":{"raw":{"variants":["Nepali models hit 95.60 on Nep-gLUE with 27.5GB corpus","2.4x bigger Nepali corpus powers SOTA transformer trio","GPT-2 and RoBERTa beat all prior Nepali NLP models","New Nepali transformers: 95.60 Nep-gLUE, better ROUGE"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00022,"raw_usage":{"total_tokens":1466,"prompt_tokens":981,"completion_tokens":485,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":597,"completion_tokens_details":{"reasoning_tokens":396}},"tokens_in":597,"tokens_out":485,"duration_ms":4032,"temperature":1.0,"reasoning_tokens":396,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T13:57:06.839401+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A decontamination scan comparing Nep-gLUE instances against the 27.5 GB corpus, or a re-evaluation with hyperparameters chosen on a held-out validation split rather than the test set, would determine whether the 95.60 and ROUGE gains are genuine.","supporting_citations":[{"cited_title":"Nepberta: Nepali language model trained in a large corpus","cited_arxiv_id":null,"evidence_quote":"Supplies the Nep-gLUE benchmark, the prior best NepBERTa baseline, and the corpus size comparison."},{"cited_title":"Bert: Pre-training of deep bidirectional transformers for language understanding","cited_arxiv_id":null,"evidence_quote":"Supplies the BERT architecture and the MLM/NSP pretraining objectives used for the encoder models."},{"cited_title":"alpaca-nepali-sft, 2024","cited_arxiv_id":null,"evidence_quote":"Supplies the publicly available Nepali Alpaca instruction dataset used for instruction tuning."},{"cited_title":"Xlsum-nepali-summerization-dataset, 2024","cited_arxiv_id":null,"evidence_quote":"Supplies the abstractive summarization dataset used to evaluate the GPT-2 models."},{"cited_title":"Neural machine translation of rare words with subword units","cited_arxiv_id":null,"evidence_quote":"Supplies the BPE subword tokenization method used for the Nepali tokenizers."},{"cited_title":"Nepalibert: Pre-training of masked language model in nepali corpus","cited_arxiv_id":null,"evidence_quote":"Supplies the NepaliBERT monolingual model used as a baseline comparator."},{"cited_title":"Distilgpt2-nepali, 2022","cited_arxiv_id":null,"evidence_quote":"Supplies the existing distilgpt-nepali model used as the generation baseline."}],"review_version":1}