Pith. sign in

REVIEW 3 major objections 5 minor 1 cited by

Development of Pre-Trained Transformer-based Models for the Nepali Language

T0 review · 3 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A new 27.5 GB Nepali corpus and three pre-trained transformers beat all prior Nepali models, scoring 95.60 on Nep-gLUE.

desk verdict Real corpus and models for Nepali, but the SOTA claim dissolves under test-set hyperparameter selection. read the letter →

arxiv 2411.15734 v2 pith:EYXIF2VD submitted 2024-11-24 cs.CL cs.LG

Prajwal Thapa , Jinu Nyachhyon , Mridul Sharma , Bal Krishna Bal This is my paper · ORCID
classification cs.CLcs.LG
keywords Nepalilanguagemodelspre-trainedtransformersNep-gLUElow-resourceNLPBERTRoBERTaGPT-2instructiontuning
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper sets out to close the resource gap for Nepali, a language of roughly 32 million speakers, by building the largest monolingual corpus to date — 27.5 GB scraped from 99 news websites — and pre-training three Transformer models on it: BERT, RoBERTa, and GPT-2, plus an instruction-tuned GPT-2 variant. On the Nep-gLUE benchmark the authors report that their RoBERTa reaches 95.60, two points above the prior best model, and that their GPT-2-Instruct reaches 20.42 ROUGE-1 on abstractive summarization. The claim is that scale and language-specific tokenization, rather than architectural novelty, are what let monolingual Nepali models surpass both multilingual and earlier monolingual baselines. If correct, this gives Nepali NLP a much stronger pretraining foundation and its first competitive decoder-based generation baselines.

What carries the argument

The load-bearing object is a 27.5 GB monolingual Nepali corpus collected by scraping 99 news websites and deduplicated, cleaned, and normalized down from 30.4 GB, with two BPE tokenizers of 30,522 and 50,256 subwords. On top sit three pretraining setups: BERT and RoBERTa (110M parameters, masked language modeling, 400k steps) and GPT-2 (124M, causal language modeling, 500k steps), followed by supervised instruction tuning of GPT-2 on a cleaned 40k-row Nepali Alpaca set. The corpus feeds both the tokenizers and the pretraining; the instruction set feeds the Instruct variant; the evaluation harness is Nep-gLUE for encoders and a 7,258-pair summarization set for decoders.

What would settle it

A decontamination scan comparing Nep-gLUE instances against the 27.5 GB corpus, or a re-evaluation with hyperparameters chosen on a held-out validation split rather than the test set, would determine whether the 95.60 and ROUGE gains are genuine.

Watch

Extended reading notes

Core claim

The paper's central claim is that a 2.4x larger Nepali corpus plus architecture-appropriate tokenizers yields state-of-the-art understanding and generation for Nepali. Concretely, the authors' RoBERTa scores 95.60 on Nep-gLUE (NER 93.74, POS 97.52, text classification 94.68, pair similarity 96.49), beating the previous best NepBERTa by 2.05 points, and both BERT (95.18) and RoBERTa outperform every multilingual and monolingual comparator on each task. On generation, their 124M GPT-2 and GPT-2-Instruct models beat the existing distilgpt-nepali on all three ROUGE metrics, with GPT-2-Instruct reaching 20.42 ROUGE-1, 15.89 ROUGE-2, and 17.76 ROUGE-L. The authors present the corpus itself — deliberately scraped from scratch to avoid overlap with prior news-based datasets — and the instruction-tuned variant as resources for future Nepali NLP.

Load-bearing premise

The reported scores assume the pretraining corpus is disjoint from the Nep-gLUE benchmark, and that choosing the best checkpoint directly on the test set does not inflate results.

Editorial extensions

If this is right

  • Nepali NLP now has a monolingual corpus roughly 2.4x larger than any prior one, giving future work a stronger pretraining base.
  • Encoder-based models for Nepali set a new state-of-the-art of 95.60 on Nep-gLUE, surpassing both multilingual models and all earlier monolingual models.
  • A Nepali GPT-2 and its instruction-tuned variant provide the first strong decoder baselines for Nepali text generation, nearly doubling ROUGE-1 over the existing Nepali GPT baseline.
  • Instruction tuning on a cleaned 40k-row Nepali instruction set improves summarization over the base GPT-2, suggesting monolingual instruction tuning works for Nepali.
  • The gap between multilingual and language-specific models on Nep-gLUE narrows further, supporting the value of dedicated monolingual modeling for low-resource languages.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the corpus and benchmark are confirmed disjoint, the results imply data scale is the dominant factor: the 2.05-point jump over NepBERTa came without architectural changes, only more data and a different tokenizer.
  • A natural next test is whether further scaling the corpus or instruction set continues to improve Nepali generation, and whether longer training sequences fix the hallucination the authors observed at inputs over 400 tokens.
  • The same corpus-building recipe could transfer to other under-resourced SOV languages, though the paper does not test this.
  • Reporting per-task test-set variance and contamination ratios would let the community assess whether the 95.60 margin is robust across seeds.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper collects a new 27.5 GB Nepali news corpus, pre-trains BERT, RoBERTa, and GPT-2 models for Nepali, performs instruction tuning on the GPT-2 model, and evaluates the resulting systems on the Nep-gLUE benchmark and an abstractive summarization task. The authors report that their RoBERTa model achieves 95.60 on Nep-gLUE, surpassing existing models, and that their GPT-2-Instruct model obtains the highest ROUGE scores in summarization. The main contributions claimed are the largest monolingual Nepali corpus to date, new monolingual encoder and decoder models, and an initial exploration of instruction tuning for Nepali.

Significance. If the reported results are reliable, the work is useful for Nepali NLP: the corpus is a substantial public resource, the encoder models improve on an established benchmark, and the decoder and instruction-tuned variants address an underserved area. The paper also contributes the first Nepali GPT-2-style model and a concrete recipe for pretraining with TPU resources. The central state-of-the-art claims, however, rest on an evaluation protocol that selects models directly on the test set and on a very small generation baseline, so the magnitude of the claimed improvements is not yet established. The resource and models would still be valuable even if the exact scores are revised after a more rigorous evaluation.

major comments (3)
  1. [Section 6.1] The evaluation protocol selects, for each Nep-gLUE task, the best-performing fine-tuned model directly on the test set ("For each task, we selected the best-performing model on the test set"). Since the search spans five learning rates and 3–10 epochs, the reported 95.60 is the maximum over many runs on the test data, which inflates the estimate. The baseline scores in Table 1 (e.g., NepBERTa 93.55) were not produced under this protocol, so the 2-point margin over the previous state of the art cannot be interpreted as a fair comparison. The authors should instead select hyperparameters on a validation split and report test performance for the chosen configuration, together with variance across seeds.
  2. [Section 3.1 / Section 6.1] No decontamination check is reported between the pretraining corpus, which was scraped from the top 99 Nepali news websites, and the Nep-gLUE tasks, which are also derived from Nepali news text. Section 3.1 only excludes other existing corpora to avoid duplication; it does not verify that Nep-gLUE instances are absent from the pretraining data. If even a fraction of the benchmark appears in the 27.5 GB corpus, the 95.60 score would be inflated by memorization. The authors should report an n-gram overlap analysis or another explicit decontamination test between the pretraining corpus and the benchmark.
  3. [Section 6.2 / Table 2] The summarization comparison uses a single baseline, distilgpt-nepali, which has 88.2M parameters while the proposed GPT-2 models have 124M. A capacity difference of this size can by itself explain the ROUGE gains, so the conclusion that the models "outperform existing models on text generation tasks" is not yet supported. The evaluation also appears to use a single 80/20 split with no multiple runs or statistical significance testing. A comparison against a same-size or multilingual GPT-2 baseline, and ideally multiple seeds with standard deviations, would be needed to substantiate the generation claim.
minor comments (5)
  1. [Section 4] There is a typo in the sentence "For our study. we pretrained a single BERT..." — the period should be a comma.
  2. [Section 7] Table 1 lists "NepBERT (Pudasaini et al., 2023)" and "NepaliBERT (Rajan, 2021)", but Section 2.2 attributes NepaliBERT to Pudasaini et al. (2023). The naming is inconsistent and should be corrected to avoid confusion about which model is being compared.
  3. [Section 6.2] The text says "resulting in average ROGUE scores" — the metric is ROUGE, not ROGUE. The same section also cites Lin (2004) and Lin and Och (2004) for ROUGE, which appears to be a duplicate citation.
  4. [Section 3.1] There is an extra closing parenthesis in "(Kafley, 2024))" in the instruction-tuning dataset description.
  5. [Table 1] The table does not define the abbreviations CC and CPS for the task columns, which makes it harder to interpret the per-task scores.

Circularity Check

1 steps flagged · score 6.0 of 10

Nep-gLUE SOTA claim reduces to test-set selection: Section 6.1 chooses the best model on the test set, so the reported 95.60 is the selection statistic rather than an unbiased prediction.

  1. fitted input called prediction [Section 6.1 (Evaluating BERT & RoBERTa), Table 1]
    "We used a batch size of 32 and fine-tuned for 3-10 epochs with multiple learning rates (5e-5, 4e-5, 3e-5, 2e-5, and 1e-5) over the data for all Nep-gLUE tasks. For each task, we selected the best-performing model on the test set."

    The headline claim that the models 'outperformed all existing models across all tasks, scoring 95.60' is a maximum over fine-tuning runs, with each run scored directly on the test set. Selecting the best model on the test set means the reported 95.60 is, by construction, the largest test score among the tried configurations, not an unbiased prediction of a single model's generalization. The baselines in Table 1 (NepBERTa 93.55, XLM-R 92.11, etc.) were not produced under this test-set-selection protocol, so the 2-point margin is at least partly manufactured by the selection procedure. The 'prediction' of state-of-the-art performance is therefore statistically forced by the evaluation protocol rather than derived independently from the pretraining corpus.

full rationale

The pretraining-and-evaluation pipeline is otherwise independent: the models are trained on a newly scraped 27.5 GB corpus and evaluated on the external Nep-gLUE benchmark and an external summarization dataset (Bhandari 2024); no derived quantity is computed from the same data that defines it. I found no load-bearing self-citation chain: the only overlapping authorship reference (Bal 2004) is a grammar description and is not load-bearing. The corpus/benchmark overlap concern is a data-contamination risk rather than a demonstrated circular reduction, because the paper reports no overlap check and the claim would be speculative; I therefore do not count it in the score. The one concrete circularity is the test-set model selection in Section 6.1: the reported Nep-gLUE score is the maximum over hyperparameter/epoch configurations evaluated on the test set, so the central SOTA claim is not an independent prediction but a selected-best test statistic. This warrants a 6: the central comparison partly reduces to construction, while the underlying model training and the summarization results retain independent content.

Assumptions & free parameters 2 free parameters · 3 assumptions · 0 invented entities

The central results rest on standard ML assumptions and two hand-chosen design decisions (tokenizer size and test-set selection). No new entities are postulated.

free parameters (2)
  • Tokenizer vocabulary sizes = 30,522 (BERT) and 50,257 (RoBERTa)
    Hand-chosen in Section 3.3; the different sizes between the two encoder models confound the BERT versus RoBERTa comparison.
  • Test-set best-model selection = not disclosed (search over 1e-5 to 5e-5 LR, 3 to 10 epochs)
    Section 6.1 states the best-performing model on the test set was chosen for each task; this is post-hoc selection that can inflate reported scores.
assumptions (3)
  • domain assumption The Transformer architecture and standard pretraining objectives (MLM, CLM) transfer to Nepali.
    The paper applies BERT/RoBERTa/GPT-2 without language-specific architectural changes; it assumes these methods work for a morphologically rich SOV language.
  • domain assumption The Nep-gLUE benchmark is a valid measure of Nepali language understanding.
    Section 6.1 adopts Nep-gLUE from Timilsina et al. 2022 as the sole NLU benchmark; the paper does not assess its construct validity.
  • domain assumption The scraped 27.5 GB corpus is sufficiently clean, diverse, and free of duplication with evaluation data.
    Section 3.1 states they scraped all content from scratch to avoid duplication, but no decontamination check against Nep-gLUE test data is described.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Development of Pre-Trained Transformer-based Models for the Nepali Language." pith.science (2026). https://pith.science/paper/EYXIF2VD

@misc{pith2026241115734,
  author       = {Pith},
  title        = {Pith review of: Development of Pre-Trained Transformer-based Models for the Nepali Language},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/EYXIF2VD}},
  note         = {Machine review of arXiv:2411.15734}
}
read the original abstract

Transformer-based pre-trained language models have dominated the field of Natural Language Processing (NLP) for quite some time now. However, the Nepali language, spoken by approximately 32 million people worldwide, remains significantly underrepresented in this domain. This underrepresentation is primarily attributed to the scarcity of monolingual data corpora and limited available resources for the Nepali language. While existing efforts have predominantly concentrated on basic encoder-based models, there is a notable gap in the exploration of decoder-based architectures. To address this gap, we have collected 27.5 GB of Nepali text data, approximately 2.4x larger than any previously available Nepali language corpus. Leveraging this data, we pre-trained three different models i.e., BERT, RoBERTa, and GPT-2, exclusively for the Nepali Language. Furthermore, we performed instruction tuning and explored its potential for monolingual Nepali data, providing a foundation for future research. Our models outperformed the existing best model by 2 points on Nep-gLUE benchmark, scoring 95.60 and also outperformed existing models on text generation tasks, demonstrating improvements in both understanding and generating Nepali text.

Figures

Figures reproduced from arXiv: 2411.15734 by the authors.

Figure 1
Figure 1. Loss and accuracy of the BERT and RoBERTA [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Loss and Perplexity of the GPT2 model com [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Prompt, Translate, Fine-Tune, Re-Initialize, or Instruction-Tune? Adapting LLMs for In-Context Learning in Low-Resource Languages

    cs.CL 2025-06

Reference graph

Works this paper leans on

39 extracted references · 25 canonical work pages · cited by 1 Pith paper

  1. [1]

    A bit of progress in language modeling

    Joshua Goodman. A bit of progress in language modeling. Expert Systems with Applications , 41(3):853--860, 2001

    work page 2001

  2. [2]

    Recurrent neural network based language model

    Tomaš Mikolov, Martin Karafiát, Lukáš Burget, Jan "Honza" Černocký, and Sanjeev Khudanpur. Recurrent neural network based language model. In Interspeech , 2010

    work page 2010

  3. [3]

    Lstm neural networks for language modeling

    Martin Sundermeyer, Ralf Schlüter, and Hermann Ney. Lstm neural networks for language modeling. In Interspeech , 2012

    work page 2012

  4. [4]

    Gomez, Łukasz Kaiser, and Illia Polosukhin

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in Neural Information Processing Systems , volume 30, 2017

    work page 2017

  5. [5]

    Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer

    Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. Deep contextualized word representations. arXiv preprint arXiv:1802.05365 , 2018

    arXiv 2018

  6. [6]

    Bert: Pre-training of deep bidirectional transformers for language understanding

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers) , pages 4171--4186. Associati...

    work page 2019

  7. [7]

    Improving language understanding with unsupervised learning

    Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. Improving language understanding with unsupervised learning. OpenAI Blog , 2018

    work page 2018

  8. [8]

    Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M

    Jason Wei, Maarten Bosma, Vincent Y. Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoc V. Le. Finetuned language models are zero-shot learners. arXiv preprint arXiv:2109.01652 , 2021

    arXiv 2021

Show all 39 references
  1. [9]

    Smith, Daniel Khashabi, and Hannaneh Hajishirzi

    Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannaneh Hajishirzi. Self-instruct: Aligning language models with self-generated instructions. arXiv preprint arXiv:2212.10560 , 2022

    2022 arXiv

  2. [10]

    Nepberta: Nepali language model trained in a large corpus

    Sulav Timilsina, Milan Gautam, and Binod Bhattarai. Nepberta: Nepali language model trained in a large corpus. In Proceedings of the 2nd Conference of the Asia-Pacific Chapter of the Association for Computational Linguistics and the 12th International Joint Conference on Natur...

    2022

  3. [11]

    Structure of Nepali Grammar

    Bal Krishna Bal. Structure of Nepali Grammar . 2004

    2004

  4. [12]

    Efficient estimation of word representations in vector space

    Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. Efficient estimation of word representations in vector space. arXiv preprint arXiv:1301.3781 , 2013

    2013 arXiv

  5. [13]

    Glove: Global vectors for word representation

    Jeffrey Pennington, Richard Socher, and Christopher Manning. Glove: Global vectors for word representation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP) , pages 1532--1543. Association for Computational Linguistics, 2014

    2014

  6. [14]

    Roberta: A robustly optimized bert pretraining approach

    Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692 , 2019

    1907 arXiv

  7. [15]

    Le, and Christopher D

    Kevin Clark, Minh-Thang Luong, Quoc V. Le, and Christopher D. Manning. Electra: Pre-training text encoders as discriminators rather than text generators. In International Conference on Learning Representations (ICLR) , 2020

    2020

  8. [16]

    Deberta: Decoding-enhanced bert with disentangled attention

    Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. Deberta: Decoding-enhanced bert with disentangled attention. arXiv preprint arXiv:2006.03654 , 2020

    2006 arXiv

  9. [17]

    How multilingual is multilingual bert? arXiv preprint arXiv:1906.01502 , 2019

    Telmo Pires, Eva Schlinger, and Dan Garrette. How multilingual is multilingual bert? arXiv preprint arXiv:1906.01502 , 2019

    1906 arXiv

  10. [18]

    Cross-lingual language model pre-training

    Guillaume Lample and Alexis Conneau. Cross-lingual language model pre-training. In Advances in Neural Information Processing Systems (NeurIPS) , 2019

    2019

  11. [19]

    Unsupervised cross-lingual representation learning at scale

    Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. Unsupervised cross-lingual representation learning at scale. arXiv preprint arXiv:1911.02116v2 , 2020

    1911 arXiv

  12. [20]

    Large-scale contextualised language modelling for norwegian

    Andrey Kutuzov, Jeremy Barnes, Erik Velldal, Lilja Øvrelid, and Stephan Oepen. Large-scale contextualised language modelling for norwegian. arXiv preprint arXiv:2104.06546 , 2021

    2021 arXiv

  13. [21]

    Multilingual is not enough: Bert for finnish

    Antti Virtanen, Jenna Kanerva, Rami Ilo, Jouni Luoma, Juhani Luotolahti, Tapio Salakoski, Filip Ginter, and Sampo Pyysalo. Multilingual is not enough: Bert for finnish. arXiv preprint arXiv:1912.07076 , 2019

    1912 arXiv

  14. [22]

    Herbert: Efficiently pretrained transformer-based language model for polish

    Robert Mroczkowski, Piotr Rybak, Alina Wróblewska, and Ireneusz Gawlik. Herbert: Efficiently pretrained transformer-based language model for polish. In Proceedings of the 8th Workshop on Balto-Slavic Natural Language Processing , pages 1--10. Association for Computational Ling...

  15. [23]

    German’s next language model

    Branden Chan, Stefan Schweter, and Timo Möller. German’s next language model. In Proceedings of the 28th International Conference on Computational Linguistics , pages 6788--6796. International Committee on Computational Linguistics, 2020

    2020

  16. [24]

    Pre-training with whole word masking for chinese bert

    Yiming Cui, Wanxiang Che, Ting Liu, Bing Qin, and Ziqing Yang. Pre-training with whole word masking for chinese bert. IEEE/ACM Transactions on Audio, Speech, and Language Processing , 2021

    2021

  17. [25]

    Khapra, Anoop Kunchukuttan, and Pratyush Kumar

    Sumanth Doddapaneni, Rahul Aralikatte, Gowtham Ramesh, Shreya Goyal, Mitesh M. Khapra, Anoop Kunchukuttan, and Pratyush Kumar. Towards leaving no indic language behind: Building monolingual corpora, benchmark and models for indic languages. arXiv preprint arXiv:2212.05409 , 2022

    2022 arXiv

  18. [26]

    Nepalibert: Pre-training of masked language model in nepali corpus

    Shushanta Pudasaini, Subarna Shakya, Aakash Tamang, Sajjan Adhikari, Sunil Thapa, and Sagar Lamichhane. Nepalibert: Pre-training of masked language model in nepali corpus. In 7th International Conference on IoT in Social, Mobile, Analytics and Cloud , 2023

    2023

  19. [27]

    Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and...

    2022

  20. [28]

    inltk: Natural language toolkit for indic languages

    Gaurav Arora. inltk: Natural language toolkit for indic languages. In Proceedings of Second Workshop for NLP Open Source Software (NLP-OSS) , pages 66--71. Association for Computational Linguistics, 2020

    2020

  21. [29]

    Asynchronous pipeline for processing huge corpora on medium to low resource infrastructures

    Pedro Ortiz Suarez, Benoît Sagot, and Laurent Romary. Asynchronous pipeline for processing huge corpora on medium to low resource infrastructures. In 7th Workshop on the Challenges in the Management of Large Corpora (CMLC-7) . Leibniz-Institut für Deutsche Sprache, 2019

    2019

  22. [30]

    alpaca-nepali-sft, 2024

    Saugat Kafley. alpaca-nepali-sft, 2024

    2024

  23. [31]

    Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, et al

    Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V. Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, et al. Google's neural machine translation system: Bridging the gap between human and machine translation. arXiv preprint arXiv:1609.08144 , 2016

    2016 arXiv

  24. [32]

    Neural machine translation of rare words with subword units

    Rico Sennrich, Barry Haddow, and Alexandra Birch. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) , pages 1715--1725. Association for Computational L...

    2016

  25. [33]

    Kingma and Jimmy Lei Ba

    Diederik P. Kingma and Jimmy Lei Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 , 2014

    2014 arXiv

  26. [34]

    Gaussian error linear units

    Dan Hendrycks and Kevin Gimpel. Gaussian error linear units. arXiv preprint arXiv:1606.08415 , 2016

    2016 arXiv

  27. [35]

    Language models are unsupervised multitask learners

    Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. OpenAI blog , 2019

    2019

  28. [36]

    Nepalibert, 2021

    Rajan. Nepalibert, 2021

    2021

  29. [37]

    Xlsum-nepali-summerization-dataset, 2024

    Sanjeev Bhandari. Xlsum-nepali-summerization-dataset, 2024

    2024

  30. [38]

    Automatic evaluation of machine translation quality using longest common subsequence and skip-bigram statistics

    Chin-Yew Lin and Franz Josef Och. Automatic evaluation of machine translation quality using longest common subsequence and skip-bigram statistics. In Proceedings of the 42nd Annual Meeting of the Association for Computational Linguistics (ACL-04) , pages 605--612, 2004

    2004

  31. [39]

    Distilgpt2-nepali, 2022

    Utsav Maskey. Distilgpt2-nepali, 2022

    2022

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.