Pith. sign in

REVIEW 4 major objections 7 minor 48 references

Domain-adaptative Continual Learning for Low-resource Tasks: Evaluation on Nepali

T0 review · 4 major / 7 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims that domain-adaptive continual learning on synthetic Nepali–English parallel data, run in 4-bit QLoRA, adapts Llama 3 8B to generate Nepali and retain latent English knowledge.

desk verdict A clean low-resource adaptation recipe for Nepali, but the central claim rests on an unvalidated GPT-4o evaluator and some loose interpretation. read the letter →

arxiv 2412.13860 v1 pith:EP5M6VZ6 submitted 2024-12-18 cs.CL cs.LG

Sharad Duwal , Suraj Prasai , Suresh Manandhar This is my paper · ORCID
classification cs.CLcs.LG
keywords domain-adaptivepretrainingcontinuallearninglow-resourcelanguagesNepalilanguagesyntheticparalleldataQLoRAcatastrophicforgettingattentionheatmaps
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that a large language model can be adapted to a low-resource language using only synthetic parallel data and a small fraction of trainable parameters. It continually trains Llama 3 8B on 5 million Nepali–English paragraph pairs and translated instruction data with 4-bit QLoRA, then compares the adapted model against the base model. The adapted model produces Nepali answers that GPT-4o scores substantially higher, especially on grammatical correctness, while English benchmarks show expected forgetting but larger few-shot improvements (up to 19.29% versus 4.98%), which the authors read as latent retention of English knowledge. If these results hold, domain-adaptive continual learning is a practical path for resource-constrained languages that lack native corpora or dedicated benchmarks.

What carries the argument

The central mechanism is a two-stage QLoRA continual pretraining on synthetic Nepali–English parallel data: first the model is trained to translate English to Nepali so it generates organic Nepali, then it is trained on bilingual next-token prediction with alternating Nepali and English sentences to reuse English knowledge. Both stages use rank-128 4-bit QLoRA, updating roughly 335M of Llama 3 8B's parameters, followed by a rank-16 instruction finetune on translated Alpaca, Dolly, and WebGLM sets. The parallel corpus is built with NLLB (8-bit) for 5M paragraph pairs and IndicTrans2 for 114K instruction triplets, filtered by a chrF++ backtranslation threshold of 50. For linguistic probing, the paper max-pools token-level self-attention into word-level attention and plots layer-head heatmaps for adjective–noun dependency pairs.

What would settle it

A native-speaker evaluation of the adapted model's Nepali outputs would settle it: ask Nepali speakers to rate grammaticality and overall quality on the same 78 questions and compare with the base model. If human ratings do not reproduce the GPT-4o advantage, the central claim fails. A second check is to back-translate a sample of the 5M parallel pairs and have a bilingual annotator judge the Nepali side; a high rate of MT artifacts or unidiomatic constructions would undermine the data-generation premise.

Watch

Extended reading notes

Core claim

On its own terms, the paper establishes that domain-adaptive pretraining on synthetic data moves a base model from almost no usable Nepali generation to recognizable, higher-scoring Nepali output. Using English-to-Nepali translation pretraining followed by bilingual next-token prediction, the final model's answers to 78 Nepali traffic-license questions score higher than the base model across correctness, grammar, usability, hallucination, and overall quality, with grammar showing the clearest improvement. On English benchmarks the adapted model forgets, but the relative gain from 0-shot to 5-shot prompting is consistently larger than the base model's, reaching 19.29% on ARC-Challenge, a pattern the paper interprets as evidence that English knowledge is retained latently and can be reactivated through in-context examples. The attention heatmaps finally show the adapted model attending from Nepali adjectives to their governing nouns in a way that resembles the base model's English patterns, supporting the claim that the model has acquired structural knowledge of Nepali rather than just token-level mimicry.

Load-bearing premise

The load-bearing premise is that the synthetic Nepali side of the parallel corpus is genuinely valid, representative Nepali: the paper reports no human quality check on the 5 million translated paragraph pairs, so if those translations are translationese or contain systematic machine-translation errors, every downstream claim about the model learning Nepali loses its foundation.

Editorial extensions

If this is right

  • A low-resource language without an instruction corpus can inherit English knowledge through a translation-and-bilingual-pretraining loop, so Nepali-style adaptation may be repeatable for other South Asian languages.
  • Catastrophic forgetting after domain-adaptive pretraining is real but not permanent in practice: the adapted model recovers more of its English benchmark performance when given a few examples, so few-shot prompting should be part of the evaluation protocol for continually trained models.
  • Attention heatmaps of adjective-noun relations offer a cheap way to check whether an adapted model has learned syntax rather than surface token statistics.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The 'latent retention' reading could be tested directly by building Nepali versions of ARC or MMLU: if few-shot gains come from retained knowledge, the adapted model should also improve on Nepali-language reasoning tasks, which the paper notes do not yet exist.
  • A native-speaker rating study of the same 78 generated answers would separate genuine Nepali competence from GPT-4o's tolerance for translationese; the paper itself names human evaluation as its main missing check.
  • Because the synthetic corpus is dominated by online news text, the adapted model's grammatical gains may be register-specific; a test on conversational or dialectal Nepali would show whether the dependency-resolution patterns generalize.
  • The method's parameter efficiency suggests a sequential-adaptation possibility the paper leaves implicit: one base model could be adapted to several low-resource languages in turn, with per-language LoRA adapters, and the few-shot reactivation effect could serve as shared retrieval across those languages.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 7 minor

Summary. The paper investigates domain-adaptive pretraining (DAPT) of Llama 3 8B for Nepali using 4-bit QLoRA and synthetic parallel data (about 5M Nepali–English pairs and 114K translated instruction triplets). The pipeline has two pretraining stages (English-to-Nepali translation and bilingual next-token prediction) followed by mixed-language instruction finetuning. The authors compare base and adapted models on three questions: Nepali generation quality (GPT-4o scores on 78 license-exam questions), English forgetting (MMLU, ARC, Winogrande, TruthfulQA), and dependency resolution (self-attention heatmaps for adjective–noun pairs). They report forgetting on English benchmarks, larger relative few-shot gains for the adapted model (up to 19.29% on ARC-Challenge), which they interpret as latent retention, and qualitatively different attention patterns for Nepali adjectives.

Significance. If the claims were fully supported, the paper would provide a useful low-resource recipe: adapting an 8B model to an unsupported language using only synthetic data and modest compute, with evidence about forgetting and knowledge transfer. The strengths are the practical setup (QLoRA, open weights), a transparent data-generation pipeline (NLLB and IndicTrans2 with chrF++ filtering), and evaluation on four standard English benchmarks. However, the headline claims ('learned Nepali', 'latent retention', 'dependency resolution') currently rest on an unvalidated automatic judge, unquantified visual analyses, and internally inconsistent tables. With added validation and statistical support, the paper could serve as a useful empirical case study for DAPT in low-resource languages.

major comments (4)
  1. [§5.3, §6, Figure 1] The central evidence for Q1 is GPT-4o scoring of 78 Nepali answers, but no scoring prompt, no human correlation, and no inter-rater reliability are reported; the Limitations section concedes that human evaluators would give a more definitive assessment. Because empty/degenerate base-model outputs are scored 0, the comparison may largely reflect fluency rather than semantic correctness, and Section 6's interpretation of higher hallucination scores as positive ('content that is more verifiable') suggests the scoring is not validated as factual accuracy. Please provide the full scoring prompt, report per-item scores, validate against at least a small set of Nepali human judgments, and justify or remove the hallucination-as-positive interpretation.
  2. [§6, Table 1] There are material inconsistencies between the table and text: Table 1 reports final-model Winogrande 0.5801/0.6275, but the text reports 0.5691/0.6022; TruthfulQA MC1/MC2 are 0.2827/0.4351 in the table but 0.2607/0.4243 in the text. These are not typo-level differences for the forgetting claim. Please correct the numbers and ensure the abstract, table, and text all use the same final values.
  3. [§6, Table 1, Abstract] The 'latent retention' interpretation is not supported by significance testing or error bars. Relative percent improvements over a much lower 0-shot baseline (e.g., ARC-Challenge: 0.3183→0.3797, +19.29%, vs. base 0.5017→0.5179, +3.23%) conflate baseline level and ceiling effects; the absolute gain is larger for the final model, but no variance or significance is reported. Please report multiple evaluation runs or standard errors, and ideally test the few-shot improvement against a control adaptation (e.g., Nepali-only continued pretraining) before claiming latent retention.
  4. [§6, Figure 2] The Q3 conclusion that the final model 'has learned to attend to Nepali adjectives the way the base model attends to English ones' is based on visual inspection of heatmaps, with no quantitative summary, no comparison to random or baseline attention, and no measure of inter-annotator agreement. Please provide a quantitative metric (e.g., mean adjective-to-noun attention strength per layer/head, with error bars) and a significance test or permutation baseline.
minor comments (7)
  1. [Title] The title contains a typo: 'Domain-adaptative' should be 'Domain-adaptive'.
  2. [§6, paragraph on few-shot gains] The sentence 'suggesting that our model leverages few-shot examples more effectively than the final model' should read '...more effectively than the base model.'
  3. [Table 1] The TruthfulQA MC1 and MC2 rows have empty 5-shot columns; please clarify why 5-shot results are not reported for these benchmarks.
  4. [§3.1] No human verification of the translated instruction sets is reported; the chrF++ backtranslation filter is a useful proxy, but the paper should state how many instruction triplets were discarded and give examples of retained and discarded items.
  5. [§3.2] Training hyperparameters (learning rate, batch size, sequence length, optimizer, number of steps, hardware) are not reported in sufficient detail to replicate the experiments; the statement 'training settings are much the same' is too vague.
  6. [§5.2.1] The claim that max-pooling is more suitable than mean-pooling is not supported by reported quantitative results; please provide the comparison that motivated this choice.
  7. [Figures 1 and 2] The heatmaps and boxplots lack axis labels and color-scale details; the score distributions in Figure 1 would be easier to interpret if the attributes were explicitly labeled on the boxplot axes.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity found: all load-bearing inputs (synthetic parallel data, external MT models, GPT-4o evaluation, English benchmarks, annotated attention probes) are external to the model being evaluated.

full rationale

The paper's derivation chain is self-contained against external benchmarks and data sources. The synthetic Nepali-English parallel corpus is generated with NLLB and IndicTrans2, both external machine-translation systems, and the 114K instruction triplets are filtered by backtranslation chrF++ against the original English instructions; no training target is defined in terms of the final evaluation scores. The training procedure (English-to-Nepali translation pretraining, bilingual next-token prediction, then mixed-instruction finetuning) is adopted from the cited external SarvamAI work, and the claimed 'latent retention' is an interpretive reading of the differences between 0-shot and 5-shot scores on ARC, MMLU, Winogrande, and TruthfulQA, which are independent English benchmarks from the LM Evaluation Harness. The Nepali-generation evaluation uses GPT-4o as an external judge over five attributes, and the attention analysis uses separately annotated adjective-noun pairs; neither evaluation is fitted from nor defined by the training data. The paper's Limitations section explicitly concedes that human evaluators, especially Nepali experts, could give a more definitive assessment and that no Nepali LM benchmarks exist, but that is a validity and calibration concern about the automatic judge, not a circularity in the derivation. No fitted parameter is renamed as a prediction, no unique theorem is imported from the authors' prior work, and no load-bearing claim reduces by construction to its own inputs.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

The central claim relies on hand-chosen hyperparameters and on assumptions that synthetic data is linguistically valid, that GPT-4o judgments are ground truth, and that attention heatmaps reflect syntax. No new entities are postulated.

free parameters (5)
  • QLoRA rank (pretraining) = 128
    Hand-chosen in Section 3.2.1; selects 335M trainable parameters; no sweep reported.
  • QLoRA rank (finetuning) = 16
    Hand-chosen in Section 3.3; updates ~41M parameters; no sweep reported.
  • chrF++ filtering threshold = 50
    Applied to backtranslated instructions in Section 3.1 to discard low-quality samples; value chosen by hand.
  • Parallel data volume = 5M pairs
    Number of Nepali-English paragraph pairs generated; 1.5M used for translation pretraining and 1.5M for bilingual NTP; hand-chosen.
  • Instruction data volume = 114K triplets
    Remaining after chrF++ filtering of translated Alpaca/Dolly/WebGLM-QA; hand-chosen.
assumptions (5)
  • domain assumption Machine-translated Nepali from NLLB and IndicTrans2 preserves organic Nepali sufficiently for language adaptation.
    Section 3.1 relies on synthetic parallel data without human quality assessment; translationese could bias the model.
  • domain assumption GPT-4o scoring is a valid measure of Nepali generation quality.
    Section 5.3 and Limitations: no human evaluators; GPT-4o's judgments are used as ground truth.
  • domain assumption Attention heatmaps from adjective-noun pairs reflect syntactic dependency resolution.
    Section 6 and Figure 2: visual comparison of 17 to 26 pairs is interpreted as evidence of grammatical knowledge, citing prior interpretability work.
  • domain assumption 4-bit QLoRA preserves enough model capacity for meaningful adaptation.
    All training uses 4-bit quantization; limitations note resource constraints but do not compare to full precision.
  • domain assumption Base Llama 3 8B English knowledge is transferable to Nepali via bilingual pretraining.
    Core premise of the method in Section 3.2; no ablation without this assumption.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Domain-adaptative Continual Learning for Low-resource Tasks: Evaluation on Nepali." pith.science (2026). https://pith.science/paper/EP5M6VZ6

@misc{pith2026241213860,
  author       = {Pith},
  title        = {Pith review of: Domain-adaptative Continual Learning for Low-resource Tasks: Evaluation on Nepali},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/EP5M6VZ6}},
  note         = {Machine review of arXiv:2412.13860}
}
read the original abstract

Continual learning has emerged as an important research direction due to the infeasibility of retraining large language models (LLMs) from scratch in the event of new data availability. Of great interest is the domain-adaptive pre-training (DAPT) paradigm, which focuses on continually training a pre-trained language model to adapt it to a domain it was not originally trained on. In this work, we evaluate the feasibility of DAPT in a low-resource setting, namely the Nepali language. We use synthetic data to continue training Llama 3 8B to adapt it to the Nepali language in a 4-bit QLoRA setting. We evaluate the adapted model on its performance, forgetting, and knowledge acquisition. We compare the base model and the final model on their Nepali generation abilities, their performance on popular benchmarks, and run case-studies to probe their linguistic knowledge in Nepali. We see some unsurprising forgetting in the final model, but also surprisingly find that increasing the number of shots during evaluation yields better percent increases in the final model (as high as 19.29% increase) compared to the base model (4.98%), suggesting latent retention. We also explore layer-head self-attention heatmaps to establish dependency resolution abilities of the final model in Nepali.

Figures

Figures reproduced from arXiv: 2412.13860 by the authors.

Figure 1
Figure 1. GPT4o scores for Nepali answers generated by the base model (Llama 3 8B 4-bit) and our model on five [PITH_FULL_IMAGE:figures/full_fig_p005_1.png] view at source ↗
Figure 2
Figure 2. Layer-head heatmaps visualizing attention from adjectives to their respective nouns in English (a,c) and [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

48 extracted references · 6 canonical work pages

  1. [1]

    Julien Abadji , Pedro Ortiz Suarez , Laurent Romary , and Beno \^ t Sagot . 2022. https://arxiv.org/abs/2201.06642 Towards a Cleaner Document-Oriented Multilingual Crawled Corpus . arXiv e-prints, arXiv:2201.06642

    arXiv 2022

  2. [2]

    Aryaman Arora, Adam Farris, Samopriya Basu, and Suresh Kolichala. 2022. https://doi.org/10.18653/v1/2022.acl-long.99 Computational historical linguistics and language diversity in S outh A sia . In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1396--1409, Dublin, Ireland. Association...

    work page doi:10.18653/v1/2022.acl-long.99 2022

  3. [3]

    Viraat Aryabumi, Yixuan Su, Raymond Ma, Adrien Morisot, Ivan Zhang, Acyr Locatelli, Marzieh Fadaee, Ahmet Üstün, and Sara Hooker. 2024. https://arxiv.org/abs/2408.10914 To code, or not to code? exploring impact of code in pre-training . Preprint, arXiv:2408.10914

    arXiv 2024

  4. [4]

    Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell

    Emily M. Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell. 2021. https://doi.org/10.1145/3442188.3445922 On the dangers of stochastic parrots: Can language models be too big? In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, FAccT '21, page 610–623, New York, NY, USA. Association for Computing...

    arXiv 2021

  5. [5]

    Cunningham

    Dan Biderman, Jacob Portes, Jose Javier Gonzalez Ortiz, Mansheej Paul, Philip Greengard, Connor Jennings, Daniel King, Sam Havens, Vitaliy Chiley, Jonathan Frankle, Cody Blakeney, and John P. Cunningham. 2024. https://arxiv.org/abs/2405.09673 Lora learns less and forgets less . Preprint, arXiv:2405.09673

    arXiv 2024

  6. [6]

    Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin...

    arXiv 2020

  7. [7]

    Jiaao Chen, Derek Tam, Colin Raffel, Mohit Bansal, and Diyi Yang. 2023. https://doi.org/10.1162/tacl_a_00542 An empirical survey of data augmentation for limited data learning in NLP . Transactions of the Association for Computational Linguistics, 11:191--211

    doi:10.1162/tacl_a_00542 2023

  8. [8]

    Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018. https://arxiv.org/abs/1803.05457 Think you have solved question answering? try arc, the ai2 reasoning challenge . Preprint, arXiv:1803.05457

    arXiv 2018

Show all 48 references
  1. [9]

    Mike Conover, Matt Hayes, Ankit Mathur, Jianwei Xie, Jun Wan, Sam Shah, Ali Ghodsi, Patrick Wendell, Matei Zaharia, and Reynold Xin. 2023. https://www.databricks.com/blog/2023/04/12/dolly-first-open-commercially-viable-instruction-tuned-llm Free dolly: Introducing the world's ...

    2023

  2. [10]

    Marta R. Costa-jussà, James Cross, Onur Çelebi, Maha Elbayad, Kenneth Heafield, Kevin Heffernan, Elahe Kalbassi, Janice Lam, Daniel Licht, Jean Maillard, Anna Sun, Skyler Wang, Guillaume Wenzek, Al Youngblood, Bapi Akula, and Loic Barrault et al. 2022. https://arxiv.org/abs/22...

    2022 arXiv

  3. [11]

    Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. 2023. https://arxiv.org/abs/2305.14314 Qlora: Efficient finetuning of quantized llms . Preprint, arXiv:2305.14314

    2023 arXiv

  4. [12]

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, and Angela Fan et al. 2024. https://arxiv.org/abs/2407.21783 The llama 3 herd of models . Preprint, arXiv:2407.21783

    2024 arXiv

  5. [13]

    Feng, Varun Gangal, Jason Wei, Sarath Chandar, Soroush Vosoughi, Teruko Mitamura, and Eduard Hovy

    Steven Y. Feng, Varun Gangal, Jason Wei, Sarath Chandar, Soroush Vosoughi, Teruko Mitamura, and Eduard Hovy. 2021. https://doi.org/10.18653/v1/2021.findings-acl.84 A survey of data augmentation approaches for NLP . In Findings of the Association for Computational Linguistics: ...

    2021 doi

  6. [14]

    Chitale, Raghavan AK, Varun Gumma, Sumanth Doddapaneni, Aswanth Kumar, Janki Nawale, Anupama Sujatha, Ratish Puduppully, Vivek Raghavan, Pratyush Kumar, Mitesh M

    Jay Gala, Pranjal A. Chitale, Raghavan AK, Varun Gumma, Sumanth Doddapaneni, Aswanth Kumar, Janki Nawale, Anupama Sujatha, Ratish Puduppully, Vivek Raghavan, Pratyush Kumar, Mitesh M. Khapra, Raj Dabre, and Anoop Kunchukuttan. 2023. https://arxiv.org/abs/2305.16307 Indictrans2...

    2023 arXiv

  7. [15]

    Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac'h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang S...

    2024 doi

  8. [16]

    Suchin Gururangan, Ana Marasovi \'c , Swabha Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, and Noah A. Smith. 2020. https://doi.org/10.18653/v1/2020.acl-main.740 Don ' t stop pretraining: Adapt language models to domains and tasks . In Proceedings of the 58th Annual Meeting o...

    2020 doi

  9. [17]

    Francisco Guzm \'a n, Peng-Jen Chen, Myle Ott, Juan Pino, Guillaume Lample, Philipp Koehn, Vishrav Chaudhary, and Marc ' Aurelio Ranzato. 2019. https://doi.org/10.18653/v1/D19-1632 The FLORES evaluation datasets for low-resource machine translation: N epali -- E nglish and S i...

    2019 doi

  10. [18]

    Rishav Hada, Varun Gumma, Adrian de Wynter, Harshita Diddee, Mohamed Ahmed, Monojit Choudhury, Kalika Bali, and Sunayana Sitaram. 2024. https://arxiv.org/abs/2309.07462 Are large language model-based evaluators the solution to scaling up multilingual evaluation? Preprint, arXi...

    2024 arXiv

  11. [19]

    Daniel Han. 2023. https://github.com/unslothai/unsloth Unsloth ai: Open source fine-tuning for llms

    2023

  12. [20]

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021. https://arxiv.org/abs/2009.03300 Measuring massive multitask language understanding . Preprint, arXiv:2009.03300

    2021 arXiv

  13. [21]

    Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen

    Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021. https://arxiv.org/abs/2106.09685 Lora: Low-rank adaptation of large language models . Preprint, arXiv:2106.09685

    2021 arXiv

  14. [22]

    Rasmus K r J rgensen, Mareike Hartmann, Xiang Dai, and Desmond Elliott. 2021. https://doi.org/10.18653/v1/2021.findings-emnlp.290 m DAPT : Multilingual domain adaptive pretraining in a single model . In Findings of the Association for Computational Linguistics: EMNLP 2021, pag...

    2021 doi

  15. [23]

    Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei

    Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B. Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. 2020. https://arxiv.org/abs/2001.08361 Scaling laws for neural language models . Preprint, arXiv:2001.08361

    2020 arXiv

  16. [24]

    Vladislav Lialin, Namrata Shivagunde, Sherin Muckatira, and Anna Rumshisky. 2023. https://arxiv.org/abs/2307.05695 Relora: High-rank training through low-rank updates . Preprint, arXiv:2307.05695

    2023 arXiv

  17. [25]

    Stephanie Lin, Jacob Hilton, and Owain Evans. 2022. https://arxiv.org/abs/2109.07958 Truthfulqa: Measuring how models mimic human falsehoods . Preprint, arXiv:2109.07958

    2022 arXiv

  18. [26]

    Liu, Matt Gardner, Yonatan Belinkov, Matthew E

    Nelson F. Liu, Matt Gardner, Yonatan Belinkov, Matthew E. Peters, and Noah A. Smith. 2019. https://doi.org/10.18653/v1/N19-1112 Linguistic knowledge and transferability of contextual representations . In Proceedings of the 2019 Conference of the North A merican Chapter of the ...

    2019 doi

  19. [27]

    Xiao Liu, Hanyu Lai, Hao Yu, Yifan Xu, Aohan Zeng, Zhengxiao Du, Peng Zhang, Yuxiao Dong, and Jie Tang. 2023. https://arxiv.org/abs/2306.07906 Webglm: Towards an efficient web-enhanced question answering system with human preferences . Preprint, arXiv:2306.07906

    2023 arXiv

  20. [28]

    Meta. 2024. https://ai.meta.com/blog/meta-llama-3/ Introducing meta llama 3: The most capable openly available llm to date

    2024

  21. [29]

    Rush, Boaz Barak, Teven Le Scao, Aleksandra Piktus, Nouamane Tazi, Sampo Pyysalo, Thomas Wolf, and Colin Raffel

    Niklas Muennighoff, Alexander M. Rush, Boaz Barak, Teven Le Scao, Aleksandra Piktus, Nouamane Tazi, Sampo Pyysalo, Thomas Wolf, and Colin Raffel. 2023. https://arxiv.org/abs/2305.16264 Scaling data-constrained language models . Preprint, arXiv:2305.16264

    2023 arXiv

  22. [30]

    OpenAI. 2024. https://arxiv.org/abs/2303.08774 Gpt-4 technical report . Preprint, arXiv:2303.08774

    2024 arXiv

  23. [31]

    Patterson, Joseph Gonzalez, Quoc V

    David A. Patterson, Joseph Gonzalez, Quoc V. Le, Chen Liang, Llu \'i s-Miquel Mungu \'i a, Daniel Rothchild, David R. So, Maud Texier, and Jeff Dean. 2021. https://api.semanticscholar.org/CorpusID:233324338 Carbon emissions and large neural network training . ArXiv, abs/2104.10350

    2021 arXiv

  24. [32]

    Haggag et al

    Angelika Romanou, Negar Foroutan, Anna Sotnikova, Zeming Chen, Sree Harsha Nelaturu, Shivalika Singh, Rishabh Maheshwary, Micol Altomare, and Mohamed A. Haggag et al. 2024. https://arxiv.org/abs/2411.19799 Include: Evaluating multilingual language understanding with regional k...

    2024 arXiv

  25. [33]

    Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2019. https://arxiv.org/abs/1907.10641 Winogrande: An adversarial winograd schema challenge at scale . Preprint, arXiv:1907.10641

    2019 arXiv

  26. [34]

    SarvamAI. 2023. https://www.sarvam.ai/blog/announcing-openhathi-series Openhathi series: An approach to build bilingual llms frugally

    2023

  27. [35]

    Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. https://doi.org/10.18653/v1/P16-1009 Improving neural machine translation models with monolingual data . In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ...

    2016 doi

  28. [36]

    Luísa Shimabucoro, Sebastian Ruder, Julia Kreutzer, Marzieh Fadaee, and Sara Hooker. 2024. https://arxiv.org/abs/2407.01490 Llm see, llm do: Guiding data generation to target non-differentiable objectives . Preprint, arXiv:2407.01490

    2024 arXiv

  29. [37]

    Emma Strubell, Ananya Ganesh, and Andrew McCallum. 2019. https://api.semanticscholar.org/CorpusID:174802812 Energy and policy considerations for deep learning in nlp . ArXiv, abs/1906.02243

    2019 arXiv

  30. [38]

    Hashimoto

    Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023. https://github.com/tatsu-lab/stanford_alpaca Stanford alpaca: An instruction-following llama model

    2023

  31. [39]

    Jesse Vig. 2019. https://doi.org/10.18653/v1/P19-3007 A multiscale visualization of attention in the transformer model . In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics: System Demonstrations, pages 37--42, Florence, Italy. Associatio...

    2019 doi

  32. [40]

    Jesse Vig and Yonatan Belinkov. 2019. https://doi.org/10.18653/v1/W19-4808 Analyzing the structure of attention in a transformer language model . In Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 63--76, Florence, It...

    2019 doi

  33. [41]

    Pablo Villalobos, Anson Ho, Jaime Sevilla, Tamay Besiroglu, Lennart Heim, and Marius Hobbhahn. 2024. https://arxiv.org/abs/2211.04325 Will we run out of data? limits of llm scaling based on human-generated data . Preprint, arXiv:2211.04325

    2024 arXiv

  34. [42]

    Elena Voita, David Talbot, Fedor Moiseev, Rico Sennrich, and Ivan Titov. 2019. https://doi.org/10.18653/v1/P19-1580 Analyzing multi-head self-attention: Specialized heads do the heavy lifting, the rest can be pruned . In Proceedings of the 57th Annual Meeting of the Associatio...

    2019 doi

  35. [43]

    Wenhan Xia, Chengwei Qin, and Elad Hazan. 2024. https://arxiv.org/abs/2401.04151 Chain of lora: Efficient fine-tuning of language models via residual learning . Preprint, arXiv:2401.04151

    2024 arXiv

  36. [44]

    Rong Zhang, Revanth Gangi Reddy, Md Arafat Sultan, Vittorio Castelli, Anthony Ferritto, Radu Florian, Efsun Sarioglu Kayi, Salim Roukos, Avi Sil, and Todd Ward. 2020. https://doi.org/10.18653/v1/2020.emnlp-main.440 Multi-stage pre-training for low-resource domain adaptation . ...

    2020 doi

  37. [45]

    Jiawei Zhao, Zhenyu Zhang, Beidi Chen, Zhangyang Wang, Anima Anandkumar, and Yuandong Tian. 2024. https://arxiv.org/abs/2403.03507 Galore: Memory-efficient llm training by gradient low-rank projection . Preprint, arXiv:2403.03507

    2024 arXiv

  38. [46]

    Çağatay Yıldız, Nishaanth Kanna Ravichandran, Prishruit Punia, Matthias Bethge, and Beyza Ermis. 2024. https://arxiv.org/abs/2402.17400 Investigating continual pretraining in large language models: Insights and implications . Preprint, arXiv:2402.17400

    2024 arXiv

  39. [47]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  40. [48]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.