Pith. sign in

REVIEW 3 major objections 5 minor 48 references

TelcoLM: collecting data, adapting, and benchmarking language models for the telecommunication domain

T0 review · 3 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Adapting Llama-2-7B to the telecommunications domain needs only a single instruction-tuning step, not continued pretraining on raw domain text, and the adapted model competes with GPT-3.5.

desk verdict A serious data-and-benchmark contribution whose central "skip DAPT" claim is undermined by a test set generated from the same corpus used for training, so the headline result is not yet established. read the letter →

arxiv 2412.15891 v1 pith:5NAFEUTN submitted 2024-12-20 cs.CL cs.AI

Camille Barboule , Viet-Phi Huynh , Adrien Bufort , Yoan Chabot , Géraldine Damnati , Gwénolé Lecorvé This is my paper · ORCID
classification cs.CLcs.AI
keywords telecommunicationsdomainadaptationinstructiontuningcontinuedpretrainingLlama-2MCQAbenchmarklanguagemodelevaluationdomain-specificdata
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that adapting a general-purpose LLM (Llama-2-7B) to the telecommunications domain does not require continued pretraining on raw domain text (DAPT). Instead, a single instruction-tuning step (IAPT) on domain-specific and general instructions is enough to reach competitive accuracy on telco question-answering, and the best adapted models match GPT-3.5. This matters because instruction-tuning is much cheaper and simpler than continued pretraining, making domain adaptation feasible for organizations with limited compute. The claim rests on a newly collected 803M-token telco corpus and 80K instructions, plus a telco evaluation suite.

What carries the argument

The central object is the comparison of six adaptation recipes: DAPT-only, IAPT-only, and DAPT+IAPT with telco-only, general-only, or blended instructions. The load-bearing result is the near-tie between IAPT-only and DAPT+IAPT, at most +0.03 accuracy difference on telco MCQs, which makes the continued-pretraining step dispensable. The data machinery comprises an 803M-token raw corpus built from standards documents, scientific paper repositories, web corpora, and DSIR-selected sources, plus an 80K-instruction set generated from those documents and existing benchmark data. Evaluation uses telco MCQA, open QA, and abstract generation tasks.

What would settle it

A human-curated set of telco MCQs drawn from documents and exam sources not present in the DAPT/IAPT corpus, administered to the best IAPT-only model and GPT-3.5; if the IAPT-only model's accuracy drops near the unadapted Llama-2 baseline while GPT-3.5 holds, the claim that IAPT alone imparts usable domain knowledge is weakened.

Watch

Extended reading notes

Core claim

The central claim is that a single instruction-tuning step on a blend of domain-specific and general instructions is sufficient to adapt Llama-2-7B to the telecommunications domain; adding continued pretraining on raw telco text yields at most +0.03 accuracy on the telco MCQ benchmark. The adapted models reach accuracy comparable to GPT-3.5 on telco knowledge questions while remaining much smaller and cheaper to deploy.

Load-bearing premise

The in-domain telco benchmark is valid as a measure of domain knowledge rather than memorization, even though the synthetic test MCQs were generated from the same technical documents used to build the training sets.

Editorial extensions

If this is right

  • Telco-specific instruction tuning alone is a viable, far cheaper alternative to continued pretraining for adapting open 7B-scale models.
  • Blending domain and general instructions gives the best adaptation recipe, outperforming telco-only and general-only mixes.
  • The adapted Llama-2-7B models reach GPT-3.5-level accuracy on telco MCQ knowledge, suggesting in-house open models can replace proprietary APIs for niche domains.
  • Gains concentrate on knowledge explicitly present in the adaptation corpus; on out-of-domain exam MCQs, adaptation gains shrink.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the same benchmark-building logic is applied elsewhere, the 'instruction tuning suffices' result is likely to hold for any technical domain where domain knowledge can be densely converted into instruction-output pairs, but this extrapolation is not tested here.
  • The synthetic test MCQs share source documents with the training data, so part of the measured accuracy may reflect retrieval of memorized text; a human-curated held-out exam set would quantify how much.
  • The paper's open-QA examples suggest adapted models also answer more concisely; a focused study could separate conciseness from factual correctness.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. TelcoLM reports an empirical study of adapting Llama-2-7B to the telecommunications domain. The authors collect a raw-text corpus of about 803M tokens and an instruction set of about 80K examples, then compare continued pretraining (DAPT), instruction tuning (IAPT), and their combinations under different data mixtures. Evaluation covers telco MCQs, open QA on TeleQnA, abstract generation, and general-domain benchmarks. The paper's main claim is that IAPT alone suffices for telco adaptation and that DAPT adds at most +0.03 accuracy, with the best adapted models approaching GPT-3.5.

Significance. If the central claim held, it would be practically valuable: skipping DAPT substantially lowers compute and simplifies adaptation pipelines. The paper also provides a transparent description of data collection and a benchmark construction process, and it checks general-domain forgetting. However, the main quantitative evidence is currently hard to verify: the result tables contain no numerical values in the submitted text, and the in-domain telco MCQ benchmark is generated from the same paragraphs used in adaptation training. The 'DAPT unnecessary' conclusion therefore needs out-of-distribution support and uncertainty quantification before it can be accepted.

major comments (3)
  1. [§3.1.3 and Table 6] The main conclusion that DAPT adds at most +0.03 accuracy is based on a telco MCQ set generated by GPT-4 from ATIS/3GPP/ETSI paragraphs that are explicitly part of the DAPT training set, while the IAPT training set contains 11,235 instructions generated from the same raw corpus plus converted TeleQnA questions. As the paper itself acknowledges, these test questions concern 'knowledges present in the DAPT training set,' so high accuracy on them can be explained by memorization of training text rather than by domain competence. The result does not establish that DAPT is unnecessary for general telco knowledge; the Nokia MCQs, whose answers are not in the adaptation data, are reported only as 'notably less favorable' (Section 4.2.1), and the DAPT-vs-IAPT comparison on that subset is not given. Please report the per-condition accuracy on Nokia and other out-of-distribution questions, and use that, not the in-distribution synthetic MCQs, to support the 'DAPT unnecessary' claim.
  2. [Tables 5–9] In the submitted manuscript, Tables 5 through 9 are present only as captions; no numerical perplexities, accuracies, ROUGE/METEOR scores, MOS scores, or abstract lengths are shown. The abstract, Section 4, and Section 5 make quantitative claims (e.g., '+0.03 maximum,' '+0.09 on average,' 'best METEOR/MOS') that cannot be checked against the provided text. Please provide the full numerical tables, including sample sizes, and state whether scores are averaged over repeated runs; without this, the central empirical comparisons are not verifiable.
  3. [§4.2.1 and §5] The design has no statistical support for the 'no gain' conclusion. The observed DAPT gain is at most +0.03 accuracy, but no confidence intervals, standard errors, or multiple-seed results are reported anywhere; on test sets of the sizes described (e.g., 632 Nokia items, 900 TeleQnA items), a 0.03 difference may be within sampling variability. Even after the numerical tables are restored, the central negative claim requires an explicit uncertainty estimate or at least a multiple-seed comparison.
minor comments (5)
  1. [Abstract and §1] There are typos and duplicated text, including 'dicarding' in the abstract and 'on a wide range of tasks, on a wide range of tasks' in the first paragraph of Section 1.
  2. [§3.1.3] The sentence 'we have selected some paragraphs having fewer than 256 words or more than 3092 words (as described in 3.1.2)' appears to invert the filtering rule stated in Section 3.1.2; please clarify which paragraphs were actually used to generate the evaluation MCQs.
  3. [§3.1.2] The description of transforming MCQs into instructions says 'we added a prefix' and then quotes a full instruction; please clarify whether the quoted text is the complete instruction template and how the input/output fields are structured.
  4. [References] Several references are incomplete or inconsistently formatted, e.g., the in-text citation for 'Smith, 2020' is listed with a garbled author string, and the entry for 'Haxo ZXhang' should be checked; please harmonize with the journal's reference style.
  5. [§4.4] There is a typo 'intereting' in the abstract-generation section; please proofread the final text.

Circularity Check

1 steps flagged · score 5.0 of 10

The 'DAPT unnecessary' conclusion rests on a telco MCQ benchmark generated from the same DAPT paragraphs used to build IAPT training, making the central evaluation loop partly circular.

  1. fitted input called prediction [Sections 3.1.2, 3.1.3, and 4.2.1 (conclusion on DAPT necessity)]
    "The corpus is created by prompting GPT3.5 and Mixtral ... from pre-training raw text (Section 3.1.1)... We obtain, with this process, a total of 11,235 instructions / output pairs. ... For the MCQA telco benchmark, we have generated synthetic MCQs from select sources within our DAPT training set, namely A TIS, 3GPP , and ETSI standards, using GPT-4 with a 32k token model. ... For this type of MCQs, questions are about knowledges present in the DAPT training set, since we have generated these MCQs based on these knowledges."

    The IAPT training set is generated from the same pre-training raw text that contains the ATIS/3GPP/ETSI paragraphs, and the telco MCQ benchmark is generated by GPT-4 from 'select sources within our DAPT training set' whose questions are explicitly 'about knowledges present in the DAPT training set'. The paper then bases its main conclusion, 'DAPT is not necessary for domain adaptation, and IAPT-only is sufficient', on accuracy over this self-generated benchmark. A model can score well by retrieving or memorizing paragraphs already seen during IAPT instruction generation, so the IAPT-vs-DAPT comparison is not an independent test of generalizable telco knowledge; the evaluation loop reduces to measuring recall of the adaptation corpus.

full rationale

The paper's central claim is not a formal derivation from equations, but it is an empirical claim whose main support is a benchmark constructed from the same corpus used to generate the IAPT training instructions. That is a genuine, partial circularity: the synthetic MCQs are generated from DAPT training paragraphs, and the IAPT training instructions are generated from the same raw text, so the benchmark largely measures how well the adapted models reproduce content from their own training data. The paper explicitly acknowledges that the questions concern 'knowledges present in the DAPT training set', and it separately notes that results on the out-of-distribution Nokia MCQs are 'notably less favorable', which confirms that the favorable in-distribution scores do not establish broad telco generalization. Some independent grounding exists: the Nokia exam questions are not derived from adaptation data, general benchmarks show no major catastrophic forgetting, and the TeleQnA split provides held-out questions from a related source. The related-work citations, including Cheng et al., are external prior work rather than load-bearing self-citations by these authors, so they do not add to the circularity score. Overall, the core 'IAPT-only suffices' conclusion is partially forced by the evaluation design, but not wholly: the paper contains independent control points and openly notes the main limitation, so a moderate score of 5 is appropriate rather than a higher one.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

The paper introduces no new theoretical entities; the only assumptions are about the validity of LLM-generated data and LLM-based evaluation.

assumptions (3)
  • domain assumption The synthetic MCQs generated by GPT-4 from training paragraphs are correct and representative of telco knowledge.
    Section 3.1.3 uses these MCQs as the primary telco benchmark; any error or bias in generation directly affects the evaluation.
  • domain assumption GPT-4-128k scores (MOS, 1-5) align with human judgment for open QA and abstract generation.
    Sections 4.3.1 and 4.4 rely on GPT4-128k as the judge for open QA and abstract quality, following Prometheus; no human agreement is reported.
  • domain assumption The collected corpus covers the telco domain sufficiently to train domain knowledge.
    Section 3.1.1 assembles 803M tokens from standards, arXiv, Common Crawl and Stack Exchange; the adequacy of this coverage for telco is assumed rather than demonstrated.

how reviews work

0 comments
Cite this review

Pith. "Pith review of TelcoLM: collecting data, adapting, and benchmarking language models for the telecommunication domain." pith.science (2026). https://pith.science/paper/5NAFEUTN

@misc{pith2026241215891,
  author       = {Pith},
  title        = {Pith review of: TelcoLM: collecting data, adapting, and benchmarking language models for the telecommunication domain},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/5NAFEUTN}},
  note         = {Machine review of arXiv:2412.15891}
}
read the original abstract

Despite outstanding processes in many tasks, Large Language Models (LLMs) still lack accuracy when dealing with highly technical domains. Especially, telecommunications (telco) is a particularly challenging domain due the large amount of lexical, semantic and conceptual peculiarities. Yet, this domain holds many valuable use cases, directly linked to industrial needs. Hence, this paper studies how LLMs can be adapted to the telco domain. It reports our effort to (i) collect a massive corpus of domain-specific data (800M tokens, 80K instructions), (ii) perform adaptation using various methodologies, and (iii) benchmark them against larger generalist models in downstream tasks that require extensive knowledge of telecommunications. Our experiments on Llama-2-7b show that domain-adapted models can challenge the large generalist models. They also suggest that adaptation can be restricted to a unique instruction-tuning step, dicarding the need for any fine-tuning on raw texts beforehand.

Figures

Figures reproduced from arXiv: 2412.15891 by the authors.

Figure 1
Figure 1. Steps for LM training and adaptation. domain-specific topics tend to be underrepre￾sented, which inevitably leads to challenges in effectively learning them for domain-specific tasks. Additionally, many domain-specific knowl￾edge resources are proprietary assets, critical to an organization’s competitive edge, and cannot be readily shared with general-purpose LLMs. Adapting a language model to a target domain, calle… view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

48 extracted references · 4 canonical work pages

  1. [1]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Lina Bariah, Qiyang Zhao, Hang Zou, Yu Tian, Faouzi Bader, and Merouane Debbah. 2023 a . Large language models for telecom: The next big thing? arXiv preprint arXiv:2306.10249

    arXiv 2023

  4. [4]

    Lina Bariah, Hang Zou, Qiyang Zhao, Belkacem Mouhouche, Faouzi Bader, and Merouane Debbah. 2023 b . Understanding telecom language through large language models. arXiv preprint arXiv:2306.07933

    arXiv 2023

  5. [5]

    Loubna Ben Allal, Anton Lozhkov, Guilherme Penedo, Thomas Wolf, and Leandro von Werra. 2024. https://huggingface.co/datasets/HuggingFaceTB/cosmopedia Cosmopedia

    2024

  6. [6]

    Lukas Blecher, Guillem Cucurull, Thomas Scialom, and Robert Stojnic. 2023. Nougat: Neural optical understanding for academic documents. arXiv preprint arXiv:2308.13418

    arXiv 2023

  7. [7]

    El Boukkouri, Ferret, Lavergne, and Zweigenbaum. 2022. Re-train or train from scratch? comparing pre-training strategies of bert in the medical domain. HAL:cea-03745341

    work page 2022

  8. [9]

    Daixuan Cheng, Shaohan Huang, and Furu Wei. 2023 b . Adapting large language models via reading comprehension. arXiv preprint arXiv:2309.09530

    arXiv 2023

Show all 48 references
  1. [10]

    Chowdhery, Narang, Devlin, Bosma, Mishra, Roberts, Barham, Won Chung, Sutton, and Gehrmann. 2022. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311

    2022 arXiv

  2. [11]

    Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. 2023. Qlora: Efficient finetuning of quantized llms. arXiv preprint arXiv:2305.14314

    2023 arXiv

  3. [12]

    Leo Gao, Stella Biderman, Sid Black, Laurence Golding, Travis Hoppe, Charles Foster, Jason Phang, Horace He, Anish Thite, Noa Nabeshima, et al. 2020. The P ile: An 800 GB dataset of diverse text for language modeling. arXiv preprint arXiv:2101.00027

    2020 arXiv

  4. [13]

    Suriya Gunasekar, Yi Zhang, Jyoti Aneja, Caio C \'e sar Teodoro Mendes, Allie Del Giorno, Sivakanth Gopi, Mojan Javaheripi, Piero Kauffmann, Gustavo de Rosa, Olli Saarikivi, et al. 2023. Textbooks are all you need. arXiv preprint arXiv:2306.11644

    2023 arXiv

  5. [14]

    Guu, Lee, Tung, Pasupat, and Chang. 2020. Realm: Retrieval-augmented language model pre-training. arXiv preprint arXiv:2002.08909

    2020 arXiv

  6. [15]

    Henrik Holm. 2021. Bidirectional encoder representations from transformers (bert) for question answering in the telecom domain.: Adapting a bert-like language model to the telecom domain using the electra pre-training approach

    2021

  7. [16]

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685

    2021 arXiv

  8. [17]

    Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas ...

    2023 arXiv

  9. [18]

    Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al. 2024. Mixtral of experts. arXiv preprint arXiv:2401.04088

    2024 arXiv

  10. [19]

    Seungone Kim, Jamin Shin, Yejin Cho, Joel Jang, Shayne Longpre, Hwaran Lee, Sangdoo Yun, Seongjin Shin, Sungdong Kim, James Thorne, et al. 2023. Prometheus: Inducing fine-grained evaluation capability in language models. In The Twelfth International Conference on Learning Repr...

    2023

  11. [20]

    Alycia Lee, Brando Miranda, and Sanmi Koyejo. 2023. Beyond scale: the diversity coefficient as a data quality metric demonstrates llms are pre-trained on formally diverse data. arXiv preprint arXiv:2306.13840

    2023 arXiv

  12. [21]

    Vladislav Lialin, Vijeta Deshpande, and Anna Rumshisky. 2023 a . Scaling down to scale up: A guide to parameter-efficient fine-tuning. arXiv preprint arXiv:2303.15647

    2023 arXiv

  13. [22]

    Vladislav Lialin, Namrata Shivagunde, Sherin Muckatira, and Anna Rumshisky. 2023 b . Stack more layers differently: High-rank training through low-rank updates. arXiv preprint arXiv:2307.05695

    2023 arXiv

  14. [23]

    Wing Lian, Bleys Goodson, Eugene Pentland, Austin Cook, Chanvichet Vong, and "Teknium". 2023. Openorca: An open dataset of gpt augmented flan reasoning traces. https://https://huggingface.co/Open-Orca/OpenOrca

    2023

  15. [24]

    Stephanie Lin, Jacob Hilton, and Owain Evans. 2021. Truthfulqa: Measuring how models mimic human falsehoods. arXiv preprint arXiv:2109.07958

    2021 arXiv

  16. [25]

    Mingjie Liu, Teodor-Dumitru Ene, Robert Kirby, Chris Cheng, Nathaniel Pinckney, Rongjian Liang, Jonah Alben, Himyanshu Anand, Sanmitra Banerjee, Ismet Bayraktaroglu, Bonita Bhaskaran, Bryan Catanzaro, Arjun Chaudhuri, Sharon Clay, Bill Dally, Laura Dang, Parikshit Deshpande, S...

    2023 arXiv

  17. [26]

    Longpre, Perisetla, Chen, Ramesh, DuBois, and Singh. 2022. Entity-based knowledge conflicts in question answering. arXiv preprint arXiv:2109.05052

    2022 arXiv

  18. [27]

    Shayne Longpre, Gregory Yauney, Emily Reif, Katherine Lee, Adam Roberts, Barret Zoph, Denny Zhou, Jason Wei, Kevin Robinson, David Mimno, et al. 2023. A pretrainer's guide to training data: Measuring the effects of data age, domain coverage, quality, & toxicity. arXiv preprint...

    2023 arXiv

  19. [28]

    Anton Lozhkov, Raymond Li, Loubna Ben Allal, Federico Cassano, Joel Lamy-Poirier, Nouamane Tazi, Ao Tang, Dmytro Pykhtar, Jiawei Liu, Yuxiang Wei, et al. 2024. Starcoder 2 and the stack v2: The next generation. arXiv preprint arXiv:2402.19173

    2024 arXiv

  20. [29]

    Ali Maatouk, Fadhel Ayed, Nicola Piovesan abd Antonio De Domenico, Merouane Debbah, and Zhi-Quan Luo. 2023 a . Teleqna: A benchmark dataset to assess large language models telecommunications knowledge. arXiv preprint arXiv:2310.15051

    2023 arXiv

  21. [30]

    Ali Maatouk, Nicola Piovesan, Fadhel Ayed, Antonio De Domenico, and Merouane Debbah. 2023 b . Large language models for telecom: Forthcoming impact on the industry. arXiv preprint arXiv:2308.06013

    2023 arXiv

  22. [31]

    Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. 2018. Can a suit of armor conduct electricity? a new dataset for open book question answering. arXiv preprint arXiv:1809.02789

    2018 arXiv

  23. [32]

    Mosin, Samenko, Tikhonov, Kozlovskii, and Yamshchikov. 2022. Fine-tuning transformers: Vocabulary transfer. arXiv preprint arXiv:2112.14569

    2022 arXiv

  24. [33]

    Subhabrata Mukherjee, Arindam Mitra, Ganesh Jawahar, Sahaj Agarwal, Hamid Palangi, and Ahmed Awadallah. 2023. Orca: Progressive learning from complex explanation traces of gpt-4. arXiv preprint arXiv:2306.02707

    2023 arXiv

  25. [34]

    openai. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774

    2023 arXiv

  26. [35]

    Penedo, Malartic, Hesslow, Cojocaru, Cappelli, Alobeidli, Pannier, Almazrouei, and Launay. 2023. The refinedweb dataset for falcon llm: Outperforming curated corpora with web data, and web data only. arXiv preprint arXiv:2306.01116

    2023 arXiv

  27. [36]

    Piktus, Petroni, Karpukhin, Goyal, Küttler, Lewis, Yih, Rocktäschel, Riedel, and Kiela. 2021. Retrieval-augmented generation for knowledge-intensive nlp tasks. arXiv preprint arXiv:2005.11401

    2021 arXiv

  28. [37]

    Jack W Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, et al. 2021. Scaling language models: Methods, analysis & insights from training gopher. arXiv preprint arXiv:2112.11446

    2021 arXiv

  29. [38]

    Jie Ren, Samyam Rajbhandari, Reza Yazdani Aminabadi, Olatunji Ruwase, Shuangyan Yang, Minjia Zhan, Dong Li, and Yuxiong He. 2021. Zero-offload: Democratizing billion-scale model training. arXiv preprint arXiv:2101.06840

    2021 arXiv

  30. [39]

    Gururangan Marasović Swayamdipta Lo Beltagy Downey Smith. 2020. Don’t stop pretraining: Adapt language models to domains and tasks. arXiv preprint arXiv:2004.10964

    2020 arXiv

  31. [40]

    Daria Soboleva, Faisal Al-Khateeb, Robert Myers, Jacob R Steeves, Joel Hestness, and Nolan Dey. 2023. https://huggingface.co/datasets/cerebras/SlimPajama-627B SlimPajama: A 627B token cleaned and deduplicated version of RedPajama . https://www.cerebras.net/blog/slimpajama-a-62...

    2023

  32. [41]

    Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonso, Agnieszka Kluska, Aitor Lewkowycz, Akshat Agarwal, Alethea Power, Alex Ray, Alex Warstadt, Alexander W

    Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R. Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonso, Agnieszka Kluska, Aitor Lewkowycz, Akshat Agarwal, Alethea Power, Alex Ray, Alex Warstadt, Alexander W. Kocurek, Ali...

    2022 arXiv

  33. [42]

    Touvron, Martin, Stone, Albert, Almahairi, Babaei, Bashlykov, and Batra. 2023. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288

    2023 arXiv

  34. [43]

    Rush, and Thomas Wolf

    Lewis Tunstall, Edward Beeching, Nathan Lambert, Nazneen Rajani, Kashif Rasul, Younes Belkada, Shengyi Huang, Leandro von Werra, Clémentine Fourrier, Nathan Habib, Nathan Sarrazin, Omar Sanseviero, Alexander M. Rush, and Thomas Wolf. 2023. https://arxiv.org/abs/2310.16944 Zeph...

    2023 arXiv

  35. [44]

    Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A Smith, Daniel Khashabi, and Hannaneh Hajishirzi. 2023. Self-instruct: Aligning language models with self-generated instructions. In Proceedings of the 61st Annual Meeting of the Association for Computational Lingui...

    2023

  36. [45]

    Sang Michael Xie, Shibani Santurkar, Tengyu Ma, and Percy Liang. 2023. Data selection for language models via importance resampling. arXiv preprint arXiv:2302.03169

    2023 arXiv

  37. [46]

    Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, and Daxin Jiang. 2023. Wizardlm: Empowering large language models to follow complex instructions. arXiv preprint arXiv:2304.12244

    2023 arXiv

  38. [47]

    Zhang and Choi. 2021. Situatedqa: Incorporating extra-linguistic contexts into qa. arXiv preprint arXiv:2109.06157

    2021 arXiv

  39. [48]

    Zhao, Lu, Deng, Zheng, Wang, Chowdhury, Li, Cui, Zhang, Zhao, Cheng, White, Gu, Pei, and Yang. 2023. Domain specialization as the key to make large language models disruptive: A comprehensive survey. arXiv preprint arXiv:2305.18703

    2023 arXiv

  40. [49]

    Haxo ZXhang. 2023. Falcon llm: a new frontier in natural language processing. AC Investment Research Journal 220

    2023

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.