Pith. sign in

REVIEW 3 major objections 5 minor 36 references

Domain Adaptation of Foundation LLMs for e-Commerce

T0 review · 3 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Continued pretraining of Llama 3.1 on one trillion e-commerce tokens yields 8B and 70B models that gain roughly 25–30% on new e-commerce benchmarks while general-language scores stay nearly flat.

desk verdict A useful industrial-scale continued-pretraining recipe with new benchmarks, but the author-built e-commerce tasks are too close to the training data to fully support the abstract's claim. read the letter →

arxiv 2501.09706 v3 pith:QMULW2BF submitted 2025-01-16 cs.CL

Christian Herold , Michael Kozielski , Tala Bazazo , Pavel Petrushkov , Patrycja Cieplicka , Dominika Basaj , Yannick Versley , Seyyed Hadi Hashemi
show 1 more author
Shahram Khadivi
This is my paper · ORCID
classification cs.CL
keywords continuedpretrainingdomainadaptatione-commercefoundationmodelsLlama3.1catastrophicforgettingmodelmergingmultilingualevaluation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Large language models trained on general web text know little about the specifics of online marketplaces — product categories, aspect attributes, prices, and how listings are phrased. The paper asks whether a general-purpose model can be turned into an e-commerce foundation model by continued pretraining, and whether that can be done without destroying its general abilities. It answers yes: after training 8B and 70B Llama 3.1 models on one trillion tokens of mixed e-commerce and general data, the adapted models improve average accuracy on five new multilingual e-commerce tasks by about 25% in English and 30% in other languages, while standard NLU aggregates move by about one point. The recipe that makes this work is a 50/50 domain/general data mix, a learning rate about one tenth of the original, and a short context window. A final trick — averaging the weights of the base and adapted models — gives a near-linear dial for trading domain gain against general competence.

What carries the argument

Three components carry the argument. Continued pretraining with a carefully chosen setup — a 50/50 mix of e-commerce and general-domain data at a maximum learning rate of about 10% of the original pretraining rate — injects domain knowledge while replaying general data to limit forgetting. The evaluation is a set of five new multilingual benchmarks built from marketplace listings: aspect prediction, aspect prediction multiple choice, price prediction multiple choice, most common aspects, and most common aspects multiple choice. Finally, model merging — averaging the parameters of the base and adapted checkpoints — provides a compute-free control knob, since the resulting performance varies almost linearly with the interpolation weight.

What would settle it

Rebuild the five e-commerce tasks on a held-out sample of marketplace listings that were created after the training data was collected, or from a different marketplace, and compare the adapted models against the base models on that sample; if the accuracy gains shrink to a few points, the reported adaptation gains are largely memorization of the training distribution rather than transferable e-commerce knowledge.

Watch

Extended reading notes

Core claim

The central discovery is that domain adaptation of a modern foundation model can be done at trillion-token scale without catastrophic forgetting, provided the training setup is tuned explicitly for retention. The e-Llama models are the evidence: continued pretraining of Llama 3.1 8B and 70B on one trillion tokens (50% e-commerce, 50% general data, learning rate capped at about one tenth of the base pretraining value) raises the average accuracy across the paper's five new e-commerce benchmarks from 35.8 to 46.8 for the 8B model and from 40.4 to 52.8 for the 70B model, corresponding to roughly 25–30% relative gains. On general-language aggregates the models stay within about one point, with one caveat: the 8B variant drops 4.6 points on the English leaderboard aggregate, a loss the authors attribute to smaller model size and distribution differences in their general data. The paper's additional claim is that merging the adapted and base checkpoints by simple weight averaging produces a near-linear performance curve, so practitioners can dial the domain/general balance without retraining.

Load-bearing premise

The whole 'adapted to e-commerce' conclusion rests on the five new e-commerce benchmarks being fair, unbiased tests of e-commerce knowledge rather than easy to game by memorizing the training data.

Editorial extensions

If this is right

  • A general-purpose base model can be repurposed for a vertical domain with one trillion tokens of mixed data, which is far cheaper than pretraining from scratch.
  • The 70B model forgets less than the 8B model while learning the domain at least as well, so larger models are the safer choice when both domain gain and general retention matter.
  • Weight averaging gives an exact, inference-time-free way to pick a point on the domain/general trade-off curve, so a single training run can serve multiple product requirements.
  • The five e-commerce benchmarks give the field a reusable multilingual evaluation for foundation-model e-commerce knowledge, filling a gap left by instruction-tuning-only evaluations.
  • Because the non-English benchmarks improve more than the English ones, the adaptation also transfers knowledge across languages within the marketplace domain.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The headline gain is only as trustworthy as the five new benchmarks; since they are constructed from the same marketplace data used for training and no overlap analysis is given, some of the improvement could reflect memorization of the training distribution rather than transferable e-commerce knowledge.
  • The near-linear merging curve suggests the base and adapted models occupy a shared loss basin; if that holds for other domain pairs, weight averaging could become a general control mechanism for continued pretraining beyond e-commerce.
  • The paper evaluates only base models, so the natural next test is whether the domain gains survive instruction tuning — the authors list this as future work, but it is also the setup that would matter for most real deployments.
  • Because the paper's general-domain data mix differs from the original pretraining distribution, the 'no significant sacrifice' claim is benchmark-dependent; the 8B leaderboard drop of 4.6 points shows the trade-off is real even when aggregate NLU scores look flat.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper presents e-Llama 8B and 70B, obtained by continued pretraining of Llama 3.1 base models on 1 trillion tokens of a mixture containing 50% e-commerce data (eBay listings, product reviews, and classifier-selected Fineweb content) and 50% general-domain data. The authors describe hyperparameter studies (learning rate, e-commerce data ratio, context size) and define five new multilingual e-commerce evaluation tasks (AP, APMC, PPMC, MCA, MCAMC). They report that e-Llama improves e-commerce benchmark accuracy by about 25% on English and 30% on non-English tasks relative to Llama 3.1, while general NLU aggregates remain roughly flat, though the 8B model drops notably on the Open LLM Leaderboard English aggregate. They also propose linear interpolation (model merging) between the base and adapted checkpoints as a way to control the general/domain performance trade-off.

Significance. If the central empirical claim is valid, the paper provides a practically useful recipe for large-scale domain adaptation of 8B and 70B foundation models, with unusually large training scale (1T tokens) and a useful set of ablations. The learning-rate and data-mixing experiments are informative, and the model-merging result is a simple but potentially valuable practical tool. The main significance, however, is conditional: all e-commerce gains are measured on author-constructed tasks built from the same data source as the continued-pretraining corpus, and the paper reports no contamination analysis, no external benchmark, and no heldout description for these tasks. The general-domain preservation claim is also overstated for the 8B model in light of a 4.6-point drop on the Open LLM Leaderboard English aggregate. The paper is therefore a useful engineering report whose central conclusion requires additional validation before it can be fully accepted.

major comments (3)
  1. [§3.2, §3.3, §7] The five e-commerce benchmarks introduced in Section 3.3 are constructed from eBay listing data, while Section 3.2 states that continued pretraining uses 'listings and product reviews from the eBay website.' The paper gives no heldout split, no deduplication procedure, no temporal split, and no analysis of overlap between the evaluation instances and the 1-trillion-token training corpus. For tasks like AP and MCA, which ask the model to generate the most probable continuation of aspect key-value pairs, and APMC/PP/MCAMC, which ask it to select the correct value among corrupted alternatives, high scores may reflect retrieval of memorized training strings rather than generalizable e-commerce competence. This is load-bearing for the abstract's claim that the models are 'adapted towards the e-commerce domain.' The limitation statement (iii) acknowledges that evaluation diversity could be improved, but the issue is not only diversity: it is the lack of any contamination control or external validation. I ask the authors to add a contamination analysis (e.g., n-gram overlap statistics, deduplication, or a temporal split) or to evaluate on an independent e-commerce benchmark suite to demonstrate that the measured gains are not in-distribution fitting.
  2. [§5.2, Table 4, Abstract] The abstract states that the models can be adapted 'without sacrificing significant performance on general domain tasks,' but Table 4 shows that the 8B model drops from 17.2 to 12.6 on the Open LLM Leaderboard English aggregate, a 4.6-point (approximately 27% relative) decrease. The NLU English aggregate is stable (71.8 to 71.6), but the leaderboard drop is substantial and should be reported prominently. The sentence should be revised to acknowledge the model-size-dependent degradation, or the authors should provide statistical significance or confidence intervals to support the word 'significant.' As written, the general-domain preservation claim is too strong for the 8B model.
  3. [§4.2, Table 3] The data-weighting ablation is run for only about 30 billion tokens and reports only a single averaged 'general benchmark' score (e.g., 43.2 to 43.3 across ratios). The paper concludes that varying the e-commerce ratio 'does not effect the model scores on the general domain benchmarks,' but this conclusion is not supported by the short training horizon or by the aggregate-only reporting. Since the final 1-trillion-token model does show a measurable leaderboard drop, the ablation should at least report per-benchmark numbers and, ideally, longer training runs, so that the choice of the 50% e-commerce ratio can be properly evaluated against the final-model trade-off.
minor comments (5)
  1. [§3.3 and Appendix A.2] The task abbreviations are inconsistent: the text uses 'PP MC' and 'PP MC' in the task list, while Table 4 and Appendix A.1 use 'PPMC' and 'PP MC'; please standardize to a single abbreviation.
  2. [§4.1, Table 2] The learning-rate comparison reports only the averaged general benchmark score. Since the final claim concerns the preservation of general capabilities, including the separate NLU and LLM Leaderboard components in Table 2 would make the learning-rate choice more transparent and reduce reliance on a potentially misleading average.
  3. [§4.3, Figure 1] Figure 1 plots perplexity against input sequence length, but the figure caption does not state whether the heldout test set is general-domain or e-commerce data, nor how many sequences are used. Please add this information, as the interpretation of the long-context behavior depends on it.
  4. [§3.1] The hardware description gives node and GPU counts but no total training time or estimated FLOPs. Reporting wall-clock time or throughput would help readers assess the practical cost of the 1-trillion-token continued pretraining and would better contextualize the paper's scale claims.
  5. [§5.3, Figure 3] The model-merging figure shows a nearly linear trade-off, but the caption does not report the evaluation tasks behind the 'general' and 'e-commerce' aggregates, nor whether this linearity holds for the 70B model. Please clarify which benchmarks are used in the figure and whether the result is consistent across model sizes.

Circularity Check

2 steps flagged · score 6.0 of 10

Central e-commerce benchmarks are constructed from the same eBay listing data used for continued pretraining, so the headline adaptation gain partly reduces to the training objective; secondary reliance on the authors' prior Lilium paper for data and non-English evaluation aggregates adds to the concern.

  1. self definitional [Section 3.2 (Data) and Section 3.3 (Evaluation), e-Commerce task definitions; abstract]
    "Regarding the e-commerce domain, we employ several data sources. On the one hand, we utilize listings and product reviews from the eBay website, as has been done by Herold et al. (2024). ... we define a total of 5 novel e-commerce benchmarks for evaluation of foundation models. ... They revolve around the listings on an e-commerce website, of which we consider title, category, price and a list of aspect key-value pairs."

    The five new e-commerce tasks are built from the same eBay listing schema (title, category, price, aspect key-value pairs) that makes up the continued-pretraining corpus. AP and MCA ask the model to generate the most probable continuation of a listing description, which is exactly the next-token prediction objective used in continued pretraining; APMC, PPMC, and MCAMC ask it to select the highest-probability listing among corrupted alternatives, i.e., to rank by in-distribution likelihood. The paper reports no deduplication, temporal split, or train/evaluation overlap analysis for these benchmarks, so the reported improvement over Llama-3.1 is at least partly a measure of fitting the training distribution.

  2. self citation load bearing [Section 3.2 (Data) and Section 3.3 (Evaluation), General Domain]
    "Regarding training data, we mostly follow Herold et al. (2024). ... For the evaluation of the non-English, general domain NLU capabilities we use the same task aggregates as Herold et al. (2024)."

    Herold et al. (2024) is the authors' own Lilium paper, with overlapping authorship. It is the source of the continued-pretraining data mixture and of the non-English general-domain NLU aggregates used to support the claim that general performance is not significantly sacrificed. The non-English half of the trade-off claim is therefore assessed on author-group-defined aggregates rather than on an independently re-derived evaluation. This is secondary to the main circularity: the English general benchmarks are external, and the e-commerce benchmarks are newly defined, so the self-citation alone does not force the central result, but it does make part of the 'no degradation' claim dependent on the authors' own prior evaluation setup.

full rationale

The paper's central e-commerce improvement claim is partly circular. The five novel benchmarks (AP, APMC, PPMC, MCA, MCAMC) are defined and implemented by the authors from eBay listing data, while the continued-pretraining corpus is also eBay listings and product reviews. Because AP and MCA are continuation tasks over the same listing schema, continued pretraining directly optimizes the benchmark objective, and without any reported contamination or deduplication control the headline ~25% (English) and ~30% (non-English) gains cannot be separated from in-distribution fitting. The general-domain preservation claim is more independent: English NLU and Open LLM Leaderboard aggregates are external, and the paper reports only modest degradation there. However, the non-English general-domain aggregates are inherited from the authors' own prior Lilium paper, adding a secondary self-citation dependency. The limitation section (iii) acknowledges the evaluation suite could be improved in diversity and quantity but does not address the source-overlap problem. Overall the training ablations, model merging, and English external benchmarks are genuine, so the paper is not wholly circular, but the main e-commerce evaluation is self-referential enough to warrant a score of 6.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The paper's central claim depends on hyperparameters chosen by ablation (LR, data mix ratio, context size, few-shot counts) and on domain assumptions about the validity and representativeness of the in-house e-commerce benchmarks. No new physical or conceptual entities are introduced.

free parameters (4)
  • maximum learning rate LRmax = 3e-5 (8B); 1.5e-5 (70B)
    Set to 10% of the original Llama pretraining LR based on the Table 2 ablations; directly controls the general/domain trade-off central to the paper's claim.
  • e-commerce data ratio = 50%
    Chosen from the 30B-token ablation in Table 3 as the point of diminishing returns for e-commerce gains before general perplexity degrades.
  • training context size = 8k tokens
    Chosen because most examples are shorter than 1k tokens and longer contexts slow training; long-context retention is only verified by perplexity in Figure 1.
  • few-shot example counts per e-commerce task = AP 5-shot; APMC 1-shot; PPMC 0-shot; MCA 20-shot; MCAMC 5-shot
    Tuned on base Llama-3 8B in Appendix A.2 and then used in final evaluation; these choices affect the reported e-commerce accuracy numbers.
assumptions (4)
  • domain assumption The five self-defined e-commerce benchmarks measure meaningful e-commerce competence in a way that supports comparison across models.
    Section 3.3 introduces them as novel benchmarks; there is no external validation, no inter-annotator agreement, and no evidence that the multiple-choice formats reflect real downstream task value.
  • domain assumption Findings from English benchmark ablations transfer to non-English languages and to the 70B model.
    Section 4 states 'we assume the non-English languages will follow the same trend' and uses English-only ablations to set global hyperparameters.
  • domain assumption Continued pretraining at 8k context preserves the base model's long-context ability.
    Figure 1 shows perplexity only on a general-domain heldout set; no downstream long-context task is evaluated despite the conclusion that degradation is an 'acceptable trade-off'.
  • domain assumption The eBay listings/reviews plus Fineweb e-commerce classifier extraction is representative of e-commerce language.
    Section 3.2 describes the data mixture qualitatively; the classifier is not specified and the proprietary split is not disclosed, so domain coverage cannot be independently assessed.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Domain Adaptation of Foundation LLMs for e-Commerce." pith.science (2026). https://pith.science/paper/QMULW2BF

@misc{pith2026250109706,
  author       = {Pith},
  title        = {Pith review of: Domain Adaptation of Foundation LLMs for e-Commerce},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/QMULW2BF}},
  note         = {Machine review of arXiv:2501.09706}
}
read the original abstract

We present the e-Llama models: 8 billion and 70 billion parameter large language models that are adapted towards the e-commerce domain. These models are meant as foundation models with deep knowledge about e-commerce, that form a base for instruction- and fine-tuning. The e-Llama models are obtained by continuously pretraining the Llama 3.1 base models on 1 trillion tokens of domain-specific data. We discuss our approach and motivate our choice of hyperparameters with a series of ablation studies. To quantify how well the models have been adapted to the e-commerce domain, we define and implement a set of multilingual, e-commerce specific evaluation tasks. We show that, when carefully choosing the training setup, the Llama 3.1 models can be adapted towards the new domain without sacrificing significant performance on general domain tasks. We also explore the possibility of merging the adapted model and the base model for a better control of the performance trade-off between domains.

Figures

Figures reproduced from arXiv: 2501.09706 by the authors.

Figure 1
Figure 1. Perplexity as a function of the input sequence [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Evolution of the model performance over the course of the training. Left: perplexity on heldout data sets [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Model merging: 8B Model performance on English general and e-commerce benchmarks as a func￾tion of the weight of the e-Llama model parameters vs the base Llama-3.1 model parameters. ing data does not allow to make very precise fore￾casts of final model performance. Instead, we uti￾lize a technique called ‘model merging’ (Worts￾man et al., 2022), where we simply average all pa￾rameters of the base Llama-3.1 model che… view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Llama-3 8B model performance on the 5 e [PITH_FULL_IMAGE:figures/full_fig_p011_4.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

36 extracted references · 4 canonical work pages

  1. [1]

    Dai, Anja Hauth, Katie Millican, David Silver, Slav Petrov, Melvin Johnson, Ioannis Antonoglou, Julian Schrittwieser, Amelia Glaese, Jilin Chen, Emily Pitler, Timothy P

    Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean - Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M. Dai, Anja Hauth, Katie Millican, David Silver, Slav Petrov, Melvin Johnson, Ioannis Antonoglou, Julian Schrittwieser, Amelia Glaese, Jilin Chen, Emily Pitler, Timothy P. Lillicrap, Angeliki Lazaridou, Orhan Firat, James Molloy, Michael ...

    arXiv doi:10.48550/arxiv.2312.11805 2023

  2. [2]

    Anthropic. 2024. The Claude 3 Model Family: Opus, Sonnet, Haiku . Technical report, Anthropic AI . https://www-cdn.anthropic.com/de8ba9b01c9ab7cbabf5c33b80b7bbc618857627/Model_Card_Claude_3.pdf

    work page 2024

  3. [3]

    Jiang, Jia Deng, Stella Biderman, and Sean Welleck

    Zhangir Azerbayev, Hailey Schoelkopf, Keiran Paster, Marco Dos Santos, Stephen Marcus McAleer, Albert Q. Jiang, Jia Deng, Stella Biderman, and Sean Welleck. 2024. https://openreview.net/forum?id=4WnqRR915j Llemma: An open language model for mathematics . In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7...

    work page 2024

  4. [4]

    Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert - Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litw...

    arXiv 2020

  5. [5]

    Zeming Chen, Alejandro Hern \' a ndez - Cano, Angelika Romanou, Antoine Bonnet, Kyle Matoba, Francesco Salvi, Matteo Pagliardini, Simin Fan, Andreas K \" o pf, Amirkeivan Mohtashami, Alexandre Sallinen, Alireza Sakhaeirad, Vinitra Swamy, Igor Krawczuk, Deniz Bayazit, Axel Marmet, Syrielle Montariol, Mary - Anne Hartley, Martin Jaggi, and Antoine Bosselut....

    arXiv doi:10.48550/arxiv.2311.16079 2023

  6. [6]

    Pierre Colombo, Telmo Pires, Malik Boudiaf, Rui Melo, Dominic Culver, Sofia Morgado, Etienne Malaboeuf, Gabriel Hautreux, Johanne Charpentier, and Michael Desa. 2024 a . https://doi.org/10.48550/ARXIV.2407.19584 Saullm-54b & saullm-141b: Scaling up domain adaptation for the legal domain . CoRR, abs/2407.19584

    arXiv doi:10.48550/arxiv.2407.19584 2024

  7. [7]

    Pierre Colombo, Telmo Pessoa Pires, Malik Boudiaf, Dominic Culver, Rui Melo, Caio Corro, Andr \' e F. T. Martins, Fabrizio Esposito, Vera L \' u cia Raposo, Sofia Morgado, and Michael Desa. 2024 b . https://doi.org/10.48550/ARXIV.2403.03883 Saullm-7b: A pioneering large language model for law . CoRR, abs/2403.03883

    arXiv doi:10.48550/arxiv.2403.03883 2024

  8. [8]

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al - Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, Aur \' e lien Rodriguez, Austen Gregerson, Ava Spataru, Baptiste Rozi \` e...

    arXiv doi:10.48550/arxiv.2407.21783 2024

Show all 36 references
  1. [9]

    Jiarui Fang and Shangchun Zhao. 2024. https://doi.org/10.48550/ARXIV.2405.07719 USP: A unified sequence parallelism approach for long context generative AI . CoRR, abs/2405.07719

    2024 doi arXiv

  2. [10]

    Clémentine Fourrier, Nathan Habib, Alina Lozovskaya, Konrad Szafer, and Thomas Wolf. 2024. Open llm leaderboard v2. https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard

    2024

  3. [11]

    Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac'h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang S...

    2023 doi

  4. [12]

    Dirk Groeneveld, Iz Beltagy, Pete Walsh, Akshita Bhagia, Rodney Kinney, Oyvind Tafjord, Ananya Harsh Jha, Hamish Ivison, Ian Magnusson, Yizhong Wang, Shane Arora, David Atkinson, Russell Authur, Khyathi Raghavi Chandu, Arman Cohan, Jennifer Dumas, Yanai Elazar, Yuling Gu, Jack...

    2024 doi arXiv

  5. [13]

    Richter, Quentin Anthony, Eugene Belilovsky, Irina Rish, and Timoth \' e e Lesort

    Kshitij Gupta, Benjamin Th \' e rien, Adam Ibrahim, Mats L. Richter, Quentin Anthony, Eugene Belilovsky, Irina Rish, and Timoth \' e e Lesort. 2023. https://doi.org/10.48550/ARXIV.2308.04014 Continual pre-training of large language models: How to (re)warm your model? CoRR, abs...

    2023 doi arXiv

  6. [14]

    Christian Herold, Michael Kozielski, Leonid Ekimov, Pavel Petrushkov, Pierre - Yves Vandenbussche, and Shahram Khadivi. 2024. https://doi.org/10.48550/ARXIV.2406.12023 Lilium: ebay's large language models for e-commerce . CoRR, abs/2406.12023

    2024 doi arXiv

  7. [15]

    Richter, Quentin Gregory Anthony, Eugene Belilovsky, Timoth \' e e Lesort, and Irina Rish

    Adam Ibrahim, Benjamin Th \' e rien, Kshitij Gupta, Mats L. Richter, Quentin Gregory Anthony, Eugene Belilovsky, Timoth \' e e Lesort, and Irina Rish. 2024. https://openreview.net/forum?id=DimPeeCxKO Simple and scalable strategies to continually pre-train large language models...

    2024

  8. [16]

    Ramasesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman - Solo, Yuhuai Wu, Behnam Neyshabur, Guy Gur - Ari, and Vedant Misra

    Aitor Lewkowycz, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay V. Ramasesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman - Solo, Yuhuai Wu, Behnam Neyshabur, Guy Gur - Ari, and Vedant Misra. 2022. http://papers.nips.cc/paper\_files/paper/2022/has...

    2022

  9. [17]

    Yangning Li, Shirong Ma, Xiaobin Wang, Shen Huang, Chengyue Jiang, Haitao Zheng, Pengjun Xie, Fei Huang, and Yong Jiang. 2024. https://doi.org/10.1609/AAAI.V38I17.29820 Ecomgpt: Instruction-tuning large language models with chain-of-task tasks for e-commerce . In Thirty-Eighth...

    2024 doi

  10. [18]

    Anton Lozhkov, Loubna Ben Allal, Leandro von Werra, and Thomas Wolf. 2024. https://doi.org/10.57967/hf/2497 Fineweb-edu

    2024 doi

  11. [19]

    Deepak Narayanan, Mohammad Shoeybi, Jared Casper, Patrick LeGresley, Mostofa Patwary, Vijay Korthikanti, Dmitri Vainbrand, Prethvi Kashinkunti, Julie Bernauer, Bryan Catanzaro, Amar Phanishayee, and Matei Zaharia. 2021. https://doi.org/10.1145/3458817.3476209 Efficient large-s...

    2021

  12. [20]

    OpenAI. 2023. https://doi.org/10.48550/ARXIV.2303.08774 GPT-4 technical report . CoRR, abs/2303.08774

    2023 doi arXiv

  13. [21]

    Jupinder Parmar, Sanjeev Satheesh, Mostofa Patwary, Mohammad Shoeybi, and Bryan Catanzaro. 2024. https://doi.org/10.48550/ARXIV.2407.07263 Reuse, don't retrain: A recipe for continued pretraining of language models . CoRR, abs/2407.07263

    2024 doi arXiv

  14. [22]

    Guilherme Penedo, Hynek Kydlíček, Loubna Ben allal, Anton Lozhkov, Margaret Mitchell, Colin Raffel, Leandro Von Werra, and Thomas Wolf. 2024. https://arxiv.org/abs/2406.17557 The fineweb datasets: Decanting the web for the finest text data at scale . Preprint, arXiv:2406.17557

    2024 arXiv

  15. [23]

    Bo Peng, Xinyi Ling, Ziru Chen, Huan Sun, and Xia Ning. 2024. https://doi.org/10.48550/ARXIV.2402.08831 ecellm: Generalizing large language models for e-commerce from large-scale, high-quality instruction data . CoRR, abs/2402.08831

    2024 doi arXiv

  16. [24]

    David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R. Bowman. 2023. https://arxiv.org/abs/2311.12022 Gpqa: A graduate-level google-proof q&a benchmark . Preprint, arXiv:2311.12022

    2023 arXiv

  17. [25]

    Baptiste Rozi \` e re, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Tal Remez, J \' e r \' e my Rapin, Artyom Kozhevnikov, Ivan Evtimov, Joanna Bitton, Manish Bhatt, Cristian Canton - Ferrer, Aaron Grattafiori, Wenhan Xiong,...

    2023 doi arXiv

  18. [26]

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. 2024. https://doi.org/10.48550/ARXIV.2402.03300 Deepseekmath: Pushing the limits of mathematical reasoning in open language models . CoRR, abs/2402.03300

    2024 doi arXiv

  19. [27]

    Mohammad Shoeybi, Mostofa Patwary, Raul Puri, Patrick LeGresley, Jared Casper, and Bryan Catanzaro. 2019. https://arxiv.org/abs/1909.08053 Megatron-lm: Training multi-billion parameter language models using model parallelism . CoRR, abs/1909.08053

    2019 arXiv

  20. [28]

    Zayne Sprague, Xi Ye, Kaj Bostrom, Swarat Chaudhuri, and Greg Durrett. 2024. https://arxiv.org/abs/2310.16049 Musr: Testing the limits of chain-of-thought with multistep soft reasoning . Preprint, arXiv:2310.16049

    2024 arXiv

  21. [29]

    Le, Ed H

    Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc V. Le, Ed H. Chi, Denny Zhou, and Jason Wei. 2022. https://arxiv.org/abs/2210.09261 Challenging big-bench tasks and whether chain-of-thought can solve them . ...

    2022 arXiv

  22. [30]

    David Thulke, Yingbo Gao, Petrus Pelser, Rein Brune, Rricha Jalota, Floris Fok, Michael Ramos, Ian van Wyk, Abdallah Nasir, Hayden Goldstein, Taylor Tragemann, Katie Nguyen, Ariana Fowler, Andrew Stanco, Jon Gabriel, Jordan Taylor, Dean Moro, Evgenii Tsymbalov, Juliette de Waa...

    2024 doi arXiv

  23. [31]

    Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chandra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, Tianle Li, Max Ku, Kai Wang, Alex Zhuang, Rongqi Fan, Xiang Yue, and Wenhu Chen. 2024. https://arxiv.org/abs/2406.01574 Mmlu-pro: A more robust and...

    2024 arXiv

  24. [32]

    Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Kornblith, and Ludwig Schmidt

    Mitchell Wortsman, Gabriel Ilharco, Samir Yitzhak Gadre, Rebecca Roelofs, Raphael Gontijo Lopes, Ari S. Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Kornblith, and Ludwig Schmidt. 2022. https://proceedings.mlr.press/v162/wortsman22a.html Model soups: averaging we...

    2022

  25. [33]

    Qianqian Xie, Qingyu Chen, Aokun Chen, Cheng Peng, Yan Hu, Fongci Lin, Xueqing Peng, Jimin Huang, Jeffrey Zhang, Vipina Kuttichi Keloth, Xingyu Zhou, Huan He, Lucila Ohno - Machido, Yonghui Wu, Hua Xu, and Jiang Bian. 2024. https://doi.org/10.48550/ARXIV.2402.12749 Me llama: F...

    2024 doi arXiv

  26. [34]

    Dong Yuan, Eti Rastogi, Gautam Naik, Sree Prasanna Rajagopal, Sagar Goyal, Fen Zhao, Bharath Chintagunta, and Jeff Ward. 2024. https://doi.org/10.18653/V1/2024.NAACL-SHORT.47 A continued pretrained LLM approach for automatic medical note generation . In Proceedings of the 2024...

    2024 doi

  27. [35]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  28. [36]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.