Pith. sign in

REVIEW 4 major objections 5 minor 57 references

Improving Continual Pre-training Through Seamless Data Packing

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Seamless Packing, a two-stage data-packing scheme, improves continual pre-training across models and domains, outperforming baselines in 99% of settings.

desk verdict A practical, mostly-consistent packing win for continual pre-training, but the headline comparison isn't compute-matched and the '99% of settings' claim oversells it. read the letter →

arxiv 2505.22018 v2 pith:OORZIV4Y submitted 2025-05-28 cs.CL

Ruicheng Yin , Xuan Gao , Changze Lv , Xiaohua Wang , Xiaoqing Zheng , Xuanjing Huang This is my paper
classification cs.CL
keywords continualpre-trainingdatapackingslidingwindowoverlapcontextualcontinuityfirst-fit-decreasingbintokendroppinglanguagemodeltraining
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Seamless Packing is a way to arrange texts into the fixed-length sequences used when a language model is continually pre-trained on a new corpus. The paper claims that replacing standard concatenate-and-truncate packing with two stages—sliding-window overlap for long texts, then first-fit-decreasing packing with a little extra bin capacity and token dropping for short texts—preserves contextual continuity and reduces both truncation and padding. In experiments with four model families across news, finance, medical, mixed-domain, general-domain, and French-language corpora, the method reports lower validation perplexity and higher downstream accuracy than concatenation-truncation and best-fit-decreasing packing in 99% of settings, with a fourfold average improvement (0.96% versus 0.24%). If this is right, data packing is a cheap and model-agnostic lever for continual pre-training that can be combined with sampling or forgetting-mitigation methods.

What carries the argument

The machinery is a two-stage packing pipeline governed by two hyperparameters: $r_{\max}$, the maximum fraction of tokens allowed to repeat between adjacent windows, and $c_{\mathrm{extra}}$, the extra bin capacity above the target sequence length $L_{\mathrm{seq}}$. Stage one applies the sliding window when $L_{\mathrm{original}}+\lceil n r_{\max} L_{\mathrm{seq}}\rceil \ge (n+1)L_{\mathrm{seq}}$, setting the actual overlap to $\lceil((n+1)L_{\mathrm{seq}}-L_{\mathrm{original}})/n\rceil$ so that a long text fills a whole number of sequences with bounded duplication. Stage two sorts the leftover short chunks by length and packs them into bins of capacity $L_{\mathrm{seq}}+c_{\mathrm{extra}}$ using First-Fit-Decreasing, discarding tokens beyond $L_{\mathrm{seq}}$ instead of padding. The first stage trades bounded redundancy for context continuity; the second trades bounded truncation for less padding.

What would settle it

Train the same model and corpus with Seamless Packing at $r_{\max}=0.3$ and with a control that inserts the same total number of repeated tokens at random positions instead of aligned overlapping windows; if the control matches Seamless Packing's perplexity and downstream accuracy, the claimed continuity mechanism is not what drives the gain.

Watch

Extended reading notes

Core claim

The paper's central claim is that contextual continuity is a trainable resource being wasted by naive packing, and that a packing scheme can recover it directly. For long texts, Seamless Packing uses a sliding window that repeats a controlled number of tokens ($L_{\max\_\mathrm{overlap}}=\lceil n r_{\max} L_{\mathrm{seq}}\rceil$) so the text fills $n+1$ complete sequences instead of leaving a fragment; for shorter texts, it packs chunks into bins of size $L_{\mathrm{seq}}+c_{\mathrm{extra}}$ with First-Fit-Decreasing and drops tokens that overflow $L_{\mathrm{seq}}$. The paper reports consistently lower validation perplexity and better downstream task accuracy than concatenation-truncation and Best-Fit-Decreasing baselines across GPT-2, Llama-3.2, Qwen2.5, and Gemma-2, in domain-specific, mixed-domain, general-domain, and cross-lingual settings, alongside a case study in which the overlap-trained model recovers an injected event's date and location while the baseline hallucinates.

Load-bearing premise

The load-bearing premise is that duplicating up to 30% of tokens across neighboring windows helps learning more than the duplication hurts it; if repeated text degrades training the way earlier duplication studies suggest, the sliding-window stage's main effect backfires.

Editorial extensions

If this is right

  • Models continually pre-trained with Seamless Packing should show lower validation perplexity than concatenation-truncation or Best-Fit-Decreasing packing on the same corpus.
  • Downstream classification and generation tasks should improve most when texts are long and truncation would otherwise split a single fact, event, or code dependency.
  • Because Seamless Packing only rearranges data, it can be layered on top of domain data selection, sampling ratios, and catastrophic-forgetting methods without changing the training loop.
  • First-Fit-Decreasing needs about 29% less packing time than Best-Fit-Decreasing on these datasets, so the method is also cheaper to run at the data-preprocessing stage.
  • The gains appear to transfer to mixed-domain, general-domain (RedPajama), and French cross-lingual continual pre-training, though the paper does not test standard pre-training from scratch.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If duplication is the true driver, the optimal $r_{\max}$ should shrink as the corpus already contains near-duplicate text; a testable extension is to measure Seamless Packing's gain on deduplicated versus raw corpora.
  • The mechanism is not limited to continual pre-training: applying the same overlap-and-drop packing to the initial pre-training phase, with a fixed token budget, would test whether continuity helps in that regime too.
  • Because stage two drops tokens that overflow $L_{\mathrm{seq}}$, rare tokens near chunk boundaries may be lost more often than common ones; checking token-level recall between the original and packed corpora would reveal any such bias.
  • The overlap stage could be combined with document-reordering methods that put related texts nearby, creating both inter-document and intra-document continuity in one packing pass.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. Seamless Packing (SP) is a two-stage data-packing method for continual pre-training. The first stage applies a sliding window with an rmax-controlled token overlap to long texts, expanding them to integer numbers of sequences; the second stage packs the remaining short chunks with First-Fit-Decreasing into bins of capacity Lseq+cextra and drops tokens that exceed Lseq. The authors compare SP against concatenation-truncation (CT) and Best-Fit-Decreasing (BFD) on GPT-2, Llama-3.2, Qwen2.5, and Gemma-2 across news, finance, medical, mixed, general, and French corpora, reporting validation perplexity and downstream task accuracy. They also present a theoretical analysis of the fraction of texts affected by the sliding window, ablations over rmax and cextra, and a small case study on hallucination.

Significance. SP is a practical and simple preprocessing idea that can be plugged into existing continual-pretraining pipelines. The paper's strengths are the breadth of the evaluation (multiple architectures, domains, and tuning regimes), the component-wise ablations (BFD-m, FFD, hyperparameter sweeps), the release of code, and the attempt to quantify how much data each stage affects. If the performance advantage survives a matched-compute comparison, the method would be a useful contribution to the growing literature on data engineering for language-model training. However, the central comparison is currently confounded by token budget, and the advertised theoretical and '99%' claims are not supported as stated.

major comments (4)
  1. [Appendix D.4 and Section 3.2] The comparison is not token-matched or step-matched. Runs are fixed at 3 epochs (Appendix D.4), but SP deliberately adds tokens: for texts satisfying condition (2), Eq. (3) expands n full sequences to n+1 sequences, duplicating Lfinal_overlap tokens per transition. CT and BFD do not duplicate. Thus SP trains on more tokens per epoch, hence more optimizer steps at fixed epochs. Table 2's lower perplexity and Table 1's higher accuracy may be due to extra compute rather than to contextual continuity. Please report per-method token counts, step counts, and wall-clock training time, and include at least one matched-budget experiment (equal tokens or equal steps, or equivalently fewer epochs for SP).
  2. [Section 3.4, Eqs. (6)-(7)] The sums in Eqs. (6) and (7) run to ceil(1/r) in the first sum and from ceil(1/r) in the second. For k = ceil(1/r), k r >= 1, so the T_k term is double-counted in Eq. (6) and the factor (1 - k r) in Eq. (7) is non-positive. The correct partition is k < 1/r for the first sum and k >= ceil(1/r) for the second (or k >= floor(1/r)+1). Because Eq. (6) directly generates the predicted Nsw=6716 in Section 4.8, the theoretical comparison to the observed 6988 should be recomputed. This is a mathematical error in an advertised contribution, not merely a typo, although the direction of the qualitative conclusion (Nsw increases with r) is unaffected.
  3. [Abstract, Section 4.2, Tables 1 and 3] The abstract's '99% of all settings' is not supported by the tables. In Table 1, Llama-3.2 Fin Topic has SP 84.58 < BFD 84.62, and Llama-3.2 ChemProt ties at 80.81. In Table 3, Qwen2.5-3B BBC News has SP 97.50 below both CT 97.54 and BFD 97.56, and Gemma-2 perplexity has SP 11.02 worse than BFD 10.97. Counting all comparisons against both baselines across the tables gives a lower fraction than 99%. Please define exactly which settings are counted and correct the abstract or the numbers accordingly.
  4. [Appendix E and Section 4.2] Statistical support is partial. Significance tests are reported only for Qwen2.5-1.5B, with several p-values above 0.1 (e.g., CT vs SP on 20 Newsgroup p=0.3154; BFD vs SP on BBC News p=0.2563 and Fin Sentiment p=0.7552), and Wilcoxon p-values cannot go below 0.0625 with five runs. Main tables (Tables 1-5) report no standard deviations or confidence intervals. Please include per-seed variation and, if possible, significance or effect-size analysis for the other models; otherwise the cross-architecture generalization claim is only impressionistic.
minor comments (5)
  1. [Section 4.6] The text says 'As illustrated in Figure 2a' for cextra, but the cextra panel is Figure 2b; please correct the cross-reference.
  2. [Section 4.5, Table 7] The comparison BFD-m vs SP mixes the sliding-window first stage with the FFD-vs-BFD choice in the second stage; since FFD and BFD perform similarly in Table 7 the confound is probably small, but an SP variant using BFD in stage 2 would isolate the first-stage contribution cleanly.
  3. [Section 4.1] The phrase 'The universal baselines consist of three configurations' is unclear; 'standard' or 'main' baselines is likely intended.
  4. [Section 4.7 and Appendix H] The case study is based on one synthetic event and five completions per model; it is illustrative rather than quantitative, and should be labeled as such in the main text.
  5. [Section 4.8] The statement that 'total dropping tokens in SP has small difference with total padding tokens in BFD' is supported only for PubMed; the BBC numbers (7K vs 140K) show a large relative difference, so the sentence should separate the two regimes explicitly.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the central claim is an empirical packing comparison, and the theoretical counts are derived from the algorithm's own definitions rather than from the target performance results.

full rationale

The paper's central claim is that Seamless Packing improves continual pre-training relative to concatenation-truncation (CT) and best-fit-decreasing (BFD) packing. This is supported by direct empirical measurements: validation perplexity in Table 2, downstream accuracies in Tables 1 and 3, and generalization results in Tables 4-6. None of these numbers are derived from the method's definitions; they are experimental outcomes, so the improvement claim is not circular. The 'theoretical analysis' in Section 3.4 computes Nsw and Ntoken_short from the algorithm's own criteria (Eqs. 1-6) and dataset length histograms; these are bookkeeping identities describing how many texts fall into each stage, explicitly cross-checked against empirical counts in Section 4.8 (6716 predicted vs. 6988 observed). This is a consistency check of a counting formula, not a prediction of model performance, so it does not constitute circularity. Hyperparameters rmax and cextra are tuned empirically in Section 4.6 on the same benchmark families later reported, which is a standard but potentially optimistic tuning practice; it is not a structural circularity because the tuning choice does not by construction force the reported accuracies or perplexities. There are no load-bearing self-citations, no imported uniqueness theorems, and no ansatz smuggled in via citation; the method's components (sliding-window overlap and FFD packing) are original combinations of published techniques and are ablated in Section 4.5. The manuscript's Limitations section explicitly acknowledges that a full theoretical framework for the dropping/padding trade-off remains undeveloped and that generalization to code and standard pre-training is unexplored, which further supports that the paper is not presenting derived results as if they were forced. The reviewer concern about unequal token/step budgets across the 3-epoch comparison is an experimental-design risk about whether the gains come from more compute, not a circularity in the derivation chain; accordingly it does not raise the circularity score.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

Seamless Packing relies on two tuned hyperparameters, a uniformity assumption for the theoretical counts, and two domain assumptions about overlap and dropping being beneficial. No new entities are introduced.

free parameters (2)
  • rmax = 0.3 (default; sweep 0.1, 0.3, 0.5)
    Maximum repetition ratio; controls how many long texts qualify for sliding window and how much token duplication is injected. Chosen in Section 4.6 by downstream accuracy, not derived.
  • cextra = 50 with Lseq=2048; 10 with Lseq=512
    Extra bin capacity beyond Lseq that permits token dropping instead of padding; tuned in Section 4.6 and scaled across datasets.
assumptions (4)
  • domain assumption Text lengths are approximately uniformly distributed within each (kLseq, (k+1)Lseq] interval.
    Used to derive Eq. (6)-(7) in Section 3.4; the paper acknowledges real distributions are not perfectly uniform, which explains the gap between theoretical and observed counts.
  • domain assumption Overlapping tokens across consecutive sequences improve or at least do not harm continual pre-training when overlap is bounded by rmax.
    Underpins the entire first stage; the paper relies on prior duplication-warning work and its own hyperparameter sweep rather than a proof.
  • domain assumption Dropping overflow tokens from bins is preferable to padding them.
    Central to the second stage; the paper provides empirical comparisons to BFD, which pads, but no controlled FFD-padding versus FFD-dropping ablation.
  • standard math Standard bin-packing heuristics (FFD and BFD) achieve near-optimal packing for the observed text length distributions.
    Assumed when claiming FFD is sufficient; no optimality bound tailored to this data is proven.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Improving Continual Pre-training Through Seamless Data Packing." pith.science (2026). https://pith.science/paper/OORZIV4Y

@misc{pith2026250522018,
  author       = {Pith},
  title        = {Pith review of: Improving Continual Pre-training Through Seamless Data Packing},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/OORZIV4Y}},
  note         = {Machine review of arXiv:2505.22018}
}
read the original abstract

Continual pre-training has demonstrated significant potential in enhancing model performance, particularly in domain-specific scenarios. The most common approach for packing data before continual pre-training involves concatenating input texts and splitting them into fixed-length sequences. While straightforward and efficient, this method often leads to excessive truncation and context discontinuity, which can hinder model performance. To address these issues, we explore the potential of data engineering to enhance continual pre-training, particularly its impact on model performance and efficiency. We propose Seamless Packing (SP), a novel data packing strategy aimed at preserving contextual information more effectively and enhancing model performance. Our approach employs a sliding window technique in the first stage that synchronizes overlapping tokens across consecutive sequences, ensuring better continuity and contextual coherence. In the second stage, we adopt a First-Fit-Decreasing algorithm to pack shorter texts into bins slightly larger than the target sequence length, thereby minimizing padding and truncation. Empirical evaluations across various model architectures and corpus domains demonstrate the effectiveness of our method, outperforming baseline method in 99% of all settings. Code is available at https://github.com/Infernus-WIND/Seamless-Packing.

Figures

Figures reproduced from arXiv: 2505.22018 by the authors.

Figure 1
Figure 1. An illustration of the proposed Seamless Packing method. [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Influence of rmax and cextra on model perfor￾mance. Figure 2a illustrates the effect of varying rmax while keeping cextra fixed at 50. Figure 2b shows the effect of varying cextra while keeping rmax fixed at 0.3. Method Date Accuracy Location Accuracy BFD 0/5 2/5 SP 3/5 5/5 [PITH_FULL_IMAGE:figures/full_fig_p009_2.png] view at source ↗
Figure 3
Figure 3. Effect of rmax on Text Count and Token Count. Text Count: The amount of texts that are able to use sliding window in stage one. Token Count: the total amount of tokens in short chunks in stage two. Dataset FFD (s) BFD (s) BBC News 2.24 2.63 Financial Article 2.83 3.61 PubMed Article 1.26 1.78 [PITH_FULL_IMAGE:figures/full_fig_p018_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Illustration of text length distribution of the [PITH_FULL_IMAGE:figures/full_fig_p018_4.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

57 extracted references · 25 canonical work pages

  1. [1]

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774

    arXiv 2023

  2. [2]

    Anton Alexandrov, Veselin Raychev, Mark Niklas Mueller, Ce Zhang, Martin Vechev, and Kristina Toutanova. 2024. https://doi.org/10.18653/v1/2024.findings-emnlp.1000 Mitigating catastrophic forgetting in language transfer via model merging . In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 17167--17186, Miami, Florida, USA. As...

    doi:10.18653/v1/2024.findings-emnlp.1000 2024

  3. [3]

    Miltiadis Allamanis. 2019. https://doi.org/10.1145/3359591.3359735 The adverse effects of code duplication in machine learning models of code . In Proceedings of the 2019 ACM SIGPLAN International Symposium on New Ideas, New Paradigms, and Reflections on Programming and Software, Onward! 2019, page 143–153, New York, NY, USA. Association for Computing Machinery

    work page arXiv 2019

  4. [4]

    Costa-jussà

    Magdalena Biesialska, Katarzyna Biesialska, and Marta R. Costa-jussà. 2020. https://doi.org/10.18653/v1/2020.coling-main.574 Continual lifelong learning in natural language processing: A survey . In Proceedings of the 28th International Conference on Computational Linguistics

    doi:10.18653/v1/2020.coling-main.574 2020

  5. [5]

    Boyer and J

    Robert S. Boyer and J. Strother Moore. 1977. https://doi.org/10.1145/359842.359859 A fast string searching algorithm . Commun. ACM, 20(10):762–772

    work page arXiv 1977

  6. [6]

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gr...

    2020

  7. [7]

    Jie Chen, Zhipeng Chen, Jiapeng Wang, Kun Zhou, Yutao Zhu, Jinhao Jiang, Yingqian Min, Wayne Xin Zhao, Zhicheng Dou, Jiaxin Mao, et al. 2024. Towards effective and efficient continual pre-training of large language models. arXiv preprint arXiv:2407.18743

    arXiv 2024

  8. [8]

    Arman Cohan, Franck Dernoncourt, Doo Soon Kim, Trung Bui, Seokhwan Kim, Walter Chang, and Nazli Goharian. 2018. https://doi.org/10.18653/v1/N18-2097 A discourse-aware attention model for abstractive summarization of long documents . In Proceedings of the 2018 Conference of the North A merican Chapter of the Association for Computational Linguistics: Human...

    doi:10.18653/v1/n18-2097 2018

Show all 57 references
  1. [9]

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. https://doi.org/10.18653/v1/N19-1423 BERT : Pre-training of deep bidirectional transformers for language understanding . In Proceedings of the 2019 Conference of the North A merican Chapter of the Associat...

    2019 doi

  2. [10]

    Hantian Ding, Zijian Wang, Giovanni Paolini, Varun Kumar, Anoop Deoras, Dan Roth, and Stefano Soatto. 2024. https://openreview.net/forum?id=kRxCDDFNpp Fewer truncations improve language modeling . In Forty-first International Conference on Machine Learning

    2024

  3. [11]

    Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783

    2024 arXiv

  4. [12]

    Samuel Eilon and Nicos Christofides. 1971. The loading problem. Management Science, 17(5):259--268

    1971

  5. [13]

    Robert M French. 1999. Catastrophic forgetting in connectionist networks. Trends in cognitive sciences, 3(4):128--135

    1999

  6. [14]

    Derek Greene and P\' a draig Cunningham. 2006. Practical solutions to the problem of diagonal dominance in kernel document clustering. In Proc. 23rd International Conference on Machine learning (ICML'06), pages 377--384. ACM Press

    2006

  7. [15]

    Antonio Gulli. 2005. Ag's corpus of news articles. http://groups.di.unipi.it/ gulli/AG_corpus_of_news_articles.html. Accessed: 15 December 2024

    2005

  8. [16]

    Kshitij Gupta, Benjamin Th \'e rien, Adam Ibrahim, Mats Leon Richter, Quentin Gregory Anthony, Eugene Belilovsky, Irina Rish, and Timoth \'e e Lesort. 2023. https://openreview.net/forum?id=pg7PUJe0Tl Continual pre-training of large language models: How to re-warm your model? I...

    2023

  9. [17]

    Suchin Gururangan, Ana Marasovi \'c , Swabha Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, and Noah A. Smith. 2020. https://doi.org/10.18653/v1/2020.acl-main.740 Don`t stop pretraining: Adapt language models to domains and tasks . In Proceedings of the 58th Annual Meeting of ...

    2020 doi

  10. [18]

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022. https://openreview.net/forum?id=nZeVKeeFYf9 Lo RA : Low-rank adaptation of large language models . In International Conference on Learning Representations

    2022

  11. [19]

    Yongxin Huang, Kexin Wang, Sourav Dutta, Raj Patel, Goran Glava s , and Iryna Gurevych. 2023. https://doi.org/10.18653/v1/2023.emnlp-main.208 A da S ent: Efficient domain-adapted sentence embeddings for few-shot classification . In Proceedings of the 2023 Conference on Empiric...

    2023 doi

  12. [20]

    Adam Ibrahim, Benjamin Th \'e rien, Kshitij Gupta, Mats Leon Richter, Quentin Gregory Anthony, Eugene Belilovsky, Timoth \'e e Lesort, and Irina Rish. 2024. https://openreview.net/forum?id=DimPeeCxKO Simple and scalable strategies to continually pre-train large language models...

    2024

  13. [21]

    Joel Jang, Seonghyeon Ye, Sohee Yang, Joongbo Shin, Janghoon Han, Gyeonghun KIM, Stanley Jungkyu Choi, and Minjoon Seo. 2022. https://openreview.net/forum?id=vfsRB5MImo9 Towards continual knowledge learning of language models . In International Conference on Learning Representations

    2022

  14. [22]

    Jeet.J. 2018. Us financial news articles. https://www.kaggle.com/datasets/jeet2016/us-financial-news-articles/data. Accessed: 10 December 2024

    2018

  15. [23]

    Fred Jelinek, Robert L Mercer, Lalit R Bahl, and James K Baker. 1977. Perplexity—a measure of the difficulty of speech recognition tasks. The Journal of the Acoustical Society of America, 62(S1):S63--S63

    1977

  16. [24]

    David S Johnson. 1973. Near-optimal bin packing algorithms. Ph.D. thesis, Massachusetts Institute of Technology

    1973

  17. [25]

    Richard M Karp and Michael O Rabin. 1987. Efficient randomized pattern-matching algorithms. IBM journal of research and development, 31(2):249--260

    1987

  18. [26]

    Zixuan Ke, Yijia Shao, Haowei Lin, Tatsuya Konishi, Gyuhak Kim, and Bing Liu. 2023. https://openreview.net/forum?id=m_GDIItaI3o Continual pre-training of language models . In The Eleventh International Conference on Learning Representations

    2023

  19. [27]

    C.K. Koç. 1995. https://doi.org/10.1016/0898-1221(95)00153-P Analysis of sliding window techniques for exponentiation . Computers & Mathematics with Applications, 30(10):17--24

    1995 doi

  20. [28]

    Krallinger, O

    M. Krallinger, O. Rabal, and A. Lourenço. 2017. https://biocreative.bioinformatics.udel.edu/tasks/biocreative-vi/track-5/ Overview of the biocreative vi chemical-protein interaction track . Proceedings of the BioCreative VI Workshop,, 141-146

    2017

  21. [29]

    Mario Michael Krell, Matej Kosec, Sergio P Perez, and Andrew Fitzgibbon. 2021. Efficient sequence packing without cross-contamination: Accelerating large language models without impacting performance. arXiv preprint arXiv:2107.02027

    2021 arXiv

  22. [30]

    Ken Lang. 1995. https://doi.org/10.1016/B978-1-55860-377-6.50048-7 Newsweeder: Learning to filter netnews . In Armand Prieditis and Stuart Russell, editors, Machine Learning Proceedings 1995, pages 331--339. Morgan Kaufmann, San Francisco (CA)

    1995 doi

  23. [31]

    Hang Le, Loïc Vial, Jibril Frej, Vincent Segonne, Maximin Coavoux, Benjamin Lecouteux, Alexandre Allauzen, Benoît Crabbé, Laurent Besacier, and Didier Schwab. 2019. https://arxiv.org/abs/1912.05372 Flaubert: Unsupervised language model pre-training for french . Preprint, arXiv...

    2019 arXiv

  24. [32]

    Katherine Lee, Daphne Ippolito, Andrew Nystrom, Chiyuan Zhang, Douglas Eck, Chris Callison-Burch, and Nicholas Carlini. 2022. https://doi.org/10.18653/v1/2022.acl-long.577 Deduplicating training data makes language models better . In Proceedings of the 60th Annual Meeting of t...

    2022 doi

  25. [33]

    Timoth \'e e Lesort, Massimo Caccia, and Irina Rish. 2021. Understanding continual learning settings with data distribution drift analysis. arXiv preprint arXiv:2104.01678

    2021 arXiv

  26. [34]

    Yucheng Li, Frank Guerin, and Chenghua Lin. 2024. Latesteval: Addressing data contamination in language model evaluation through dynamic and time-sensitive test construction. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 18600--18607

    2024

  27. [35]

    Yong Lin, Hangyu Lin, Wei Xiong, Shizhe Diao, Jianmeng Liu, Jipeng Zhang, Rui Pan, Haoxiang Wang, Wenbin Hu, Hanning Zhang, Hanze Dong, Renjie Pi, Han Zhao, Nan Jiang, Heng Ji, Yuan Yao, and Tong Zhang. 2024. https://doi.org/10.18653/v1/2024.emnlp-main.35 Mitigating the alignm...

    2024 doi

  28. [36]

    Ilya Loshchilov and Frank Hutter. 2019. https://openreview.net/forum?id=Bkg6RiCqY7 Decoupled weight decay regularization . In International Conference on Learning Representations

    2019

  29. [37]

    P. Malo, A. Sinha, P. Korhonen, J. Wallenius, and P. Takala. 2014. Good debt or bad debt: Detecting semantic orientations in economic texts. Journal of the Association for Information Science and Technology, 65

    2014

  30. [38]

    Silvano Martello and Paolo Toth. 1990. https://doi.org/10.1016/0166-218X(90)90094-S Lower bounds and reduction procedures for the bin packing problem . Discrete Applied Mathematics, 28(1):59--70

    1990 doi

  31. [39]

    Sanket Vaibhav Mehta, Darshan Patil, Sarath Chandar, and Emma Strubell. 2023. An empirical investigation of the role of pre-training in lifelong learning. J. Mach. Learn. Res., 24(1)

    2023

  32. [40]

    Alessio Miaschi, Dominique Brunato, Felice Dell ' Orletta, and Giulia Venturi. 2021. https://doi.org/10.18653/v1/2021.deelio-1.5 What makes my model perplexed? a linguistic investigation on neural language models perplexity . In Proceedings of Deep Learning Inside Out (DeeLIO)...

    2021 doi

  33. [41]

    Jupinder Parmar, Sanjev Satheesh, Mostofa Patwary, Mohammad Shoeybi, and Bryan Catanzaro. 2024. Reuse, don't retrain: A recipe for continued pretraining of language models. arXiv preprint arXiv:2407.07263

    2024 arXiv

  34. [42]

    Alec Radford and Karthik Narasimhan. 2018. https://api.semanticscholar.org/CorpusID:49313245 Improving language understanding by generative pre-training

    2018

  35. [43]

    Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners

    2019

  36. [44]

    Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21(140):1--67

    2020

  37. [45]

    Anthony V. Robins. 1995. https://api.semanticscholar.org/CorpusID:22882861 Catastrophic forgetting, rehearsal and pseudorehearsal . Connect. Sci., 7:123--146

    1995

  38. [46]

    Smith, Luke Zettlemoyer, Wen tau Yih, and Mike Lewis

    Weijia Shi, Sewon Min, Maria Lomeli, Chunting Zhou, Margaret Li, Xi Victoria Lin, Noah A. Smith, Luke Zettlemoyer, Wen tau Yih, and Mike Lewis. 2024. https://openreview.net/forum?id=LXVswInHOo In-context pretraining: Language modeling beyond document boundaries . In The Twelft...

    2024

  39. [47]

    Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, L \'e onard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ram \'e , et al. 2024. Gemma 2: Improving open language models at a practical size. arXiv preprint arXiv:2408.00118

    2024 arXiv

  40. [48]

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, W...

    2023 arXiv

  41. [49]

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, ukasz Kaiser, and Illia Polosukhin. 2017. https://proceedings.neurips.cc/paper_files/paper/2017/file/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf Attention is all you need . In Advances in Ne...

    2017

  42. [50]

    Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2018. https://doi.org/10.18653/v1/W18-5446 GLUE : A multi-task benchmark and analysis platform for natural language understanding . In Proceedings of the 2018 EMNLP Workshop B lackbox NLP : A...

    2018 doi

  43. [51]

    Maurice Weber, Daniel Y. Fu, Quentin Anthony, Yonatan Oren, Shane Adams, Anton Alexandrov, Xiaozhong Lyu, Huu Nguyen, Xiaozhe Yao, Virginia Adams, Ben Athiwaratkun, Rahul Chalamala, Kezhen Chen, Max Ryabinin, Tri Dao, Percy Liang, Christopher Ré, Irina Rish, and Ce Zhang. 2024...

    2024

  44. [52]

    Genta Winata, Lingjue Xie, Karthik Radhakrishnan, Shijie Wu, Xisen Jin, Pengxiang Cheng, Mayank Kulkarni, and Daniel Preotiuc-Pietro. 2023. https://doi.org/10.18653/v1/2023.findings-acl.48 Overcoming catastrophic forgetting in massively multilingual continual learning . In Fin...

    2023 doi

  45. [53]

    An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, Guanting Dong, Haoran Wei, Huan Lin, Jialong Tang, Jialin Wang, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Ma, Jianxin Yang, Jin Xu, Jingren Zhou, Jinze...

    2024 arXiv

  46. [54]

    An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le Yu, Me...

    2024 arXiv

  47. [55]

    C a g atay Y ld z, Nishaanth Kanna Ravichandran, Prishruit Punia, Matthias Bethge, and Beyza Ermis. 2024. Investigating continual pretraining in large language models: Insights and implications. arXiv preprint arXiv:2402.17400

    2024 arXiv

  48. [56]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  49. [57]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.