Pith. sign in

REVIEW 4 major objections 3 minor 2 cited by

Quantized language models forget less than full-precision ones in continual learning, with 8-bit precision striking the best balance between retaining old tasks and learning new ones.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-03 14:49 UTC pith:CXV3YNSL

load-bearing objection Single-seed runs and inconsistent numbers undercut an otherwise novel precision-vs-replay result; the mechanism claim is not backed by the setup. the 4 major comments →

arxiv 2512.18934 v2 pith:CXV3YNSL submitted 2025-12-22 cs.LG cs.AI

When Less is More: 8-bit Quantization Improves Continual Learning in Large Language Models

Michael S. Zhang , Rishi A. Ruia , Arnav Kewalram , Saathvik Dharmapuram , Utkarsh Sharma , Kevin Zhu This is my paper
classification cs.LG cs.AI
keywords continual learningcatastrophic forgettingquantization8-bit precisionreplay bufferlow-rank adaptationimplicit regularizationlarge language models
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper asks whether lower numerical precision can actually help continual learning, not just save memory. Across a three-stage sequence of NLU, Math, and Code training on an 8-billion-parameter language model, the authors find that FP16 wins on the first task, but quantized models outperform it by 8–15% on later-task accuracy, with INT4 nearly doubling FP16's Code-generation score. INT8 consistently offers the best trade-off between learning new tasks and retaining old ones, and even a 0.1% replay buffer dramatically lifts retention across all precisions. The paper challenges the default assumption that higher precision is always better, and suggests quantization noise may act as a form of implicit regularization.

Core claim

In a three-stage continual learning setup—train on NLU, then Math with replay of NLU, then Code with replay of both—the authors observe a precision inversion. Full-precision FP16 achieves the highest initial NLU accuracy (74.44%), but after subsequent tasks quantized models beat FP16 by 8–15% on final-task forward accuracy, with INT4 reaching about twice FP16's Code pass@1 (40% vs 20%). Across replay buffer sizes from 0% to 20%, INT8 gives the best balance of plasticity and retention. A replay buffer as small as 0.1% raises NLU retention after Math training from 45% to 65% at every precision level, and quantized models generally require less replay than FP16 to reach comparable retention. Th

What carries the argument

The experimental engine is a three-stage continual-learning schedule—NLU, then Math with replayed NLU data, then Code with replayed NLU and Math data—run at three precisions (FP16, INT8, INT4) using frozen base weights and trained low-rank adapters. The paper's proposed mechanism is quantization-induced noise as implicit regularization: discretization error is said to smooth the loss landscape and bias the model toward flatter minima, which amplifies the influence of replayed samples and helps anchor prior knowledge. Replay buffer size is the second lever, interleaving up to 20% of prior-task data into later training stages.

Load-bearing premise

The central claim relies on quantization noise actively regularizing learning, but since the quantized base weights are frozen and the adapters are trained in high precision, the noise is static and never directly perturbs the adapter updates—so the benefit could instead come from altered forward-pass scales or optimization interactions, and if so, the precision ordering may not transfer to other setups.

What would settle it

Run the same continual-learning protocol in FP16 while injecting noise into the adapter gradients or forward activations with the same magnitude as INT8's quantization error; if noise-injected FP16 reproduces the 8–15% retention advantage, the implicit-regularization story is supported, whereas if it does not, the effect is driven by quantization's changed scales or optimization dynamics, not by noise. Repeating across multiple seeds would also confirm whether the INT8-over-FP16 ordering is stable.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • INT8 quantization should be considered a default for continual fine-tuning of large language models, since it combines lower memory cost with better retention and comparable learning of new tasks.
  • Small replay buffers (1–2%) are sufficient for NLU retention across precisions, while Math and Code benefit from moderate buffers (5–10%).
  • Quantized models need less replay data than FP16 to achieve comparable retention, reducing storage overhead in deployed continual-learning systems.
  • The precision ordering observed (INT8 over FP16 and INT4 on later tasks) implies that quantization choices interact with learning dynamics in ways static benchmarks miss.
  • If the implicit-regularization hypothesis is correct, precision becomes a tunable knob for balancing plasticity and stability in sequential training.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • A direct way to test the proposed mechanism is to inject controlled noise into FP16 adapter gradients or forward activations, matched in magnitude to INT8's quantization error; if that reproduces the retention advantage, the regularization story is supported, whereas a failure would point to changed scales or optimization dynamics as the true cause.
  • The effect may be specific to frozen quantized base weights with high-precision adapters; adaptive or mixed-precision schemes might produce different orderings, so the practical recommendation should be validated across adapter and quantization configurations.
  • The same lens could be applied to other continual-learning scenarios, such as vision or multimodal models, where quantization noise might interact differently with replay and task order.
  • If quantization noise really biases toward flatter minima, then explicit flatness-promoting regularizers (e.g., sharpness-aware updates) might confer similar benefits in full precision, offering an alternative to quantization for stability.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 3 minor

Summary. The paper studies the interaction between weight quantization precision (FP16, INT8, INT4) and replay-buffer size in continual learning of a LLaMA-3.1-8B model. Using LoRA adapters on frozen base weights, the authors fine-tune sequentially on NLU, Math (GSM8K), and Code (CodeAlpaca/HumanEval), varying the replay buffer from 0% to 20%. They report that, contrary to the usual assumption that higher precision is better, quantized models outperform FP16 on final-task forward accuracy by 8–15%, with INT4 achieving roughly double FP16's HumanEval pass@1 (40% vs 20%), and INT8 offering the best plasticity–retention balance. They attribute the effect to quantization-induced noise acting as implicit regularization and give practical buffer-size recommendations. The paper includes a public code repository and proposes a benchmark for quantized continual learning.

Significance. If the empirical inversion is correct, the paper identifies a genuinely useful and counterintuitive design choice: low-precision weights could be not only efficient but also beneficial for continual learning, and small replay buffers might suffice under quantization. The systematic sweep over precision × buffer size is a useful contribution, and the public code link strengthens reproducibility. However, the central quantitative claims are currently unverified: all cells are single-seed runs, the abstract and Section 3 report conflicting numbers, and the proposed mechanistic explanation is not tested against simpler alternatives. The paper's practical recommendations therefore rest on an empirical foundation that the text itself acknowledges is incomplete.

major comments (4)
  1. [Section 5, Limitations; Abstract; Section 3] The central 8–15% inversion and the 40% vs 20% Code gap rest on one training run per precision–buffer condition. Section 5 admits 'our experiments lack multi-seed runs and confidence intervals.' Since HumanEval pass@1 has only 164 problems, a 20-point gap is roughly 33 samples, and initialization/order noise can be several points, the reported ordering cannot be distinguished from seed noise. This is the load-bearing claim of the paper; multi-seed runs with intervals (or at least repeated runs for the headline cells) are required.
  2. [Section 3, 'Baseline degradation' and 'Amplification under continual learning'] Reported numbers are internally inconsistent. The abstract states FP16 achieves 74.44% on initial NLU, but Section 3 says FP16 drops 'from 77.26% to 42.50%'. The same paragraph says '4-bit models exhibited sharper degradation,' yet the numbers show FP16 losing 34.76 points (77.26→42.50) versus 4-bit losing 29.81 points (72.31→42.50). These contradictions prevent the reader from determining even the direction of the precision effect, let alone its magnitude.
  3. [Section 1 vs Abstract/Section 3] The introduction states that 'at higher precision, minimal replay is sufficient, while under 4-bit quantization, buffer size becomes a decisive factor' and that '8-bit quantization noise acts as a natural regularizer.' The abstract and Section 3, by contrast, claim FP16 is markedly worse under replay and that INT8 is the best balance. These are substantively different claims: one implies FP16 needs little replay, the other implies FP16 underperforms even with replay. The narrative needs to be reconciled and the precise claim stated consistently.
  4. [Section 2; Section 3, 'Interpretation'] The proposed mechanism — 'quantization-induced noise acts as implicit regularization' — is not supported by the experimental setup. With frozen quantized base weights and high-precision LoRA adapters, the quantization noise is static and does not enter the adapter gradient updates; it merely changes the forward activations. The observed effects could instead come from altered scales, activation ranges, or LoRA initialization/optimization interactions. The authors correctly label this as a hypothesis ('We speculate'), but the paper's abstract and guidelines lean on it as if established. I recommend either removing the mechanistic claim or testing it directly, e.g., by adding equivalent static noise to FP16 forward passes.
minor comments (3)
  1. [References] Several citations are misassigned: [4] is used for both CodeAlpaca and A-GEM; [11] is cited for the eight NLU datasets but is a GFlowNet paper; [16] is cited for AWQ but is a self-attention paper. Please verify all references.
  2. [Figures 1 and 2] The figure captions appear duplicated ('Forward Accuracy. Forward Accuracy.') and the axes are not described in the caption. Please clarify what is plotted and label the axes or describe them in the caption.
  3. [Throughout] Typo: 'Low-Ranking Adaptation' should be 'Low-Rank Adaptation.' Also, 'with out the mentorship' in the acknowledgments should be 'without the mentorship.'

Circularity Check

0 steps flagged

No circularity found: the measured accuracy outcomes are produced by a fixed evaluation protocol, and the implicit-regularization hypothesis is explicitly post-hoc rather than an input to the measurements.

full rationale

The paper's central claims are empirical observations generated by a defined experimental protocol, not by a derivation chain. Section 2 fixes the model (LLaMA-3.1-8B), adapter setup (LoRA rank 8, alpha 16, dropout 0), precisions (FP16/8-bit/4-bit), and replay buffers (20% down to 0%), then reports forward and backward accuracy from those runs. The abstract numbers (74.44% NLU, 40% vs 20% Code, 45% to 65% retention) are outputs of evaluation, not outputs of any fitted equation or of the paper's own theoretical premise. The 'quantization-induced noise acts as implicit regularization' account in Section 3 is explicitly labeled a hypothesis and speculation ('We hypothesize...', 'We speculate...', 'Further investigation is needed to validate this hypothesis'), and it is not used to generate the accuracy numbers, so it cannot make the results circular. The recommendations in Section 5 are summaries of the observed trade-off curves rather than predictions derived from a fitted model. No load-bearing self-citation chain appears: the LoRI protocol [26] is an external benchmark protocol, and no uniqueness theorem, ansatz, or fitted parameter is imported from the authors' own prior work. The admitted limitations in Section 5 — single model, limited tasks, and especially 'our experiments lack multi-seed runs and confidence intervals' — are statistical-reproducibility concerns, not instances of a prediction reducing by construction to its input. Therefore no specific circular step can be exhibited, and the appropriate finding is no significant circularity.

Axiom & Free-Parameter Ledger

1 free parameters · 4 axioms · 1 invented entities

No physical entities are introduced; one explanatory construct—quantization-induced implicit regularization—is postulated post hoc with no independent evidence. The central claim rests on experimental setup choices rather than fitted parameters: LoRA with frozen base weights, one epoch per task, a fixed dataset sequence, and uniform random replay.

free parameters (1)
  • LoRA fine-tuning hyperparameters = r=8, alpha=16, lr=2e-4, batch=8, one epoch
    Hand-set for all runs; no tuning or sensitivity analysis, so the precision effects could be confounded by suboptimal hyperparameters for one precision.
axioms (4)
  • domain assumption LoRA with frozen quantized base weights and high-precision adapters is a valid setup for isolating quantization effects in continual learning
    The paper's mechanism story presumes quantization noise influences learning, but with frozen base weights the noise is static and may not directly affect adapter gradients. (Section 2, Methods)
  • domain assumption The three-stage NLU→Math→Code sequence with 1 epoch per task is representative of continual learning
    Results may not transfer to other task orders, domains, or training budgets; the paper itself limits evaluation to LLaMA-3.1-8B and three task families. (Section 5, Limitations)
  • domain assumption Uniform random replay sampling per dataset is sufficient to characterize replay effects
    The paper acknowledges that more sophisticated selection strategies (herding, clustering) may shift the trade-offs. (Section 5, Limitations)
  • ad hoc to paper Quantization-induced noise acts as implicit regularization
    This is proposed as a hypothesis/speculation in Section 3 and is not directly measured; it is used to explain the observed precision inversion.
invented entities (1)
  • Implicit regularization from quantization noise no independent evidence
    purpose: Explains why quantized models retain knowledge better than FP16; not directly measured
    Proposed as a hypothesis (Section 3 Interpretation); no independent test distinguishes it from alternative explanations such as altered optimization dynamics.

pith-pipeline@v1.3.0-alltime-deepseek · 5876 in / 13602 out tokens · 125451 ms · 2026-08-03T14:49:24.235758+00:00 · methodology

0 comments
read the original abstract

Catastrophic forgetting poses a fundamental challenge in continual learning, particularly when models are quantized for deployment efficiency. We systematically investigate the interplay between quantization precision (FP16, INT8, INT4) and replay buffer strategies in large language models, revealing unexpected dynamics. While FP16 achieves superior initial task performance (74.44% on NLU), we observe a striking inversion on subsequent tasks: quantized models outperform FP16 by 8-15% on final task forward accuracy, with INT4 achieving nearly double FP16's performance on Code generation (40% vs 20%). Critically, even minimal replay buffers (0.1%) dramatically improve retention - increasing NLU retention after Math training from 45% to 65% across all precision levels - with INT8 consistently achieving the optimal balance between learning plasticity and knowledge retention. We hypothesize that quantization-induced noise acts as implicit regularization, preventing the overfitting to new task gradients that plagues high-precision models. These findings challenge the conventional wisdom that higher precision is always preferable, suggesting instead that INT8 quantization offers both computational efficiency and superior continual learning dynamics. Our results provide practical guidelines for deploying compressed models in continual learning scenarios: small replay buffers (1-2%) suffice for NLU tasks, while Math and Code benefit from moderate buffers (5-10%), with quantized models requiring less replay than FP16 to achieve comparable retention. Code is available at https://github.com/Festyve/LessIsMore.

Figures

Figures reproduced from arXiv: 2512.18934 by Arnav Kewalram, Kevin Zhu, Michael S. Zhang, Rishi A. Ruia, Saathvik Dharmapuram, Utkarsh Sharma.

Figure 1
Figure 1. Figure 1: Forward Accuracy. Forward Accuracy. Current task performance across different quantization levels and replay buffer sizes (log scale). NLU performance is unaffected by replay buffers [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Backward Accuracy. Retention of previous task performance after training on subsequent tasks. Dashed lines indicate retention metrics. X-axis uses log scale to emphasize critical low-replay region. We analyze the effect of quantization and replay on continual learning performance, reporting both average forward accuracy (performance immediately after training on each task, [PITH_FULL_IMAGE:figures/full_fi… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. SuperLocalMemory V3.3: The Living Brain -- Biologically-Inspired Forgetting, Cognitive Quantization, and Multi-Channel Retrieval for Zero-LLM Agent Memory Systems

    cs.AI 2026-04 unverdicted novelty 7.0

    SuperLocalMemory V3.3 implements a cognitive memory taxonomy with mathematical forgetting and multi-channel retrieval, reaching 70.4% on LoCoMo in zero-LLM mode.

  2. What Survives When You Compress a Recursive Reasoner for the Edge?

    cs.LG 2026-06 unverdicted novelty 6.0

    Aggressive compression of recursive reasoners keeps local predictions intact but destroys global reasoning accuracy, recoverable with calibrated INT4 and detectable via carry-trajectory fidelity.

Reference graph

Works this paper leans on

26 extracted references · 25 linked inside Pith · cited by 2 Pith papers

  1. [1]

    Post-training 4-bit quantization of convolution networks for rapid deployment.arXiv preprint arXiv:1810.05723, 2018

    Ron Banner, Yury Nahshan, Elad Hoffer, and Daniel Soudry. Post-training 4-bit quantization of convolution networks for rapid deployment.arXiv preprint arXiv:1810.05723, 2018

    Pith/arXiv arXiv 2018

  2. [2]

    Experience grounds language.arXiv preprint arXiv:2004.10151, (arXiv:2004.10151), November 2020

    Yonatan Bisk, Ari Holtzman, Jesse Thomason, Jacob Andreas, Yoshua Bengio, Joyce Chai, Mirella Lapata, Angeliki Lazaridou, Jonathan May, Aleksandr Nisnevich, Nicolas Pinto, and Joseph Turian. Experience grounds language.arXiv preprint arXiv:2004.10151, (arXiv:2004.10151), November 2020. arXiv:2004.10151

    Pith/arXiv arXiv 2004

  3. [3]

    Ex-model: Continual learning from a stream of trained models.arXiv preprint arXiv:2112.06511, (arXiv:2112.06511), December 2021

    Antonio Carta, Andrea Cossu, Vincenzo Lomonaco, and Davide Bacciu. Ex-model: Continual learning from a stream of trained models.arXiv preprint arXiv:2112.06511, (arXiv:2112.06511), December 2021. arXiv:2112.06511

    Pith/arXiv arXiv 2021

  4. [4]

    Dokania, Philip H

    Arslan Chaudhry, Marcus Rohrbach, Mohamed Elhoseiny, Thalaiyasingam Ajanthan, Puneet K. Dokania, Philip H. S. Torr, and Marc’Aurelio Ranzato. On tiny episodic memories in continual learning.arXiv preprint arXiv:1902.10486, (arXiv:1902.10486), June 2019. arXiv:1902.10486

    Pith/arXiv arXiv 1902

  5. [5]

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian...

    Pith/arXiv arXiv 2021

  6. [6]

    Overcoming forgetting catastrophe in quantization-aware training

    Ting-An Chen, De-Nian Yang, and Ming-Syan Chen. Overcoming forgetting catastrophe in quantization-aware training. InProceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pages 17358–17367, 2023

    2023

  7. [7]

    Think you have solved question answering? try arc, the ai2 reasoning chal- lenge.arXiv preprint arXiv:1803.05457, (arXiv:1803.05457), March 2018

    Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning chal- lenge.arXiv preprint arXiv:1803.05457, (arXiv:1803.05457), March 2018. arXiv:1803.05457

    Pith/arXiv arXiv 2018

  8. [8]

    Training verifiers to solve math word problems.arXiv preprint arXiv:2110.14168, (arXiv:2110.14168), November 2021

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. Training verifiers to solve math word problems.arXiv preprint arXiv:2110.14168, (arXiv:2110.14168), November 2021. arXiv:2110.14168

    Pith/arXiv arXiv 2021

  9. [9]

    Llm.int8(): 8-bit matrix multiplication for transformers at scale.arXiv preprint arXiv:2208.07339, (arXiv:2208.07339), November 2022

    Tim Dettmers, Mike Lewis, Younes Belkada, and Luke Zettlemoyer. Llm.int8(): 8-bit matrix multiplication for transformers at scale.arXiv preprint arXiv:2208.07339, (arXiv:2208.07339), November 2022. arXiv:2208.07339

    Pith/arXiv arXiv 2022

  10. [10]

    8-bit optimizers via block-wise quantization.arXiv preprint arXiv:2110.02861, (arXiv:2110.02861), June 2022

    Tim Dettmers, Mike Lewis, Sam Shleifer, and Luke Zettlemoyer. 8-bit optimizers via block-wise quantization.arXiv preprint arXiv:2110.02861, (arXiv:2110.02861), June 2022. arXiv:2110.02861. 5

    Pith/arXiv arXiv 2022

  11. [11]

    Hu, Nikolay Malkin, Moksh Jain, Katie Everett, Alexandros Graikos, and Yoshua Bengio

    Edward J. Hu, Nikolay Malkin, Moksh Jain, Katie Everett, Alexandros Graikos, and Yoshua Bengio. Gflownet-em for learning compositional latent variable models.arXiv preprint arXiv:2302.06576, (arXiv:2302.06576), June 2023. arXiv:2302.06576

    Pith/arXiv arXiv 2023

  12. [12]

    Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen

    Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models.arXiv preprint arXiv:2106.09685, (arXiv:2106.09685), October 2021. arXiv:2106.09685

    Pith/arXiv arXiv 2021

  13. [13]

    Quantization and training of neural networks for efficient integer-arithmetic-only inference.arXiv preprint arXiv:1712.05877, 2018

    Benoit Jacob, Skirmantas Kligys, Bo Chen, Menglong Zhu, Matthew Tang, Andrew Howard, Hartwig Adam, and Dmitry Kalenichenko. Quantization and training of neural networks for efficient integer-arithmetic-only inference.arXiv preprint arXiv:1712.05877, 2018

    Pith/arXiv arXiv 2018

  14. [14]

    Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, Demis Hassabis, Claudia Clopath, Dharshan Kumaran, and Raia Hadsell

    James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, An- drei A. Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, Demis Hassabis, Claudia Clopath, Dharshan Kumaran, and Raia Hadsell. Overcoming catastrophic forgetting in neural networks.arXiv preprint arXiv:1612.00796, (arXiv:1612.00796), January

    Pith/arXiv arXiv

  15. [15]

    Learning without forgetting.arXiv preprint arXiv:1606.09282, (arXiv:1606.09282), February 2017

    Zhizhong Li and Derek Hoiem. Learning without forgetting.arXiv preprint arXiv:1606.09282, (arXiv:1606.09282), February 2017. arXiv:1606.09282

    Pith/arXiv arXiv 2017

  16. [16]

    A structured self-attentive sentence embedding.arXiv preprint arXiv:1703.03130, 2017

    Zhouhan Lin, Minwei Feng, Cicero Nogueira dos Santos, Mo Yu, Bing Xiang, Bowen Zhou, and Yoshua Bengio. A structured self-attentive sentence embedding.arXiv preprint arXiv:1703.03130, 2017

    Pith/arXiv arXiv 2017

  17. [17]

    Gradient episodic memory for continual learning

    David Lopez-Paz and Marc’Aurelio Ranzato. Gradient episodic memory for continual learning. arXiv preprint arXiv:1706.08840, (arXiv:1706.08840), September 2022. arXiv:1706.08840

    Pith/arXiv arXiv 2022

  18. [18]

    An empirical study of catastrophic forgetting in large language models during continual fine-tuning.arXiv preprint arXiv:2308.08747, (arXiv:2308.08747), January 2025

    Yun Luo, Zhen Yang, Fandong Meng, Yafu Li, Jie Zhou, and Yue Zhang. An empirical study of catastrophic forgetting in large language models during continual fine-tuning.arXiv preprint arXiv:2308.08747, (arXiv:2308.08747), January 2025. arXiv:2308.08747

    Pith/arXiv arXiv 2025

  19. [19]

    Piggyback: Adapting a single net- work to multiple tasks by learning to mask weights.arXiv preprint arXiv:1801.06519, (arXiv:1801.06519), March 2018

    Arun Mallya, Dillon Davis, and Svetlana Lazebnik. Piggyback: Adapting a single net- work to multiple tasks by learning to mask weights.arXiv preprint arXiv:1801.06519, (arXiv:1801.06519), March 2018. arXiv:1801.06519

    Pith/arXiv arXiv 2018

  20. [20]

    Can a suit of armor conduct electricity? a new dataset for open book question answering.arXiv preprint arXiv:1809.02789, (arXiv:1809.02789), September 2018

    Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can a suit of armor conduct electricity? a new dataset for open book question answering.arXiv preprint arXiv:1809.02789, (arXiv:1809.02789), September 2018. arXiv:1809.02789

    Pith/arXiv arXiv 2018

  21. [21]

    Sylvestre-Alvise Rebuffi, Alexander Kolesnikov, Georg Sperl, and Christoph H. Lampert. icarl: Incremental classifier and representation learning.arXiv preprint arXiv:1611.07725, (arXiv:1611.07725), April 2017. arXiv:1611.07725

    Pith/arXiv arXiv 2017

  22. [22]

    Winogrande: An adversarial winograd schema challenge at scale.arXiv preprint arXiv:1907.10641, (arXiv:1907.10641), November 2019

    Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adversarial winograd schema challenge at scale.arXiv preprint arXiv:1907.10641, (arXiv:1907.10641), November 2019. arXiv:1907.10641

    Pith/arXiv arXiv 1907

  23. [23]

    Socialiqa: Commonsense reasoning about social interactions.arXiv preprint arXiv:1904.09728, (arXiv:1904.09728), September 2019

    Maarten Sap, Hannah Rashkin, Derek Chen, Ronan LeBras, and Yejin Choi. Socialiqa: Commonsense reasoning about social interactions.arXiv preprint arXiv:1904.09728, (arXiv:1904.09728), September 2019. arXiv:1904.09728

    Pith/arXiv arXiv 1904

  24. [24]

    Hellaswag: Can a machine really finish your sentence?arXiv preprint arXiv:1905.07830, (arXiv:1905.07830), May 2019

    Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a machine really finish your sentence?arXiv preprint arXiv:1905.07830, (arXiv:1905.07830), May 2019. arXiv:1905.07830

    Pith/arXiv arXiv 1905

  25. [25]

    Continual learning through synaptic intelli- gence.arXiv preprint arXiv:1703.04200, (arXiv:1703.04200), June 2017

    Friedemann Zenke, Ben Poole, and Surya Ganguli. Continual learning through synaptic intelli- gence.arXiv preprint arXiv:1703.04200, (arXiv:1703.04200), June 2017. arXiv:1703.04200

    Pith/arXiv arXiv 2017

  26. [26]

    Lori: Reducing cross-task interference in multi-task low-rank adaptation.arXiv preprint arXiv:2504.07448, (arXiv:2504.07448), August 2025

    Juzheng Zhang, Jiacheng You, Ashwinee Panda, and Tom Goldstein. Lori: Reducing cross-task interference in multi-task low-rank adaptation.arXiv preprint arXiv:2504.07448, (arXiv:2504.07448), August 2025. arXiv:2504.07448. 6

    Pith/arXiv arXiv 2025