Pith. sign in

REVIEW 4 major objections 5 minor 3 cited by

The Future of Continual Learning in the Era of Foundation Models: Three Key Directions

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Continual learning's future lies in composing and orchestrating many models, not in scaling one monolithic model.

desk verdict A useful and readable position paper whose headline claim—that continual compositionality and orchestration is the most promising direction—goes beyond what the paper's own evidence supports. read the letter →

arxiv 2506.03320 v1 pith:IFY5T4RP submitted 2025-06-03 cs.LG cs.AI

Jack Bell , Luigi Quarantiello , Eric Nuertey Coleman , Lanpei Li , Malio Li , Mauro Madeddu , Elia Piccoli , Vincenzo Lomonaco This is my paper
classification cs.LGcs.AI
keywords continuallearningfoundationmodelspre-trainingfine-tuningcompositionalityorchestrationmulti-agentsystemscatastrophicforgetting
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that continual learning remains essential even when foundation models have access to internet-scale knowledge, for three reasons: models must keep absorbing new data (continual pre-training, CPT), they must specialize and personalize cheaply (continual fine-tuning, CFT), and they must be dynamically composed with other models to solve higher-level tasks (continual compositionality and orchestration, CCO). The paper's central contention is that CCO is the most promising and necessary of the three, because recent AI progress has come less from added compute than from orchestration-like mechanisms such as reasoning, longer context windows, and multi-agent coordination. If correct, the field should pivot from scaling or weight-level continual learning toward modular, decentralized ecosystems of continually evolving and interacting agents, making continual learning more relevant than ever.

What carries the argument

The central object is Continual Compositionality & Orchestration (CCO), defined as the dynamic integration of multiple AI agents over time to solve higher-level tasks. It carries the argument by supplying the high-frequency adaptation loop that CPT and CFT lack: instead of adjusting a single network's weights, an orchestrator selects and composes modules, including models, experts, tools, and memories, on the fly, while continual learning contributes rapid adaptation and memory consolidation to stabilize useful compositions. Supporting evidence includes test-time scaling laws that favor added inference compute over parameter growth, advances in reasoning and long context windows, and the diversity argument that a diverse set of agents can outperform a sample of the best individual performers.

What would settle it

Run a controlled benchmark that spends the same compute budget either on continued pre-training or fine-tuning of one model or on composing and orchestrating a team of smaller models; if the single model consistently achieves equal or larger accuracy gains across tasks, the paper's priority claim collapses.

Watch

Extended reading notes

Core claim

The central claim is that continual learning in the foundation-model era splits into three directions, and that the third direction, Continual Compositionality & Orchestration, will mark the rebirth of continual learning. CPT keeps a large model current against knowledge staleness and distribution shift; CFT enables task-specific specialization and personalization without full retraining; CCO treats an AI system as a living assembly of modules, prompt routers, domain experts, tools, and episodic memories that can be composed, recombined, and adapted at high frequency. The authors contend that CCO is indispensable because real-world change often appears at the level of orchestration, such as new tools, revised chains of thought, or reorganized agent teams, requiring updates in minutes or seconds and making repeated offline training cycles impractical. The future they foresee is not a single static model but an ecosystem of continually evolving and interacting models.

Load-bearing premise

The argument assumes that orchestration-level gains, such as test-time scaling, multi-agent coordination, and longer context windows, will keep outpacing gains from training bigger or better single models, so the field should prioritize CCO over the other two directions.

Editorial extensions

If this is right

  • If correct, research emphasis in continual learning should shift from weight-level methods to orchestration and composition.
  • Academic labs can contribute to CCO without owning huge compute, since CCO is where the paper places primary emphasis for research.
  • Foundation-model systems become living assemblies whose components can be updated, replaced, or recombined as new tools, regulations, or tasks appear.
  • Continual learning becomes a foundational requirement rather than an optional enhancement, repositioning the field from niche research to the core of AI's next paradigm.
  • Specialized models distilled for particular domains will be composed rather than replaced by monolithic generalists.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural test of the priority claim is to measure, on standard benchmarks, whether a fixed budget spent on orchestrating a team of smaller models reliably beats the same budget spent on continued pre-training or fine-tuning of one larger model.
  • The emphasis on CCO implies that evaluation protocols should move beyond task-incremental and domain-incremental benchmarks toward dynamic composition benchmarks where the set of available modules and the orchestration graph change over time.
  • If orchestration gains continue to outpace parameter scaling, weight-level continual learning could become a supporting technology that supplies the modules to be composed, rather than the main locus of continual learning research.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. This paper is a position/survey paper on continual learning in the era of foundation models. It reviews the history of continual learning, organizes current work into three directions—continual pre-training (CPT), continual fine-tuning (CFT), and continual compositionality and orchestration (CCO)—and argues that CCO is the most promising and necessary direction for future research. Section 2 provides background on classical continual learning methods and scenarios; Section 3 motivates continual learning for foundation models; Section 4 surveys each of the three directions, with CPT and CFT receiving substantial literature coverage and CCO framed around multi-agent orchestration, memory modules, and communication protocols; Section 5 and the conclusion recommend that academic research should prioritize CCO. The paper contains no experiments, benchmarks, or formal derivations beyond a restatement of the LoRA update.

Significance. The paper is valuable as a structured overview of an active debate and as a research agenda. Its strengths are the breadth of the CPT/CFT survey, the explicit enumeration of open problems (e.g., the stability gap, bias drift during CPT, PEFT-based continual learning, model merging), and a genuinely different architectural proposal that goes beyond the usual 'continual learning for LLMs' framing: a decentralized ecosystem of continually interacting models. The CCO discussion connects to relevant prior work on agents, test-time inference, memory modules (Voyager, GITM, AppAgent), and model merging. However, the central priority claim is not tested: no experiment, benchmark, or formal model separates CCO from CPT/CFT, and the key motivating premise—that recent FM progress is 'orchestration rather than scale'—is supported by references that, on inspection, concern within-model inference techniques. The paper is best read as a position piece proposing a research direction, not as a validated result.

major comments (4)
  1. [Section 1] The premise that 'recent advances in FMs have primarily emerged not from additional computational resources, but rather through enhanced reasoning abilities [15] and longer context windows [16], both of which are forms of orchestration rather than scale' is not supported by the cited evidence. Reference [12] is a test-time compute scaling study within a single multimodal model; reference [15] is a survey of post-training reasoning LLMs; reference [16] reports a longer context window for one model. None of these involves composing multiple models or agents, and test-time compute scaling is literally a form of inference-time scaling rather than orchestration. Because this premise is the main argument for the conclusion that CCO is 'the most promising and necessary direction,' the paper should either re-classify these results as within-model inference-time computation and argue separately that cross-model coordination will dominate, or soften the claim to a hypothesis with explicit scope conditions.
  2. [Section 4.3.3] The paper asserts that CCO 'inherently supports high-frequency adaptation' and that current frameworks such as LangGraph and AutoGen are 'largely static and predefined,' but it presents no evidence that dynamically re-routing or recombining modules over time improves performance over a static composition. A concrete falsifiable design is needed: compare, on a sequence of tasks with known distribution shifts, (i) a static orchestration graph with fixed routing, (ii) the same modules with a learned dynamic router that updates over time, and (iii) a single monolithic FM scaled to comparable total compute; report accuracy, forgetting, and adaptation latency. Until such a comparison exists, the priority claim remains an untested assumption.
  3. [Section 5] The load-bearing assumption that orchestration-level adaptation will outpace further parameter scaling is not examined. Section 5 states that real-world change is 'expressed at the level of orchestration' and that repeated offline training is impractical, citing [15]; however, reference [15] is about post-training large models with RL, which itself consumes substantial computational resources. The paper offers no evidence that high-frequency re-composition—new tools, revised chains of thought, reorganized agents—will continue to yield larger gains than improved weight-level continual learning or larger scale. To make the claim defensible, the authors should state a conditional thesis (e.g., 'if gains from coordination continue to grow as module pools and context lengths grow, then CCO is the highest-value direction') and identify data or benchmarks that would falsify it.
  4. [Sections 4.2.3 and 4.3] The boundary between CFT and CCO is not crisply defined, which makes the priority claim hard to test. Model merging and adaptive routing methods (TIES-MERGING, DARE, MagMax, Adaptive LoRA Merging) are described under CFT in Section 4.2.3, yet they also compose models or modules dynamically; Section 4.3.3 then proposes 'dynamic selection and composition' as a CCO contribution. If module merging and routing are already CFT tools, then CCO is not a separate direction but an application of CFT, and the statement that CCO is 'not merely advantageous - it is indispensable' loses its contrast. Please state explicit inclusion criteria—for example, whether the modules being composed are frozen, whether the composition itself is learned, and what time scale counts as 'continual'—that separate CCO from CFT.
minor comments (5)
  1. [Section 4.2.1] There is a duplicated word: 'Fine-tuning is attractive because it it far cheaper' should read 'because it is far cheaper.'
  2. [Section 4.2.3] There is a typo: 'while preserving data privay' should read 'while preserving data privacy.'
  3. [Section 1] Reference [14] is cited together with [13] as evidence of multi-agent frameworks, but [14] is a chain-of-thought paper, not a multi-agent framework; please re-cite or rephrase the sentence.
  4. [Section 3] The cost estimate for training models 'comparable to GPT-4.5' is attributed to [47], which is Hoffmann et al.'s compute-optimal scaling paper rather than a GPT-4.5 cost estimate; the citation should be updated or the claim softened.
  5. [Section 1] The term 'orchestration' is used in two different senses: within-model inference-time computation (following [12], [15], [16]) and cross-agent composition (Section 4.3). This ambiguity should be resolved, for example by distinguishing 'intra-model orchestration' from 'inter-model orchestration'.

Circularity Check

0 steps flagged · score 2.0 of 10

No significant circularity: the paper is a programmatic position piece; its priority claim rests on external trends and its self-citations are secondary, not load-bearing.

full rationale

This is a perspective/survey paper without a derivation chain: there are no fitted parameters, no uniqueness theorems, and no predictions that reduce to their inputs by construction. The only equation, W = W0 + BA, is the standard LoRA update and is not used to derive any claim. The central assertion that Continual Compositionality and Orchestration (CCO) is the most promising direction is argued from external references about test-time scaling, reasoning post-training, and long context windows ([12], [15], [16]); those references are independent of the present authors and are not re-fitted or redefined here. The paper explicitly concedes that existing orchestration frameworks such as LangGraph and AutoGen are 'largely static and predefined' (Section 4.3.3), so CCO is presented as a proposed future direction rather than a result already established by the paper's own machinery. The self-citations that appear ([7], [84], [96], [132]) are used as supporting empirical or methodological evidence for narrower points, such as self-supervised continual pre-training mitigating forgetting, online continual learning evaluations, adaptive LoRA merging, and model-stream continual learning; none of these is the load-bearing premise for the paper's headline claim. The skeptic's objection that 'orchestration' is stretched to include within-model reasoning and context scaling is a classification and evidence-weight concern, not a circular reduction: the paper does not define CCO in terms of the cited trends and then re-derive those trends from CCO. Accordingly, the paper shows no significant circularity, and the score of 2 reflects only the presence of minor, non-load-bearing self-citations.

Assumptions & free parameters 0 free parameters · 4 assumptions · 1 invented entities

The argument rests on assumptions about model staticity, test-time scaling trends, and agent diversity, plus the practical constraints of CPT/CFT. No free parameters are fitted. The only introduced conceptual entity is the CCO framework, which has no independent falsifiable evidence in the paper.

assumptions (4)
  • domain assumption Foundation models are static snapshots that become stale as the world changes.
    Used throughout Sections 1 and 3 to motivate the need for continual learning. It is a reasonable claim, but not demonstrated in this paper.
  • domain assumption Test-time scaling and orchestration-level improvements currently outpace gains from further parameter growth.
    Invoked in Section 1 via refs [12-16]; this is load-bearing for prioritizing CCO over CPT/CFT, and the paper does not critically evaluate counter-trends.
  • domain assumption A diverse group of agents can outperform a group of the best individual agents.
    Cited from Hong and Page [17] in Section 1 and extended to AI agents without direct evidence.
  • domain assumption Continual pre-training and fine-tuning are inherently lower-frequency and resource-hungry, limiting their potential.
    Sections 1 and 5 use this to argue CCO superiority; it is a practical trend, not a proven limit.
invented entities (1)
  • Continual Compositionality and Orchestration (CCO)
    purpose: Proposed organizing framework where AI systems are dynamic assemblies of modules, agents, routers, and memories, composed at test time.
    Named and defined in this paper; no falsifiable predictions or concrete implementation are given, so it functions as a research agenda rather than a measurable entity.

how reviews work

0 comments
Cite this review

Pith. "Pith review of The Future of Continual Learning in the Era of Foundation Models: Three Key Directions." pith.science (2026). https://pith.science/paper/IFY5T4RP

@misc{pith2026250603320,
  author       = {Pith},
  title        = {Pith review of: The Future of Continual Learning in the Era of Foundation Models: Three Key Directions},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/IFY5T4RP}},
  note         = {Machine review of arXiv:2506.03320}
}
read the original abstract

Continual learning--the ability to acquire, retain, and refine knowledge over time--has always been fundamental to intelligence, both human and artificial. Historically, different AI paradigms have acknowledged this need, albeit with varying priorities: early expert and production systems focused on incremental knowledge consolidation, while reinforcement learning emphasised dynamic adaptation. With the rise of deep learning, deep continual learning has primarily focused on learning robust and reusable representations over time to solve sequences of increasingly complex tasks. However, the emergence of Large Language Models (LLMs) and foundation models has raised the question: Do we still need continual learning when centralised, monolithic models can tackle diverse tasks with access to internet-scale knowledge? We argue that continual learning remains essential for three key reasons: (i) continual pre-training is still necessary to ensure foundation models remain up to date, mitigating knowledge staleness and distribution shifts while integrating new information; (ii) continual fine-tuning enables models to specialise and personalise, adapting to domain-specific tasks, user preferences, and real-world constraints without full retraining, avoiding the need for computationally expensive long context-windows; (iii) continual compositionality offers a scalable and modular approach to intelligence, enabling the orchestration of foundation models and agents to be dynamically composed, recombined, and adapted. While continual pre-training and fine-tuning are explored as niche research directions, we argue it is continual compositionality that will mark the rebirth of continual learning. The future of AI will not be defined by a single static model but by an ecosystem of continually evolving and interacting models, making continual learning more relevant than ever.

Figures

Figures reproduced from arXiv: 2506.03320 by the authors.

Figure 1
Figure 1. In (a), we see a base model pre-trained on video data is Continually Pre-trained on general corpora spanning different modalities (e.g. audio, images and text). Then in (b), this base model is Continually Fine-tuned over time, resulting in specialised fine-tuning (FT) modules trained on domain-specific datasets, such as medical texts. Finally in (c)—looking at model inference, an orchestrator routes a user’s query t… view at source ↗

Discussion (0). Sign in to comment.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Continual Learning in Transition

    cs.LG 2026-08 accept novelty 5.0 of 10

    A tri-axial framework of When, Where, and How organizes the ongoing transition of continual learning from parameter-centric updates to system-level capability evolution.

  2. Lifelong Representations: A Survey on Continual Self-Supervised Learning for Vision Models

    cs.CV 2026-07 accept novelty 5.0 of 10

    CSSL for vision is more robust to forgetting than supervised CL due to task-agnostic features and flatter losses, yet still needs better protocols and scaling beyond small benchmarks to foundation-model continual pretraining.

  3. A Compositional Paradigm for Foundation Models: Towards Smarter Robotic Agents

    cs.RO 2025-10 conditional novelty 3.0 of 10

    The paper proposes continual learning plus compositional adapters as the path to adaptable robot foundation models, but its two supporting experiments are underspecified and reliant on the authors' earlier work.

Reference graph

Works this paper leans on

136 extracted references · 14 canonical work pages · cited by 3 Pith papers

  1. [12]

    Z. Chen, W. Wang, Y. Cao, Y. Liu, Z. Gao, E. Cui, J. Zhu, S. Ye, H. Tian, Z. Liu, L. Gu, X. Wang, Q. Li, Y. Ren, Z. Chen, J. Luo, J. Wang, T. Jiang, B. Wang, C. He, B. Shi, X. Zhang, H. Lv, Y. Wang, W. Shao, P. Chu, Z. Tu, T. He, Z. Wu, H. Deng, J. Ge, K. Chen, K. Zhang, L. Wang, M. Dou, L. Lu, X. Zhu, T. Lu, D. Lin, Y. Qiao, J. Dai, W. Wang, Expanding Pe...

    arXiv doi:10.48550/arxiv.2412.05271 2025

  2. [15]

    Kumar, T

    K. Kumar, T. Ashraf, O. Thawakar, R. M. Anwer, H. Cholakkal, M. Shah, M.-H. Yang, P. H. S. Torr, F. S. Khan, S. Khan, LLM Post-Training: A Deep Dive into Reasoning Large Language Models, 2025. URL: http://arxiv.org/abs/2502.21321. doi: 10.48550/arXiv.2502.21321, arXiv:2502.21321 [cs]

    arXiv doi:10.48550/arxiv.2502.21321 2025

  3. [16]

    G. Team, P. Georgiev, V. I. Lei, R. Burnell, L. Bai, A. Gulati, G. Tanzer, D. Vincent, Z. Pan, S. Wang, et al., Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context, 2024

    2024

  4. [1]

    R. M. French, Catastrophic forgetting in connectionist networks, Trends in Cognitive Sciences 3 (1999) 128–135. URL: https://www.cell.com/trends/cognitive-sciences/abstract/S1364-6613(99) 01294-2. doi:10.1016/S1364-6613(99)01294-2, publisher: Elsevier

    doi:10.1016/s1364-6613(99)01294-2 1999

  5. [2]

    Ditzler, M

    G. Ditzler, M. Roveri, C. Alippi, R. Polikar, Learning in nonstationary environments: A survey, IEEE Computational Intelligence Magazine 10 (2015) 12–25

    2015

  6. [3]

    Ring, Continual learning in reinforcement environments, 1994

    M. Ring, Continual learning in reinforcement environments, 1994. URL: https://www.proquest. com/openview/2d2f13eb52fc09d3eadfd0c81fe5f181/1?cbl=18750&diss=y&pq-origsite=gscholar

    1994

  7. [4]

    Learning Task Grouping and Overlap in Multi-task Learning

    A. Kumar, H. D. III, Learning Task Grouping and Overlap in Multi-task Learning, 2012. URL: http://arxiv.org/abs/1206.6417. doi:10.48550/arXiv.1206.6417, arXiv:1206.6417 [cs]

    work page Pith review arXiv doi:10.48550/arxiv.1206.6417 2012

  8. [5]

    Giannini, G

    F. Giannini, G. Ziffer, A. Cossu, V. Lomonaco, Streaming Continual Learning for Unified Adap- tive Intelligence in Dynamic Environments, IEEE Intelligent Systems 39 (2024) 81–85. URL: https://ieeexplore.ieee.org/document/10779199/?arnumber=10779199. doi:10.1109/MIS.2024. 3479469, conference Name: IEEE Intelligent Systems

    work page arXiv 2024

Show all 136 references
  1. [6]

    W. Li, F. Qi, R. Yan, H. Zhang, W. Lei, J. Tang, J. Luo, Continual Learning meets Multimodal Foundation Models: Fundamentals and Advances, in: Proceedings of the 1st on Continual Learning meets Multimodal Foundation Models: Fundamentals and Advances, ACMMM CL’24, Association f...

    2024

  2. [7]

    Cossu, A

    A. Cossu, A. Carta, L. Passaro, V. Lomonaco, T. Tuytelaars, D. Bacciu, Continual pre-training mitigates forgetting in language and vision, Neural Networks 179 (2024) 106492

    2024

  3. [8]

    Zheng, C

    J. Zheng, C. Shi, X. Cai, Q. Li, D. Zhang, C. Li, D. Yu, Q. Ma, Lifelong Learning of Large Language Model based Agents: A Roadmap, 2025. URL: http://arxiv.org/abs/2501.07278. doi: 10.48550/ arXiv.2501.07278, arXiv:2501.07278 [cs]

    2025 doi

  4. [9]

    H. Shi, Z. Xu, H. Wang, W. Qin, W. Wang, Y. Wang, Z. Wang, S. Ebrahimi, H. Wang, Continual learning of large language models: A comprehensive survey, arXiv preprint arXiv:2404.16789 (2024)

    2024 arXiv

  5. [10]

    G. M. Van de Ven, T. Tuytelaars, A. S. Tolias, Three types of incremental learning, Nature Machine Intelligence 4 (2022) 1185–1197

    2022

  6. [11]

    T. K. Hensch, Critical period plasticity in local cortical circuits, Nature reviews neuroscience 6 (2005) 877–888

    2005

  7. [13]

    Schmidgall, M

    S. Schmidgall, M. Moor, AgentRxiv: Towards Collaborative Autonomous Research, 2025. URL: http://arxiv.org/abs/2503.18102. doi:10.48550/arXiv.2503.18102, arXiv:2503.18102 [cs]

    2025 doi arXiv

  8. [14]

    Z. Cai, B. Chang, W. Han, Human-in-the-loop through chain-of-thought, arXiv preprint arXiv:2306.07932 (2023)

    2023 arXiv

  9. [17]

    L. Hong, S. E. Page, Groups of diverse problem solvers can outperform groups of high-ability problem solvers, Proceedings of the National Academy of Sciences 101 (2004) 16385–16389

    2004

  10. [18]

    Lesort, V

    T. Lesort, V. Lomonaco, A. Stoian, D. Maltoni, D. Filliat, N. Díaz-Rodríguez, Continual Learning for Robotics: Definition, Framework, Learning Strategies, Opportunities and Challenges, 2019. URL: http://arxiv.org/abs/1907.00182. doi:10.48550/arXiv.1907.00182, arXiv:1907.00182 [cs]

    2019 doi arXiv

  11. [19]

    McCloskey, N

    M. McCloskey, N. J. Cohen, Catastrophic Interference in Connectionist Networks: The Se- quential Learning Problem, in: G. H. Bower (Ed.), Psychology of Learning and Motivation, volume 24, Academic Press, 1989, pp. 109–165. URL: https://www.sciencedirect.com/science/ article/pi...

    1989 doi

  12. [20]

    Ratcliff, Connectionist models of recognition memory: constraints imposed by learning and forgetting functions., Psychological review 97 (1990) 285

    R. Ratcliff, Connectionist models of recognition memory: constraints imposed by learning and forgetting functions., Psychological review 97 (1990) 285

    1990

  13. [21]

    R. K. Srivastava, J. Masci, S. Kazerounian, F. Gomez, J. Schmidhuber, Compete to Compute, in: C. J. Burges, L. Bottou, M. Welling, Z. Ghahramani, K. Q. Weinberger (Eds.), Advances in Neural Infor- mation Processing Systems, volume 26, Curran Associates, Inc., 2013. URL: https:...

    2013

  14. [22]

    I. J. Goodfellow, M. Mirza, D. Xiao, A. Courville, Y. Bengio, An Empirical Investigation of Catas- trophic Forgetting in Gradient-Based Neural Networks, 2015. URL: http://arxiv.org/abs/1312.6211. doi:10.48550/arXiv.1312.6211, arXiv:1312.6211 [stat]

    2015 doi arXiv

  15. [23]

    Z. Li, D. Hoiem, Learning without Forgetting, 2017. URL: http://arxiv.org/abs/1606.09282. doi:10. 48550/arXiv.1606.09282, arXiv:1606.09282 [cs]

    2017 doi arXiv

  16. [24]

    Kirkpatrick, R

    J. Kirkpatrick, R. Pascanu, N. Rabinowitz, J. Veness, G. Desjardins, A. A. Rusu, K. Milan, J. Quan, T. Ramalho, A. Grabska-Barwinska, D. Hassabis, C. Clopath, D. Kumaran, R. Hadsell, Overcoming catastrophic forgetting in neural networks, Proceedings of the National Academy of ...

    2017 arXiv

  17. [25]

    L. Wang, X. Zhang, H. Su, J. Zhu, A comprehensive survey of continual learning: Theory, method and application, IEEE Transactions on Pattern Analysis and Machine Intelligence (2024)

    2024

  18. [26]

    Wickramasinghe, G

    B. Wickramasinghe, G. Saha, K. Roy, Continual learning: A review of techniques, challenges, and future directions, IEEE Transactions on Artificial Intelligence 5 (2023) 2526–2546

    2023

  19. [27]

    Zenke, B

    F. Zenke, B. Poole, S. Ganguli, Continual Learning Through Synaptic Intelligence, 2017. URL: http://arxiv.org/abs/1703.04200. doi:10.48550/arXiv.1703.04200, arXiv:1703.04200 [cs]

    2017 doi arXiv

  20. [28]

    A. A. Rusu, N. C. Rabinowitz, G. Desjardins, H. Soyer, J. Kirkpatrick, K. Kavukcuoglu, R. Pascanu, R. Hadsell, Progressive Neural Networks, 2022. URL: http://arxiv.org/abs/1606.04671. doi: 10. 48550/arXiv.1606.04671, arXiv:1606.04671 [cs]

    2022 doi arXiv

  21. [29]

    Mallya, S

    A. Mallya, S. Lazebnik, PackNet: Adding Multiple Tasks to a Single Network by Itera- tive Pruning, 2018. URL: http://arxiv.org/abs/1711.05769. doi:10.48550/arXiv.1711.05769, arXiv:1711.05769 [cs]

    2018 doi arXiv

  22. [30]

    Mallya, D

    A. Mallya, D. Davis, S. Lazebnik, Piggyback: Adapting a Single Network to Multiple Tasks by Learning to Mask Weights, 2018. URL: http://arxiv.org/abs/1801.06519. doi:10.48550/arXiv. 1801.06519, arXiv:1801.06519 [cs]

    2018 doi arXiv

  23. [31]

    Frankle, M

    J. Frankle, M. Carbin, The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks, 2019. URL: http://arxiv.org/abs/1803.03635. doi: 10.48550/arXiv.1803.03635, arXiv:1803.03635 [cs]

    2019 doi arXiv

  24. [32]

    Serrà, D

    J. Serrà, D. Surís, M. Miron, A. Karatzoglou, Overcoming catastrophic forgetting with hard attention to the task, 2018. URL: http://arxiv.org/abs/1801.01423. doi:10.48550/arXiv.1801. 01423, arXiv:1801.01423 [cs]

    2018 doi arXiv

  25. [33]

    Wortsman, V

    M. Wortsman, V. Ramanujan, R. Liu, A. Kembhavi, M. Rastegari, J. Yosinski, A. Farhadi, Super- masks in Superposition, in: Advances in Neural Information Processing Systems, volume 33, Curran Associates, Inc., 2020, pp. 15173–15184. URL: https://proceedings.neurips.cc/paper/202...

    2020

  26. [34]

    Prabhu, P

    A. Prabhu, P. H. S. Torr, P. K. Dokania, GDumb: A Simple Approach that Questions Our Progress in Continual Learning, in: A. Vedaldi, H. Bischof, T. Brox, J.-M. Frahm (Eds.), Computer Vision – ECCV 2020, volume 12347, Springer International Publishing, Cham, 2020, pp. 524–540. ...

    2020 doi

  27. [35]

    Lopez-Paz, M

    D. Lopez-Paz, M. Ranzato, Gradient Episodic Memory for Continual Learning, 2022. URL: http: //arxiv.org/abs/1706.08840. doi:10.48550/arXiv.1706.08840, arXiv:1706.08840 [cs]

    2022 doi arXiv

  28. [36]

    Rebuffi, A

    S.-A. Rebuffi, A. Kolesnikov, G. Sperl, C. H. Lampert, iCaRL: Incremental Classifier and Representa- tion Learning, 2017. URL: http://arxiv.org/abs/1611.07725. doi:10.48550/arXiv.1611.07725, arXiv:1611.07725 [cs]

    2017 doi arXiv

  29. [37]

    Pellegrini, G

    L. Pellegrini, G. Graffieti, V. Lomonaco, D. Maltoni, Latent Replay for Real-Time Contin- ual Learning, 2020. URL: http://arxiv.org/abs/1912.01100. doi:10.48550/arXiv.1912.01100, arXiv:1912.01100 [cs]

    2020 doi arXiv

  30. [38]

    Graffieti, D

    G. Graffieti, D. Maltoni, L. Pellegrini, V. Lomonaco, Generative negative replay for continual learning, Neural Networks 162 (2023) 369–383

    2023

  31. [39]

    G. M. Van de Ven, A. S. Tolias, Three scenarios for continual learning, arXiv preprint arXiv:1904.07734 (2019)

    2019 arXiv

  32. [40]

    Devlin, M.-W

    J. Devlin, M.-W. Chang, K. Lee, K. Toutanova, BERT: Pre-training of Deep Bidirectional Trans- formers for Language Understanding, 2019. URL: http://arxiv.org/abs/1810.04805. doi:10.48550/ arXiv.1810.04805, arXiv:1810.04805 [cs]

    2019 doi arXiv

  33. [41]

    T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert-Voss, G. Krueger, T. Henighan, R. Child, A. Ramesh, D. M. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. ...

    2020 doi arXiv

  34. [42]

    Radford, J

    A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, G. Krueger, I. Sutskever, Learning Transferable Visual Models From Natural Language Supervision, 2021. URL: http://arxiv.org/abs/2103.00020. doi:10.48550/arXiv.2103.00...

    2021 doi arXiv

  35. [43]

    Bommasani, D

    R. Bommasani, D. A. Hudson, E. Adeli, R. Altman, S. Arora, S. v. Arx, M. S. Bernstein, J. Bohg, A. Bosselut, E. Brunskill, E. Brynjolfsson, S. Buch, D. Card, R. Castellon, N. Chatterji, A. Chen, K. Creel, J. Q. Davis, D. Demszky, C. Donahue, M. Doumbouya, E. Durmus, S. Ermon, ...

    2022 doi arXiv

  36. [44]

    C. Zhou, Q. Li, C. Li, J. Yu, Y. Liu, G. Wang, K. Zhang, C. Ji, Q. Yan, L. He, et al., A comprehensive survey on pretrained foundation models: A history from bert to chatgpt, International Journal of Machine Learning and Cybernetics (2024) 1–65

    2024

  37. [45]

    Y. Yang, J. Zhou, X. Ding, T. Huai, S. Liu, Q. Chen, Y. Xie, L. He, Recent advances of foundation language models-based continual learning: A survey, ACM Computing Surveys 57 (2025) 1–38

    2025

  38. [46]

    Plaat, A

    A. Plaat, A. Wong, S. Verberne, J. Broekens, N. v. Stein, T. Back, Reasoning with Large Lan- guage Models, a Survey, 2024. URL: http://arxiv.org/abs/2407.11511. doi:10.48550/arXiv.2407. 11511, arXiv:2407.11511 [cs] version: 1

    2024 doi

  39. [47]

    Hoffmann, S

    J. Hoffmann, S. Borgeaud, A. Mensch, E. Buchatskaya, T. Cai, E. Rutherford, D. d. L. Casas, L. A. Hendricks, J. Welbl, A. Clark, T. Hennigan, E. Noland, K. Millican, G. v. d. Driessche, B. Damoc, A. Guy, S. Osindero, K. Simonyan, E. Elsen, J. W. Rae, O. Vinyals, L. Sifre, Trai...

    2022 doi arXiv

  40. [48]

    Schwartz, J

    R. Schwartz, J. Dodge, N. A. Smith, O. Etzioni, Green AI, Commun. ACM 63 (2020) 54–63. URL: https://dl.acm.org/doi/10.1145/3381831. doi:10.1145/3381831

    2020 doi

  41. [49]

    Strubell, A

    E. Strubell, A. Ganesh, A. McCallum, Energy and Policy Considerations for Modern Deep Learning Research, Proceedings of the AAAI Conference on Artificial Intelligence 34 (2020) 13693–13696. URL: https://ojs.aaai.org/index.php/AAAI/article/view/7123. doi:10.1609/aaai.v34i09.712...

    2020 doi

  42. [50]

    H. R. Kirk, B. Vidgen, P. Röttger, S. A. Hale, Personalisation within bounds: A risk taxonomy and policy framework for the alignment of large language models with personalised feedback, 2023. URL: http://arxiv.org/abs/2303.05453. doi: 10.48550/arXiv.2303.05453, arXiv:2303.05453 [cs]

    2023 doi arXiv

  43. [51]

    Y. Sun, X. Wang, Z. Liu, J. Miller, A. A. Efros, M. Hardt, Test-Time Training for Out-of-Distribution Generalization (2019). URL: https://openreview.net/forum?id=HyezmlBKwr

    2019

  44. [52]

    E. M. Bender, T. Gebru, A. McMillan-Major, S. Shmitchell, On the Dangers of Stochastic Parrots: Can Language Models Be Too Big?, in: Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, FAccT ’21, Association for Computing Machinery, New York, ...

    2021

  45. [53]

    Pedreschi, L

    D. Pedreschi, L. Pappalardo, E. Ferragina, R. Baeza-Yates, A.-L. Barabási, F. Dignum, V. Dignum, T. Eliassi-Rad, F. Giannotti, J. Kertész, et al., Human-ai coevolution, Artificial Intelligence (2024) 104244

    2024

  46. [54]

    Raheja, N

    T. Raheja, N. Pochhi, Foundation models meet continual learning: Recent advances, challenges, and future directions, in: NeurIPS 2024 Workshop on Scalable Continual Learning for Lifelong Foundation Models, 2024

    2024

  47. [55]

    Y. Shen, K. Song, X. Tan, D. Li, W. Lu, Y. Zhuang, HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in Hugging Face, 2023. URL: http://arxiv.org/abs/2303.17580. doi: 10.48550/ arXiv.2303.17580, arXiv:2303.17580 [cs]

    2023 doi arXiv

  48. [56]

    K. Roth, V. Udandarao, S. Dziadzio, A. Prabhu, M. Cherti, O. Vinyals, O. Hénaff, S. Albanie, M. Bethge, Z. Akata, A practitioner’s guide to continual multimodal pretraining, arXiv preprint arXiv:2408.14471 (2024)

    2024 arXiv

  49. [57]

    Küchemann, K

    S. Küchemann, K. E. Avila, Y. Dinc, C. Hortmann, N. Revenga, V. Ruf, N. Stausberg, S. Steinert, F. Fischer, M. Fischer, et al., On opportunities and challenges of large multimodal foundation models in education, npj Science of Learning 10 (2025) 11

    2025

  50. [58]

    Parmar, S

    J. Parmar, S. Satheesh, M. Patwary, M. Shoeybi, B. Catanzaro, Reuse, don’t retrain: A recipe for continued pretraining of language models, arXiv preprint arXiv:2407.07263 (2024)

    2024 arXiv

  51. [59]

    The Llama 4 herd: The beginning of a new era of natively multimodal AI innovation, https://ai.meta.com/blog/llama-4-multimodal-intelligence/, 2025

    2025

  52. [60]

    Golatkar, A

    A. Golatkar, A. Achille, S. Soatto, Eternal sunshine of the spotless net: Selective forgetting in deep networks, in: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2020, pp. 9304–9312

    2020

  53. [61]

    L. Wang, X. Zeng, J. Guo, K.-F. Wong, G. Gottlob, Selective forgetting: Advancing machine un- learning techniques and evaluation in language models, in: Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, 2025, pp. 843–851

    2025

  54. [62]

    H. Zhao, B. Ni, J. Fan, Y. Wang, Y. Chen, G. Meng, Z. Zhang, Continual forgetting for pre-trained vision models, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024, pp. 28631–28642

    2024

  55. [63]

    B. Zhu, G. Cui, Y. Chen, Y. Qin, L. Yuan, C. Fu, Y. Deng, Z. Liu, M. Sun, M. Gu, Removing backdoors in pre-trained models by regularized continual pre-training, Transactions of the Association for Computational Linguistics 11 (2023) 1608–1623

    2023

  56. [64]

    Brinner, T

    M. Brinner, T. A. Mustafa, S. Zarrieß, Enhancing domain-specific encoder models with llm- generated data: How to leverage ontologies, and how to do without them, arXiv preprint arXiv:2503.22006 (2025)

    2025

  57. [65]

    Li, H.-Y

    C.-A. Li, H.-Y. Lee, Examining forgetting in continual pre-training of aligned large language models, arXiv preprint arXiv:2401.03129 (2024)

    2024 arXiv

  58. [66]

    S. V. Mehta, D. Patil, S. Chandar, E. Strubell, An empirical investigation of the role of pre-training in lifelong learning, Journal of Machine Learning Research 24 (2023) 1–50

    2023

  59. [67]

    T. Wu, L. Luo, Y.-F. Li, S. Pan, T.-T. Vu, G. Haffari, Continual learning for large language models: A survey, arXiv preprint arXiv:2402.01364 (2024)

    2024 arXiv

  60. [68]

    H. Zhu, Y. Wei, X. Liang, C. Zhang, Y. Zhao, Ctp: Towards vision-language continual pretraining via compatible momentum contrast and topology preservation, in: Proceedings of the IEEE/CVF International Conference on Computer Vision, 2023, pp. 22257–22267

    2023

  61. [69]

    Zhang, L

    G. Zhang, L. Wang, G. Kang, L. Chen, Y. Wei, Slca: Slow learner with classifier alignment for continual learning on a pre-trained model, in: Proceedings of the IEEE/CVF International Conference on Computer Vision, 2023, pp. 19148–19158

    2023

  62. [70]

    Yıldız, N

    Ç. Yıldız, N. K. Ravichandran, N. Sharma, M. Bethge, B. Ermis, Investigating continual pretraining in large language models: Insights and implications, arXiv preprint arXiv:2402.17400 (2024)

    2024 arXiv

  63. [71]

    Y. Guo, J. Fu, H. Zhang, D. Zhao, Y. Shen, Efficient continual pre-training by mitigating the stability gap, arXiv preprint arXiv:2406.14833 (2024)

    2024 arXiv

  64. [72]

    X. Li, L. Peng, Y.-P. Wang, W. Zhang, Open challenges and opportunities in federated foundation models towards biomedical healthcare, BioData Mining 18 (2025) 2

    2025

  65. [73]

    Mendieta, B

    M. Mendieta, B. Han, X. Shi, Y. Zhu, C. Chen, Towards geospatial foundation models via continual pretraining, in: Proceedings of the IEEE/CVF International Conference on Computer Vision, 2023, pp. 16806–16816

    2023

  66. [74]

    N. D. Noce, M. Resta, D. Bacciu, Sequential continual pre-training for neural machine translation, in: 32nd European Symposium on Artificial Neural Networks, Computational Intelligence and Machine Learning, ESANN, 2024. URL: https://doi.org/10.14428/esann/2024.ES2024-165

    2024 doi

  67. [75]

    Y. Xie, K. Aggarwal, A. Ahmad, Efficient continual pre-training for building domain specific large language models, in: Findings of the Association for Computational Linguistics ACL 2024, 2024, pp. 10184–10201

    2024

  68. [76]

    Ostapenko, T

    O. Ostapenko, T. Lesort, P. Rodriguez, M. R. Arefin, A. Douillard, I. Rish, L. Charlin, Continual learning with foundation models: An empirical study of latent replay, in: Conference on lifelong learning agents, PMLR, 2022, pp. 60–91

    2022

  69. [77]

    Houlsby, A

    N. Houlsby, A. Giurgiu, S. Jastrzebski, B. Morrone, Q. de Laroussilhe, A. Gesmundo, M. Attariyan, S. Gelly, Parameter-efficient transfer learning for nlp, 2019. URL: https://arxiv.org/abs/1902.00751. arXiv:1902.00751

    2019 arXiv

  70. [78]

    E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, W. Chen, Lora: Low-rank adapta- tion of large language models, 2021. URL: https://arxiv.org/abs/2106.09685. arXiv:2106.09685

    2021 arXiv

  71. [79]

    Aggarwal, S

    D. Aggarwal, S. Damle, N. Goyal, S. Lokam, S. Sitaram, Exploring continual fine-tuning for enhancing language ability in large language model, 2024. URL: https://arxiv.org/abs/2410.16006. arXiv:2410.16006

    2024 arXiv

  72. [80]

    Beaulieu, L

    S. Beaulieu, L. Frati, T. Miconi, J. Lehman, K. O. Stanley, J. Clune, N. Cheney, Learning to continually learn, 2020. URL: https://arxiv.org/abs/2002.09571. arXiv:2002.09571

    2020 arXiv

  73. [81]

    Zhang, L

    X. Zhang, L. Bai, X. Yang, J. Liang, C-lora: Continual low-rank adaptation for pre-trained models,

  74. [82]

    T. Fan, Y. Kang, G. Ma, W. Chen, W. Wei, L. Fan, Q. Yang, Fate-llm: A industrial grade federated learning framework for large language models, 2023. URL: https://arxiv.org/abs/2310.10049. arXiv:2310.10049

    2023 arXiv

  75. [83]

    J. Liu, J. Ren, R. Jin, Z. Zhang, Y. Zhou, P. Valduriez, D. Dou, Fisher information-based efficient curriculum federated learning with large language models, 2024. URL: https://arxiv.org/abs/2410. 00131. arXiv:2410.00131

    2024 arXiv

  76. [84]

    Soutif-Cormerais, A

    A. Soutif-Cormerais, A. Carta, A. Cossu, J. Hurtado, V. Lomonaco, J. Van de Weijer, H. Hemati, A comprehensive empirical evaluation on online continual learning, in: Proceedings of the IEEE/CVF International Conference on Computer Vision, 2023, pp. 3518–3528

    2023

  77. [85]

    G. I. Parisi, R. Kemker, J. L. Part, C. Kanan, S. Wermter, Continual lifelong learning with neural networks: A review, Neural Netw. 113 (2019) 54–71. URL: https://doi.org/10.1016/j.neunet.2019. 01.012. doi:10.1016/j.neunet.2019.01.012

    2019 doi

  78. [86]

    H. Du, S. Liu, L. Zheng, Y. Cao, A. Nakamura, L. Chen, Privacy in fine-tuning large language models: Attacks, defenses, and future directions, 2025. URL: https://arxiv.org/abs/2412.16504. arXiv:2412.16504

    2025 arXiv

  79. [87]

    Z. Wang, Z. Zhang, C.-Y. Lee, H. Zhang, R. Sun, X. Ren, G. Su, V. Perot, J. Dy, T. Pfister, Learning to prompt for continual learning, 2022. URL: https://arxiv.org/abs/2112.08654.arXiv:2112.08654

    2022 arXiv

  80. [88]

    J. S. Smith, L. Karlinsky, V. Gutta, P. Cascante-Bonilla, D. Kim, A. Arbelle, R. Panda, R. Feris, Z. Kira, Coda-prompt: Continual decomposed attention-based prompting for rehearsal-free continual learning, 2023. URL: https://arxiv.org/abs/2211.13218. arXiv:2211.13218

    2023 arXiv

  81. [89]

    H. Chen, J. Li, N. Gazagnadou, W. Zhuang, C. Chen, L. Lyu, Dual low-rank adaptation for continual learning with pre-trained models, 2024. URL: https://arxiv.org/abs/2411.00623. arXiv:2411.00623

    2024

  82. [90]

    Zhang, Y

    W. Zhang, Y. Huang, T. Zhang, Q. Zou, W.-S. Zheng, R. Wang, Adapter learning in pretrained feature extractor for continual learning of diseases, 2023. URL: https://arxiv.org/abs/2304.09042. arXiv:2304.09042

    2023 arXiv

  83. [91]

    X. Gao, S. Dong, Y. He, Q. Wang, Y. Gong, Beyond prompt learning: Continual adapter for efficient rehearsal-free continual learning, 2024. URL: https://arxiv.org/abs/2407.10281. arXiv:2407.10281

    2024 arXiv

  84. [92]

    Yadav, D

    P. Yadav, D. Tam, L. Choshen, C. Raffel, M. Bansal, Ties-merging: Resolving interference when merging models, 2023. URL: https://arxiv.org/abs/2306.01708. arXiv:2306.01708

    2023 arXiv

  85. [93]

    L. Yu, B. Yu, H. Yu, F. Huang, Y. Li, Language models are super mario: Absorbing abili- ties from homologous models as a free lunch, 2024. URL: https://arxiv.org/abs/2311.03099. arXiv:2311.03099

    2024 arXiv

  86. [94]

    Marczak, B

    D. Marczak, B. Twardowski, T. Trzciński, S. Cygert, Magmax: Leveraging model merging for seamless continual learning, 2024. URL: https://arxiv.org/abs/2407.06322. arXiv:2407.06322

    2024 arXiv

  87. [95]

    E. Yang, L. Shen, Z. Wang, G. Guo, X. Chen, X. Wang, D. Tao, Representation surgery for multi-task model merging, 2024. URL: https://arxiv.org/abs/2402.02705. arXiv:2402.02705

    2024 arXiv

  88. [96]

    E. N. Coleman, L. Quarantiello, J. Hurtado, V. Lomonaco, Adaptive LoRA merging for efficient domain incremental learning, in: Adaptive Foundation Models: Evolving AI for Personalized and Efficient Learning, 2024. URL: https://openreview.net/forum?id=tlB5eonGEk

    2024

  89. [97]

    C. Finn, P. Abbeel, S. Levine, Model-agnostic meta-learning for fast adaptation of deep networks,

  90. [98]

    Zhang, R

    R. Zhang, R. Qiang, S. A. Somayajula, P. Xie, Autolora: Automatically tuning matrix ranks in low-rank adaptation based on meta learning, 2024. URL: https://arxiv.org/abs/2403.09113. arXiv:2403.09113

    2024 arXiv

  91. [99]

    W. Li, L. Zou, M. Tang, Q. Yu, W. Li, C. Li, META-LORA: Memory-efficient sample reweighting for fine-tuning large language models, in: O. Rambow, L. Wanner, M. Apidianaki, H. Al-Khalifa, B. D. Eugenio, S. Schockaert (Eds.), Proceedings of the 31st International Conference on C...

    2025

  92. [100]

    Palazzo, M

    L. Palazzo, M. Pennisi, F. P. Salanitri, G. Bellitto, S. Palazzo, C. Spampinato, Fedrewind: Rewinding continual model exchange for decentralized federated learning, 2024. URL: https://arxiv.org/abs/ 2411.09842. arXiv:2411.09842

    2024 arXiv

  93. [101]

    Chollet, Abstraction and Reasoning Corpus for Artificial General Intelligence (ARC-AGI), https://github.com/fchollet/ARC-AGI, 2019

    F. Chollet, Abstraction and Reasoning Corpus for Artificial General Intelligence (ARC-AGI), https://github.com/fchollet/ARC-AGI, 2019. URL: https://github.com/fchollet/ARC-AGI

    2019

  94. [102]

    Srivastava, A

    A. Srivastava, A. Rastogi, A. Rao, A. A. M. Shoeb, A. Abid, A. Fisch, A. R. Brown, A. Santoro, A. Gupta, A. Garriga-Alonso, et al., Beyond the imitation game: Quantifying and extrapolating the capabilities of language models, arXiv preprint arXiv:2206.04615 (2022)

    2022 arXiv

  95. [103]

    Luo, Has llm reached the scaling ceiling yet? unified insights into llm regularities and constraints, arXiv preprint arXiv:2412.16443 (2024)

    C. Luo, Has llm reached the scaling ceiling yet? unified insights into llm regularities and constraints, arXiv preprint arXiv:2412.16443 (2024)

    2024 arXiv

  96. [104]

    K.-T. Tran, D. Dao, M.-D. Nguyen, Q.-V. Pham, B. O’Sullivan, H. D. Nguyen, Multi-agent collabo- ration mechanisms: A survey of llms, arXiv preprint arXiv:2501.06322 (2025)

    2025 arXiv

  97. [105]

    L. Wang, C. Ma, X. Feng, Z. Zhang, H. Yang, J. Zhang, Z. Chen, J. Tang, X. Chen, Y. Lin, et al., A survey on large language model based autonomous agents, Frontiers of Computer Science 18 (2024) 186345

    2024

  98. [106]

    B. Liu, X. Li, J. Zhang, J. Wang, T. He, S. Hong, H. Liu, S. Zhang, K. Song, K. Zhu, Y. Cheng, S. Wang, X. Wang, Y. Luo, H. Jin, P. Zhang, O. Liu, J. Chen, H. Zhang, Z. Yu, H. Shi, B. Li, D. Wu, F. Teng, X. Jia, J. Xu, J. Xiang, Y. Lin, T. Liu, T. Liu, Y. Su, H. Sun, G. Berset...

    2025 arXiv

  99. [107]

    J. Wei, X. Wang, D. Schuurmans, M. Bosma, F. Xia, E. Chi, Q. V. Le, D. Zhou, et al., Chain-of- thought prompting elicits reasoning in large language models, Advances in neural information processing systems 35 (2022) 24824–24837

    2022

  100. [108]

    Z. Xi, W. Chen, X. Guo, W. He, Y. Ding, B. Hong, M. Zhang, J. Wang, S. Jin, E. Zhou, et al., The rise and potential of large language model based agents: A survey, Science China Information Sciences 68 (2025) 121101

    2025

  101. [109]

    ichter, A

    b. ichter, A. Brohan, Y. Chebotar, C. Finn, K. Hausman, A. Herzog, D. Ho, J. Ibarz, A. Irpan, E. Jang, R. Julian, D. Kalashnikov, S. Levine, Y. Lu, C. Parada, K. Rao, P. Sermanet, A. T. Toshev, V. Vanhoucke, F. Xia, T. Xiao, P. Xu, M. Yan, N. Brown, M. Ahn, O. Cortes, N. Sieve...

    2023

  102. [110]

    B. Xu, Z. Peng, B. Lei, S. Mukherjee, Y. Liu, D. Xu, Rewoo: Decoupling reasoning from observations for efficient augmented language models, CoRR abs/2305.18323 (2023). URL: https://doi.org/10. 48550/arXiv.2305.18323

    2023 doi arXiv

  103. [111]

    S. S. Raman, V. Cohen, E. Rosen, I. Idrees, D. Paulius, S. Tellex, Planning with large language models via corrective re-prompting, in: NeurIPS 2022 Foundation Models for Decision Making Workshop, 2022. URL: https://openreview.net/forum?id=cMDMRBe1TKs

    2022

  104. [112]

    S. Yao, D. Yu, J. Zhao, I. Shafran, T. Griffiths, Y. Cao, K. Narasimhan, Tree of thoughts: Deliberate problem solving with large language models, Advances in neural information processing systems 36 (2023) 11809–11822

    2023

  105. [113]

    N. Du, Y. Huang, A. M. Dai, S. Tong, D. Lepikhin, Y. Xu, M. Krikun, Y. Zhou, A. W. Yu, O. Firat, et al., Glam: Efficient scaling of language models with mixture-of-experts, in: International conference on machine learning, PMLR, 2022, pp. 5547–5569

    2022

  106. [114]

    B. Lin, Z. Tang, Y. Ye, J. Cui, B. Zhu, P. Jin, J. Huang, J. Zhang, Y. Pang, M. Ning, et al., Moe-llava: Mixture of experts for large vision-language models, arXiv preprint arXiv:2401.15947 (2024)

    2024 arXiv

  107. [115]

    F. Xue, Z. Zheng, Y. Fu, J. Ni, Z. Zheng, W. Zhou, Y. You, Openmoe: An early effort on open mixture-of-experts language models, arXiv preprint arXiv:2402.01739 (2024)

    2024 arXiv

  108. [116]

    D. Dai, C. Deng, C. Zhao, R. Xu, H. Gao, D. Chen, J. Li, W. Zeng, X. Yu, Y. Wu, et al., Deepseekmoe: Towards ultimate expert specialization in mixture-of-experts language models, arXiv preprint arXiv:2401.06066 (2024)

    2024 arXiv

  109. [117]

    K. A. Fischer, Reflective linguistic programming (rlp): A stepping stone in socially-aware agi (socialagi), arXiv preprint arXiv:2305.12647 (2023)

    2023 arXiv

  110. [118]

    C. Li, J. Wang, K. Zhu, Y. Zhang, W. Hou, J. Lian, X. Xie, Emotionprompt: Leveraging psychology for large language models enhancement via emotional stimulus, arXiv preprint arXiv:2307.11760 (2023)

    2023 arXiv

  111. [119]

    X. Wang, S. Zhang, S. Li, K. Li, K. Kallidromitis, Y. Kato, K. Kozuka, T. Darrell, Segllm: Multi-round reasoning segmentation with large language models, in: The Thirteenth International Conference on Learning Representations, ????

  112. [120]

    Z. Zhou, X. Zhang, S. Tan, L. Zhang, C. Li, Collaborative evolution: Multi-round learning between large and small language models for emergent fake news detection, in: Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, 2025, pp. 1210–1218

    2025

  113. [121]

    Q. Wu, G. Bansal, J. Zhang, Y. Wu, B. Li, E. Zhu, L. Jiang, X. Zhang, S. Zhang, J. Liu, et al., Autogen: Enabling next-gen llm applications via multi-agent conversation, arXiv preprint arXiv:2308.08155 (2023)

    2023 arXiv

  114. [122]

    G. Wang, Y. Xie, Y. Jiang, A. Mandlekar, C. Xiao, Y. Zhu, L. Fan, A. Anandkumar, Voyager: An open-ended embodied agent with large language models, Transactions on Machine Learning Research (2024). URL: https://openreview.net/forum?id=ehfRiF0R3a

    2024

  115. [123]

    X. Zhu, Y. Chen, H. Tian, C. Tao, W. Su, C. Yang, G. Huang, B. Li, L. Lu, X. Wang, Y. Qiao, Z. Zhang, J. Dai, Ghost in the minecraft: Generally capable agents for open-world environments via large language models with text-based knowledge and memory, 2023. URL: https://arxiv.o...

    2023 arXiv

  116. [124]

    Zhang, Z

    C. Zhang, Z. Yang, J. Liu, Y. Han, X. Chen, Z. Huang, B. Fu, G. Yu, Appagent: Multimodal agents as smartphone users, arXiv preprint arXiv:2312.13771 (2023)

    2023 arXiv

  117. [125]

    Madaan, N

    A. Madaan, N. Tandon, P. Clark, Y. Yang, Memory-assisted prompt editing to improve gpt-3 after deployment, in: Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, 2022, pp. 2833–2861

    2022

  118. [126]

    Sumers, S

    T. Sumers, S. Yao, K. Narasimhan, T. Griffiths, Cognitive architectures for language agents, Transactions on Machine Learning Research (2024). URL: https://openreview.net/forum?id= 1i6ZCvflQJ, survey Certification

    2024

  119. [127]

    G. Li, H. Hammoud, H. Itani, D. Khizbullin, B. Ghanem, Camel: Communicative agents for" mind" exploration of large language model society, Advances in Neural Information Processing Systems 36 (2023) 51991–52008

    2023

  120. [128]

    H. Liu, Y. Zhou, B. Liu, J. Zhao, R. Yao, Z. Shao, Incremental learning with neural networks for computer vision: a survey, Artificial intelligence review 56 (2023) 4557–4589

    2023

  121. [129]

    Khetarpal, M

    K. Khetarpal, M. Riemer, I. Rish, D. Precup, Towards continual reinforcement learning: A review and perspectives, Journal of Artificial Intelligence Research 75 (2022) 1401–1476

    2022

  122. [130]

    J. Gou, B. Yu, S. J. Maybank, D. Tao, Knowledge distillation: A survey, International Journal of Computer Vision 129 (2021) 1789–1819

    2021

  123. [131]

    C. Yang, Y. Zhu, W. Lu, Y. Wang, Q. Chen, C. Gao, B. Yan, Y. Chen, Survey on knowledge distillation for large language models: methods, evaluation, and application, ACM Transactions on Intelligent Systems and Technology (2024)

    2024

  124. [132]

    Carta, A

    A. Carta, A. Cossu, V. Lomonaco, D. Bacciu, Ex-model: Continual learning from a stream of trained models, in: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2022, pp. 3790–3799

    2022

  125. [133]

    Kenton, T

    Z. Kenton, T. Everitt, L. Weidinger, I. Gabriel, V. Mikulik, G. Irving, Alignment of language agents, arXiv preprint arXiv:2103.14659 (2021)

    2021 arXiv

  126. [134]

    W. Du, Z. M. Kim, V. Raheja, D. Kumar, D. Kang, Read, revise, repeat: A system demonstration for human-in-the-loop iterative text revision, in: Proceedings of the First Workshop on Intelligent and Interactive Writing Assistants (In2Writing 2022), 2022, pp. 96–108

    2022

  127. [2017]

    arXiv:1703.03400

    URL: https://arxiv.org/abs/1703.03400. arXiv:1703.03400

    arXiv

  128. [2025]

    arXiv:2502.17920

    URL: https://arxiv.org/abs/2502.17920. arXiv:2502.17920

    arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.