Pith. sign in

REVIEW 3 major objections 7 minor 69 references

Task-Core Memory Management and Consolidation for Long-term Continual Learning

T0 review · 3 major / 7 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Long-CL nearly halts catastrophic forgetting across 21 and 30 sequential task benchmarks.

desk verdict A useful long-horizon CL benchmark pair and a plausible method, but two load-bearing ambiguities (replay-buffer persistence, Eq. 5) and missing code/data mean the paper needs revision before the numbers can be trusted. read the letter →

arxiv 2505.09952 v1 pith:UAA7IU2R submitted 2025-05-15 cs.LG cs.AI

Tianyu Huai , Jie Zhou , Yuxuan Cai , Qin Chen , Wen Wu , Xingjiao Wu , Xipeng Qiu , Liang He This is my paper · ORCID
classification cs.LGcs.AI
keywords continuallearningcatastrophicforgettinglargelanguagemodelsvision-languageLoRAexperiencereplaymemoryconsolidationlong-termbenchmark
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper introduces long-term continual learning—learning from a stream of dozens of heterogeneous tasks—and argues that existing continual learning methods collapse under this load. To fix it, the paper proposes Long-CL, a two-part memory framework: MemMan protects task-critical LoRA parameters by masking and adaptively fusing old and new weights, while MemCon replays a curated mix of hard and cross-task-aligned samples. The central claim is that Long-CL achieves state-of-the-art average performance (AP) of 51.93 on the new 21-task multimodal benchmark and 60.12 on the new 30-task text benchmark, beating the previous best method by 7.4 and 6.5 AP. At a 20% replay buffer it reaches 94.5% of the multitask upper bound. The upshot is a parameter-mask plus selective-replay recipe for letting 7B-scale generative models keep learning over long streams without task identity at inference time.

What carries the argument

The machinery is a two-part memory loop around LoRA adapters. Task-Core Memory Management (MemMan) first computes the per-unit Euclidean drift between the model before and after tuning on the current task, records the top-K drifted units in a cumulative mask $\mathbf{Mask}_t$, and then fuses old and new LoRA weights element-wise as $\theta_t = \beta_t \odot \phi_t + (1-\beta_t) \odot \theta_{t-1}$ with $\beta_t = \alpha_t \mathbf{Mask}_t + (1-\alpha_t)(1-\mathbf{Mask}_t)$. The scalar $\alpha_t$ is set by the distance of the current task's prototype to previous prototypes, so early novel tasks get larger updates and later tasks become more conservative. Long-term Memory Consolidation (MemCon) builds a replay set $R_t = H_t \cup G_t$: $H_t$ holds the hardest samples (largest distance to the current prototype) and $G_t$ holds differential samples (smallest cumulative distance to previous prototypes but not too close to any one of them). These replayed samples are provided during current-task training, which is what lets the model refresh old knowledge while learning new tasks.

What would settle it

Run Long-CL on MMLongCL-Bench with the replay buffer reset to each task's own hard and differential samples, so no samples from earlier tasks are replayed; if AP remains at 51.93, the selective replay of old samples is not what drives the reported retention.

Watch

Extended reading notes

Core claim

Long-CL establishes that catastrophic forgetting over long task streams can be largely controlled by protecting a small set of task-critical parameters and replaying a carefully chosen subset of old samples. MemMan computes the per-unit drift between the LoRA model before and after tuning on the current task, records the top-K drifted units in a cumulative mask, and fuses old and new weights with a prototype-derived weight. MemCon selects hard samples (far from the current task prototype) and differential samples (close to previous task prototypes) and uses them as replay data. On MMLongCL-Bench this reaches 51.93 AP with -9.93 average forgetting, and on TextLongCL-Bench it reaches 60.12 AP with -0.89 average forgetting. With a 20% replay buffer, Long-CL reaches 94.5% of the multitask upper bound.

Load-bearing premise

The reported retention depends on the replay buffer accumulating samples across tasks: samples chosen for early tasks must still be replayed while later tasks train, but the paper defines R_t only for the current task and never states that earlier selections are carried forward.

Editorial extensions

If this is right

  • At a 20% replay buffer, Long-CL reaches 94.5% of the multitask-learning upper bound on MMLongCL-Bench, so the memory cost can be modest while keeping most of the benefit.
  • The reported negative average forgetting (-9.93 on MMLongCL-Bench, -0.89 on TextLongCL-Bench) means earlier tasks end up scoring higher after later training, a backward-transfer effect rather than mere retention.
  • Because inference requires no task identifier, Long-CL applies to streams where tasks arrive unlabeled and may overlap in unknown ways.
  • The two released benchmarks, with 21 vision-language and 30 text-only datasets, give later work a common yardstick for long-horizon continual learning, where prior benchmarks covered at most a handful of tasks.
  • Long-CL outperforms CL-MoE across three random task orders, indicating the method is not tuned to one particular sequence.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Not tested in the paper: the same drift-mask and prototype-distance machinery should transfer to other parameter-efficient adapters (adapters, soft prompts), since nothing in the fusion equations is LoRA-specific; running that variant would test the mechanism's generality.
  • Not tested in the paper: the ablation does not isolate cross-task replay from within-task replay; a version that resets the buffer after each task would separate how much of the gain comes from accumulated old-task samples versus the selection rule alone.
  • Not tested in the paper: because $\alpha_t$ decays with the number of accumulated tasks, the framework predicts memory updates become more conservative as the stream grows; one could test whether the optimal schedule depends mainly on task count or on true task similarity.
  • Not tested in the paper: the negative average forgetting values suggest the fused parameter trajectory keeps improving old tasks during later training; if that holds, Long-CL behaves like progressive learning rather than merely protecting an old snapshot.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 7 minor

Summary. This paper addresses long-term continual learning (CL) for large language and vision-language models, where a model must learn sequentially from a large number of heterogeneous tasks without catastrophic forgetting. The authors propose Long-CL, consisting of two components: MemMan, which identifies task-critical parameter updates via discrepancy estimation (Eq. 2) and fuses old and new LoRA adapters with an adaptive interpolation weight α_t (Eqs. 5–7); and MemCon, which selects hard samples and differential samples for a replay buffer (Eqs. 8–9). The paper also introduces two benchmarks, MMLongCL-Bench (21 multimodal datasets, 503k training samples) and TextLongCL-Bench (30 textual datasets, 397k samples), and reports experiments with LLaVA-7B and Qwen2.5-7B. The main results claim state-of-the-art final average performance of 51.93 AP on MMLongCL-Bench and 60.12 AP on TextLongCL-Bench, with corresponding average forgetting values of −9.93 and −0.89, approaching the multitask upper bounds.

Significance. The paper targets a relevant and underexplored setting: continual learning over dozens of heterogeneous tasks with multi-billion-parameter generative models. The proposed combination of parameter-space masking (MemMan) and selective replay (MemCon) is plausible, and the empirical gains over standard baselines (EWC, ER, LWF, CL-MoE, etc.) are substantial. The construction and planned release of the two benchmarks is in itself a useful community contribution, and the paper includes ablations (Table 3), sensitivity analysis (Figure 4, Table 5), and robustness to task order (Table 4). If the mechanism and results hold up, the work would provide a practical recipe for reducing catastrophic forgetting in long-horizon instruction tuning.

major comments (3)
  1. [§4.2, Eq. (5)] The adaptive weight α_t is undefined at t=2 because the denominator ∑_{1≤i<j≤t−1} ||A_i − A_j||_2 is an empty sum, evaluating to 0. Since the continual learning procedure starts at task 1 and needs an interpolation weight for every task, the update in Eq. (7) cannot be executed for the second task as written. The lower bound α_min does not resolve this, as it is a one-sided bound and α_t would be infinite rather than merely small. Please specify the initialization for t=1 and t=2 (e.g., set α_2=1 or take a limit), and ensure the formula is well-defined for all t.
  2. [§4.3 and §5.4] The paper never states whether the replay buffer R_t is accumulated across tasks. Section 4.3 defines R_t = H_t ∪ G_t for the current task only, and the implementation details mention a buffer size of 20% of the training dataset without specifying whether the buffer persists. Figure 3 shows 'Task t-1 Buffer' and 'Append', which suggests persistence, but the text is ambiguous. This ambiguity is load-bearing: the ablation in Table 3 shows MemCon alone yields AP 39.94, and the paper's central claim that selective replay of old examples largely eliminates forgetting depends on old task samples being replayed during later tasks. If the buffer is reset each task, MemCon does not replay any old task data, and the reported retention must be explained by MemMan alone, which the ablation does not support. The authors must state explicitly how buffers persist and how the 20% budget is shared between the current and historical tasks, and adjust the text or the method accordingly.
  3. [§5.1, Tables 2–5] All reported results appear to be from a single run; no standard deviations, confidence intervals, or seed information are provided. Table 4 shows performance across three task orders but without variance estimates, and the sensitivity analyses in Figure 4 and Table 5 also lack error bars. The headline claims of state-of-the-art performance and of reaching 94.5% of multitask performance are quantitative comparative statements, so without a measure of run-to-run variability it is impossible to judge whether the reported gaps (e.g., Long-CL 45.74 vs. EWC 37.40 on MMLongCL-Bench) are statistically meaningful. Please report means and standard deviations over at least three random seeds, or otherwise justify that the evaluation is deterministic.
minor comments (7)
  1. [§4.3, Eq. (9)] The notation Min_{R_g} in Eq. (9) is unclear; R_g is later stated as a percentage, but the set construction with a threshold δ needs a precise definition of how many samples are selected and how ties are broken.
  2. [§5.4] The phrase 'When Rt is set to 20%' conflates the buffer set R_t with its size; please clarify that R_t denotes the buffer and 20% refers to its budget relative to the training set.
  3. [§5.1] The hyperparameter δ is defined as 0.8*Dmax / 2, but Dmax is not defined; specify how the maximum distance of prototypes is computed.
  4. [Table 2] The column header order is confusing because the TextLongCL-Bench AP/AF columns appear among the category columns; reformat the table so that the benchmark-level metrics are clearly separated from category-level accuracies.
  5. [Abstract, §5.2] The abstract claims the method 'outperforms the previous state-of-the-art by 7.4% and 6.5% AP', but the percentage-point calculation in Section 5.2 (36.58% and 15.92%) refers to improvements over Vanilla, not over the previous SOTA; please make the comparison basis explicit.
  6. [§1] The paper states that the benchmarks are released, but no download URL or repository is provided; please include the public access information.
  7. [§5.3] There are several typographical issues, such as 'Long—CL' in the heading of Section 5.3 and inconsistent spacing in Table 2; please proofread the manuscript.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity; results are measured on external benchmarks and no fitted quantity is relabeled as a prediction.

full rationale

The paper's headline numbers (AP 51.93 and 60.12, AF -9.93 and -0.89) are direct measurements on MMLongCL-Bench and TextLongCL-Bench, not outputs of a fitted regression or of equations that take the claimed result as input. MemMan's interpolation (Eq. 7) and MemCon's replay selection (Eqs. 8-9) combine parameters and samples but do not encode the benchmark accuracies; the ablation in Table 3 is an independent attribution-style comparison. Hyperparameters (K=10%, alpha_min=0.3, Rg=Rh=10%, delta=0.8*Dmax/2) are stated constants; even if tuned on the same benchmarks, that is benchmark overfitting rather than constructional circularity, and no quantity fitted to a subset is renamed as a prediction of the held-out result. The only self-citations (CL-MoE [6] and the authors' survey [4]) are used as related work and as a baseline; neither is invoked to justify the method's correctness or to forbid alternatives. No uniqueness theorem, ansatz imported by citation, or renaming of a known result is present. There is a genuine definitional bug in Eq. (5) at t=2 (empty denominator), and the persistence of the replay buffer across tasks is underspecified, but both are correctness and completeness concerns, not circular reductions.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

Long-CL depends on several hand-selected proportions and thresholds that are not derived from data or theory, and on domain assumptions about Euclidean distances in parameter and feature spaces. No new physical or mathematical entities are introduced; 'task-core memory' and 'long-term memory' are conceptual metaphors.

free parameters (5)
  • K = 10%
    Proportion of LoRA units selected as task-core in Eq. (2); the paper sets K=10% and shows a sensitivity curve in Figure 4.
  • alpha_min = 0.3
    Lower bound on the adaptive weight alpha_t in Eq. (7); chosen by hand as lambda in Section 5.1.
  • R_h = 10%
    Percentage of current-task samples selected as hard samples in Eq. (8).
  • R_g = 10%
    Percentage of current-task samples selected as differential samples in Eq. (9).
  • delta = 0.8 * Dmax / 2
    Distance threshold in Eq. (9) to discard samples too close to a single prior prototype; Dmax is the maximum prototype distance.
assumptions (5)
  • domain assumption Euclidean distance between LoRA parameters before and after a task identifies the memory units most critical to that task.
    Used in Eq. (2) to select top-K indices; if parameter shift magnitude does not track functional importance, the mask in Eq. (3) loses its justification.
  • domain assumption Mean-pooling encoder features over all samples produces a prototype that captures the task's semantic identity.
    Eq. (4) defines A_t; alpha_t in Eq. (5) and the sample selection in Eq. (8)-(9) all assume these prototypes are meaningful in Euclidean space.
  • ad hoc to paper Linear element-wise interpolation between LoRA adapters (Eq. (7)) preserves both old and new task knowledge without destructive interference.
    The central update combines theta_{t-1} and phi_t with weights beta_t; this is a modeling choice with no proof of stability over 21-30 tasks.
  • ad hoc to paper The adaptive weight alpha_t in Eq. (5) is well-defined for all tasks; the t=2 case, where the denominator is an empty sum, is not addressed.
    For t=2 the sum over pairs 1<=i<j<=1 is zero, so alpha_2 is undefined. The paper never states a special-case rule, yet the method is applied at every task transition.
  • ad hoc to paper Persistent replay of previously selected samples is implicit in the MemCon design; the paper defines R_t as current-task selections only.
    Section 4.3 defines R_t = H_t union G_t without stating that previous R_j are retained; if the buffer is not persistent, MemCon cannot replay old task data.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Task-Core Memory Management and Consolidation for Long-term Continual Learning." pith.science (2026). https://pith.science/paper/UAA7IU2R

@misc{pith2026250509952,
  author       = {Pith},
  title        = {Pith review of: Task-Core Memory Management and Consolidation for Long-term Continual Learning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/UAA7IU2R}},
  note         = {Machine review of arXiv:2505.09952}
}
read the original abstract

In this paper, we focus on a long-term continual learning (CL) task, where a model learns sequentially from a stream of vast tasks over time, acquiring new knowledge while retaining previously learned information in a manner akin to human learning. Unlike traditional CL settings, long-term CL involves handling a significantly larger number of tasks, which exacerbates the issue of catastrophic forgetting. Our work seeks to address two critical questions: 1) How do existing CL methods perform in the context of long-term CL? and 2) How can we mitigate the catastrophic forgetting that arises from prolonged sequential updates? To tackle these challenges, we propose a novel framework inspired by human memory mechanisms for long-term continual learning (Long-CL). Specifically, we introduce a task-core memory management strategy to efficiently index crucial memories and adaptively update them as learning progresses. Additionally, we develop a long-term memory consolidation mechanism that selectively retains hard and discriminative samples, ensuring robust knowledge retention. To facilitate research in this area, we construct and release two multi-modal and textual benchmarks, MMLongCL-Bench and TextLongCL-Bench, providing a valuable resource for evaluating long-term CL approaches. Experimental results show that Long-CL outperforms the previous state-of-the-art by 7.4\% and 6.5\% AP on the two benchmarks, respectively, demonstrating the effectiveness of our approach.

Figures

Figures reproduced from arXiv: 2505.09952 by the authors.

Figure 1
Figure 1. The performance of our method and O￾LoRA on MMLongCL-Bench. Although previous studies have proven effec￾tive, most existing methods perform CL un￾der a few task types with relatively short data streams. In contrast, real-world scenarios al￾ways require long-term CL, which requires mod￾els to continually learning across heterogeneous task types and non-stationary, long-term data streams. There are two primary challen… view at source ↗
Figure 2
Figure 2. The performance heatmap of our method and O-LoRA on [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. The framework of our Long-CL. First, we introduce MemMan to index the task-core memory and adjust the model’s memory adaptively based on the relationship between the current and previous tasks. Then, MemCon learns the key knowledge using hard sample selection and differential sample selection to choose task-relevant informative (i.e., the samples in the green annulus) and cross-task generalizable samples (i.e., the … view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Performance(%) of Long-CL with different buffer size and K value on MMLongCL-Bench. 5.3 Ablation Study Method MMLongCL-Bench AP AF MemMan MemCon VQA VE MSA TR a ✗ ✗ 12.34 27.92 0.00 0.30 15.35 28.42 b ✓ ✗ 31.01 47.74 48.85 12.70 39.91 2.31 c ✗ ✓ 39.94 50.09 58.57 24.80…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

69 extracted references · 20 canonical work pages

  1. [1]

    Continual lifelong learning with neural networks: A review.Neural networks, 113:54–71, 2019

    German I Parisi, Ronald Kemker, Jose L Part, Christopher Kanan, and Stefan Wermter. Continual lifelong learning with neural networks: A review.Neural networks, 113:54–71, 2019

    2019

  2. [2]

    Catastrophic forgetting in connectionist networks.Trends in cognitive sciences, 3(4): 128–135, 1999

    Robert M French. Catastrophic forgetting in connectionist networks.Trends in cognitive sciences, 3(4): 128–135, 1999

    1999

  3. [3]

    Overcoming catastrophic forgetting in neural networks.Proceedings of the national academy of sciences, 114(13):3521–3526, 2017

    James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, et al. Overcoming catastrophic forgetting in neural networks.Proceedings of the national academy of sciences, 114(13):3521–3526, 2017

    2017

  4. [4]

    Recent advances of foundation language models-based continual learning: A survey.ACM Computing Surveys, 57(5):1–38, 2025

    Yutao Yang, Jie Zhou, Xuanwen Ding, Tianyu Huai, Shunyu Liu, Qin Chen, Yuan Xie, and Liang He. Recent advances of foundation language models-based continual learning: A survey.ACM Computing Surveys, 57(5):1–38, 2025

    2025

  5. [5]

    Orthogonal subspace learning for language model continual learning

    Xiao Wang, Tianze Chen, Qiming Ge, Han Xia, Rong Bao, Rui Zheng, Qi Zhang, Tao Gui, and Xuanjing Huang. Orthogonal subspace learning for language model continual learning. InThe 2023 Conference on Empirical Methods in Natural Language Processing

    work page 2023

  6. [6]

    Cl-moe: Enhancing multimodal large language model with dual momentum mixture-of-experts for continual visual question answering.arXiv preprint arXiv:2503.00413, 2025

    Tianyu Huai, Jie Zhou, Xingjiao Wu, Qin Chen, Qingchun Bai, Ze Zhou, and Liang He. Cl-moe: Enhancing multimodal large language model with dual momentum mixture-of-experts for continual visual question answering.arXiv preprint arXiv:2503.00413, 2025

    arXiv 2025

  7. [7]

    Sentence embedding alignment for lifelong relation extraction

    Hong Wang, Wenhan Xiong, Mo Yu, Xiaoxiao Guo, Shiyu Chang, and William Yang Wang. Sentence embedding alignment for lifelong relation extraction. InAnnual Conference of the North American Chapter of the Association for Computational Linguistics. Association for Computational Linguistics (ACL), 2019

    work page 2019

  8. [8]

    Learning to prompt for continual learning

    Zifeng Wang, Zizhao Zhang, Chen-Yu Lee, Han Zhang, Ruoxi Sun, Xiaoqi Ren, Guolong Su, Vincent Perot, Jennifer Dy, and Tomas Pfister. Learning to prompt for continual learning. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 139–149, 2022

    2022

Show all 69 references
  1. [9]

    Bert: Pre-training of deep bidi- rectional transformers for language understanding

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidi- rectional transformers for language understanding. InProceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human lang...

    2019

  2. [10]

    Language models are few-shot learners

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901, 2020

    1901

  3. [11]

    Exploring the limits of transfer learning with a unified text-to-text transformer

    Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21(140):1–67, 2020. 10

    2020

  4. [12]

    Ernie 3.0: Large-scale knowledge enhanced pre-training for language understanding and generation.arXiv preprint arXiv:2107.02137, 2021

    Yu Sun, Shuohuan Wang, Shikun Feng, Siyu Ding, Chao Pang, Junyuan Shang, Jiaxiang Liu, Xuyi Chen, Yanbin Zhao, Yuxiang Lu, et al. Ernie 3.0: Large-scale knowledge enhanced pre-training for language understanding and generation.arXiv preprint arXiv:2107.02137, 2021

    2021 arXiv

  5. [13]

    Bloom: A 176b-parameter open-access multilingual language model

    Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ili ´c, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, Matthias Gallé, et al. Bloom: A 176b-parameter open-access multilingual language model. 2023

    2023

  6. [14]

    Gpt-4 technical report.arXiv preprint arXiv:2303.08774, 2023

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report.arXiv preprint arXiv:2303.08774, 2023

    2023 arXiv

  7. [15]

    Palm: Scaling language modeling with pathways.Journal of Machine Learning Research, 24(240):1–113, 2023

    Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways.Journal of Machine Learning Research, 24(240):1–113, 2023

    2023

  8. [16]

    Llama: Open and efficient foundation language models.arXiv preprint arXiv:2302.13971, 2023

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models.arXiv preprint arXiv:2302.13971, 2023

    2023 arXiv

  9. [18]

    The llama 3 herd of models

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al- Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024

    2024 arXiv

  10. [19]

    Ocean-omni: To understand the world with omni-modality.arXiv preprint arXiv:2410.08565, 2024

    Yadong Li, Haoze Sun, Mingan Lin, Tianpeng Li, Guosheng Dong, Tao Zhang, Bowen Ding, Wei Song, Zhenglin Cheng, Yuqi Huo, et al. Ocean-omni: To understand the world with omni-modality.arXiv preprint arXiv:2410.08565, 2024

    2024

  11. [20]

    Molmo and pixmo: Open weights and open data for state-of-the-art multimodal models.arXiv preprint arXiv:2409.17146, 2024

    Matt Deitke, Christopher Clark, Sangho Lee, Rohun Tripathi, Yue Yang, Jae Sung Park, Mohammadreza Salehi, Niklas Muennighoff, Kyle Lo, Luca Soldaini, et al. Molmo and pixmo: Open weights and open data for state-of-the-art multimodal models.arXiv preprint arXiv:2409.17146, 2024

    2024 arXiv

  12. [21]

    Gemini: A family of highly capable multimodal models.arXiv preprint arXiv:2312.11805, 1, 2023

    Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. Gemini: A family of highly capable multimodal models.arXiv preprint arXiv:2312.11805, 1, 2023

    2023 arXiv

  13. [22]

    Next-gpt: Any-to-any multimodal llm

    Shengqiong Wu, Hao Fei, Leigang Qu, Wei Ji, and Tat-Seng Chua. Next-gpt: Any-to-any multimodal llm. InForty-first International Conference on Machine Learning, 2024

    2024

  14. [23]

    Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks

    Zhe Chen, Jiannan Wu, Wenhai Wang, Weijie Su, Guo Chen, Sen Xing, Muyan Zhong, Qinglong Zhang, Xizhou Zhu, Lewei Lu, et al. Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. InProceedings of the IEEE/CVF conference on computer visi...

    2024

  15. [24]

    Gpt-4o system card.arXiv preprint arXiv:2410.21276, 2024

    Aaron Hurst, Adam Lerer, Adam P Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, et al. Gpt-4o system card.arXiv preprint arXiv:2410.21276, 2024

    2024 arXiv

  16. [25]

    Visual instruction tuning.Advances in neural information processing systems, 36:34892–34916, 2023

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning.Advances in neural information processing systems, 36:34892–34916, 2023

    2023

  17. [26]

    Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al. Qwen2. 5-vl technical report.arXiv preprint arXiv:2502.13923, 2025

    2025 arXiv

  18. [27]

    Gradient episodic memory for continual learning.Advances in neural information processing systems, 30, 2017

    David Lopez-Paz and Marc’Aurelio Ranzato. Gradient episodic memory for continual learning.Advances in neural information processing systems, 30, 2017

    2017

  19. [28]

    Lamol: Language modeling for lifelong language learning.arXiv preprint arXiv:1909.03329, 2019

    Fan-Keng Sun, Cheng-Hao Ho, and Hung-Yi Lee. Lamol: Language modeling for lifelong language learning.arXiv preprint arXiv:1909.03329, 2019

    1909 arXiv

  20. [29]

    Conpet: Continual parameter-efficient tuning for large language models.arXiv preprint arXiv:2309.14763, 2023

    Chenyang Song, Xu Han, Zheni Zeng, Kuai Li, Chen Chen, Zhiyuan Liu, Maosong Sun, and Tao Yang. Conpet: Continual parameter-efficient tuning for large language models.arXiv preprint arXiv:2309.14763, 2023

    2023 arXiv

  21. [30]

    Lifelong language pretraining with distribution-specialized experts

    Wuyang Chen, Yanqi Zhou, Nan Du, Yanping Huang, James Laudon, Zhifeng Chen, and Claire Cui. Lifelong language pretraining with distribution-specialized experts. InInternational Conference on Machine Learning, pages 5383–5395. PMLR, 2023

    2023

  22. [31]

    Class incremental learning with pre-trained vision-language models.arXiv preprint arXiv:2310.20348, 2023

    Xialei Liu, Xusheng Cao, Haori Lu, Jia-wen Xiao, Andrew D Bagdanov, and Ming-Ming Cheng. Class incremental learning with pre-trained vision-language models.arXiv preprint arXiv:2310.20348, 2023

    2023 arXiv

  23. [32]

    Clap4clip: Continual learning with probabilistic finetuning for vision-language models.arXiv preprint arXiv:2403.19137, 2024

    Saurav Jha, Dong Gong, and Lina Yao. Clap4clip: Continual learning with probabilistic finetuning for vision-language models.arXiv preprint arXiv:2403.19137, 2024. 11

    2024 arXiv

  24. [33]

    Vlm-pl: Advanced pseudo labeling approach for class incremental object detection via vision-language model

    Junsu Kim, Yunhoe Ku, Jihyeon Kim, Junuk Cha, and Seungryul Baek. Vlm-pl: Advanced pseudo labeling approach for class incremental object detection via vision-language model. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4170–4181, 2024

    2024

  25. [34]

    Experience replay for continual learning.Advances in neural information processing systems, 32, 2019

    David Rolnick, Arun Ahuja, Jonathan Schwarz, Timothy Lillicrap, and Gregory Wayne. Experience replay for continual learning.Advances in neural information processing systems, 32, 2019

    2019

  26. [35]

    Learning without forgetting.IEEE transactions on pattern analysis and machine intelligence, 40(12):2935–2947, 2017

    Zhizhong Li and Derek Hoiem. Learning without forgetting.IEEE transactions on pattern analysis and machine intelligence, 40(12):2935–2947, 2017

    2017

  27. [36]

    Mind the interference: Retaining pre-trained knowledge in parameter efficient continual learning of vision-language models

    Longxiang Tang, Zhuotao Tian, Kai Li, Chunming He, Hantao Zhou, Hengshuang Zhao, Xiu Li, and Jiaya Jia. Mind the interference: Retaining pre-trained knowledge in parameter efficient continual learning of vision-language models. InEuropean Conference on Computer Vision, pages 3...

    2024

  28. [37]

    Continual multimodal knowledge graph construction

    Xiang Chen, Jingtian Zhang, Xiaohan Wang, Ningyu Zhang, Tongtong Wu, Yuxiang Wang, Yongheng Wang, and Huajun Chen. Continual multimodal knowledge graph construction. InProceedings of the Thirty-Third International Joint Conference on Artificial Intelligence, pages 6225–6233, 2024

    2024

  29. [38]

    Symbolic replay: Scene graph as prompt for continual learning on vqa task

    Stan Weixian Lei, Difei Gao, Jay Zhangjie Wu, Yuxuan Wang, Wei Liu, Mengmi Zhang, and Mike Zheng Shou. Symbolic replay: Scene graph as prompt for continual learning on vqa task. InProceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 1250–1259, 2023

    2023

  30. [39]

    Vqacl: A novel visual question answering continual learning setting

    Xi Zhang, Feifei Zhang, and Changsheng Xu. Vqacl: A novel visual question answering continual learning setting. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 19102–19112, 2023

    2023

  31. [40]

    Climb: A continual learning benchmark for vision-and-language tasks.Advances in Neural Information Processing Systems, 35:29440–29453, 2022

    Tejas Srinivasan, Ting-Yun Chang, Leticia Pinto Alva, Georgios Chochlakis, Mohammad Rostami, and Jesse Thomason. Climb: A continual learning benchmark for vision-and-language tasks.Advances in Neural Information Processing Systems, 35:29440–29453, 2022

    2022

  32. [41]

    Adapting BERT for continual learning of a sequence of aspect sentiment classification tasks

    Zixuan Ke, Hu Xu, and Bing Liu. Adapting BERT for continual learning of a sequence of aspect sentiment classification tasks. In Kristina Toutanova, Anna Rumshisky, Luke Zettlemoyer, Dilek Hakkani-Tur, Iz Belt- agy, Steven Bethard, Ryan Cotterell, Tanmoy Chakraborty, and Yichao...

    2021

  33. [42]

    Episodic memory in lifelong language learning.Advances in Neural Information Processing Systems, 32, 2019

    Cyprien de Masson D’Autume, Sebastian Ruder, Lingpeng Kong, and Dani Yogatama. Episodic memory in lifelong language learning.Advances in Neural Information Processing Systems, 32, 2019

    2019

  34. [43]

    Lfpt5: A unified framework for lifelong few-shot language learning based on prompt tuning of t5

    Chengwei Qin and Shafiq Joty. Lfpt5: A unified framework for lifelong few-shot language learning based on prompt tuning of t5. InInternational Conference on Learning Representations

  35. [44]

    Clevr: A diagnostic dataset for compositional language and elementary visual reasoning

    Justin Johnson, Bharath Hariharan, Laurens Van Der Maaten, Li Fei-Fei, C Lawrence Zitnick, and Ross Girshick. Clevr: A diagnostic dataset for compositional language and elementary visual reasoning. In Proceedings of the IEEE conference on computer vision and pattern recognitio...

    2017

  36. [45]

    Dvqa: Understanding data visualizations via question answering

    Kushal Kafle, Brian Price, Scott Cohen, and Christopher Kanan. Dvqa: Understanding data visualizations via question answering. InCVPR, 2018

    2018

  37. [46]

    Gqa: A new dataset for real-world visual reasoning and compositional question answering

    Drew A Hudson and Christopher D Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 6700–6709, 2019

    2019

  38. [47]

    Imagenet: A large-scale hierarchical image database

    Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In2009 IEEE conference on computer vision and pattern recognition, pages 248–255. Ieee, 2009

    2009

  39. [48]

    Infographicvqa

    Minesh Mathew, Viraj Bagal, Rubèn Tito, Dimosthenis Karatzas, Ernest Valveny, and CV Jawahar. Infographicvqa. InProceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 1697–1706, 2022

    2022

  40. [49]

    Icdar 2015 competition on robust reading

    Dimosthenis Karatzas, Lluis Gomez-Bigorda, Anguelos Nicolaou, Suman Ghosh, Andrew Bagdanov, Masakazu Iwamura, Jiri Matas, Lukas Neumann, Vijay Ramaseshan Chandrasekhar, Shijian Lu, et al. Icdar 2015 competition on robust reading. In2015 13th international conference on documen...

    2015

  41. [50]

    Semeval-2020 task 8: Memotion analysis-the visuo-lingual metaphor! InProceedings of the Fourteenth Workshop on Semantic Evaluation, pages 759–773, 2020

    Chhavi Sharma, Deepesh Bhageria, William Scott, Srinivas Pykl, Amitava Das, Tanmoy Chakraborty, Viswanath Pulabaigari, and Björn Gambäck. Semeval-2020 task 8: Memotion analysis-the visuo-lingual metaphor! InProceedings of the Fourteenth Workshop on Semantic Evaluation, pages 7...

    2020

  42. [51]

    A corpus of natural language for visual reasoning

    Alane Suhr, Mike Lewis, James Yeh, and Yoav Artzi. A corpus of natural language for visual reasoning. InProceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 217–223, 2017. 12

    2017

  43. [52]

    Plug- and-play grounding of reasoning in multimodal large language models.arXiv preprint arXiv:2403.19322, 2024

    Jiaxing Chen, Yuxuan Liu, Dehu Li, Xiang An, Weimo Deng, Ziyong Feng, Yongle Zhao, and Yin Xie. Plug- and-play grounding of reasoning in multimodal large language models.arXiv preprint arXiv:2403.19322, 2024

    2024 arXiv

  44. [53]

    Learn to explain: Multimodal reasoning via thought chains for science question answering.Advances in Neural Information Processing Systems, 35:2507–2521, 2022

    Pan Lu, Swaroop Mishra, Tanglin Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering.Advances in Neural Information Processing Systems, 35:2507–2521, 2022

    2022

  45. [54]

    Are deep neural networks smarter than second graders? InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10834–10844, 2023

    Anoop Cherian, Kuan-Chuan Peng, Suhas Lohit, Kevin A Smith, and Joshua B Tenenbaum. Are deep neural networks smarter than second graders? InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10834–10844, 2023

    2023

  46. [55]

    Visual entailment: A novel task for fine-grained image understanding.arXiv preprint arXiv:1901.06706, 2019

    Ning Xie, Farley Lai, Derek Doran, and Asim Kadav. Visual entailment: A novel task for fine-grained image understanding.arXiv preprint arXiv:1901.06706, 2019

    1901 arXiv

  47. [56]

    Cross-media learning for image sentiment analysis in the wild

    Lucia Vadicamo, Fabio Carrara, Andrea Cimino, Stefano Cresci, Felice Dell’Orletta, Fabrizio Falchi, and Maurizio Tesconi. Cross-media learning for image sentiment analysis in the wild. InProceedings of the IEEE international conference on computer vision workshops, pages 308–317, 2017

    2017

  48. [57]

    Adaptive co-attention network for named entity recognition in tweets

    Qi Zhang, Jinlan Fu, Xiaoyu Liu, and Xuanjing Huang. Adaptive co-attention network for named entity recognition in tweets. InProceedings of the AAAI conference on artificial intelligence, volume 32, 2018

    2018

  49. [58]

    Improving multimodal named entity recognition via entity span detection with unified multimodal transformer

    Jianfei Yu, Jing Jiang, Li Yang, and Rui Xia. Improving multimodal named entity recognition via entity span detection with unified multimodal transformer. Association for Computational Linguistics, 2020

    2020

  50. [59]

    Visual choice of plausible alternatives: An evaluation of image-based common- sense causal reasoning

    Jinyoung Yeo, Gyeongbok Lee, Gengyu Wang, Seungtaek Choi, Hyunsouk Cho, Reinald Kim Amplayo, and Seung-won Hwang. Visual choice of plausible alternatives: An evaluation of image-based common- sense causal reasoning. InProceedings of the Eleventh International Conference on Lan...

    2018

  51. [60]

    From recognition to cognition: Visual commonsense reasoning

    Rowan Zellers, Yonatan Bisk, Ali Farhadi, and Yejin Choi. From recognition to cognition: Visual commonsense reasoning. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 6720–6731, 2019

    2019

  52. [61]

    Vizwiz grand challenge: Answering visual questions from blind people

    Danna Gurari, Qing Li, Abigale J Stangl, Anhong Guo, Chi Lin, Kristen Grauman, Jiebo Luo, and Jeffrey P Bigham. Vizwiz grand challenge: Answering visual questions from blind people. InProceedings of the IEEE conference on computer vision and pattern recognition, pages 3608–3617, 2018

    2018

  53. [62]

    Making the v in vqa matter: Elevating the role of image understanding in visual question answering

    Yash Goyal, Tejas Khot, Douglas Summers-Stay, Dhruv Batra, and Devi Parikh. Making the v in vqa matter: Elevating the role of image understanding in visual question answering. InProceedings of the IEEE conference on computer vision and pattern recognition, pages 6904–6913, 2017

    2017

  54. [63]

    Visual spatial reasoning.Transactions of the Association for Computational Linguistics, 11:635–651, 2023

    Fangyu Liu, Guy Emerson, and Nigel Collier. Visual spatial reasoning.Transactions of the Association for Computational Linguistics, 11:635–651, 2023

    2023

  55. [64]

    Instructuie: Multi-task instruction tuning for unified information extraction

    Xiao Wang, Weikang Zhou, Can Zu, Han Xia, Tianze Chen, Yuansen Zhang, Rui Zheng, Junjie Ye, Qi Zhang, Tao Gui, et al. Instructuie: Multi-task instruction tuning for unified information extraction. arXiv preprint arXiv:2304.08085, 2023

    2023 arXiv

  56. [65]

    Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen

    Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. InThe Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. Ope...

    2022

  57. [66]

    Riemannian walk for incremental learning: Understanding forgetting and intransigence

    Arslan Chaudhry, Puneet K Dokania, Thalaiyasingam Ajanthan, and Philip HS Torr. Riemannian walk for incremental learning: Understanding forgetting and intransigence. InECCV, pages 532–547, 2018

    2018

  58. [67]

    Coin: A benchmark of continual instruction tuning for multimodel large language models.Advances in Neural Information Processing Systems, 37:57817–57840, 2024

    Cheng Chen, Junchen Zhu, Xu Luo, Hengtao Shen, Jingkuan Song, and Lianli Gao. Coin: A benchmark of continual instruction tuning for multimodel large language models.Advances in Neural Information Processing Systems, 37:57817–57840, 2024

    2024

  59. [68]

    Analyzing and reducing catastrophic forgetting in parameter efficient tuning

    Xinlong Li, Weijieying Ren, Wei Qin, Lei Wang, Tianxiang Zhao, and Richang Hong. Analyzing and reducing catastrophic forgetting in parameter efficient tuning. InICASSP 2025-2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1–5. IEEE, 2025

    2025

  60. [69]

    Visual instruction tuning.NeurIPS, 36, 2024

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning.NeurIPS, 36, 2024

    2024

  61. [70]

    An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2. 5 technical report.arXiv preprint arXiv:2412.15115, 2024. 13

    2024 arXiv

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.