{"id":"7146bf6f-ccc5-4da1-bb38-b8c2fd7e911d","arxiv_id":"2505.09952","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"Long-CL, a masking plus selective replay method, reduces catastrophic forgetting across 21 vision-language and 30 text tasks, outperforming baselines on two new long-term continual learning benchmarks.","lead":"This paper builds two large multi-task continual learning benchmarks, one with 21 vision-and-language tasks and one with 30 text tasks, and proposes Long-CL, a memory-management method that masks important parameters and replays selected examples. On these benchmarks, Long-CL reports final average performance of 51.9% and 60.1%, beating the prior baselines and approaching the multitask upper bound.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Replay buffer persistence across tasks is never specified; if R_t is not accumulated, MemCon does not replay old task data, and the reported SOTA retention rests on an unsupported mechanism.","rationale":"The paper makes a concrete, falsifiable claim: a 7B-scale generative model can retain nearly all old task performance over 21 or 30 tasks via parameter masking plus selective replay. The ablation study is the only evidence isolating the two components, and its interpretation hinges entirely on whether the replay buffer accumulates old samples. Figure 3 and the phrase 'Append' suggest it does, but no formal definition appears in Section 4.3. This is not a matter of consensus; it is an internal specification gap. The Eq. (5) division by zero at t=2 is a separate, concrete mathematical error that would prevent the method from running as described, though it is likely a boundary-condition fix. I agree with the reader's CONDITIONAL verdict: the concerns are addressable without rejecting the work, but neither the benchmark release nor the code is currently available to verify the buffer semantics. No machine-checked proof or shipped code is present, so the reproducibility burden falls on the text.","tokens_in":14837,"tokens_out":11891,"duration_ms":112321,"concrete_test":"Inspect the released code (or supplement) for the buffer update rule: is buffer_t = buffer_{t-1} ∪ H_t ∪ G_t, or is it reset at each task? If the code shows accumulation, the concern is resolved. If not, re-run the MMLongCL-Bench experiment with an explicit persistent buffer (same 20% budget) and compare AP to the reported 51.93; if AP drops toward the MemMan-only level (31.01), the central claim about replay of old tasks is unsupported. Also, as a secondary check, evaluate Eq. (5) at t=2 with nonzero prototype distances: if no special case is implemented, the denominator is zero and the update step is undefined for the second task.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing ambiguity is in Section 4.3: 'The final replay buffer for task t is constructed by combining both sets, Rt = Ht ∪ Gt.' The paper never states whether Rt is accumulated across tasks or reset each task. Figure 3 shows 'Task t-1 Buffer' and 'Append,' suggesting a persistent buffer, but the text and implementation details ('we set the memory buffer size to 20% of the training dataset') do not specify that samples selected for earlier tasks are replayed during later tasks. If Rt is only the current task's selection, then MemCon's 'rehearsal mechanism' trains on current-task hard and differential samples only; it does not replay old task data. In that case, the reported retention (AF = -9.93 on MMLongCL-Bench) must be produced by MemMan's parameter interpolation (Eq. 7) alone, yet the ablation (Table 3) shows MemMan alone yields AP 31.01 vs. 39.94 for MemCon alone and 45.74 combined. Without accumulation, the contribution of MemCon to retention is unexplained and the central claim that selective replay of old samples largely eliminates forgetting is unsupported. This is directly load-bearing for the headline SOTA numbers (51.93 and 60.12 AP). Additionally, Eq. (5) is undefined at t=2 because the denominator is an empty sum (0), making the adaptive weight infinite; the paper only imposes a lower bound α_min, no upper bound, so the method as written cannot be executed for the second task. Both issues are addressable in revision, but they must be resolved before the central claim can be accepted.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper addresses long-term continual learning (CL) for large language and vision-language models, where a model must learn sequentially from a large number of heterogeneous tasks without catastrophic forgetting. The authors propose Long-CL, consisting of two components: MemMan, which identifies task-critical parameter updates via discrepancy estimation (Eq. 2) and fuses old and new LoRA adapters with an adaptive interpolation weight α_t (Eqs. 5–7); and MemCon, which selects hard samples and differential samples for a replay buffer (Eqs. 8–9). The paper also introduces two benchmarks, MMLongCL-Bench (21 multimodal datasets, 503k training samples) and TextLongCL-Bench (30 textual datasets, 397k samples), and reports experiments with LLaVA-7B and Qwen2.5-7B. The main results claim state-of-the-art final average performance of 51.93 AP on MMLongCL-Bench and 60.12 AP on TextLongCL-Bench, with corresponding average forgetting values of −9.93 and −0.89, approaching the multitask upper bounds.","tokens_in":15231,"tokens_out":13771,"duration_ms":99694,"significance":"The paper targets a relevant and underexplored setting: continual learning over dozens of heterogeneous tasks with multi-billion-parameter generative models. The proposed combination of parameter-space masking (MemMan) and selective replay (MemCon) is plausible, and the empirical gains over standard baselines (EWC, ER, LWF, CL-MoE, etc.) are substantial. The construction and planned release of the two benchmarks is in itself a useful community contribution, and the paper includes ablations (Table 3), sensitivity analysis (Figure 4, Table 5), and robustness to task order (Table 4). If the mechanism and results hold up, the work would provide a practical recipe for reducing catastrophic forgetting in long-horizon instruction tuning.","major_comments":[{"comment":"The adaptive weight α_t is undefined at t=2 because the denominator ∑_{1≤i<j≤t−1} ||A_i − A_j||_2 is an empty sum, evaluating to 0. Since the continual learning procedure starts at task 1 and needs an interpolation weight for every task, the update in Eq. (7) cannot be executed for the second task as written. The lower bound α_min does not resolve this, as it is a one-sided bound and α_t would be infinite rather than merely small. Please specify the initialization for t=1 and t=2 (e.g., set α_2=1 or take a limit), and ensure the formula is well-defined for all t.","section":"§4.2, Eq. (5)"},{"comment":"The paper never states whether the replay buffer R_t is accumulated across tasks. Section 4.3 defines R_t = H_t ∪ G_t for the current task only, and the implementation details mention a buffer size of 20% of the training dataset without specifying whether the buffer persists. Figure 3 shows 'Task t-1 Buffer' and 'Append', which suggests persistence, but the text is ambiguous. This ambiguity is load-bearing: the ablation in Table 3 shows MemCon alone yields AP 39.94, and the paper's central claim that selective replay of old examples largely eliminates forgetting depends on old task samples being replayed during later tasks. If the buffer is reset each task, MemCon does not replay any old task data, and the reported retention must be explained by MemMan alone, which the ablation does not support. The authors must state explicitly how buffers persist and how the 20% budget is shared between the current and historical tasks, and adjust the text or the method accordingly.","section":"§4.3 and §5.4"},{"comment":"All reported results appear to be from a single run; no standard deviations, confidence intervals, or seed information are provided. Table 4 shows performance across three task orders but without variance estimates, and the sensitivity analyses in Figure 4 and Table 5 also lack error bars. The headline claims of state-of-the-art performance and of reaching 94.5% of multitask performance are quantitative comparative statements, so without a measure of run-to-run variability it is impossible to judge whether the reported gaps (e.g., Long-CL 45.74 vs. EWC 37.40 on MMLongCL-Bench) are statistically meaningful. Please report means and standard deviations over at least three random seeds, or otherwise justify that the evaluation is deterministic.","section":"§5.1, Tables 2–5"}],"minor_comments":[{"comment":"The notation Min_{R_g} in Eq. (9) is unclear; R_g is later stated as a percentage, but the set construction with a threshold δ needs a precise definition of how many samples are selected and how ties are broken.","section":"§4.3, Eq. (9)"},{"comment":"The phrase 'When Rt is set to 20%' conflates the buffer set R_t with its size; please clarify that R_t denotes the buffer and 20% refers to its budget relative to the training set.","section":"§5.4"},{"comment":"The hyperparameter δ is defined as 0.8*Dmax / 2, but Dmax is not defined; specify how the maximum distance of prototypes is computed.","section":"§5.1"},{"comment":"The column header order is confusing because the TextLongCL-Bench AP/AF columns appear among the category columns; reformat the table so that the benchmark-level metrics are clearly separated from category-level accuracies.","section":"Table 2"},{"comment":"The abstract claims the method 'outperforms the previous state-of-the-art by 7.4% and 6.5% AP', but the percentage-point calculation in Section 5.2 (36.58% and 15.92%) refers to improvements over Vanilla, not over the previous SOTA; please make the comparison basis explicit.","section":"Abstract, §5.2"},{"comment":"The paper states that the benchmarks are released, but no download URL or repository is provided; please include the public access information.","section":"§1"},{"comment":"There are several typographical issues, such as 'Long—CL' in the heading of Section 5.3 and inconsistent spacing in Table 2; please proofread the manuscript.","section":"§5.3"}],"recommendation":"major_revision","confidential_remarks":"The manuscript addresses an important problem and the benchmarks are potentially valuable. The two technical issues (Eq. (5) and buffer persistence) are fixable by clarification and small additions, but they are central to the method as written, so I recommend major revision. I also note that the paper does not mention code release, which would strengthen reproducibility for a community that values open benchmarks."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a useful empirical paper for the continual learning community. It offers two genuinely new long-horizon benchmarks and a method that appears to work, but two specification holes need fixing before the headline claims can be trusted.\n\nWhat's new: MMLongCL-Bench (21 vision-language tasks across four task types) and TextLongCL-Bench (30 information-extraction tasks) are larger and more heterogeneous than existing CL suites. Long-CL combines top-K LoRA shift masking, a prototype-distance adaptive interpolation weight, and replay of hard plus cross-task samples. The components are known, but the assembly is new, and the ablations show both pieces matter: MemMan alone gets 31.01 AP on MMLongCL-Bench, MemCon alone 39.94, and the combination 45.74. They also test task-order permutations and buffer-size/K sensitivity, which is more than many CL papers do.\n\nSoft spots, in order of severity. First, replay buffer persistence is underspecified. Section 4.3 defines Rt = Ht ∪ Gt for the current task only; it never says whether earlier buffers are retained. Figure 3 shows 'Append' and 'Task t-1 Buffer', suggesting accumulation, but the text and the implementation note ('we set the memory buffer size to 20% of the training dataset') do not confirm it. If the buffer resets at each task, MemCon never replays old data and the reported retention must come from MemMan alone, which the ablation does not establish. This is load-bearing. Second, Eq. (5) is undefined at t=2 because the denominator is an empty sum; the lower bound on α doesn't fix an infinite value. The method as written cannot be executed on the second task. Easy fix, but it must be stated. Third, results are single-run with no error bars, and the benchmarks and code are not yet accessible. For a paper whose main contribution includes benchmarks, that is a concrete reproducibility gap. Fourth, the 'previous state-of-the-art' turns out to be their own CL-MoE; not a flaw by itself, but the comparison is less external than it seems.\n\nNone of these is fatal. The central claim—that a parameter-mask plus selective-replay method reaches 94.5% of multitask performance over 21 tasks—is plausible, and the ablations support the mechanism qualitatively. But the buffer ambiguity and the Eq. (5) bug mean the paper needs revision before the numbers can be taken at face value. Who it's for: CL researchers who want a longer benchmark and a solid baseline. I'd send it to serious referees, and I'd insist on code and data release.","headline":"A useful long-horizon CL benchmark pair and a plausible method, but two load-bearing ambiguities (replay-buffer persistence, Eq. 5) and missing code/data mean the paper needs revision before the numbers can be trusted.","tokens_in":15704,"tokens_out":3244,"would_cite":false,"duration_ms":29143,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Long-CL nearly halts catastrophic forgetting across 21 and 30 sequential task benchmarks.","keywords":["continual learning","catastrophic forgetting","large language models","vision-language models","LoRA","experience replay","memory consolidation","long-term continual learning benchmark"],"falsifier":"Run Long-CL on MMLongCL-Bench with the replay buffer reset to each task's own hard and differential samples, so no samples from earlier tasks are replayed; if AP remains at 51.93, the selective replay of old samples is not what drives the reported retention.","tokens_in":14626,"feed_emoji":"🧠","tokens_out":12750,"duration_ms":108683,"temperature":0.7,"pith_summary":"The paper introduces long-term continual learning—learning from a stream of dozens of heterogeneous tasks—and argues that existing continual learning methods collapse under this load. To fix it, the paper proposes Long-CL, a two-part memory framework: MemMan protects task-critical LoRA parameters by masking and adaptively fusing old and new weights, while MemCon replays a curated mix of hard and cross-task-aligned samples. The central claim is that Long-CL achieves state-of-the-art average performance (AP) of 51.93 on the new 21-task multimodal benchmark and 60.12 on the new 30-task text benchmark, beating the previous best method by 7.4 and 6.5 AP. At a 20% replay buffer it reaches 94.5% of the multitask upper bound. The upshot is a parameter-mask plus selective-replay recipe for letting 7B-scale generative models keep learning over long streams without task identity at inference time.","feed_headline":"Method hits 94.5% of multitask accuracy in long task streams","feed_subtitle":"Parameter-drift masking plus selective replay beats prior methods by 7.4 and 6.5 average points.","key_machinery":"The machinery is a two-part memory loop around LoRA adapters. Task-Core Memory Management (MemMan) first computes the per-unit Euclidean drift between the model before and after tuning on the current task, records the top-K drifted units in a cumulative mask $\\mathbf{Mask}_t$, and then fuses old and new LoRA weights element-wise as $\\theta_t = \\beta_t \\odot \\phi_t + (1-\\beta_t) \\odot \\theta_{t-1}$ with $\\beta_t = \\alpha_t \\mathbf{Mask}_t + (1-\\alpha_t)(1-\\mathbf{Mask}_t)$. The scalar $\\alpha_t$ is set by the distance of the current task's prototype to previous prototypes, so early novel tasks get larger updates and later tasks become more conservative. Long-term Memory Consolidation (MemCon) builds a replay set $R_t = H_t \\cup G_t$: $H_t$ holds the hardest samples (largest distance to the current prototype) and $G_t$ holds differential samples (smallest cumulative distance to previous prototypes but not too close to any one of them). These replayed samples are provided during current-task training, which is what lets the model refresh old knowledge while learning new tasks.","core_discovery":"Long-CL establishes that catastrophic forgetting over long task streams can be largely controlled by protecting a small set of task-critical parameters and replaying a carefully chosen subset of old samples. MemMan computes the per-unit drift between the LoRA model before and after tuning on the current task, records the top-K drifted units in a cumulative mask, and fuses old and new weights with a prototype-derived weight. MemCon selects hard samples (far from the current task prototype) and differential samples (close to previous task prototypes) and uses them as replay data. On MMLongCL-Bench this reaches 51.93 AP with -9.93 average forgetting, and on TextLongCL-Bench it reaches 60.12 AP with -0.89 average forgetting. With a 20% replay buffer, Long-CL reaches 94.5% of the multitask upper bound.","pith_inferences":["Not tested in the paper: the same drift-mask and prototype-distance machinery should transfer to other parameter-efficient adapters (adapters, soft prompts), since nothing in the fusion equations is LoRA-specific; running that variant would test the mechanism's generality.","Not tested in the paper: the ablation does not isolate cross-task replay from within-task replay; a version that resets the buffer after each task would separate how much of the gain comes from accumulated old-task samples versus the selection rule alone.","Not tested in the paper: because $\\alpha_t$ decays with the number of accumulated tasks, the framework predicts memory updates become more conservative as the stream grows; one could test whether the optimal schedule depends mainly on task count or on true task similarity.","Not tested in the paper: the negative average forgetting values suggest the fused parameter trajectory keeps improving old tasks during later training; if that holds, Long-CL behaves like progressive learning rather than merely protecting an old snapshot."],"forward_implications":["At a 20% replay buffer, Long-CL reaches 94.5% of the multitask-learning upper bound on MMLongCL-Bench, so the memory cost can be modest while keeping most of the benefit.","The reported negative average forgetting (-9.93 on MMLongCL-Bench, -0.89 on TextLongCL-Bench) means earlier tasks end up scoring higher after later training, a backward-transfer effect rather than mere retention.","Because inference requires no task identifier, Long-CL applies to streams where tasks arrive unlabeled and may overlap in unknown ways.","The two released benchmarks, with 21 vision-language and 30 text-only datasets, give later work a common yardstick for long-horizon continual learning, where prior benchmarks covered at most a handful of tasks.","Long-CL outperforms CL-MoE across three random task orders, indicating the method is not tuned to one particular sequence."],"supporting_citations":[{"why":"Supplies the LoRA adapter parameterization that all memory units, masks, and weight fusions operate on.","marker":"[65]"},{"why":"Provides the LLaVA-7B backbone used for the MMLongCL-Bench experiments.","marker":"[69]"},{"why":"Provides the Qwen2.5-7B backbone used for the TextLongCL-Bench experiments.","marker":"[70]"},{"why":"O-LoRA is the regularization baseline Long-CL is compared against and outperforms on both benchmarks.","marker":"[5]"},{"why":"EWC is a regularization baseline and frames the catastrophic-forgetting problem the paper addresses.","marker":"[3]"},{"why":"Experience replay is the rehearsal baseline that MemCon's selective replay mechanism builds on.","marker":"[34]"},{"why":"CL-MoE is the strongest architecture baseline, used for the main comparison and task-order robustness tests.","marker":"[6]"},{"why":"Supplies the final average performance (AP) metric used for the headline results.","marker":"[27]"},{"why":"Supplies the average forgetting (AF) metric used to report backward transfer.","marker":"[66]"}],"fun_headline_variants":["Long-CL beats prior CL by 7.4% and 6.5% on two benchmarks","Memory management and consolidation boost long-term continual learning","Selective replay and drift masking hit 94.5% multitask accuracy","New framework curbs forgetting in long task streams","Task-core memory strategy surpasses SOTA in long-term CL"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The reported retention depends on the replay buffer accumulating samples across tasks: samples chosen for early tasks must still be replayed while later tasks train, but the paper defines R_t only for the current task and never states that earlier selections are carried forward.","fun_headline_variants_meta":{"raw":{"variants":["Long-CL beats prior CL by 7.4% and 6.5% on two benchmarks","Memory management and consolidation boost long-term continual learning","Selective replay and drift masking hit 94.5% multitask accuracy","New framework curbs forgetting in long task streams","Task-core memory strategy surpasses SOTA in long-term CL"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000512,"raw_usage":{"total_tokens":2500,"prompt_tokens":968,"completion_tokens":1532,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":584,"completion_tokens_details":{"reasoning_tokens":1440}},"tokens_in":584,"tokens_out":1532,"duration_ms":9567,"temperature":1.0,"reasoning_tokens":1440,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T21:21:42.605466+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run Long-CL on MMLongCL-Bench with the replay buffer reset to each task's own hard and differential samples, so no samples from earlier tasks are replayed; if AP remains at 51.93, the selective replay of old samples is not what drives the reported retention.","supporting_citations":[{"cited_title":"Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen","cited_arxiv_id":null,"evidence_quote":"Supplies the LoRA adapter parameterization that all memory units, masks, and weight fusions operate on."},{"cited_title":"Orthogonal subspace learning for language model continual learning","cited_arxiv_id":null,"evidence_quote":"O-LoRA is the regularization baseline Long-CL is compared against and outperforms on both benchmarks."},{"cited_title":"Riemannian walk for incremental learning: Understanding forgetting and intransigence","cited_arxiv_id":null,"evidence_quote":"Supplies the average forgetting (AF) metric used to report backward transfer."}],"review_version":1}