{"id":"40b840ca-70de-4ecb-8b6a-32feff7437a9","arxiv_id":"2607.04711","paper_version":1,"verdict":"ACCEPT","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"low","formal_verification":"none","parameter_count":3,"one_line_summary":"Modeling prompts as probabilistic distributions and sampling from query-conditioned mixtures eliminates prompt collapse and sets new state-of-the-art accuracy in class-incremental learning.","lead":"Prompt-based continual learning methods suffer from prompt collapse, where learned prompts become nearly identical and fail to capture diverse image patterns across tasks. This paper models prompts as Gaussians, samples from a query-conditioned mixture, and regularizes distribution drift, yielding higher accuracy on ImageNet-R, CIFAR-100 and CUB-200.","discovery_kind":"new_method","skeptic_critique":{"model":"grok-4.5","headline":"No significant objection identified","rationale":"The reader's identification of the collapse-to-accuracy causal premise as the weakest link is accurate, yet the manuscript already supplies the natural controls (ablation of each modeling choice, comparison against output-level and diversity-regularized baselines, soft-vs-hard mixture, Ns sensitivity). Those controls keep correctness risk low and leave the ACCEPT verdict intact. No stronger technical soft spot (e.g., hidden non-differentiability, metric inconsistency between Mahalanobis relevance and cosine aggregation, or unacknowledged capacity inflation) survives scrutiny of Sec. 3.3–3.4 and the supplement. A modest verification of the diversity–accuracy correlation under re-implementation remains worthwhile for reproducibility, but does not alter the evaluation.","tokens_in":23822,"tokens_out":466,"duration_ms":20597,"concrete_test":"Re-implement the three variants of Supp. Table G (deterministic, output-level, prompt-level) under identical random seeds and measure both FAA and the entropy of the relevance scores s(n) across all tasks; if the prompt-level variant loses its similarity reduction or the FAA gap shrinks below 1 point, the causal link between prompt-level diversity and accuracy would be weakened.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim (prompt collapse as primary failure mode of deterministic prompt pools, remedied by query-conditioned diagonal-Gaussian mixtures + moment matching + cosine re-weighting + KL step-wise regularization) is supported by the paper's own evidence chain: Fig. 2 / Fig. F document near-1.0 pairwise similarities even after scaling N; Table 4 isolates progressive gains from probabilistic sampling, mixture formation, and LDR; Supp. Tables G/H show that output-level probabilistic modeling, isotropic perturbation, or an explicit diversity loss leave both similarity and FAA essentially unchanged. No internal inconsistency or untested regime that would reverse the reported margins on ImageNet-R / CIFAR-100 / CUB-200 is apparent. The reader's weakest assumption is therefore real but already stress-tested inside the manuscript; it does not rise to a load-bearing threat.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.5","summary":"The paper identifies a 'prompt collapse' problem in existing prompt-based class-incremental learning methods (L2P, DualPrompt, CODA-P, VQ-Prompt), where learned prompt tokens become highly correlated (pairwise cosine similarities near 1.0) and fail to capture diverse query-feature distributions across tasks. To address this, the authors model each prompt component as a diagonal Gaussian, form a query-conditioned mixture via Mahalanobis relevance scores and moment matching (Eqs. 2–7), sample Ns prompts via the reparameterization trick, aggregate them by cosine similarity to the query (Eqs. 9–10), and add a KL distribution-regularization loss (Eqs. 13–14) that penalizes abrupt changes between successive training steps. The resulting framework is evaluated on ImageNet-R (5/10/20-task), CIFAR-100 and CUB-200 under frozen ViT-B/16 backbones, reporting consistent gains in final average accuracy (FAA) and cumulative average accuracy (CAA) over strong prompt-based baselines, with supporting ablations, per-task forgetting curves, and hyper-parameter studies.","tokens_in":24096,"tokens_out":783,"duration_ms":11547,"significance":"If the empirical claims hold, the work supplies a clean, practically usable diagnosis and remedy for a previously under-appreciated failure mode of prompt pools in continual learning. The multi-seed tables, component-wise ablations (Table 4), comparisons under three pre-training regimes (Table 3), and supplementary checks against output-level probabilistic modeling, isotropic perturbation and explicit diversity losses constitute a solid evidence chain. The method adds only negligible memory and compute (Table 7) while remaining rehearsal-free and backbone-frozen, making it immediately relevant to the growing literature on efficient continual adaptation of large vision transformers. The distribution-regularization term is a simple, well-motivated online regularizer that further reduces forgetting without restricting plasticity.","major_comments":[],"minor_comments":[{"comment":"Several places in the provided manuscript text exhibit missing spaces (e.g., 'Continuallearningaimsto', 'promptcollapseproblem'). These are almost certainly formatting artifacts, but should be cleaned for the camera-ready version.","section":null},{"comment":"Figure 1(a) caption and the corresponding t-SNE plots in the supplement would benefit from a brief note on how the query features were extracted (frozen ViT-B/16, which layer) so that the diversity claim can be reproduced exactly.","section":null},{"comment":"In Sec. 3.3 the notation for the mixture covariance (Eqs. 6–7) is correct under the diagonal assumption, yet a one-sentence reminder that the between-component term is the law of total variance would help readers less familiar with moment matching.","section":null},{"comment":"Table 5 (effect of top-K) is useful; adding the corresponding CAA numbers would make the soft-mixture versus hard-selection comparison fully consistent with the rest of the paper.","section":null},{"comment":"The hyper-parameter table in the supplement lists Ns=30 and λ=1e-6 for all datasets; a short remark that these values were selected on a held-out split of ImageNet-R (and then frozen) would strengthen the claim of robustness.","section":null}],"recommendation":"accept","confidential_remarks":"The manuscript is already in good shape for a top-tier venue. The central empirical claim is well-supported and the supplementary material closes the most obvious alternative explanations. I see no reason to request additional experiments that would alter the recommendation."},"author_rebuttal":null,"desk_editor":{"model":"grok-4.5","letter":"The useful thing here is the diagnosis: existing prompt pools (L2P, DualPrompt, CODA-P, VQ-Prompt) collapse to near-identical vectors (pairwise cosine ~1.0 even after scaling N), so they cannot cover the high intra-task diversity you actually see in the t-SNEs. They fix it by turning each prompt into a diagonal Gaussian, building a query-conditioned mixture via moment matching, sampling Ns tokens, re-weighting them by cosine to the query, and adding a cheap KL that keeps successive Gaussians from jumping. That combination is new relative to ProDA/PPL (which keep the prompts themselves deterministic) and it works.\n\nWhat they do well: multi-seed tables (5 runs) on ImageNet-R 5/10/20-task, CIFAR-100 and CUB-200, plus three different pre-trains (supervised, iBOT, DINO). Ablations isolate each piece (Table 4), Ns and λ sweeps are honest, per-task forgetting curves show the KL actually helps, and the overhead is negligible because they average the samples before the single Pre-T forward. Supp. Tables G/H kill the obvious alternatives (output-level probabilistic modeling, isotropic noise, explicit diversity loss). Math is standard reparameterization + moment matching; citations are fair.\n\nSoft spots are real but secondary. The weakest premise—that high cosine similarity is the main accuracy bottleneck and that diagonal Gaussians + moment matching are enough—is stress-tested inside the paper itself and does not reverse the reported margins. No public code is a reproducibility ding, not a validity one. Scope is narrow (frozen ViT, rehearsal-free CIL); they do not claim more.\n\nThis is for people already working on prompt-based continual learning who want a low-overhead diversity fix. It is not a new scientific paradigm, but it is a clean, reproducible engineering advance that a serious referee should see. I would accept it for review and expect light-to-moderate revision mainly around code release and a couple of extra ablations.","headline":"Clean empirical fix for a real, previously unreported failure mode (prompt collapse) in rehearsal-free prompt-based CIL; solid multi-seed gains, no load-bearing holes.","tokens_in":24664,"tokens_out":530,"would_cite":true,"duration_ms":5229,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.5","headline":"Prompt-based continual learning collapses when prompts become nearly identical; modeling them as Gaussians and sampling from a query-conditioned mixture restores diversity and lifts accuracy.","keywords":["continual learning","class-incremental learning","prompt tuning","prompt collapse","probabilistic prompts","Gaussian mixture","distribution regularization"],"falsifier":"Train the same architecture on ImageNet-R 10-task but replace the learned Gaussians with deterministic vectors that are forced to the same pairwise cosine similarities as the probabilistic model; if final average accuracy remains unchanged, the collapse diagnosis is false.","tokens_in":24759,"feed_emoji":"🔀","tokens_out":602,"duration_ms":4686,"temperature":0.7,"pith_summary":"When a frozen vision transformer is taught new classes over time by learning a small set of prompt tokens, those tokens tend to become almost identical to one another. The paper shows that this “prompt collapse” prevents the model from representing the highly varied image patterns that appear as tasks accumulate. The remedy is to replace each fixed prompt vector with a diagonal Gaussian, form a query-conditioned mixture of those Gaussians by moment matching, draw several samples, and re-weight them by cosine similarity to the query. A simple KL penalty keeps successive distributions from jumping too far. On ImageNet-R, CIFAR-100 and CUB-200 the resulting prompts stay diverse and final average accuracy rises by clear margins over previous prompt methods, with almost no extra memory or compute.","feed_headline":"Prompt collapse is fixed by sampling from Gaussians","feed_subtitle":"Query-conditioned mixtures keep continual-learning prompts diverse and raise accuracy on ImageNet-R","key_machinery":"Query-conditioned Gaussian mixture of prompt distributions (Eqs. 1–10): each of M×N prompts is a diagonal Gaussian; relevance scores form a mixture by moment matching; Ns samples are drawn by the reparameterization trick and re-weighted by cosine similarity to the query before prefix-tuning.","core_discovery":"Existing prompt-based continual learners suffer from prompt collapse: pairwise cosine similarities among learned prompts approach 1.0, so the prompts cannot capture the diverse query-feature distributions that arise across tasks. Parameterizing each prompt as a diagonal Gaussian, building a soft mixture via Mahalanobis relevance scores, sampling multiple prompts and aggregating them by query similarity, together with a KL distribution-regularization loss, eliminates the collapse and yields higher final and cumulative average accuracy.","pith_inferences":[],"forward_implications":[],"fun_headline_variants":["Prompt Collapse Fixed by Sampling Gaussian Mixtures","Query-Mixture Prompts Stop Collapse in Continual Learning","Diagonal Gaussians Keep Prompts Diverse Across Tasks","Mahalanobis Mixtures End Prompt Collapse","KL-Regularized Prompt Distributions Raise Continual Accuracy"],"cache_read_input_tokens":16512,"weakest_assumption_plain":"That the high pairwise similarity among deterministic prompts is the main reason accuracy stalls, and that replacing those vectors by diagonal Gaussians whose mixture is formed by Mahalanobis scores is enough to restore useful diversity.","fun_headline_variants_meta":{"raw":{"variants":["Prompt Collapse Fixed by Sampling Gaussian Mixtures","Query-Mixture Prompts Stop Collapse in Continual Learning","Diagonal Gaussians Keep Prompts Diverse Across Tasks","Mahalanobis Mixtures End Prompt Collapse","KL-Regularized Prompt Distributions Raise Continual Accuracy"]},"model":"grok-4.5","effort":"low","cost_usd":0.00151,"raw_usage":{"total_tokens":835,"prompt_tokens":779,"num_sources_used":0,"completion_tokens":56,"cost_in_usd_ticks":15100000,"prompt_tokens_details":{"text_tokens":779,"audio_tokens":0,"image_tokens":0,"cached_tokens":256},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":0,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":779,"tokens_out":56,"duration_ms":1120,"temperature":1.0,"reasoning_tokens":0,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-07-11T14:45:40.981824+00:00","model_set":{"reader":"grok-4.5"},"falsifier":"Train the same architecture on ImageNet-R 10-task but replace the learned Gaussians with deterministic vectors that are forced to the same pairwise cosine similarities as the probabilistic model; if final average accuracy remains unchanged, the collapse diagnosis is false.","supporting_citations":[],"review_version":1}