{"id":"8a705d7d-335b-4550-b559-628c0b546b61","arxiv_id":"2505.06524","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":4,"one_line_summary":"CPC-SAM reweights random prompts to enforce segmentation consistency across prompt variants, claiming this yields causal prompts that improve open-vocabulary multi-entity segmentation with SAM.","lead":"This paper argues that prompt bias, not the backbone, makes SAM fail on open-vocabulary multi-entity segmentation, and proposes CPC-SAM to reweight random prompts using a causal consistency objective. The reported empirical gains are consistent, but the central causal proof is invalid, so the method currently stands as a heuristic prompt-tuning recipe rather than a proven causal calibration.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Theorem 1's proof assumes random prompts create independent confounders E_i, but all D_i share the same image X; image-level confounders are constant across prompt perturbations and cannot be canceled by consistency, so the causal-prompt conclusion is unsupported.","rationale":"The reader's weakest_assumption identifies the same load-bearing flaw: all perturbed datasets D_i share the same image X, so random prompt annotations do not create independent environments with varying confounders E_i. My read of Appendix A.2 confirms this: the proof explicitly writes X_i = C ⊕ E_i and treats E_i as if it varies across i, but Section 4.1 defines each prompt group as the same sample x_i,j with a different prompt p_i,j. The decomposition is therefore a modeling assertion, not a consequence of the setup. The proof also relies on the claim that random perturbations make E_i and E_j uncorrelated; with a fixed image, image-level confounders are perfectly correlated across prompt groups. Consequently, the key gradient-cancellation argument in Eqs. (7)-(9) is invalid, and Theorem 1 does not establish that fθ(P_i) contains only causal factors. This is not merely a disagreement with an outside consensus; it is an internal inconsistency between the theorem's proof and the method's construction. I also considered whether the empirical results could independently support the causal claim, but the ablations compare prompt types rather than manipulating confounders, and the 32.4% expert-refinement gap only shows prompt quality matters, not that the learned prompts are causal. Therefore the central theoretical contribution is unsupported, and rejection as a theoretical paper is the appropriate verdict.","tokens_in":29538,"tokens_out":2147,"duration_ms":25843,"concrete_test":"Analytically re-derive Appendix A.2 with E_i = E for all i, as required by D_i = {X, P_i} sharing the same image X. If the gradient cancellation step PNt i=1 ∇θL_Ei ≈ 0 is replaced by N_t ∇θL_E, show whether Eq. (9) still follows; it will not unless the loss itself forces E-independence, which is precisely the unproven claim. Complement this with a synthetic experiment: construct data where a class-correlated background is constant across all images, train CPC-SAM with Nt = 2 random box prompts, and measure whether the learned prompts suppress the background. If the background remains encoded, the theorem's causal conclusion fails on its own setup.","verdict_should_be":"REJECT","load_bearing_attack":"The central claim is that Theorem 1 (Section 3.3) proves that minimizing cumulative segmentation loss across randomly perturbed prompts yields prompts containing only task-relevant causal factors. The proof in Appendix A.2 models each perturbed dataset as X_i = C ⊕ E_i, with E_i an environment-specific confounder perturbed by random prompt annotations P_i, and then argues that the gradient sum over i cancels confounder terms because E_i are uncorrelated across i. This premise is false under the paper's own construction: Section 4.1 defines D_i = {x_i,j, p_i,j} with exactly the same image x_i,j across all prompt groups. Random boxes, points, or masks change the conditioning signal to SAM; they do not alter the generating factors of the image. Any image-level confounder, e.g., deep-sea background texture correlated with class, is therefore identical in every D_i, so E_i = E for all i. The confounder gradient sum in Eq. (8) becomes N_t ∇θ L_E, not approximately zero, and Eq. (9), fθ(P_i) = C + δ_i with δ_i → 0, does not follow. Moreover, the theorem's antecedent, L(f*_phi; X, f_theta(P_i)) ≈ ε for all i, is essentially the L_task objective in Eq. (1); the proof does not independently establish that invariant-under-prompt representations exclude shared image-level confounders. Because Theorem 1 is the paper's stated basis for calling the learned prompts causal, the rejection of the theoretical contribution is justified. The empirical method may still be a useful heuristic, but its causal interpretation is not established.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper addresses open-vocabulary multi-entity segmentation (OVMS) with SAM, arguing that prompt bias is the primary cause of SAM's poor generalization to unseen classes. It introduces a causal framework in which good prompts are 'causal prompts' containing only task-relevant generating factors, and it proposes CPC-SAM, a method that learns such prompts by enforcing segmentation consistency across multiple randomly annotated prompt groups. The method consists of a causal prompt learner (CaPL) with task-level and entity-level calibration modules, optimized jointly with a LoRA-fine-tuned SAM via bi-level optimization. Experiments on nine datasets report consistent improvements over prior SAM-based and OVMS baselines.","tokens_in":29837,"tokens_out":2862,"duration_ms":31095,"significance":"If Theorem 1 were valid, the paper would provide a principled causal justification for prompt calibration in interactive segmentation, going beyond heuristic prompt tuning. The empirical study is extensive: nine datasets, standard and few-shot settings, an OOD transfer setting, ablations of prompt types, hyperparameters, LoRA rank, and a plug-and-play analysis with several SAM variants. The paper also commits to releasing code, which would aid reproducibility. However, the central theoretical claim is load-bearing for the paper's framing: the causal interpretation of the learned prompts rests entirely on Theorem 1 and its proof. Because that proof is invalid under the paper's own data construction, the contribution reduces to an empirical prompt-calibration heuristic with an unsupported causal narrative.","major_comments":[{"comment":"The proof of Theorem 1 assumes that random prompt annotations create independent confounders E_i with X_i = C ⊕ E_i, and then concludes that confounder gradients cancel in the sum over i, giving f_theta(P_i) = C + δ_i with δ_i → 0. This premise is false under the paper's own construction in Section 4.1: each D_i = {X, P_i} contains exactly the same image X. Random boxes, points, or masks change the conditioning signal to SAM; they do not change the generating factors of the image. Any image-level confounder, such as background texture correlated with class in the deep-sea dataset, is therefore identical in every D_i, so E_i = E for all i. Eq. (8) then becomes N_t ∇_θ L_E rather than a sum of uncorrelated terms, and Eq. (9) does not follow. Since Theorem 1 is the stated basis for calling the learned prompts causal, the central theoretical contribution is unsupported.","section":"Section 3.3, Theorem 1; Appendix A.2, Eqs. (7)-(9)"},{"comment":"The theorem's antecedent, L(f*_phi; X, f_theta(P_i)) ≈ ε for all i, is essentially the objective minimized by L_task_CaPL in Eq. (1): minimizing the cumulative segmentation loss across the perturbed prompt groups. The proof does not independently establish that near-zero training loss plus consistency across prompts excludes shared image-level confounders; it asserts that any residual confounder gradient would violate the consistency constraint, which is exactly the point in question. Consequently, the conclusion that the resulting prompts contain 'only causal factors' is an interpretation of the training objective rather than a derived result. This circularity undermines the causal claim in the title and abstract.","section":"Section 3.3 and Eq. (1)"},{"comment":"The motivating experiment attributes the 32.4% performance gap to 'task-irrelevant generating factors acting as confounders.' The experiment compares prompts from a fixed generator with expert-refined prompts, but this manipulation changes many properties at once, including prompt coverage, boundary accuracy, and annotation noise. The results show that prompt quality matters, but they do not isolate a causal confounder mechanism. The paper uses this experiment to justify the SCM and the 'causal prompt' terminology, so the empirical evidence does not provide independent support for the causal interpretation.","section":"Section 3.2, Figure 1b; Appendix F.1"},{"comment":"The justification of the entity causal module states that if f^1_theta is better than f^2_theta then L_entity_CaPL(f^1_theta, f_phi; D) < L_entity_CaPL(f^2_theta, f_phi; D), and that bi-level optimization therefore forces the module to accurately capture the target similarities. This is tautological: the loss defines 'better' in terms of the same segmentation objective, so optimizing it does not establish that the learned attention calibration matrices correspond to true causal factor–entity relationships. The entity-level calibration may be a useful regularizer, but its causal interpretation is not justified by the presented argument.","section":"Section 4.3, Eq. (4)"}],"minor_comments":[{"comment":"The caption lists '(a) Existence of generalization issue' twice and then labels the prompt-bias panel as (b) and (c); the in-text references to panels should be reconciled with the actual figure layout.","section":"Figure 1 caption"},{"comment":"The claim that random prompt perturbations make confounder gradients 'noisy and canceling in expectation' is attributed to [46] (Mohri, Foundations of Machine Learning), but that reference does not establish this property for the present setup; a derivation or a more specific citation is needed.","section":"Appendix A.2, reference [46]"},{"comment":"The qualitative section mentions a 'children's dental panoramic radiographs dataset' that is not described in Appendix B or listed among the nine datasets; either the dataset should be documented or the reference removed.","section":"Appendix F.5"},{"comment":"The OOD column reports single Dice scores without standard deviations, in contrast to the other columns; reporting the average over the five runs with a deviation would be consistent with the rest of the paper.","section":"Table 3"},{"comment":"The theorem statement uses the informal phrase 'is believed to be causal prompts'; for a formal theorem the conclusion should be stated as a precise mathematical property of f_theta(P_i), especially since the proof appendix attempts a formal derivation.","section":"Section 3.3, Theorem 1"}],"recommendation":"reject","confidential_remarks":"The paper's theoretical centerpiece, Theorem 1, is invalid under the paper's own construction because all perturbed datasets share the same image, so the confounder-cancellation argument cannot remove image-level confounders. This is a load-bearing error: without Theorem 1, the paper's causal framing and the name 'causal prompt' are not supported. The empirical results may still describe a useful prompt-calibration heuristic, but reframing the paper as empirical would be a substantial rewrite beyond a minor revision. I therefore recommend rejection, despite the breadth of the experiments."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Honest take: the empirical core is solid and the prompt-bias phenomenon is worth knowing about, but the causal theorem that should animate the paper is not valid as stated. I would send this to referees, but I would not let Theorem 1 pass. What is actually new: the controlled experiment showing expert-refined prompts beating generator prompts by 32.4 Dice on unseen classes is a clean, useful demonstration. The CPC-SAM recipe—random prompt perturbation, consistency reweighting, entity-level sparsity, LoRA, bi-level optimization—is a reasonable engineering combination, and the reported gains on nine datasets are consistent. The plug-and-play experiment, adding CaPL to several SAM baselines and seeing +3.9 to +7.6 Dice, is good evidence that the calibration signal is doing something. The citation pattern looks fine, and the ablations are reasonably thorough. Where it falls apart: Theorem 1. The proof in Appendix A.2 writes X_i = C ⊕ E_i and claims random prompt annotations perturb E_i independently across i. But Section 4.1 defines D_i with exactly the same image X across all prompt groups. Random boxes and points change SAM's conditioning signal, not the image's generating factors. Any image-level confounder (deep-sea background, lighting) is identical in every D_i, so the gradient sum in Eq. (8) does not cancel and Eq. (9) does not follow. On top of that, the theorem's antecedent—loss near zero on all perturbed prompts—is essentially the L_task_CaPL objective, so the proof is close to restating the training loss as a causal conclusion. This is the load-bearing claim, and it does not hold. Smaller issues: Section 3.2 says 46 deep-sea species in OceanCOCO, while Appendix B says 38; the code is promised but not released; the inference-time prompt-generation protocol using a 10% subset needs clarification to rule out label leakage. None of these are fatal by themselves, but they add friction. Bottom line: treat this as an empirical method paper, not a causal theory paper. With the theorem demoted and a few details cleaned up, it is a useful contribution for people building on SAM for open-vocabulary segmentation. I would accept it for peer review and push for major revision, with the causal claims softened.","headline":"The prompt-bias story is real and the empirical package is decent, but the causal theorem that frames the paper does not survive its own setup.","tokens_in":726,"tokens_out":2095,"would_cite":false,"duration_ms":50046,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper's central claim is that reweighting randomly annotated prompts under a consistency loss yields causal prompts containing only task-relevant factors, and that this resolves the Segment Anything Model's open-vocabulary…","keywords":["open-vocabulary multi-entity segmentation","causal prompt","prompt bias","Segment Anything Model","causal multi-distribution consistency","bi-level optimization","prompt calibration","confounder"],"falsifier":"Train CPC-SAM on images in which a background cue, such as sediment color in the deep-sea set, is strongly correlated with the target class, then evaluate on held-out images with that cue removed; a sharp drop in Dice on the target while the same prompts still satisfy the consistency loss would show the learned prompt retained the confounder, refuting the causal reading of Theorem 1.","tokens_in":29206,"feed_emoji":"🎯","tokens_out":13185,"duration_ms":115794,"temperature":0.7,"pith_summary":"The paper aims to show that the reason the Segment Anything Model (SAM) fails to generalize to unseen classes in open-vocabulary multi-entity segmentation is prompt bias, not the vision backbone. It defines the ideal prompt, the causal prompt, as one containing only task-relevant causal factors, and claims that this ideal can be reached without extra labeled data. The load-bearing step is a theorem stating that if the optimally fine-tuned segmentation model attains near-zero loss under every randomly perturbed prompt for the same image, then the reweighted prompts must be causal. The paper builds CPC-SAM, which learns such reweighting through a light causal prompt learner and a bi-level training loop, and reports consistent Dice-score gains over strong baselines on nine datasets. A reader should care because this offers a data-free route to making promptable foundation models reliable in open-vocabulary settings.","feed_headline":"Reweighting random prompts cancels SAM's unseen-class failures","feed_subtitle":"A causal-consistency loss over random prompt annotations removes task-irrelevant factors; Dice rises by up to 13 points OOD.","key_machinery":"The load-bearing object is the causal prompt, a prompt embedding that encodes the task-relevant generating factors $F^p_r$ and is independent of task-irrelevant factors $F^p_{ir}$, defined through an invertible mapping from the sample's causal factors and an irrelevance-exclusion condition. The argument is carried by Theorem 1, which converts loss consistency across randomly annotated prompts into a claim about causal content; its proof uses the decomposition $X_i = C \\oplus E_i$, with $C$ invariant and $E_i$ environment-specific, and the gradient cancellation $\\sum_i \\nabla_\\theta L_{E_i} \\approx 0$ under random perturbations. In the method, the same principle is operationalized by the two-stage Causal Prompt Learner (CaPL): a task causal module reweights prompt embeddings under the cumulative segmentation loss $L^{task}_{\\mathrm{CaPL}}$ over $N_t$ prompt groups, and an entity causal module sparsifies cross-attention matrices so each entity responds mainly to its own factors; the two are coordinated by the bi-level objective in Eq. (4).","core_discovery":"The paper's central claim is Theorem 1 of Section 3.3: if, for every perturbed prompt dataset $D_i = \\{X, P_i\\}$ built from the same samples $X$, the optimal segmentation model $f^*_\\phi$ satisfies $L(f^*_\\phi; X, f_\\theta(P_i)) \\approx \\varepsilon$ with $\\varepsilon \\ge 0$, then the prompt reweighting $f_\\theta(P_i)$ produces causal prompts, i.e., prompts carrying only the causal factors $C$ that drive the label. The argument decomposes each input as $X_i = C \\oplus E_i$ and shows that gradients from the environment-specific confounders $E_i$ are inconsistent across the random prompt distributions, so enforcing consistency drives their contribution to zero and leaves only the invariant causal factors. On this basis the paper proposes CPC-SAM, where the causal prompt learner CaPL generates several random annotations per sample, reweights prompt embeddings at the task level, reweights attention maps at the entity level with an $\\ell^1$ sparsity constraint, and is alternated with a LoRA-fine-tuned SAM through bi-level optimization. The experiments then claim that this calibrated prompt transfers to unseen classes, with the largest reported gains in few-shot and out-of-distribution medical settings, for example about 13 Dice points over the strongest baseline in an OOD chest-X-ray transfer.","pith_inferences":["Beyond the paper: a sharp test of the proof's mechanism would measure the variance of gradient contributions across prompt groups during CaPL training; the theorem predicts confounder gradients approach zero while causal gradients stay aligned.","Beyond the paper: the same consistency argument could be reused for text-prompted vision-language models, with random prompt paraphrases playing the role of the random boxes; that would be a direct transfer of Theorem 1's logic.","Beyond the paper: the entity-level sparsity constraint is a heuristic implementation of disentanglement rather than a consequence of Theorem 1, so ablating it would isolate how much of the gain is causal consistency and how much is architectural regularization."],"forward_implications":["If Theorem 1 is correct, a prompt reweighter trained only to minimize the cumulative segmentation loss over random prompts yields prompts invariant to task-irrelevant factors, with no external annotations or expert refinement.","Open-vocabulary multi-entity segmentation accuracy on unseen classes should improve; the paper reports average Dice gains of 3.9 points in the standard setting and 5.1 points in the few-shot setting over the strongest SAM-based baseline, and about 13 points in an OOD medical transfer.","Entity-level calibration is a necessary part of the claim, since the same prompt affects different entities unequally; the entity causal module's sparsified attention is what links causal factors to the correct entity.","The calibration component is plug-and-play: inserting CaPL after the prompt-optimization stage of SAM, Med-SA, SAMed, BLO-SAM, and OVSAM raises their Dice by at least 3.9 points, which the paper reads as evidence that those methods had not reached causal prompts."],"supporting_citations":[{"why":"supplies the segmentation model whose prompt-induced generalization gap is the problem being solved.","marker":"[29]"},{"why":"supplies the structural causal model and causal-invariance view used to define the causal prompt.","marker":"[47]"},{"why":"provides the causal multi-distribution consistency principle that Theorem 1 relies on.","marker":"[7]"},{"why":"supports the claim that near-optimal loss implies the model has captured the task-essential causal factors.","marker":"[54]"},{"why":"provides the prompt generator and OVMS baseline used in the motivating experiment and comparisons.","marker":"[67]"},{"why":"the bi-level-optimization SAM baseline whose strategy the paper adapts and outperforms.","marker":"[70]"},{"why":"the learnable-prompt SAM baseline that CPC-SAM compares against in prompt-embedding ablations.","marker":"[53]"},{"why":"frames the open-vocabulary segmentation task that the paper targets.","marker":"[68]"},{"why":"cited in the proof as the basis for claiming that random perturbations make confounder gradients inconsistent.","marker":"[46]"},{"why":"the COCO benchmark that underlies the OceanCOCO dataset used to demonstrate the generalization issue.","marker":"[39]"}],"fun_headline_variants":["Causal prompt calibration: SAM's OOV dice rises up to 13 points","Reweight random prompts, strip confounders: SAM sees unseen classes","CPC-SAM: causal consistency erases prompt bias for open-vocabulary","SAM's OVMS fixed by enforcing causal multi-distribution consistency","Prompt reweighting via causal theory boosts SAM out-of-distribution"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The proof assumes that randomly sliding or resizing a prompt box changes only the background, never the object itself, and that the background effects from different random trials cancel out during training.","fun_headline_variants_meta":{"raw":{"variants":["Causal prompt calibration: SAM's OOV dice rises up to 13 points","Reweight random prompts, strip confounders: SAM sees unseen classes","CPC-SAM: causal consistency erases prompt bias for open-vocabulary","SAM's OVMS fixed by enforcing causal multi-distribution consistency","Prompt reweighting via causal theory boosts SAM out-of-distribution"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000281,"raw_usage":{"total_tokens":1757,"prompt_tokens":1127,"completion_tokens":630,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":743,"completion_tokens_details":{"reasoning_tokens":532}},"tokens_in":743,"tokens_out":630,"duration_ms":6494,"temperature":1.0,"reasoning_tokens":532,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T22:40:30.232376+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train CPC-SAM on images in which a background cue, such as sediment color in the deep-sea set, is strongly correlated with the target class, then evaluate on held-out images with that cue removed; a sharp drop in Dice on the target while the same prompts still satisfy the consistency loss would show the learned prompt retained the confounder, refuting the causal reading of Theorem 1.","supporting_citations":[{"cited_title":"Segment any- thing","cited_arxiv_id":null,"evidence_quote":"supplies the segmentation model whose prompt-induced generalization gap is the problem being solved."},{"cited_title":"Causality","cited_arxiv_id":null,"evidence_quote":"supplies the structural causal model and causal-invariance view used to define the causal prompt."},{"cited_title":"Causal invariance as an essential constraint for creating representation of the world: generalizing the invariance of causal power","cited_arxiv_id":null,"evidence_quote":"provides the causal multi-distribution consistency principle that Theorem 1 relies on."},{"cited_title":"A feature-integration theory of attention","cited_arxiv_id":null,"evidence_quote":"supports the claim that near-optimal loss implies the model has captured the task-essential causal factors."},{"cited_title":"Blo-sam: Bi-level optimization based finetuning of the segment anything model for overfitting- preventing semantic segmentation","cited_arxiv_id":null,"evidence_quote":"the bi-level-optimization SAM baseline whose strategy the paper adapts and outperforms."},{"cited_title":"Fourier features let networks learn high frequency functions in low dimen- sional domains","cited_arxiv_id":null,"evidence_quote":"the learnable-prompt SAM baseline that CPC-SAM compares against in prompt-embedding ablations."},{"cited_title":"A simple framework for open-vocabulary segmentation and detection","cited_arxiv_id":null,"evidence_quote":"frames the open-vocabulary segmentation task that the paper targets."},{"cited_title":"Foundations of machine learning, 2018","cited_arxiv_id":null,"evidence_quote":"cited in the proof as the basis for claiming that random perturbations make confounder gradients inconsistent."},{"cited_title":"Microsoft coco: Common objects in context","cited_arxiv_id":null,"evidence_quote":"the COCO benchmark that underlies the OceanCOCO dataset used to demonstrate the generalization issue."}],"review_version":1}