{"id":"cf5cc604-e10d-4350-8b16-fe71c31f5668","arxiv_id":"2506.13831","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"high","formal_verification":"none","parameter_count":2,"one_line_summary":"A rotation-sensitivity hypothesis test plus Varimax rotation produces sparse concept dictionaries from CLIP embeddings and improves worst-group accuracy after spurious concept removal.","lead":"This paper proposes a statistical test for rotation-sensitive structure in CLIP embeddings and a Varimax-based decomposition that turns embeddings into sparse, interpretable concepts while preserving reconstruction fidelity. The practical payoff, if the claims hold, is a principled way to validate concept methods and to remove spurious concepts, with a 22.6% worst-group accuracy gain on Waterbirds.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The null resampling is not the null: under the paper's own Gaussian model, U is Haar on the Stiefel manifold, and row-wise independent rotations (Algorithm 4) destroy the orthogonality constraint, so the reported p-values do not control Type I error.","rationale":"I read the paper's central claim as: the hypothesis test certifies rotation-sensitive structure, and Varimax then gives concepts that are robust rather than method-specific. That claim requires the resampling distribution under H0 to coincide with the distribution of U under Gaussian noise. It does not. The reader's weakest assumption is the same as my concern; I agree. The decomposition and identification material (Theorem 2) may be a reasonable extension of Varimax identifiability, and the reconstruction-fidelity comparison could be useful, but it does not repair the hypothesis test. The spurious-removal experiments are uncontrolled, but the invalid null is the more fundamental issue: it is what the abstract's 'statistical guarantees' rest on. Verdict unchanged (REJECT).","tokens_in":17448,"tokens_out":7141,"duration_ms":78962,"concrete_test":"Run a null-calibration simulation: for B=200 trials, draw A with n=10,000, d=512, entries i.i.d. N(0,1); compute U_k with k=50 and drop the first column as in App. C.2; compute observed TS1/TS2; generate 1,000 Algorithm-4 resamples; record p-values. Under a valid test, p-values are Uniform(0,1), so about 5% of trials should have p<0.05 (95% CI for B=200 roughly 2.2%-8.8%). If the empirical rejection rate exceeds this, or a KS test rejects uniformity, the resampling null is misspecified. As a direct diagnostic, compare the distribution of TS1/TS2 on actual Haar U versus on Algorithm-4 rotations of the same U; any significant difference shows the resampler is not sampling the null.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Proposition 1 (Sec. 2.2) is the only stated justification for Algorithm 4. It assumes i.i.d. rows x_i from a rotationally invariant density; for such rows, rotating each row independently is a valid conditional randomization. But Algorithm 4 is applied to the left singular-vector matrix U, not to raw rows. Under the paper's null (Example 2, proof in App. G.2), U is Haar-distributed on the Stiefel manifold: rows are dependent because they must satisfy U^T U = I_k. Independently rotating each row of U by R_i ~ Uniform(SO_k) yields a matrix whose rows are conditionally independent given their norms and whose columns are almost surely not orthonormal. Such matrices have measure zero under the null. Hence the null distribution used for TS1/TS2 is not the distribution of U under H0, and the Monte-Carlo p-values are not calibrated. Theorem 1 does not fix this: its proof treats the columns as i.i.d. (the 'X_i are i.i.d.' step), but columns of Haar U are dependent, and the normalized statistic TS3 in Eq. (6) is not the TS1 defined in Sec. 2.3. The empirical white-noise controls cannot validate the procedure because the same misspecified resampler is used to generate both null samples and p-values.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a statistical framework for interpreting CLIP embeddings: a hypothesis test for rotation-sensitive structure in the left singular vectors of an embedding matrix, followed by a Varimax-based post-hoc concept decomposition with automatic text labeling. It claims that the test validates that discovered concepts are robust and reproducible rather than method-specific artifacts, that the decomposition achieves better reconstruction fidelity than SpLiCE, and that removing the identified spurious concepts improves Waterbirds worst-group accuracy by 22.6%. The paper also contains an identification theorem for Varimax rotations, a reconstruction-error lower bound for fixed-concept methods, and supporting empirical studies on ImageNet, Waterbirds, iWildCam, and CelebA.","tokens_in":17742,"tokens_out":11078,"duration_ms":124161,"significance":"If the statistical guarantees were valid, the framework would be a useful contribution: it offers a way to validate concept decomposition without retraining, a principled link between rotation invariance and the absence of concept structure, and an external comparison against SpLiCE. The empirical spurious-correlation removal results are interesting and the reconstruction-fidelity comparison is clearly presented. The paper also extends a published Varimax identification result and provides a clean lower bound for fixed-concept methods. However, the central statistical claim is not supported as written because the resampling procedure does not generate draws from the null model it defines, and the proof of the main distributional theorem does not cover the test statistic actually used.","major_comments":[{"comment":"Algorithm 4 does not sample from the null distribution defined by the paper. Under Example 2, which the paper proves in Appendix G.2, the left singular-vector matrix U of Gaussian noise is Haar-distributed on the Stiefel manifold: its rows are dependent and its columns are almost surely orthonormal. Algorithm 4 instead independently rotates each row of U by a uniform rotation in SO_k, producing a matrix whose columns are almost surely not orthonormal; such matrices have measure zero under the null. Consequently, the Monte-Carlo p-values in Algorithm 1 are not calibrated, because TS1 and TS2 are evaluated on matrices that are not draws from the null model. The white-noise control experiments cannot validate the procedure because the same misspecified resampler generates both the null samples and the p-values. A valid resampling for the paper's null would right-multiply U by a single Haar-distributed rotation in SO_k, which preserves the Stiefel constraint.","section":"Section 2.2, Algorithm 4, Proposition 1, Example 2"},{"comment":"The theorem claims a standard normal limit for an equivalent rescaled version of TS1, but the proof does not establish this. Equation (6) defines TS3 using |kurtosis|, but the proof immediately replaces it with n * sum_j U_ji^4 - 3n/(n+2), dropping the absolute value. The proof then states that the relevant variables are i.i.d. and invokes the central limit theorem. However, the columns of a Haar-distributed U are dependent (they are exchangeable, not independent), and the absolute value is a nonlinear transformation that is not accounted for in the rescaling. Thus the proof does not cover the absolute-kurtosis statistic TS1 defined in Section 2.3.","section":"Theorem 1, Eq. (6), Appendix G.6"},{"comment":"The equality characterization in the proof of Theorem 2 is not valid under the stated assumptions. The proof's leading term has coefficient proportional to eta_j - 3*sigma_j^4, and the equality claim requires this coefficient to be strictly positive. Assumption 1 allows eta_j = 3*sigma_j^4 (with equal variances this makes the objective independent of the rotation), so the claimed uniqueness up to permutation can fail. In addition, Assumption 1 states kurtosis kappa >= 3 using the excess-kurtosis convention of Section 2.3, while the proof uses eta_j >= 3*sigma_j^4, which corresponds to raw kurtosis at least 3; these conditions are inconsistent. The theorem needs a strict super-Gaussianity condition stated with the same kurtosis convention used in the paper.","section":"Assumption 1, Theorem 2, Appendix G.7"},{"comment":"The proof that Gaussian-mixture singular vectors are rotation-sensitive contains a numerical error. For the mixture (1/2)N(1,1) + (1/2)N(-1,1), each entry has variance 2, not 1, so Var((Av)_i) = Var((Av')_i) = 2 for the two directions considered in the proof. The claim that the variance of (Av')_i is 'apparently smaller than 1' is therefore false, and the proof does not establish the stated rotation sensitivity as written.","section":"Example 4, Appendix G.4"}],"minor_comments":[{"comment":"The claimed 22.6% improvement on Waterbirds is the difference between the full zero-shot worst-group accuracy (38.1) and the spurious-removed accuracy (60.7); the baseline used for the comparison should be stated explicitly in the text, since Table 2 also reports an SVD-reconstructed baseline of 39.0.","section":"Section 5.3 / Table 2"},{"comment":"Algorithm 2 performs Varimax on U D, whereas the hypothesis test in Section 2 operates on U. The relationship between the two objects should be clarified, especially because the theoretical identification result in Theorem 2 is stated for Z, not for U D.","section":"Section 3 / Algorithm 2"},{"comment":"References [23] and [24] appear to cite the same work in two versions; they should be consolidated to avoid duplicate entries.","section":"References [23] and [24]"},{"comment":"The paper says its method 'outperforms other techniques in terms of reconstruction error' but only compares against SpLiCE. The claim should be restricted to the methods actually evaluated, or additional baselines should be included.","section":"Section 5.2"},{"comment":"The notation in the proof of Theorem 1 switches between kurtosis of columns and sums of fourth powers without defining the centered or scaled quantities consistently; this makes it difficult to verify the moment calculations.","section":"Appendix G.6"}],"recommendation":"major_revision","confidential_remarks":"The empirical concept-decomposition pipeline is interesting and may survive a corrected resampling procedure, but the statistical claims are currently not supported. The most important fix is to replace Algorithm 4 with a Stiefel-preserving rotation and rework Theorem 1; if the authors cannot provide a valid null resampling, the paper should be rejected. The spurious-concept removal results depend on dataset-specific heuristic detection strategies, so the automated selection procedure should be described in enough detail for the results to be reproduced independently."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know two things about this paper. First, the hypothesis test that the statistical-guarantee claim rests on is not valid as written. Second, the concept-decomposition part is actually decent: the Varimax approach preserves reconstruction fidelity, and the identification theory extends earlier work in a real way. It's a mixed bag, not a reject on all counts.\n\nWhat's new is the combination of a rotation-sensitivity bootstrap with Varimax rotation for CLIP embeddings; I don't see that in the cited baselines. Theorem 2 generalizes Rohe-Zeng to unequal variances, which is a legitimate extension even though it's the author's own prior work. The paper also gives a neat lower bound (Theorem 3) for fixed-vocabulary methods like SpLiCE, and the qualitative concept clusters look coherent.\n\nThe soft spot is load-bearing. Proposition 1 assumes i.i.d. rows from a rotationally invariant density. Algorithm 4 applies it to the rows of U, the left singular vectors, but under the paper's own Gaussian null U is Haar-distributed on the Stiefel manifold: rows are dependent, and columns are orthonormal. Rotating each row independently destroys orthonormality, so the null samples are not draws from the null model. The p-values from that bootstrap don't control Type I error, and the white-noise control experiments can't fix that because they use the same misspecified resampler. Theorem 1 also isn't proven for the statistic it states: the appendix derives a CLT for a non-absolute, centered version, not for the absolute-kurtosis TS1.\n\nThe empirical section is weaker than the rest: Table 2 has no error bars, no validation protocol, and doesn't say how many concepts were removed. The 22.6% worst-group improvement is a demonstration, not a controlled estimate.\n\nWhat's good: the decomposition itself is sensible, the sparsity-fidelity trade-off plots are informative, and the identifiability theory in Section 4 stands. Strip out the hypothesis-test claims and there's useful material here.\n\nThis paper is for people doing concept-based interpretability on CLIP-like embeddings. It deserves a serious referee, but the referee should focus on the resampling null before anything else. With that fixed, or with the test reframed honestly as a heuristic, the paper could be salvageable. I'd send it to review with that issue flagged.","headline":"The rotation test is broken as written, but the decomposition and identification theory are worth a second look.","tokens_in":18268,"tokens_out":4164,"would_cite":false,"duration_ms":42658,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["62H25","62F03","68T07"],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that meaningful concepts in CLIP embeddings are directional structures that break rotational invariance, and that this property can be turned into a statistically grounded post-hoc decomposition, with removing flagged…","keywords":["CLIP embeddings","concept decomposition","Varimax rotation","rotation invariance","hypothesis testing","spurious correlation","reconstruction fidelity","interpretability"],"falsifier":"Generate an $n \\times d$ matrix with i.i.d. standard Gaussian entries, which the paper itself treats as its null model, compute the truncated left singular vectors $U$, and run the paper's test procedure (Algorithm 4 resampling plus TS1/TS2). If the row-wise rotation is a valid null, p-values across many independent datasets should be approximately uniform over $[0,1]$; if the null is invalid, small p-values will appear far more often than 5% of the time.","tokens_in":17225,"feed_emoji":"🔄","tokens_out":6726,"duration_ms":65813,"temperature":0.7,"pith_summary":"The paper tries to establish that meaningful concepts inside CLIP embeddings are directional structures that break rotational invariance, and that this property can be turned into a hypothesis test. It argues that random noise is statistically unchanged by rotation, whereas true semantic concepts prefer certain directions, so a test that rejects rotation invariance signals real structure. On top of the test, the paper develops a post-hoc decomposition that applies Varimax rotation to the singular vectors of the embedding matrix, yielding sparse, interpretable concept loadings while keeping near-SVD reconstruction fidelity. If the central claim is right, concept decompositions can be validated statistically and corrected without retraining, and the paper reports a 22.6% improvement in worst-group accuracy on Waterbirds after removing the concepts it flags as spurious.","feed_headline":"Rotation test validates CLIP concepts, lifting accuracy 22.6%","feed_subtitle":"Statistical null separates real concept directions from noise, letting spurious cues be removed without retraining.","key_machinery":"The load-bearing object is the left singular vector matrix $U$ of the normalized embedding matrix, together with the Varimax objective $v(U,R)$, which measures the maximum sparsity achievable by rotating the columns. The test generates a null distribution by independently rotating each row of $U$ with a random orthogonal matrix (Algorithm 4), then compares TS1 and TS2 against that null to produce p-values. The decomposition uses the Varimax-rotated SVD factors: image loadings are $\\hat{Z} = UDR$ and the concept dictionary is $\\hat{Y} = VR$, so concepts are orthogonal directions that are sparse over data points and interpretable through top-loading images and text descriptions.","core_discovery":"The central claim is that rotation-sensitivity testing plus Varimax rotation yields concepts that represent robust, reproducible patterns rather than method-specific artifacts, while preserving SVD-level reconstruction fidelity. The paper formalizes \"no concept structure\" as a null model in which singular vectors are rotationally invariant, and proposes two test statistics: average absolute kurtosis across singular-vector columns (TS1) and the maximum Varimax sparsity objective over rotations (TS2). It then decomposes the embedding matrix as sparse loadings times an orthogonal concept dictionary obtained by rotating the SVD factors, and proves that under stated assumptions the Varimax objective recovers the true concept rotation up to permutation and sign. It also proves a lower bound showing that fixed, pre-defined concept vocabularies suffer unavoidable reconstruction error when misspecified, which motivates learning concepts from data. Empirically, the paper reports that removing the spurious concepts identified by the method improves Waterbirds worst-group accuracy by 22.6% and triples iWildCam prediction accuracy.","pith_inferences":["A natural next check, not reported in the paper, is whether the row-wise rotation null yields calibrated p-values when $U$ is drawn from the paper's own Gaussian null; under that null $U$ is Haar-distributed on the Stiefel manifold, so the independent row rotations may not match the null distribution.","If calibrated, the same test could be applied to other foundation models' embeddings to audit which interpretable directions are statistically real rather than artifacts of a particular SVD run.","The demonstrated concept arithmetic (group-of-dogs minus single-dog plus single-bird retrieving groups of birds) suggests concepts behave like word embeddings do for analogies, which could be tested systematically across concept pairs.","Removing spurious concepts improved worst-group accuracy on three benchmarks, which points toward a general bias-mitigation recipe, but the paper does not examine whether removal degrades other axes of behavior, so that trade-off is an open question."],"forward_implications":["Concepts extracted by the method come with a p-value for rotation-sensitive structure, so a practitioner can tell whether a reported concept is distinguishable from noise.","Because reconstruction fidelity stays near the SVD baseline at moderate concept counts, downstream zero-shot classification can be run in the concept space with little information loss.","Deleting the loadings of flagged spurious concepts is a retraining-free intervention that the paper shows lifts Waterbirds worst-group accuracy by 22.6% and triples iWildCam accuracy.","The reconstruction lower bound for fixed concept vocabularies implies that word-based or otherwise pre-specified concept dictionaries will leave an irreducible error whenever the true concepts are not a projection of that dictionary.","The identification theorem implies that when loadings are independent and super-Gaussian, the Varimax rotation finds the true concept axes up to permutation and sign, which is what makes the decomposition reproducible across runs."],"supporting_citations":[{"why":"Supplies the CLIP embedding model whose internal representations are the object of study.","marker":"[21]"},{"why":"Provides the theory that a Varimax rotation of singular vectors recovers sparse, interpretable factor structure, which the paper extends.","marker":"[23]"},{"why":"Defines SpLiCE, the fixed-vocabulary sparse decomposition that serves as the main baseline for reconstruction fidelity comparisons.","marker":"[3]"},{"why":"Supplies the curated text description set and the text-based decomposition approach used to interpret the discovered concepts.","marker":"[6]"},{"why":"Introduces the Waterbirds benchmark on which removing spurious concepts yields the reported 22.6% worst-group accuracy improvement.","marker":"[25]"},{"why":"Defines the Varimax criterion that the paper uses both as a test statistic and as the rotation objective in its decomposition.","marker":"[11]"},{"why":"Establishes that learned representations are identifiable only up to a linear map, motivating the rotation-based analysis of embedding axes.","marker":"[22]"}],"fun_headline_variants":["Statistical test confirms CLIP concepts, boosts worst-group accuracy","Varimax rotation yields robust CLIP concepts, +22.6% accuracy","Rigorous concept detection in CLIP spaces lifts accuracy 22.6%","Hypothesis testing for CLIP concepts cuts spurious bias, +22.6%","Statistical null for CLIP concept structure, remove spurious cues"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The test's validity rests on one premise: that independently spinning each row of the singular-vector matrix around a random rotation mimics what the matrix would look like if the data had no preferred directions, and if that simulated \"noise\" is not the right comparison, every p-value in the paper is on shaky ground.","fun_headline_variants_meta":{"raw":{"variants":["Statistical test confirms CLIP concepts, boosts worst-group accuracy","Varimax rotation yields robust CLIP concepts, +22.6% accuracy","Rigorous concept detection in CLIP spaces lifts accuracy 22.6%","Hypothesis testing for CLIP concepts cuts spurious bias, +22.6%","Statistical null for CLIP concept structure, remove spurious cues"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001517,"raw_usage":{"total_tokens":6064,"prompt_tokens":916,"completion_tokens":5148,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":532,"completion_tokens_details":{"reasoning_tokens":5050}},"tokens_in":532,"tokens_out":5148,"duration_ms":37166,"temperature":1.0,"reasoning_tokens":5050,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T00:38:20.463159+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Generate an $n \\times d$ matrix with i.i.d. standard Gaussian entries, which the paper itself treats as its null model, compute the truncated left singular vectors $U$, and run the paper's test procedure (Algorithm 4 resampling plus TS1/TS2). If the row-wise rotation is a valid null, p-values across many independent datasets should be approximately uniform over $[0,1]$; if the null is invalid, small p-values will appear far more often than 5% of the time.","supporting_citations":[{"cited_title":"Vintage factor analysis with varimax performs statistical inference","cited_arxiv_id":null,"evidence_quote":"Provides the theory that a Varimax rotation of singular vectors recovers sparse, interpretable factor structure, which the paper extends."},{"cited_title":"Interpreting clip’s image representation via text-based decomposition","cited_arxiv_id":null,"evidence_quote":"Supplies the curated text description set and the text-based decomposition approach used to interpret the discovered concepts."},{"cited_title":"The varimax criterion for analytic rotation in factor analysis","cited_arxiv_id":null,"evidence_quote":"Defines the Varimax criterion that the paper uses both as a test statistic and as the rotation objective in its decomposition."},{"cited_title":"On linear identifiability of learned representations","cited_arxiv_id":null,"evidence_quote":"Establishes that learned representations are identifiable only up to a linear map, motivating the rotation-based analysis of embedding axes."}],"review_version":1}