{"id":"19a9aed5-5988-429b-97f0-d581da66fd4d","arxiv_id":"2608.08418","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"DeepMORSE learns a single self-expressive coefficient matrix across image and text embeddings and regularizes each modality into a union of subspaces, improving image clustering on several CLIP-based benchmarks.","lead":"DeepMORSE is a clustering method that learns one shared self-expressive coefficient matrix for both image and text CLIP representations while shaping each modality into class-specific low-dimensional subspaces. It reports large clustering gains on three benchmarks and competitive transfer to retrieval and zero-shot classification, with code released.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Theory analyzes the least-squares C in Eq. (7), but the deployed C in Eq. (17) is a signed Sinkhorn projection of Z_mix^T Z_mix; no proof transfers the noise-suppression or implicit-regularization guarantees, and the final clustering uses |Z_img^T Z_img| instead.","rationale":"The reader's verdict is CONDITIONAL and I agree. The reason this concern is load-bearing rather than cosmetic is that contribution 2 of the paper—the theoretical justification for suppressing inter-class noise and the implicit regularization—is what distinguishes DeepMORSE from a heuristic combination of Sinkhorn affinity and rate reduction. If the actual C in Eq. (17) does not satisfy the inequalities proven for the least-squares C, then the paper's stated mechanism ('the modality-shared self-expressive coefficients suppress inter-class noise') is not established for the method that is trained and evaluated. The final use of |Z_img^T Z_img| for clustering makes the link even more indirect, because the reported ACC/NMI do not directly measure the quality of C. None of this invalidates the empirical results: the ablations, transfer experiments, and Figure 4 show real behavior and are valuable. But the central claim of a principled, theoretically grounded model currently rests on a bridge between Eq. (7) and Eq. (17) that the paper does not build. The proposed retraining check with the exact least-squares C would determine whether the Sinkhorn projection is a benign computational surrogate (gap is only a missing proof) or a load-bearing component with properties outside the theory (gap is substantive). If the latter, the paper should either extend the theory to Sinkhorn-style projections or substantially soften the theoretical claims and reposition the contribution as empirical with a heuristic justification. I do not raise the strong covariance assumption in Prop. 1 as the primary issue, because it is a common simplifying condition; the Sinkhorn gap is more fundamental.","tokens_in":29647,"tokens_out":13354,"duration_ms":131580,"concrete_test":"Retrain DeepMORSE on ImageNet-Dogs and DTD-47 with Eq. (17) replaced by the exact mini-batch least-squares solution of Eq. (7) (per-column leave-one-out ridge solution with λ = 2(N−n_b)/n_b from Prop. 3), keeping all other settings fixed, and report ACC/NMI for spectral clustering on both |Z_img^T Z_img| and |C|+|C^T|. If the least-squares variant performs far worse or diverges, the reported gains depend on the untheorized Sinkhorn C; if it performs comparably, the Sinkhorn C behaves as a proxy for the analyzed solution, and the gap is a missing proof rather than a substantive mismatch.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central theoretical contribution (Prop. 1, Section III-B) is a guarantee that the modality-shared self-expressive coefficients obtained by solving the least-squares subproblem (7) suppress inter-class noise relative to modality-specific solutions (inequality (10)). However, the deployed DeepMORSE never solves (7). In Eq. (17), C is defined as a signed Sinkhorn-Knopp projection of Z_mix^T Z_mix, with Z_mix = (Z_img + Z_text)/2. Sinkhorn-Knopp is an entropy-regularized optimal-transport row/column normalization; it is not the minimizer of the self-expressive objective, and no proposition or argument in the paper shows that this C inherits inequality (10) or the implicit-regularization result of Prop. 3. The training loss (18) uses this Sinkhorn C, so the gradients that shape Z_img and Z_text are driven by coefficients whose subspace-preservation properties are unproven. Furthermore, the test-time procedure (Section III-C) discards C and applies spectral clustering to |Z_img^T Z_img|, so the affinity actually clustered on is not the matrix the theory analyzes. Prop. 3 contains an additional disconnect: it analyzes a mini-batch scheme that solves for C in subproblem (14), whereas in the actual forward pass C is a deterministic function of Z_mix, not a solved variable; the empirical check in Fig. 5 therefore measures a gap for the implemented C that the derivation does not cover without extra assumptions. The empirical ablations and Fig. 4 provide suggestive evidence, but they do not supply the missing link between the least-squares theory and the Sinkhorn implementation.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes DeepMORSE, a deep subspace-clustering method that uses a modality-shared self-expressive coefficient matrix to align image and text representations, while learning structured representations that form modality-specific unions of subspaces. The model is trained by minimizing a self-expressive reconstruction loss plus a negative coding-rate regularization, with coefficients reparameterized via a signed Sinkhorn-Knopp projection of the modality-mixed Gram matrix. The authors provide theoretical results on noise suppression (Proposition 1), non-collapse of representations (Proposition 2), and implicit regularization (Proposition 3), and they report extensive experiments on six image clustering benchmarks, achieving state-of-the-art accuracy on five of them, along with strong transfer to image retrieval and zero-shot classification.","tokens_in":30096,"tokens_out":7824,"duration_ms":80420,"significance":"If the empirical results hold, DeepMORSE is a competitive and simple method for language-assisted image clustering, with clear gains over existing multimodal baselines and useful transfer to downstream tasks. The paper is commendable for its thorough experimental protocol: multi-seed averages, ablations over components, sensitivity analyses, time comparisons, and comparisons to many modern baselines, plus publicly available code. The theoretical framework, even with the gaps noted below, offers a plausible mechanism for why sharing self-expressive coefficients across modalities helps, and the implicit-regularization analysis is insightful. A caveat is that the textual counterparts are generated from the image embeddings themselves via sparse coding, so the cross-modal benefit may partly reflect the sparse-coding prior rather than independent textual information; the ablations with MLLM-generated text partially mitigate this concern.","major_comments":[{"comment":"The theoretical analysis in Propositions 1 and 3 studies the least-squares self-expressive subproblem (7), whose closed-form solution is given in Eq. (20), but the implemented DeepMORSE computes C as the signed Sinkhorn-Knopp projection C = P(Z_mix^T Z_mix) in Eq. (17). This is not the minimizer of the self-expressive objective, and the paper provides no proof or argument that this reparameterized C inherits the noise-suppression property of inequality (10) or the implicit regularization of Proposition 3. Since the training loss (18) uses this Sinkhorn-based C, the gradients that shape the representations are driven by coefficients whose theoretical guarantees are not established. The authors should either extend the theory to cover the Sinkhorn reparameterization or clearly state that the theoretical results apply to an idealized version of the model, relying on the empirical validation in Figure 4 for the actual method.","section":"Section III-B (Eq. 7) vs. Section III-C (Eq. 17)"},{"comment":"The proof of Proposition 1 requires the cross-covariance condition E[delta_img delta_text^T] = 0 to drop the cross terms in Eq. (27), but the assumptions of Proposition 1 state only that both deviation vectors have zero mean and covariance sigma^2 I_d. Without the uncorrelatedness assumption, the error covariance of c_share includes additional terms and inequality (9) may fail. The assumption should be stated explicitly, or the proof should handle the correlated case.","section":"Supplementary Material, proof of Proposition 1 (Eq. 27)"},{"comment":"The final clustering is performed by spectral clustering on the affinity |Z_img^T Z_img|, not on the learned shared self-expressive matrix C. The theoretical guarantee in Proposition 1 concerns the coefficients C, so the paper does not establish that the affinity actually used inherits the subspace-preserving or noise-suppressing properties. The authors should either use a clustering affinity derived from C (e.g., a symmetrized absolute value of C) or provide an explicit argument and experimental evidence that the inner-product affinity of the learned image representations preserves the class structure.","section":"Section III-C, 'Evaluation on Test Data'"}],"minor_comments":[{"comment":"The header 'Proof for Proportion 1' should read 'Proof for Proposition 1'.","section":"Supplementary Material, header"},{"comment":"Table B.1 lists hyperparameters for STL-10 and ImageNet-10, which do not appear among the six benchmarks in Table I; please clarify or remove these rows to avoid confusion.","section":"Appendix B, Table B.1"},{"comment":"The conclusion states that DeepMORSE achieves state-of-the-art clustering performance on six benchmarks, but Table I shows it is not state-of-the-art on CIFAR-10; the wording should be 'five of six' or 'competitive on all six'.","section":"Section V, Conclusion"},{"comment":"The textual counterparts are generated from the image embeddings themselves via sparse coding, so the phrase 'textual information' should be qualified. The ablations in Tables II and V help, but the main narrative should acknowledge that the text modality is constructed from the image query and an external text dictionary.","section":"Section III-C, Eq. (15)"}],"recommendation":"major_revision","confidential_remarks":"The empirical work is solid and the paper is likely to be a good fit for the journal once the theory-practice gap is addressed. The major concerns are the mismatch between the analyzed least-squares subproblem and the deployed Sinkhorn reparameterization, the missing assumption in Proposition 1, and the disconnect between the learned C and the affinity used for final clustering. These issues are fixable but require substantive revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe key thing to know: this is a paper with a genuinely new construction and solid-looking experiments, but its theory does not actually cover the method it analyzes. The paper deserves a serious referee, but it needs a major revision before the theoretical claims can be trusted.\n\nWhat's new: sharing a self-expressive coefficient matrix across image and text modalities, with each modality having its own subspace structure. That's a clean idea, and the results—3–5% ACC gains on three benchmarks, competitive transfer to retrieval and zero-shot—are well-ablated and reported with multiple seeds. The ablation showing the importance of the rate regularization and the tradeoff vs. TAC is useful.\n\nWhere it's soft: the stress-test note is on target. Proposition 1 analyzes the least-squares solution of the subproblem in eq. (7), but the actual C is a signed Sinkhorn projection of Z_mix^T Z_mix (eq. 17). There is no proof that this projection inherits the noise-suppression or subspace-preserving properties of the least-squares solution. Worse, test-time clustering does not use C at all; it uses |Z_img^T Z_img|. So the theory and the implementation are two different methods. Proposition 3 has the same issue: it derives an implicit regularizer for a mini-batch least-squares solver, but the forward pass computes C as a deterministic function of Z_mix. Figure 5's empirical check is suggestive, but it is measuring the gap for the implemented C, not the solved C in the derivation.\n\nOne more soft spot, in proportion: the text modality is generated from the image via sparse coding (eq. 15). That means the \"shared structure\" is partly constructed by the method itself, not discovered from an independent text source. This is not fatal—the dictionary is from WordNet and the selection uses image-text similarity—but it weakens the claim of cross-modal discovery.\n\nBottom line: the empirical method is interesting and the experiments are careful enough that I'd send this to review. But I'd flag the theory-practice gap as a load-bearing issue. Either bridge it (analyze the Sinkhorn projection, or connect C to the affinity used at test time) or soften the theoretical claims to what is actually shown. For a reader working on multimodal representation learning, the empirical results are worth a look.\n\nRecommendation: accept for serious peer review, expect heavy revision.","headline":"A useful but over-claimed theory wraps a solid empirical contribution; referee it, but expect the theory claims to need shrinking.","tokens_in":30544,"tokens_out":3212,"would_cite":true,"duration_ms":31919,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"DeepMORSE claims that sharing one self-expressive coefficient matrix across image and text embeddings, while shaping each modality into a union of subspaces, improves clustering and transfers to retrieval and zero-shot classification.","keywords":["image clustering","multimodal representation learning","subspace clustering","self-expressive model","union of subspaces","vision-language models","textual counterpart generation","implicit regularization"],"falsifier":"Take a dataset with captions, corrupt a fraction of the text counterparts by swapping them with captions from other classes, and measure the inter-class energy of the learned coefficients and the clustering accuracy. The theory predicts the shared least-squares coefficients suppress this added noise; if the implemented Sinkhorn-based coefficients show no suppression, or accuracy falls as fast as an image-only model, the bridge between the theorem and the implementation is broken.","tokens_in":29441,"feed_emoji":"🖼️","tokens_out":9367,"duration_ms":90672,"temperature":0.7,"pith_summary":"The paper sets out to show that image clustering with textual information works better when the two modalities are not directly aligned but are coupled by one shared set of self-expressive coefficients, while each modality is shaped into its own union of low-dimensional subspaces. The proposed method, DeepMORSE, jointly learns image and text embeddings together with a shared coefficient matrix $C$ that expresses each point as a linear combination of other points with the same weights in both modalities. The authors prove that sharing coefficients across modalities suppresses inter-class noise relative to either modality alone, and that mini-batch training induces an implicit regularization on $C$. If the claim is right, multimodal clustering no longer has to trust the often unreliable raw similarities of pretrained vision-language embeddings. On six benchmarks the paper reports state-of-the-art accuracy except on CIFAR-10, with gains of 3.8, 3.2, and 4.9 accuracy points on DTD-47, UCF-101, and ImageNet-Dogs.","feed_headline":"Shared self-expression lifts image clustering by up to 4.9%","feed_subtitle":"Giving image and text embeddings a common subspace structure beats direct cross-modal alignment on five of six benchmarks.","key_machinery":"The load-bearing object is the modality-shared self-expressive coefficient matrix $C$: for every data point, the same weights must express the image embedding and its textual counterpart as linear combinations of the other points. In the implementation $C$ is reparameterized rather than solved directly: it is the signed Sinkhorn-Knopp projection of $Z_{\\text{mix}}^\\top Z_{\\text{mix}}$, where $z_{\\text{mix}} = (z_{\\text{img}}+z_{\\text{text}})/2$, with diagonal entries removed; the projection balances rows and columns of the similarity matrix and then restores the original signs. Textual counterparts themselves are produced by sparse coding each image embedding against a concept dictionary of text embeddings, so each text is a sparse combination of semantically relevant concepts. Around this object, the self-expression losses enforce cross-modal agreement on structure while the negative total coding rates $-\\rho(Z_{\\text{img}})-\\rho(Z_{\\text{text}})$ prevent collapsed solutions and push each modality toward a union of subspaces. Propositions 1-3 supply the theoretical support: shared coefficients suppress inter-class noise, optimal modality-specific representations are non-collapsed with equal singular-value profiles, and mini-batch optimization introduces an implicit regularizer $2(N-n_b)/n_b\\,\\|C\\|_F^2$.","core_discovery":"DeepMORSE's central claim is that a single coefficient matrix shared by image and text modalities recovers the partition structure that is invariant across modalities and suppresses modality-specific noise. With fixed representations, the shared least-squares coefficients solve $\\min_{c_j} \\|z^j_{\\text{img}}-Z^{-j}_{\\text{img}}c_j\\|_2^2 + \\|z^j_{\\text{text}}-Z^{-j}_{\\text{text}}c_j\\|_2^2$, and Proposition 1 shows these coefficients are closer to the true modality-invariant coefficients than either modality's own solution, with no larger expected energy on inter-class entries. The embeddings are trained so that each modality approximately occupies a union of subspaces, using negative total coding rate terms to prevent collapse, and the final clusters are obtained by spectral clustering on the image-only affinity $|Z_{\\text{img}}^\\top Z_{\\text{img}}|$. The paper reports clustering accuracy improvements over prior multimodal methods on five of six benchmarks and shows the learned image representations transfer to retrieval and zero-shot classification without task-specific losses or post-processing.","pith_inferences":["Editorial inference: the theory and implementation are bridged by an unproven step, because Proposition 1 analyzes the closed-form least-squares solution while training computes $C$ through a signed Sinkhorn-Knopp projection; a head-to-head comparison of the two coefficient solvers on the same learned representations would test whether the implementation inherits the noise-suppression guarantee.","Editorial inference: if the shared-coefficient mechanism is right, the same pretraining should also improve other intra-modal tasks hurt by unreliable pretrained vision-language geometry, such as nearest-neighbor few-shot classification, by using the learned structured image representations.","Editorial inference: the implicit regularization depends on batch size, so smaller batches should act as a stronger coefficient regularizer; measuring cluster purity versus batch size would give a direct, tunable test of Proposition 3.","Editorial inference: the union-of-subspaces geometry suggests a way to inspect clusters: each class corresponds to a low-dimensional subspace, so class structure could be probed by principal angles between subspaces rather than by centroids."],"forward_implications":["Textual information can improve image clustering even when raw image-text similarities are unreliable, because the shared coefficients enforce structural agreement instead of direct embedding alignment.","If the central claim holds, the learned image representations are reusable as-is for retrieval and zero-shot classification, so one unsupervised training phase serves several downstream tasks.","Mini-batch training supplies an implicit $\\|C\\|_F^2$ regularization, making explicit coefficient regularization unnecessary and explaining why added coefficient penalties have little effect.","The framework is modality-agnostic: additional modalities with a shared embedding space can be mixed into $z_{\\text{mix}}$ and trained with the same loss, with modest gains or no degradation."],"supporting_citations":[{"why":"Defines self-expressive subspace clustering, the model that the paper extends from one modality to a shared multi-modality coefficient.","marker":"[60]"},{"why":"Supplies the deep subspace clustering framework with total coding rate regularization and the non-collapse theory that DeepMORSE generalizes.","marker":"[72]"},{"why":"Provides the pretrained vision-language encoder whose image and text embeddings are used to build the concept dictionary and textual counterparts.","marker":"[83]"},{"why":"Gives the external-guidance baseline and the concept dictionary construction scheme that DeepMORSE adapts for textual counterpart generation.","marker":"[28]"},{"why":"Introduces the Sinkhorn-Knopp projection used to compute the shared coefficient matrix in the reparameterized implementation.","marker":"[87]"},{"why":"Proposes learning self-expressive coefficients with a network; its soft-thresholding variant is compared against the signed Sinkhorn projection in an ablation.","marker":"[68]"},{"why":"Shows how to reparameterize affinities with a Sinkhorn projection and is the source of the modality-mixed reparameterization idea.","marker":"[10]"},{"why":"Documents the unreliable intra-modal geometry of pretrained vision-language embeddings, motivating structured representations and providing retrieval comparisons.","marker":"[30]"},{"why":"Matching pursuit algorithm solves the sparse coding problem that generates each textual counterpart from the concept dictionary.","marker":"[84]"}],"fun_headline_variants":["Modality-shared self-expressiveness improves image clustering","Shared subspace model aligns images and text for clustering","DeepMORSE: a common self-expressiveness for image-text clusters","One shared coefficient matrix for image-text clustering","Modality-shared subspaces yield transferable image clusters"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the coefficient matrix actually computed in training, a signed Sinkhorn-Knopp projection of the modality-mixed similarity, inherits the noise-suppression and structure-preservation properties that are proven only for the closed-form least-squares solution, and that clustering from the image-only affinity still reflects those shared coefficients.","fun_headline_variants_meta":{"raw":{"variants":["Modality-shared self-expressiveness improves image clustering","Shared subspace model aligns images and text for clustering","DeepMORSE: a common self-expressiveness for image-text clusters","One shared coefficient matrix for image-text clustering","Modality-shared subspaces yield transferable image clusters"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00022,"raw_usage":{"total_tokens":1502,"prompt_tokens":1057,"completion_tokens":445,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":673,"completion_tokens_details":{"reasoning_tokens":365}},"tokens_in":673,"tokens_out":445,"duration_ms":4622,"temperature":1.0,"reasoning_tokens":365,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T04:36:13.134750+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a dataset with captions, corrupt a fraction of the text counterparts by swapping them with captions from other classes, and measure the inter-class energy of the learned coefficients and the clustering accuracy. The theory predicts the shared least-squares coefficients suppress this added noise; if the implemented Sinkhorn-based coefficients show no suppression, or accuracy falls as fast as an image-only model, the bridge between the theorem and the implementation is broken.","supporting_citations":[{"cited_title":"Sparse subspace clustering,","cited_arxiv_id":null,"evidence_quote":"Defines self-expressive subspace clustering, the model that the paper extends from one modality to a shared multi-modality coefficient."},{"cited_title":"Sinkhorn distances: Lightspeed computation of opti- mal transport,","cited_arxiv_id":null,"evidence_quote":"Introduces the Sinkhorn-Knopp projection used to compute the shared coefficient matrix in the reparameterized implementation."},{"cited_title":"Learning a self-expressive network for subspace clustering,","cited_arxiv_id":null,"evidence_quote":"Proposes learning self-expressive coefficients with a network; its soft-thresholding variant is compared against the signed Sinkhorn projection in an ablation."},{"cited_title":"Cross the gap: Exposing the intra-modal misalignment in clip via modality inversion,","cited_arxiv_id":null,"evidence_quote":"Documents the unreliable intra-modal geometry of pretrained vision-language embeddings, motivating structured representations and providing retrieval comparisons."},{"cited_title":"Matching pursuits with time-frequency dictionaries,","cited_arxiv_id":null,"evidence_quote":"Matching pursuit algorithm solves the sparse coding problem that generates each textual counterpart from the concept dictionary."}],"review_version":1}