{"id":"841b73ca-6523-43c9-9577-5a68492acc00","arxiv_id":"2412.18930","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"CgMCR2 jointly learns image embeddings and cluster assignments by adding a differentiable normalized-cut loss to the maximal coding rate reduction objective, improving clustering accuracy on CLIP and MoCo-v2 features.","lead":"The paper combines maximal coding rate reduction with a differentiable graph-cut clustering loss so that image embeddings and cluster assignments are trained together instead of in separate stages. It reports higher clustering accuracy than prior representation-based methods on several standard and out-of-domain image datasets.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Soft-label Rc in Eq. (8) is an unproven relaxation of MCR2 theory; it is the only coupling between feature and cluster heads, so the 'principled' joint-optimization claim rests on it.","rationale":"The reader identifies the soft-label use of MCR2 as the weakest assumption, and I agree. This is load-bearing because Eq. (8) is presented as a unified principled objective, and Rc is the only place where the cluster membership influences the learned embedding in the fine-tuning stage, since the affinity graph in L_Ncut is detached from Theta. Without a valid soft-label Rc, the framework reduces to a spectral-clustering warm-up plus a generic representation loss, and the claimed 'graph cut-guided MCR2' novelty is unsupported. I am not claiming the method is wrong: the empirical results, ablations, released code, and visualizations are real evidence that the pipeline works. The issue is that the theoretical framing is ahead of the analysis. A synthetic test, as described, would settle whether the relaxation is benign or merely a heuristic. I therefore keep the CONDITIONAL verdict rather than rejecting: the empirical claims are plausible, but the paper should either prove or explicitly derive the soft-label relaxation, or reposition the contribution as a heuristic with strong empirical support.","tokens_in":18074,"tokens_out":8259,"duration_ms":83275,"concrete_test":"On synthetic data drawn from a known union of two orthogonal subspaces, train only the fine-tuning objective Eq. (8) from random initialization with a Gumbel-Softmax head and measure the final subspace angle and the within-cluster coding rate. Separately, compute Rc for the uniform soft assignment Pi = 1/k on a fixed Z; if Rc_soft equals R(Z), the soft relaxation admits a trivial zero-gradient solution for the MCR2 term. If the learned subspaces are not orthogonal or the uniform assignment nullifies Rc, the soft-label Rc is not a faithful surrogate for the theory in [48].","verdict_should_be":"UNCHANGED","load_bearing_attack":"Eq. (8) minimizes -R(Z_Theta;epsilon) + Rc(Z_Theta,Pi_Phi;epsilon) + L_Ncut(Pi_Phi;A,gamma) with Pi_Phi produced by a Gumbel-Softmax cluster head. In MCR2 [48], Rc is defined for a hard 0/1 partition: each column of Pi* is an indicator, N_ell is the cardinality of class ell, and the orthogonal-subspace optimality proof relies on that. The paper does not define Rc for soft probabilities, nor does it state how N_ell is computed from the soft memberships (e.g., as the sum of probabilities). No derivation or citation is offered for this relaxation. Because Rc is the only gradient path from Pi_Phi back to the feature head Theta in the fine-tuning stage (the affinity A inside L_Ncut is explicitly detached in Algorithm 1), the claimed integration of clustering and representation learning depends entirely on this unexamined soft-label assumption. If a soft assignment can drive Rc close to R (for example, the uniform assignment Pi = (1/k) * 1 makes the weighted per-class covariance sum to the total covariance, so Rc = R), then the MCR2 term yields no gradient and the 'rate reduction' mechanism is a heuristic rather than the principled objective claimed in Section 3.2.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes CgMCR2, a unified framework for jointly learning structured image embeddings and cluster memberships by combining the Maximal Coding Rate Reduction (MCR2) objective with a differentiable relaxation of normalized cut. The method uses a frozen pre-trained encoder (CLIP or MoCo-v2), a feature head that produces embeddings, and a cluster head with Gumbel-Softmax output that yields soft memberships. Training is two-stage: a warm-up with a simplified objective (negative rate plus normalized-cut loss), followed by fine-tuning with the full objective in Eq. (8). Experiments on five standard datasets and four out-of-domain datasets report state-of-the-art clustering accuracy, with ablation studies on the output activation, affinity construction, post-processing, hyperparameters, model size, and training strategy. The code is publicly available.","tokens_in":18377,"tokens_out":7484,"duration_ms":69277,"significance":"If the central objective is well-founded, the paper makes a useful contribution: it provides a scalable, differentiable, and unified treatment of representation learning and clustering, with strong empirical results (e.g., 97.7% ACC on CIFAR-10, 77.8% on CIFAR-100, 72.9% on TinyImageNet, and 67.5% on ImageNet-1k using CLIP features) and a broad set of ablations. The release of code and the detailed hyperparameter tables are commendable. However, the theoretical grounding of the soft-label rate-reduction term is currently missing, and a core component of the clustering loss is attributed to an unpublished manuscript; both issues need to be addressed before the claimed 'principled' joint optimization can be accepted.","major_comments":[{"comment":"The rate term Rc(ZTheta, Pi_Phi; epsilon) is used with soft Gumbel-Softmax memberships Pi_Phi, but the MCR2 formulation in Eq. (1) and the theory in [48] are defined for a hard 0/1 partition, with N_l the cardinality of class l. The paper does not define how N_l is computed from soft probabilities, nor does it justify that the intended within-class compression and between-class discrimination properties survive the relaxation. This is load-bearing: in Algorithm 1 the affinity A is detached, so Rc is the only term that couples the feature head Theta and the cluster head Phi during fine-tuning; the claimed principled joint optimization rests entirely on this unexamined step. Please provide the precise definition of Rc for soft memberships and either a derivation or a citation to a published analysis, or explicitly state that the rate term is used heuristically.","section":"Section 3.2, Eq. (8)"},{"comment":"The relaxation of the normalized-cut problem and the practical rule for choosing the feasible range of gamma are attributed to the authors' own unpublished manuscript [16] (\"Submitted to Pattern Recognition\"). For a journal publication, this core component must be self-contained: include the derivation of the penalty-based relaxation and the gamma-selection procedure, or cite a published source. As written, a central part of the proposed objective is not verifiable from the public literature, which hinders reproducibility and assessment of novelty.","section":"Section 3.2, Eq. (6) and gamma selection"},{"comment":"The affinity A is explicitly detached from backpropagation, so the normalized-cut loss L_Ncut provides no direct gradient to the feature head Z_Theta. The only pathway from the clustering module to the representation during fine-tuning is through the rate-reduction term Rc(Z_Theta, Pi_Phi; epsilon). The paper should state this explicitly and discuss how the 'joint learning' claim is affected, especially in view of the missing soft-label definition for Rc. If the authors intend L_Ncut to refine the embeddings, they should either remove the detachment or explain why the indirect pathway through Pi_Phi and Rc is sufficient.","section":"Algorithm 1, line 7 and Section 3.2"}],"minor_comments":[{"comment":"The heading 'Relate Work' should be 'Related Work'.","section":"Section 2"},{"comment":"The sentence 'a softmax function is used is used for the output' contains a duplicated 'is used' and should be corrected.","section":"Section 3.2, after Eq. (6)"},{"comment":"The Gumbel-Softmax temperature is not reported in the paper or in the hyperparameter table; this parameter controls the hardness of the memberships and should be stated for reproducibility.","section":"Section 3.3 and Table A.3"},{"comment":"The conclusion states that experiments were conducted on 'five benchmark datasets', but the paper uses nine datasets (five standard and four out-of-domain); please correct the count.","section":"Section 5"},{"comment":"Eq. (7) defines the volume matrix V using sums over i = 1 to N, while Algorithm 1 operates on mini-batches of size n; please clarify whether V is computed on the mini-batch or the full dataset, since this affects the normalization in the normalized-cut loss.","section":"Algorithm 1 and Eq. (7)"},{"comment":"In the MoCo-v2 comparison, NMCE, MLC, and IMC-SwAV use their own specially designed pre-trained models, while CgMCR2 uses MoCo-v2 features; the text mentions this but should emphasize the limited comparability of these baselines.","section":"Table 4"}],"recommendation":"major_revision","confidential_remarks":"The dependence on the authors' unpublished manuscript [16] for the core normalized-cut relaxation is a reviewability concern; the current manuscript should include the full derivation or a published reference. The soft-label Rc issue is significant but appears fixable within the scope of a revision: the authors could define Rc for soft memberships, prove or cite the relaxation's properties, or switch to hard pseudo-labels for the rate term. The empirical study is broad and well documented, and the central idea is promising, so major revision rather than rejection seems appropriate."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Hi — quick take on arXiv:2412.18930 (CgMCR2). The empirical core is solid and the experiments are broad. The paper's actual contribution is a joint training objective for clustering with pretrained features: a differentiable normalized-cut term added to the MCR2 rate-reduction loss, with a two-stage warm-up/fine-tune. That's new among the MCR2 clustering papers I know. Results on CLIP features are strong — 97.7 ACC on CIFAR-10, 77.8 on CIFAR-100, 72.9 on TinyImageNet, 67.5 on ImageNet-1k — and it beats CPP by several points in most settings. The ablations are thorough: activation, affinity construction, post-processing, hyperparameters, training strategy, architecture size. Code is released. This is real work.\n\nThe soft spot is the theory. The objective in Eq. (8) uses Pi_Phi from a Gumbel-Softmax head, i.e., a soft probability matrix. But MCR2's rate reduction Rc is defined and proved for hard 0/1 partitions; the paper never redefines N_ell or Rc for soft memberships, and cites nothing for this relaxation. The stress-test concern is on point: a uniform assignment makes the weighted per-class covariance sum to the total covariance, so Rc=R and the rate-reduction gradient vanishes. Because the affinity A inside LNcut is explicitly detached, Rc is the only gradient path from the cluster head back to the feature head in fine-tuning. If that term is degenerate, the 'joint learning' claim reduces to a heuristic. That doesn't kill the empirical contribution, but it does mean the paper should either prove or cite a soft-label version of MCR2, or soften its claims about principled joint optimization.\n\nOther issues are minor but real: the objective is written for the full dataset while training uses mini-batch graphs; the differentiable normalized-cut module is borrowed from an unpublished self-citation [16]; and per-dataset hyperparameters are tuned without a described validation split. None of these are fatal.\n\nBottom line: this deserves a serious referee. The experiments are extensive, the code is available, and the core idea is worth engineering attention even if the theoretical framing needs a major cleanup. I'd send it to review, with the expectation of major revisions rather than acceptance as-is.","headline":"A well-executed clustering paper whose empirical results are strong, but whose principled joint-optimization claim rests on an unproven soft-label extension of MCR2 theory.","tokens_in":18922,"tokens_out":4035,"would_cite":true,"duration_ms":35534,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"By folding a differentiable graph-cut loss into the maximal coding rate reduction objective, the paper jointly learns embeddings and cluster assignments, reaching 97.7% clustering accuracy on CIFAR-10 and 67.5% on ImageNet-1k.","keywords":["image clustering","maximal coding rate reduction","normalized cut","structured representation learning","pre-trained vision models","deep clustering","Gumbel-Softmax","CLIP features"],"falsifier":"Run the same two-stage CgMCR2 procedure but replace the soft membership in $R_c$ with a hard one-hot assignment taken from the argmax of the cluster head, with gradients stopped or passed through straight-through. If the hard-membership version matches or beats the soft version, then the soft relaxation is not what carries the result. Alternatively, train only with $-R+L_{\\mathrm{Ncut}}$ and then evaluate with the rate-reduction term frozen so that the partition cannot influence the embeddings; if this performs as well as the full joint training, the claimed mutual guidance between partition and embedding is not load-bearing.","tokens_in":17872,"feed_emoji":"🧩","tokens_out":5996,"duration_ms":51854,"temperature":0.7,"pith_summary":"This paper proposes CgMCR2, a single objective that trains a feature head and a cluster head at the same time. The objective combines the maximal coding rate reduction (MCR2) loss, which pushes embeddings into a union of low-dimensional subspaces, with a differentiable normalized-cut loss that asks the cluster head for a clean partition. The two terms are coupled: the partition guides which regions of the embedding space get compressed, and the structured embeddings supply the affinity matrix used by the graph cut. On CLIP features the method reaches 97.7% clustering accuracy on CIFAR-10, 77.8% on CIFAR-100, 72.9% on TinyImageNet and 67.5% on ImageNet-1k, and it also works from MoCo-v2 features trained from scratch. The point of the paper is that clustering and representation learning should be one joint problem, not two sequential stages.","feed_headline":"Joint graph-cut and rate-reduction loss hits 97.7% on CIFAR-10","feed_subtitle":"A differentiable normalized-cut term lets one network learn both structured embeddings and cluster assignments from CLIP features.","key_machinery":"The load-bearing object is the joint objective in Eq. (8), which couples two existing principles. Maximal coding rate reduction supplies $R(Z;\\epsilon)-R_c(Z,\\Pi;\\epsilon)$: expanding the whole embedding while compressing each cluster into a low-dimensional subspace. A relaxed normalized-cut loss $L_{\\mathrm{Ncut}}(\\Pi;A,\\gamma)$, with affinity $A=P_s(Z^\\top Z)$ (sparsified cosine similarity) and volume-normalized membership, supplies a differentiable clustering signal. A Gumbel-Softmax cluster head reparameterizes the soft partition, and a two-stage schedule first warms up with $-R+L_{\\mathrm{Ncut}}$ and then fine-tunes with the full objective. The affinity is detached from backpropagation so the graph-cut term trains the cluster head without distorting the feature head through the graph.","core_discovery":"The paper's central claim is that the joint objective\n$$\n\\min_{\\Theta,\\Phi}\\,-R(Z_\\Theta;\\epsilon)+R_c(Z_\\Theta,\\Pi_\\Phi;\\epsilon)+L_{\\mathrm{Ncut}}(\\Pi_\\Phi;A,\\gamma)\n$$\nturns clustering and structured representation learning into a single differentiable problem. The rate-reduction terms make the feature-head output $Z$ a union of low-dimensional orthogonal subspaces, while the graph-cut term makes the cluster-head output $\\Pi$ a soft partition whose affinity $A$ is built from $Z$. Because every term is differentiable, both heads can be trained end-to-end with gradient descent, the cluster head can label unseen test points directly, and the learned membership actively guides which subspaces are compressed. The paper reports that this joint training outperforms prior MCR2-based clustering, classical clustering methods, and recent deep clustering methods on most tested datasets.","pith_inferences":["The same coupling could be applied to a trainable encoder: nothing in the objective requires the backbone to be frozen, so end-to-end fine-tuning of CLIP or a vision transformer is a direct next test.","Straight-through hard assignment during fine-tuning would isolate whether the soft Gumbel-Softmax membership is actually providing the cluster-wise compression the paper claims, since the theoretical guarantees in the cited MCR2 work are stated for hard partitions.","The mini-batch affinity could be stabilized with a memory bank or an exponential moving average; this would likely matter at larger scales where the current top-s sparsified batch affinity is a noisy graph.","Because the normalized-cut loss only sees the current batch's affinity, an interesting transfer test is node clustering on graph data, where the affinity is given and the same joint objective could be applied directly."],"forward_implications":["If the central claim holds, deep image clustering can be trained end-to-end as one objective, removing the separate pseudo-label phase used by SCAN-style pipelines.","The cluster head becomes a label predictor for unseen data, so no extra k-means or spectral step is needed at test time.","On frozen CLIP features the method reaches 97.7% ACC on CIFAR-10, 77.8% on CIFAR-100, 72.9% on TinyImageNet and 67.5% on ImageNet-1k, beating the CLIP-based MCR2 baseline CPP in every setting.","On MoCo-v2 features, CgMCR2 improves CIFAR-10 ACC to 92.8, ahead of the prior MCR2-based MLC (86.3) and other deep clustering baselines.","The two-stage warm-up is important: ablations show that fine-tuning the full objective without the warm-up drops CIFAR-100 ACC from 77.8 to 66.0.","The neural cluster head is about 100 times faster per training iteration than replacing it with k-means, spectral clustering, or EnSC, while achieving comparable or better accuracy."],"supporting_citations":[{"why":"Defines the MCR2 objective and the structured-subspace geometry that the joint framework builds on.","marker":"[48]"},{"why":"Supplies the differentiable relaxation of normalized cut used as the graph-cut loss.","marker":"[16]"},{"why":"Introduces the normalized-cut criterion whose relaxed form is the clustering loss.","marker":"[41]"},{"why":"Provides Gumbel-Softmax for differentiable categorical reparameterization of the cluster head.","marker":"[19]"},{"why":"Is the CLIP-based MCR2 clustering baseline (CPP) that CgMCR2 compares against and extends.","marker":"[8]"},{"why":"Provides the frozen CLIP image encoder used to produce the pre-features.","marker":"[39]"},{"why":"NMCE is an earlier MCR2-based deep clustering method that initializes the partition randomly, providing a contrast for the principled initialization.","marker":"[28]"},{"why":"MLC is an earlier MCR2-based clustering method that initializes the cluster head by copying feature-head parameters.","marker":"[11]"}],"fun_headline_variants":["Graph-cut guides rate reduction for joint embedding and clustering","One differentiable loss unifies image embedding and clustering","Graph-cut and rate reduction jointly learn clusters and embeddings","Single loss for both structured embeddings and cluster assignments","End-to-end graph-cut and rate reduction for better clustering"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The central assumption is that a soft, probabilistic cluster-membership matrix produced by Gumbel-Softmax can stand in for the hard 0/1 assignment that the rate-reduction theory assumes; the paper offers no derivation or citation for this relaxation, and the joint objective depends on it.","fun_headline_variants_meta":{"raw":{"variants":["Graph-cut guides rate reduction for joint embedding and clustering","One differentiable loss unifies image embedding and clustering","Graph-cut and rate reduction jointly learn clusters and embeddings","Single loss for both structured embeddings and cluster assignments","End-to-end graph-cut and rate reduction for better clustering"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000655,"raw_usage":{"total_tokens":2974,"prompt_tokens":891,"completion_tokens":2083,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":507,"completion_tokens_details":{"reasoning_tokens":2008}},"tokens_in":507,"tokens_out":2083,"duration_ms":13886,"temperature":1.0,"reasoning_tokens":2008,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T04:19:59.101835+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same two-stage CgMCR2 procedure but replace the soft membership in $R_c$ with a hard one-hot assignment taken from the argmax of the cluster head, with gradients stopped or passed through straight-through. If the hard-membership version matches or beats the soft version, then the soft relaxation is not what carries the result. Alternatively, train only with $-R+L_{\\mathrm{Ncut}}$ and then evaluate with the rate-reduction term frozen so that the partition cannot influence the embeddings; if this performs as well as the full joint training, the claimed mutual guidance between partition and embedding is not load-bearing.","supporting_citations":[{"cited_title":"In: Advances in Neural Information Processing Systems","cited_arxiv_id":null,"evidence_quote":"Defines the MCR2 objective and the structured-subspace geometry that the joint framework builds on."},{"cited_title":"Submitted to Pattern Recognition (2024)","cited_arxiv_id":null,"evidence_quote":"Supplies the differentiable relaxation of normalized cut used as the graph-cut loss."},{"cited_title":"In: 5th International Conference on Learning Representations (2017)","cited_arxiv_id":null,"evidence_quote":"Provides Gumbel-Softmax for differentiable categorical reparameterization of the cluster head."},{"cited_title":"In: International Conference on Learning Representations (2024)","cited_arxiv_id":null,"evidence_quote":"Is the CLIP-based MCR2 clustering baseline (CPP) that CgMCR2 compares against and extends."},{"cited_title":"In: International Conference on Machine Learning","cited_arxiv_id":null,"evidence_quote":"Provides the frozen CLIP image encoder used to produce the pre-features."},{"cited_title":"In: IEEE/CVF International Conference on Computer Vision","cited_arxiv_id":null,"evidence_quote":"MLC is an earlier MCR2-based clustering method that initializes the cluster head by copying feature-head parameters."}],"review_version":1}