{"id":"4d6ffe29-ffad-457c-8e9e-b9f8bf5f093e","arxiv_id":"2504.16140","paper_version":1,"verdict":"REJECT","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":4,"one_line_summary":"SparseJEPA applies an oi-VAE style sparsity penalty to JEPA and reports modest linear-probe gains, but its theoretical justification is flawed.","lead":"This paper adds a sparsity penalty to JEPA, a self-supervised vision model, and reports small accuracy gains on four classification benchmarks. The theoretical proof it offers for why grouping helps contains mathematical errors, and the experiments lack error bars or code.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central claim is not established: Lemma 1's strictness is false (XOR counterexample), so Theorem 1(1) fails; the only empirical support also lacks seeds, configs, and code.","rationale":"The reader correctly suspected Lemma 1's strictness condition, but the supplied identical-variables example is not a valid counterexample: original multiinformation is 2 bits and grouped multiinformation is 1 bit, so strict reduction holds there. A parity construction is a valid counterexample, and it makes Theorem 1(1) false as stated. This is a genuine mathematical defect in the only theoretical argument offered. The empirical side is also under-specified, with L_KL never defined, no beta/lambda/G values, no seeds, no error bars, and no code, so Table I cannot verify the central accuracy claim. I therefore agree with a reject verdict, but the strongest precise objection is the invalid strictness claim, with the missing experimental protocol as a second, independent reason. The non-strict inequality in Lemma 1 may be true, and the general idea of adding oi-VAE-style penalties to JEPA is not inherently impossible; the problem is that the paper's proof and evidence do not establish its central claim.","tokens_in":5603,"tokens_out":13805,"duration_ms":129919,"concrete_test":"Run the multiinformation calculation for X1, X2 independent Bernoulli(1/2) and X3 = X1 xor X2, comparing I(X1;X2;X3) with I((X1,X2);X3). Both equal 1 bit, so Lemma 1 strictness fails and Theorem 1(1)'s promised strict reduction is not guaranteed. This settles the theoretical concern; separately, the authors should be asked for code and exact hyperparameters and Table I rerun with five seeds to determine whether the empirical gain survives.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim has two supports: Theorem 1, which is supposed to prove that grouping enhances representations, and Table I, which reports accuracy gains. The theorem's strict inequality (Eq. 1) depends on Lemma 1's strictness clause, and that clause is false. Take X1, X2 independent Bernoulli(1/2) and X3 = X1 xor X2. Then H(Xi)=1 for each i and H(X1,X2,X3)=2, so the original multiinformation is 3 - 2 = 1 bit. Grouping {X1,X2} as G1 and X3 as G2 gives H(G1)=2, H(G2)=1, H(G1,G2)=2, so the grouped multiinformation is also 1 bit. There are nontrivial inter-group dependencies, yet no reduction occurs. Thus Lemma 1's provided condition does not imply strict reduction, and Theorem 1(1) is false as stated; the proof's phrase 'discards inter-group dependencies' does unjustified work. The empirical side does not rescue the claim: Section 4 never defines L_KL, beta, lambda, G, the block mask geometry, or the training schedule, and Table I has no seeds, error bars, or code release. The reported gains cannot be distinguished from regularization-strength tuning or run-to-run variance, and no argument connects lower multiinformation to higher linear-probe accuracy.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes SparseJEPA, an extension of Joint Embedding Predictive Architectures that adds an oi-VAE-inspired sparsity penalty to the JEPA loss, intended to make latent embeddings sparse and grouped along semantically related image patches. The authors claim that this grouping reduces multiinformation among latent variables, thereby enhancing representation quality, and they report linear-probe accuracy gains over a JEPA baseline on CIFAR-100, Place205, iNaturalist-2018, and CLEVR/COUNT. The theoretical contribution is presented in Section 3 via Lemma 1 and Theorem 1, and the empirical contribution is in Section 5 with a four-row comparison table.","tokens_in":5880,"tokens_out":7404,"duration_ms":64834,"significance":"The problem addressed—interpretability and structured sparsity in JEPA-style self-supervised representations—is relevant and timely. The non-strict version of Lemma 1, that deterministic grouping cannot increase multiinformation, is a correct and standard data-processing-type result, and the idea of transplanting an oi-VAE-style penalty into a JEPA framework is worth exploring. However, the paper's central theoretical claim of a strict reduction under inter-group dependencies is false, and Theorem 1 as stated does not follow from Lemma 1. The empirical evaluation is also too thin to establish the claimed improvements: no seeds, error bars, hyperparameters, or code are provided. If the theoretical and experimental gaps were fully addressed, the underlying idea could be of interest, but in its current form the manuscript does not support its headline claims.","major_comments":[{"comment":"The strictness clause of Lemma 1 is false. Let X1 and X2 be independent Bernoulli(1/2) and let X3 = X1 XOR X2. There is a nontrivial inter-group dependency between the group G1=(X1,X2) and G2=X3, yet I(G1;G2) = I(X1;X2;X3) = 1 bit, equal to the original multiinformation. The inequality is not strict in this example, contradicting the lemma's second sentence. The proof's statement that the inequality is strict 'whenever the grouping process discards inter-group dependencies' is therefore unjustified and is load-bearing for Theorem 1(1).","section":"§3, Lemma 1"},{"comment":"Theorem 1 misapplies Lemma 1. In Lemma 1, the grouped variables G_j are deterministic functions of subsets of the variables whose multiinformation appears on the right-hand side. In Theorem 1, however, G_j is defined as a subset of the latent variables Z, while the right-hand side of Eq. (1) is the multiinformation of the observed variables X. No connection between I(Z1;...;Zk) and I(X1;...;Xn) is established, so the claimed inequality I(G1;...;Gm) < I(X1;...;Xn) does not follow. Statement 2 is also not proven: since G is simply a rearrangement of Z, I(Z;G)=H(Z) and I(Z;G) ≥ I(Z;X) holds trivially, but this is not the mechanism described in the proof.","section":"§3, Theorem 1"},{"comment":"The empirical evaluation is not reproducible and does not support the reported gains. The loss in §4.2 includes βL_KL but L_KL is never defined, and the hyperparameters λ, β, the number of groups G, the size and geometry of the latent-to-group matrix W, the block-mask construction, and the training schedule are all unspecified. Table I gives a single point estimate per dataset without seeds, error bars, or ablations, so the improvement (e.g., 45.4 vs 40.01 on CIFAR-100) cannot be distinguished from run-to-run variance or from tuning the regularizer strength.","section":"§4.2 and §5, Table I"},{"comment":"The abstract claims SparseJEPA 'enhances the quality of learned representations' and demonstrates transfer to 'low-level tasks,' but Table I only reports top-1 linear-probe accuracy on four datasets, three of which are image-classification benchmarks. No low-level tasks such as depth estimation or counting beyond CLEVR/COUNT (which is presented as a classification accuracy) are shown. The abstract also says the paper includes a proof of the Data Processing Inequality for Multiinformation, but the paper only invokes known data-processing results and does not provide such a proof.","section":"Abstract and §5"}],"minor_comments":[{"comment":"There is a typo: 'multiiformation' should be 'multiinformation.'","section":"§3"},{"comment":"The phrase 'including proofing the Data Processing Inequality' should be 'including proving the Data Processing Inequality' or, more accurately, 'citing the Data Processing Inequality.'","section":"Abstract"},{"comment":"The caption 'SparseJEPA Architecure' contains a typo: 'Architecure' should be 'Architecture.'","section":"Figure 1"},{"comment":"The notation W^(g)_{·,j} is ambiguous: it is unclear whether W is a tensor with per-group matrices or a single matrix, and the index '·' should be clarified.","section":"§4.2"},{"comment":"The conclusion says 'as demonstrated on the CIFAR-100 benchmark' even though the results section reports results on four datasets; this should be corrected.","section":"§6"}],"recommendation":"reject","confidential_remarks":"The manuscript is not ready for publication. The central theoretical claim relies on a false strictness assertion and a misapplication of Lemma 1, and the experimental section lacks the detail needed to evaluate the reported gains. The non-strict inequality is standard and does not constitute a novel contribution as presented. The paper would need a substantial rewrite of the theory and a much more rigorous experimental protocol before it could be considered; in its current form, the load-bearing errors are not local fixes."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The one thing to know: this paper is a mix of a reasonable empirical idea and an under-supported write-up. The new bit is grafting the oi-VAE sparsity penalty onto JEPA and checking linear-probe transfer on four datasets. That is a legitimate experiment to run, and the reported gains (e.g., CIFAR-100 45.4 vs 40.01) are directionally interesting if real. The paper is honest about not scaling up and about being an early-stage extension. It is not a field-reshaping result, but it is not a non-result either.\n\nWhat deserves credit: the problem is worth attacking. Dense JEPA latents are hard to interpret, and asking for group-structured sparsity is a sensible prior. The choice to test on low-level transfer tasks (CLEVR/COUNT, iNat18) alongside classification is also reasonable. The paper correctly identifies that the grouping mechanism is what matters, rather than sparsity for its own sake. The literature connections to oi-VAE and to multiinformation are the right touchstones.\n\nNow the soft spots, in proportion. The theoretical section is the weakest part. Lemma 1's strictness clause is false as stated: identical variables or the XOR example show nontrivial inter-group dependencies can exist without strict multiinformation reduction. The proof hand-waves with “discards inter-group dependencies,” which does not do the work. Since Theorem 1(1) relies on that strictness, that part of the theorem fails as stated. The second inequality (mutual information with Z) is asserted rather than proven; it is not obvious that grouping a sufficient statistic increases MI with Z, and no argument beyond intuition is given. These are load-bearing flaws because the abstract and conclusion claim a proof that grouping enhances representation quality.\n\nThe empirical section is also thinner than it should be. Section 4 never defines L_KL, the block-mask geometry, the number of groups G, the latent-to-group matrix W's initialization, the training schedule, or the hyperparameter ranges for lambda and beta. Table I has no seeds, no error bars, no code release. The reader cannot tell whether the gains come from the sparsity penalty or from regularization tuning or run-to-run variance. The claim that lower multiinformation leads to higher linear-probe accuracy is never empirically substantiated; the paper only shows a correlation in one direction.\n\nThe citation pattern is fine: the core result does not lean on self-citation, and the references are standard. I would flag that the proof of DPI for multiinformation is standard and does not need a new proof; the novelty claim there is overstated.\n\nWho is this for? A reader working on interpretable self-supervised learning might find the empirical direction worth a quick look, provided the authors or a future version supply the missing experimental detail. The theoretical contribution as written is not usable. My honest recommendation: do not cite the theorem as a proof; treat Table I as a preliminary result that needs confirmation. If this comes to a journal, I would send it to peer review only after a major revision—verifiable experiments, corrected theorem statements, and a real comparison against plain JEPA with comparable regularization.\n\nFor you: I would not bring it to the reading group as a model of how to write theory, but the empirical question is a fair one to discuss.","headline":"SparseJEPA is a plausible-in-principle combination of oi-VAE sparsity with JEPA, but the theory has a false strictness claim and the empirical table lacks the controls needed to support the headline accuracy gains.","tokens_in":6354,"tokens_out":836,"would_cite":false,"duration_ms":9527,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Adding a sparsity penalty to JEPA's latent space improves downstream accuracy and interpretability.","keywords":["SparseJEPA","Joint Embedding Predictive Architecture","sparse representation learning","multiinformation","linear probe transfer","Vision Transformer","self-supervised learning","latent space interpretability"],"falsifier":"Run the lemma with m=n, each subset containing exactly one variable: the group variables are then the original variables, so multiinformation is identical, and the strict inequality asserted for any nontrivial inter-group dependencies fails; a two-variable distribution with nonzero mutual information is enough to check this.","tokens_in":5413,"feed_emoji":"🧩","tokens_out":9561,"duration_ms":86282,"temperature":0.7,"pith_summary":"The paper proposes SparseJEPA, an extension of Joint Embedding Predictive Architectures (JEPA) that adds a sparsity penalty to the latent space so that dimensions shared by semantically related image patches stay grouped while redundant dimensions are suppressed. The goal is to show that sparse latent codes improve both the interpretability and the transfer quality of JEPA's learned representations. On CIFAR-100 with a lightweight Vision Transformer, the authors report top-1 linear-probe accuracy rising from 40.01 to 45.4 when the penalty is added, with similar gains on Place205, CLEVR/Count, and iNaturalist. The authors also present a theoretical argument that grouping latent variables reduces multiinformation, the total shared information across variables, and thereby leaves more capacity for meaningful structure.","feed_headline":"Sparsity lifts JEPA transfer accuracy on CIFAR-100","feed_subtitle":"New penalty gives 45.4 vs 40.0 top-1 accuracy and reveals semantic groupings in latent space.","key_machinery":"The load-bearing object is the augmented JEPA loss, $L=\\frac{1}{M}\\sum_{i=1}^{M}\\sum_{j\\in\\mathcal{B}_i}\\|\\hat{s}_y^j-s_y^j\\|_2^2+\\beta L_{\\mathrm{KL}}+\\lambda\\sum_{g=1}^{G}\\sum_{j=1}^{K}\\|W^{(g)}_{\\cdot,j}\\|_2$, where the last term penalizes each latent dimension for being strongly weighted in many groups. The grouping argument rests on multiinformation, $I(X_1;\\dots;X_n)=D_{\\mathrm{KL}}(p(x_1,\\dots,x_n)\\|\\prod_i p(x_i))$, which measures how much shared information the variables carry. The paper's lemma applies the data processing inequality to deterministic group mappings to claim $I(G_1;\\dots;G_m)\\le I(X_1;\\dots;X_n)$, giving the theoretical route from grouping to reduced redundancy.","core_discovery":"On its own terms, the paper claims that sparsity is not a trade-off: a penalty that pushes each latent dimension to be active in only a few semantic groups simultaneously sharpens the representation and improves downstream transfer. The authors call this architecture SparseJEPA. The reported evidence is a comparison against a same-backbone JEPA baseline, with top-1 linear-probe accuracy rising from 40.01 to 45.4 on CIFAR-100 and improving on every other tested dataset. The paper also asserts a theorem: partitioning latent variables into groups reduces the multiinformation among them, and the reduction is strict when cross-group dependencies exist, which is offered as a formal explanation for why sparse grouping helps.","pith_inferences":["If the grouping mechanism, rather than generic regularization, drives the gain, then ablating the penalty on random group assignments should remove most of the accuracy improvement; the paper does not report this control.","A direct test of the theoretical claim would estimate latent multiinformation with and without the sparsity penalty; the theory predicts a measurable drop whenever the grouping is non-trivial.","The authors point toward object-centric learning; one concrete extension is to make the penalty's groups correspond to detected objects or patch clusters, turning the loss into a soft object-segmentation signal."],"forward_implications":["If the reported accuracy holds under controlled re-runs, SparseJEPA offers a drop-in penalty for JEPA models that improves linear-probe transfer without changing the backbone.","The grouping of latent dimensions by shared semantics gives a route to inspect what a JEPA model has learned: each active group can be traced to image patches with high mutual information.","Because the penalty acts on the latent code rather than the architecture, it can be combined with larger Vision Transformers and other JEPA variants as compute allows.","Downstream tasks like object counting and depth estimation should inherit the gain, since the paper reports improvements on CLEVR/Count and other benchmarks."],"supporting_citations":[{"why":"defines the JEPA framework whose latent-space prediction objective SparseJEPA extends.","marker":"(Assran et al., 2023)"},{"why":"supplies the oi-VAE sparsity loss adapted for the latent-to-group penalty.","marker":"(Ainsworth et al., 2018)"},{"why":"one source for the multiinformation definition used in the grouping theorem.","marker":"(Raman et al., 2017)"},{"why":"provides the multiinformation function as a measure of stochastic dependence and the data-processing inequality context.","marker":"(Studený and Vejnarová, 1998)"},{"why":"cited to support the claim that sparsity reduces redundant information in feature selection.","marker":"(Fang et al., 2018)"},{"why":"provides the CIFAR-100 dataset used for pretraining and evaluation.","marker":"(Krizhevsky, 2009)"}],"fun_headline_variants":["Sparse grouping sharpens JEPA transfer on CIFAR-100","SparseJEPA: 45.4 vs 40.0 top-1 accuracy","Grouping reduces multiinformation in JEPA latents","SparseJEPA: Sparsity penalty yields semantic grouping","Less dense, more accurate: SparseJEPA on CIFAR-100"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The theorem's claim that grouping strictly lowers redundant multiinformation assumes the partition actually merges dependent variables; if the 'grouping' leaves variables in their own singleton groups, the quantity does not change, so strictness is not automatic.","fun_headline_variants_meta":{"raw":{"variants":["Sparse grouping sharpens JEPA transfer on CIFAR-100","SparseJEPA: 45.4 vs 40.0 top-1 accuracy","Grouping reduces multiinformation in JEPA latents","SparseJEPA: Sparsity penalty yields semantic grouping","Less dense, more accurate: SparseJEPA on CIFAR-100"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000387,"raw_usage":{"total_tokens":2028,"prompt_tokens":912,"completion_tokens":1116,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":528,"completion_tokens_details":{"reasoning_tokens":1020}},"tokens_in":528,"tokens_out":1116,"duration_ms":8717,"temperature":1.0,"reasoning_tokens":1020,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T11:23:25.723880+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the lemma with m=n, each subset containing exactly one variable: the group variables are then the original variables, so multiinformation is identical, and the strict inequality asserted for any nontrivial inter-group dependencies fails; a two-variable distribution with nonzero mutual information is enough to check this.","supporting_citations":[{"cited_title":"Varshney","cited_arxiv_id":null,"evidence_quote":"one source for the multiinformation definition used in the grouping theorem."},{"cited_title":"On the role of sparsity in feature selection and an innovative method LRMI","cited_arxiv_id":null,"evidence_quote":"cited to support the claim that sparsity reduces redundant information in feature selection."}],"review_version":1}