{"id":"58a97863-7d87-4a84-99e9-97a12c3de5d7","arxiv_id":"2608.12748","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"A unified visual grounding framework combining a broadcast cross-attention head, a JEPA auxiliary loss, and an MLLM-generated caption dataset preserves representation diversity and generalizes across RefCOCO/+/g.","lead":"This paper trains one visual grounding model on images and machine-generated referring descriptions, adding a predictive visual 'JEPA' loss to keep learned features diverse. The authors report that a compact 75M model matches much larger multimodal LLMs on RefCOCO benchmarks without per-dataset fine-tuning.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Theorem 8's dim=C claim is guaranteed only by Assumption 6, which is the conclusion in disguise; the empirical effective rank of 83 at C=768 does not establish full-support JEPA gradient fluctuation.","rationale":"The reader's weakest_assumption identifies exactly the same load-bearing point: Assumption 6 in Appendix B is essentially the conclusion, and the measured effective rank of 83 does not demonstrate full C-dimensional gradient support. My reading agrees with the reader's assessment that the empirical framework and ablations are internally consistent and re-examinable, but that the theoretical guarantee is conditional on an unproven premise. The concern is concrete and testable: one can directly estimate the covariance of the JEPA gradient fluctuation under mask/noise resampling and check whether it has positive variance in every direction. If the test fails, the theorem's strong claim should be weakened, but the empirical contributions and the conditional verdict remain appropriate. Since the reader already marked the paper CONDITIONAL for this reason, no verdict change is needed.","tokens_in":22532,"tokens_out":5115,"duration_ms":60778,"concrete_test":"After training the default mACH+JEPA model to convergence, freeze all parameters and hold out a fixed batch of images with their referring expressions. Resample the mask pattern Ω and noise ε from Equation (A5) for K ≥ 2000 iterations, computing the JEPA gradient with respect to the shared visual features X for each sample. Form the empirical fluctuation covariance Γ̂_J = (1/K) Σ (g_k − ḡ)(g_k − ḡ)ᵀ and estimate its eigenvalue spectrum, especially after projecting out the top text-driven subspace. If the minimum eigenvalue is at the float32 floor, or if the effective rank of Γ̂_J is far below C=768, then Assumption 6 fails and Theorem 8's dim=C claim is unsupported. Repeat the same measurement on ConvNeXt-Base with C=1024; a genuine full-support fluctuation should keep λ_min bounded away from zero rather than shrinking as C grows.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central theoretical claim—that only the dual-stream objective is almost surely free of alignment-blind directions and preserves alignment capacity in every direction (Theorem 8, Section 4)—rests entirely on Assumption 6 in Appendix B: the mask-noise fluctuation ξ_J of the JEPA gradient is not almost surely confined to any hyperplane, i.e., Γ_J ⪰ c_J I with c_J > 0. This assumption is not derived from the architecture; it asserts the exact property that Theorem 8 is supposed to establish. Lemma 7 only converts Assumption 6 into the theorem's conclusion via the bound Γ ⪰ α² c_J I, so the proof is conditional on an unverified premise. No constructive argument is given that stochastic mask resampling plus the text-conditioned predictor produces fluctuation in every one of the C=768 feature directions. The empirical evidence in Section 5.5 is suggestive but not conclusive: the measured effective rank is 83 for C=768, and the reported non-vanishing spectral tail (~1e-5) sits close to the float32 accumulation floor that the paper itself places near ~1e-6, so most of the 768 directions carry variance barely above numerical precision. Even a strictly positive c_J could be so small that the sustained-excitation term is numerically negligible. The authors' own Appendix F concedes that the theory addresses representational capacity rather than optimization dynamics or supervision quality. If Assumption 6 fails for even one unit direction, the claimed 'only paradigm preserves alignment capacity in every direction' is false; the empirical results would still stand, but the theoretical contribution would reduce to a restatement of the assumption.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript proposes a framework for unified open-vocabulary visual grounding based on three components: a Modulated Attention-Contrastive Head (mACH) for token-level vision-language alignment, a text-conditioned JEPA auxiliary stream intended to provide complementary gradient support without inference overhead, and a newly constructed dataset O365-Caption that converts Objects365 category labels into dense referring expressions. The paper also presents a theoretical analysis, culminating in Theorem 8, which claims that only the dual-stream objective (mACH plus JEPA) is almost surely free of alignment-blind directions and preserves alignment capacity in every feature direction. Experiments on RefCOCO/+/g under zero-shot and fine-tuning protocols report strong results, including 85.3/89.0/82.5 zero-shot and 91.7/93.0/90.2 fine-tuned on RefCOCO with a 75M model, plus ablations and spectral analyses intended to support the representation-diversity argument.","tokens_in":22806,"tokens_out":3051,"duration_ms":34078,"significance":"If the central theoretical claim were established, the paper would make a useful conceptual contribution: a precise, dimension-based account of why an auxiliary predictive objective can preserve alignment capacity beyond what contrastive or dot-alignment objectives provide. The O365-Caption resource, the careful ablations, and the public release of code and data are also valuable, and the reported zero-shot numbers are striking relative to the model size. However, the main theorem is conditional on an assumption that essentially states the conclusion, and the empirical spectral evidence is explicitly interpreted by the authors as not supporting depth comparisons near the noise floor. These issues currently prevent the paper from substantiating its strongest claims.","major_comments":[{"comment":"Assumption 6 (Eq. A20-A22) states that the mask-noise fluctuation ξ_J is not almost surely confined to any hyperplane, i.e., Γ_J ⪰ c_J I with c_J > 0. This is exactly the property needed to conclude that the joint gradient second moment has full rank, and Lemma 7 and Theorem 8 only propagate that assumption to the conclusion. The proof therefore does not establish that the dual-stream objective 'preserves alignment capacity in every direction' from architectural or algorithmic principles; it merely states the conclusion as an assumption. The paper needs either a constructive argument showing that stochastic mask resampling plus the predictor architecture yields fluctuation in every one of the C=768 directions, or a clearly framed conditional theorem stating that full-support fluctuation is a sufficient condition rather than a derived property.","section":"Appendix B, Assumption 6 and Theorem 8"},{"comment":"The spectral evidence does not provide empirical support for Assumption 6. The effective rank is 83 for C=768, and the non-vanishing tail is around 1e-5, while the paper itself places the float32 accumulation floor near 1e-6; the authors also note that 'cliff positions, not depths, are interpretable.' This means that most of the 768 directions carry variance close to the numerical floor, which is compatible with a strictly positive but negligible c_J, and does not demonstrate that fluctuation spans every direction. A stronger test would compare the measured spectrum against a properly estimated noise floor across multiple training seeds, and report whether the number of directions above that floor is consistent with C.","section":"Section 5.5, Figure 5"},{"comment":"The evaluation protocol is not apples-to-apples. Section 5.1 states that all evaluated models are benchmarked on a cleaned version of RefCOCO/+/g, but Table 2 appears to cite baseline numbers from the original papers, which typically report on the standard, uncleaned annotations. If the baselines were not re-evaluated on the same cleaned data, the reported margins over GLIP-T, GDINO-T, and PropVG may be inflated. The authors should re-run all baselines on the same cleaned benchmark, or clearly mark which numbers were re-evaluated and which were copied, and discuss the direction of the resulting bias.","section":"Section 5.1 and Table 2"},{"comment":"There is an internal inconsistency in the reported size of the dataset: Table 1 lists 9.6M annotations for O365-Caption, while Appendix C states that the pipeline produces '10.0M precise spatial-textual alignment targets.' The authors should reconcile these numbers and specify which count is used in training.","section":"Table 1 vs. Appendix C"},{"comment":"No error bars, seeds, or variance are reported for any experiment, and several ablation differences are small (for example, α=0.1 vs. α=0.2 differs by only 0.3-0.6 points on most splits). Without multiple seeds or some uncertainty quantification, it is difficult to tell which differences are meaningful. At minimum, the main results and key ablations should be repeated for at least three seeds and reported as mean ± std.","section":"Section 5.3 and Table 3"}],"minor_comments":[{"comment":"The architecture diagram contains many tiny labels and abbreviations (e.g., 'B,P,L,C', 'E', 'R', 'S') that are not defined in the caption; please enlarge the figure and add a legend or explain the notation in the caption.","section":"Figure 2"},{"comment":"The column 'FT' is not defined in the caption; please state explicitly that it indicates whether the model was fine-tuned on the target benchmark.","section":"Table 2"},{"comment":"The inference latency numbers in the appendix (33.2 ms fixed text, 140.5 ms dynamic text) are not reconciled with the latency table in Section 5.5, which reports 26-27 ms for mACH. Please clarify the hardware, batch size, and caching assumptions so the two sets of numbers are comparable.","section":"Appendix E, Table A2(d)"},{"comment":"Equation (7) says the predictor receives language embedding W, but Appendix B, Lemma 5, describes the JEPA driving signal as 'text-free' and states that text-side outputs receive no gradient. The relationship between these two descriptions should be clarified in the main text, since the phrase 'text-conditioned JEPA' could mislead readers about where language information enters the gradient.","section":"Section 3.2"}],"recommendation":"major_revision","confidential_remarks":"The paper has a useful dataset and an interesting hypothesis, but the headline theoretical claim is not currently supported by the proof, and the benchmark comparisons need to be made fair. I would also ask the editor to verify that the baselines in Table 2 were indeed copied from their original papers without re-evaluation on the cleaned benchmarks, since this changes the interpretation of the reported gains. The inconsistency between 9.6M and 10.0M annotations and the absence of seed variance should be fixed before resubmission."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"This paper is worth a serious look, but the headline theoretical claim does not survive contact with its own assumption. The real contributions are empirical: O365-Caption, a large-scale upgrade of Objects365 labels into referring expressions, and the mACH+JEPA co-design that appears to improve zero-shot grounding at 75M parameters. The ablations are internally consistent, the dataset statistics are carefully documented, and the code and data links make the work re-examinable. Credit where due: the data pipeline is a genuine asset, and the broadcast cross-attention head is a sensible engineering choice for multi-query edge inference.\n\nThe soft spots are real and need airing. Theorem 8, the claim that only the dual-stream objective preserves alignment capacity in every direction, rests entirely on Assumption 6: that the JEPA gradient fluctuation is not almost surely confined to any hyperplane. That is the conclusion restated as a premise. Lemma 7 just converts it into the theorem. No constructive argument shows that mask resampling plus the text-conditioned predictor actually spans all 768 feature directions. The empirical spectral evidence is suggestive but not conclusive: effective rank 83 out of 768, with a tail around 1e-5, sits close to the float32 accumulation floor the paper itself cites. A strictly positive but tiny lower bound on the spectral floor would be numerically meaningless.\n\nAlso worth flagging: the method is evaluated on the cleaned RefCOCO benchmark while many baseline numbers appear to be cited from the original papers, which were not cleaned. That asymmetry can inflate apparent gains. The lack of error bars or seeds makes it hard to judge how much of the margin is stable. The 9.6M vs 10.0M annotation count discrepancy is minor but should be corrected.\n\nThe paper is honest about the theory's scope in Appendix F, which helps, but the main text oversells it. If the theory is reframed as a conditional statement ('if the JEPA fluctuation is full-rank, then...') and the overclaim about being the only paradigm is softened, the paper is in decent shape. As is, the empirical system and dataset deserve a serious referee; the theory does not.\n\nMy recommendation: send it to peer review. The data asset and the reproducible experiments justify the time. A good referee can push the authors to demote the theory to a motivating framework and to run a fairer baseline comparison on the same cleaned benchmark.","headline":"A useful empirical recipe and dataset for edge visual grounding, but the theoretical centerpiece is an assumption in disguise and the benchmark comparisons have a protocol wrinkle.","tokens_in":23411,"tokens_out":1362,"would_cite":true,"duration_ms":16656,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A compact 75M visual grounding model can match 13B multimodal LLMs by preserving representation diversity through a dual-stream training objective.","keywords":["referring expression comprehension","open-vocabulary grounding","representation diversity","representation collapse","JEPA","cross-modal attention","visual grounding","Objects365-Caption"],"falsifier":"For a fixed image and fixed text, sample many JEPA masks and compute the gradient of $\\mathcal{L}_{JEPA}$ with respect to the shared visual features; estimate the minimum eigenvalue of the covariance of these centered gradients across masks. If that minimum sits at the numerical floor rather than at $c_J > 0$, Assumption 6 fails and the claimed dimension-$C$ support collapses. A simpler proxy is the empirical effective rank of $\\Xi_X$: if the mACH+JEPA spectral tail above the float32 floor does not persist across all 768 dimensions, the predicted spectral floor is contradicted.","tokens_in":1635,"feed_emoji":"🎯","tokens_out":8357,"duration_ms":130809,"temperature":0.7,"pith_summary":"This paper argues that the main obstacle to scaling a single open-vocabulary visual grounding model is representation degeneration: contrastive-style training compresses the shared visual features into a low-rank subspace, leaving many feature directions with no alignment signal. To counter this, the authors jointly train the same visual features with two complementary objectives: a token-level modulated attention-contrastive head for grounding, plus a reconstruction-based JEPA auxiliary stream that is removed at inference. On the data side, they convert Objects365 category tags into dense, context-aware referring expressions (O365-Caption). The central claim is that only the dual-stream objective keeps gradient support in all feature directions, so a single 75M checkpoint can match or beat much larger systems on standard referring-expression benchmarks without dataset-specific adaptation.","feed_headline":"75M grounding model matches 13B LLMs by keeping features diverse","feed_subtitle":"Adding a reconstruction loss keeps all 768 feature directions alive; the 75M model then matches 13B systems at zero-shot grounding.","key_machinery":"The argument runs on three coupled objects. First, the broadcast cross-attention head mACH reformulates standard cross-attention so one visual forward pass scores many text queries at once, expanding supervision from pooled text vectors to token-level centered subspaces. Second, a text-conditioned JEPA auxiliary stream reconstructs masked visual features under language guidance using an EMA teacher; the fluctuation of its gradient under mask resampling is what supplies the full-support spectral floor $\\Gamma_J \\succeq c_J I$. Third, O365-Caption replaces discrete detection tags with context-aware referring expressions. The theoretical load-bearing identity is directional alignment capacity $\\mathrm{cap}(k) = \\mathrm{Var}_m(x_m^\\top k) = k^\\top \\Xi_X k$, with Lemma 2 showing that directions without gradient support decay exponentially under weight decay.","core_discovery":"The core claim, formalized as Theorem 8, is that training objectives differ in which feature directions they supervise. A pooled contrastive loss spans only the pooled text-vector directions (dimension at most $N_c$), the mACH head spans centered token-key subspaces (dimension at most $N-N_c$), and the dual-stream objective with the text-conditioned JEPA stream spans the full feature space $\\mathbb{R}^C$ almost surely. Under weight decay, only the dual-stream objective avoids alignment-blind directions, so representation diversity, counted as the number of alignment-active directions, scales along the ladder $N_c < N-N_c < C$. The empirical counterpart is a 75M static checkpoint reaching 85.3/89.0/82.5 on RefCOCO zero-shot and 91.7/93.0/90.2 after fine-tuning, matching or beating 13B multimodal LLMs on several splits.","pith_inferences":["Because the proof only requires the auxiliary objective's gradient fluctuation to have full support, any reconstructive or self-supervised auxiliary loss with that property could plausibly substitute for JEPA; the specific architecture may matter less than the full-support property.","The O365-Caption pipeline is described by the authors as dataset-agnostic, applicable to other detection corpora and to repairing noisy phrase-grounding corpora, so the data-side contribution extends beyond Objects365.","Even if the full-support proof holds, the paper's own limitation appendix warns that representational capacity is not the same as end-task accuracy under shift; using effective rank as a training-time monitor for transfer readiness would be a natural testable extension."],"forward_implications":["A single 75M checkpoint can do zero-shot referring-expression comprehension across RefCOCO, RefCOCO+, and RefCOCOg, and fine-tuning the same weights yields competitive state-of-the-art numbers on all three benchmarks.","The JEPA branch adds training-time gradient diversity at no inference cost because it is removed after training; the reported latency stays close to the mACH-only head.","Feature covariance rank should climb from at most $N_c$ under contrastive training to at most $N-N_c$ under mACH and to $C=768$ under mACH+JEPA; the paper reports effective ranks 36, 44, and 83, consistent with that ladder.","Novel out-of-distribution expressions whose key directions lie outside the training subspace stay alignment-blind for discriminative-only heads but retain positive capacity under the dual-stream objective.","Scaling the head feature dimension $C$ from 768 to 1024 removes representation bottlenecks and yields further gains, as shown by the ConvNeXt-Base results."],"supporting_citations":[{"why":"Provides the Objects365 corpus with 9.6M annotations that O365-Caption rewrites into referring expressions.","marker":"(Shao et al. 2019)"},{"why":"Introduces the Joint Embedding Predictive Architecture that the auxiliary stream adapts for text-conditioned reconstruction.","marker":"(LeCun et al. 2022)"},{"why":"Supplies the masking and EMA-teacher recipe the JEPA branch follows.","marker":"(Assran et al. 2023)"},{"why":"Documents multimodal representation collapse, the failure mode the theoretical analysis targets.","marker":"(Chaudhuri et al. 2025)"},{"why":"Provides the GLIP contrastive alignment baseline whose subspace limitation Lemma 3 formalizes.","marker":"(Li et al. 2022)"},{"why":"Provides the Grounding DINO baseline and open-set grounding recipe the method compares against.","marker":"(Liu et al. 2024)"},{"why":"Defines the cleaned RefCOCO evaluation protocol used for all reported accuracy numbers.","marker":"(Chen et al. 2025b)"},{"why":"Provides the effective-rank metric used to measure spectral diversity of the learned features.","marker":"(Garrido et al. 2023)"}],"fun_headline_variants":["Feature diversity lets a 75M model match 13B LLMs at zero-shot grounding","Reconstruction loss keeps feature directions alive for scalable grounding","Keep all feature directions active: 75M grounding model rivals 13B LLMs","Feature diversity, not model size, is key for open-vocabulary grounding"],"cache_read_input_tokens":25472,"weakest_assumption_plain":"The theorem's full-support claim collapses if the JEPA mask-noise fluctuation is confined to a lower-dimensional plane; the paper assumes this does not happen (Assumption 6, $\\Gamma_J \\succeq c_J I$ with $c_J > 0$), and its own Appendix F warns that the theory addresses representation capacity rather than final accuracy or calibration.","fun_headline_variants_meta":{"raw":{"variants":["Feature diversity lets a 75M model match 13B LLMs at zero-shot grounding","Reconstruction loss keeps feature directions alive for scalable grounding","Keep all feature directions active: 75M grounding model rivals 13B LLMs","Feature diversity, not model size, is key for open-vocabulary grounding"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000799,"raw_usage":{"total_tokens":3502,"prompt_tokens":917,"completion_tokens":2585,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":533,"completion_tokens_details":{"reasoning_tokens":2502}},"tokens_in":533,"tokens_out":2585,"duration_ms":17260,"temperature":1.0,"reasoning_tokens":2502,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T00:04:14.292733+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"For a fixed image and fixed text, sample many JEPA masks and compute the gradient of $\\mathcal{L}_{JEPA}$ with respect to the shared visual features; estimate the minimum eigenvalue of the covariance of these centered gradients across masks. If that minimum sits at the numerical floor rather than at $c_J > 0$, Assumption 6 fails and the claimed dimension-$C$ support collapses. A simpler proxy is the empirical effective rank of $\\Xi_X$: if the mACH+JEPA spectral tail above the float32 floor does not persist across all 768 dimensions, the predicted spectral floor is contradicted.","supporting_citations":[],"review_version":1}