{"id":"a95c199f-4632-47e7-80bf-d3c7bfb10140","arxiv_id":"2505.05626","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"A multimodal LLM trained with an auxiliary visual loss, input token masking, modality-disentangled weights, and synthetic spatial data shows spatial-reasoning gains that are not cleanly attributable to the proposed method.","lead":"This paper trains multimodal LLMs with an extra visual loss, masked input tokens, separate weights for image and text, plus synthetic spatial data, and reports gains on spatial reasoning benchmarks. The headline gains are confounded because the final model also swaps in a stronger vision encoder, and it does not beat its own earlier variants on two of four benchmark subsets.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Encoder confound in Table 2: PERCEPTLLM switches the input encoder from CLIP to AIM-v2 without an ablation, so the headline gains cannot be attributed to the proposed training innovations.","rationale":"The paper presents a coherent recipe and useful diagnostics, but its central claim is causal: the proposed training innovations produce the improvement. The most load-bearing assumption is that the final row's gains are attributable to those innovations rather than to the simultaneous switch to the AIM-v2 input encoder. The reader's weakest_assumption identifies exactly this confound, and the missing control is concrete: an encoder-swap ablation under the same recipe. In addition, Table 2 itself undermines the 'strongest results' sentence, since the +VISUALLOSSADV row outperforms PERCEPTLLM on Map and Real. These are addressable gaps—an encoder-swap ablation plus re-running the comparison baseline under the modified benchmark protocol would settle them—so a conditional verdict is appropriate. No change to the reader's verdict is needed.","tokens_in":9854,"tokens_out":9079,"duration_ms":95533,"concrete_test":"Train the full PERCEPTLLM pipeline twice with identical data, auxiliary teacher, BLANKTOKENS, disentangled weights, hyperparameters, and the same modified SpatialEval evaluation harness, changing only the input vision encoder between CLIP and AIM-v2. Compare the four subset scores. If the CLIP-input variant reproduces the PERCEPTLLM row (50.8/35.0/55.13/52.6), the encoder swap is not the cause; if it falls back toward the pre-AIM-v2 rows (about 43.5/26.7/64.1/53.3), then the stronger encoder explains the headline Grid/MazeNav gains and the training-innovation attribution fails.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is causal: the proposed training techniques (VISUALLOSS, BLANKTOKENS, disentangled weights, synthetic data) are responsible for PERCEPTLLM's gains. Section 4.2/Table 2 never support that attribution directly. Rows 1-5 keep the CLIP input encoder while adding the proposed components, but the final PERCEPTLLM row simultaneously introduces the AIM-v2 input encoder. The row-to-row jumps on Grid (43.5 to 50.8) and MazeNav (26.7 to 35.0), and the drops on Map (64.1 to 55.13) and Real (53.3 to 52.6), could be entirely due to that encoder swap. No ablation keeps the training recipe fixed and varies only the input encoder. Thus the condition required for the central claim, that the innovations rather than the stronger encoder cause the reported boost, is untested. The footnote saying the benchmark prompting/parsing was modified also weakens the comparison to Llama 3.2 unless that model was re-run under the same protocol, which is not reported. The 'strongest results' statement is therefore not established by the evidence presented.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes PERCEPTLLM, a multimodal LLM built on the LLaVA recipe with a Llama 3.1 8B backbone, and attributes its accuracy gains to four training interventions: VISUALLOSS (an auxiliary MSE loss matching the LLM's visual-token features to features from an independently pretrained vision encoder), BLANKTOKENS (masking the first N text tokens and a random 20% of later tokens during training), independent weights for image and text pathways within the transformer, and synthetic grid data generated from OpenImages objects. The evaluation has two parts: an upstream next-token-prediction loss on SpatialMM and subset accuracies on SpatialEval. The headline claim is that PERCEPTLLM obtains a more than 10 percentage point boost over the baseline and outperforms Llama 3.2 11B while being smaller and lower-resolution. The causal claim that the proposed techniques, rather than the simultaneous switch to the AIM-v2 input encoder, drive the improvement is the central point that needs scrutiny.","tokens_in":10104,"tokens_out":9653,"duration_ms":105369,"significance":"If established, the paper would be a useful empirical contribution to visually grounded MLLM training: VISUALLOSS is self-supervised in the sense of not requiring region annotations, BLANKTOKENS is a simple regularization, and the released prompting and parsing code aids reproducibility. The upstream token-level analysis is a nice microscopic check. However, as submitted the evidence does not yet support the causal attribution, because the final model confounds the training recipe with an input encoder swap and the modified SpatialEval protocol is not re-baselined. The evaluation is not circular: the auxiliary representations come from independently pretrained encoders and the benchmark is not used for training. The contribution therefore hinges on additional controlled experiments rather than on a logical flaw.","major_comments":[{"comment":"The row labeled PERCEPTLLM simultaneously introduces the AIM-v2 input encoder, while all earlier rows use the CLIP input encoder; there is no ablation that changes only the input encoder under a fixed training recipe. Consequently, the gains on Grid (43.5 to 50.8) and MazeNav (26.7 to 35.0) cannot be attributed to VISUALLOSS, BLANKTOKENS, independent weights, or synthetic data; they may be due entirely to the stronger encoder. The same confound makes the drops on Map (64.1 to 55.13) and Real (53.3 to 52.6) between VISUALLOSSADV and PERCEPTLLM uninterpretable. Please add an ablation that applies the proposed training recipe with the CLIP encoder and with AIM-v2, and report all four subsets per configuration.","section":"Section 4.2, Table 2"},{"comment":"The paper states that the SpatialEval prompting and parsing were modified for this work and that the code is released, but it does not re-run the baseline or Llama 3.2 under the modified protocol. Without such a re-baselining, the comparison against Llama 3.2 11B and even the baseline is not controlled, because changes to prompt templates and answer parsing can change measured accuracy independently of model capability. Please re-evaluate the baseline and at least Llama 3.2 with the released protocol, or quantify the effect of the prompt and parser changes.","section":"Section 4.2, footnote 2"},{"comment":"No error bars, confidence intervals, or significance tests are reported for either the next-token-prediction losses in Table 1 or the accuracies in Table 2. Given that several adjacent rows in Table 2 differ by only a few points (for example, the Real subset values 52.6 versus 53.3) and some baseline values are in the 15 to 50 range, the claimed improvements and the 1.52 versus 1.365 average comparison rest on point estimates whose variability is unknown. Please report variance across at least three evaluation runs or a statistical test.","section":"Tables 1 and 2"},{"comment":"The statement that PERCEPTLLM 'yields our strongest results' is not supported by Table 2 on two subsets: PERCEPTLLM scores 55.13 on Map, below the 64.1 of +VISUALLOSSADV, and 52.6 on Real, below the 53.3 of +Synthetic and +VISUALLOSSADV. The aggregate Average does not override these subset-level regressions; the text should either report the per-subset trade-offs honestly or provide a stated criterion under which the aggregate is the primary metric.","section":"Section 4.2 and Section 5"}],"minor_comments":[{"comment":"The semantics of the mask M appear inverted: the text says M determines whether to blank a token, but the formula keeps Tin when M is True and substitutes bId otherwise. Please align the formula with the intended behavior or reword the definition.","section":"Section 6.2, Eq. (3)"},{"comment":"The relationships among the auxiliary encoders I-JEPA and AM-Radio and the final input encoder AIM-v2 should be stated in one place; as written, the reader cannot tell which encoder is used in which row of Table 2 without cross-referencing the appendix.","section":"Sections 3.2, 3.4, and Table 2"},{"comment":"The Average column is labeled 'mean of normalized accuracy improvement'; please define the normalization in the caption and note whether the average is over the four subset ratios or a different quantity.","section":"Table 2 caption"},{"comment":"The claim that the disentangled architecture incurs 'no additional computational or memory access overhead during inference' should be quantified or softened; doubling the weight matrices increases memory residency even if per-token FLOPs are unchanged.","section":"Section 3.4"},{"comment":"The training setup lists beta, N, and the synthetic-data fraction, but not batch size, learning rate, number of steps or epochs, pretraining mixture size, or compute; please add these details for reproducibility.","section":"Section 6.4"},{"comment":"The title and abstract promise an analysis of visual attention, but the paper never measures attention weights; the evidence is per-token prediction loss and accuracy. Please add an attention analysis or adjust the framing.","section":"Title and Abstract"}],"recommendation":"major_revision","confidential_remarks":"The core issue is a confound rather than circularity: the auxiliary loss is matched to independently pretrained encoders and the SpatialEval benchmark is not in the training mixture. The missing AIM-v2-only ablation is blocking, and the modified evaluation protocol needs re-baselining. I would not accept without these experiments."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Worth a look if you work on multimodal grounding, but budget your time. The paper puts together a clean, practical recipe: an auxiliary MSE loss that makes the LLM's visual tokens predict features from a self-supervised encoder (I-JEPA, later AM-Radio), blanking the first N text tokens plus random masking to weaken language priors, disentangled vision/text transformer weights, and synthetic grid data from Open Images. None of that is individually revolutionary, but the combination is new, and the upstream analysis on SpatialMM (NTP loss) is a nice diagnostic. The writing is readable and the recipe is concrete enough to reproduce.\n\nThe soft spots matter more. The core claim is causal: the proposed techniques, not the stronger input encoder, drive the gains. Table 2 doesn't support that. Rows 1-5 keep the CLIP input encoder; the final PERCEPTLLM row simultaneously introduces AIM-v2 as the input encoder. So the jumps on Grid and MazeNav, and the drops on Map and Real, could be entirely an encoder effect. No ablation fixes the training recipe and varies only the input encoder. On top of that, the 'strongest results' statement is contradicted by the paper's own table on two of four subsets: PERCEPTLLM's Map is 55.13 vs 64.1 for the row above, and Real is 52.6 vs 53.3. Also, the benchmark prompt/parsing modifications are not re-run on Llama 3.2 under the same protocol, so that comparison is weakened, and there are no error bars or statistical tests across runs.\n\nI don't think this is a broken idea; the confound is addressable. The paper gives enough detail that an ablation would settle it, and the auxiliary-visual-loss idea is worth testing even if the current evidence doesn't isolate it. But as written, the headline claim is under-supported, and the overclaim about strongest results should be fixed.\n\nWho's this for? People designing training objectives for MLLMs, especially those interested in using self-supervised encoders as supervision targets. It deserves a serious referee: it's concrete, falsifiable, and the central recipe is plausible. But I would not cite it in its current form, and I'd want the ablation before relying on the numbers.","headline":"A plausible MLLM training recipe for visual grounding, but the headline SpatialEval gains are confounded by swapping the input encoder to AIM-v2 without an ablation; the central causal claim is not yet supported.","tokens_in":10633,"tokens_out":2165,"would_cite":false,"duration_ms":22556,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that blanking input tokens, adding a visual loss, using separate vision/text weights, and training on synthetic grids can make an 8B multimodal model surpass an 11B model on SpatialEval at lower input resolution.","keywords":["multimodal large language models","visual grounding","language priors","spatial reasoning","auxiliary visual loss","token masking","disentangled vision-text architecture","synthetic data"],"falsifier":"Run the same baseline with only the input encoder changed from CLIP to AIM-v2 and none of the proposed training modifications, then evaluate on Grid and MazeNav; if that model approaches 50.8 and 35.0 respectively, the training innovations are not the cause of the headline gains.","tokens_in":1691,"feed_emoji":"👁️","tokens_out":1810,"duration_ms":83881,"temperature":0.7,"pith_summary":"This paper tries to establish that the visual grounding weakness of multimodal LLMs is fixable through training rather than only through larger models or stronger encoders. It argues that MLLMs already develop an unlabeled, patch-level sense of image content, and that this latent ability can be amplified by four techniques: an auxiliary visual loss that makes the language backbone predict features from a strong image encoder, blanking leading text tokens so responses must start from the image, separate attention weights for image and text tokens, and synthetic grid images that force spatial reading. If the paper is right, the payoff is concrete: an 8B model at 224x224 resolution beats the Llama 3.2 11B model on SpatialEval, with more than 10 percentage points of gain on the grid task and consistent gains on other spatial subsets.","feed_headline":"Smaller 8B model beats 11B rival on spatial reasoning","feed_subtitle":"Training with auxiliary visual loss, blanked input tokens, and synthetic grids cuts reliance on language priors.","key_machinery":"The argument is carried by four training-time mechanisms acting on the LLM backbone, plus a diagnostic. VISUALLOSS adds an MSE term that forces the backbone's visual-token representations to match the representations produced by a pretrained I-JEPA encoder, so the language model must encode the whole image rather than only text-relevant regions. BLANKTOKENS replaces the first five input tokens and about 20% of later tokens with a designated blank token, preventing the model from autocompleting from language cues and forcing it to start from the image. The disentangled architecture gives image and text tokens separate self-attention weights, resolving the conflict between next-token prediction and the visual representation objective; this duplicates parameters but adds no inference-time compute because only the pathway for the current token's modality is active. Synthetic grid images built from Open Images objects with programmed questions supply training examples whose answers cannot be guessed from language priors. The upstream probe, reading the LM head's top token for each visual patch, shows the model already attaches semantic labels to image patches, motivating the interventions. The final PERCEPTLLM replaces the input encoder with AIM-v2, which the paper credits for the strongest results.","core_discovery":"The central claim is that MLLMs can be trained to rely on what they actually see instead of on language priors, and the paper supports this with a proposed model, PERCEPTLLM. The authors first probe internal visual tokens by passing them through the LM head and find that the model already attaches meaningful semantic labels to image patches without explicit supervision. Building on that, they add an auxiliary visual loss that forces the LLM backbone's visual-token representations to match I-JEPA's representations, blank out initial and random input tokens to block language-based autocompletion, give image and text tokens independent attention weights, and augment training with synthetic grid images whose answers cannot be guessed from text. The final configuration also swaps the input encoder to AIM-v2. The paper reports that this combination lowers next-token loss on the visually dependent SpatialMM dataset and raises SpatialEval accuracy across all subsets, with the largest gains on Grid and MazeNav, so that PERCEPTLLM outperforms the larger Llama 3.2 11B model while being more efficient in size, architecture, and input resolution.","pith_inferences":["Editorial inference: because the final PERCEPTLLM row swaps the input encoder to AIM-v2 under the same headline as the other techniques, the Grid and MazeNav gains are not cleanly attributable to the training modifications; an ablation with AIM-v2 alone under the same recipe would settle it.","Editorial inference: BLANKTOKENS is a form of input corruption during training, so a natural control is standard dropout or random token masking without a dedicated blank token; if that control matched the gains, the specific blank-token mechanism would not be the cause.","Editorial inference: the paper's Future Work passage acknowledges that the larger capacity of PERCEPTLLM may make the baseline data mixture and hyperparameters sub-optimal and could increase overfitting on synthetic data, a caveat that qualifies the generality of the reported configuration.","Editorial inference: the synthetic grid data could be varied in object count, grid size, and question type to test whether the technique generalizes from spatial layout to counting, matching, and relation tasks."],"forward_implications":["If PERCEPTLLM is right, visual grounding in MLLMs can be improved during training without extra visual annotations and without additional encoders at inference time.","A smaller model at lower input resolution can beat a larger, higher-resolution model on spatial reasoning benchmarks, which matters for deployment efficiency.","The consistent per-subset gains on SpatialEval suggest the techniques transfer across different kinds of spatial questions, not just grid layouts.","The upstream analysis predicts that visually dependent tokens, such as objects, directions, colors, and shapes, will be predicted with lower loss after training, giving a finer-grained diagnostic than overall accuracy.","Because the auxiliary encoder used for the visual loss can be swapped, the framework offers a path for future encoder advances to flow into the language backbone."],"supporting_citations":[{"why":"I-JEPA, the auxiliary image encoder whose representations the visual loss forces the LLM backbone to predict.","marker":"[2]"},{"why":"AIM-v2, the input encoder used in the final PERCEPTLLM configuration.","marker":"[5]"},{"why":"Mixture-of-Transformers, the basis for the disentangled vision/text pathways.","marker":"[9]"},{"why":"LLaVA, the base architecture and training framework that PERCEPTLLM layers its changes on.","marker":"[10]"},{"why":"SpatialMM dataset, used for the upstream next-token prediction analysis.","marker":"[16]"},{"why":"SpatialEval benchmark, whose Grid, MazeNav, Map, and Real subsets carry the headline accuracy comparisons.","marker":"[20]"},{"why":"Open Images v4, the source of objects and segmentation masks for the synthetic grid data.","marker":"[8]"},{"why":"Llama 3.1/3.2, the LLM backbone and the larger comparison model.","marker":"[6]"},{"why":"AM-Radio, the auxiliary encoder used in the VISUALLOSSADV variant that improves SpatialMap accuracy.","marker":"[13]"}],"fun_headline_variants":["Seeing over guessing: 8B model beats 11B with visual-focused training","Forcing visual focus yields 10-pt gain and beats larger 11B model","Multimodal model taught to see: curbs language priors, tops larger rival","See it, don't guess: training curbs language priors in multimodal AI","Pixels over priors: new training boosts multimodal visual smarts"],"cache_read_input_tokens":12800,"weakest_assumption_plain":"The load-bearing premise is that the reported gains come from the proposed training modifications, yet the final model also switches the input encoder to AIM-v2 and the paper never ablates that encoder swap alone under the same training recipe.","fun_headline_variants_meta":{"raw":{"variants":["Seeing over guessing: 8B model beats 11B with visual-focused training","Forcing visual focus yields 10-pt gain and beats larger 11B model","Multimodal model taught to see: curbs language priors, tops larger rival","See it, don't guess: training curbs language priors in multimodal AI","Pixels over priors: new training boosts multimodal visual smarts"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00106,"raw_usage":{"total_tokens":4408,"prompt_tokens":871,"completion_tokens":3537,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":487,"completion_tokens_details":{"reasoning_tokens":3432}},"tokens_in":487,"tokens_out":3537,"duration_ms":24523,"temperature":1.0,"reasoning_tokens":3432,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T23:00:36.638264+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same baseline with only the input encoder changed from CLIP to AIM-v2 and none of the proposed training modifications, then evaluate on Grid and MazeNav; if that model approaches 50.8 and 35.0 respectively, the training innovations are not the cause of the headline gains.","supporting_citations":[{"cited_title":"Self-supervised learning from images with a joint-embedding predictive architecture","cited_arxiv_id":null,"evidence_quote":"I-JEPA, the auxiliary image encoder whose representations the visual loss forces the LLM backbone to predict."},{"cited_title":"Visual instruction tuning","cited_arxiv_id":null,"evidence_quote":"LLaVA, the base architecture and training framework that PERCEPTLLM layers its changes on."},{"cited_title":"An empirical analysis on spatial reason- ing capabilities of large multimodal models","cited_arxiv_id":null,"evidence_quote":"SpatialMM dataset, used for the upstream next-token prediction analysis."},{"cited_title":"Is a picture worth a thousand words? delving into spatial reasoning for vi- sion language models","cited_arxiv_id":null,"evidence_quote":"SpatialEval benchmark, whose Grid, MazeNav, Map, and Real subsets carry the headline accuracy comparisons."},{"cited_title":"The open images dataset v4: Unified image classification, object detection, and visual relationship detection at scale","cited_arxiv_id":null,"evidence_quote":"Open Images v4, the source of objects and segmentation masks for the synthetic grid data."},{"cited_title":"Am-radio: Agglomerative vision foundation model reduce all domains into one","cited_arxiv_id":null,"evidence_quote":"AM-Radio, the auxiliary encoder used in the VISUALLOSSADV variant that improves SpatialMap accuracy."}],"review_version":1}