{"id":"1c90b9ac-1dda-4c9c-ba90-d8c944d47d36","arxiv_id":"2412.09817","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"Simignore improves multimodal LLM complex question answering on ScienceQA by masking image tokens whose embeddings have low cosine similarity to the text prompt.","lead":"The paper presents Simignore, a method that removes image tokens a vision-language model is unlikely to need by measuring how similar each image patch is to the question text. In tests on ScienceQA, pruning these low-similarity tokens improved accuracy for LLaVA and Mipha models while also cutting compute.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The paper's central causal claim is unverified: it never shows that the image tokens selected by cosine similarity are the tokens whose removal changes the answer, and K is tuned on the test set.","rationale":"The reader's weakest_assumption identifies the causal gap: the paper never verifies that similarity-selected tokens are the tokens whose masking changes the answer. I agree with this as the most load-bearing concern. Table 3 does provide some evidence that selection matters (unimportant > random > important), but it is confounded by using the same similarity metric to define both the selection and the importance labels, and it is performed at a single K on a single model. The test-set selection of K is a further independent problem: Table 2 sweeps K from 72 to 576 on what appears to be the evaluation split, so the reported best accuracy at K=124 is not a held-out result. The absence of error bars and comparison with FastV/LLaVA-PruMerge makes it impossible to distinguish the proposed mechanism from generic token pruning or noise reduction. These weaknesses are addressable with the proposed validation-split K selection and the counterfactual overlap test. Because the reader already assigned CONDITIONAL based on these issues, my stress-test does not change the verdict; it sharpens the specific experiment needed to move from correlation to causation.","tokens_in":12299,"tokens_out":3403,"duration_ms":40270,"concrete_test":"Run a counterfactual causality check on a validation split of ScienceQA. For a random sample of 200 questions, compute for each image token a causal importance score: mask that token alone, then measure the change in the model's log-probability of the correct answer. Build the set of the K=124 most causally important tokens per question, and compute its overlap (e.g., Jaccard index or rank correlation) with the set of K=124 tokens selected by Simignore's cosine similarity. If the overlap is no better than a random baseline, the mechanism is not supported. Also select K on the validation split and report the test accuracy for that K; if the gain over baseline shrinks, the test-set K sweep is the source of the improvement.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that Simignore improves complex reasoning by ignoring image tokens that are irrelevant/unimportant to the text. For this to hold, the top-K tokens selected by Eq. 11 must coincide with tokens whose removal (masking) makes the model answer correctly. The paper provides only qualitative evidence: attention maps in Fig. 2, a single case study in Fig. 7, and one worked example (id 1879). Table 3 shows that ignoring 'unimportant' tokens (defined by low similarity) outperforms ignoring 'important' or 'random' tokens, but this is correlational: both the selection and the evaluation are based on the same similarity score, so it does not establish that the selected tokens are causally responsible. A different mechanism, such as regularization from dropping a fixed number of tokens, could produce the same accuracy gain. Moreover, Table 2 selects K=124 by sweeping K on the ScienceQA test set itself; the reported 65.15→68.02 gain may be an artifact of test-set overfitting. No error bars or multiple seeds are reported, and no comparison is made with prior token-reduction baselines (FastV, LLaVA-PruMerge) even though they are cited. Together, these gaps leave the central claim under-supported: the method works (if it works) but the paper does not demonstrate that it works for the stated reason.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Simignore, a training-free image token reduction method for large vision-language models (LVLMs). During inference, it computes cosine similarity between image token embeddings and text token embeddings, then masks out the K image tokens with the lowest similarity to the text, aiming to improve complex reasoning accuracy by removing irrelevant or unimportant visual information. The method is evaluated on the ScienceQA-image benchmark with LLaVA1.5-7B, LLaVA1.5-13B, and Mipha-3B, reporting accuracy gains (e.g., LLaVA1.5-7B from 65.15% to 68.02%) and reduced runtime. The paper also includes attention visualization, a k-means clustering case study, and ablations on the number of ignored tokens, similarity algorithms, and importance levels.","tokens_in":12560,"tokens_out":6068,"duration_ms":63322,"significance":"If the reported gains are robust, Simignore offers an attractive zero-training plug-in for improving LVLMs on visual reasoning tasks, and the information-flow observation provides a useful interpretability perspective. The method is conceptually simple and has a single hyperparameter K. The paper promises public code, which would aid reproducibility. However, the significance is currently tempered by several issues: evaluation is limited to a single benchmark, no comparison is made with prior token-reduction methods, K is selected on the test set, and the causal link between similarity-based token removal and accuracy improvement is not directly verified. The central claim is plausible but not yet established; the paper reads as a promising workshop-level contribution rather than a fully validated research result.","major_comments":[{"comment":"The hyperparameter K=124 is selected as the best value from a ten-point scan over the ScienceQA-image test set, and the reported improvement (65.15% to 68.02%) is the best-case result from that scan. Because no separate validation split is used for hyperparameter selection, the reported accuracy is an optimistic estimate and does not reflect the performance of the method with a fixed K chosen a priori. Please report results on a held-out validation set for selecting K, or provide a clear selection rule and discuss how the reported gain changes under that rule.","section":"Ablation Study, Table 2"},{"comment":"The paper cites FastV and LLaVA-PruMerge as prior token-reduction methods in the Related Work, but Table 1 only compares against whole-model baselines and does not include any token-reduction baseline. Since the central contribution is a token-reduction method, the claim of effectiveness requires direct comparison with the most relevant prior work under the same evaluation protocol. Please add these comparisons, as their absence leaves the relative improvement unclear.","section":"Experiment, Table 1"},{"comment":"The causal mechanism is not validated. The paper claims Simignore works by ignoring irrelevant image tokens, but the evidence is correlational: Table 3 contrasts ignoring tokens with high versus low similarity, using the same similarity score for selection and evaluation. This does not establish that the selected tokens are the ones whose removal is causally responsible for better answers. A direct test would be to check whether masking the top-K similar tokens changes the model's answer more often than masking K random tokens or K low-attention tokens, and whether the tokens that actually flip the answer coincide with the low-similarity tokens. As it stands, the accuracy gain could arise from a different mechanism such as noise reduction or length regularization.","section":"Method / Ablation Study, Table 3"},{"comment":"The token-selection procedure is under-specified and potentially flawed. The flatten-and-argsort in Eqs. (13)-(14) selects the top-K image-text pairs, not the top-K distinct image tokens; an image token can appear multiple times among the top-K pairs, so the final attention mask may ignore fewer than K distinct tokens. Additionally, the paper does not specify which embeddings are used for the similarity computation (e.g., the LLM input embeddings after the projector, or the vision encoder outputs before projection) and how the feature-alignment function F⅁ is implemented for LLaVA. Please clarify the selection procedure so that exactly K distinct image tokens are ignored, and specify the exact embedding sources used.","section":"Method, Eqs. (11)-(15)"},{"comment":"All accuracy numbers are reported as single values without variance, error bars, or significance tests. The gains are small (e.g., 0.45 points for Mipha-3B and 1.14 points for LLaVA1.5-13B) and could be within run-to-run variation. Please report the mean and standard deviation over multiple runs (at least three random seeds) and, if feasible, a paired significance test for the main comparisons.","section":"Experiment, Tables 1-4"}],"minor_comments":[{"comment":"The title uses 'Complex Reason'; it should be 'Complex Reasoning'.","section":"Title"},{"comment":"The left-hand side λj_img is indexed by j, but the right-hand side sums over j ∈ I, so the index is inconsistent; the left-hand side should likely be λ_i_img or another index.","section":"Method, Eq. (3)"},{"comment":"The word 'unsequenced' appears to be a typo for 'unsqueezed' or 'reshaped' when describing the attention matrix transformation.","section":"Method, paragraph after Eq. (3)"},{"comment":"The caption uses 'regularization' where 'normalization' is meant.","section":"Figure 3 caption"},{"comment":"The reference list contains an incomplete entry: 'Advancing Pose-Guided Image Synthesis with Progressive Conditional Diffusion Models' has '????' instead of a year, and the citation is malformed.","section":"References"},{"comment":"The limitations section only mentions future work on adaptively choosing K; it should also acknowledge the test-set selection of K, the lack of comparison with token-reduction baselines, and the correlational nature of the evidence.","section":"Discussion and Limitations"}],"recommendation":"major_revision","confidential_remarks":"The stress-test concerns largely land. The most serious issue is the selection of K on the test set, which inflates the reported gains and undermines the paper's central effectiveness claim. The method is plausible and the idea is interesting, but the experimental validation is currently too weak for a top-tier venue: single-run numbers, no baselines for token reduction, and no direct test of the causal mechanism. The paper would need a proper validation split for K, multiple seeds, baseline comparisons, and a more rigorous analysis of why the selected tokens are causally relevant. The scope is also narrow (only ScienceQA-image), which further limits significance. I recommend major revision, not rejection, because the issues are fixable within the manuscript's scope."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First thing you should know: this paper proposes a genuinely different token-reduction rule—mask image tokens whose embeddings have low cosine similarity to the text prompt—and reports consistent accuracy gains on ScienceQA across three LVLMs (e.g., LLaVA-1.5-7B from 65.15 to 68.02). That's a real departure from FastV (attention-based) and LLaVA-PruMerge (CLS-based), and the idea is simple enough to be deployment-friendly. The ablations are directionally coherent: ignoring low-similarity tokens helps more than ignoring random or high-similarity ones, and the paper includes a nice case study with clustering.\n\nThe soft spots are three, and the first two are load-bearing. One: K (the number of ignored tokens) is chosen on the test set. Table 2 sweeps ten values and picks the best accuracy on ScienceQA; that inflates the reported gain. There's no validation split or principled criterion. Two: no error bars or multiple seeds, so we don't know if 68.02 vs. 65.15 is noise. Three: the paper cites FastV and LLaVA-PruMerge but never compares against them. Without that, we can't tell whether the gain comes from text-conditioning or just from dropping a fixed number of tokens.\n\nThe stress-test note worries that the causal story is unverified—the paper never shows that the masked tokens are exactly the ones whose removal flips the answer. That's fair as a secondary concern, but I don't think it's fatal: the paper's primary claim is that the method improves accuracy, and the ablations at least support the direction of the effect. The mechanism explanation (\"spy tokens\") is speculative, but the paper flags it as conjecture.\n\nThe writing is rough in places (typos, some unclear notation), and the evaluation is single-benchmark. Still, the core idea is new and the experiments, while under-controlled, are clearly described. For anyone working on token reduction or LVLM efficiency, this is a useful data point. This is exactly the kind of paper that should get a serious referee: the method is worth testing, but it needs a validation split, seeds, and baseline comparisons before the numbers can be trusted. I'd send it to review with a request for major revision.","headline":"New idea (text-conditioned token pruning) with a plausible mechanism, but test-set-tuned K and missing baselines keep the main effect from being fully trusted.","tokens_in":13084,"tokens_out":3415,"would_cite":true,"duration_ms":36418,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that text-relevant image tokens are the ones that carry reasoning in vision-language models, and that masking out the rest with a text-similarity filter improves accuracy on complex visual questions.","keywords":["multimodal large language models","image token reduction","visual reasoning","attention interpretability","cosine similarity","ScienceQA","chain-of-thought reasoning","token pruning"],"falsifier":"On a fixed set of ScienceQA image questions, compare Simignore's selected K tokens with the K tokens whose individual masking actually changes the model's predicted answer. If the two sets do not overlap substantially, the accuracy gain cannot be attributed to removing causally relevant tokens.","tokens_in":12117,"feed_emoji":"🖼️","tokens_out":7268,"duration_ms":67847,"temperature":0.7,"pith_summary":"The paper argues that when a vision-language model answers a complex reasoning question, the image patches that matter are exactly those whose embeddings sit closest to the text embeddings, and that deliberately ignoring the rest improves both accuracy and speed. It proposes Simignore, which computes the cosine similarity between image-token embeddings and text-token embeddings, keeps the K most similar image tokens, and zeroes out the attention mask of the rest. On ScienceQA image questions, applying Simignore to LLaVA1.5-7B raises accuracy from 65.15 to 68.02 percent, LLaVA1.5-13B from 72.09 to 73.23 percent, and Mipha-3B from 70.40 to 70.85 percent, while reducing inference time from 303 to 279 seconds at K=124 ignored tokens. Ablations show that ignoring high-similarity tokens hurts accuracy while ignoring low-similarity tokens helps, and that cosine similarity beats Euclidean and Manhattan distances for token selection.","feed_headline":"Drop text-irrelevant image tokens; LLaVA accuracy climbs 2.9","feed_subtitle":"Simignore masks question-unrelated image patches, lifting ScienceQA accuracy from 65.15 to 68.02 percent.","key_machinery":"The load-bearing object is the cosine similarity matrix $S(i,j)$ between normalized image-token embeddings and normalized text-token embeddings (Eq. 11). Flattening and argsorting $S$ yields the indices of the $K$ image tokens with the strongest text affinity; the attention mask for every other image token is set to zero. The paper motivates this selection with an attention-convergence observation: overlaying decoder attention scores on the input image shows concentrated attention on text-related objects, such as a mushroom or copepod mentioned in the options.","core_discovery":"The paper's central claim is that in large vision-language models such as LLaVA1.5, image tokens semantically related to the prompt receive higher attention scores in the LLM decoding layers, a phenomenon the paper calls information-flow convergence, while unrelated tokens receive very little attention. Simignore exploits this by treating the top-K image tokens under normalized cosine similarity to the text as the essential visual context and masking out the rest before decoding. The authors report that this masking is not only a speed optimization: it improves complex reasoning accuracy across three backbones, and ablations show the effect depends on the similarity ranking, since dropping important tokens degrades accuracy and cosine similarity outperforms distance-based alternatives.","pith_inferences":["Because Simignore needs no training, its selected tokens could double as a zero-shot saliency map for diagnosing which image region a model used, though the paper does not evaluate this use.","The random-ignoring ablation (65.12 percent at K=124) is close to baseline, so the gain is not merely length regularization; an independent test would be to compare Simignore against a mask that removes tokens with the lowest attention scores rather than lowest text similarity.","An adaptive K, chosen per question from the similarity distribution or the model's confidence, could improve the accuracy-runtime frontier beyond the fixed K=124 setting the paper reports.","The method's reliance on text-embedding similarity may transfer poorly to questions where the relevant visual detail is not lexicalized in the prompt; benchmarks with implicit visual reasoning would test that boundary."],"forward_implications":["At K=124 of 576 image tokens ignored, ScienceQA image accuracy rises from 65.15 to 68.02 for LLaVA1.5-7B, from 72.09 to 73.23 for LLaVA1.5-13B, and from 70.40 to 70.85 for Mipha-3B.","Runtime on the LLaVA1.5-7B evaluation drops monotonically as more tokens are ignored, from 303 seconds at baseline to 279 seconds at the best-accuracy setting.","Ablations indicate that the benefit comes from removing low-similarity tokens: ignoring the most important tokens drops accuracy to 61.73, while ignoring unimportant tokens gives 68.02, and cosine similarity outperforms Euclidean and Manhattan distance.","A case study suggests that low-similarity image tokens can form a distinct embedding cluster whose removal can flip a wrong answer to the correct one.","Simignore requires no fine-tuning or training, operating purely through attention masks at inference time."],"supporting_citations":[{"why":"Supplies the ScienceQA benchmark used in all accuracy comparisons.","marker":"(Lu et al. 2022)"},{"why":"Defines the LLaVA and LLaVA1.5 backbones that Simignore is applied to.","marker":"(Liu et al. 2023)"},{"why":"Describes FastV, the attention-score-based token-reduction baseline that Simignore contrasts with.","marker":"(Chen et al. 2024)"},{"why":"Describes LLaVA-PruMerge, the clustering-based token-pruning baseline that Simignore contrasts with.","marker":"(Shang et al. 2024)"},{"why":"Provides the CLIP visual encoder that produces the image tokens used by the models.","marker":"(Radford et al. 2021)"}],"fun_headline_variants":["Similarity-based image pruning lifts LLaVA accuracy by 2.9 points","Simignore masks irrelevant image tokens, boosting LLaVA reasoning","Drop text-unrelated visual tokens to gain 2.9 on ScienceQA","Text-similarity token reduction sharpens LVLM complex reasoning","Ignore 70% image tokens via similarity; LLaVA complex QA up 2.9"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that text similarity is a faithful proxy for causal importance: the image tokens with the highest cosine similarity to the prompt are the ones that must stay for the model to answer correctly.","fun_headline_variants_meta":{"raw":{"variants":["Similarity-based image pruning lifts LLaVA accuracy by 2.9 points","Simignore masks irrelevant image tokens, boosting LLaVA reasoning","Drop text-unrelated visual tokens to gain 2.9 on ScienceQA","Text-similarity token reduction sharpens LVLM complex reasoning","Ignore 70% image tokens via similarity; LLaVA complex QA up 2.9"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000329,"raw_usage":{"total_tokens":1811,"prompt_tokens":899,"completion_tokens":912,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":515,"completion_tokens_details":{"reasoning_tokens":812}},"tokens_in":515,"tokens_out":912,"duration_ms":10194,"temperature":1.0,"reasoning_tokens":812,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T16:41:10.324123+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"On a fixed set of ScienceQA image questions, compare Simignore's selected K tokens with the K tokens whose individual masking actually changes the model's predicted answer. If the two sets do not overlap substantially, the accuracy gain cannot be attributed to removing causally relevant tokens.","supporting_citations":[],"review_version":1}