{"id":"cb34b385-e72f-4f81-8108-2206fa2d8a02","arxiv_id":"2501.14276","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A semantic-similarity weighting module for sub-images improves high-resolution vision-language model performance, pending clarification of whether evaluation benchmarks overlap with training data.","lead":"This paper proposes a module, GSWA, that assigns different weights to sub-images of high-resolution photos based on their semantic similarity to the whole image, instead of treating all sub-images equally. The authors report improved scores on several vision-language benchmarks when adding GSWA to the InternVL2-2B model, but they also fine-tune on some of the same benchmark datasets they later evaluate on.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Training/evaluation overlap on AI2D and DocVQA is not addressed; without disjoint-split evidence, the headline gains over untrained baselines may reflect data exposure rather than GSWA.","rationale":"The reader's CONDITIONAL verdict is appropriate. I initially looked for a technical flaw in Equation 7's use of the global <cls> attention as an information-density proxy; that assumption is plausible but under-validated. However, the more immediate threat to the central claim is empirical: the fine-tuning mixture in Section V-A contains AI2D and DocVQA, which are also evaluation benchmarks. The paper does not report split identifiers or whether the original InternVL2-2B checkpoint had prior exposure to these benchmarks. Because the main comparison tables pit SleighVL against the original InternVL2-2B and other models not fine-tuned on this mixture, the observed gains on those two benchmarks cannot be attributed to GSWA without additional evidence. This does not automatically falsify the paper: many other benchmarks, such as MME, MMB-CN, OCRBench, TextVQA, and HallusionBench, also improve, and the Table VI ablation uses a fine-tuned no-GSWA baseline. But the headline claim of being competitive with larger models leans on DocVQA, so the concern is load-bearing. The numerical inconsistency in the Section V-F text (MME 1907 vs Table VI 1913, OCRBench 800 vs 803) is secondary but reinforces the need for a careful re-check. Therefore I recommend keeping the CONDITIONAL verdict until the overlap question is answered; no change to the reader's verdict is needed.","tokens_in":16514,"tokens_out":7046,"duration_ms":68911,"concrete_test":"Request or reconstruct the exact image/question identifiers used in the GSWA training mixture and in the AI2D and DocVQA evaluation splits; compute the intersection. If any overlap exists, retrain SleighVL and the fine-tuned no-GSWA baseline on a version with those samples removed and re-report Tables IV and V. If no overlap exists, additionally fine-tune the original InternVL2-2B on the identical 602K mixture without GSWA and report the per-benchmark deltas; this isolates GSWA's contribution from training-data exposure and from the added module capacity.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section V-A lists AI2D, DVQA, ChartQA, DocVQA, GeoQA+, Synthdog-en, and others as GSWA fine-tuning data, while Sections V-D and V-E evaluate DocVQA Test and AI2D. The main comparisons in Tables II and V use the original InternVL2-2B checkpoint, which is not fine-tuned on this mixture, so SleighVL's DocVQA (87.1 vs 86.9) and AI2D (74.6 vs 74.1) numbers, and the claim of being competitive with larger models on DocVQA, can be explained by training-set exposure even if the official test split is nominally disjoint. DocVQA is a known case where train and test images can share document sources, so explicit image/question ID checks are necessary. The ablation baseline in Table VI is fine-tuned on the same data, so that comparison is less affected, but it is not the comparison used to support the headline model-vs-baseline and model-vs-SOTA claims. This makes the empirical core of the central claim unverifiable without split-level disclosure.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Global Semantic-guided Weight Allocator (GSWA), a module inserted between the vision encoder and projector of a sub-image-partitioning LVLM. GSWA takes the <cls> tokens of sub-images and a global thumbnail, passes them through transformer blocks, and uses the global token's attention over sub-image tokens to compute per-sub-image weights, which are then multiplied onto the patch features. The module is integrated into InternVL2-2B to form SleighVL. The authors first provide observational evidence (Table I) that removing sub-images with high global semantic similarity degrades performance more than removing low-similarity sub-images. They then evaluate SleighVL on 21 benchmarks, report gains over the original InternVL2-2B and competitive results with larger models, and include an ablation (Table VI) comparing GSWA to cross-attention, cosine-similarity weighting, and a fine-tuned baseline without GSWA.","tokens_in":16755,"tokens_out":6724,"duration_ms":57404,"significance":"If the reported results hold, GSWA provides a simple, modular way to make sub-image partitioning in high-resolution LVLMs adaptive, with a modest parameter overhead. The preliminary observation in Table I is a useful empirical study of how sub-image importance varies. The ablation in Table VI consistently shows a benefit of the learned attention-based weighting over uniform fine-tuning on eight benchmarks, which is the central evidence for the proposed mechanism. The paper also benefits from evaluation across a wide range of benchmarks. However, the lack of explicit train/test split disclosure for AI2D and DocVQA, which appear in both the training mixture and the evaluation tables, undermines the specific claims of competitiveness on those benchmarks, and the absence of variance estimates makes the small reported improvements difficult to interpret. These issues are addressable but require additional experiments and disclosure.","major_comments":[{"comment":"The training data list in Section V-A includes AI2D and DocVQA, and the same datasets are used in the evaluation tables (Table IV: DocVQA Test; Table V: AI2D). The paper never states whether the training and evaluation splits are disjoint. Because the primary comparisons in these tables are against the original InternVL2-2B checkpoint, which was not fine-tuned on this mixture, the reported gains on DocVQA (87.1 vs 86.9) and AI2D (74.6 vs 74.1) could be partly due to exposure to training data rather than to GSWA. DocVQA in particular has known cases where test images share document sources with training images. The authors must either verify and explicitly state that the official test splits are disjoint from the training samples (including image-level and question-level deduplication) or, failing that, report the fine-tuned baseline (w/o GSWA) on these benchmarks. Without this, the claims of competitiveness on DocVQA and AI2D are not verifiable.","section":"V-A, Tables IV and V"},{"comment":"The ablation study is the cleanest support for the central claim because it compares GSWA against a baseline fine-tuned on the same data. However, the ablation table omits DocVQA and AI2D, which are exactly the benchmarks where training/evaluation overlap is a concern. Please add these two benchmarks to Table VI so that the effect of GSWA on the contested sets can be assessed independently of fine-tuning. This would directly address whether the module itself helps on these datasets or only the added training data.","section":"V-F, Table VI"},{"comment":"No error bars, confidence intervals, or multiple-seed results are reported anywhere in the paper. Several improvements in Table VI are small in absolute terms (e.g., RealWorldQA 57.8 vs 56.6; HallusionBench 39.8 vs 37.5), and the benchmark scores are averages over test sets. Without an estimate of variance, it is not possible to determine whether these differences are statistically reliable. The authors should either provide multiple runs with standard deviations or use a significance test, at least for the central ablation table.","section":"V-F, Table VI"}],"minor_comments":[{"comment":"The text says SleighVL 'surpasses several larger models, such as Monkey (72.4/65.1), Cambrian-8B (75.9/64.9), and IDEFICS2-8B (76.8/65.1)' on MMB_EN and MMB_CN_V11. On MMB_EN, SleighVL (73.4) is actually lower than both Cambrian-8B (75.9) and IDEFICS2-8B (76.8); the statement is only correct for MMB_CN_V11. Please rephrase to avoid overstating the result.","section":"V-B"},{"comment":"In the discussion of Table V, IDEFICS2-8B is quoted as (39.1/86.2/88.7) when comparing HallusionBench/POPE/AI2D, but the table lists IDEFICS2-8B's AI2D score as 72.3 and its SQA-I score as 88.7. The 88.7 appears to be the SQA-I value, not AI2D. Please correct the text to match the table.","section":"V-E"},{"comment":"The text describing the cosine-similarity ablation says 'the MME score decreasing from 1907 to 1868', but Table VI reports MME=1902 for the cross-attention variant and MME=1868 for the cosine variant. The value 1907 does not appear in the table. Please reconcile the text with the table.","section":"V-F"},{"comment":"The term 'Average Decline' in Table VI is not defined. Please specify how it is computed (e.g., average relative decrease across the listed benchmarks).","section":"V-F"},{"comment":"Equation (7) is hard to parse: the notation {·}_global is introduced but not rigorously defined, and it is unclear whether the softmax is taken over all sub-image tokens for each head before averaging. Please clarify the exact tensor dimensions and the normalization steps.","section":"IV-B"},{"comment":"The paper describes SleighVL as 'lightweight' but does not report the parameter increase or inference-time overhead introduced by the GSWA module. Section VI mentions additional computational cost qualitatively; please provide concrete numbers (e.g., parameter count, FLOPs or latency) to justify the lightweight claim.","section":"V-A and VI"},{"comment":"There are a few typos: 'employe' in Section V-E and 'dierctly' in Section V-F. Please copy-edit the manuscript.","section":"II-A and V-E"}],"recommendation":"major_revision","confidential_remarks":"The paper is promising but the training/evaluation overlap on AI2D and DocVQA is a serious oversight that must be fixed before publication. The central ablation in Table VI is encouraging, but the contested benchmarks need to be added to that table or the splits need to be verified. I would also suggest the authors report variance across seeds; the reported margins are often small. If the authors can provide disjoint-split verification and a corrected ablation, the paper would be a solid contribution to high-resolution LVLM processing."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Knew you'd want a heads-up on this one. The genuine nugget is Table I: remove the sub-images most semantically similar to the global thumbnail and InternVL2-2B's scores crater; remove the least similar and they barely move. That is a clean demonstration that sub-image information density is not uniform, and it motivates a simple, plug-in fix—GSWA, which reads the global <cls> token's attention over sub-image <cls> tokens and weights each sub-image's features accordingly. That mechanism is new in the sub-image-partitioning literature, and the ablation shows it beats cosine-similarity weighting and cross-attention, and beats a fine-tuned baseline without the module. Credit where due: the idea is simple, the experiments are broad (21 benchmarks), and the module is cheap. I would actually run it on my own setup to see if the gains hold.\n\nThe soft spot is the one the stress-test note flags, and it is real. Section V-A lists AI2D, DocVQA, and DVQA as fine-tuning data; Tables V and IV report evaluation on AI2D and DocVQA. The comparison baseline, InternVL2-2B, is the released checkpoint, not fine-tuned on that mixture. So every point SleighVL gains on those benchmarks over the baseline is potentially just training-set exposure. The paper never states whether the official test splits were used and whether image/question IDs were checked for overlap. DocVQA is a known leak-prone dataset. If the splits are disjoint, fine; but the authors do not say, and the burden is on them. The Table VI ablation is less affected—the w/o-GSWA row is fine-tuned on the same data—but that is not the comparison the abstract and Section V-B use to sell the model.\n\nTwo smaller things. No error bars, no significance tests; with gains of 37 points on MME and 19 on OCRBench, that is tolerable, but the field has moved past bare point estimates. And the ablation text contradicts the table: it says MME dropped from 1907 to 1868 and OCRBench from 800 to 783, while Table VI shows 1913 to 1868 and 803 to 783. Sloppy, fixable, but it corrodes trust.\n\nBottom line: if the train/test splits are clean, this is a decent lightweight contribution. If they are not, the empirical core is inflated. The authors need to disclose splits; a reviewer will ask. The idea itself is worth a serious look.\n\nMy call: send it to review, but with a request for split-level disclosure and a consistency pass. I would bring it to reading group mainly as a case study in how train/eval overlap can quietly decide a paper's verdict.","headline":"A plausible lightweight weighting module for sub-image LVLMs with a genuinely clean motivating experiment, but the train/test overlap on AI2D and DocVQA must be disclosed before the headline gains can be trusted.","tokens_in":17284,"tokens_out":2524,"would_cite":false,"duration_ms":21116,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"GSWA, a learnable module that weights image tiles by their global semantic relevance, improves high-resolution vision-language understanding.","keywords":["high-resolution vision-language models","sub-image partitioning","global semantic guidance","per-tile weight allocation","human visual attention","GSWA module","SleighVL","multimodal understanding"],"falsifier":"Take high-resolution images where the task-relevant content is deliberately placed in a low-salience region, such as a small sign at the image edge, run SleighVL, and compare its per-tile weights with an independent ranking of tiles by how much the answer depends on them; if the lowest-weighted tiles turn out to be the ones whose removal hurts accuracy most, the central claim is wrong.","tokens_in":16315,"feed_emoji":"🧩","tokens_out":10895,"duration_ms":85934,"temperature":0.7,"pith_summary":"Sub-image partitioning lets high-resolution vision-language models (LVLMs) work within fixed-resolution vision encoders, but standard pipelines treat every tile equally. This paper claims that tiles whose content is more semantically aligned with the whole image carry more of the information needed for understanding, and that a lightweight module can learn to up-weight those tiles. The proposed Global Semantic-guided Weight Allocator (GSWA) reads only the summary tokens of the tiles and the global thumbnail, computes per-tile weights from the global token's attention, and rescales tile features before they reach the language model. Inserted into the InternVL2-2B framework, the resulting SleighVL model improves on the baseline across general, real-world, text-rich, and hallucination benchmarks while staying competitive with models several times larger. If the claim holds, adaptive tile weighting is a cheap, general upgrade for any sub-image-partitioning LVLM.","feed_headline":"Semantic tile weighting lifts a 2B vision-language model","feed_subtitle":"Adding GSWA to InternVL2-2B beats same-size rivals and closes the gap to much larger models.","key_machinery":"The GSWA module is the central object: a lightweight stack of four transformer blocks followed by a multi-head self-attention layer that operates only on the <cls> tokens extracted from the $N$ sub-images and the global thumbnail after pixel shuffle. Its output is the weight vector $w$ in Equation 7, where the global token's average attention over the sub-image tokens is normalized to sum to one; Equation 8 then scales each sub-image's patch embeddings by its weight. The design converts the paper's semantic-relevance hypothesis into a differentiable, task-tuned weighting mechanism, and because it touches only <cls> tokens, it adds a small parameter count and can be inserted into any sub-image-partitioning LVLM.","core_discovery":"The paper's central claim is that in high-resolution LVLMs that partition an image into tiles, the tiles whose <cls> token is most similar to the <cls> token of the whole-image thumbnail are the ones that preserve the model's visual understanding, and that this signal can be turned into a learnable per-tile weight. It supports the first part with a removal experiment: deleting the three most globally similar tiles from InternVL2-2B drops MME from 1876 to 1734 and OCRBench from 784 to 440, whereas deleting the three least similar tiles leaves scores almost unchanged. It then builds GSWA, which passes the <cls> tokens of all tiles plus the thumbnail through transformer blocks and uses the thumbnail token's average attention over tile tokens as a normalized weight vector; every tile's pixel-shuffled patch embeddings are multiplied by its weight before projection into the LLM. The resulting model, SleighVL, reports higher scores than InternVL2-2B on every benchmark where the baseline is listed in the comparison tables, including MME (1913), OCRBench (803), and TextVQA (75.9), and it matches or beats several 4B-13B models on real-world and text-rich tasks.","pith_inferences":["A text-conditioned version of the weight allocator, which the paper lists as future work, would likely outperform the image-only GSWA on question answering, since tile relevance is often question-dependent; this is an extension the authors did not test.","The weight vector could double as a saliency map for interpretability: inspecting which tiles receive high weight in SleighVL would give a post-hoc explanation of where the model looks.","If the semantic-relevance hypothesis transfers across encoders, the same weighting idea could apply to other tiled domains such as medical or satellite imagery, where informative regions are sparse.","Because the paper's removal experiment shows that low-relevance tiles can be dropped with little loss, the weighting signal may also serve as a token-compression schedule, reducing inference cost without retraining."],"forward_implications":["Any sub-image-partitioning LVLM can adopt GSWA as a plug-in, since it consumes only the <cls> tokens that such encoders already produce and leaves the vision encoder untouched.","The ablation results imply that the learnable attention interaction matters: replacing it with fixed cosine similarity costs about 2.39% average performance, and removing the module costs about 3.49% relative to the fine-tuned baseline.","The same weight signal could later be reused as a pruning criterion: tiles with low learned weight are candidates for token dropping, which would offset the extra visual tokens that partitioning introduces.","The reported gains are largest on OCRBench, where the score moves from 784 to 803, suggesting that document-heavy and text-rich tasks are where the semantic-relevance weighting pays off most.","Because GSWA sits between the vision encoder and the projector, it can be trained or fine-tuned without changing the rest of the model's weights."],"supporting_citations":[{"why":"Supplies the InternVL2-2B baseline, dynamic cropping configuration, and comparison numbers into which GSWA is inserted.","marker":"[11]"},{"why":"InternViT is the vision encoder that produces the <cls> and patch tokens GSWA consumes.","marker":"[2]"},{"why":"SalGAN saliency analysis motivates the bottom-up human-attention premise that information-dense regions are the important ones.","marker":"[6]"},{"why":"Monkey exemplifies the uniform sub-image partitioning approach that GSWA aims to improve on.","marker":"[4]"},{"why":"MME is a central benchmark in the removal experiment and the main comparison tables.","marker":"[19]"},{"why":"OCRBench shows the largest drop (784 to 440) when top-semantic tiles are removed, anchoring the information-density claim.","marker":"[20]"},{"why":"InternLM-XComposer2-4KHD is the larger high-resolution model that SleighVL is compared against on real-world and HRBench tasks.","marker":"[5]"},{"why":"ShareGPT4V-100K is part of the fine-tuning data used to train SleighVL.","marker":"[3]"}],"fun_headline_variants":["Global semantic tile weights lift high-res LVLMs","Learn to weight tiles by global similarity, not uniformly","SleighVL: semantic-guided tile weighting for sharper vision","Tile weight from global relevance beats uniform processing","Semantic similarity to whole image guides tile weighting"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"Everything rests on the assumption that the thumbnail's summary token, after self-attention, reliably represents the image's global semantics and that its attention to each tile's summary token tracks how much task-relevant information that tile actually contains.","fun_headline_variants_meta":{"raw":{"variants":["Global semantic tile weights lift high-res LVLMs","Learn to weight tiles by global similarity, not uniformly","SleighVL: semantic-guided tile weighting for sharper vision","Tile weight from global relevance beats uniform processing","Semantic similarity to whole image guides tile weighting"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000229,"raw_usage":{"total_tokens":1504,"prompt_tokens":999,"completion_tokens":505,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":615,"completion_tokens_details":{"reasoning_tokens":431}},"tokens_in":615,"tokens_out":505,"duration_ms":5501,"temperature":1.0,"reasoning_tokens":431,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T15:14:20.784812+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take high-resolution images where the task-relevant content is deliberately placed in a low-salience region, such as a small sign at the image edge, run SleighVL, and compare its per-tile weights with an independent ranking of tiles by how much the answer depends on them; if the lowest-weighted tiles turn out to be the ones whose removal hurts accuracy most, the central claim is wrong.","supporting_citations":[{"cited_title":"Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks","cited_arxiv_id":null,"evidence_quote":"InternViT is the vision encoder that produces the <cls> and patch tokens GSWA consumes."},{"cited_title":"Salgan: Visual saliency prediction with adversarial networks","cited_arxiv_id":null,"evidence_quote":"SalGAN saliency analysis motivates the bottom-up human-attention premise that information-dense regions are the important ones."},{"cited_title":"Monkey: Image resolution and text label are important things for large multi-modal models","cited_arxiv_id":null,"evidence_quote":"Monkey exemplifies the uniform sub-image partitioning approach that GSWA aims to improve on."},{"cited_title":"Ocrbench: on the hidden mystery of ocr in large multimodal models","cited_arxiv_id":null,"evidence_quote":"OCRBench shows the largest drop (784 to 440) when top-semantic tiles are removed, anchoring the information-density claim."},{"cited_title":"InternLM-XComposer2-4KHD: A pioneering large vision-language model handling resolutions from 336 pixels to 4k HD","cited_arxiv_id":null,"evidence_quote":"InternLM-XComposer2-4KHD is the larger high-resolution model that SleighVL is compared against on real-world and HRBench tasks."},{"cited_title":"Sharegpt4v: Improving large multi- modal models with better captions","cited_arxiv_id":null,"evidence_quote":"ShareGPT4V-100K is part of the fine-tuning data used to train SleighVL."}],"review_version":1}