{"id":"f1510c53-8353-49ed-946a-ee4300860456","arxiv_id":"2506.07138","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A learned spatial token fusion plus multi-block features lets a multimodal model use only 25% of its vision tokens while matching or exceeding baseline accuracy on eight benchmarks.","lead":"This paper shrinks the number of image tokens a large multimodal model must process by fusing neighboring tokens with a learned convolution, cutting vision token count to one quarter. The authors report accuracy that matches or beats the full-token baseline on eight visual question answering benchmarks, suggesting vision tokens are highly redundant.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 'lossless token reduction' claim in §3.4 is contradicted by the paper's own Table 3: STF adds a lossy strided conv, and MBTF+STF (66.3) underperforms MBTF-only (66.6).","rationale":"The paper's central empirical claim is plausible: a 25%-token model matches or exceeds the LLaVA-1.5 full-token baseline on average across 8 benchmarks. The method is clearly described, comes with released code and weights, and includes ablations. The main load-bearing weakness is the theoretical justification: Section 3.4 asserts 'lossless token reduction' by treating a strided convolution as a learnable concatenation, which is not a valid information-preserving operation. Table 3 itself reveals the loss: MBTF+STF (66.3) is below MBTF-only (66.6), so the STF stage discards information that the multi-block fusion would otherwise retain. This does not falsify the empirical result, but it removes the paper's principled argument for why aggressive downsampling should not hurt. Additionally, the reported superiority margin is small and based on single runs without error bars, so the central 'comparable or even superior' claim is fragile under seed variance. The proposed multi-seed ablation directly addresses both the lossless inconsistency and the statistical reliability of the comparison. I therefore keep the reader's conditional verdict: the paper is likely publishable after revision, but the 'lossless' terminology and the strength of the empirical claim need to be scaled back or supported with error bars.","tokens_in":13660,"tokens_out":10822,"duration_ms":117051,"concrete_test":"Run the MBTF-only and MBTF+STF configurations from Table 3 with at least 3 random seeds each, using the released code and identical hyperparameters, and report average score and standard deviation on the 8 benchmarks. If MBTF+STF is consistently lower than MBTF-only, the STF stage is lossy and the §3.4 'lossless' claim is empirically refuted; if the gap is within noise, the claim should be qualified as approximate rather than lossless.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that 25% vision tokens can match the full-token baseline is justified in the paper as 'lossless' token reduction (Sections 3.1 and 3.4). However, Eq. (2) defines STF as a 2x2 strided convolution with GeLU, not a concatenation. A strided convolutional pooling is a learned nonlinear map that is not information-preserving in general, so the 'lossless' framing is mathematically unsupported. More importantly, the paper's own ablation in Table 3 provides direct evidence that the STF stage loses information: MBTF+STF averages 66.3 across the 8 benchmarks, while MBTF-only averages 66.6. Thus the spatial reduction is lossy, and the empirical 'comparable or even superior' result is not explained by the proposed lossless mechanism. The quantitative support for the central claim also rests on single-run evaluations without error bars; the 0.8-point average gain over LLaVA-1.5 full tuning could be within run-to-run variance. Since the code is publicly released, these concerns are testable.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a two-stage vision token compression method for LLaVA-style large multimodal models. A Multi-Block Token Fusion (MBTF) module concatenates features from eight evenly sampled CLIP ViT-L/14 blocks and fuses them with 1x1 convolutions, and a Spatial Token Fusion (STF) module applies a strided k x k convolution (default k=2) to fuse adjacent tokens, reducing the token count to 25% of the original. The reduced model is trained with the standard LLaVA-1.5 two-stage procedure and evaluated on eight vision-language benchmarks. The authors report an average of 66.3% at 1.9 TFLOPs, slightly above the full-token LLaVA-1.5 full-tuning baseline (65.5%) and above several other efficient LMM methods, while ablations examine the contribution of each module, the kernel size k, the number of fused tokens E, and alternative fusion strategies.","tokens_in":13946,"tokens_out":6335,"duration_ms":49067,"significance":"If the reported results are robust, the paper provides evidence for strong spatial redundancy in CLIP visual features and offers a practical way to cut the compute of LMM inference by roughly 4x with minimal accuracy change. The release of code and weights is a concrete strength, as is the inclusion of a simple average-pooling baseline that helps calibrate how much redundancy exists. The ablations are fairly thorough, covering module contributions, fusion kernel size, number of output tokens, and fusion strategies. The main weaknesses are that the central 'lossless' claim is mathematically unsupported and internally contradicted by Table 3, and that all headline comparisons rest on single runs without variance estimates, so the claimed superiority over the full-token baseline is not yet established with statistical confidence.","major_comments":[{"comment":"The claim that token reduction is 'lossless' is not justified. Section 3.1 states the aim of reducing aligned tokens 'in a lossless manner', and Section 3.4 argues that a 2x2 convolution with C2 = k^2 * C1 achieves 'lossless token reduction'. However, Eq. (2) defines STF as a strided 2x2 convolution followed by GeLU; a nonlinear strided convolution is not an information-preserving map in general. Moreover, the paper's own ablation in Table 3 shows that adding STF to MBTF decreases the average score from 66.6 to 66.3, which is direct evidence that the spatial reduction is lossy. The authors should reframe the contribution as learned lossy compression that preserves task-relevant information, and remove the 'lossless' terminology.","section":"Sections 3.1 and 3.4, Eq. (2)"},{"comment":"The central empirical claim of 'comparable or even superior performance' to LLaVA-1.5 relies on single training runs with no error bars or significance testing. The headline 0.8-point average gain over full-token LLaVA-1.5 full tuning (66.3 vs. 65.5) could plausibly be within run-to-run variance, especially since several individual benchmarks are lower for the proposed method (e.g., GQA 61.9 vs. 62.0, TextVQA 57.4 vs. 58.2, POPE 86.0 vs. 85.9 is a tie within rounding). The authors should report results over multiple seeds or at least provide a statistical significance test (e.g., paired bootstrap) to support the superiority claim.","section":"Tables 2 and 3"},{"comment":"The best configuration (k=2, E=1) is selected by looking at performance on the same eight benchmarks used for the final comparison, which introduces selection bias. The reported 66.3 average is therefore an optimistic estimate of the method's expected performance on those benchmarks. The authors should either use a validation split for hyperparameter choice or report performance across a small grid of hyperparameters without cherry-picking the best row, to make the comparison in Table 2 fair.","section":"Tables 2 and 4"}],"minor_comments":[{"comment":"The FastV row has a missing SQA score and a missing VQAT score, yet an average is reported; the authors should state how averages are computed when entries are missing, or fill in the missing values, so that the comparison is apples-to-apples.","section":"Table 2"},{"comment":"The explanation that larger kernel sizes overfit 'due to the deficit of training data' is speculative; no training loss, generalization gap, or variance evidence is provided. It would be better to report the number of parameters added by each kernel size and, if possible, training curves.","section":"Section 4.3.2"},{"comment":"MME-Perception is listed as an evaluation benchmark but does not appear in Table 2; either add the corresponding result or remove the benchmark from the setup description.","section":"Section 4.1.2 and Table 2"},{"comment":"The text refers to the cattle-counting example as 'Figure 3 (d)', but in the figure the cattle image is panel (e); the panel references should be corrected. There are also several typos, including 'pretrainig', 'finetuing', and 'acheves'.","section":"Figure 3 and Section 4.4"}],"recommendation":"major_revision","confidential_remarks":"The 'lossless' framing in Sections 3.1 and 3.4 is likely to draw criticism from reviewers and should be softened regardless of the empirical results. The single-run results are the main risk for the central claim; asking for multiple seeds is feasible given the released code. The contribution is somewhat incremental relative to TokenPacker and LLaVA-Mini, but the multi-block fusion with STF is a reasonable design choice and the public code is a plus."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a useful, honest efficiency paper whose main result—matching or beating LLaVA-1.5 on 8 benchmarks with 25% of the vision tokens—looks credible, especially since code and weights are public. But the paper overclaims by calling the spatial fusion 'lossless,' and its own ablation undercuts that word. I'd send it to review, with the expectation of a revision.\n\nWhat's actually new is the combination of two ideas: MBTF, which pulls features from 8 evenly spaced ViT blocks and fuses them with 1x1 convs, and STF, a learned strided kxk conv that reduces the token grid. Neither component is brand new—TokenPacker and LLaVA-Mini do adjacent things—but the specific pairing and the thorough ablations (fusion modules, kernel size, E, fusion strategies) give a clear picture of what matters. The authors also deserve credit for releasing code and trained weights; that makes the single-run results independently checkable.\n\nThe main soft spot is the 'lossless' claim in Sections 3.1 and 3.4. A 2x2 strided conv with GeLU is not a concatenation and is not information-preserving in general. The paper's own Table 3 confirms this: MBTF+STF averages 66.3, while MBTF alone averages 66.6. So the spatial fusion is empirically lossy—it trades a small accuracy drop for a 4x FLOP reduction. That's a perfectly reasonable trade, but calling it lossless is wrong and should be fixed. The central claim doesn't depend on the word; the model still beats the full-token baseline. But the framing needs to change.\n\nTwo other issues. First, the 0.8-point average gain over full-tuned LLaVA-1.5 comes from single runs without error bars. On several benchmarks the differences are within a plausible run-to-run noise band. Since the code is out, adding multiple seeds or at least acknowledging this would strengthen the paper. Second, TokenPacker is cited but never compared, which is a noticeable omission given how similar the approach is. A direct comparison would help position the work.\n\nMinor: some typos ('cases study'), and the MME benchmark is listed in Section 4.1.2 but not used in the tables. None of these are fatal.\n\nBottom line: this paper is for people working on efficient LMMs who want another solid data point on token reduction. It deserves a serious referee, and with a revised lossless claim, a TokenPacker comparison, and some uncertainty quantification I'd be comfortable citing it.","headline":"A solid efficiency result with a mislabeled 'lossless' fusion and a missing TokenPacker comparison; worth refereeing after those are fixed.","tokens_in":14423,"tokens_out":2340,"would_cite":true,"duration_ms":25891,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Fusing neighboring vision tokens with a learnable convolution and supplementing features from eight frozen-encoder layers lets a LLaVA-1.5 model match its full-token baseline using only 25% of the vision tokens, at 1.9 TFLOPs instead of…","keywords":["large multimodal models","vision token compression","spatial token fusion","multi-block token fusion","token reduction","inference acceleration","LLaVA","vision-language benchmarks"],"falsifier":"A concrete test: on a dense fine-grained benchmark—say TextVQA at higher resolution or a small-object counting set—compare the 25%-token STC model against the full-token LLaVA-1.5 baseline and against the MBTF-only model at full FLOPs. If STC drops substantially below the baseline while MBTF-only does not, the spatial-redundancy premise fails; if both drop together, the loss lies in multi-block feature fusion rather than in the 4x spatial reduction.","tokens_in":13495,"feed_emoji":"⚡","tokens_out":12789,"duration_ms":113852,"temperature":0.7,"pith_summary":"This paper tries to establish that the long vision-token sequences fed into large multimodal models are heavily spatially redundant, so a learnable fusion of neighboring tokens can cut the sequence to a quarter of its original length without sacrificing benchmark performance. The authors show that even plain average pooling to 25% of the tokens loses little on most benchmarks, and that their two-stage fusion—concatenating features from eight layers of the vision encoder, whose weights remain frozen, then fusing 2x2 neighborhoods with a learnable convolution—matches or exceeds the full-token LLaVA-1.5-7B on the average of eight vision-language benchmarks while dropping inference cost from 7.6 to 1.9 TFLOPs. If correct, this means much of the compute spent on long vision prefixes is unnecessary, and that fixed vision encoders can be compensated by multi-layer feature fusion instead of being retrained.","feed_headline":"One fusion trick matches LLaVA-1.5 with 25% of its vision tokens","feed_subtitle":"Learnable 2x2 fusion plus multi-layer features holds 66.3 percent average at 1.9 TFLOPs, versus 65.5 at 7.6.","key_machinery":"Spatial Token Fusion (STF) is a convolution with kernel and stride 2x2 that maps a 24x24x1024 token map to 12x12x4096, so each compact token absorbs four adjacent tokens and its 4096-channel width matches the text-embedding width of the LLM, making the reduction lossless in the channel dimension. Multi-Block Token Fusion (MBTF) concatenates the outputs of blocks 3, 6, 9, 12, 15, 18, 21, and 24 of the frozen CLIP ViT-L/14 encoder and fuses them with two 1x1 convolutions (4096 then 1024 channels) before STF runs. A reshape step allows E fused tokens per neighborhood, with E=1 by default, and the reported gains use that setting.","core_discovery":"The central claim is that spatial redundancy, not token count, dominates the vision-token cost of LLaVA-style models. Concretely, the paper's combined model (STC) compresses the 576 vision tokens of LLaVA-1.5-7B to 144 tokens—25% of the original—and still averages 66.3% across GQA, SQA, TextVQA, POPE, MMBench, MMBench-CN, VQAv2, and VisWiz, against 65.5% for the full-token LLaVA-1.5 full-tuning baseline, at 1.9 TFLOPs versus 7.6 TFLOPs. The authors attribute the result to two complementary modules: MBTF, which fuses intermediate-block features and alone lifts the full-token baseline by 1.1 points, and STF, which alone holds the baseline at 25% tokens. They caution that on some individual benchmarks the compressed model is still inferior even though the average is higher.","pith_inferences":["An extension the paper does not run: if 4x spatial fusion is nearly lossless at 336x336 resolution, the same fusion ratio at higher input resolutions (e.g., 672x672) should preserve proportionally more detail and may push the average further above baseline.","Because STF removes spatial redundancy before the LLM while text-guided methods remove task-irrelevant tokens, combining both families should compound savings; the paper's analysis does not rule this out.","MBTF's success suggests that fusing a few intermediate blocks of a frozen encoder could serve as a general drop-in replacement for unfreezing the encoder in other LMMs, not just LLaVA-1.5.","The 4096-to-4096 channel match means the fusion bottleneck is purely spatial; if that is right, further gains at 25% tokens must come from better multi-block fusion, not from keeping more tokens per neighborhood."],"forward_implications":["A 4x reduction in vision tokens cuts the measured inference cost of LLaVA-1.5-7B from 7.6 to 1.9 TFLOPs while holding the 8-benchmark average at or above the full-token baseline.","Plain average pooling to 25% of the tokens already outperforms several published efficient-LLaVA methods, indicating that the redundancy is structural and not specific to the learnable fusion.","MBTF's standalone improvement over the full-token baseline implies that intermediate-block features carry information the final block alone misses, so token compression can be paired with layer fusion rather than encoder unfreezing.","Aggressive fusion (k=4 or k=8) degrades accuracy, so the paper's claim is for a moderate 4x spatial reduction, not for arbitrary token compression."],"supporting_citations":[{"why":"Defines the full-token LLaVA-1.5-7B baseline and training recipe that STC modifies and is measured against.","marker":"[21]"},{"why":"Supplies the LLaVA-style two-stage pretrain-then-instruction-tune architecture and the data schedule the method inherits.","marker":"[22]"},{"why":"Provides the frozen CLIP ViT-L/14 encoder whose blocks 3 through 24 are sampled by MBTF.","marker":"[27]"},{"why":"Provides the Vicuna-1.5-7B LLM backbone that consumes the compact vision tokens.","marker":"[5]"},{"why":"An efficient-LLaVA token-reduction baseline at 1.9 TFLOPs that STC must outperform in Table 2.","marker":"[29]"},{"why":"A plug-and-play token-pruning baseline at 1.9 TFLOPs used for comparison in Table 2.","marker":"[4]"},{"why":"A one-vision-token efficient LMM baseline reproduced and compared at 1.9 TFLOPs.","marker":"[40]"}],"fun_headline_variants":["Vision tokens cut to 25% match LLaVA-1.5 on 8 benchmarks","Fusing 2x2 vision tokens keeps LLaVA-1.5 accuracy at quarter cost","Spatial token fusion slashes vision tokens, preserves multimodal performance","LLaVA-1.5 holds its own with only 25% of vision tokens via fusion","Learnable fusion shrinks vision tokens to 25% without a performance dip"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method assumes that the fine visual details lost when four neighboring image tokens are squeezed into one are already stored redundantly across the eight sampled layers of the vision encoder, whose weights are left unchanged; if a task needs a detail that lives in only one layer and the squeezing destroys it, the quarter-token model will fall behind the full-token one.","fun_headline_variants_meta":{"raw":{"variants":["Vision tokens cut to 25% match LLaVA-1.5 on 8 benchmarks","Fusing 2x2 vision tokens keeps LLaVA-1.5 accuracy at quarter cost","Spatial token fusion slashes vision tokens, preserves multimodal performance","LLaVA-1.5 holds its own with only 25% of vision tokens via fusion","Learnable fusion shrinks vision tokens to 25% without a performance dip"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00042,"raw_usage":{"total_tokens":2186,"prompt_tokens":994,"completion_tokens":1192,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":610,"completion_tokens_details":{"reasoning_tokens":1079}},"tokens_in":610,"tokens_out":1192,"duration_ms":9505,"temperature":1.0,"reasoning_tokens":1079,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T05:40:31.976766+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A concrete test: on a dense fine-grained benchmark—say TextVQA at higher resolution or a small-object counting set—compare the 25%-token STC model against the full-token LLaVA-1.5 baseline and against the MBTF-only model at full FLOPs. If STC drops substantially below the baseline while MBTF-only does not, the spatial-redundancy premise fails; if both drop together, the loss lies in multi-block feature fusion rather than in the 4x spatial reduction.","supporting_citations":[{"cited_title":"Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, march 2023","cited_arxiv_id":null,"evidence_quote":"Provides the Vicuna-1.5-7B LLM backbone that consumes the compact vision tokens."},{"cited_title":"Llava-prumerge: Adaptive token reduction for efficient large multimodal models","cited_arxiv_id":null,"evidence_quote":"An efficient-LLaVA token-reduction baseline at 1.9 TFLOPs that STC must outperform in Table 2."},{"cited_title":"An image is worth 1/2 tokens after layer 2: Plug-and-play inference acceleration for large vision-language models","cited_arxiv_id":null,"evidence_quote":"A plug-and-play token-pruning baseline at 1.9 TFLOPs used for comparison in Table 2."},{"cited_title":"Llava-mini: Efficient image and video large multimodal models with one vision token","cited_arxiv_id":null,"evidence_quote":"A one-vision-token efficient LMM baseline reproduced and compared at 1.9 TFLOPs."}],"review_version":1}