{"id":"197413a5-ef4a-45a2-b241-80773bd0750e","arxiv_id":"2411.14164","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"Focal Pruning (FoPru) retains as few as 25% of visual tokens, selected by vision-encoder attention, and keeps accuracy within about 1% on three LVLM benchmarks while cutting time-to-first-token by up to 2.5x.","lead":"FoPru speeds up vision-language models by deleting most visual tokens before they reach the language model, choosing which to keep from the image encoder's own attention maps. A smart generalist might care because it offers a training-free way to roughly double response speed while keeping accuracy stable on several benchmarks.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Table 1's accuracy is a per-dataset oracle over rank and row strategies while its speedup is rank-only; the reported accuracy is not achievable by one FoPru configuration.","rationale":"The reader's weakest assumption -- that CLIP's prompt-free attention is a reliable proxy for task-relevant visual tokens -- is real and is partly acknowledged by the authors' own results on TextVQA and Ocrbench. That concern is conceptual and limits generality, but it does not by itself invalidate the central quantitative claim at 25% retention. The more concrete and decisive issue is the reporting inconsistency in Table 1: the accuracy column is a per-dataset best over two strategies, while the speedup column is rank-only. This makes the main evidence table an unattainable upper envelope, not the result of a single FoPru setting. Because Table 4 already contains all per-strategy numbers, the issue is directly checkable and fixable; a corrected Table 1 would clarify whether the claimed accuracy-efficiency trade-off survives under one consistent strategy. The 25% headline appears to survive, so the verdict should stay CONDITIONAL rather than moving to reject, but the paper should be revised to report rank and row separately and to add uncertainty estimates for small benchmarks such as MMMU-val. This is a good-faith, evidence-focused concern about the presentation of results, not an accusation of misconduct.","tokens_in":14585,"tokens_out":14339,"duration_ms":131347,"concrete_test":"Recompute Table 1 twice using Table 4's published per-strategy numbers: once with rank-only and once with row-only for every model and retention ratio. For each version, check (i) whether the 'less than 1% drop on MMMU, SQA, POPE' statement still holds at 25%, and (ii) what the largest per-dataset drop is at 50% and 75%. If the mixed Table 1 entries exceed both single-strategy versions, Table 1 must be relabeled as an upper envelope rather than a method result. Additionally, bootstrap over the 150 MMMU-val questions (e.g., 1000 resamples) or run 5 evaluation seeds to determine whether the observed 1.00-point difference is within sampling noise.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 5.2/Table 1 claims FoPru maintains accuracy across retention ratios, but the table does not report a single pruning configuration. Cross-referencing Table 1 with Table 4 (which gives full rank and row results) shows that each accuracy cell at 75% (and some at 50%) is the per-dataset maximum of the rank and row strategies. For example, LLaVA-NeXT-8B at 75% takes Ai2D 70.69, SQA 79.91, TextVQA 64.14, and Ocrbench 53.20 from row pruning, while taking GQA 65.21, MMMU 39.78, and POPE 87.87 from rank pruning. The inference-efficiency columns, however, are explicitly stated to be rank-only ('we only present the results for the rank strategy here'). Thus Table 1 attaches rank-based speedups to an oracle accuracy obtained by combining two different pruning patterns. The abstract's headline 1%-margin claim for 25% retention is safe because those entries are all from the rank strategy, but the broader claim of 'maintaining high accuracy while pruning large numbers of tokens' at 50% and 75% is not supported by any single, reproducible configuration. The absence of error bars compounds this: MMMU-val has only about 150 questions, so a 1.00-point drop corresponds to roughly 1-2 items, which may be within sampling noise.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"FoPru proposes a training-free visual token pruning method for LVLMs. It computes per-token significance from the penultimate-layer attention maps of the frozen vision encoder, selects tokens either by global rank or by row-preserving local structure, and then reorders the survivors to retain spatial positions. The method is evaluated on LLaVA-NeXT-8B, LLaVA-1.6-7B, and LLaVA-1.6-13B across seven multimodal benchmarks at retention ratios of 25%, 50%, and 75%, with additional comparisons to FastV, PruMerge, and PruMerge+ on LLaVA-1.5-7B. The headline claims are that 25% visual token retention stays within a 1% accuracy margin on MMMU, SQA, and POPE while yielding up to 2.52x TTFT speedup, and that FoPru generally maintains accuracy while pruning at 50% and 75% retention.","tokens_in":14895,"tokens_out":4021,"duration_ms":36376,"significance":"If the claims hold, FoPru is a simple, model-agnostic, training-free optimization that could be plugged into existing LVLMs without retraining or architecture changes. The paper's strengths are its breadth of models and datasets, its use of the LMMS-Eval platform for consistent evaluation, direct comparison with FastV and PruMerge/PruMerge+, and an honest limitations section that admits retention ratios and strategies are task-dependent. The method is not fitted to evaluation data, so circularity is not a concern. However, the empirical support is weakened by the oracle-style reporting in the main table, the absence of any repeated-run statistics, and the ad hoc variance rule in Eq. (4). These issues affect the central claims about accuracy retention and inference speedup, so they need to be resolved before the paper can be recommended for publication.","major_comments":[{"comment":"Table 1 reports accuracy as the per-dataset best of the rank and row strategies (cross-referencing with Table 4), while the inference-efficiency columns are explicitly presented for the rank strategy only. For example, LLaVA-NeXT-8B at 75% retention takes Ai2D 70.69, SQA 79.91, TextVQA 64.14, and Ocrbench 53.20 from row pruning, but GQA 65.21, MMMU 39.78, and POPE 87.87 from rank pruning. Consequently no single FoPru configuration attains both the reported accuracy and the reported speedup at 50% and 75% retention, so the claim that FoPru maintains high accuracy at those ratios is not supported as stated. Please report a single consistent strategy with its own speedups, or clearly label the oracle selection and provide efficiency numbers for the exact configuration that produced each accuracy column.","section":"§5.2, Table 1"},{"comment":"No error bars, confidence intervals, or repeated-run statistics are reported for any accuracy number, and several benchmarks are small (MMMU-val is roughly 150 questions). Under that noise scale, a 1.0-point drop corresponds to about one or two items, which makes the abstract's 'within a 1% margin' claim statistically fragile; moreover, for LLaVA-NeXT-8B at 25% retention the MMMU drop is exactly 1.00 point (40.22 to 39.22), which contradicts the bullet's 'less than 1%' wording. Please add repeated-run statistics or at least an explicit sample-size caveat, and correct the wording.","section":"§5.2, Table 1; §5.4, Table 3"},{"comment":"The variance rule for choosing between s1 and s2 is presented without a principled justification, and the ablation labeled 'w/o Variance' tests selecting the low-variance direction, not whether the variance heuristic itself adds value. It is therefore unknown whether Eq. (4) improves over always using s1 or always using s2. Please compare the fixed choices and the variance rule across models, datasets, and retention ratios, or provide a principled basis for the heuristic.","section":"§4.2, Eq. (4); §5.4, Table 3"}],"minor_comments":[{"comment":"The limitation paragraph already concedes that optimal ratios vary by task and model; consider moving this observation into the main-results discussion so the dataset-specific behavior in Figure 5 is framed as a design constraint rather than an aside.","section":"§6, Conclusion"},{"comment":"There are typos in the figure axis ('Relative Accrucy') and in the Table 1 header ('Inference Efficience'); in §5.2 the phrase 'accuracy accuracy after pruning' should be corrected.","section":"Figure 6; Table 1"},{"comment":"The related-work discussion of 'mode collapse' cites [11] and uses it as motivation, but the connection to FoPru's design is descriptive; consider stating explicitly where the concept is used (Eq. (4)) so the reader can assess the strength of the link.","section":"§2.2"}],"recommendation":"major_revision","confidential_remarks":"The paper is a reasonable empirical contribution for a computer vision venue, and the central 25%-retention claim appears to be supported by the rank-strategy numbers. The main concern is the presentation of Table 1 as a single configuration when it is actually a per-dataset oracle; this is fixable by restructuring the table or reporting rank-only results consistently. The variance heuristic in Eq. (4) also needs a more informative ablation. I do not see grounds for rejection, provided the authors address the reproducibility and statistical-significance issues."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: FoPru is a solid, incremental systems paper, and it deserves a serious referee. The stress-test note holds up. Table 1 reports accuracy as the per-dataset maximum of the rank and row strategies from Table 4, while the speedup columns are explicitly rank-only. That means the table attaches rank-based TTFT/TPOP numbers to an oracle accuracy that no single FoPru configuration actually produces. The 25% “within 1%” claim on MMMU, SQA, and POPE survives because those cells come from rank, but the broader “maintains accuracy at 50/75%” claim is not supported by a single reproducible setup.\n\nWhat is genuinely new: pruning before the projector using the CLIP encoder’s internal attention map is not in FastV, PruMerge, or the training-based compressors cited here. The two strategies—global rank and local row-based—plus token reordering form a clean, training-free framework. The evaluation is broad: three LVLMs, seven datasets, comparisons against FastV and PruMerge on LLaVA-1.5, and an ablation of the variance rule, significance stage, and reordering. No part of the method is fitted to the evaluation data; the retention ratio is swept. That is real, honest empirical work, and the speedups are plausible.\n\nSoft spots beyond the Table 1 issue: no error bars anywhere, and MMMU-val is small enough that a 1-point drop could be a few questions. The variance heuristic in Eq. 4 is ad hoc—the ablation shows it helps, but there is no analysis of when it is the right direction to pick. The static encoder attention, computed without the text prompt, degrades on TextVQA and OCRbench; the authors acknowledge this in the limitations but should state it more directly in the main results. Also, no code is shipped, though the method is simple enough to reimplement.\n\nBottom line: the central mechanism is sound and the paper is readable. It needs a revision that reports both strategies consistently per ratio, or pre-registers one, and adds error bars or confidence intervals. With those fixes, it is a useful contribution to the LVLM efficiency literature. I would accept it for peer review as a conditional.","headline":"A simple, training-free token-pruning method with real experimental breadth, but Table 1 mixes per-dataset best-of-two strategies with rank-only speedups, so the headline accuracy claim is not tied to one reproducible configuration.","tokens_in":15386,"tokens_out":1868,"would_cite":true,"duration_ms":20172,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that large vision-language models contain many redundant visual tokens, and that the vision encoder's attention map alone can identify which tokens are worth keeping before they enter the language model.","keywords":["visual token pruning","large vision-language models","attention map","training-free inference acceleration","token significance","mode collapse","CLIP encoder","token reordering"],"falsifier":"A direct test: construct a dataset of images in which the answer depends on a small object or text string located in a region that the CLIP encoder's attention scores rank low, and measure whether FoPru at 25% retention drops sharply relative to keeping all tokens. Alternatively, compare FoPru's selected tokens against the tokens the LLM's own cross-attention would select for the given prompt; if the two sets disagree on a large fraction of queries, the text-free significance assumption is the weak link.","tokens_in":14415,"feed_emoji":"✂️","tokens_out":5727,"duration_ms":47243,"temperature":0.7,"pith_summary":"FoPru is a training-free method that prunes visual tokens in large vision-language models before they reach the language model. It computes a significance score for each visual token from the vision encoder's attention distribution, then keeps the top-scoring tokens either globally (rank strategy) or row-by-row (row strategy), and reorders them to preserve spatial layout. With only 25% of the visual tokens, the method keeps accuracy within one point on MMMU, SQA, and POPE, while cutting time-to-first-token by up to 2.52x on LLaVA-1.6-13B. The paper motivates this by showing that deep encoder layers concentrate over 80% of attention on less than 25% of tokens, a mode collapse that makes a small subset of tokens carry the critical visual information. The reason to care is that this offers a retraining-free inference speedup that can be applied to existing LVLMs.","feed_headline":"Prune 75% of visual tokens with under 1% accuracy loss","feed_subtitle":"FoPru uses the vision encoder's own attention map to keep only the tokens that matter, cutting time-to-first-token by up to 2.52x.","key_machinery":"The central object is the attention-based token significance score. Starting from the multi-head attention map of the encoder's penultimate layer, FoPru averages attention over heads, then computes the average attention per token along both the row and column directions; it picks the direction with the larger variance as the significance vector, reasoning that a more dispersed distribution makes important tokens stand out. This score is used in two pruning strategies: rank pruning, which keeps the top $N \\times r\\%$ tokens globally, and row pruning, which reshapes the score into a grid and keeps the highest-scoring rows. Finally, token reordering sorts the kept indices back into ascending order so the pruned sequence preserves the original spatial arrangement. The mechanism works because, as the paper shows, deep encoder layers exhibit an imbalanced attention distribution (mode collapse), with most attention concentrated on a small subset of tokens that carry the critical visual information.","core_discovery":"FoPru establishes that visual token pruning for LVLMs can be guided entirely by the attention distribution inside the CLIP vision encoder, without any training or access to the LLM. The token significance score is computed by averaging multi-head attention from the penultimate encoder layer, then taking the row or column marginal that has the larger variance; this score, computed without seeing the question text, is used to select the most important tokens. The rank strategy keeps the globally top-scoring tokens, while the row strategy keeps entire image rows with the highest summed significance, preserving horizontal continuity for text-heavy content. After selection, tokens are reordered by their original indices to keep relative spatial positions. The paper's experiments on three LLaVA-1.6 variants and seven benchmarks show that at 25% retention the method stays within one accuracy point of the full-token baseline on MMMU, SQA, and POPE, and that on GQA, MMMU, SQA, and POPE pruning can even slightly exceed the unfiltered baseline, suggesting that redundant tokens can actively distract the model.","pith_inferences":["The text-free significance score could be made more robust by conditioning on the prompt: for example, using the LLM's first-layer cross-attention with the question text as a second scoring signal, then fusing the two ranks. The paper does not explore this, and the degradation on TextVQA/OCRbench suggests where the gain would show up.","The row strategy's advantage on text-heavy images at high retention suggests a hybrid scheme: use row pruning when the image has dense horizontal content (e.g., documents or scenes with text) and rank pruning for natural images with a clear focal object; the paper reports the two separately but does not combine them adaptively.","If the mode-collapse observation holds across other vision encoders (e.g., SigLIP, DINOv2), FoPru's scoring rule should transfer directly, but the optimal significance formula might need re-tuning since attention distributions differ; this is a testable extension beyond the LLaVA/CLIP family reported.","The near-60% accuracy at 0.2% retention on SQA and Ai2D hints that those benchmarks contain images where the answer is localized in a single small region; measuring the correlation between FoPru's selected patch and human-annotated answer regions would turn this hint into a diagnostic."],"forward_implications":["At 25% visual-token retention, FoPru cuts time-to-first-token by up to 2.52x (LLaVA-1.6-13B) and time-per-output-token by up to 1.24x, with GPU memory also reduced, enabling faster deployment of LVLMs without retraining.","Because FoPru prunes before the projector, it composes with LLM-level pruning like FastV; the paper's FoPru+FastV combination at 25% retention beats FastV alone on six datasets.","On GQA, MMMU, SQA, and POPE, pruning can slightly improve accuracy over the full-token baseline, implying that redundant visual tokens can add noise that interferes with the model's judgments.","The optimal retention ratio is task-dependent: text-heavy datasets like TextVQA and OCRbench need more tokens, while SQA and Ai2D remain near 60% accuracy even at 0.2% retention (about five tokens), so a practical deployment would tune the ratio per dataset or per task family."],"supporting_citations":[{"why":"Defines FastV, the layer-level training-free pruning baseline that FoPru is compared against and combined with.","marker":"[7]"},{"why":"Documents the imbalanced attention / over-smoothing phenomenon in transformers that motivates deriving significance from encoder attention.","marker":"[11]"},{"why":"Introduces PruMerge and PruMerge+, adaptive training-free token reduction baselines that FoPru outperforms.","marker":"[24]"},{"why":"Provides the unified evaluation platform used for all accuracy and efficiency measurements.","marker":"[34]"},{"why":"Defines LLaVA-NeXT, one of the three LVLM backbones on which FoPru is evaluated.","marker":"[14]"}],"fun_headline_variants":["FoPru prunes 75% of visual tokens with <1% accuracy loss","Training-free token pruning speeds LVLMs 2.52x","Keep only the visual tokens that matter: FoPru","Cut LVLM inference cost with attention-based pruning","Prune visual tokens via vision encoder attention, no training"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The significance score comes from the vision encoder alone, without seeing the question or instruction text, so the method assumes that the tokens that matter for the final answer are the same ones the encoder's attention highlights on the image itself.","fun_headline_variants_meta":{"raw":{"variants":["FoPru prunes 75% of visual tokens with <1% accuracy loss","Training-free token pruning speeds LVLMs 2.52x","Keep only the visual tokens that matter: FoPru","Cut LVLM inference cost with attention-based pruning","Prune visual tokens via vision encoder attention, no training"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000677,"raw_usage":{"total_tokens":3105,"prompt_tokens":998,"completion_tokens":2107,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":614,"completion_tokens_details":{"reasoning_tokens":2020}},"tokens_in":614,"tokens_out":2107,"duration_ms":13966,"temperature":1.0,"reasoning_tokens":2020,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T15:27:06.814018+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A direct test: construct a dataset of images in which the answer depends on a small object or text string located in a region that the CLIP encoder's attention scores rank low, and measure whether FoPru at 25% retention drops sharply relative to keeping all tokens. Alternatively, compare FoPru's selected tokens against the tokens the LLM's own cross-attention would select for the given prompt; if the two sets disagree on a large fraction of queries, the text-free significance assumption is the weak link.","supporting_citations":[{"cited_title":"Llava-next: Stronger llms supercharge multimodal capa- bilities in the wild, 2024","cited_arxiv_id":null,"evidence_quote":"Defines LLaVA-NeXT, one of the three LVLM backbones on which FoPru is evaluated."}],"review_version":1}