{"id":"4543a0ad-8209-4c86-99a2-7cfa4b9bdd02","arxiv_id":"2412.08771","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":3.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"DFMR uses the mean within-patch standard deviation of CLIP features to pick a pooling factor of 1, 2, or 3 per image, and training LLaVA-1.5 with this dynamic compression improves average benchmark scores over a random-compression baseline.","lead":"This paper introduces Dynamic Feature Map Reduction (DFMR), a module that chooses how aggressively to average-pool visual tokens in LLaVA based on the variance of the image features, and reports that training LLaVA with this adaptive compression improves average scores on eight multimodal benchmarks. The straightforward idea matters because it offers a cheap way to free token budget for multi-image and video prompts in resource-limited settings.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Table 1 never evaluates DFMR's adaptive rule: each row fixes s for all images, so the reported gains may reflect an unreported training-time skew toward low compression rather than the sigma metric.","rationale":"The reader's weakest assumption identifies essentially the same confound: DFMR's training distribution over s is unreported, so Random and DFMR may operate at different average token budgets. My reading sharpens this into a more fundamental measurement gap: the experimental evaluation never exercises the adaptive mechanism. Because Section 4.1 fixes s for all images in each evaluation row, Table 1 cannot support the paper's central claim that DFMR dynamically compresses based on intrinsic image information. The proposed matched-histogram control would settle both concerns: it equates the training token-budget distributions and evaluates both models under identical adaptive per-image choices. I therefore keep the reader's conditional verdict; the paper needs this reanalysis before the adaptive-claim can be accepted.","tokens_in":7755,"tokens_out":5047,"duration_ms":47919,"concrete_test":"Report the empirical histogram of DFMR-selected s on the training set and on each evaluation benchmark at tau=5e-2. Then train a matched-control baseline, LLaVA-1.5-Random-Matched, that samples s per training image from that same histogram instead of uniformly, and evaluate it under the same adaptive per-image s choices that DFMR would make for each test image. If DFMR's average advantage over Random-Matched shrinks to noise or reverses, the Table 1 gains are a token-budget distribution artifact rather than evidence for the sigma-based selection rule.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing problem is in the evaluation protocol, Section 4.1: \"For each model, we evaluate its performance on images with compression ratios of 1, 2, and 3.\" This means every image in a row receives the same fixed s, so Table 1 does not test the claimed dynamic, per-image selection of s by DFMR at all. It only tests robustness to fixed token budgets after training with a stochastic compression policy. The sole training difference between LLaVA-1.5-Random and LLaVA-1.5-DFMR is the distribution of s per image: Random samples s uniformly from {1,2,3}, while DFMR chooses s via the threshold rule in Section 3.2 with tau=5e-2. The paper never reports how often DFMR selects each s, so the training token-budget distributions are unmatched and unknown. If DFMR selects s=1 for most training images, it is trained mostly at 576 tokens; that would explain why Table 1 shows DFMR ahead on all eight benchmarks at s=1 and s=2 but behind Random on five of eight benchmarks at s=3. The reported \"average\" equally weights the three fixed-s rows, so it can be inflated purely by a training-distribution artifact, independent of whether the sigma metric identifies informative images. No significance testing or error bars are reported, so \"significantly improves\" is not statistically established either.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Dynamic Feature Map Reduction (DFMR), a module inserted between the vision encoder and projector of LLaVA-1.5. DFMR computes the mean within-patch standard deviation of CLIP visual features and uses a manually chosen threshold tau=5e-2 to select a compression factor s in {1,2,3}, then applies s x s average pooling. The authors train three models: the standard LLaVA-1.5, LLaVA-1.5 Random in which s is sampled uniformly per image, and LLaVA-1.5 DFMR in which s is selected by the proposed rule. They evaluate all models at fixed token counts of 576, 144, and 64 on eight benchmarks and report that DFMR improves over the Random baseline on average and across most rows. The central claim is that using intrinsic image information to choose per-image compression improves performance under varied visual token budgets.","tokens_in":8070,"tokens_out":3171,"duration_ms":33534,"significance":"The idea of using a cheap, intrinsic image statistic to guide visual token compression is appealing and potentially useful for resource-constrained multimodal models. The comparison against a stochastic random-compression baseline is a reasonable experimental design and gives the paper some independent grounding. The COCO distribution analysis in Section 4.4 also shows that the proposed standard-deviation metric separates images in a plausible way. However, the main empirical claim is not supported by the evaluation as reported: Table 1 fixes s for all images at evaluation time and never exercises the dynamic per-image selection rule, the training-time distribution of s for DFMR is not reported, and no variance or significance information is given. The threshold tau is hand-picked without ablation. As a result, the reported gains could plausibly stem from an uncontrolled difference in average token budgets during training rather than from the information-content metric. With a corrected evaluation protocol, the method could still be of interest, but the current evidence is insufficient.","major_comments":[{"comment":"The evaluation protocol fixes the compression factor for every image in a given row: “we evaluate its performance on images with compression ratios of 1, 2, and 3.” This means Table 1 never tests DFMR’s claimed dynamic, per-image selection of s; it tests only how well each model performs when forced to use a fixed token budget. The differences between LLaVA-1.5 Random and LLaVA-1.5 DFMR in each row can be fully explained by the different distributions of s seen during training, independent of whether the standard-deviation metric identifies informative images. The paper must report how often DFMR selects each s during training and, ideally, evaluate DFMR’s actual adaptive rule on the test images, rather than only fixed-s rows.","section":"Section 4.1 (Evaluation Details) and Table 1"},{"comment":"The comparison between LLaVA-1.5 Random and LLaVA-1.5 DFMR is confounded by an unreported and likely unequal distribution of training token budgets. Random samples s uniformly from {1,2,3}, so its average training budget is 576*(1/3)+144*(1/3)+64*(1/3) = 261.3 tokens. If DFMR’s threshold rule selects s=1 for a large fraction of images, DFMR is trained predominantly at 576 tokens, which would explain why DFMR is ahead at evaluation token counts of 576 and 144 but often behind Random at 64 tokens. The paper must report the empirical distribution of s selected by DFMR over the training data and show the average training token budget; otherwise the average row of Table 1 does not support the claim that DFMR “enhances model performance across all tasks.”","section":"Section 4.1 (Training Details) and Section 4.3 (Performance)"},{"comment":"The abstract states that DFMR “significantly improves” performance, and Section 4.3 repeats “significantly better results,” but no error bars, standard deviations across seeds, or statistical significance tests are reported anywhere. Each benchmark is a single number per configuration, so the reader cannot assess whether the observed differences of a few points are meaningful. The authors should provide at least three training runs or per-benchmark bootstrap confidence intervals, and should temper the word “significantly” unless such evidence is supplied.","section":"Section 4.3 (Performance) and Abstract"},{"comment":"The threshold tau=5e-2 is a manually defined hyperparameter, and the compression factor set {1,2,3} is fixed in advance. The paper gives no ablation or sensitivity analysis for tau, and no principled argument that 5e-2 is the right value. Because tau controls the distribution of s during training, the reported gains could partly reflect a threshold that was tuned with the evaluation benchmarks in mind. The authors should report an ablation over tau, or justify the choice independently of the test benchmarks, and should show that the qualitative conclusions are stable across reasonable threshold values.","section":"Section 3.2 (DFMR) and Section 4.1 (Implementation Details)"},{"comment":"The statement “Our proposed method operates with a computational cost of O(1)” is incorrect for the DFMR module itself: Equations (1)–(3) require computing patch standard deviations over all spatial positions and embedding dimensions, which is O(N_v D_v) in the number of visual tokens and embedding dimension. The actual cost may be small relative to the LLM, but it is not O(1). This should be corrected to an accurate complexity statement, and the claimed absence of dataloader burden should be supported by the relevant measurement if it is meant to be a practical efficiency result.","section":"Section 5 (Discussion)"}],"minor_comments":[{"comment":"The method is called Dynamic Feature Map Reduction in the abstract and Section 3.2, but Section 4.4 refers to “Dynamic Feature Mixing Ratio (DFMR)”; the acronym expansion should be made consistent throughout.","section":"Section 4.4 (Analysis)"},{"comment":"The caption labels the panels as “Bottom 10 Samples” and “Top 10 Samples” with the top being high-standard-deviation images, but the body text says Figure 2(b) shows images “composed of background content or repetitive patterns”, which would correspond to the bottom 10. The panel-to-description mapping is unclear and should be clarified.","section":"Figure 2 caption and Section 4.4"},{"comment":"The paper title is “LLaVA-Zip” but the proposed module is DFMR and “LLaVA-Zip” is not defined or used in the text; either the method name should be aligned with the title or the title should be changed to avoid confusion.","section":"Title and Abstract"},{"comment":"The threshold is described as “fixed threshold of 5e-2”, but Section 3.2 and Figure 2 also discuss thresholds 7e-2 and 9e-2; the relationship between these values and the experiments in Table 1 should be stated explicitly.","section":"Section 4.1 (Implementation Details)"},{"comment":"Some references appear incomplete or informally formatted, such as [14], where author names are run together; please format all entries according to the journal style.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The main concern is the gap between the claimed dynamic mechanism and the fixed-s evaluation protocol in Table 1. This is fixable in principle by reporting the DFMR s-distribution, evaluating the adaptive rule at test time, adding error bars or significance tests, and ablating the threshold. If the authors cannot show that DFMR’s advantages survive a token-budget-matched comparison, the central claim would collapse. The paper is otherwise within the journal’s scope and the underlying idea is simple enough to be worth a revised look."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the module itself is simple and reasonable, but the evaluation never actually tests the adaptive behavior. The paper's Table 1 fixes the compression factor s per row for every image, so DFMR's per-image selection is never exercised at test time. What the table shows is robustness to fixed token budgets after training with a stochastic compression policy. That is a real but different claim.\n\nThe good parts: the idea of using intrinsic image statistics (mean within-patch standard deviation of visual features) to decide pooling strength is sensible, cheap, and O(1). The observation that training the model with variable token counts (Random) makes it much more robust to aggressive compression than the vanilla LLaVA-1.5, which is only trained at 576 tokens, is a useful empirical point, and it seems to hold across all eight benchmarks. Figures 2(a) and the top/bottom image examples give some qualitative support for the metric.\n\nThe load-bearing problem is the central comparison. DFMR and Random differ by the distribution of s during training: Random samples s uniformly from {1,2,3}; DFMR picks s via a threshold rule with tau=5e-2. The paper never reports how often DFMR selects each value. If, as the numbers suggest, DFMR mostly selects s=1 for natural images, then it is effectively trained near 576 tokens most of the time. That would explain why DFMR is competitive at 576 tokens, mildly better at 144, and actually worse than Random on five of eight benchmarks at 64 tokens. The average across the three fixed-s rows then says little about the sigma metric, and everything about the training token-budget distribution. There are also no error bars or significance tests, so \"significantly improves\" is not statistically supported, and the threshold tau is hand-picked with no ablation.\n\nThe paper would be worth a rework rather than a full reject: rerun evaluation with the adaptive rule actually applied per image, report the empirical distribution of s, match token budgets across models, and compare against existing token-compression methods like HiRED or Matryoshka. As it stands, I wouldn't send this to peer review; the main claim is confounded and the current experiments cannot support it.","headline":"The DFMR module is a cheap, sensible idea, but Table 1 never evaluates the adaptive rule it proposes, so the headline result is an artifact of the training distribution, not a validated improvement.","tokens_in":8523,"tokens_out":2695,"would_cite":false,"duration_ms":27618,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A standard-deviation statistic computed from each image's visual features decides how many tokens LLaVA retains, and the paper reports that this beats random compression on eight benchmarks.","keywords":["visual token compression","multimodal large language models","LLaVA","dynamic feature map reduction","average pooling","standard deviation metric","adaptive token budgets","efficient vision-language models"],"falsifier":"Count how often DFMR selects each compression factor on a held-out image set under $\\tau = 5\\times10^{-2}$, compute the average number of visual tokens DFMR and the random baseline actually use, and re-run the eight benchmarks with DFMR's per-image choices matched by token count to the random baseline. If the benchmark averages converge once token budgets are equalized, the content metric is not the cause of the improvement.","tokens_in":7568,"feed_emoji":"🗜️","tokens_out":5203,"duration_ms":50918,"temperature":0.7,"pith_summary":"This paper proposes a plug-in module, Dynamic Feature Map Reduction (DFMR), for the LLaVA-1.5 multimodal model. DFMR decides how aggressively to average-pool visual tokens by measuring the variability of each image's visual feature map: images with high patch-to-patch variance keep more tokens, while flat, repetitive images are compressed more. The paper reports that training LLaVA-1.5 with this adaptive scheme improves average scores across eight visual question-answering and multimodal benchmarks compared with both the uncompressed model and a baseline that compresses tokens by a fixed random factor. The motivation is practical: if visual tokens can be cut without losing accuracy, the same model can handle multiple images or video within a fixed token limit and with less memory.","feed_headline":"Adaptive token compression beats random token-cutting in LLaVA","feed_subtitle":"Per-image compression guided by visual variance lifts scores across eight multimodal benchmarks at 64, 144, and 576 tokens.","key_machinery":"The load-bearing object is the mean patch standard deviation $\\sigma$ of the visual feature map. For each window of the feature map, DFMR computes the standard deviation of the token values, averages these over all windows, and compares the average to a manually set threshold $\\tau = 5\\times10^{-2}$. A low $\\sigma$ signals repetitive or uniform content and licenses a larger pooling factor; a high $\\sigma$ signals fine detail and forces a smaller pooling factor. The threshold is the single control knob that trades token savings against fidelity, and the paper shows its effect on the distribution of chosen compression factors.","core_discovery":"The central claim is that intrinsic image statistics alone can guide how many visual tokens a multimodal language model needs. DFMR computes, for each candidate compression factor $s$, the mean of the per-patch standard deviations over the visual feature map, and increases $s$ only while this value stays below a fixed threshold. The resulting per-image choice of 576, 144, or 64 tokens is used during both pretraining and fine-tuning. On the eight benchmarks reported, LLaVA-1.5 with DFMR outperforms the random-compression baseline at each evaluated token length and on average, with the largest gains appearing at heavier compression. The paper therefore claims that compression guided by image content is better than compression guided by chance.","pith_inferences":["If DFMR's gains come from content-aware selection rather than extra tokens, then a matched-budget comparison would isolate the contribution of the variance metric; the paper does not report how often each compression factor is chosen.","The same standard-deviation heuristic could be ported to other vision encoders and pooling schemes, with the threshold re-calibrated per encoder.","A testable extension is to use the metric to choose compression per region or per frame in video, rather than per whole image.","Images with low $\\sigma$ are likely compressible before tokenization, so the metric could also guide storage or transmission savings, not just in-model token counts."],"forward_implications":["A single LLaVA-style model can process one image, several images, or video frames without exceeding the LLM's token limit, by assigning each image its own compression level.","Training with adaptive compression acts as a form of data augmentation: the same image contributes tokens of several lengths, which the paper argues can ease the scarcity of open-domain image-text pairs in continued pretraining.","The threshold $\\tau$ gives practitioners a tunable knob to shift the operating point between accuracy and memory, with lower thresholds favoring more compression.","Because the decision rule is a simple statistic computed before the projector, DFMR adds only constant-time overhead during data loading and training."],"supporting_citations":[{"why":"Defines the LLaVA-1.5 base model, training recipe, and instruction data that DFMR is inserted into and compared against.","marker":"[2]"},{"why":"Provides the CLIP vision encoder whose output feature map DFMR measures and pools.","marker":"[12]"},{"why":"Provides the evaluation harness used to score all models on the eight benchmarks.","marker":"[14]"},{"why":"GQA benchmark, one of the eight tasks used to compare DFMR with baselines.","marker":"[15]"},{"why":"TextVQA benchmark, the OCR-focused task where DFMR shows the largest relative gains.","marker":"[16]"},{"why":"VQAv2 benchmark, one of the general VQA tasks in the comparison.","marker":"[17]"},{"why":"POPE benchmark, used to measure hallucination-related errors across token lengths.","marker":"[18]"},{"why":"MME benchmark, a comprehensive MLLM evaluation used in the average comparison.","marker":"[20]"},{"why":"SEED-Bench, another comprehensive benchmark in the eight-task comparison.","marker":"[21]"}],"fun_headline_variants":["LLaVA-Zip: image variance picks token count per image","DFMR: adaptive token reduction for LLaVA on modest hardware","Intrinsic image statistics decide LLaVA's token budget","Compress LLaVA tokens by image content, not random","Per-image token compression: variance beats chance in LLaVA"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The comparison treats DFMR and the random baseline as if they use the same average number of tokens, but the paper never reports how often DFMR picks each compression level; if DFMR mostly keeps 576 tokens, its higher benchmark scores may come from more tokens rather than from the content metric.","fun_headline_variants_meta":{"raw":{"variants":["LLaVA-Zip: image variance picks token count per image","DFMR: adaptive token reduction for LLaVA on modest hardware","Intrinsic image statistics decide LLaVA's token budget","Compress LLaVA tokens by image content, not random","Per-image token compression: variance beats chance in LLaVA"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000613,"raw_usage":{"total_tokens":2822,"prompt_tokens":891,"completion_tokens":1931,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":507,"completion_tokens_details":{"reasoning_tokens":1844}},"tokens_in":507,"tokens_out":1931,"duration_ms":14469,"temperature":1.0,"reasoning_tokens":1844,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T17:35:02.925569+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Count how often DFMR selects each compression factor on a held-out image set under $\\tau = 5\\times10^{-2}$, compute the average number of visual tokens DFMR and the random baseline actually use, and re-run the eight benchmarks with DFMR's per-image choices matched by token count to the random baseline. If the benchmark averages converge once token budgets are equalized, the content metric is not the cause of the improvement.","supporting_citations":[{"cited_title":"Lmms-eval: Accelerating the development of large multimoal models, March 2024","cited_arxiv_id":null,"evidence_quote":"Provides the evaluation harness used to score all models on the eight benchmarks."},{"cited_title":"Gqa: A new dataset for real-world visual reasoning and com- positional question answering","cited_arxiv_id":null,"evidence_quote":"GQA benchmark, one of the eight tasks used to compare DFMR with baselines."}],"review_version":1}