{"id":"1c833dec-5c0c-4d99-bc8d-6a9d1408b9e6","arxiv_id":"2507.20454","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"SparseVAR is a training-free acceleration method for next-scale prediction models that excludes low-frequency tokens in high-resolution stages, yielding up to 2x speedup on Infinity-2B with a GenEval drop of 0.0051.","lead":"SparseVAR speeds up high-resolution image generation by skipping low-frequency tokens during later generation stages, while keeping a few anchor tokens to fill in skipped regions. It achieves up to 2x faster inference on Infinity-2B with minimal quality loss and needs no retraining.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Missing same-budget random-mask control leaves the central frequency-awareness premise untested; Eq. (2)'s max-normalized threshold also shows fragility (HART tau=0.7 collapse).","rationale":"The paper has real independent support: Tables 1-3 report a consistent speed/quality tradeoff on two models, the MSE metric is parameter-free and cheap, and appendix Fig. 10 at least compares against several token-reduction baselines. However, the paper's own Section 5 and Appendix E.2 contain an unresolved discrepancy about which block index is used (the main text says the 16th block for MSE-based frequency estimation after an experiment claimed to be on Infinity; the caption of Fig. 8 says HART; Appendix E.2 says block 3 for Infinity-2B and block 16 for HART-0.7B). This matters because Fig. 8 shows GenEval score varies roughly from 0.30 to 0.50 with block index, so the reported tables cannot be reproduced from the text alone. The missing random-mask control is more fundamental: the method's contribution is the frequency ordering of Eq. (2), and no experiment isolates that ordering from the pruning budget and the anchor-copy repair. The HART tau=0.7 collapse in Table 3 shows the same threshold can be far outside a safe operating range on another model, so reporting only the mean Infinity score at the chosen tau understates the risk. My recommended check (random same-budget masks, plus excluded-fraction distributions) would directly settle whether the frequency ordering is doing the work. Because the speedup itself is plausible and the evidence is mostly consistent, I do not move the reader's conditional verdict; I would only make the stated conditions explicit.","tokens_in":14653,"tokens_out":11592,"duration_ms":125292,"concrete_test":"Re-run the Table 1 Infinity-2B GenEval configuration (tau=0.7, alpha=4, beta=0.9, P=10) with Eq. (2) replaced by a per-image random mask of exactly the same size as M_low_k, keeping the anchor-copying step unchanged. If the random-mask score stays within about 0.005 of 0.7209, the frequency ordering is not load-bearing; if it drops below 0.7209 by more than the reported 0.0051 drop, the MSE proxy matters. In the same runs, report the distribution of excluded-token fractions across the GenEval prompts; if the fraction varies by more than about 20 percentage points at fixed tau=0.7, the max-normalized threshold is unstable and mean latency/quality numbers are not representative.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Table 1's headline result (Infinity-2B, tau=0.7: 2.78s to 1.36s, GenEval 0.7260 to 0.7209) rests on Eq. (2) marking exactly the tokens that are safe to drop. The paper never compares SparseVAR against a same-budget control that preserves the sparsity level but randomizes which tokens are removed; Fig. 6 compares MSE vs logits vs L1 and Fig. 10 compares against ToMe/PiToMe/ZipVL/SparseViT, but no experiment holds the number of excluded tokens fixed and varies only the identity of the excluded tokens. The observed tradeoff could therefore be produced by the anchor-copying repair (Sec. 4.3, alpha=4, beta=0.9) plus any sparsity pattern, and the claimed low-frequency specificity would not be load-bearing. The fragility of the max-normalized threshold in Eq. (2) is visible in Table 3: HART at tau=0.7 collapses (ImageReward 0.866 to 0.433, HPSv2 29.53 to 25.71), so the same tau does not define a safe operating point across models. Only mean scores are reported for Infinity; the per-prompt distribution of |M_low_k| and of quality change is absent. Without a random-mask ablation and per-prompt mask statistics, the Infinity 2x result is a favorable operating point rather than evidence that low-frequency tokens are intrinsically skippable.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes SparseVAR, a training-free inference acceleration framework for next-scale prediction visual autoregressive models. The method uses the per-token MSE change of features in a selected transformer block to identify low-frequency tokens, excludes those tokens from computation in high-resolution stages, and retains a small set of uniformly sampled anchor tokens whose logits are copied to excluded tokens when cosine similarity is sufficiently high. Experiments on Infinity-2B and HART-0.7B report roughly 2x speedup on Infinity with a GenEval overall score drop from 0.7260 to 0.7209 at tau=0.7, as well as smaller speedups on HART and FlexVAR. The paper also compares against token-reduction baselines and ablates the metric choice, anchor grid size, and starting stage.","tokens_in":14998,"tokens_out":5657,"duration_ms":59607,"significance":"If the frequency-aware exclusion claim is validated, SparseVAR is a practically useful plug-and-play accelerator for high-resolution autoregressive image generation: it requires no retraining, ships with code, and reports comparisons against ToMe, PiToMe, ZipVL, and SparseViT. The additional FlexVAR experiment is a thoughtful check that the redundancy is not solely an artifact of the residual VAE used by HART and Infinity. However, the manuscript's central conceptual claim, that low-frequency tokens are intrinsically skippable, is not yet directly supported by a same-budget randomized control, and a specification inconsistency in the block-selection setup must be fixed before the results are reproducible.","major_comments":[{"comment":"The experiments do not include a same-budget control that randomizes which tokens are excluded while holding the exclusion count, anchor placement, and anchor-copy repair mechanism fixed. Figure 6 compares MSE, logits, and L1 as scoring metrics, and Figure 10 compares against ToMe, PiToMe, ZipVL, and SparseViT, but no experiment preserves the sparsity ratio and anchor strategy while varying only the identity of the dropped tokens. Without such a control, the reported quality-latency tradeoff could in principle be produced by the anchor-copying mechanism in Section 4.3 with any sparsity pattern, leaving the paper's central frequency-awareness premise untested. I ask the authors to add a random-mask ablation at matched latency/sparsity operating points.","section":"Section 5, Figures 6 and 10"},{"comment":"The selected block index s, which controls the MSE map in Eq. (1), is specified inconsistently. The main text states that experiments were conducted on Infinity with tau=0.6 and that the 16th block achieved the best results, but the caption of Figure 8 says the evaluation uses HART-0.7B, and Appendix E.2 states that the 3rd block is selected for Infinity-2B while the 16th block is selected for HART-0.7B. Since s is a free parameter that affects the exclusion mask and the reported latency, the manuscript must state unambiguously which block is used for each model and reconcile the text, figure caption, and appendix.","section":"Section 5, 'Impact of block selection'; Appendix E.2; Figure 8"},{"comment":"The headline operating point is selected on the evaluation benchmarks, and the tau threshold does not transfer across models. Table 1 reports tau=0.7 for Infinity, but Table 3 shows that HART-0.7B at tau=0.7 collapses on human-preference metrics (ImageReward drops from 0.8656 to 0.4333 and HPSv2.1 from 29.53 to 25.71). Because tau, alpha, beta, P, and the block index are all chosen after inspecting GenEval/DPG-Bench results, the 'minimal degradation' claim is at risk of overfitting to the reported settings. The authors should either tune on a separate calibration split or report a selection rule based on a target latency budget, and they should report the per-prompt distribution of the exclusion ratio and quality change for the chosen operating point.","section":"Tables 1, 3, 4, 5; Figure 8"}],"minor_comments":[{"comment":"The heading 'Empircal Insights' contains a typo and should read 'Empirical Insights'.","section":"Section 3 heading"},{"comment":"The implementation details state beta=0.9 and P=10 but do not state the value of alpha in the main text; alpha=4 appears only in the Table 1 caption. Please state all fixed hyperparameters for each model in Section 5.","section":"Section 5, 'Implementation details'"},{"comment":"Please specify the interpolation method used to map Delta F^s_{k-1} to the resolution of stage k, and clarify that the max in Eq. (2) is taken per image over the interpolated map; this affects the reproducibility of the exclusion masks.","section":"Section 4.2, Eq. (2)"},{"comment":"Please clarify whether Eq. (1) is computed on the pruned feature map or the full feature map after anchor-copy assignment, since this affects the exclusion mask propagated to the next stage. Also, the '-' row in Table 4 should explicitly state that excluded tokens receive zero residuals when no anchors are kept.","section":"Algorithm 1 and Section 4.3"}],"recommendation":"major_revision","confidential_remarks":"The paper is a reasonable empirical contribution and the reported speedups on Infinity are intriguing, but the missing random-mask control leaves the central frequency-awareness claim unsupported, and the contradictory block-index specifications must be resolved. I would like the revision to add the randomized ablation, fix the block-selection description, and strengthen the hyperparameter-selection discussion before publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Punchline: the speedup is real at the stated operating point — Infinity-2B goes from 2.78s to 1.36s at tau=0.7 with GenEval dropping from 0.7260 to 0.7209, and the DPG-Bench and human-preference tables tell the same story. The mechanism claim, that low-frequency tokens are the ones that can be safely skipped, is plausible but under-tested: there is no same-budget random-mask control, so we don't know how much of the gain comes from frequency-awareness versus from the sparsity-plus-anchor-copying recipe.\n\nWhat is actually new: using block-wise MSE change as a cheap proxy for frequency and excluding those tokens from later high-resolution stages, with a sparse anchor grid that supplies copied logits to excluded positions. The observation that later stages mostly refine high-frequency regions is well visualized, and the FlexVAR experiment in the appendix is a good check that the redundancy is inherent to next-scale prediction and not an artifact of the residual VAE. Table 4 confirms the anchors earn their keep, especially for HART's residual diffusion. The authors also report the failure modes openly: Table 3 shows tau=0.7 wrecks HART (ImageReward halves), so the threshold clearly does not transfer across models. The math is simple and correct; the citation pattern is fine except that ZipAR — the most relevant prior work on spatial-locality acceleration for autoregressive generation — is cited but never compared.\n\nSoft spots, in order. First, the missing random-mask control. Figure 6 compares three informed metrics and Figure 10 compares methods with different mechanisms; neither fixes sparsity and randomizes only token identity. Second, tau fragility: the same value that works for Infinity collapses HART, meaning per-model tuning is required and the abstract's phrasing overstates generality. Third, an internal inconsistency about block selection: the main text says block 16 was chosen on Infinity, the Figure 8 caption says HART, and the appendix says Infinity uses block 3 and HART block 16. A careful reader cannot tell which block drives the headline result. Fourth, hyperparameters (tau, alpha, beta, P, s) are tuned on GenEval, the latency numbers have no error bars, and the code link is unverified. Minor, but worth stating.\n\nAudience: practitioners working on efficient inference for VAR models; the efficiency subfield. This deserves a serious referee rather than a desk reject. I'd send it out, with the random-mask control, a ZipAR comparison, per-prompt mask statistics, and the block-selection cleanup as requested revisions.","headline":"Useful, honest efficiency paper: the 2x training-free speedup on Infinity-2B holds at its chosen operating point, but the frequency-awareness mechanism is under-tested without a random-mask control and tau needs per-model tuning.","tokens_in":15523,"tokens_out":8224,"would_cite":true,"duration_ms":80735,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"SparseVAR claims that low-frequency tokens at high-resolution stages can be safely skipped, cutting next-scale image-model latency roughly in half with negligible quality loss.","keywords":["next-scale prediction","visual autoregressive model","token reduction","inference acceleration","low-frequency tokens","anchor tokens","text-to-image generation","sparse inference"],"falsifier":"Take a set of generated images and, for each, compute the interpolated MSE map at the selected block, threshold it at $\\tau$, then run inference with the excluded tokens forced to zero residual or to anchor-copied logits. If images with fine detail inside the excluded regions such as text, faces, or texture on a non-flat background degrade visibly while the mask still excludes most tokens, the frequency proxy is misaligned. Quantitatively, one could measure the correlation between each token's interpolated block-MSE value and the actual change that removing that token produces in the final decoded image; a low or negative correlation would refute the paper's premise.","tokens_in":14408,"feed_emoji":"⚡","tokens_out":10841,"duration_ms":97781,"temperature":0.7,"pith_summary":"The paper sets out to show that the high-resolution stages of next-scale prediction image models spend most of their computation on redundant tokens. Decoded-image differences between consecutive stages concentrate in high-frequency regions such as edges, while smooth background areas barely change, so low-frequency tokens can be dropped once a model reaches higher resolutions. SparseVAR identifies those tokens from the mean-squared change of features in a selected transformer block, excludes them from the current and all later stages, and keeps a small regular grid of anchor tokens whose logits are copied into the excluded positions when similar enough. The key result is that this training-free mask cuts Infinity-2B inference from 2.78 s to 1.36 s at threshold $\\tau=0.7$ while the GenEval overall score moves only from 0.7260 to 0.7209. This matters because late-stage high-resolution tokens dominate inference cost in visual autoregressive models, so a cheap frequency mask offers a practical speedup without a new training run.","feed_headline":"Skipping low-frequency tokens nearly halves image-model latency","feed_subtitle":"A no-retraining mask cuts Infinity-2B inference from 2.78 s to 1.36 s with a GenEval drop of 0.0051.","key_machinery":"The load-bearing object is the MSE change map computed inside one block of the model: $$\\$\\Delta$ $F^{{s}}$_{k-1}(i,j) = \\frac{1}{C}\\sum_{c=1}^{C}\\left($F^{{s}}$_{k-1}(i,j,c) - $F^{{s-1}}$_{k-1}(i,j,c)\\right)^2,$$ where $s$ is the selected block and $C$ the channel dimension. After interpolation to stage $k$, the exclusion set $$\\mathcal{M}^{k}_{\\mathrm{low}} = \\{(i,j) \\mid \\tilde{F}_k(i,j) < \\tau \\max(\\tilde{F}_k)\\}$$ sends those positions out of the computation for stage $k$ and later stages. Anchor retention is the second mechanism: the top-left corner of every $\\alpha \\times \\alpha$ grid is kept, and each excluded token receives the logits of its most similar anchor when the cosine similarity of the interpolated previous-stage logits exceeds threshold $\\beta$, otherwise its residual is set to zero. These two mechanisms turn the sparsity decision into a lightweight per-image choice of one block's MSE map and a threshold, avoiding clustering or attention ranking that would itself be expensive at high resolution.","core_discovery":"SparseVAR's central claim is that low-frequency tokens are largely inert during the high-resolution stages of next-scale prediction and can be excluded with almost no image-quality cost. The method rests on two observations: the per-stage residuals of HART, Infinity, and FlexVAR concentrate their effect in high-frequency regions, and different transformer blocks within the model attend to different spatial frequencies. A selected block's feature MSE change map from stage $k-1$ is interpolated to stage $k$; tokens whose interpolated values fall below $\\tau$ times the map maximum are marked low-frequency and skipped, while the skipped positions either receive the logits of their most similar anchor token or a zero residual. At $\\tau=0.7$ on Infinity-2B the latency drops from 2.78 s to 1.36 s with GenEval moving from 0.7260 to 0.7209, and at $\\tau=0.6$ on HART-0.7B the latency drops from 1.32 s to 0.99 s with the GenEval score essentially flat. The appendix applies the same procedure to FlexVAR, supporting the stronger claim that the redundancy is a property of multi-stage refinement itself, not of the residual VAE designs used by HART and Infinity.","pith_inferences":["A natural untested extension is to combine frequency masking with KV-cache or attention sparsification, since the skipped low-frequency tokens are precisely the positions whose long-range attention is weakest.","Because anchors are placed on a fixed grid, an adaptive scheme that places anchors along detected edges or texture boundaries would likely match or exceed the same quality while using fewer anchor tokens.","The reported latencies are single-GPU, batch-1 measurements on an NVIDIA 3090; production serving with larger batches or tensor-parallel execution may compress the wall-clock gain, even though the reduction in computed tokens should transfer.","The same frequency-proxy idea may carry over to high-resolution diffusion transformers, which also waste computation on flat backgrounds, though the paper deliberately restricts its claim to next-scale autoregressive models."],"forward_implications":["Any next-scale prediction model can be accelerated at inference time by adding this mask and a small number of anchor copies, with no retraining and no change to the pretrained weights.","At 1024x1024, users of Infinity-2B can expect roughly a doubling of generative speed at threshold $\\tau=0.7$ while text-to-image alignment on GenEval shifts by only about half a percent.","Because the redundancy appears in HART, Infinity, and FlexVAR, the speedup should become larger at even higher resolutions, where late stages process many more tokens and low-frequency regions dominate the token count.","The quality of the acceleration depends on reading the right block's MSE map, so a practical deployment should search the block index once per model family rather than treating the same block as universally valid.","Anchor tokens are load-bearing for HART's residual diffusion path: without them, low-frequency regions lose the inputs the diffusion refinement expects, and quality degrades more than it does for direct next-scale models."],"supporting_citations":[{"why":"Defines the next-scale prediction paradigm that SparseVAR operates on and supplies the VAR baseline.","marker":"[29]"},{"why":"Provides HART-0.7B, the hybrid residual-diffusion model used for the main quality-speed trade-off experiments.","marker":"[28]"},{"why":"Provides Infinity-2B, the 1024x1024 bitwise model on which the 2x speedup is measured.","marker":"[10]"},{"why":"Supplies the pretrained codebook that maps predicted logits to residual feature maps and establishes the multi-scale tokenizer idea.","marker":"[30]"},{"why":"Documents the strong local-dependency pattern of attention in autoregressive image generation, motivating why attention-score token selection fails in this setting.","marker":"[11]"},{"why":"GenEval is the primary text-to-image alignment benchmark whose overall score quantifies the quality change of SparseVAR.","marker":"[9]"},{"why":"ToMe is the token-merging baseline compared in the appendix, showing merging degrades quality at high sparsity.","marker":"[3]"},{"why":"ZipVL is the token-selection baseline used in the appendix comparison, representing the attention-score pruning alternative.","marker":"[12]"}],"fun_headline_variants":["SparseVAR skips low-freq tokens, nearly doubles image synthesis speed","No-retrain token pruning halves latency in next-scale image models","Low-frequency token dropout speeds up image generation up to 2x","SparseVAR: discard inert tokens, keep image quality, cut compute","Skipping low-frequency tokens cuts image generation latency in half"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The acceleration stands or falls with the assumption that a selected block's interpolated feature MSE change at the previous scale correctly identifies which tokens can be skipped at the current and later scales without visible harm; the paper supports this with qualitative visualizations and a block-selection experiment rather than a direct per-token measurement.","fun_headline_variants_meta":{"raw":{"variants":["SparseVAR skips low-freq tokens, nearly doubles image synthesis speed","No-retrain token pruning halves latency in next-scale image models","Low-frequency token dropout speeds up image generation up to 2x","SparseVAR: discard inert tokens, keep image quality, cut compute","Skipping low-frequency tokens cuts image generation latency in half"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00084,"raw_usage":{"total_tokens":3702,"prompt_tokens":1030,"completion_tokens":2672,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":646,"completion_tokens_details":{"reasoning_tokens":2580}},"tokens_in":646,"tokens_out":2672,"duration_ms":17849,"temperature":1.0,"reasoning_tokens":2580,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T17:42:36.193083+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a set of generated images and, for each, compute the interpolated MSE map at the selected block, threshold it at $\\tau$, then run inference with the excluded tokens forced to zero residual or to anchor-copied logits. If images with fine detail inside the excluded regions such as text, faces, or texture on a non-flat background degrade visibly while the mask still excludes most tokens, the frequency proxy is misaligned. Quantitatively, one could measure the correlation between each token's interpolated block-MSE value and the actual change that removing that token produces in the final decoded image; a low or negative correlation would refute the paper's premise.","supporting_citations":[{"cited_title":"Visual autoregressive modeling: Scalable image generation via next-scale prediction","cited_arxiv_id":null,"evidence_quote":"Defines the next-scale prediction paradigm that SparseVAR operates on and supplies the VAR baseline."},{"cited_title":"Neural discrete representation learning","cited_arxiv_id":null,"evidence_quote":"Supplies the pretrained codebook that maps predicted logits to residual feature maps and establishes the multi-scale tokenizer idea."},{"cited_title":"Geneval: An object-focused framework for evaluating text-to-image alignment","cited_arxiv_id":null,"evidence_quote":"GenEval is the primary text-to-image alignment benchmark whose overall score quantifies the quality change of SparseVAR."},{"cited_title":"Token merging: Your vit but faster","cited_arxiv_id":null,"evidence_quote":"ToMe is the token-merging baseline compared in the appendix, showing merging degrades quality at high sparsity."}],"review_version":2}