{"id":"032bb304-07d2-4e64-8d28-d2352ac6c0d2","arxiv_id":"2504.16922","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Generalized Neighborhood Attention adds a stride parameter to neighborhood attention, aligning windows with GPU tiles so sparse attention achieves its full theoretical speedup; a Blackwell kernel delivers 1.26x-1.63x end-to-end speedups on three generative models.","lead":"A new stride parameter for neighborhood attention aligns sliding windows with GPU tile boundaries, letting sparse attention reach its theoretical speedup for images and video. The authors build a Blackwell kernel and a simulator, and show 26-63% end-to-end speedups on Cosmos, HunyuanVideo, and FLUX without fine-tuning.","discovery_kind":"unification","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Token-permutation equivariance is the load-bearing assumption for the off-the-shelf end-to-end claims; it is unproven and questionable for RoPE-based models like FLUX and HunyuanVideo.","rationale":"The reader's weakest assumption was that runtime scales with the number of visited KV tiles, with token-permutation overhead being a possible violation. The paper's own measurements, however, show that in the perfectly block-sparse cases (e.g., HunyuanVideo stride 16x8x8) the actual op-level speedup is 11.1x, matching the analytical and FLOP-wise predictions, and the end-to-end speedups in Tables 2-4 also match analytical values to within a few percent. So the runtime-scaling assumption is empirically validated for the tested cases; the residual gap is small and already disclosed. The more serious, and less examined, assumption is the permutation equivariance claim in Sec. 3.3 point 4, which is load-bearing for the entire 'no fine-tuning' application story. FLUX and HunyuanVideo are known to use RoPE, and RoPE is not invariant to an arbitrary reordering of the sequence. If the permutation changes the positions used by RoPE, the model's behavior changes, and the reported quality/end-to-end speedup numbers would not correspond to the original model. This is directly testable and would settle whether the headline application claims are valid. The kernel-level speedup contribution is independent and appears sound; the concern targets the applicability claims, so the verdict remains CONDITIONAL rather than a rejection.","tokens_in":22499,"tokens_out":22126,"duration_ms":195219,"concrete_test":"Pick FLUX.1-dev or HunyuanVideo with a fixed seed and a fixed prompt/latent. Run the dense (self-attention) baseline forward pass, recording the output latents. Then apply the exact GNA token permutation (as implemented) to the input, run the same dense model in the permuted order, and reverse the permutation at the end. Compare the reversed output to the baseline output. If the maximum absolute difference is not at the noise floor (e.g., <1e-4 relative), the architecture is not permutation-equivariant and the paper's use of token permutation without fine-tuning is invalid. Also report whether the model implementation applies RoPE from the permuted sequence index or from the original grid coordinates.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The paper's end-to-end speedups without fine-tuning (Tables 2-6) depend on Sec. 3.3 point 4: 'Most transformer architectures are equivariant to permutation, and this holds true for both ViT and DiT.' This is only correct for additive absolute positional encodings. DiT/MMDiT models such as FLUX and HunyuanVideo use rotary position embeddings (RoPE), where Q and K are rotated by angles determined by their index in the current sequence order. The GNA token permutation reorders the sequence before the first layer; every subsequent attention layer computes RoPE from the permuted indices, changing the relative geometry used by the attention scores. The result is a different model, not a permuted version of the same model. The paper gives no equivalence test showing that applying the permutation and its inverse reproduces the baseline latents, and it does not mention any modification to preserve original coordinates for RoPE. The reported quality metrics show only small differences from baseline, which is inconsistent with a full break of positional semantics; this suggests either the models used are not RoPE-based (contradicting FLUX/Hunyuan's known architectures) or the equivalence was silently ensured. Without this support, the end-to-end speedup numbers may be measured on different model behavior, and the 'without any fine-tuning' claim is not established.","agreement_with_reader":"disagree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Generalized Neighborhood Attention (GNA), an extension of Neighborhood Attention with a stride parameter that groups queries into shared context windows, unifying sliding-window, strided, and blocked attention. It introduces NATTENSim, a simulation tool that computes speedup upper bounds by counting KV tiles visited under configurable tiling choices. The authors implement GNA within a CUTLASS-based FMHA kernel for NVIDIA Blackwell, using token permutation instead of multidimensional tiling, and report op-level speedups up to 11.1x at 91% sparsity and end-to-end speedups of 1.26x (Cosmos-7B), 1.63x (HunyuanVideo), and 1.45x (FLUX-1.dev) on a B200 without fine-tuning. The central performance claim—that perfectly block-sparse GNA configurations realize the FLOP-wise speedup—is supported by Tables 2-4 and Fig. 5.","tokens_in":22726,"tokens_out":12959,"duration_ms":115788,"significance":"If the claims hold, this is a valuable systems contribution: a clean unification of static local-attention patterns, a realistic tiling-level simulator, and a Blackwell kernel that, in perfectly block-sparse settings, matches analytical speedups with no fitted parameters. The paper honestly identifies the conditions (TKV dividing window size and TQ dividing stride) under which FLOP-proportional speedup is achievable, and the op-level kernel measurements in Fig. 5 corroborate the simulator. The main risk is the token-permutation equivariance assumption used to transfer these op-level gains to unmodified generative models; that assumption is load-bearing for the without-fine-tuning end-to-end claims.","major_comments":[{"comment":"The claim that 'most transformer architectures are equivariant to permutation, and this holds true for both ViT and DiT' is load-bearing for the off-the-shelf, no-fine-tuning end-to-end results in Tables 2-6. It is only valid for architectures whose positional information is carried additively in the token rows. FLUX and HunyuanVideo use rotary position embeddings, where Q and K are rotated by angles derived from the token index in the current sequence order. Permuting the tokens before the first layer changes those indices and therefore changes the relative geometry of every attention head, so the permuted model is not a permutation of the baseline model. The manuscript reports no equivalence test (e.g., comparing hidden states or output latents with and without a permutation and its inverse), and Sec. 6's description of token permutation does not mention preserving original RoPE coordinates. Please either provide such a test, explain how RoPE coordinates are kept fixed under the permuted layout, or restrict the no-fine-tuning model-level claims to architectures for which permutation equivariance provably holds.","section":"Sec. 3.3, item 4"},{"comment":"The analytical speedups in Tables 2-4 and Fig. 5 treat runtime as proportional to the number of visited KV tiles and do not explicitly model the token permutation and reverse permutation. Section 3.3 point 3 estimates this cost as 1-2% of FMHA time using half of the 8 TB/s HBM bandwidth, but Section 6 states that the current PyTorch copy achieves only about 1/8 of peak bandwidth. At 91% sparsity, the attention time after sparsification is a small fraction of the full attention time, so the permutation transfer time can be comparable to or larger than the remaining attention time. The fact that Table 3 reports an actual end-to-end speedup equal to the analytical value (2.23x) for stride 16x8x8 therefore needs a quantitative explanation: please report the measured time for permute/reverse, state whether these costs were included in the analytical E2E numbers, and, if they were excluded, annotate the tables accordingly.","section":"Sec. 3.3, point 3 vs. Sec. 6"}],"minor_comments":[{"comment":"The caption contains typos: 'full the analytical speedup' and 'the the memory operation'; please copyedit.","section":"Figure 5 caption"},{"comment":"The abstract states '28% to 46% end-to-end speedup', while the introduction and Table 3 report 1.63x (63%) for HunyuanVideo; please align these numbers.","section":"Abstract and Sec. 1"},{"comment":"The text says GNA's reduced workload 'limits achievable end-to-end speedup to approximately 1.64x' for Cosmos and '1.72x' for HunyuanVideo; these are the bounds when the GNA attention time is zero, not the FLOP-wise bounds for the stated sparsity levels (1.28x and 1.63x in Tables 2-3). Please rephrase to avoid confusion.","section":"Sec. 4.2"},{"comment":"The abstract and conclusion report 1.3 petaFLOPs/s for the FP16 kernel, while Sec. 3.3 states the CUTLASS FMHA baseline achieves up to 1.2 petaFLOPs/s; please reconcile the numbers.","section":"Sec. 3.3 and Conclusion"},{"comment":"The GNA mask is defined verbally; a formal equation or pseudocode for the neighborhood relation (given window left/right, stride, and coordinate space) would make the definition precise and reproducible.","section":"Sec. 3.1"},{"comment":"Please clarify how token permutation is scheduled in the mixed SA/GNA runs (e.g., first 12/15/9 self-attention steps followed by GNA), since the description 'permute before the first layer and reverse after the last' does not apply when the initial steps must run in the original layout.","section":"Sec. 3.3"},{"comment":"NATTENSim's 'upper-bound speedup' terminology is confusing: using the maximum number of KV tiles visited by any Q tile yields a conservative estimate of speedup rather than an upper bound on speedup; please define the quantity precisely.","section":"Sec. 3.2"}],"recommendation":"major_revision","confidential_remarks":"The token-permutation equivariance issue is the main risk to the paper's headline claims. If the authors cannot demonstrate equivalence for RoPE-based models such as FLUX and HunyuanVideo, the without-fine-tuning model-level claims should be removed or substantially reframed as a systems result; the kernel and simulator contributions would still be valuable. I would look for a concrete equivalence test in the revision rather than a purely textual rebuttal."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Couple of things to know. The core performance claim holds: in perfectly block-sparse configurations, the Blackwell FNA kernel matches or nearly matches the FLOP-wise upper bound on op-level speedup (Tables 2-4, Fig. 5). That is a genuine engineering result. The GNA parameterization with independent TQ and TKV tile sizes is a real step beyond STA's equal tile size assumption, and NATTENSim is a sensible tile-level simulator that gives more honest speedup upper bounds than raw FLOP counts.\n\nThe soft spot is Sec. 3.3 point 4. The paper justifies doing token permutation once before the first transformer layer by saying \"most transformer architectures are equivariant to permutation\" and that this holds for ViT and DiT. For ViT with absolute positional embeddings, fine. But FLUX and HunyuanVideo both use rotary position embeddings (RoPE). Permuting the sequence before the first layer and then computing RoPE from the permuted indices changes the position geometry in every layer. The reported quality stays close to baseline, which is not what you'd expect from a full break of positional semantics; either the permutation preserves RoPE coordinates in a way the paper doesn't describe, or the models are not being evaluated under genuinely equivalent conditions. Either way, the paper provides no equivalence test, and the \"without fine-tuning\" end-to-end claim is not established as written.\n\nOther issues are smaller but worth fixing. The abstract's 28-46% end-to-end speedup range contradicts the body's 1.26x (26%), 1.63x (63%), and 1.45x (45%). The \"1.3 petaFLOPs/s\" figure is undefined and inconsistent with the body's 1.2 petaFLOPs/s FP16 number. Quality evaluation is thin: VBench and GenEval have no error bars, and Cosmos has only qualitative results. The token-permutation bandwidth estimate assumes half of 8TB/s while the implementation uses about 1/8th of peak; the paper admits this, so it's a known limitation rather than a hidden flaw.\n\nThe central speedup result is solid and the simulator is a useful contribution for anyone building sparse attention kernels or deploying diffusion models at high resolution. I'd send this to a serious referee, but the revision needs to reconcile the headline numbers and either prove or qualify the permutation equivariance claim. As is, I wouldn't cite the no-fine-tuning speedups without checking the implementation.","headline":"The kernel speedup is real and the simulator is useful, but the paper's off-the-shelf end-to-end claims rest on a false token-permutation equivariance assumption for RoPE-based models.","tokens_in":23375,"tokens_out":8150,"would_cite":true,"duration_ms":68000,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A stride parameter shares context windows across query groups, making neighborhood attention perfectly block-sparse, so measured speedup hits the FLOP-wise maximum: 11.1x at 91% sparsity on attention ops, 1.26x–1.63x end-to-end.","keywords":["generalized neighborhood attention","sparse attention","sliding window attention","block sparsity","fused multi-head attention","Blackwell GPU kernels","diffusion model acceleration"],"falsifier":"Run the kernel on a B200 with two perfectly block-sparse GNA settings that visit the same number of key-value tiles but use different tile shapes or window/stride combinations: the model predicts identical speedup, so any measurable runtime difference falsifies the claim that realized speedup equals the tile-count upper bound. A quantitative second check: time the token-permutation copies alone for HunyuanVideo's roughly 10.5 GB of attention tensors; at the paper's admitted one-eighth-of-bandwidth implementation these take on the order of 20 ms per transformer pass, several times the assumed 1–2% overhead, and if that overhead grows with sparsity the measured end-to-end speedups will fall short of the reported values.","tokens_in":22251,"feed_emoji":"⚡","tokens_out":28367,"duration_ms":228420,"temperature":0.7,"pith_summary":"Sparse attention has long promised to escape the quadratic cost of self attention, but sliding-window patterns spend much of their saving on masked-out FLOPs, so runtime never drops as much as the sparsity ratio suggests. The paper claims this gap can be closed by adding a single 'stride' parameter to neighborhood attention: stride groups queries that must share a context window, aligning window boundaries so the mask becomes perfectly block-sparse, meaning every computed FLOP is used. For such configurations, the paper's Blackwell kernel matches the analytically predicted upper bound — 11.1x at the attention-op level at 91% sparsity, sustaining 1.3 petaFLOPs/s in FP16 — and off-the-shelf diffusion models Cosmos-7B, HunyuanVideo, and FLUX run 1.26x, 1.63x, and 1.45x faster end-to-end with no fine-tuning. The paper also contributes a simulator that predicts these upper-bound speedups by counting the key-value tiles each query tile must visit under different tile shapes and tiling strategies. The upshot is that sparse local attention can finally be as fast as its arithmetic, which matters because attention dominates the workload in these generative models.","feed_headline":"At 91% sparsity, attention runs 11x faster","feed_subtitle":"Stride packs sliding windows into dense tiles, yielding up to 63% faster generation with no retraining.","key_machinery":"The load-bearing object is the stride parameter of Generalized Neighborhood Attention, which takes any positive integer up to the window size: stride 1 reproduces neighborhood attention, stride equal to the window size reproduces blocked attention (window self attention), and intermediate values produce strided sliding-window patterns, including HaloNet-style blocked local attention and sliding tile attention. Mechanically, stride partitions queries into groups that share one context window; when $T_Q$ divides the stride and $T_{KV}$ divides the window size, every shared window aligns to tile boundaries, making the mask perfectly block-sparse — masked-out regions coincide exactly with the kernel's tile grid, so no computation is spent on entries that are later discarded. Around this sits the paper's simulator, NATTENSim, which counts the key-value tiles each query tile visits under choices of tile shapes, static versus dynamic key-value tiling, and one- versus multi-dimensional tiling, producing upper-bound speedups finer than the FLOP-wise ratio. The implementation is a fused-neighborhood-attention-style kernel for the Blackwell architecture that takes multi-dimensional tiling out of the kernel and implements it as a token-permutation step, keeping static, one-dimensional tile indexing for tensor-memory loads; the paper reports that it fully realizes the simulated speedup in perfectly block-sparse cases.","core_discovery":"The central discovery is that the distance between sliding-window attention and blocked attention reduces to one parameter: stride. Standard neighborhood attention moves the context window one query at a time, so windows straddle tile boundaries and some FLOPs land on entries that are later masked away; GNA's stride moves the window once per query group, forcing every query in the group to share one context window. Whenever the query tile size $T_Q$ divides the stride and the key-value tile size $T_{KV}$ divides the window size, shared windows align exactly with tile boundaries, the mask is perfectly block-sparse, and fine-grained masking can be skipped entirely. The authors show that a kernel built by adapting the best fused multi-head attention kernel for the Blackwell architecture to this pattern, moving multi-dimensional tiling into a token-permutation step, fully realizes the predicted speedup in these cases: on the HunyuanVideo workload at 91% sparsity, GNA with stride $16\\times 8\\times 8$ reaches 11.1x at the op level and 2.23x end-to-end when applied to all diffusion steps, which the paper calls 'the maximum speedup theoretically possible for this exact level of sparsity.' The same recipe gives 1.26x on Cosmos-7B and 1.45x on FLUX 4K without any fine-tuning.","pith_inferences":["Editorial inference: the perfectly block-sparse condition is a divisibility constraint tied to hardware tile shapes, so future GPUs with different tensor-core tile sizes will redefine which strides count as perfect; the paper's specific numbers are B200-bound, but its method of hunting for tile-aligned patterns should carry over.","Editorial inference: the paper's admission that token permutation currently uses only about one-eighth of the card's memory bandwidth implies that partially block-sparse configurations still have measurable headroom; a faster permutation kernel should shrink the recorded gap between 'actual' and 'actual excluding memory ops' speedups without any change to the attention math.","Editorial inference: because stride trades translational equivariance for computational density, quality and speed share a single knob; the zero-fine-tuning results hint that fine-tuning with GNA — the paper's 'all diffusion steps' setting — could push workable sparsity beyond the roughly 90% explored here and make settings like HunyuanVideo's 2.23x the realistic operating point.","Editorial inference: the same tile-count simulation could be applied to other fine-grained masks — dilated windows, causal video masks — to locate their hidden perfectly block-sparse configurations, generalizing the sparsity-proportional speedup result beyond the three pattern families GNA currently unifies."],"forward_implications":["Perfectly block-sparse GNA configurations deliver the full FLOP-wise speedup at the op level — 11.1x at 91% sparsity on the HunyuanVideo attention workload — so the gap between sparse attention's promised and measured runtime is closed in these cases.","Off-the-shelf diffusion models gain real end-to-end speed without retraining: 1.26x on Cosmos-7B, 1.63x on HunyuanVideo, and 1.45x on FLUX 4K on a B200, because attention is the dominant share of these workloads.","GNA subsumes sliding-window, strided sliding-window, and blocked attention in one definition, turning a family of incompatible implementations into a single window/stride parameter sweep guided by the simulator's tile-count predictions.","The design choices are presented as architecture-agnostic, so the same recipe should transfer to other GPU architectures and future fused-attention kernels, making sparsity-proportional speedup a design target rather than a special case."],"supporting_citations":[{"why":"Defines neighborhood attention, the sliding-window pattern that GNA extends with its stride parameter.","marker":"[21]"},{"why":"Introduced the delay-step (strided) sliding-window idea that GNA's stride generalizes, and set the memory-cost framing the paper revisits.","marker":"[43]"},{"why":"Supplies the fused multi-headed attention kernel for the Blackwell architecture on which the GNA implementation is built.","marker":"[41]"},{"why":"The fused neighborhood attention kernel whose multi-dimensional tiling the new kernel adapts by moving tiling into a token-permutation step.","marker":"[19]"},{"why":"Sliding tile attention, the closest comparative method, which GNA subsumes and is benchmarked against on VBench.","marker":"[54]"},{"why":"Defines window self attention, the blocked-attention case GNA reproduces exactly at stride equal to window size.","marker":"[31]"},{"why":"Provides the HunyuanVideo workload behind the headline 91%-sparsity results: 11.1x op-level and 2.23x end-to-end.","marker":"[27]"},{"why":"Provides the Cosmos-7B workload for the 1.26x end-to-end result.","marker":"[1]"},{"why":"Provides the FLUX workload for the 1.45x end-to-end image-generation result.","marker":"[28]"}],"fun_headline_variants":["Sparse attention 11x faster at 91% sparsity","Stride-aligned windows yield 11x attention speedup","No retraining: 11x faster attention with GNA","GNA achieves 11x speedup at 91% sparsity","Attention goes 11x faster via stride-packing"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole argument rests on the assumption that attention time shrinks in proportion to the number of tiles the kernel must visit, so every skipped tile saves real time, and that the token-permutation step which enables static tiling is nearly free — the paper assumes it costs 1–2% of attention time at half memory bandwidth, while its own current implementation of that step reaches only about one-eighth of the bandwidth.","fun_headline_variants_meta":{"raw":{"variants":["Sparse attention 11x faster at 91% sparsity","Stride-aligned windows yield 11x attention speedup","No retraining: 11x faster attention with GNA","GNA achieves 11x speedup at 91% sparsity","Attention goes 11x faster via stride-packing"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000597,"raw_usage":{"total_tokens":2877,"prompt_tokens":1115,"completion_tokens":1762,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":731,"completion_tokens_details":{"reasoning_tokens":1675}},"tokens_in":731,"tokens_out":1762,"duration_ms":12928,"temperature":1.0,"reasoning_tokens":1675,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T10:52:44.259026+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the kernel on a B200 with two perfectly block-sparse GNA settings that visit the same number of key-value tiles but use different tile shapes or window/stride combinations: the model predicts identical speedup, so any measurable runtime difference falsifies the claim that realized speedup equals the tile-count upper bound. A quantitative second check: time the token-permutation copies alone for HunyuanVideo's roughly 10.5 GB of attention tensors; at the paper's admitted one-eighth-of-bandwidth implementation these take on the order of 20 ms per transformer pass, several times the assumed 1–2% overhead, and if that overhead grows with sparsity the measured end-to-end speedups will fall short of the reported values.","supporting_citations":[{"cited_title":"Neighborhood attention transformer","cited_arxiv_id":null,"evidence_quote":"Defines neighborhood attention, the sliding-window pattern that GNA extends with its stride parameter."},{"cited_title":"Scaling local self-attention for parameter efficient visual backbones","cited_arxiv_id":null,"evidence_quote":"Introduced the delay-step (strided) sliding-window idea that GNA's stride generalizes, and set the memory-cost framing the paper revisits."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the fused multi-headed attention kernel for the Blackwell architecture on which the GNA implementation is built."},{"cited_title":"Faster neighborhood attention: Reducing the O(n2) cost of self at- tention at the threadblock level","cited_arxiv_id":null,"evidence_quote":"The fused neighborhood attention kernel whose multi-dimensional tiling the new kernel adapts by moving tiling into a token-permutation step."},{"cited_title":"Swin transformer: Hierarchical vision transformer using shifted windows","cited_arxiv_id":null,"evidence_quote":"Defines window self attention, the blocked-attention case GNA reproduces exactly at stride equal to window size."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the FLUX workload for the 1.45x end-to-end image-generation result."}],"review_version":1}