{"id":"677ba876-668d-48d2-9b2c-2b588db64735","arxiv_id":"2506.18999","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"The authors distill a diffusion transformer into a mostly-Mamba hybrid model, reaching teacher-level GenEval scores while generating up to 4K images with linear-complexity speed.","lead":"This paper trains a fast image generation model based on Mamba, a linear-time architecture, by copying knowledge from a slower transformer model called PixArt. The result is a hybrid model that generates 2048x2048 images with lower compute and quality close to its teacher.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 'bidirectional' scan in Sec 3.3 is actually two causal scans; the 4 SA layers carry the global non-causal context, so the feasibility claim for sequential causal Mamba is not established.","rationale":"The reader's conditional verdict is appropriate. I focused on the bidirectional scanning assumption because it underlies the paper's headline conceptual contribution, stated in the abstract's final sentence: that sequential and causal Mamba models can generate non-causal visual output. The GenEval inconsistency (0.28 in the abstract vs 0.184 in the table caption) is a numerical error that should be corrected, and the absence of quantitative high-resolution metrics is a reporting gap, but neither directly challenges the method's central argument as much as the scanning issue. Section 3.3 describes 'bidirectional' scanning as width-first and height-first interleaved scans with shared weights and linear fusion. Both are causal flattenings of the 2D grid; no reverse pass is present, so a token cannot see tokens to its right or below at any Mamba layer, and depth does not reverse that directionality. The four self-attention blocks are therefore the only truly non-causal global mixers, and Table 1's 'No SA' row demonstrates that removing them sharply degrades GenEval. This means the results may justify a hybrid-model claim, but they do not establish that causal Mamba alone models non-causal visual dependencies. A true-bidirectional ablation on the 'No SA' variant would settle the question directly. I keep the verdict conditional, as the method may still work as a hybrid, but the paper must either provide that evidence or soften its feasibility claim.","tokens_in":13475,"tokens_out":8078,"duration_ms":84805,"concrete_test":"Retrain the 'No SA' variant from Table 1 under identical settings with true bidirectional scans (forward and reverse along both width and height, still weight-shared and linearly fused) in place of the current width-first/height-first causal scans, and compare GenEval. If true bidirectional does not improve over the published 0.420, the current scan is already sufficient; if it improves substantially, the claimed 'bidirectional' scheme is not delivering global context and the feasibility conclusion must be restricted to the hybrid with 4 SA layers.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central feasibility claim—that sequential causal Mamba can generate non-causal visual output—rests on Sec 3.3's 'bidirectional scanning strategy' (width-first and height-first interleaved scans with shared weights and a linear fusion). As described, both scans are causal in their traversal order. A width-first raster scan gives a token at position (i,j) access only to tokens in earlier rows and to its left in the same row; a height-first column scan gives access only to tokens in earlier columns and above in the same column. Neither includes a reverse pass, so no Mamba layer can receive information from tokens to the right or below. Stacking layers does not reverse this directionality: information still flows only from earlier to later positions in each scan order. Consequently, the 24 Mamba blocks are not actually bidirectional and do not supply global 2D context. The only truly non-causal global mechanism is the 4 self-attention blocks in the Sec 4.1 pattern SA-(HM-WM)x3 repeated. Table 1's 'No SA' row drops to 0.420 GenEval, confirming that the attention layers are doing substantial work. The reported 0.485 vs 0.481 GenEval gap therefore supports a hybrid model with 14% attention, not the claim that causal Mamba alone can cover non-causal visual dependencies. The paper provides no receptive-field analysis and no reverse-scan ablation to settle this.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes T2MD, a multi-stage distillation pipeline that transfers a pre-trained PixArt-α diffusion transformer into a 0.7B hybrid diffusion model containing 24 Mamba layers and 4 self-attention layers. The stages are layer-level teacher forcing, feature-based knowledge distillation, component adaptation, and high-resolution fine-tuning, with the end goal of efficient 2048x2048 and zero-shot 4K text-to-image generation. The paper reports a GenEval score of 0.485 for the distilled model versus 0.481 for the teacher, an MS-COCO FID-30K of 8.63, and 1.5x to 2.1x latency speedups over a DiT baseline at high resolutions. It also makes a broader feasibility claim: that sequential and causal Mamba models can generate non-causal visual output.","tokens_in":13771,"tokens_out":5270,"duration_ms":51541,"significance":"If the claims hold, the work would be a useful contribution to efficient high-resolution text-to-image generation: it offers a concrete training recipe for avoiding expensive from-scratch training of state-space diffusion models and provides quantitative evidence that a hybrid Mamba model can approach a DiT teacher on GenEval. The layer-level teacher-forcing idea is plausible and the ablation table gives useful evidence about which components contribute. However, the strongest scientific claim, that causal Mamba alone can handle non-causal visual dependencies, is not supported by the architecture as described, since the two scan directions are both causal and the model contains self-attention layers. The high-resolution quality claims also currently rest on qualitative examples only. The paper is therefore potentially valuable, but the central claims need significant additional evidence or careful restatement.","major_comments":[{"comment":"The \"bidirectional scanning strategy\" is described only as width-first and height-first interleaved scans with shared weights and a linear fusion; both scans are causal in their traversal order, so no Mamba block has access to tokens to the right or below. The only non-causal global mechanism in the model is therefore the four self-attention blocks in the Sec 4.1 pattern. The GenEval drop in the \"No SA\" row (0.420 vs 0.485) and the \"Bi-dir→Uni-dir\" row (0.448 vs 0.485) show that these design choices matter, but they do not test whether a pure causal-Mamba stack can handle non-causal visual dependencies. Please add a reverse-scan ablation and a receptive-field analysis, or restrict the feasibility claim to the hybrid architecture.","section":"Sec 3.3, Table 1"},{"comment":"The abstract and Sec 1 state that T2MD improves GenEval by 0.28, while Table 1 reports 0.301 to 0.485, an improvement of 0.184. This discrepancy concerns the headline quantitative result and must be corrected in the final version.","section":"Abstract and Table 1"},{"comment":"High-resolution and zero-shot 4K claims are supported only by qualitative figures and by the latency/throughput measurements in Table 3; no FID, GenEval, or user-study numbers are reported for 1024, 2048, or 4K, and no comparison is made to high-resolution-capable baselines such as SDXL, PixArt-sigma, or LinFusion at those resolutions. Since the first beyond-2k SSM text-to-image claim is a central contribution, quantitative quality evaluation at high resolution is required.","section":"Sec 4.3, Figs 1 and 6"},{"comment":"The claim that the student \"reaches the teacher model in quality\" is only supported on GenEval; on MS-COCO FID-30K the student is worse than the teacher (8.63 vs 7.32, where lower is better). Please qualify the claim or provide additional metrics that substantiate teacher-level quality.","section":"Sec 4.3.2, Table 2"}],"minor_comments":[{"comment":"The word \"casual\" appears where \"causal\" is meant in the introduction and in Sec 4.3.2; please fix throughout.","section":"Sec 1 and Sec 4.3.2"},{"comment":"The model is referred to as \"T2DM\" twice in Sec 4.3.2; use \"T2MD\" consistently.","section":"Sec 4.3.2"},{"comment":"The \"SA latency\" column is not defined; label the baseline explicitly and state whether it is the teacher PixArt-α or another DiT configuration.","section":"Table 3"},{"comment":"The \"Bi-dir→Uni-dir\" ablation is not described in the text; please define exactly what change is applied and what the row is meant to show.","section":"Table 1"},{"comment":"The token-mixer loss is written in terms of ε[n] without defining the per-block output convention; clarify the notation so the reader can tell what is compared at each layer.","section":"Sec 3.4.2, Eq (10)"},{"comment":"The contribution claim of being the first SSM-based beyond-2k text-to-image model is not verified against recent concurrent work; consider a more careful survey or a qualified statement.","section":"Sec 1"}],"recommendation":"major_revision","confidential_remarks":"The paper is likely salvageable. The distillation pipeline and GenEval ablations are solid, but the manuscript currently overclaims on two fronts: the causal-Mamba feasibility statement does not follow from the described architecture, and the high-resolution quality evidence is only qualitative. A major revision that adds quantitative high-resolution evaluation and either ablates true bidirectional scanning or restates the feasibility claim would substantially strengthen the paper."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's the short version. The layer-level teacher forcing idea is the real contribution: feeding each student Mamba layer the teacher's intermediate input prevents the usual error accumulation when distilling non-causal attention into causal state-space layers. The ablations are honest and the 512x512 numbers are credible—GenEval 0.485 versus the PixArt teacher's 0.481, with the baseline hybrid Mamba at 0.301. The multi-stage path to 2048 and 4K is practical and the speedups, while only measured on a single H100, are plausible.\n\nBut the paper's headline claim—that sequential causal Mamba models can generate non-causal visual output—is not backed by the architecture they actually built. The 'bidirectional scanning' in Sec 3.3 interleaves a width-first scan and a height-first scan, and both are causal in traversal order. Neither gives a token access to anything to its right or below. The only non-causal global mechanism in the model is the 4 self-attention blocks in the SA-(HM-WM)x3 pattern. The 'No SA' ablation drops GenEval to 0.420, and the Bi-dir->Uni-dir ablation only costs 0.037, so the two scans contribute less than the 4 attention layers. The paper would need a true forward+reverse scan ablation or a receptive-field analysis before the sequential-causal claim can stand.\n\nTwo more concrete problems. The abstract says the method improves GenEval by 0.28, but Table 1 shows 0.184 (0.301 to 0.485); the numbers don't match. And the high-resolution and 4K results are qualitative only—no FID or GenEval at 2048 or 4K, so those claims are suggestive, not measured. The speed comparison also needs more baseline detail (exact DiT variant, sampling steps, VAE), and there's no code or data.\n\nNet: the method is useful and the 512 results are solid, but the paper overreaches in its conclusion. It deserves a serious referee who will ask for a real bidirectional scan ablation, quantitative high-res evaluation, and a corrected abstract. The core distillation story is strong enough to be worth engaging.","headline":"Useful distillation recipe with credible 512 results, but the headline claim about causal Mamba standing alone is not supported—the model's global context really comes from its 4 self-attention layers.","tokens_in":14318,"tokens_out":4574,"would_cite":true,"duration_ms":45577,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"By distilling a diffusion transformer layer-by-layer into a hybrid Mamba model, this paper claims a 0.7B model matches teacher-level text-to-image quality while gaining 1.5–2.1× speed at 2048 and 4K resolution.","keywords":["diffusion transformers","state space models","Mamba","knowledge distillation","high-resolution image generation","text-to-image synthesis","teacher forcing","hybrid architecture"],"falsifier":"Measure the trained student's effective receptive field by perturbing one patch of the noisy latent and measuring how the prediction for a distant patch changes, and compare against the teacher's attention map; if the student's influence decays sharply with distance while the teacher's does not, the model is not actually capturing long-range non-causal dependencies, and the GenEval match would be explained by local statistics rather than the claimed mechanism.","tokens_in":13267,"feed_emoji":"🖼️","tokens_out":7535,"duration_ms":69047,"temperature":0.7,"pith_summary":"The paper tries to show that a diffusion image-generation model built mostly from causal Mamba layers can be trained to match a transformer-based teacher by distilling the teacher layer by layer, rather than training the Mamba model from scratch. It claims the resulting 0.7B hybrid model, with 86% Mamba and 14% self-attention blocks, reaches teacher-level text-to-image quality (GenEval 0.485 vs teacher 0.481) while generating 2048×2048 images 1.5× faster and 4K images 2.1× faster. The deeper claim is that a sequential, causal state-space model can capture the non-causal global dependencies needed for coherent images, if the distillation feeds it the teacher's intermediate features as pseudo-ground truth. If this holds, it opens a cheaper path to high-resolution generation without quadratic attention cost.","feed_headline":"Hybrid Mamba matches diffusion transformer quality at 2x speed","feed_subtitle":"Layer-by-layer distillation lets a 0.7B Mamba model hit teacher-level GenEval while generating 4K images faster.","key_machinery":"The load-bearing mechanism is layer-level teacher forcing: for each Mamba block, the student receives the teacher's intermediate input $h_{\\theta'}^{(n)}$ as pseudo-ground-truth input and is trained to minimize the distance between its token-mixer output and the teacher's self-attention output at that same layer, $\\lVert MA_\\theta^{(n)}(h_{\\theta'}^{(n)}) - SA_{\\theta'}^{(n)}(h_{\\theta'}^{(n)}) \\rVert_2^2$, with the loss applied only at Mamba blocks. This converts a long causal chain into a set of per-layer imitation tasks, preventing early deviations from snowballing. The second mechanism is the bidirectional scanning scheme: width-first and height-first interleaved scans with shared Mamba weights whose outputs are combined by a linear projection, giving the causal 1D state-space model access to 2D context in both orientations.","core_discovery":"The paper's central claim is that a diffusion model whose backbone is 86% causal Mamba layers can be brought to teacher-level generation quality by distilling a pretrained non-causal diffusion transformer instead of training the Mamba model from scratch. The recipe has three stages. First, layer-level teacher forcing feeds each student Mamba block the intermediate features that the teacher's corresponding self-attention block receives, and trains the Mamba block to output that block's attention result, so errors from earlier layers do not compound. Second, knowledge distillation keeps all non-token-mixer weights frozen and trains the token mixers against a diffusion MSE loss, teacher pseudo-labels, and per-layer mixer outputs. Third, model adaptation replaces the positional encoding and VAE before two rounds of high-resolution fine-tuning to 1024×1024 and 2048×2048. The resulting 0.7B model scores GenEval 0.485 versus the teacher's 0.481, reaches 8.63 FID-30K on COCO versus 7.32 for the teacher, and speeds up sampling by 1.5× at 2048×2048 and 2.1× at 3840×2160.","pith_inferences":["If the 'Mamba is hard to train' problem is mainly cumulative error propagation rather than a representational limit, the same layerwise teacher-forcing recipe should transfer to other non-causal-to-causal distillations, such as video or audio diffusion.","A direct testable extension would be to apply T2MD to a larger or stronger teacher and measure whether the speed advantage grows with token count; the linear-complexity argument predicts the gap widens at higher resolutions.","The paper stops short of analyzing effective receptive fields; one would expect the student's bidirectional scans to approximate the teacher's global attention on natural-image statistics but to diverge on images with long-range periodic or relational structure.","The model-adaptation stage recovers within 100k steps after swapping text encoder, VAE, and positional encoding, suggesting the distilled Mamba backbone is a stable platform; swapping in new conditioning encoders could be a cheap way to test that generality."],"forward_implications":["A pretrained diffusion transformer can be converted into a mostly-Mamba diffusion model by training only the token mixers, yielding a GenEval score (0.485) that slightly exceeds the teacher (0.481).","High-resolution generation no longer requires quadratic self-attention: the hybrid model produces 2048×2048 images 1.5× faster and 3840×2160 images 2.1× faster than the DiT baseline on the same GPU.","Keeping a small number of attention blocks matters: removing them drops the overall GenEval score from 0.485 to 0.420, while the 86/14 hybrid retains global dependencies.","After model adaptation, the distilled model can generate zero-shot 4K images even though it was fine-tuned only up to 2048×2048.","The layerwise teacher forcing is the key ingredient: adding it raises the two-object GenEval subscore and the overall score, confirming that it prevents error accumulation across Mamba layers."],"supporting_citations":[{"why":"Supplies the pretrained diffusion transformer used as teacher and the quality upper bound the student is distilled toward.","marker":"[4]"},{"why":"Provides the Mamba 2 architecture and the structured state space duality rationale for training Mamba layers to mimic transformer layers.","marker":"[9]"},{"why":"Prior Mamba-based diffusion model used as a comparison baseline for image quality.","marker":"[11]"},{"why":"Defines the GenEval benchmark used in ablations and in the claim that the student reaches teacher-level text-to-image alignment.","marker":"[17]"},{"why":"Introduces the Mamba selective state space model that the student's token mixers are built from.","marker":"[19]"},{"why":"Supplies the higher-resolution VAE used during the model adaptation stage.","marker":"[47]"},{"why":"Empirical study showing Mamba models are harder to train, motivating the hybrid design and the distillation approach.","marker":"[60]"},{"why":"Provides the causal-to-causal Mamba initialization method used as the comparison baseline for the non-causal-to-causal setting.","marker":"[61]"}],"fun_headline_variants":["Distilled Mamba matches diffusion transformer at 2x speed","Teacher forcing turns Mamba into a high-res image generator","Mamba matches DiT quality after layer-wise distillation, faster","86% Mamba backbone hits teacher-level GenEval via distillation","Distillation pipeline: DiT to Mamba for 2048px text-to-image"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The training pipeline works only if the bidirectional interleaved scanning gives the causal Mamba layers enough global, non-causal 2D context to imitate the teacher's self-attention; if that scan cannot represent long-range dependencies, the quality match and the feasibility claim collapse even though training remains stable.","fun_headline_variants_meta":{"raw":{"variants":["Distilled Mamba matches diffusion transformer at 2x speed","Teacher forcing turns Mamba into a high-res image generator","Mamba matches DiT quality after layer-wise distillation, faster","86% Mamba backbone hits teacher-level GenEval via distillation","Distillation pipeline: DiT to Mamba for 2048px text-to-image"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000799,"raw_usage":{"total_tokens":3542,"prompt_tokens":1002,"completion_tokens":2540,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":618,"completion_tokens_details":{"reasoning_tokens":2449}},"tokens_in":618,"tokens_out":2540,"duration_ms":20594,"temperature":1.0,"reasoning_tokens":2449,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T18:39:48.797994+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Measure the trained student's effective receptive field by perturbing one patch of the noisy latent and measuring how the prediction for a distant patch changes, and compare against the teacher's attention map; if the student's influence decays sharply with distance while the teacher's does not, the model is not actually capturing long-range non-causal dependencies, and the GenEval match would be explained by local statistics rather than the claimed mechanism.","supporting_citations":[],"review_version":2}