REVIEW 4 major objections 6 minor 40 references
Brick-Diffusion: Generating Long Videos with Brick-to-Wall Denoising
T0 review · 4 major / 6 minor · reviewed 2026-08-10 · deepseek-v4-flash
Pith's one-line read Brick-Diffusion claims that shifting which frames are denoised together at each step lets a pre-trained short-video diffusion model produce long, consistent videos with no training.
desk verdict A simple, clearly explained training-free trick for long video generation that is plausibly useful, but the current evidence does not yet support the headline margin over FIFO-Diffusion. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The brick-to-wall denoising procedure: at each denoising step the long latent is sliced into segments of f frames (f = 16, the pre-trained model's native length), each segment is denoised independently, then the entire latent is shifted by a stride (set to 1 in the main experiments) and re-sliced for the next step. The offset at step t is offset_t = stride × (T − t) mod f, which cycles across the frame dimension, so a frame that was near the edge of one segment becomes interior in a later segment. A shorter first or last segment is handled by temporarily extending it to f frames, denoising, and keeping only the valid part. This shifting is what converts independent segment denoising from plain concatenation into a global communication mechanism.
What would settle it
Generate a 128-frame video of a static scene with a known fixed background and compare it with the same scene generated by the native 16-frame model; if the brick-to-wall video shows periodic flicker or content drift aligned with the 16-frame segment boundaries, the segment-wise approximation is not repairing boundary effects. A more direct check is to denoise the same overlapping frames under two different segment alignments at the same timestep and measure the latent difference, since large differences would mean the model's predictions are inconsistent across segmentations.
Extended reading notes
Core claim
On the paper's own terms, the central discovery is that shifting the segmentation of a long latent by a fixed stride between successive denoising steps — instead of denoising fixed, disjoint slices — is enough to keep a long video temporally consistent and visually sharp. Each f-frame segment is denoised independently by the pre-trained model for the current noise level, then the latent is re-sliced with a one-frame offset and denoised again; over many steps every frame is repeatedly co-denoised with many different sets of neighbours, which the paper likens to a staggered brick wall. This simple re-slicing communicates information between segments and avoids both the blurring caused by averaging overlapping windows and the training-inference gap of diagonal denoising. The paper reports that the method achieves the highest dynamic degree, aesthetic quality, and overall text-video consistency among the training-free baselines tested, while FreeNoise retains an edge in subject consistency.
Load-bearing premise
The method assumes that denoising f-frame segments independently at each timestep, with only a shifted regrouping between steps, approximates the joint denoising the pre-trained model would perform on the whole clip; if segment-wise processing shifts the latent distribution or leaves boundary artifacts that the shifting cannot repair, the quality claim fails.
Editorial extensions
If this is right
- A pre-trained short-video diffusion model can produce videos of arbitrary length with no fine-tuning, as long as the inference loop re-slices the latent with a non-zero stride.
- The stride can be varied (values 1, 3, 5, 7, and 9 all score comparably in the ablation), so the method is robust to the choice of shift.
- Because each segment is denoised independently, the procedure can be run in parallel, unlike sequential frame-by-frame or full temporal-attention methods.
- The main scores improve with a non-zero stride: average 81.70 at stride 1 versus 72.20 at stride 0 (concatenation), showing the shift is the active ingredient.
- The short first and last segments are handled by padding, so the method supports any output length, not only multiples of f.
Reading between the lines
- The same shifting-segment trick could be applied to other iterative generative models with a fixed context length; the stride acts as a cheap way to couple overlapping contexts.
- Combining brick-to-wall denoising with noise rescheduling of the kind FreeNoise uses might improve subject consistency further, since the paper shows FreeNoise's advantage lies there while brick-to-wall leads in dynamics and fidelity.
- A direct test of the approximation would be to compare co-denoised overlapping regions: at a given timestep, a frame denoised as part of segment A and the same frame denoised as part of segment B should agree; the paper does not report such an error measure, and this would pinpoint where the joint-denoiser approximation is weakest.
- Because the offset formula depends on the total timestep T, the sequence of segmentations is deterministic; a random or adaptive stride schedule might smooth out any residual periodic artifacts every f frames.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. Brick-Diffusion proposes a training-free inference procedure that generates long videos by slicing a latent into f-frame segments, denoising each segment independently with a pretrained short-video diffusion model, and shifting the slicing grid by a stride at every denoising step. The authors argue that this 'brick-to-wall' shifting lets frames from different segments communicate, and they evaluate the method with VideoCrafter2 on the VBench benchmark for 128-frame videos. In their quantitative results, the method attains an average score of 81.70, ahead of FIFO-Diffusion's 79.12, driven mainly by a large dynamic-degree advantage and a small text-consistency advantage, while FreeNoise remains best on subject consistency. A stride ablation shows that non-zero strides greatly improve over stride 0, with s=1 selected as the best.
Significance. Brick-Diffusion is a simple and potentially useful contribution: if its reported gains are reproducible, it offers a parallelizable, fine-tuning-free way to extend existing short-video diffusion models to longer videos, and the paper makes this claim falsifiable through the stride ablation. The algorithm is described clearly enough to reimplement, and the evaluation against four training-free baselines on a standard benchmark is a strength. However, the significance is currently limited by the lack of statistical robustness in the headline comparison, the selection of the stride on the same test set used for reporting, and the absence of direct evidence that the independent-segment approximation in Eq. (6) preserves the joint statistics required for temporal consistency. These issues are addressable and do not invalidate the core idea, but they preclude acceptance of the superiority claims as stated.
major comments (4)
- [Table I and Section III-D] The headline comparison reports a single numeric value per metric and method with no error bars, confidence intervals, or number of independent runs. The paper states that 825 videos are generated per method, but it does not say whether these come from one seed per prompt or from repeated draws; if they are single draws, the reported 2.58-point average margin over FIFO-Diffusion and the 9.16-point dynamic-degree margin could change substantially under a different initial noise. Since FIFO-Diffusion actually leads on subject consistency (93.91 vs 93.00), the 'superiority' claim needs a statistical basis. Please report means and standard errors over at least three seeds, or otherwise justify why a single run is representative.
- [Table II and Section III-C] The stride s=1 is selected as the configuration with the highest average score on the same VBench prompts used for the final comparison against baselines. This is an instance of tuning on the test set, and it makes the reported advantage over the baselines optimistic because the baselines receive no comparably tuned hyperparameter. The authors should pre-register the stride, choose it on a separate validation set, or report all stride values alongside the baseline comparison and discuss the selection bias. The near-flat results for s=3,5,7,9 suggest robustness, but the selection protocol must still be stated.
- [Section II-B, Eq. (6)] The core of the method is the assumption that denoising f-frame segments independently at each timestep, with the grid shifted between steps, is a valid approximation of the joint denoising the pretrained model would perform on the whole clip. The paper does not provide direct evidence for this assumption: the stride ablation in Table II only shows that non-zero stride beats concatenation, while VBench's aggregate metrics (DINOv2-based subject consistency, RAFT-based dynamic degree, aesthetic score, InternVid text-video similarity) are unlikely to be sensitive to localized flicker or discontinuity exactly at the old segment boundaries. Please add a diagnostic that isolates boundary artifacts, for example comparing per-frame metrics near versus far from previous slice boundaries, testing a control in which segment order is randomly permuted between steps, or comparing the final latent against a jointly denoised short-clip reference. Without such evidence, the 'communication between segments' mechanism remains a heuristic.
- [Section III-D, Fig. 3] The qualitative evaluation consists of a single text prompt ('a cute raccoon playing guitar in a boat on the ocean'). A single example cannot support the claim of consistently high-fidelity output, especially when the advertised differences are temporal (flicker, object deformation, disappearance). Please include a supplementary video set or a small perceptual study (e.g., pairwise preference among methods over several prompts), and show frames from at least a few diverse prompts.
minor comments (6)
- [References] Reference [30] contains a typo: 'ane et al.' should be 'and et al.' or the correct author list; please check all references for consistency.
- [Section II-B] The symbol 'offsett' is used as both a word and a variable name; please denote the offset at timestep t by a consistent symbol, such as o_t, and use it consistently in Eqs. (3)-(8).
- [Section III-C] The implementation details do not state the number of DDIM steps, the classifier-free guidance scale, or whether the same random seed was used across all methods; these choices affect relative performance and should be reported.
- [Section III-D] The claim about 'highest scores in dynamic degree, aesthetic quality, and overall video-text consistency' is faithful to Table I, but the text should also note that FreeNoise attains the highest subject consistency (95.09) and FIFO-Diffusion is higher than the proposed method on subject consistency as well; this nuance is important for a balanced reading.
- [Section II-B] The padding of the initial latent to F+2f frames and the cropping of the middle F frames of the output is described in one sentence; please elaborate on how this interacts with the offset schedule and how the first/last short segments are handled in detail.
- [Section I] The word 'parallized' appears in the contribution list; please correct to 'parallelized'.
Circularity Check
No derivation-level circularity; the only in-sample element is stride selection on the same VBench metric later reported as the headline score.
-
fitted input called prediction
[Section III-D-c (Ablation Study), Table II, and Table I]
"The results for s = 1, s = 3, s = 5, s = 7, and s = 9 vary across the four metrics, but overall, the average scores are relatively close. We select a stride of s = 1 for our final results, as it achieves the highest average score."
The stride is chosen by evaluating the method on the VBench prompts and comparing average scores in Table II, and then the same VBench average for s=1 (81.70) is reported in Table I as the method's headline result. Thus the reported 'prediction' is the selected maximum of the same evaluation metric used for selection, not an out-of-sample test of a predetermined hyperparameter. This is a mild form of in-sample selection rather than a forced construction. It is not load-bearing because every non-zero stride (s=3,5,7,9) also yields an average around 81.1, still above FIFO-Diffusion's 79.12.
full rationale
Brick-Diffusion is a training-free inference heuristic: no model parameter is learned from data and no first-principles prediction is derived from fitted constants. The core operation in Eq. (6) simply applies the pretrained 16-frame denoiser to shifted segments, and the claimed 'communication' between frames is an empirically grounded heuristic rather than a circular reduction. The only evaluation-related choice is the stride, which is tuned on the same VBench benchmark later used for the final comparison; because the method's advantage over baselines is stable across all tested non-zero strides, this selection does not force the reported outcome. There is no load-bearing self-citation chain or imported uniqueness theorem. The main vulnerabilities of the paper are empirical, such as unmeasured boundary artifacts and the insensitivity of aggregate VBench scores to localized flicker, not circular reasoning.
Assumptions & free parameters
free parameters (1)
- denoising stride s =
1
assumptions (3)
- domain assumption Each 16-frame segment's marginal latent distribution matches VideoCrafter2's training distribution, so denoising segments independently with Equation (6) is approximately valid.
- domain assumption VBench normalized metrics are a valid proxy for video quality and motion fidelity.
- standard math The offset recurrence in Equation (4) correctly cycles segment boundaries through all frame positions, enabling full communication between frames.
Cite this review
Pith. "Pith review of Brick-Diffusion: Generating Long Videos with Brick-to-Wall Denoising." pith.science (2026). https://pith.science/paper/RKS23OAB
@misc{pith2026250102741,
author = {Pith},
title = {Pith review of: Brick-Diffusion: Generating Long Videos with Brick-to-Wall Denoising},
year = {2026},
howpublished = {\url{https://pith.science/paper/RKS23OAB}},
note = {Machine review of arXiv:2501.02741}
}
read the original abstract
Recent advances in diffusion models have greatly improved text-driven video generation. However, training models for long video generation demands significant computational power and extensive data, leading most video diffusion models to be limited to a small number of frames. Existing training-free methods that attempt to generate long videos using pre-trained short video diffusion models often struggle with issues such as insufficient motion dynamics and degraded video fidelity. In this paper, we present Brick-Diffusion, a novel, training-free approach capable of generating long videos of arbitrary length. Our method introduces a brick-to-wall denoising strategy, where the latent is denoised in segments, with a stride applied in subsequent iterations. This process mimics the construction of a staggered brick wall, where each brick represents a denoised segment, enabling communication between frames and improving overall video quality. Through quantitative and qualitative evaluations, we demonstrate that Brick-Diffusion outperforms existing baseline methods in generating high-fidelity videos.
Figures
Reference graph
Works this paper leans on
-
[1]
Deep unsupervised learning using nonequilibrium thermodynamics,
J. Sohl-Dickstein, E. Weiss, N. Maheswaranathan, and S. Ganguli, “Deep unsupervised learning using nonequilibrium thermodynamics,” in ICML, 2015
2015
-
[2]
Denoising diffusion probabilistic models,
J. Ho, A. Jain, and P. Abbeel, “Denoising diffusion probabilistic models,” in NeurIPS, 2020
2020
-
[3]
Score-based generative modeling through stochastic differential equations,
Y . Song, J. Sohl-Dickstein, D. P. Kingma, and et al., “Score-based generative modeling through stochastic differential equations,” in ICLR, 2021
work page 2021
-
[4]
High- resolution image synthesis with latent diffusion models,
R. Rombach, A. Blattmann, D. Lorenz, P. Esser, and B. Ommer, “High- resolution image synthesis with latent diffusion models,” in CVPR, 2022
2022
-
[5]
GLIDE: towards photorealistic image generation and editing with text-guided diffusion models,
A. Q. Nichol, P. Dhariwal, A. Ramesh, and et al., “GLIDE: towards photorealistic image generation and editing with text-guided diffusion models,” in ICML, 2022
work page 2022
-
[6]
Cascaded Diffusion Models for High Fidelity Image Generation,
J. Ho, C. Saharia, W. Chan, D. J. Fleet, M. Norouzi, and T. Salimans, “Cascaded Diffusion Models for High Fidelity Image Generation,” JMLR, 2022
work page 2022
-
[7]
Scalable diffusion models with transformers,
W. Peebles and S. Xie, “Scalable diffusion models with transformers,” in ICCV, 2023
2023
-
[8]
SDXL: improving latent diffusion models for high-resolution image synthesis,
D. Podell, Z. English, K. Lacey, and et al., “SDXL: improving latent diffusion models for high-resolution image synthesis,” in ICLR, 2024
work page 2024
Show all 40 references
-
[9]
PixArt- α: fast training of diffusion transformer for photorealistic text-to-image synthesis,
J. Chen, J. YU, C. GE, and et al., “PixArt- α: fast training of diffusion transformer for photorealistic text-to-image synthesis,” in ICLR, 2024
2024
-
[10]
Videocrafter2: overcoming data limitations for high-quality video diffusion models,
H. Chen, Y . Zhang, X. Cun, and et al., “Videocrafter2: overcoming data limitations for high-quality video diffusion models,” in CVPR, 2024
2024
-
[11]
AnimateDiff: animate your personalized text-to-image diffusion models without specific tuning,
Y . Guo, C. Yang, A. Rao, and et al., “AnimateDiff: animate your personalized text-to-image diffusion models without specific tuning,” in ICLR, 2024
2024
-
[12]
Stable video dif- fusion: scaling latent video diffusion models to large datasets,
A. Blattmann, T. Dockhorn, S. Kulal, and et al., “Stable video dif- fusion: scaling latent video diffusion models to large datasets,” 2023, arXiv:2311.15127. [Online]. Available: https://arxiv.org/abs/2311.15127
2023 arXiv
-
[13]
ModelScope text-to- video technical report,
J. Wang, H. Yuan, D. Chen, and et al., “ModelScope text-to- video technical report,” 2023, arXiv:2308.06571. [Online]. Available: https://arxiv.org/abs/2308.06571
2023 arXiv
-
[14]
Mag- icVideo: efficient video generation with latent diffusion models,
D. Zhou, W. Wang, H. Yan, W. Lv, Y . Zhu, and J. Feng, “Mag- icVideo: efficient video generation with latent diffusion models,” 2023, arXiv:2211.11018. [Online]. Available: https://arxiv.org/abs/2211.11018
2023 arXiv
-
[15]
Video diffusion models,
J. Ho, T. Salimans, A. Gritsenko, W. Chan, M. Norouzi, and D. J. Fleet, “Video diffusion models,” in NeurIPS, 2022
2022
-
[16]
Align your latents: high-resolution video synthesis with latent diffusion models,
A. Blattmann, R. Rombach, H. Ling, and et al., “Align your latents: high-resolution video synthesis with latent diffusion models,” in CVPR, 2023
2023
-
[17]
MCVD: masked condi- tional video diffusion for prediction, generation, and interpolation,
V . V oleti, A. Jolicoeur-Martineau, and C. Pal, “MCVD: masked condi- tional video diffusion for prediction, generation, and interpolation,” in NeurIPS, 2022
2022
-
[18]
Latent Video Diffusion Models for High-Fidelity Long Video Generation,
Y . He, T. Yang, Y . Zhang, Y . Shan, and Q. Chen, “Latent Video Diffusion Models for High-Fidelity Long Video Generation,” 2023, arXiv:2211.13221. [Online]. Available: https://arxiv.org/abs/2211.13221
2023 arXiv
-
[19]
Latte: latent diffusion transformer for video generation,
X. Ma, Y . Wang, G. Jia, and et al., “Latte: latent diffusion transformer for video generation,” 2024, arXiv:2401.03048. [Online]. Available: https://arxiv.org/abs/2401.03048
2024 arXiv
-
[20]
Make-A-Video: text-to-video generation without text-video data,
U. Singer, A. Polyak, T. Hayes, and et al., “Make-A-Video: text-to-video generation without text-video data,” in ICLR, 2023
2023
-
[21]
Preserve your own correlation: a noise prior for video diffusion models,
S. Ge, S. Nah, G. Liu, and et al., “Preserve your own correlation: a noise prior for video diffusion models,” in ICCV, 2023
2023
-
[22]
StyleGAN-V: a con- tinuous video generator with the price, image quality and perks of StyleGAN2,
I. Skorokhodov, S. Tulyakov, and M. Elhoseiny, “StyleGAN-V: a con- tinuous video generator with the price, image quality and perks of StyleGAN2,” in CVPR, 2022
2022
-
[23]
Generating long videos of dynamic scenes,
T. Brooks, J. Hellsten, M. Aittala, and et al., “Generating long videos of dynamic scenes,” in NeurIPS, 2022
2022
-
[24]
Long video generation with time- agnostic VQGAN and time-sensitive transformer,
S. Ge, T. Hayes, H. Yang, and et al., “Long video generation with time- agnostic VQGAN and time-sensitive transformer,” in ECCV, 2022
2022
-
[25]
Flexible diffusion modeling of long videos,
W. Harvey, S. Naderiparizi, V . Masrani, C. Weilbach, and F. Wood, “Flexible diffusion modeling of long videos,” in NeurIPS, 2022
2022
-
[26]
NUW A-XL: diffusion over diffusion for extremely long video generation,
S. Yin, C. Wu, H. Yang, and et al., “NUW A-XL: diffusion over diffusion for extremely long video generation,” 2023, arXiv:2303.12346. [Online]. Available: https://arxiv.org/abs/2303.12346
2023 arXiv
-
[27]
Phenaki: variable length video generation from open domain textual descriptions,
R. Villegas, M. Babaeizadeh, P. J. Kindermans, and et al., “Phenaki: variable length video generation from open domain textual descriptions,” in ICLR, 2023
2023
-
[28]
SEINE: short-to-long video diffusion model for generative transition and prediction,
X. Chen, Y . Wang, L. Zhang, and et al., “SEINE: short-to-long video diffusion model for generative transition and prediction,” in ICLR, 2024
2024
-
[29]
Gen-L-Video: multi-text to long video generation via temporal co-denoising,
F. Wang, W. Chen, G. Song, and et al., “Gen-L-Video: multi-text to long video generation via temporal co-denoising,” 2023, arXiv.2305.18264. [Online]. Available: https://arxiv.org/abs/2305.18264
2023 arXiv
-
[30]
FreeNoise: tuning-free longer video diffusion via noise rescheduling,
H. Qiu, M. Xia, Y . Zhang, ane et al., “FreeNoise: tuning-free longer video diffusion via noise rescheduling,” in ICLR, 2024
2024
-
[31]
FIFO-Diffusion: generating infinite videos from text without training,
J. Kim, J. Kang, J. Choi, and B. Han, “FIFO-Diffusion: generating infinite videos from text without training,” 2024, arXiv:2405.11473. [Online]. Available: https://arxiv.org/abs/2405.11473
2024 arXiv
-
[32]
Denoising diffusion implicit models,
J. Song, C. Meng, and S. Ermon, “Denoising diffusion implicit models,” in ICLR, 2021
2021
-
[33]
DPM-Solver: a fast ODE solver for diffusion probabilistic model sampling in around 10 steps,
C. Lu, Y . Zhou, F. Bao, J. Chen, C. LI, and J. Zhu, “DPM-Solver: a fast ODE solver for diffusion probabilistic model sampling in around 10 steps,” in NeurIPS, 2022
2022
-
[34]
Elucidating the design space of diffusion-based generative models,
T. Karras, M. Aittala, T. Aila, and S. Laine, “Elucidating the design space of diffusion-based generative models,” in NeurIPS, 2022
2022
-
[35]
Pseudo numerical methods for diffusion models on manifolds,
L. Liu, Y . Ren, Z. Lin, and Z. Zhao, “Pseudo numerical methods for diffusion models on manifolds,” in ICLR, 2022
2022
-
[36]
VBench: comprehensive benchmark suite for video generative models,
Z. Huang, Y . He, J. Yu, and et al., “VBench: comprehensive benchmark suite for video generative models,” in CVPR, 2024
2024
-
[37]
DINOv2: learning robust visual features without supervision,
M. Oquab, T. Darcet, T. Moutakanni, and et al., “DINOv2: learning robust visual features without supervision,” TMLR, 2024
2024
-
[38]
RAFT: recurrent all-pairs field transforms for optical flow,
Z. Teed and J. Deng, “RAFT: recurrent all-pairs field transforms for optical flow,” in ECCV, 2020
2020
-
[39]
Aesthetic predictor,
LAION-AI, “Aesthetic predictor,” 2022, gitHub repository. [Online]. Available: https://github.com/LAION-AI/aesthetic-predictor
2022
-
[40]
InternVid: a large-scale video-text dataset for multimodal understanding and generation,
Y . Wang, Y . He, Y . Li, and et al., “InternVid: a large-scale video-text dataset for multimodal understanding and generation,” in ICLR, 2024
2024
Reviewed August 10, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.