Pith. sign in

REVIEW 3 major objections 5 minor 1 cited by

GenMAC: Compositional Text-to-Video Generation with Multi-Agent Collaboration

T0 review · 3 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Agent loop tops all seven compositional video benchmarks

desk verdict A useful self-correction framework for compositional T2V, but the SOTA claim rests on unmatched baseline numbers and needs a controlled rerun or softer language. read the letter →

arxiv 2412.04440 v1 pith:MKKSBJRZ submitted 2024-12-05 cs.CV

Kaiyi Huang , Yukun Huang , Xuefei Ning , Zinan Lin , Yu Wang , Xihui Liu This is my paper · ORCID
classification cs.CV
keywords compositionaltext-to-videogenerationmulti-agentcollaborationMLLMagentsiterativerefinementself-routingT2V-CompBenchlayoutguidance
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper proposes GenMAC, a framework that wraps an off-the-shelf text-to-video model in a loop of design, generation, and redesign, so that complex prompts with multiple objects, attributes, spatial relations, and motions are checked and corrected iteratively. The central claim is that decomposing the redesign step into four specialized MLLM agents—verification, suggestion, correction, and output structuring—plus a self-routing choice among correction experts, makes video outputs align with compositional prompts much better than single-pass models. On the seven T2V-CompBench aspects, GenMAC reports the best score in every category, with the largest margin in generative numeracy. If the comparison is fair, this would mean that compositional video fidelity can be improved substantially without retraining the backbone generator.

What carries the argument

The load-bearing mechanism is the Design-Generation-Redesign loop with a decomposed redesign stage. The redesign stage is split into four sequentially executed MLLM agents (verification, suggestion, correction, and output structuring), and a self-routing mechanism selects one of three correction experts (consistency, temporal dynamics, or spatial dynamics). The generation stage injects the layout into the diffusion model by gradient-guiding cross-attention maps toward bounding boxes, with a guidance scale that the redesign stage adjusts. This machinery converts an open-ended alignment problem into a series of bounded sub-tasks and a structured output that the generator can consume.

What would settle it

Re-run every baseline in Table 1 under GenMAC's exact protocol (65 frames, 512x512, same prompt set, same metric versions, and same judge settings) and check whether GenMAC still beats each one; if regenerating the baseline scores changes the 0.5166 versus 0.2928 numeracy margin, the state-of-the-art claim does not stand.

Watch

Extended reading notes

Core claim

GenMAC's claim is that a complex text-to-video alignment task can be handled by orchestration rather than by stronger generation alone. The workflow first uses an LLM to design per-frame bounding boxes and prompts, generation uses an off-the-shelf video diffusion model with attention-map layout guidance, and a redesign stage then verifies the video against the prompt on existence, quantity, attributes, and relationships, suggests changes, selects among correction agents specialized for consistency, temporal dynamics, or spatial dynamics, and emits structured layouts, revised prompts, and guidance scales for the next iteration. The loop runs until the verification agent is satisfied. The paper reports that this Design-Generation-Redesign loop achieves the best scores on all seven T2V-CompBench categories, for example generative numeracy of 0.5166 against 0.2928 for the second-best model, and the ablations attribute the gains to iterative refinement, role specialization, and self-routing.

Load-bearing premise

The claim that GenMAC is state-of-the-art assumes the baseline scores copied from T2V-CompBench are directly comparable to GenMAC's own runs, even though the baselines were evaluated separately and may have used different frame counts, resolutions, or settings.

Editorial extensions

If this is right

  • If GenMAC's reported results hold, compositional text-to-video quality can be improved by orchestration rather than by scaling or retraining the generator.
  • The self-routing correction scheme suggests that verification and correction for generative models are best handled by role-specialized agents instead of a single generalist.
  • Iterative refinement with structured layout, guidance scale, and prompt revision gives the largest gains on object count, spatial relations, and motion binding, while dynamic attribute changes remain the hardest aspect.
  • The framework should transfer to newer backbones and MLLMs whenever their base capabilities improve, because it only wraps generation and correction around them.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A testable extension is to apply the same agent decomposition to text-to-image generation, where layout and correction machinery are simpler and the benchmark is already available.
  • Because the agents rely on GPT-4o, the reported gains are capped by the verifier's perceptual accuracy; a weaker or open-source MLLM would likely shrink the margins.
  • If the baseline scores sourced from T2V-CompBench were produced under different resolutions or frame counts, the headline margins could shift; re-running the comparison under one identical protocol would settle this.
  • Accepted iterations of the loop are near-correct clip-prompt pairs, which could be collected as training data to fine-tune the backbone generator without human annotation.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes GenMAC, an iterative multi-agent framework for compositional text-to-video generation. The workflow consists of a Design stage, a Generation stage using an off-the-shelf video diffusion model (VideoCrafter2), and a Redesign stage in which four sequentially executed MLLM-based agents verify alignment, suggest corrections, apply corrections, and structure the output for the next generation iteration. A self-routing mechanism adaptively selects among specialized correction agents for consistency, temporal dynamics, and spatial dynamics. The authors evaluate on T2V-CompBench and report the best scores in all seven compositional aspects, claiming state-of-the-art performance, and provide ablations showing the contribution of the multi-stage pipeline, role specialization, iterative refinement, and self-routing.

Significance. If the state-of-the-art claim were firmly established, GenMAC would be a notable contribution: it is among the first frameworks to apply heterogeneous multi-agent collaboration with iterative verification-and-redesign to compositional text-to-video generation, without retraining the backbone. The internal ablations in Table 2 consistently support the value of each design component (Design stage, Redesign stage, iterative loop, task decomposition, and self-routing), and the main benchmark uses external metrics (Grid-LLaVA, D-LLaVA, G-Dino, DOT), so the central comparison is not circular. The principal weakness is that the quantitative SOTA claim rests on importing baseline scores from T2V-CompBench [46] without a shared evaluation protocol, and all numbers are single runs without variance reporting.

major comments (3)
  1. [Table 1 and Section 4.1] The central state-of-the-art claim depends on comparing GenMAC's scores against baseline scores that are not recomputed under GenMAC's settings. The Table 1 caption states 'The baseline data are sourced from [46]', while GenMAC is evaluated at 65 frames, 512x512, with VideoCrafter2 and GPT-4o. Because the baselines were evaluated separately, differences in resolution, frame count, prompt templates, or metric implementations (especially for detection-based G-Dino and DOT scores) can change the numeric values. The reported margin in generative numeracy (0.5166 vs 0.2928, about 76% relative) may therefore overstate the true improvement. The authors should either re-run all baselines under a common protocol or explicitly report the protocol differences and provide sensitivity analysis (e.g., how scores vary with frame count and resolution).
  2. [Section 4.2 and Table 1] Every reported number in Table 1 appears to come from a single run, with no error bars, no seed variations, and no statistical significance tests. Given the stochasticity of video diffusion models and the LLM/detector-based metrics, the observed differences between GenMAC and the second-best baseline could be within run-to-run noise, particularly for the smaller margins such as dynamic attribute binding (5.13%) and object interactions (4.43%). Reporting multiple seeds with means and standard deviations, or at least per-prompt score distributions, is required to support the SOTA claim.
  3. [Section 4.4 and Figure 7] The 'cumulative corrected ratio' in Figure 7 counts a prompt as corrected when the paper's own verification agent judges the video aligned and the loop exits. This is a circular success signal for measuring refinement progress, because the judge is itself part of the system being evaluated. Additionally, the stopping rule for the iterative loop is not formally defined; the paper mentions a maximum of nine iterations in Figure 7 but does not describe the termination criterion in the text. To substantiate the iterative-refinement analysis, the authors should track an external metric at each iteration and specify the stopping condition independently of the verification agent's judgment.
minor comments (5)
  1. [Section 1] There are several typographical and grammatical errors, e.g., 'and and tet prompts' near the Generation stage description, and 'an iterative process is composed of three stages' should be 'an iterative process composed of three stages'.
  2. [Figure 4 caption] The word 'comparision' should be 'comparison'.
  3. [Section 3.2] The labels '/tasks', '/edit', '/tools', and '/user-check' appear as stray annotations in the agent descriptions; these should be removed or clearly explained if they are meant to indicate tool calls.
  4. [Appendix B.1] The guidance scale initialization (1.0) and increment (0.05) are only given in the appendix but are important for reproducing the method; they should also appear in the main-text implementation details.
  5. [References] Several references list incorrect final page numbers, such as [8], [29], and [66] ending in '1, 2'; these should be corrected.

Circularity Check

1 steps flagged · score 4.0 of 10

Partial circularity in the self-reported corrected-ratio analysis (Figure 7); the central SOTA comparison uses external metrics but carries a same-author benchmark comparability risk.

  1. self definitional [Section 4.4, Figure 7 (Analysis on Iterative Generation)]
    "We calculate the cumulative corrected ratio within each subset at every iteration, which is the ratio of prompts that have completed the refinement and exited the G ENMAC loop to the total size of the subset."

    The 'corrected' status is decided by GenMAC's own verification agent: a prompt exits the loop only when that agent reports alignment. Therefore the cumulative corrected ratio measures the verification agent's self-approval rate, not an externally judged correction. The paper uses the rising ratio to 'demonstrat[e] the necessity of iterative refinement', but the success signal is the same subsystem whose corrections are being counted, so the improvement is partly by construction. This does not contaminate Table 1/2, which use external metrics (Grid-LLaVA, D-LLaVA, G-Dino, DOT).

full rationale

The main SOTA claim in Table 1 rests on external detector/LLM metrics and newly measured GenMAC scores, so it is not circular by construction. The ablations in Table 2 are also measured with those external metrics. The one genuinely self-referential analysis is Figure 7's corrected ratio, where 'corrected' is defined by the system's own exit condition (verification-agent agreement), making the reported improvement partly a measure of self-consistency. The benchmark [46] is co-authored by GenMAC authors and baseline scores are imported rather than re-run; this is a comparability/rigor risk, not circularity under the required definition, because no result is derived from the citation by construction. Overall, partial circularity is confined to an auxiliary analysis, not the central SOTA claim.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The central result depends on hand-set guidance hyperparameters, a benchmark whose baseline numbers are imported from a paper with overlapping authors, and the reliability of GPT-4o as both verifier and corrector. The framework itself does not introduce new physical or mathematical entities.

free parameters (3)
  • guidance_scale_init = 1.0
    Hand-set in Appendix B.1; initial scale for bounding-box guidance in the diffusion process.
  • guidance_scale_increment = 0.05
    Hand-set in Appendix B.1; how much the Redesign stage increases emphasis per iteration.
  • max_iterations_or_stopping_criterion = up to 9 (Figure 7); exit condition not specified
    The loop exits when the verification agent reports alignment; this threshold is not quantified, and it defines the cumulative corrected ratio.
assumptions (4)
  • domain assumption The baseline scores from T2V-CompBench [46] are directly comparable to GenMAC's scores despite different generation settings.
    Invoked implicitly in Table 1; the caption states baseline data are sourced from [46], and the paper does not re-run baselines under its own protocol.
  • domain assumption The LVD attention-guidance energy function (Eq. A1) effectively enforces bounding box layouts in VideoCrafter2.
    Adopted from LVD [29] in Section A; the paper does not verify layout adherence independently.
  • domain assumption GPT-4o's video understanding is reliable enough to verify text-video alignment and produce useful corrections.
    Section 3.2 assumes the agents can detect misalignments; Section C admits the method fails when GPT-4o exceeds its capability.
  • domain assumption The verification agent's decision to exit the loop indicates genuine correction.
    Figure 7's cumulative corrected ratio is based on the verification agent's own judgment, not external metrics.

how reviews work

0 comments
Cite this review

Pith. "Pith review of GenMAC: Compositional Text-to-Video Generation with Multi-Agent Collaboration." pith.science (2026). https://pith.science/paper/MKKSBJRZ

@misc{pith2026241204440,
  author       = {Pith},
  title        = {Pith review of: GenMAC: Compositional Text-to-Video Generation with Multi-Agent Collaboration},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/MKKSBJRZ}},
  note         = {Machine review of arXiv:2412.04440}
}
read the original abstract

Text-to-video generation models have shown significant progress in the recent years. However, they still struggle with generating complex dynamic scenes based on compositional text prompts, such as attribute binding for multiple objects, temporal dynamics associated with different objects, and interactions between objects. Our key motivation is that complex tasks can be decomposed into simpler ones, each handled by a role-specialized MLLM agent. Multiple agents can collaborate together to achieve collective intelligence for complex goals. We propose GenMAC, an iterative, multi-agent framework that enables compositional text-to-video generation. The collaborative workflow includes three stages: Design, Generation, and Redesign, with an iterative loop between the Generation and Redesign stages to progressively verify and refine the generated videos. The Redesign stage is the most challenging stage that aims to verify the generated videos, suggest corrections, and redesign the text prompts, frame-wise layouts, and guidance scales for the next iteration of generation. To avoid hallucination of a single MLLM agent, we decompose this stage to four sequentially-executed MLLM-based agents: verification agent, suggestion agent, correction agent, and output structuring agent. Furthermore, to tackle diverse scenarios of compositional text-to-video generation, we design a self-routing mechanism to adaptively select the proper correction agent from a collection of correction agents each specialized for one scenario. Extensive experiments demonstrate the effectiveness of GenMAC, achieving state-of-the art performance in compositional text-to-video generation.

Figures

Figures reproduced from arXiv: 2412.04440 by the authors.

Figure 1
Figure 1. The first row illustrates our multi-agent collaboration approach, showcasing the collaborative workflow, task decomposition in [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Framework of GENMAC. Collaborative workflow includes three stages with an iterative loop: DESIGN, GENERATION, and REDESIGN (Section 3.1). Task decomposition decomposes the redesign stage into four sub-tasks, handled by four agents: verification agent, suggestion agent, correction agent, and output structuring agent (Section 3.2). Self-routing mechanism allows for adaptive selection of suitable correction agent to ad… view at source ↗
Figure 3
Figure 3. Illustration of Task Decomposition for the REDESIGN stage (Section 3.2). The diagram illustrates the allocation of roles: verification agent, suggestion agent, correction agent, and output structuring agent within a sequential task breakdown, highlighting the clear responsibilities of each agent. out and guidance scale predicted from the DESIGN stage. Specifically, we employ an off-the-shelf text-to-video gen￾eratio… view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: Qualitative Comparison. Our proposed GENMAC generates videos that accurately adhere to complex compositional scenarios, demonstrating a clear advantage in handling such requirements in comparision with SOTA text-to-video models. (7.27%), consistent attribute binding (6…
Figure 5
Figure 5. Figure 5: Qualitative Results. Our proposed GENMAC generates videos that highly aligned with complex compositional prompts, including attribute binding, multiple objects, quantity, and dynamic motion binding. VideoTetris [51] [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 6
Figure 6. Figure 6: Visualization of the iterative refinement process in our multi-agent framework, demonstrating iterations enhance scene accuracy by progressively aligning video content with composi￾tional prompts. 1 2 3 4 5 6 7 8 9 Iteration Count 0.3 0.4 0.5 0.6 0.7 0.8 0.9 Cumulative…
Figure 7
Figure 7. Figure 7: Cumulative Corrected Ratio. For each subset in T2V [PITH_FULL_IMAGE:figures/full_fig_p008_7.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. AgentHOI: Multi-Agent Reasoning for Human-Object-Interaction Video Generation via Implicit Representation Alignment

    cs.CV 2026-07 conditional novelty 6.0 of 10

    AgentHOI generates human-object interaction videos from text plus one human image and one object image, using multi-agent action planning and implicit text-to-motion feature alignment inside a video diffusion model.

Reference graph

Works this paper leans on

77 extracted references · 43 canonical work pages · cited by 1 Pith paper

  1. [46]

    T2v-compbench: A comprehen- sive benchmark for compositional text-to-video generation,

    Kaiyue Sun, Kaiyi Huang, Xian Liu, Yue Wu, Zihan Xu, Zhenguo Li, and Xihui Liu. T2v-compbench: A comprehen- sive benchmark for compositional text-to-video generation,

  2. [1]

    https : / / huggingface

    ZeroScope. https : / / huggingface . co / cerspense / zeroscope _ v2 _ 576w/, 2023. Ac- cessed: 2024-11-14. 5, 6

    work page 2023

  3. [2]

    https://pika.art/, 2023

    Pika Art. https://pika.art/, 2023. Accessed: 2024- 11-14. 5, 6

    work page 2023

  4. [3]

    Stable video diffusion: Scaling latent video diffusion models to large datasets

    Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Dominik Lorenz, Yam Levi, Zion English, Vikram V oleti, Adam Letts, et al. Stable video diffusion: Scaling latent video diffusion models to large datasets. arXiv preprint arXiv:2311.15127, 2023. 2

    arXiv 2023

  5. [4]

    Align your latents: High-resolution video synthesis with la- tent diffusion models

    Andreas Blattmann, Robin Rombach, Huan Ling, Tim Dock- horn, Seung Wook Kim, Sanja Fidler, and Karsten Kreis. Align your latents: High-resolution video synthesis with la- tent diffusion models. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pages 22563–22575, 2023. 2

    2023

  6. [5]

    Maskgit: Masked generative image transformer

    Huiwen Chang, Han Zhang, Lu Jiang, Ce Liu, and William T Freeman. Maskgit: Masked generative image transformer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11315–11325, 2022. 2

    2022

  7. [6]

    Muse: Text-to-image generation via masked generative transform- ers

    Huiwen Chang, Han Zhang, Jarred Barber, AJ Maschinot, Jose Lezama, Lu Jiang, Ming-Hsuan Yang, Kevin Mur- phy, William T Freeman, Michael Rubinstein, et al. Muse: Text-to-image generation via masked generative transform- ers. arXiv preprint arXiv:2301.00704, 2023. 2

    arXiv 2023

  8. [7]

    Attend-and-excite: Attention-based se- mantic guidance for text-to-image diffusion models

    Hila Chefer, Yuval Alaluf, Yael Vinker, Lior Wolf, and Daniel Cohen-Or. Attend-and-excite: Attention-based se- mantic guidance for text-to-image diffusion models. InACM Trans. Graph., 2023. 2

    work page 2023

Show all 77 references
  1. [8]

    Videocrafter2: Overcoming data limitations for high-quality video diffusion models

    Haoxin Chen, Yong Zhang, Xiaodong Cun, Menghan Xia, Xintao Wang, Chao Weng, and Ying Shan. Videocrafter2: Overcoming data limitations for high-quality video diffusion models. arXiv preprint arXiv:2401.09047, 2024. 5, 6, 7, 1, 2

    2024 arXiv

  2. [9]

    Training- free layout control with cross-attention guidance

    Minghao Chen, Iro Laina, and Andrea Vedaldi. Training- free layout control with cross-attention guidance. In WACV,

  3. [10]

    Reason out your layout: Evoking the layout master from large lan- guage models for text-to-image synthesis

    Xiaohui Chen, Yongfei Liu, Yingxiang Yang, Jianbo Yuan, Quanzeng You, Li-Ping Liu, and Hongxia Yang. Reason out your layout: Evoking the layout master from large lan- guage models for text-to-image synthesis. arXiv preprint arXiv:2311.17126, 2023. 2

    2023 arXiv

  4. [11]

    Mind2web: Towards a generalist agent for the web, 2023

    Xiang Deng, Yu Gu, Boyuan Zheng, Shijie Chen, Samuel Stevens, Boshi Wang, Huan Sun, and Yu Su. Mind2web: Towards a generalist agent for the web, 2023. 3

    2023

  5. [12]

    Danny Driess, Fei Xia, Mehdi S. M. Sajjadi, Corey Lynch, Aakanksha Chowdhery, Brian Ichter, Ayzaan Wahid, Jonathan Tompson, Quan Vuong, Tianhe Yu, Wenlong Huang, Yevgen Chebotar, Pierre Sermanet, Daniel Duck- worth, Sergey Levine, Vincent Vanhoucke, Karol Hausman, Marc Toussai...

    2023

  6. [13]

    Training-free structured diffu- sion guidance for compositional text-to-image synthesis

    Weixi Feng, Xuehai He, Tsu-Jui Fu, Varun Jampani, Arjun Akula, Pradyumna Narayana, Sugato Basu, Xin Eric Wang, and William Yang Wang. Training-free structured diffu- sion guidance for compositional text-to-image synthesis. In ICLR, 2023. 2

    2023

  7. [14]

    Llm blueprint: Enabling text-to-image generation with complex and detailed prompts

    Hanan Gani, Shariq Farooq Bhat, Muzammal Naseer, Salman Khan, and Peter Wonka. Llm blueprint: Enabling text-to-image generation with complex and detailed prompts. arXiv preprint arXiv:2310.10640, 2023. 2

    2023 arXiv

  8. [15]

    Animatediff: Animate your personalized text-to-image diffusion models without specific tuning

    Yuwei Guo, Ceyuan Yang, Anyi Rao, Yaohui Wang, Yu Qiao, Dahua Lin, and Bo Dai. Animatediff: Animate your personalized text-to-image diffusion models without specific tuning. arXiv preprint arXiv:2307.04725, 2023. 2, 5, 6

    2023 arXiv

  9. [16]

    Latent video diffusion models for high-fidelity long video generation

    Yingqing He, Tianyu Yang, Yong Zhang, Ying Shan, and Qifeng Chen. Latent video diffusion models for high-fidelity long video generation. arXiv preprint arXiv:2211.13221 ,

    arXiv

  10. [17]

    Denoising diffu- sion probabilistic models, 2020

    Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffu- sion probabilistic models, 2020. 2

    2020

  11. [18]

    Imagen video: High definition video generation with diffusion mod- els

    Jonathan Ho, William Chan, Chitwan Saharia, Jay Whang, Ruiqi Gao, Alexey Gritsenko, Diederik P Kingma, Ben Poole, Mohammad Norouzi, David J Fleet, et al. Imagen video: High definition video generation with diffusion mod- els. arXiv preprint arXiv:2210.02303, 2022. 2

    arXiv

  12. [19]

    Metagpt: Meta programming for a multi-agent collaborative framework, 2024

    Sirui Hong, Mingchen Zhuge, Jiaqi Chen, Xiawu Zheng, Yuheng Cheng, Ceyao Zhang, Jinlin Wang, Zili Wang, Steven Ka Shing Yau, Zijuan Lin, Liyang Zhou, Chenyu Ran, Lingfeng Xiao, Chenglin Wu, and J¨urgen Schmidhuber. Metagpt: Meta programming for a multi-agent collaborative fram...

    2024

  13. [20]

    Cogvideo: Large-scale pretraining for text-to-video generation via transformers

    Wenyi Hong, Ming Ding, Wendi Zheng, Xinghan Liu, and Jie Tang. Cogvideo: Large-scale pretraining for text-to-video generation via transformers. arXiv preprint arXiv:2205.15868, 2022. 2

    2022 arXiv

  14. [21]

    Open-sora: Democratizing efficient video pro- duction for all, 2024

    hpcaitech. Open-sora: Democratizing efficient video pro- duction for all, 2024. 5, 6

    2024

  15. [22]

    T2i-compbench: A comprehensive bench- mark for open-world compositional text-to-image genera- tion

    Kaiyi Huang, Kaiyue Sun, Enze Xie, Zhenguo Li, and Xihui Liu. T2i-compbench: A comprehensive bench- mark for open-world compositional text-to-image genera- tion. NeurIPS, 2024. 2

    2024

  16. [23]

    Text2video-zero: Text- to-image diffusion models are zero-shot video generators

    Levon Khachatryan, Andranik Movsisyan, Vahram Tade- vosyan, Roberto Henschel, Zhangyang Wang, Shant Navasardyan, and Humphrey Shi. Text2video-zero: Text- to-image diffusion models are zero-shot video generators. In Proceedings of the IEEE/CVF International Conference on Comput...

    2023

  17. [24]

    Dense text-to-image generation with attention modulation

    Yunji Kim, Jiyoung Lee, Jin-Hwa Kim, Jung-Woo Ha, and Jun-Yan Zhu. Dense text-to-image generation with attention modulation. In ICCV, 2023. 2

    2023

  18. [25]

    Videopoet: A large language model for zero-shot video generation

    Dan Kondratyuk, Lijun Yu, Xiuye Gu, Jos ´e Lezama, Jonathan Huang, Rachel Hornung, Hartwig Adam, Hassan Akbari, Yair Alon, Vighnesh Birodkar, et al. Videopoet: A large language model for zero-shot video generation. arXiv preprint arXiv:2312.14125, 2023. 2

    2023 arXiv

  19. [26]

    Open-sora-plan, 2024

    PKU-Yuan Lab and Tuzhan AI etc. Open-sora-plan, 2024. 5, 6, 2

    2024

  20. [27]

    Gligen: Open-set grounded text-to-image generation

    Yuheng Li, Haotian Liu, Qingyang Wu, Fangzhou Mu, Jian- wei Yang, Jianfeng Gao, Chunyuan Li, and Yong Jae Lee. Gligen: Open-set grounded text-to-image generation. In ICCV, 2023. 2 9

    2023

  21. [28]

    Stylet2i: Toward compositional and high-fidelity text- to-image synthesis

    Zhiheng Li, Martin Renqiang Min, Kai Li, and Chenliang Xu. Stylet2i: Toward compositional and high-fidelity text- to-image synthesis. In CVPR, 2022

    2022

  22. [29]

    Llm-grounded video diffusion models

    Long Lian, Baifeng Shi, Adam Yala, Trevor Darrell, and Boyi Li. Llm-grounded video diffusion models. arXiv preprint arXiv:2309.17444, 2023. 2, 3, 4, 5, 6, 1

    2023 arXiv

  23. [30]

    Videodirectorgpt: Consistent multi-scene video generation via llm-guided planning

    Han Lin, Abhay Zala, Jaemin Cho, and Mohit Bansal. Videodirectorgpt: Consistent multi-scene video generation via llm-guided planning. In COLM, 2024. 3

    2024

  24. [31]

    Compositional visual generation with composable diffusion models

    Nan Liu, Shuang Li, Yilun Du, Antonio Torralba, and Joshua B Tenenbaum. Compositional visual generation with composable diffusion models. In ECCV, 2022. 2

    2022

  25. [32]

    Referee can play: An alterna- tive approach to conditional generation via model inversion

    Xuantong Liu, Tianyang Hu, Wenjia Wang, Kenji Kawaguchi, and Yuan Yao. Referee can play: An alterna- tive approach to conditional generation via model inversion. arXiv preprint arXiv:2402.16305, 2024. 2

    2024 arXiv

  26. [33]

    Videofusion: Decomposed diffusion mod- els for high-quality video generation

    Zhengxiong Luo, Dayou Chen, Yingya Zhang, Yan Huang, Liang Wang, Yujun Shen, Deli Zhao, Jingren Zhou, and Tieniu Tan. Videofusion: Decomposed diffusion mod- els for high-quality video generation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognit...

    2023

  27. [34]

    Latte: Latent diffusion transformer for video generation

    Xin Ma, Yaohui Wang, Gengyun Jia, Xinyuan Chen, Zi- wei Liu, Yuan-Fang Li, Cunjian Chen, and Yu Qiao. Latte: Latent diffusion transformer for video generation. arXiv preprint arXiv:2401.03048, 2024. 5, 6

    2024 arXiv

  28. [35]

    Conform: Contrast is all you need for high-fidelity text-to-image diffusion models

    Tuna Han Salih Meral, Enis Simsar, Federico Tombari, and Pinar Yanardag. Conform: Contrast is all you need for high-fidelity text-to-image diffusion models. arXiv preprint arXiv:2312.06059, 2023. 2

    2023 arXiv

  29. [36]

    Hello GPT-4o

    OpenAI. Hello GPT-4o. https : / / openai . com / index/hello-gpt-4o/ , 2024. Accessed: 2024-11-14. 5

    2024

  30. [37]

    Benchmark for compositional text-to- image synthesis

    Dong Huk Park, Samaneh Azadi, Xihui Liu, Trevor Darrell, and Anna Rohrbach. Benchmark for compositional text-to- image synthesis. In NeurIPS, 2021. 2

    2021

  31. [38]

    O’Brien, Carrie J

    Joon Sung Park, Joseph C. O’Brien, Carrie J. Cai, Mered- ith Ringel Morris, Percy Liang, and Michael S. Bernstein. Generative agents: Interactive simulacra of human behavior,

  32. [39]

    Eclipse: A resource-efficient text-to-image prior for image generations

    Maitreya Patel, Changhoon Kim, Sheng Cheng, Chitta Baral, and Yezhou Yang. Eclipse: A resource-efficient text-to-image prior for image generations. arXiv preprint arXiv:2312.04655, 2023. 2

    2023 arXiv

  33. [40]

    Chatdev: Communicative agents for software development,

    Chen Qian, Wei Liu, Hongzhang Liu, Nuo Chen, Yufan Dang, Jiahao Li, Cheng Yang, Weize Chen, Yusheng Su, Xin Cong, Juyuan Xu, Dahai Li, Zhiyuan Liu, and Maosong Sun. Chatdev: Communicative agents for software development,

  34. [41]

    Linguistic bind- ing in diffusion models: Enhancing attribute correspondence through attention map alignment

    Royi Rassin, Eran Hirsch, Daniel Glickman, Shauli Rav- fogel, Yoav Goldberg, and Gal Chechik. Linguistic bind- ing in diffusion models: Enhancing attribute correspondence through attention map alignment. NeurIPS, 2024. 2

    2024

  35. [42]

    Runway AI. Gen-3. https : / / runwayml . com / blog / introducing - gen - 3 - alpha/, 2024. Ac- cessed: 2024-11-14. 5, 6, 2

    2024

  36. [43]

    Make-a-video: Text-to-video generation without text-video data

    Uriel Singer, Adam Polyak, Thomas Hayes, Xi Yin, Jie An, Songyang Zhang, Qiyuan Hu, Harry Yang, Oron Ashual, Oran Gafni, et al. Make-a-video: Text-to-video generation without text-video data. arXiv preprint arXiv:2209.14792 ,

    arXiv

  37. [44]

    Weiss, Niru Mah- eswaranathan, and Surya Ganguli

    Jascha Sohl-Dickstein, Eric A. Weiss, Niru Mah- eswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics, 2015. 2

    2015

  38. [45]

    Kingma, Ab- hishek Kumar, Stefano Ermon, and Ben Poole

    Yang Song, Jascha Sohl-Dickstein, Diederik P. Kingma, Ab- hishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equa- tions, 2021. 2

    2021

  39. [47]

    A survey of neural code intelligence: Paradigms, advances and beyond, 2024

    Qiushi Sun, Zhirui Chen, Fangzhi Xu, Kanzhi Cheng, Chang Ma, Zhangyue Yin, Jianing Wang, Chengcheng Han, Renyu Zhu, Shuai Yuan, Qipeng Guo, Xipeng Qiu, Pengcheng Yin, Xiaoli Li, Fei Yuan, Lingpeng Kong, Xiang Li, and Zhiy- ong Wu. A survey of neural code intelligence: Paradigm...

    2024

  40. [48]

    Corex: Pushing the boundaries of complex reasoning through multi-model collaboration, 2024

    Qiushi Sun, Zhangyue Yin, Xiang Li, Zhiyong Wu, Xipeng Qiu, and Lingpeng Kong. Corex: Pushing the boundaries of complex reasoning through multi-model collaboration, 2024. 3

    2024

  41. [49]

    Box it to bind it: Unified layout control and at- tribute binding in t2i diffusion models

    Ashkan Taghipour, Morteza Ghahremani, Mohammed Ben- namoun, Aref Miri Rekavandi, Hamid Laga, and Farid Bous- said. Box it to bind it: Unified layout control and at- tribute binding in t2i diffusion models. arXiv preprint arXiv:2402.17910, 2024. 2

    2024 arXiv

  42. [50]

    Prioritizing safeguarding over autonomy: Risks of llm agents for science, 2024

    Xiangru Tang, Qiao Jin, Kunlun Zhu, Tongxin Yuan, Yichi Zhang, Wangchunshu Zhou, Meng Qu, Yilun Zhao, Jian Tang, Zhuosheng Zhang, Arman Cohan, Zhiyong Lu, and Mark Gerstein. Prioritizing safeguarding over autonomy: Risks of llm agents for science, 2024. 3

    2024

  43. [51]

    Videotetris: Towards compo- sitional text-to-video generation, 2024

    Ye Tian, Ling Yang, Haotian Yang, Yuan Gao, Yufan Deng, Jingmin Chen, Xintao Wang, Zhaochen Yu, Xin Tao, Pengfei Wan, Di Zhang, and Bin Cui. Videotetris: Towards compo- sitional text-to-video generation, 2024. 2, 3, 5, 6, 7

    2024

  44. [52]

    Phenaki: Variable length video generation from open domain textual descriptions

    Ruben Villegas, Mohammad Babaeizadeh, Pieter-Jan Kin- dermans, Hernan Moraldo, Han Zhang, Mohammad Taghi Saffar, Santiago Castro, Julius Kunze, and Dumitru Erhan. Phenaki: Variable length video generation from open domain textual descriptions. In International Conference on Le...

    2022

  45. [53]

    V oyager: An open-ended embodied agent with large language models, 2023

    Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and Anima Anandku- mar. V oyager: An open-ended embodied agent with large language models, 2023. 3

    2023

  46. [54]

    Modelscope text-to-video technical report

    Jiuniu Wang, Hangjie Yuan, Dayou Chen, Yingya Zhang, Xiang Wang, and Shiwei Zhang. Modelscope text-to-video technical report. arXiv preprint arXiv:2308.06571, 2023. 2, 5, 6

    2023 arXiv

  47. [55]

    Compositional text-to-image synthe- sis with attention map control of diffusion models

    Ruichen Wang, Zekang Chen, Chen Chen, Jian Ma, Haonan Lu, and Xiaodong Lin. Compositional text-to-image synthe- sis with attention map control of diffusion models. arXiv preprint arXiv:2305.13921, 2023. 2 10

    2023 arXiv

  48. [56]

    Xu, Xian- gru Tang, Mingchen Zhuge, Jiayi Pan, Yueqi Song, Bowen Li, Jaskirat Singh, Hoang H

    Xingyao Wang, Boxuan Li, Yufan Song, Frank F. Xu, Xian- gru Tang, Mingchen Zhuge, Jiayi Pan, Yueqi Song, Bowen Li, Jaskirat Singh, Hoang H. Tran, Fuqiang Li, Ren Ma, Mingzhang Zheng, Bill Qian, Yanjun Shao, Niklas Muen- nighoff, Yizhe Zhang, Binyuan Hui, Junyang Lin, Robert Br...

    2024

  49. [57]

    Genartist: Multimodal llm as an agent for unified image gen- eration and editing, 2024

    Zhenyu Wang, Aoxue Li, Zhenguo Li, and Xihui Liu. Genartist: Multimodal llm as an agent for unified image gen- eration and editing, 2024. 2, 3

    2024

  50. [58]

    Divide and conquer: Language mod- els can plan and self-correct for compositional text-to-image generation

    Zhenyu Wang, Enze Xie, Aoxue Li, Zhongdao Wang, Xihui Liu, and Zhenguo Li. Divide and conquer: Language mod- els can plan and self-correct for compositional text-to-image generation. arXiv preprint arXiv:2401.15688, 2024. 2

    2024 arXiv

  51. [59]

    Godiva: Gen- erating open-domain videos from natural descriptions

    Chenfei Wu, Lun Huang, Qianxi Zhang, Binyang Li, Lei Ji, Fan Yang, Guillermo Sapiro, and Nan Duan. Godiva: Gen- erating open-domain videos from natural descriptions. arXiv preprint arXiv:2104.14806, 2021. 2

    2021 arXiv

  52. [60]

    N ¨uwa: Visual synthesis pre- training for neural visual world creation

    Chenfei Wu, Jian Liang, Lei Ji, Fan Yang, Yuejian Fang, Daxin Jiang, and Nan Duan. N ¨uwa: Visual synthesis pre- training for neural visual world creation. In European con- ference on computer vision, pages 720–736. Springer, 2022. 2

    2022

  53. [61]

    Autogen: Enabling next-gen llm ap- plications via multi-agent conversation, 2023

    Qingyun Wu, Gagan Bansal, Jieyu Zhang, Yiran Wu, Beibin Li, Erkang Zhu, Li Jiang, Xiaoyun Zhang, Shaokun Zhang, Jiale Liu, Ahmed Hassan Awadallah, Ryen W White, Doug Burger, and Chi Wang. Autogen: Enabling next-gen llm ap- plications via multi-agent conversation, 2023. 3

    2023

  54. [62]

    Harnessing the spatial- temporal attention of diffusion models for high-fidelity text- to-image synthesis

    Qiucheng Wu, Yujian Liu, Handong Zhao, Trung Bui, Zhe Lin, Yang Zhang, and Shiyu Chang. Harnessing the spatial- temporal attention of diffusion models for high-fidelity text- to-image synthesis. In ICCV, 2023. 2

    2023

  55. [63]

    Gonzalez, Boyi Li, and Trevor Darrell

    Tsung-Han Wu, Long Lian, Joseph E. Gonzalez, Boyi Li, and Trevor Darrell. Self-correcting llm-controlled diffusion models, 2023. 2

    2023

  56. [64]

    Mastering text-to-image diffu- sion: Recaptioning, planning, and generating with multi- modal llms

    Ling Yang, Zhaochen Yu, Chenlin Meng, Minkai Xu, Ste- fano Ermon, and Bin Cui. Mastering text-to-image diffu- sion: Recaptioning, planning, and generating with multi- modal llms. arXiv preprint arXiv:2401.11708, 2024. 2, 3

    2024 arXiv

  57. [65]

    Compositional video gen- eration as flow equalization, 2024

    Xingyi Yang and Xinchao Wang. Compositional video gen- eration as flow equalization, 2024. 2, 3, 5, 6

    2024

  58. [66]

    Cogvideox: Text-to-video diffusion models with an expert transformer

    Zhuoyi Yang, Jiayan Teng, Wendi Zheng, Ming Ding, Shiyu Huang, Jiazheng Xu, Yuanming Yang, Wenyi Hong, Xiao- han Zhang, Guanyu Feng, et al. Cogvideox: Text-to-video diffusion models with an expert transformer. arXiv preprint arXiv:2408.06072, 2024. 5, 6, 7, 2

    2024 arXiv

  59. [67]

    Webshop: Towards scalable real-world web interaction with grounded language agents, 2023

    Shunyu Yao, Howard Chen, John Yang, and Karthik Narasimhan. Webshop: Towards scalable real-world web interaction with grounded language agents, 2023. 3

    2023

  60. [68]

    Magvit: Masked generative video transformer

    Lijun Yu, Yong Cheng, Kihyuk Sohn, Jos ´e Lezama, Han Zhang, Huiwen Chang, Alexander G Hauptmann, Ming- Hsuan Yang, Yuan Hao, Irfan Essa, et al. Magvit: Masked generative video transformer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p...

    2023

  61. [69]

    Language model beats diffusion–tokenizer is key to visual generation

    Lijun Yu, Jos ´e Lezama, Nitesh B Gundavarapu, Luca Ver- sari, Kihyuk Sohn, David Minnen, Yong Cheng, Agrim Gupta, Xiuye Gu, Alexander G Hauptmann, et al. Language model beats diffusion–tokenizer is key to visual generation. arXiv preprint arXiv:2310.05737, 2023. 2

    arXiv

  62. [70]

    Mag- ictime: Time-lapse video generation models as metamorphic simulators

    Shenghai Yuan, Jinfa Huang, Yujun Shi, Yongqi Xu, Ruijie Zhu, Bin Lin, Xinhua Cheng, Li Yuan, and Jiebo Luo. Mag- ictime: Time-lapse video generation models as metamorphic simulators. arXiv preprint arXiv:2404.05014, 2024. 5, 6

    2024 arXiv

  63. [71]

    Mora: En- abling generalist video generation via a multi-agent frame- work, 2024

    Zhengqing Yuan, Yixin Liu, Yihan Cao, Weixiang Sun, Hao- long Jia, Ruoxi Chen, Zhaoxu Li, Bin Lin, Li Yuan, Lifang He, Chi Wang, Yanfang Ye, and Lichao Sun. Mora: En- abling generalist video generation via a multi-agent frame- work, 2024. 3

    2024

  64. [72]

    Show-1: Marrying pixel and latent dif- fusion models for text-to-video generation

    David Junhao Zhang, Jay Zhangjie Wu, Jia-Wei Liu, Rui Zhao, Lingmin Ran, Yuchao Gu, Difei Gao, and Mike Zheng Shou. Show-1: Marrying pixel and latent dif- fusion models for text-to-video generation. arXiv preprint arXiv:2309.15818, 2023. 5, 6

    2023 arXiv

  65. [73]

    Big Ben”, one on the left and the other on the right. In the second example of Figure A10 (the second row), VideoTetris [51] and Gen-3 [42] do not de- pict “sitting on haunches

    Daquan Zhou, Weimin Wang, Hanshu Yan, Weiwei Lv, Yizhe Zhu, and Jiashi Feng. Magicvideo: Efficient video generation with latent diffusion models. arXiv preprint arXiv:2211.11018, 2022. 2 11 GENMAC: Compositional Text-to-Video Generation with Multi-Agent Collaboration Supplemen...

    2022 arXiv

  66. [74]

    a glass sculpture

    fail to generate “a glass sculpture”, while Gen-3 [42] omits “ancient vase”. For the second prompt in Fig- ure A11 (the second row), Gen-3 [42] and Open-Sora-Plan

  67. [75]

    porcelain rabbit

    lack “porcelain rabbit” in some or all frames, while VideoCrafter2 [8], CogVideoX [66], and VideoTetris [51] generate several “golden cactus” instead of “one” indicated in the prompt. For the third prompt in Figure A11 (the third row), Open-Sora-Plan [26] omits the “butterfly”...

  68. [76]

    Rabbit police officer directs traffic

    Suggest corrections for the bounding boxes:- Remove one of the cars to correct the quantity issue.- Ensure the remaining car is clearly shown moving from right to left by adjusting its position across frames to depict motion.2. Choose the suitable correction agent: B1. (correc...

  69. [77]

    This could involve positioning the rabbit with an arm raised or using a gesture to indicate traffic direction.2

    **Suggest corrections for the bounding boxes**:- Add a bounding box to include a scene or action where the rabbit is directing traffic. This could involve positioning the rabbit with an arm raised or using a gesture to indicate traffic direction.2. **Choose the suitable correc...

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.