{"id":"24f2884e-66a5-4c17-ba3d-350fdd76bce3","arxiv_id":"2412.04440","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"An iterative design-generate-redesign pipeline with four specialized LLM agents and self-routing correction improves compositional text-to-video generation on T2V-CompBench, with the largest gains in object numeracy.","lead":"GenMAC is a multi-agent system that breaks text-to-video generation into design, generation, and a redesign loop where specialized AI agents check the output, propose fixes, and update layouts and prompts. It reports stronger compositional fidelity, especially for object counts and spatial relationships, over 17 baselines on T2V-CompBench.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"SOTA claim rests on importing baseline scores from T2V-CompBench, but no controlled evaluation protocol is shared; reported margins may be overstated.","rationale":"The reader correctly identifies the weakest assumption: GenMAC's reported state-of-the-art performance depends on comparing its own scores against baseline numbers imported from T2V-CompBench [46], which are not generated under identical evaluation conditions. This concern is load-bearing because the paper's central claim is the SOTA comparison, not just the internal architecture. Without a controlled re-evaluation, the large margins in Table 1 could be an artifact of different resolutions, frame counts, or metric pipelines. The internal ablations in Table 2 show that each component helps on GenMAC's own pipeline, providing directional evidence for the framework's effectiveness, but they do not establish external superiority. The concrete test proposed, rerunning the baselines under GenMAC's exact evaluation settings, would directly settle whether the headline comparisons hold. Since this concern matches the reader's weakest_assumption, the verdict remains conditional; the paper would be acceptable once the evaluation protocol is made rigorous and reproducible.","tokens_in":535,"tokens_out":4069,"duration_ms":84262,"concrete_test":"Rerun the 17 baseline models, or at least the top competitors (VideoCrafter2, CogVideoX-5B, VideoTetris, Vico, LVD, Open-Sora-Plan, and Gen-3), on the same T2V-CompBench prompts and the same automatic metric implementations, at the same resolution (512x512), frame count (65), and a fixed inference budget, then recompute Table 1. Report results over at least 3 seeds. If the relative ordering changes materially, for example the numeracy gap drops below 20% relative or changes sign, the state-of-the-art claim is not supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is 'state-of-the-art performance in compositional text-to-video generation'. The quantitative evidence for this is Table 1, where every baseline score is taken from T2V-CompBench [46] rather than re-run under a common protocol. GenMAC is evaluated at 65 frames, 512x512, using VideoCrafter2 and GPT-4o, while the baselines were evaluated separately, possibly at different resolutions, frame counts, prompt templates, or metric implementations. Because the metric values are numeric scores on detection/LLM-based measures, they are sensitive to such protocol differences. For instance, the headline gap in generative numeracy (0.5166 vs 0.2928, a 76% relative margin) could shrink or disappear if baselines were run at the same frame count and with identical G-Dino inference settings. The paper reports no error bars, no seed variations, and no code or agent-prompt release. The internal ablations (Table 2) show consistent improvements of GenMAC's components on its own pipeline, but that does not establish superiority over external baselines. Figure 7 uses the verification agent to decide when a prompt has 'completed refinement', which is a circular success signal for the loop, though it does not directly contaminate the external metric evaluation; however, the lack of a defined stopping rule adds ambiguity. The load-bearing condition is that the Table 1 baseline numbers are directly comparable to GenMAC's own numbers; this is not demonstrated and is not safe to assume.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes GenMAC, an iterative multi-agent framework for compositional text-to-video generation. The workflow consists of a Design stage, a Generation stage using an off-the-shelf video diffusion model (VideoCrafter2), and a Redesign stage in which four sequentially executed MLLM-based agents verify alignment, suggest corrections, apply corrections, and structure the output for the next generation iteration. A self-routing mechanism adaptively selects among specialized correction agents for consistency, temporal dynamics, and spatial dynamics. The authors evaluate on T2V-CompBench and report the best scores in all seven compositional aspects, claiming state-of-the-art performance, and provide ablations showing the contribution of the multi-stage pipeline, role specialization, iterative refinement, and self-routing.","tokens_in":23634,"tokens_out":2240,"duration_ms":26558,"significance":"If the state-of-the-art claim were firmly established, GenMAC would be a notable contribution: it is among the first frameworks to apply heterogeneous multi-agent collaboration with iterative verification-and-redesign to compositional text-to-video generation, without retraining the backbone. The internal ablations in Table 2 consistently support the value of each design component (Design stage, Redesign stage, iterative loop, task decomposition, and self-routing), and the main benchmark uses external metrics (Grid-LLaVA, D-LLaVA, G-Dino, DOT), so the central comparison is not circular. The principal weakness is that the quantitative SOTA claim rests on importing baseline scores from T2V-CompBench [46] without a shared evaluation protocol, and all numbers are single runs without variance reporting.","major_comments":[{"comment":"The central state-of-the-art claim depends on comparing GenMAC's scores against baseline scores that are not recomputed under GenMAC's settings. The Table 1 caption states 'The baseline data are sourced from [46]', while GenMAC is evaluated at 65 frames, 512x512, with VideoCrafter2 and GPT-4o. Because the baselines were evaluated separately, differences in resolution, frame count, prompt templates, or metric implementations (especially for detection-based G-Dino and DOT scores) can change the numeric values. The reported margin in generative numeracy (0.5166 vs 0.2928, about 76% relative) may therefore overstate the true improvement. The authors should either re-run all baselines under a common protocol or explicitly report the protocol differences and provide sensitivity analysis (e.g., how scores vary with frame count and resolution).","section":"Table 1 and Section 4.1"},{"comment":"Every reported number in Table 1 appears to come from a single run, with no error bars, no seed variations, and no statistical significance tests. Given the stochasticity of video diffusion models and the LLM/detector-based metrics, the observed differences between GenMAC and the second-best baseline could be within run-to-run noise, particularly for the smaller margins such as dynamic attribute binding (5.13%) and object interactions (4.43%). Reporting multiple seeds with means and standard deviations, or at least per-prompt score distributions, is required to support the SOTA claim.","section":"Section 4.2 and Table 1"},{"comment":"The 'cumulative corrected ratio' in Figure 7 counts a prompt as corrected when the paper's own verification agent judges the video aligned and the loop exits. This is a circular success signal for measuring refinement progress, because the judge is itself part of the system being evaluated. Additionally, the stopping rule for the iterative loop is not formally defined; the paper mentions a maximum of nine iterations in Figure 7 but does not describe the termination criterion in the text. To substantiate the iterative-refinement analysis, the authors should track an external metric at each iteration and specify the stopping condition independently of the verification agent's judgment.","section":"Section 4.4 and Figure 7"}],"minor_comments":[{"comment":"There are several typographical and grammatical errors, e.g., 'and and tet prompts' near the Generation stage description, and 'an iterative process is composed of three stages' should be 'an iterative process composed of three stages'.","section":"Section 1"},{"comment":"The word 'comparision' should be 'comparison'.","section":"Figure 4 caption"},{"comment":"The labels '/tasks', '/edit', '/tools', and '/user-check' appear as stray annotations in the agent descriptions; these should be removed or clearly explained if they are meant to indicate tool calls.","section":"Section 3.2"},{"comment":"The guidance scale initialization (1.0) and increment (0.05) are only given in the appendix but are important for reproducing the method; they should also appear in the main-text implementation details.","section":"Appendix B.1"},{"comment":"Several references list incorrect final page numbers, such as [8], [29], and [66] ending in '1, 2'; these should be corrected.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The main gate for acceptance is the rigor of the quantitative comparison. The internal ablations and the proposed framework are credible, but the SOTA claim as currently formulated depends on an uncontrolled comparison with imported baseline scores. If the authors can re-run baselines under a common protocol, or convincingly bound the effect of protocol differences, and add variance reporting, the paper could reach the bar for this journal. The circularity in the iterative-analysis figure is a secondary but real concern."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a solid system paper whose internal evidence is consistent, but the headline SOTA claim is not nailed down because every baseline number in Table 1 is imported from the T2V-CompBench paper rather than re-run under GenMAC's protocol. The four-agent decomposition of the redesign stage with self-routing among consistency, temporal, and spatial correction agents is genuinely new relative to the cited GenArtist, LVD, VideoDirectorGPT, and Mora. The paper does a good job of explaining the workflow, and the ablations in Table 2 show each added component moves scores in the right direction on its own evaluation pipeline. That is real evidence for the mechanism, not just a vibe.\n\nWhere it gets soft: the central SOTA claim. Table 1's caption says baseline data are sourced from [46], and the numbers are single-run with no error bars. GenMAC runs at 65 frames, 512x512, with VideoCrafter2 and GPT-4o; the baselines were evaluated separately, possibly at different resolutions, frame counts, and metric implementations. These are detector/LLM scores, so protocol differences could explain a chunk of the reported margins. The numeracy gap (0.5166 vs 0.2928) is dramatic and would need a controlled rerun to believe. The internal ablations are not affected by this problem, but they do not establish superiority over external baselines either.\n\nFigure 7's 'cumulative corrected ratio' is a softer flaw. The loop exits when GenMAC's own verification agent judges the video aligned, so the ratio partly measures the agent's self-consistency rather than external alignment. It does not contaminate the Table 1 metrics, but the paper should frame it as an internal convergence measure and define the stopping rule clearly. The supplementary material is honest about limitations (MLLM capability ceiling, base model limits), though code and agent prompts are not released. That is addressable.\n\nWho it is for: people building production text-to-video pipelines or working on self-correction with MLLMs. It does not settle a theoretical question, but it gives a practical recipe with believable internal evidence. I would send it to a serious referee. Conditional acceptance with a request to either rerun baselines under a matched protocol or soften the SOTA language, plus report variance and release artifacts. For my own work, I would cite the framework if writing about agent-based video generation, but I would not quote the numeracy number as established.","headline":"A useful self-correction framework for compositional T2V, but the SOTA claim rests on unmatched baseline numbers and needs a controlled rerun or softer language.","tokens_in":24247,"tokens_out":2230,"would_cite":true,"duration_ms":72496,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Agent loop tops all seven compositional video benchmarks","keywords":["compositional text-to-video generation","multi-agent collaboration","MLLM agents","iterative refinement","self-routing","T2V-CompBench","layout guidance"],"falsifier":"Re-run every baseline in Table 1 under GenMAC's exact protocol (65 frames, 512x512, same prompt set, same metric versions, and same judge settings) and check whether GenMAC still beats each one; if regenerating the baseline scores changes the 0.5166 versus 0.2928 numeracy margin, the state-of-the-art claim does not stand.","tokens_in":23144,"feed_emoji":"🎬","tokens_out":3114,"duration_ms":31967,"temperature":0.7,"pith_summary":"The paper proposes GenMAC, a framework that wraps an off-the-shelf text-to-video model in a loop of design, generation, and redesign, so that complex prompts with multiple objects, attributes, spatial relations, and motions are checked and corrected iteratively. The central claim is that decomposing the redesign step into four specialized MLLM agents—verification, suggestion, correction, and output structuring—plus a self-routing choice among correction experts, makes video outputs align with compositional prompts much better than single-pass models. On the seven T2V-CompBench aspects, GenMAC reports the best score in every category, with the largest margin in generative numeracy. If the comparison is fair, this would mean that compositional video fidelity can be improved substantially without retraining the backbone generator.","feed_headline":"Agent loop tops all seven compositional video benchmarks","feed_subtitle":"GenMAC splits redesign into four specialized roles and iterates until the clip matches the prompt.","key_machinery":"The load-bearing mechanism is the Design-Generation-Redesign loop with a decomposed redesign stage. The redesign stage is split into four sequentially executed MLLM agents (verification, suggestion, correction, and output structuring), and a self-routing mechanism selects one of three correction experts (consistency, temporal dynamics, or spatial dynamics). The generation stage injects the layout into the diffusion model by gradient-guiding cross-attention maps toward bounding boxes, with a guidance scale that the redesign stage adjusts. This machinery converts an open-ended alignment problem into a series of bounded sub-tasks and a structured output that the generator can consume.","core_discovery":"GenMAC's claim is that a complex text-to-video alignment task can be handled by orchestration rather than by stronger generation alone. The workflow first uses an LLM to design per-frame bounding boxes and prompts, generation uses an off-the-shelf video diffusion model with attention-map layout guidance, and a redesign stage then verifies the video against the prompt on existence, quantity, attributes, and relationships, suggests changes, selects among correction agents specialized for consistency, temporal dynamics, or spatial dynamics, and emits structured layouts, revised prompts, and guidance scales for the next iteration. The loop runs until the verification agent is satisfied. The paper reports that this Design-Generation-Redesign loop achieves the best scores on all seven T2V-CompBench categories, for example generative numeracy of 0.5166 against 0.2928 for the second-best model, and the ablations attribute the gains to iterative refinement, role specialization, and self-routing.","pith_inferences":["A testable extension is to apply the same agent decomposition to text-to-image generation, where layout and correction machinery are simpler and the benchmark is already available.","Because the agents rely on GPT-4o, the reported gains are capped by the verifier's perceptual accuracy; a weaker or open-source MLLM would likely shrink the margins.","If the baseline scores sourced from T2V-CompBench were produced under different resolutions or frame counts, the headline margins could shift; re-running the comparison under one identical protocol would settle this.","Accepted iterations of the loop are near-correct clip-prompt pairs, which could be collected as training data to fine-tune the backbone generator without human annotation."],"forward_implications":["If GenMAC's reported results hold, compositional text-to-video quality can be improved by orchestration rather than by scaling or retraining the generator.","The self-routing correction scheme suggests that verification and correction for generative models are best handled by role-specialized agents instead of a single generalist.","Iterative refinement with structured layout, guidance scale, and prompt revision gives the largest gains on object count, spatial relations, and motion binding, while dynamic attribute changes remain the hardest aspect.","The framework should transfer to newer backbones and MLLMs whenever their base capabilities improve, because it only wraps generation and correction around them."],"supporting_citations":[{"why":"Supplies the T2V-CompBench benchmark and the baseline scores that GenMAC claims to beat on all seven categories.","marker":"[46]"},{"why":"Supplies the layout-conditioned attention-map guidance method that the Generation stage uses to inject bounding boxes into the diffusion model.","marker":"[29]"},{"why":"Serves as the off-the-shelf VideoCrafter2 backbone that GenMAC wraps for the Generation stage.","marker":"[8]"},{"why":"Provides GPT-4o, the MLLM used to instantiate the verification, suggestion, correction, and output structuring agents.","marker":"[36]"},{"why":"Is a compositional text-to-video method that GenMAC compares against and reports outperforming.","marker":"[51]"},{"why":"Is a compositional text-to-video method that GenMAC compares against and reports outperforming.","marker":"[65]"}],"fun_headline_variants":["Multi-agent loop beats all seven T2V-CompBench scores","GenMAC's agent redesign loop tops all 7 compositional benchmarks","Iterative multi-agent loop beats all seven T2V-CompBench metrics","Agent collaboration and self-routing lift compositional T2V scores"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The claim that GenMAC is state-of-the-art assumes the baseline scores copied from T2V-CompBench are directly comparable to GenMAC's own runs, even though the baselines were evaluated separately and may have used different frame counts, resolutions, or settings.","fun_headline_variants_meta":{"raw":{"variants":["Multi-agent loop beats all seven T2V-CompBench scores","GenMAC's agent redesign loop tops all 7 compositional benchmarks","Iterative multi-agent loop beats all seven T2V-CompBench metrics","Agent collaboration and self-routing lift compositional T2V scores"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00075,"raw_usage":{"total_tokens":3362,"prompt_tokens":990,"completion_tokens":2372,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":606,"completion_tokens_details":{"reasoning_tokens":2297}},"tokens_in":606,"tokens_out":2372,"duration_ms":18106,"temperature":1.0,"reasoning_tokens":2297,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T21:23:34.573981+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run every baseline in Table 1 under GenMAC's exact protocol (65 frames, 512x512, same prompt set, same metric versions, and same judge settings) and check whether GenMAC still beats each one; if regenerating the baseline scores changes the 0.5166 versus 0.2928 numeracy margin, the state-of-the-art claim does not stand.","supporting_citations":[{"cited_title":"Hello GPT-4o","cited_arxiv_id":null,"evidence_quote":"Provides GPT-4o, the MLLM used to instantiate the verification, suggestion, correction, and output structuring agents."},{"cited_title":"Videotetris: Towards compo- sitional text-to-video generation, 2024","cited_arxiv_id":null,"evidence_quote":"Is a compositional text-to-video method that GenMAC compares against and reports outperforming."},{"cited_title":"Compositional video gen- eration as flow equalization, 2024","cited_arxiv_id":null,"evidence_quote":"Is a compositional text-to-video method that GenMAC compares against and reports outperforming."}],"review_version":1}