Pith. sign in

REVIEW 4 major objections 6 minor 1 cited by

ComfyMind: Toward General-Purpose Generation via Tree-Based Planning and Reactive Feedback

T0 review · 4 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read ComfyMind claims that composing prebuilt, semantically described workflow modules with tree-based planning and localized feedback lets an open-source system match or beat a closed unified generative model on standard generation, editing…

desk verdict A solid systems paper whose external-benchmark results deserve attention, but the ComfyBench resolve rate is self-scored by the system's own evaluator, which makes the central claim unsecured until independently verified. read the letter →

arxiv 2505.17908 v1 pith:YGWB3APK submitted 2025-05-23 cs.AI cs.CV

Litao Guo (1) , Xinli Xu (1) , Luozhou Wang (1) , Jiantao Lin (1) , Jinsong Zhou (1) , Zixin Zhang (1) , Bolan Su (3) , Ying-Cong Chen (1
show 3 more authors
2) ((1) HKUST (GZ) (2) HKUST (3) Bytedance)
This is my paper
classification cs.AIcs.CV
keywords general-purposegenerationComfyUIworkflowssemanticworkflowinterfacesearchtreeplanninglocalfeedbackexecutiontext-to-imageimageeditingmulti-agentsystems
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

ComfyMind is trying to show that general-purpose visual generation can be built by composing pre-existing workflow modules rather than by training a new unified model. The paper's central claim is that a Semantic Workflow Interface plus Search Tree Planning with local feedback lets an open-source system achieve a 100% workflow pass rate on ComfyBench, an overall score of 0.90 on GenEval (above GPT-Image-1's 0.84), and a Reason-Edit GPT-score of 0.906, roughly matching the closed model. If true, this matters because it suggests open-source systems can reach the capability level of closed generative models by orchestrating modular tools, and can extend across image and video generation, editing, and reasoning tasks. The gain is attributed to semantic-level composition, which avoids low-level JSON errors, and to local reactive correction, which prevents one failed step from forcing a full restart.

What carries the argument

The load-bearing mechanism is the pair of the Semantic Workflow Interface (SWI) and Search Tree Planning with Local Feedback Execution. SWI abstracts low-level ComfyUI node graphs into atomic functions with natural-language descriptions, input/output specifications, and parameters; the search tree treats a task as a hierarchy of sub-tasks, with each edge an SWI invocation, and an Evaluation Agent returning a pass/fail signal plus diagnostic feedback so the planner backtracks locally rather than restarting. Together they replace token-level workflow synthesis with semantic composition and adaptive correction.

What would settle it

Have an independent human or a fixed third-party judge re-score ComfyMind's ComfyBench outputs using the original task rubrics, and compare the resulting resolve rate with the reported 83%; a large drop would show the internal evaluator is not a reliable success signal and would undermine the performance claims.

Watch

Extended reading notes

Core claim

On the paper's own terms, ComfyMind's central claim is that general-purpose visual generation does not require a single unified model; it can be achieved by a planner that composes atomic, community-validated workflow modules. The Semantic Workflow Interface turns low-level node graphs into callable natural-language functions, and the Search Tree Planning mechanism executes one step at a time, evaluates each intermediate result with a vision-language evaluator, and backtracks locally on failure. On ComfyBench the system reaches a 100% workflow pass rate and 83% task resolution, on GenEval an overall score of 0.90 (above GPT-Image-1's 0.84), and on Reason-Edit a GPT-score of 0.906, comparable to GPT-Image-1's 0.929.

Load-bearing premise

The system's own Evaluation Agent decides whether a generated output counts as successful, and that same judgment is used to compute the headline resolve rates, so a lenient or biased evaluator would inflate the claimed performance.

Editorial extensions

If this is right

  • An open-source system built from existing workflow modules can match or exceed a closed unified model on compositional image generation, without training a new foundation model.
  • Complex multi-stage generation becomes more robust because failures trigger local backtracking at the current tree level, preserving valid partial results.
  • Adding new capabilities reduces to writing one semantic description for a workflow, so the system can track community-contributed tools quickly.
  • The same framework extends beyond text-to-image to video generation, image editing, reasoning-based editing, and world-knowledge synthesis, as shown by the additional WISE benchmark results.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If an independent evaluator were used across all baselines, the ranking might shift; the paper's internal Evaluation Agent is part of the system being tested, so cross-system comparisons inherit its judgment.
  • The approach makes capability scaling a matter of library curation rather than model training: expanding the set of atomic workflows could improve performance faster than scaling any single generative model.
  • The same semantic-interface-plus-search-tree design could be applied to other node-based or tool-based execution environments, because the interface hides the underlying syntax from the planner.
  • Because prompt preprocessing optimizes every input before generation, ablating that preprocessing would reveal how much of the GenEval gain comes from the planner versus the prompt-enhancement workflow.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. ComfyMind proposes a collaborative AI system built on ComfyUI for general-purpose visual generation. The two main components are a Semantic Workflow Interface (SWI), which abstracts prebuilt atomic node graphs into natural-language callable functions, and a Search Tree Planning mechanism with local feedback execution, which composes those functions through hierarchical planning and iterative correction. The paper evaluates ComfyMind on ComfyBench, GenEval, and Reason-Edit, reporting a 100% pass rate and 83% resolve rate on ComfyBench, an overall GenEval score of 0.90, and a Reason-Edit GPT-score of 0.906. The central claim is that an open-source modular orchestrator can outperform existing open-source frameworks and reach performance comparable to GPT-Image-1.

Significance. If the reported results are valid, the contribution is significant: it demonstrates a practical path toward general-purpose generation without training a new foundation model, by composing community-validated workflows through semantic planning. The SWI abstraction and tree-based local feedback are well-motivated design ideas, and the paper includes useful ablations, a robustness check across two LLMs, and an unusually detailed appendix of system prompts that aids reproducibility. However, the strongest performance claims currently rest on a measurement that appears partly circular and on protocol details that are not reported. The GenEval and Reason-Edit benchmarks use external metrics and provide independent evidence, but the ComfyBench resolve rate, which is the centerpiece of the comparison against ComfyAgent and GPT-Image-1, needs independent verification before the headline claims can be accepted.

major comments (4)
  1. [§3.3 and Appendix C.3.3; Table 1] The ComfyBench resolve rate (Table 1: 83.0% overall) is measured by the system's own Evaluation Agent, which also decides when the search terminates. Section 3.3 states: "If the result passes evaluation, the search ends successfully." Appendix C.3.3 describes the evaluator as returning "a binary judgment indicating whether the current task is considered complete" and supporting "dynamic threshold adjustment." Consequently, the reported resolve rate is the rate at which the system accepted its own outputs, not an externally verified rate of satisfying ComfyBench task requirements; a lenient or miscalibrated evaluator inflates the number directly by terminating search on outputs an independent rater would reject. This is load-bearing because the claims of outperforming ComfyAgent and matching GPT-Image-1 rest on these labels. Please report the exact evaluation protocol, fix the threshold rather than allowing it to vary, and provide independent scoring, for example using ComfyBench's official judge or human raters blind to system identity.
  2. [§3.2 and §4.1; Table 1] The 100% pass rate on ComfyBench is partly by construction and does not measure the same capability as the baselines. The Semantic Workflow Interface restricts the planner to selecting prebuilt atomic workflows and preserving their DAGs during execution (Section 3.2 and Appendix C.3.2), so the system does not synthesize low-level node graphs from scratch in the way ComfyAgent must. The pass rate therefore largely reflects the coverage and quality of the atomic workflow library and the planner's retrieval accuracy, not the ability to construct valid ComfyUI workflows. The phrase "autonomous workflow construction" in Section 4.1 should be qualified, and Table 1 should be framed as a comparison between module composition over prevalidated workflows and generative construction of node-level workflows.
  3. [§4.2.1 and §4.3.1] The GenEval and Reason-Edit comparisons may not be protocol-matched. ComfyMind's pipeline includes prompt optimization, parameter adjustment, and iterative local feedback that can re-execute workflows until the internal Evaluation Agent passes (Section 3.3), whereas the baselines are single-pass generative models. If ComfyMind is allowed multiple generation attempts per prompt, the GenEval score of 0.90 and Reason-Edit score of 0.906 are not comparable to the baselines under equal inference budget. The paper should state the number of attempts allowed per prompt, whether only the first output is scored, and the compute or API budget consumed by each system.
  4. [§4, Tables 1-5] No statistical uncertainty or protocol details are reported for any of the main tables: no number of independent runs, seeds, trials, or confidence intervals. The only hardware description is one RTX A6000 in Appendix C.1. For a 200-task benchmark, point estimates such as 83.0% versus 32.5% need at least a confidence interval or an exact resolved-task count (e.g., 166/200). Additionally, key free parameters of the system are not reported: the Evaluation Agent's threshold, the search-tree budget, and the contents of the prompt-optimization dictionary. Since these parameters directly affect the reported numbers, the sensitivity of the headline results to them is unknown. Please provide run-level data and a precise specification of all tunable parameters.
minor comments (6)
  1. [§4.2.2 and Table 2] The term "Attribution Binding" is inconsistent with "Attr. Binding" in Table 2 and with the standard term "Attribute Binding" used in the GenEval benchmark; please standardize.
  2. [Figure 3] Figure 3 contains typos ("anima style", "Wam2.1" for what is presumably Wan2.1), and the small handwritten-style feedback text is difficult to read; please clean up the figure and increase legibility.
  3. [Appendix E] The system prompt for "Workflow Adaptive Adjustment" contains the typo "upsacle rate" (should be "upscale rate"), and the JSON schema in Figure 16 has the invalid entry "required(ALL PARAMETERS ARE OPTIONAL EXCEPT WORKFLOW_NAME)"; this should be corrected to a proper JSON Schema formulation.
  4. [Project page and code] No code, atomic workflow library, or evaluation scripts are released; including these would substantially improve reproducibility, especially given that the reported system behavior depends on the exact workflow metadata.
  5. [Appendix A.2, Table 5] The WISE results are presented without a description of the evaluation protocol (e.g., number of generations per prompt, whether retries are allowed, or how WiScore is computed for this system); the abstract-level claim of surpassing GPT-Image-1 on WISE should be either fully specified or removed from the claims.
  6. [References and Figure 5] Reference [14] is cited as "GPT-eval" in Section 4.3.2 but appears in the bibliography as "Gpt-imgeval"; please align the name and title. In Figure 5, the bar labels "GPT Ours" are ambiguous and should be "GPT-Image-1" and "Ours".

Circularity Check

2 steps flagged · score 6.0 of 10

ComfyBench resolve and pass rates reduce to the system's own evaluator and to a workflow library curated from ComfyBench itself; GenEval and Reason-Edit remain externally scored.

  1. self definitional [Section 3.3; Appendix C.3.3; Section 4.1 (Table 1)]
    "If the result passes evaluation, the search ends successfully. ... Appendix C.3.3: Each evaluation returns two components: a binary judgment indicating whether the current task is considered complete, which guides the system on whether to proceed or backtrack; and a detailed failure analysis ... To accommodate varying tasks and application requirements, the Evaluation Agent supports dynamic threshold adjustment."

    The ComfyBench resolve rate is defined in Section 4.1 as 'whether the output satisfies all task requirements,' but the paper does not describe an external scorer for that judgment. The described scorer is the in-system Evaluation Agent, and the same agent's 'binary judgment' also terminates the search (Section 3.3). A task is counted as resolved exactly when the system's own threshold-adjustable evaluator approves the output; the planner may keep revising until that internal approval is obtained. Consequently the 83% resolve figure is partially a measure of the system accepting its own outputs rather than an independent measure of task satisfaction. This is definitional circularity for the ComfyBench claim, not for GenEval or Reason-Edit, which use external metrics.

  2. fitted input called prediction [Appendix C.2 (workflow library construction) vs. Section 4.1 (ComfyBench evaluation)]
    "First, standard workflows from the ComfyBench benchmark suite were systematically tested. Suboptimal examples were discarded to ensure overall quality. Second, a curated selection of high-quality workflows was extracted from the official ComfyUI website and documentation ... workflows with strong user ratings and proven effectiveness were sourced from popular communities such as Civitai and OpenArt, enhancing the practicality and robustness of the workflow library."

    The atomic workflow library is the system's only source of executable modules, and it was built by testing and curating workflows taken from the ComfyBench benchmark suite itself. The same benchmark is then used in Table 1 to report 100% pass and 83% resolve. This is a fitted-input form of circularity: the test distribution contributed directly to selecting the modules that later 'resolve' its tasks, so the reported ComfyBench numbers are not predictions on an unseen benchmark. The effect is analogous to fitting a retrieval list or module set to the test set and then evaluating with that test set.

full rationale

The strongest independent evidence in the paper is GenEval (0.90) and Reason-Edit (0.906), both of which rely on external scoring conventions and would stand even if the ComfyBench numbers were discounted. The circularity is therefore partial: it attaches to the ComfyBench claim, which is the headline benchmark for autonomous workflow construction. There is no load-bearing self-citation here: ComfyBench is by other authors, and the claimed contributions are not justified by citing the authors' own prior work. However, the ComfyBench numbers are doubly entangled with the method: (i) the only completion signal described for final outputs is the system's own Evaluation Agent, which also decides when search stops and has adjustable thresholds; and (ii) the module library was curated by testing ComfyBench's own workflows. For these reasons the central '100% pass / 83% resolve' result is not a fully external validation of the planning mechanism, though it does not make the whole paper circular.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central claim depends on a hand-curated workflow library, an internal self-evaluation loop, and the reliability of natural-language workflow descriptions. None of these are externally fixed or released, so the system is not fully specified by the paper. No new physical or conceptual entities are introduced; the inventions are architectural.

free parameters (4)
  • Atomic workflow library composition = not released
    The set and quality of atomic workflows determine success; suboptimal workflows were discarded by the authors, making this a hand-selected resource rather than a fixed input.
  • Evaluation dynamic threshold = not reported
    Appendix C.3.3 states the system can raise or lower evaluation standards depending on task complexity; no thresholds are given, and this directly controls the reported resolution rate.
  • Search tree search budget = not reported
    The stopping and backtracking policy depends on unstated hyperparameters such as maximum depth, branching, and retry count; a larger budget can inflate success rates on GenEval and ComfyBench.
  • Prompt optimization dictionary = not released
    The hand-authored style, camera, and atmosphere dictionaries in Appendix E affect prompt quality and therefore benchmark scores, but they are not treated as variables in the evaluation.
assumptions (4)
  • domain assumption The curated atomic workflow library covers the space of user tasks encountered in the benchmarks and beyond.
    ComfyMind can only compose workflows from its library; if a task has no decomposable workflow chain, the system fails. The library is assembled by the authors (Appendix C.2) and not independently validated.
  • domain assumption The internal Evaluation Agent's binary judgments and failure diagnoses are reliable.
    The planning loop stops when this agent approves, and ComfyBench resolve rates are reported using it, so the accuracy of this judge is load-bearing.
  • domain assumption Atomic workflows retrieved from the community are executable and semantically correct for their stated purpose.
    SWI treats each workflow as a callable function; the paper says workflows were tested and suboptimal ones discarded (Appendix C.2), but no automated guarantee of correctness is given.
  • domain assumption Natural-language function descriptions in the SWI document are unambiguous enough for the LLM planner to select the right module.
    The entire planning layer relies on semantic matching between user intent and workflow descriptions; no evidence of disambiguation accuracy is provided.

how reviews work

0 comments
Cite this review

Pith. "Pith review of ComfyMind: Toward General-Purpose Generation via Tree-Based Planning and Reactive Feedback." pith.science (2026). https://pith.science/paper/YGWB3APK

@misc{pith2026250517908,
  author       = {Pith},
  title        = {Pith review of: ComfyMind: Toward General-Purpose Generation via Tree-Based Planning and Reactive Feedback},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/YGWB3APK}},
  note         = {Machine review of arXiv:2505.17908}
}
read the original abstract

With the rapid advancement of generative models, general-purpose generation has gained increasing attention as a promising approach to unify diverse tasks across modalities within a single system. Despite this progress, existing open-source frameworks often remain fragile and struggle to support complex real-world applications due to the lack of structured workflow planning and execution-level feedback. To address these limitations, we present ComfyMind, a collaborative AI system designed to enable robust and scalable general-purpose generation, built on the ComfyUI platform. ComfyMind introduces two core innovations: Semantic Workflow Interface (SWI) that abstracts low-level node graphs into callable functional modules described in natural language, enabling high-level composition and reducing structural errors; Search Tree Planning mechanism with localized feedback execution, which models generation as a hierarchical decision process and allows adaptive correction at each stage. Together, these components improve the stability and flexibility of complex generative workflows. We evaluate ComfyMind on three public benchmarks: ComfyBench, GenEval, and Reason-Edit, which span generation, editing, and reasoning tasks. Results show that ComfyMind consistently outperforms existing open-source baselines and achieves performance comparable to GPT-Image-1. ComfyMind paves a promising path for the development of open-source general-purpose generative AI systems. Project page: https://github.com/LitaoGuo/ComfyMind

Figures

Figures reproduced from arXiv: 2505.17908 by the authors.

Figure 1
Figure 1. Overview of generative and editing capabilities supported by ComfyMind. [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Structural comparison between ours and ComfyAgent. Building on prior work based on ComfyUI [17], such as ComfyAgent [16], an automated solution for generating workflows from natural language instructions was proposed. ComfyAgent employs a multi-pronged mechanism to convert natural language instructions into executable workflows. This approach involves pseudo-code translation to convert JSON structures into Python-li… view at source ↗
Figure 3
Figure 3. Overview of ComfyMind pipeline. Given a user instruction, the system first parses the task [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (16 more)
Figure 4
Figure 4. Figure 4: Qualitative comparison on challenging GenEval [ [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: Quantitative Comparison on Reason-Edit [6] benchmark. Moreover, our method achieves performance competitive with GPT-Image-1 (0.929), narrowing the gap between open and closed models. This gain arises from our sys￾tem’s planning and feedback mechanism, which enables it…
Figure 6
Figure 6. Figure 6: Qualitative Comparison on Reason-edit [6] benchmark. [PITH_FULL_IMAGE:figures/full_fig_p009_6.png]
Figure 7
Figure 7. Figure 7: More examples generated by ComfyMind 15 [PITH_FULL_IMAGE:figures/full_fig_p015_7.png]
Figure 8
Figure 8. Figure 8: More examples generated by ComfyMind 16 [PITH_FULL_IMAGE:figures/full_fig_p016_8.png]
Figure 9
Figure 9. Figure 9: More examples generated by ComfyMind [PITH_FULL_IMAGE:figures/full_fig_p017_9.png]
Figure 10
Figure 10. Figure 10: System Prompt for Preprocessing, Part 1 20 [PITH_FULL_IMAGE:figures/full_fig_p020_10.png]
Figure 11
Figure 11. Figure 11: System Prompt for Preprocessing, Part 2 21 [PITH_FULL_IMAGE:figures/full_fig_p021_11.png]
Figure 12
Figure 12. Figure 12: System Prompt for Preprocessing, Part 3 22 [PITH_FULL_IMAGE:figures/full_fig_p022_12.png]
Figure 13
Figure 13. Figure 13: System Prompt for Planning, Part 1 23 [PITH_FULL_IMAGE:figures/full_fig_p023_13.png]
Figure 14
Figure 14. Figure 14: System Prompt for Planning, Part 2 24 [PITH_FULL_IMAGE:figures/full_fig_p024_14.png]
Figure 15
Figure 15. Figure 15: System Prompt for Planning, Part 3 25 [PITH_FULL_IMAGE:figures/full_fig_p025_15.png]
Figure 16
Figure 16. Figure 16: System Prompt for Tools Definition 26 [PITH_FULL_IMAGE:figures/full_fig_p026_16.png]
Figure 17
Figure 17. Figure 17: System Prompt for Updating Workspace 27 [PITH_FULL_IMAGE:figures/full_fig_p027_17.png]
Figure 18
Figure 18. Figure 18: System Prompt for Workflow Adaptive Adjustment, Part 1 [PITH_FULL_IMAGE:figures/full_fig_p028_18.png]
Figure 19
Figure 19. Figure 19: System Prompt for Workflow Adaptive Adjustment, Part 2 [PITH_FULL_IMAGE:figures/full_fig_p029_19.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Knowledge-Centric Agents for Workflow Generation in ComfyUI

    cs.AI 2026-07 conditional novelty 6.0 of 10

    A knowledge-centric pipeline distills strategies and pseudo-codes from real workflows, fine-tunes a language model on those levels, and reconstructs executable ComfyUI graphs from task descriptions.

Reference graph

Works this paper leans on

80 extracted references · 35 canonical work pages · cited by 1 Pith paper

  1. [1]

    Sdxl: Improving latent diffusion models for high-resolution image synthesis.arXiv preprint arXiv:2307.01952, 2023

    Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis.arXiv preprint arXiv:2307.01952, 2023

    arXiv 2023

  2. [2]

    Scaling rectified flow transformers for high-resolution image synthesis

    Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, et al. Scaling rectified flow transformers for high-resolution image synthesis. InForty-first international conference on machine learning, 2024

    2024

  3. [3]

    Flux, 2024

    Black Forest Labs. Flux, 2024. URLhttps://github.com/black-forest-labs/flux

    2024

  4. [4]

    Instructpix2pix: Learning to follow image editing instructions

    Tim Brooks, Aleksander Holynski, and Alexei A Efros. Instructpix2pix: Learning to follow image editing instructions. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 18392–18402, 2023

    2023

  5. [5]

    Cosxl, 2024

    Stability Ai. Cosxl, 2024. URLhttps://huggingface.co/stabilityai/cosxl

    work page 2024

  6. [6]

    Smartedit: Exploring complex instruction-based image editing with multimodal large language models

    Yuzhou Huang, Liangbin Xie, Xintao Wang, Ziyang Yuan, Xiaodong Cun, Yixiao Ge, Jiantao Zhou, Chao Dong, Rui Huang, Ruimao Zhang, et al. Smartedit: Exploring complex instruction-based image editing with multimodal large language models. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8362–8371, 2024

    2024

  7. [7]

    Ltx-video: Realtime video latent diffusion.arXiv preprint arXiv:2501.00103, 2024

    Yoav HaCohen, Nisan Chiprut, Benny Brazowski, Daniel Shalem, Dudu Moshe, Eitan Richardson, Eran Levin, Guy Shiran, Nir Zabari, Ori Gordon, et al. Ltx-video: Realtime video latent diffusion.arXiv preprint arXiv:2501.00103, 2024

    arXiv 2024

  8. [8]

    Hunyuanvideo, 2024

    Tencent. Hunyuanvideo, 2024. URLhttps://github.com/Tencent/HunyuanVideo

    work page 2024

Show all 80 references
  1. [9]

    Wan: Open and advanced large-scale video generative models.arXiv preprint arXiv:2503.20314, 2025

    Ang Wang, Baole Ai, Bin Wen, Chaojie Mao, Chen-Wei Xie, Di Chen, Feiwu Yu, Haiming Zhao, Jianxiao Yang, Jianyuan Zeng, et al. Wan: Open and advanced large-scale video generative models.arXiv preprint arXiv:2503.20314, 2025

    2025 arXiv

  2. [10]

    Seed-x: Multimodal models with unified multi-granularity comprehension and generation.arXiv preprint arXiv:2404.14396, 2024

    Yuying Ge, Sijie Zhao, Jinguo Zhu, Yixiao Ge, Kun Yi, Lin Song, Chen Li, Xiaohan Ding, and Ying Shan. Seed-x: Multimodal models with unified multi-granularity comprehension and generation.arXiv preprint arXiv:2404.14396, 2024

    2024 arXiv

  3. [11]

    Janus: Decoupling visual encoding for unified multimodal understanding and generation.arXiv preprint arXiv:2410.13848, 2024

    Chengyue Wu, Xiaokang Chen, Zhiyu Wu, Yiyang Ma, Xingchao Liu, Zizheng Pan, Wen Liu, Zhenda Xie, Xingkai Yu, Chong Ruan, et al. Janus: Decoupling visual encoding for unified multimodal understanding and generation.arXiv preprint arXiv:2410.13848, 2024

    arXiv

  4. [12]

    Got: Unleashing reasoning capability of multimodal large language model for visual generation and editing.arXiv preprint arXiv:2503.10639, 2025

    Rongyao Fang, Chengqi Duan, Kun Wang, Linjiang Huang, Hao Li, Shilin Yan, Hao Tian, Xingyu Zeng, Rui Zhao, Jifeng Dai, et al. Got: Unleashing reasoning capability of multimodal large language model for visual generation and editing.arXiv preprint arXiv:2503.10639, 2025

    2025 arXiv

  5. [13]

    gpt-image-1, 2025

    OpenAI. gpt-image-1, 2025. URLhttps://platform.openai.com/docs/models/gpt-image-1

    2025

  6. [14]

    Gpt-imgeval: A comprehensive benchmark for diagnosing gpt4o in image generation.arXiv preprint arXiv:2504.02782, 2025

    Zhiyuan Yan, Junyan Ye, Weijia Li, Zilong Huang, Shenghai Yuan, Xiangyang He, Kaiqing Lin, Jun He, Conghui He, and Li Yuan. Gpt-imgeval: A comprehensive benchmark for diagnosing gpt4o in image generation.arXiv preprint arXiv:2504.02782, 2025

    2025 arXiv

  7. [15]

    Comfygen: Prompt-adaptive workflows for text-to-image generation.arXiv preprint arXiv:2410.01731, 2024

    Rinon Gal, Adi Haviv, Yuval Alaluf, Amit H Bermano, Daniel Cohen-Or, and Gal Chechik. Comfygen: Prompt-adaptive workflows for text-to-image generation.arXiv preprint arXiv:2410.01731, 2024

    2024 arXiv

  8. [16]

    Comfybench: Benchmark- ing llm-based agents in comfyui for autonomously designing collaborative ai systems.URL https://arxiv

    Xiangyuan Xue, Zeyu Lu, Di Huang, Zidong Wang, Wanli Ouyang, and Lei Bai. Comfybench: Benchmark- ing llm-based agents in comfyui for autonomously designing collaborative ai systems.URL https://arxiv. org/abs/2409.01392, 2024

    2024

  9. [17]

    ComfyUI: A powerful and modular stable-diffusion gui

    ComfyUI Contributors. ComfyUI: A powerful and modular stable-diffusion gui. https://github.com/ comfyanonymous/ComfyUI, 2023. Accessed: 2025-05-14. 10

    2023

  10. [18]

    Retrieval-augmented generation for knowledge- intensive nlp tasks.Advances in neural information processing systems, 33:9459–9474, 2020

    Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. Retrieval-augmented generation for knowledge- intensive nlp tasks.Advances in neural information processing systems...

    2020

  11. [19]

    Adding conditional control to text-to-image diffusion models

    Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. InProceedings of the IEEE/CVF international conference on computer vision, pages 3836–3847, 2023

    2023

  12. [20]

    Ip-adapter: Text compatible image prompt adapter for text-to-image diffusion models.arXiv preprint arXiv:2308.06721, 2023

    Hu Ye, Jun Zhang, Sibo Liu, Xiao Han, and Wei Yang. Ip-adapter: Text compatible image prompt adapter for text-to-image diffusion models.arXiv preprint arXiv:2308.06721, 2023

    2023 arXiv

  13. [21]

    An image is worth one word: Personalizing text-to-image generation using textual inversion

    Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit H Bermano, Gal Chechik, and Daniel Cohen-Or. An image is worth one word: Personalizing text-to-image generation using textual inversion. arXiv preprint arXiv:2208.01618, 2022

    2022 arXiv

  14. [22]

    Janusflow: Harmonizing autoregression and rectified flow for unified multimodal understanding and generation.arXiv preprint arXiv:2411.07975, 2024

    Yiyang Ma, Xingchao Liu, Xiaokang Chen, Wen Liu, Chengyue Wu, Zhiyu Wu, Zizheng Pan, Zhenda Xie, Haowei Zhang, Liang Zhao, et al. Janusflow: Harmonizing autoregression and rectified flow for unified multimodal understanding and generation.arXiv preprint arXiv:2411.07975, 2024

    2024 arXiv

  15. [23]

    Chameleon: Mixed-modal early-fusion foundation models.arXiv preprint arXiv:2405.09818, 2024

    Chameleon Team. Chameleon: Mixed-modal early-fusion foundation models.arXiv preprint arXiv:2405.09818, 2024

    2024 arXiv

  16. [24]

    Emu3: Next-token prediction is all you need.arXiv preprint arXiv:2409.18869, 2024

    Xinlong Wang, Xiaosong Zhang, Zhengxiong Luo, Quan Sun, Yufeng Cui, Jinsheng Wang, Fan Zhang, Yueze Wang, Zhen Li, Qiying Yu, et al. Emu3: Next-token prediction is all you need.arXiv preprint arXiv:2409.18869, 2024

    2024 arXiv

  17. [25]

    Vila-u: a unified foundation model integrating visual understanding and generation.arXiv preprint arXiv:2409.04429, 2024

    Yecheng Wu, Zhuoyang Zhang, Junyu Chen, Haotian Tang, Dacheng Li, Yunhao Fang, Ligeng Zhu, Enze Xie, Hongxu Yin, Li Yi, et al. Vila-u: a unified foundation model integrating visual understanding and generation.arXiv preprint arXiv:2409.04429, 2024

    2024 arXiv

  18. [26]

    Show-o: One single transformer to unify multimodal understanding and generation.arXiv preprint arXiv:2408.12528, 2024

    Jinheng Xie, Weijia Mao, Zechen Bai, David Junhao Zhang, Weihao Wang, Kevin Qinghong Lin, Yuchao Gu, Zhijie Chen, Zhenheng Yang, and Mike Zheng Shou. Show-o: One single transformer to unify multimodal understanding and generation.arXiv preprint arXiv:2408.12528, 2024

    2024 arXiv

  19. [27]

    Monoformer: One transformer for both diffusion and autoregression.arXiv preprint arXiv:2409.16280, 2024

    Chuyang Zhao, Yuxing Song, Wenhao Wang, Haocheng Feng, Errui Ding, Yifan Sun, Xinyan Xiao, and Jingdong Wang. Monoformer: One transformer for both diffusion and autoregression.arXiv preprint arXiv:2409.16280, 2024

    2024 arXiv

  20. [28]

    Omnigen: Unified image generation.arXiv preprint arXiv:2409.11340, 2024

    Shitao Xiao, Yueze Wang, Junjie Zhou, Huaying Yuan, Xingrun Xing, Ruiran Yan, Chaofan Li, Shut- ing Wang, Tiejun Huang, and Zheng Liu. Omnigen: Unified image generation.arXiv preprint arXiv:2409.11340, 2024

    2024 arXiv

  21. [29]

    Janus-pro: Unified multimodal understanding and generation with data and model scaling.arXiv preprint arXiv:2501.17811, 2025

    Xiaokang Chen, Zhiyu Wu, Xingchao Liu, Zizheng Pan, Wen Liu, Zhenda Xie, Xingkai Yu, and Chong Ruan. Janus-pro: Unified multimodal understanding and generation with data and model scaling.arXiv preprint arXiv:2501.17811, 2025

    2025 arXiv

  22. [30]

    AutoGPT.https://github.com/Significant-Gravitas/AutoGPT, 2023

    Toran Bruce Richards. AutoGPT.https://github.com/Significant-Gravitas/AutoGPT, 2023

    2023

  23. [31]

    Autogen: Enabling next-gen llm applications via multi-agent conversation

    Qingyun Wu, Gagan Bansal, Jieyu Zhang, Yiran Wu, Beibin Li, Erkang Zhu, Li Jiang, Xiaoyun Zhang, Shaokun Zhang, Jiale Liu, et al. Autogen: Enabling next-gen llm applications via multi-agent conversation. arXiv preprint arXiv:2308.08155, 2023

    2023 arXiv

  24. [32]

    Openagents: An open platform for language agents in the wild.arXiv preprint arXiv:2310.10634, 2023

    Tianbao Xie, Fan Zhou, Zhoujun Cheng, Peng Shi, Luoxuan Weng, Yitao Liu, Toh Jing Hua, Junning Zhao, Qian Liu, Che Liu, et al. Openagents: An open platform for language agents in the wild.arXiv preprint arXiv:2310.10634, 2023

    arXiv

  25. [33]

    Re- act: Synergizing reasoning and acting in language models

    Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. Re- act: Synergizing reasoning and acting in language models. InInternational Conference on Learning Representations (ICLR), 2023

    2023

  26. [34]

    Metagpt: Meta programming for multi-agent collaborative framework.arXiv preprint arXiv:2308.00352, 3(4):6, 2023

    Sirui Hong, Xiawu Zheng, Jonathan Chen, Yuheng Cheng, Jinlin Wang, Ceyao Zhang, Zili Wang, Steven Ka Shing Yau, Zijuan Lin, Liyang Zhou, et al. Metagpt: Meta programming for multi-agent collaborative framework.arXiv preprint arXiv:2308.00352, 3(4):6, 2023

    2023 arXiv

  27. [35]

    Flux.1-redux-dev, 2024

    Black Forest Labs. Flux.1-redux-dev, 2024. URL https://huggingface.co/black-forest-labs/ FLUX.1-Redux-dev

    2024

  28. [36]

    Ace++: Instruction-based image creation and editing via context-aware content filling.arXiv preprint arXiv:2501.02487, 2025

    Chaojie Mao, Jingfeng Zhang, Yulin Pan, Zeyinzi Jiang, Zhen Han, Yu Liu, and Jingren Zhou. Ace++: Instruction-based image creation and editing via context-aware content filling.arXiv preprint arXiv:2501.02487, 2025. 11

    2025 arXiv

  29. [37]

    Segment anything

    Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer Whitehead, Alexander C Berg, Wan-Yen Lo, et al. Segment anything. In2023 IEEE/CVF International Conference on Computer Vision (ICCV), pages 3992–4003. IEEE, 2023

    2023

  30. [38]

    Sam 2: Segment anything in images and videos.arXiv preprint arXiv:2408.00714, 2024

    Nikhila Ravi, Valentin Gabeur, Yuan-Ting Hu, Ronghang Hu, Chaitanya Ryali, Tengyu Ma, Haitham Khedr, Roman Rädle, Chloe Rolland, Laura Gustafson, et al. Sam 2: Segment anything in images and videos.arXiv preprint arXiv:2408.00714, 2024

    2024 arXiv

  31. [39]

    React: Synergizing reasoning and acting in language models

    Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. InThe Eleventh International Conference on Learning Representations

  32. [40]

    Geneval: An object-focused framework for evaluating text-to-image alignment.Advances in Neural Information Processing Systems, 36:52132–52152, 2023

    Dhruba Ghosh, Hannaneh Hajishirzi, and Ludwig Schmidt. Geneval: An object-focused framework for evaluating text-to-image alignment.Advances in Neural Information Processing Systems, 36:52132–52152, 2023

    2023

  33. [41]

    Language models are few-shot learners

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877–1901, 2020

    1901

  34. [42]

    Chain-of-thought prompting elicits reasoning in large language models.Advances in neural information processing systems, 35:24824–24837, 2022

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models.Advances in neural information processing systems, 35:24824–24837, 2022

    2022

  35. [43]

    Self-consistency improves chain of thought reasoning in language models.arXiv preprint arXiv:2203.11171, 2022

    Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models.arXiv preprint arXiv:2203.11171, 2022

    2022 arXiv

  36. [44]

    High-resolution image synthesis with latent diffusion models

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10684–10695, 2022

    2022

  37. [45]

    Hierarchical text-conditional image generation with clip latents.arXiv preprint arXiv:2204.06125, 1(2):3, 2022

    Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents.arXiv preprint arXiv:2204.06125, 1(2):3, 2022

    2022 arXiv

  38. [46]

    Autore- gressive model beats diffusion: Llama for scalable image generation.arXiv preprint arXiv:2406.06525, 2024

    Peize Sun, Yi Jiang, Shoufa Chen, Shilong Zhang, Bingyue Peng, Ping Luo, and Zehuan Yuan. Autore- gressive model beats diffusion: Llama for scalable image generation.arXiv preprint arXiv:2406.06525, 2024

    2024 arXiv

  39. [47]

    World model on million-length video and language with ringattention.arXiv e-prints, pages arXiv–2402, 2024

    Hao Liu, Wilson Yan, Matei Zaharia, and Pieter Abbeel. World model on million-length video and language with ringattention.arXiv e-prints, pages arXiv–2402, 2024

    2024

  40. [48]

    Guiding instruction- based image editing via multimodal large language models.arXiv preprint arXiv:2309.17102, 2023

    Tsu-Jui Fu, Wenze Hu, Xianzhi Du, William Yang Wang, Yinfei Yang, and Zhe Gan. Guiding instruction- based image editing via multimodal large language models.arXiv preprint arXiv:2309.17102, 2023

    2023 arXiv

  41. [49]

    Magicbrush: A manually annotated dataset for instruction-guided image editing.Advances in Neural Information Processing Systems, 36:31428–31449, 2023

    Kai Zhang, Lingbo Mo, Wenhu Chen, Huan Sun, and Yu Su. Magicbrush: A manually annotated dataset for instruction-guided image editing.Advances in Neural Information Processing Systems, 36:31428–31449, 2023

    2023

  42. [50]

    Wise: A world knowledge-informed semantic evaluation for text-to-image generation.arXiv preprint arXiv:2503.07265, 2025

    Yuwei Niu, Munan Ning, Mengren Zheng, Bin Lin, Peng Jin, Jiaqi Liao, Kunpeng Ning, Bin Zhu, and Li Yuan. Wise: A world knowledge-informed semantic evaluation for text-to-image generation.arXiv preprint arXiv:2503.07265, 2025

    2025 arXiv

  43. [51]

    Fast training of diffusion transformer for photorealistic text-to-image synthesis.arXiv preprint arXiv:2310.00426, 2023

    Junsong Chen, Jincheng Yu, Chongjian Ge, Lewei Yao, Enze Xie, Yue Wu, Zhongdao Wang, James Kwok, Ping Luo, Huchuan Lu, et al. Fast training of diffusion transformer for photorealistic text-to-image synthesis.arXiv preprint arXiv:2310.00426, 2023

    arXiv

  44. [52]

    Playground v2

    Daiqing Li, Aleks Kamko, Ehsan Akhgari, Ali Sabet, Linmiao Xu, and Suhail Doshi. Playground v2. 5: Three insights towards enhancing aesthetic quality in text-to-image generation.arXiv preprint arXiv:2402.17245, 2024

    2024 arXiv

  45. [53]

    if execution succeeds, proceed to the next subtask; if it fails, return to the current node for local backtracking

    Siqi Kou, Jiachun Jin, Zhihong Liu, Chang Liu, Ye Ma, Jian Jia, Quan Chen, Peng Jiang, and Zhijie Deng. Orthus: Autoregressive interleaved image-text generation with modality-specific heads.arXiv preprint arXiv:2412.00127, 2024. 12 A Supplementary Experiments A.1 Supplementary...

    2024 arXiv

  46. [54]

    A close-up of an old scholar's hands delicately flipping the pages of an ancient manuscript

    Camera Language Shot Types (Framing): Close-up Shot: Captures fine details, expressions, or objects in high focus.Example: "A close-up of an old scholar's hands delicately flipping the pages of an ancient manuscript." Medium Shot: Shows the subject from the waist up, providing...

  47. [55]

    A lively marketplace where children laugh and vendors showcase colorful handmade goods under warm sunlight

    Atmosphere (Mood & Emotion) Energetic / Joyful / Uplifting: Bright lighting, vibrant colors, and lively movement.Example: "A lively marketplace where children laugh and vendors showcase colorful handmade goods under warm sunlight." Dreamlike / Surreal / Mystical: Soft focus, f...

  48. [56]

    A hacker in a hooded jacket, surrounded by glowing holographic data streams in a futuristic Tokyo street

    Style (Artistic Direction) Cyberpunk: Neon lights, dark cityscapes, high-tech elements.Example: "A hacker in a hooded jacket, surrounded by glowing holographic data streams in a futuristic Tokyo street." Post-Apocalyptic (Wasteland Style): Rugged, destroyed environments, muted...

  49. [57]

    Carefully analyze the current input and search history

  50. [58]

    Evaluate the applicability of available workflows

  51. [59]

    *Must* consider previous failed workflow attempts—workflows that have been recorded as incapable of completing a task should not be selected again

  52. [60]

    Apply logical reasoning: if a workflow has been recorded as failing for a follow-up task (not the current one due to possible workflow errors), selecting a similar workflow should be done with caution

  53. [61]

    Achieving the desired result is the top priority! If there are no suitable workflows to proceed with the task, but reordering the workflow sequence can help, the generation order requested by the user can be adjusted when necessary

  54. [62]

    If no single workflow can advance the task, think deeply and creatively—combine multiple workflows and execute them sequentially to complete the task

  55. [63]

    If no suitable workflow can proceed with the task, return a failure signal

    Do not arbitrarily select workflows—only choose them if they can advance the task. If no suitable workflow can proceed with the task, return a failure signal

  56. [64]

    Think step by step: if only one workflow call is needed, remaining_steps = 0

    Based on the planned workflow sequence, determine the remaining steps (remaining_steps). Think step by step: if only one workflow call is needed, remaining_steps = 0. If two sequential workflow calls are needed, remaining_steps = 1, and so on

  57. [65]

    The Object

    You need to be aware of the difference between instructions and prompts. Instructions are descriptions of tasks, while prompts are just descriptions of the images in the tasks. For example, a task to remove an object should have the prompt "The Object" instead of "Remove the object"

  58. [66]

    Analyze whether the user has additional requirements for the generated result, such as video duration, resolution, frame rate, upscaling factor, object placement, position, or any other requirements not covered by tool_input

  59. [67]

    Note: Additional requirements for generation results do not include vague terms like high quality, seamless integration, without visible artifacts, etc

  60. [68]

    Replace Object

    If the context indicates that a certain workflow cannot complete the task, it is strictly forbidden to select that workflow again. Instead, try a more complex, multi-step workflow chain to solve the problem. For example, if a direct "Replace Object" workflow fails, do not sele...

  61. [69]

    Create a video of the cityscape with the perspective changing based on the image

    Use past failed results to refine workflow selection and modify additional requirement parameters. For example, if an evaluation shows that the generated video duration is too short, recognize the user's intended duration and add it to additional_requirements. # Tools You have...

  62. [70]

    Totally follow the JSON schema of the input

    Your output should be a JSON object. Totally follow the JSON schema of the input

  63. [71]

    Then add the content of the new added input parameters and them to instruction

    You should read the information of the workflow and the chain of thought, accroding the workflow's function guess what steps you have completed and what steps you may next complete. Then add the content of the new added input parameters and them to instruction. This may includ...

  64. [72]

    For example, the user asked you to generate an image first and then upscale it

    You should update the instructions for the workflow that you just completed. For example, the user asked you to generate an image first and then upscale it. At this time, you noticed that the workflow you just ran performed the task of generating an image. At this time, you sh...

  65. [73]

    For example, The instruction:generate an image with a resolution of XXX or a video with a duration of XXX

    You should pay attention to the timeliness of the user's instructions. For example, The instruction:generate an image with a resolution of XXX or a video with a duration of XXX. Such instructions are permanent. Therefore, it should continue to be passed, and at the same time r...

  66. [74]

    file_meta_info

    *IMPORTANT* You MUST add information and introduction for the new generated file to "file_meta_info"(*Do not* leave it empty)

  67. [75]

    the image, the video, ...etc.)

    *IMPORTANT* You MUST maintain ALL Previous step 'file_meta_info' of the previous files(e.g. the image, the video, ...etc.). If the previous files are not mentioned in the 'file_meta_info', you should add them to the 'file_meta_info'. E.g: This image is the original input image...

  68. [76]

    - Identify any specific features or areas relevant to the user's request (e.g., determining where to place an object like a small dog)

    Analyze Visual Input: - Carefully examine the provided visual content to understand its key elements and context. - Identify any specific features or areas relevant to the user's request (e.g., determining where to place an object like a small dog)

  69. [77]

    - Ensure you fully grasp the requirements before proceeding with any changes

    Understand User Requirements: - Clearly interpret the user's specific demands, such as adding an object to an image or modifying certain aspects of the workflow. - Ensure you fully grasp the requirements before proceeding with any changes. - Ignore requests about generation qu...

  70. [78]

    - Identify which hyperparameters control the aspects you need to modify based on the user's request (e.g

    Review Workflow Structure: - Analyze the given workflow to understand its current structure and node connections. - Identify which hyperparameters control the aspects you need to modify based on the user's request (e.g. The length of the video is controled by video frames and ...

  71. [79]

    - Base your modifications on the analysis of the visual input and the user's specific demands

    Modify Hyperparameters: - Adjust only the necessary hyperparameters to meet the user's requirements, ensuring the workflow's structure and node connections remain unchanged. - Base your modifications on the analysis of the visual input and the user's specific demands

  72. [80]

    - Ensure the output is easy to understand and directly addresses the user's needs

    Provide Clear Output: - Return the modified workflow with a clear explanation of the changes made. - Ensure the output is easy to understand and directly addresses the user's needs. Workflow Adaptive Adjustment Figure 18: System Prompt for Workflow Adaptive Adjustment, Part 1 ...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.