Pith. sign in

REVIEW 3 major objections 4 minor 3 cited by

ComfyUI-R1: Exploring Reasoning Models for Workflow Generation

T0 review · 3 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A 7-billion-parameter model writes executable ComfyUI workflows at a 97 percent validity rate.

desk verdict A solid RL-for-workflow application with honest scoping on the main test set, but the ComfyBench comparison is uncontrolled and the node reward ignores precision; deserves peer review with revisions. read the letter →

arxiv 2506.09790 v1 pith:BT46BYYO submitted 2025-06-11 cs.CL cs.CVcs.SE

Zhenran Xu , Yiyu Wang , Xue Yang , Longyue Wang , Weihua Luo , Kaifu Zhang , Baotian Hu , Min Zhang This is my paper
classification cs.CLcs.CVcs.SE
keywords workflowgenerationComfyUIchain-of-thoughtreasoningreinforcementlearningGRPOnodeselectionmultimodalcontentcreationmodel
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper's thesis is that automated workflow generation for ComfyUI, turning a natural-language instruction into an executable graph of image- and video-generation nodes, does not require a powerful closed API; a 7B open-weights model can be post-trained to do it better. ComfyUI-R1 starts from Qwen2.5-Coder-7B and is first fine-tuned on long chain-of-thought traces that select nodes, state a design principle, and emit a Python-like code representation of the workflow. A reinforcement-learning stage with GRPO then shapes the reasoning, using a veto-based hybrid reward that penalizes format errors, invalid directed acyclic graphs, and hallucinated nodes, while rewarding correct node selection. The trained model reaches 97% format validity, node-level F1 of 0.62 and graph-level F1 of 0.51 on the authors' test set, above every GPT-4o and Claude prompting baseline, and raises ComfyBench pass rate to 0.67 from ComfyAgent's 0.56. If these results hold, they suggest that reasoning-model training can transfer from math to structured creative tasks with only a well-shaped reward.

What carries the argument

The load-bearing machinery is two-stage post-training with a structured action space. Stage one is supervised fine-tuning on distilled long chain-of-thought data: each example asks the model to output a node-selection block, a design-principle rationale, and a workflow expressed as Python-like function calls in topological order, reversibly convertible to ComfyUI JSON. Stage two is Group Relative Policy Optimization (GRPO), a reinforcement-learning method that samples a group of outputs for the same input and assigns each an advantage from its reward relative to the group. The reward used in training is a veto-based hybrid: format reward checks that all required tags parse, structure reward checks that the workflow forms a valid directed acyclic graph, node-fidelity reward penalizes nodes outside the candidate set or inconsistent between the selection block and the code, and only if all three pass is a precision-style node-selection reward applied; any single veto sets total reward to -1. This design is what lets the model maximize reasoning quality while hard constraints remain non-negotiable.

What would settle it

Give the model the same 600 test instructions but remove the candidate-node menu entirely, or perturb Vcand so that some gold nodes are missing, and recompute format validity, node-level F1, and execution pass rate; if these collapse toward the few-shot baselines, the claimed reasoning advantage depends on the candidate-menu protocol.

Watch

Extended reading notes

Core claim

The central claim, stated on the authors' terms, is that ComfyUI-R1 is the first large reasoning model for automated workflow generation and that its two-stage recipe, cold-start CoT supervised fine-tuning followed by GRPO reinforcement learning with a fine-grained rule-metric hybrid reward, is what enables a 7B parameter model to outperform prior state-of-the-art systems built on GPT-4o and Claude. On the in-house test set, format validity reaches 97% versus 41% for the untrained backbone, while node-level F1 is 0.62 and graph-level F1 is 0.51, both above all baselines. The ablation shows that the RL stage improves format validity from 95% to 97%, and that representing workflows as Python-like code outperforms representing them as JSON. On ComfyBench, which tests end-to-end retrieval and generation without a supplied candidate node set, the model achieves a 0.67 pass rate, an absolute 11% gain over the GPT-4o-based ComfyAgent.

Load-bearing premise

The load-bearing premise is that the evaluation protocol, which hands the model a candidate node list that already contains every correct node plus a small random set of decoys, fairly represents how users will actually get node suggestions; if retrieval errors or the full-size node space make the real candidate list less helpful, the reported gains shrink.

Editorial extensions

If this is right

  • A 7B open-weights model can beat prompt-only systems powered by GPT-4o and Claude 3.5/3.7 on a structured generation task, so workflow-generation capability is trainable rather than dependent on API-scale models.
  • With 97% format validity, the model's outputs can almost always be parsed into executable ComfyUI graphs, making workflow generation usable as a component in a copilot rather than a suggestion tool.
  • The code-representation advantage over JSON indicates that a structured, semantically rich action space helps the model both plan and conform to graph constraints.
  • The veto-based hybrid reward provides a transferable recipe: enforce hard constraints as binary gates and let reinforcement learning optimize only the soft quality metric inside those gates.
  • The 0.67 ComfyBench pass rate means a majority of generated workflows actually execute on the ComfyUI server, which is the strongest evidence that the benefit reaches real retrieval-plus-generation use.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The main test set's candidate-node protocol, which injects the gold nodes plus a small random decoy set, isolates node selection from retrieval; a reader should treat the ComfyBench number as the more realistic estimate of deployed performance, and even 0.67 likely degrades as the node space grows.
  • The same SFT-plus-GRPO recipe with a code-level DAG action space should transfer to other low-code creative tools, such as Blender geometry nodes or video-editing graphs, where the hard constraints are format, graph validity, and node fidelity.
  • A cheap testable extension is to vary the number of random candidates in the protocol: if performance holds when decoys are many or when gold nodes are partly withheld, retrieval is not the bottleneck; if it collapses, retrieval becomes the priority.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper presents ComfyUI-R1, a 7B-parameter reasoning model for generating ComfyUI workflows from natural-language instructions. The authors collect a knowledge base of 3,917 workflows and 7,238 nodes, synthesize long chain-of-thought training data (node selection, design rationale, code representation), fine-tune Qwen2.5-Coder-7B-Instruct on this data, and then apply GRPO with a rule-metric hybrid reward combining format validity, DAG structure, node fidelity, and node-selection accuracy. Evaluation on a self-built 600-sample test set reports 97% format validity, node-level F1 0.62, and graph-level F1 0.51, outperforming few-shot and CoT baselines with GPT-4o and Claude models. An end-to-end experiment on ComfyBench reports a 0.67 pass rate versus 0.56 for ComfyAgent, leading the authors to claim that their 7B model significantly surpasses prior state-of-the-art methods built on larger closed-source models.

Significance. If the results hold, the paper would be a useful demonstration that a 7B open model can outperform much larger closed models on structured workflow generation through SFT followed by reinforcement learning with a custom reward. The comparison of code-based versus JSON-based workflow representations, the detailed knowledge-base construction, and the integration into ComfyUI-Copilot are practical contributions. The reasoning-data generation and reward design are described clearly enough to be replicated, modulo the use of proprietary LLMs to produce rationales and descriptions. However, the strongest end-to-end claim rests on a comparison that is not currently controlled, and the node-selection reward is recall-only, so the evidence for the headline superiority is weaker than the text suggests.

major comments (3)
  1. [Sec. 4.5, Table 3] The 0.67 versus 0.56 pass-rate comparison is not a controlled head-to-head. The ComfyAgent and GPT-4o numbers are taken from the original paper, while ComfyUI-R1 is evaluated with its own retrieval pipeline (top-3 workflows from a 3,917-workflow KB embedded with text-embedding-3-small) and an execution harness that is not shown to be identical to the one used for the published baselines. Because the KB is scraped from ComfyUI community sites (Sec. 3.1) and ComfyBench tasks are themselves ComfyUI workflows, the paper must quantify overlap or near-duplicates between the KB and the ComfyBench test set; if gold workflows leak into retrieval, Vcand can contain the correct nodes and inflate the pass rate. I ask for an overlap analysis, a re-run of ComfyAgent under the same retrieval and execution conditions, or an ablation that isolates the generator from the retrieval component.
  2. [Sec. 3.3, Eq. (5)] The node-selection reward R_correct = |Vp∩Vg|/|Vg| - 1 is recall-only. It reaches its maximum of 0 whenever all gold nodes are included, regardless of how many additional valid but non-gold nodes are selected; R_fidelity only penalizes nodes outside Vcand and mismatches between the selected_nodes block and the workflow block, not extra in-candidate nodes. Thus the reward does not penalize a significant class of over-generation, which is also consistent with the case study's observation that ComfyUI-R1 produces more nodes than ComfyAgent. I recommend adding a precision or cardinality term and, at minimum, reporting reward variants in the ablation study.
  3. [Sec. 4.1, Table 1] The main test set is evaluated with an oracle candidate set that always contains the gold node set, which the paper acknowledges as an evaluation of reasoning over pre-retrieved nodes. This is a fair across-model protocol, but it means the high absolute F1 scores and the format-validity rate do not by themselves establish end-to-end usability; the only end-to-end evidence is the ComfyBench experiment, whose control issues are raised above. The paper should either add end-to-end results on its own test set or clearly state in the abstract and conclusion that the superiority claims are for the pre-retrieved-node setting.
minor comments (4)
  1. [Eq. (9)] The displayed KL divergence is not the standard KL(πθ||πref); as written it places expectations under πref and appears to be the reverse KL with a -1 offset. Please correct the equation or clarify the convention used.
  2. [Sec. 3.2] The split statement is numerically inconsistent: 3,717 workflows split into training and 200 test should yield 3,517 training workflows, and with three generated samples per workflow (Qwen-Max, Claude 3.5, GPT-4o) the expected counts are 10,551 training and 600 test samples, not 11,140. Please clarify the sample-generation process.
  3. [Sec. 4.3] The implementation details report the main hyperparameters, but the paper does not state the exact RL training data size, the inference prompt template, or whether the model and training code will be released. Please provide a repository link or state the release plan for reproducibility.
  4. [Sec. 4.6, Fig. 3-5] The qualitative claims that ComfyUI-R1's outputs 'adhere' to style and 'seamlessly combine' images are subjective; adding a small quantitative metric, such as CLIP similarity of generated images to the instruction, would strengthen the case study.

Circularity Check

0 steps flagged · score 1.0 of 10

No material circularity: the training and evaluation pipeline is metric-aligned but not circular; borrowed ComfyBench baseline numbers are a comparison-fairness issue, not a definitional reduction.

full rationale

I walked the paper's derivation chain: SFT data are constructed from a curated workflow KB (Sec. 3.1-3.2), the model is trained with a rule-metric hybrid reward (Sec. 3.3) under GRPO (Sec. 3.4), and evaluation uses held-out test instructions plus ComfyBench (Sec. 4.5). No step defines X in terms of Y such that the claimed prediction is identical to its input by construction. The candidate-node protocol Vcand = Vg ∪ Vrandom (Sec. 3.2) does give the model the gold nodes during both training and test, and the paper explicitly discloses that this "evaluation setup is designed to assess the model's ability to reason over pre-retrieved node information, rather than its retrieval capability" (Sec. 4.1). That is a scoped evaluation choice, not a circular derivation. Likewise, R_correct = |Vp∩Vg|/|Vg| - 1 (Eq. 5) is directly aligned with the node-level F1 metric, but aligning a reward with the target metric is standard training practice, not a fitted-parameter-called-prediction. The strongest non-circularity concern is Table 3: "The numbers of GPT-4o methods are taken from the original paper [Xue et al. 2024]," while ComfyUI-R1 uses its own retrieval over a 3,917-workflow KB. This makes the 0.67 vs 0.56 comparison uncontrolled and should be weighed as a correctness/robustness risk, but it is not an input-output equivalence. Self-citations (ComfyUI-Copilot, Marco-o1, Anim-Director, etc.) are contextual or related-work citations and are not load-bearing for the central claim. Overall, the central result is an empirical trained-model comparison, self-contained against the external ComfyBench benchmark modulo the borrowed baseline numbers, with no identifiable circular step.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The central empirical claims rest on a constructed dataset, parser correctness, and an evaluation protocol; no physics-like axioms are involved. The most important assumptions are the reliability of the parsed workflow graph, the accuracy of LLM-generated labels, the validity of structural graph matching as a quality proxy, and the correctness of the community-sourced ground truth.

free parameters (2)
  • Distractor ratio in candidate node set = 0.8 x |V_g|
    Sec. 3.2 constructs candidate sets by adding 80% as many random nodes as gold nodes. This ratio is chosen by hand and shapes the difficulty of all SFT and RL examples; no sensitivity analysis is given.
  • Number of retrieved workflows in end-to-end setting = 3
    Sec. 4.5 retrieves the top 3 semantically similar workflows to build Vcand; this number is chosen without ablation and affects the ComfyBench comparison.
assumptions (4)
  • domain assumption The JSON-to-code and code-to-JSON parsers are assumed reversible and semantically faithful.
    Sec. 3.1 retains only workflows that survive round-trip conversion, and all training and evaluation labels depend on this parser.
  • domain assumption LLM-generated workflow descriptions and reasoning rationales are assumed accurate reflections of what the workflows do.
    Sec. 3.2 uses Qwen-Max, Claude 3.5, and GPT-4o to write instructions and rationales for 3,917 workflows; if these are noisy, the SFT labels and the test set inherit that noise.
  • domain assumption WorFEval-style LIS and MCIS graph matching is assumed to measure workflow quality.
    Sec. 4.1 uses these graph-level metrics as the primary quality signal; they are structural matches, not direct measures of whether the final image matches the user instruction.
  • domain assumption Community-sourced workflows used as ground truth are assumed correct and well-designed.
    Sec. 3.1 collects 27K workflows from community sources and keeps 3,917 after cleaning; the gold node sets for training and testing come from these community workflows.

how reviews work

0 comments
Cite this review

Pith. "Pith review of ComfyUI-R1: Exploring Reasoning Models for Workflow Generation." pith.science (2026). https://pith.science/paper/BT46BYYO

@misc{pith2026250609790,
  author       = {Pith},
  title        = {Pith review of: ComfyUI-R1: Exploring Reasoning Models for Workflow Generation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/BT46BYYO}},
  note         = {Machine review of arXiv:2506.09790}
}
read the original abstract

AI-generated content has evolved from monolithic models to modular workflows, particularly on platforms like ComfyUI, enabling customization in creative pipelines. However, crafting effective workflows requires great expertise to orchestrate numerous specialized components, presenting a steep learning curve for users. To address this challenge, we introduce ComfyUI-R1, the first large reasoning model for automated workflow generation. Starting with our curated dataset of 4K workflows, we construct long chain-of-thought (CoT) reasoning data, including node selection, workflow planning, and code-level workflow representation. ComfyUI-R1 is trained through a two-stage framework: (1) CoT fine-tuning for cold start, adapting models to the ComfyUI domain; (2) reinforcement learning for incentivizing reasoning capability, guided by a fine-grained rule-metric hybrid reward, ensuring format validity, structural integrity, and node-level fidelity. Experiments show that our 7B-parameter model achieves a 97\% format validity rate, along with high pass rate, node-level and graph-level F1 scores, significantly surpassing prior state-of-the-art methods that employ leading closed-source models such as GPT-4o and Claude series. Further analysis highlights the critical role of the reasoning process and the advantage of transforming workflows into code. Qualitative comparison reveals our strength in synthesizing intricate workflows with diverse nodes, underscoring the potential of long CoT reasoning in AI art creation.

Figures

Figures reproduced from arXiv: 2506.09790 by the authors.

Figure 1
Figure 1. We introduce ComfyUI-R1, a large reasoning model for automated workflow generation. Given a user instruction, ComfyUI-R1 performs long chain-of-thought reasoning to generate a code representation of a ComfyUI workflow. The generated workflow adheres to the correct format, executes successfully, and produces an image that aligns with the user’s instruction. ComfyUI-R1 is integrated in https://github.com/AIDC-AI/Comfy… view at source ↗
Figure 2
Figure 2. An example of SFT data. workflow generation, propose a novel reward mechanism for RL in this task, and finally develop ComfyUI-R1. 3 ComfyUI-R1 ComfyUI-R1 is a large reasoning model designed for automated workflow generation. Given a user query about task description, based on a set of retrieved nodes, ComfyUI-R1 first performs step￾by-step reasoning to select relevant nodes and plan the workflow. It then generates … view at source ↗
Figure 3
Figure 3. Comparison between ComfyAgent and our ComfyUI-R1. The execution result of ComfyUI-R1 accurately adheres to the “anime-style” and “cartoon” attributes in the user instruction. In contrast, ComfyAgent fails to follow these stylistic guidelines. outperforms few-shot, CoT prompting and multi-agent methods, which all rely on top-tier closed-source LLMs such as GPT-4o and the Claude series. Compared to the original Qwen2.… view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Case study: multi-image combination. Here ComfyUI-R1 seamlessly combines the input images by generating a correct and executable workflow. SA Conference Papers ’25, December 15–18, 2025, Hong Kong, China [PITH_FULL_IMAGE:figures/full_fig_p010_4.png]
Figure 5
Figure 5. Figure 5: Case study: multi-image combination (continued). This figure shows the workflow and the corresponding execution result generated by ComfyAgent [Xue et al. 2024]. Categories Image Editing [PITH_FULL_IMAGE:figures/full_fig_p011_5.png]
Figure 6
Figure 6. Figure 6: An illustration of the task categories in our constructed workflow KB, including the subcategories under image editing. [PITH_FULL_IMAGE:figures/full_fig_p011_6.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Knowledge-Centric Agents for Workflow Generation in ComfyUI

    cs.AI 2026-07 conditional novelty 6.0 of 10

    A knowledge-centric pipeline distills strategies and pseudo-codes from real workflows, fine-tunes a language model on those levels, and reconstructs executable ComfyUI graphs from task descriptions.

  2. Na\"ive PAINE: Lightweight Text-to-Image Generation Improvement with Prompt Evaluation

    cs.CV 2026-03 conditional novelty 6.0 of 10

    A lightweight predictor ranks initial noises by expected human-preference score for a prompt, selecting the best few for diffusion generation and reporting prompt difficulty.

  3. JarvisHub: An Open Harness for Canvas-Native Multimodal Creative Agents

    cs.CV 2026-07 conditional novelty 5.0 of 10

    JarvisHub open-sources a three-layer canvas-state, protocol-bridge, and agent-runtime harness so multimodal creative agents can inspect and update a shared editable project graph over long workflows.

Reference graph

Works this paper leans on

2 extracted references · 2 linked inside Pith · cited by 3 Pith papers

  1. [2024]

    arXiv:2403.17804 [cs.CV] https://arxiv.org/abs/2403.17804 Qian Ning, Weisheng Dong, Guangming Shi, Leida Li, and Xin Li

    Improving Text-to-Image Consistency via Automatic Prompt Optimization. arXiv:2403.17804 [cs.CV] https://arxiv.org/abs/2403.17804 Qian Ning, Weisheng Dong, Guangming Shi, Leida Li, and Xin Li. 2021. Accurate and Lightweight Image Super-Resolution With Model-Guided Deep Unfolding Network. IEEE Journal of Selected Topics in Signal Processing15, 2 (2021), 240...

    arXiv 2021

  2. [2025]

    Aha Moment

    Flow: Modularized Agentic Workflow Automation. InThe Thirteenth Interna- tional Conference on Learning Representations. https://openreview.net/forum?id= sLKDbuyq99 OpenAI. 2024. Introducing OpenAI o1-preview. https://openai.com/index/introducing- openai-o1-preview/. OpenAI. 2025a. Introducing Deep Research. https://openai.com/index/introducing- deep-resea...

    arXiv 2025

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.