Pith. sign in

REVIEW 3 major objections 5 minor 1 cited by

MermaidFlow: Redefining Agentic Workflow Generation via Safety-Constrained Evolutionary Programming

T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read MermaidFlow argues that moving agentic workflow search from code space to statically typed Mermaid graphs—with safety-constrained evolutionary operators—raises success rates and convergence speed on four agent reasoning benchmarks, and…

desk verdict Useful, honest paper about Mermaid-DSL workflow search, but the 'correctness by construction' guarantee is a filter, not a guarantee. read the letter →

arxiv 2505.22967 v1 pith:ST5NVMWL submitted 2025-05-29 cs.LG cs.MA

Chengqi Zheng , Jianda Chen , Yueming Lyu , Wen Zheng Terence Ng , Haopeng Zhang , Yew-Soon Ong , Ivor Tsang , Haiyan Yin This is my paper
classification cs.LGcs.MA
keywords agenticworkflowsevolutionaryprogrammingMermaidgraphsstaticverificationmulti-agentsystemsLLMagentsworkflowsearchLLM-as-judge
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Existing methods generate agent workflows as Python code or JSON trees, where structure is implicit and mistakes surface only at runtime. MermaidFlow instead represents a workflow as a typed Mermaid graph, with nodes carrying roles and type signatures and edges carrying semantic labels, so that structural properties can be checked before execution. On top of this representation, the paper builds an evolutionary search whose operators are designed to preserve type compatibility, and whose candidates are validated by a static checker before evaluation. On GSM8K, MATH, HumanEval, and MBPP, MermaidFlow reports the best solve rates among the baselines tested, with the largest gains on the harder math and code benchmarks, plus faster convergence to executable plans. If correct, this means workflow search can be moved from fragile code space into a statically verifiable graph space without changing the task setup or evaluation protocol.

What carries the argument

The central object is the declarative workflow graph $G(V[\tau,\alpha], E[\rho])$ written in Mermaid: nodes are typed and annotated, with $\tau(v) = T_{\mathrm{in}} \to T_{\mathrm{out}}$ and attributes such as role or analysis, and edges carry semantic labels such as "input" or "problem". The search space $S = \{G \in G_{\mathrm{Mermaid}} \mid G \models C_{\mathrm{static}}\}$ is enforced by a static checker: soft regex checks W1 through W5 verify interface presence, connectivity, node types, and ensemble fan-in, while hard checks compile the diagram with the Mermaid compiler. The evolutionary operators—node substitution, node addition, edge rewiring, node deletion, subgraph mutation, and crossover—are defined with type-compatibility side conditions, supporting the paper's Lemma 1 that the space is closed under the operators. The paper also states that when an LLM proposes a new graph, the checker rejects violations and requests regeneration, so "valid by construction" here means valid by check-and-regenerate rather than by unconstrained generation.

What would settle it

Measure the fraction of Mermaid candidates that pass both soft and hard checks but then fail when translated to executable Python on the same four benchmarks; if that fraction approaches the failure rate of code-based search, the static-verification guarantee is not the source of the reported gains.

Watch

Extended reading notes

Core claim

The central claim is that the search space itself, rather than the LLM's raw generation quality, is the main bottleneck in agentic workflow construction. By encoding workflows as typed declarative graphs, MermaidFlow defines local edit operators with interface conditions such as $T_{\mathrm{out}}(v_a) = T_{\mathrm{in}}(v_c)$, so that each edit stays inside a statically valid subspace, and a checker filters any LLM-proposed candidate that violates structural rules. The paper reports consistent improvements over thirteen baselines, an average score of 80.75 percent against 79.35 percent for the strongest baseline, and a much higher rate of producing executable Python from Mermaid graphs than from direct code-based search. The authors further claim this is the first agentic workflow framework to guarantee static graph-level correctness across the entire generation process.

Load-bearing premise

The whole method depends on the assumption that a diagram passing its structure checks and Mermaid compilation is genuinely safe and executable, not merely syntactically valid.

Editorial extensions

If this is right

  • Searching in the typed graph space raises the fraction of evaluated candidates that are executable, so more of each optimization budget goes to scoring real workflows rather than discarding broken code.
  • On benchmarks where a single-agent baseline is already high, such as GSM8K and HumanEval, gains are smaller, while the approach helps most on tasks where workflow design matters, such as MATH and MBPP.
  • Because each validated Mermaid workflow is stored as a scored experience, successful substructures can be reused across tasks without revalidation, enabling cross-task transfer.
  • Stronger optimization LLMs yield larger performance gains within the same search space, since the verifiable structure lets capability improvements pass through to final workflow quality.
  • The Mermaid artifact serves as a single source for visualization, static checking, and Python code generation, improving interpretability and debugging without an extra representation.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A testable extension is to replace the LLM proposer with a grammar-constrained sampler that only emits type-compatible edits; if the gains persist, the graph representation rather than the proposal model is the active ingredient.
  • The same typed-graph substrate could apply to non-agent pipelines such as data-processing DAGs or evaluation harnesses, where static type checks and crossover over subgraphs are equally natural.
  • The convergence claim suggests a measurable prediction: the number of regeneration rounds per accepted candidate should decrease as checker rules are strengthened, and adding explicit edge-compatibility checks should shorten search further.
  • Since the evaluation uses one fixed closed-source optimization LLM at temperature zero, separating the representation's contribution from the optimizer's choice remains an open test worth running across multiple model families.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper introduces MermaidFlow, a framework that represents agentic workflows as typed, declarative Mermaid graphs and searches over this space using evolutionary operators (node substitution, addition, rewiring, deletion, subgraph mutation, crossover). The authors claim that this representation yields a statically verifiable workflow space in which every candidate is valid by construction, and they support this with a closure lemma (Lemma 1). The system is evaluated on GSM8K, MATH, HumanEval, and MBPP, reporting consistent improvements over single-agent, hand-crafted multi-agent, and automated workflow baselines, including AFlow.

Significance. If the static-correctness guarantee were sound, the paper would make a useful contribution by introducing a verifiable intermediate representation for agentic workflow search and demonstrating its empirical value. The experimental setup is solid in that final workflows are executed on validation and test splits, and the comparison to AFlow under matched settings is informative. The code is promised and the benchmark selections are standard. However, the central theoretical claim does not match the implemented system, and this gap materially weakens the claimed novelty. The empirical results stand on their own but are less interesting without a credible safety guarantee; the paper as written overstates what is shown.

major comments (3)
  1. [Section 4.1, Lemma 1 and Algorithms 1-3] The paper claims that all candidates are valid by construction because the evolutionary operators preserve the space S. However, the actual pipeline (Algorithms 1-3) does not apply these formal operators; it asks an LLM to generate new Mermaid diagrams and then filters them with the checker described in Appendix A.2. The checker verifies W1-W5 (interface presence, connectivity, node types, ensemble fan-in) and Mermaid CLI syntax, but it never checks the interface type compatibility conditions (e.g., Tout(va)=Tin(vc)) that the operators in Section 4.1 rely on. Therefore Lemma 1's closure argument does not establish correctness for the deployed system, and the statement that 'every candidate is valid by construction' is not supported. The authors should either implement the operators explicitly as graph transformations or revise the claims to describe a checker-filtered generation process.
  2. [Section 3.2, Eq. (2) and Appendix A.2] The search space S is defined as graphs satisfying Cstatic, where Cstatic is said to be 'automatically enforced by Mermaid's parser and extended structural schema.' In practice, the Mermaid CLI only checks syntax; the soft checks W1-W5 are a fixed set of regex-based rules. No formal definition of Cstatic is given, and no proof is provided that the implemented checker is equivalent to Cstatic. The node types in Appendix A.1 are described informally, with no concrete type signatures or compatibility rules, so the Tout=Tin conditions used in the operators are not well-defined for the actual node types. This gap breaks the chain from the formal space S to the filtered candidates in the implementation.
  3. [Section 5.3, Figure 3 and abstract] The abstract claims 'faster convergence to executable plans,' but the supporting evidence in Figure 3 shows learning curves without error bars, without a definition of 'convergence,' and without a statistical test. The y-axis label is also unclear ('highest solve rates'). The paper should report variance across the three runs mentioned in Table 1, define a convergence metric (e.g., iterations to reach a threshold), and test whether the apparent advantage over AFlow is significant.
minor comments (5)
  1. [Abstract] The abstract contains a grammar error: 'MermaidFlow represent workflows' should be 'MermaidFlow represents workflows.'
  2. [Section 5.3] The text refers to 'Figure 5.3' when the figure is labeled 'Figure 3'; please correct the cross-reference.
  3. [Appendix A.3] There are several typos in the prompt templates: 'exmaples' should be 'examples,' 'Guidence' should be 'Guidance,' and 'Mermiad' should be 'Mermaid.' These do not affect the experiments but should be cleaned up.
  4. [Section 4.2] The statement that 'all candidates in Scandidates are statically verified by the Mermaid compiler' is imprecise because the checker also includes soft checks (W1-W5) that are not part of the Mermaid compiler. Please clarify that the combined checker is used.
  5. [Section 4.2 and Algorithm 2] The paper says the checker is always used, but Algorithm 2 includes a branch that accepts the first response if no checker exists. Since the experimental setup uses a checker, this branch is not exercised, but it weakens the formal description.

Circularity Check

1 steps flagged · score 2.0 of 10

Benchmark claims are externally anchored; only Lemma 1 is a definitional tautology, and the stronger 'valid by construction' wording is undercut by the paper's own checker admission.

  1. self definitional [Section 4.1, Lemma 1 (Transformation Invariance), Eq. (4); compare Section 3.2, Eq. (2)]
    "Lemma 1 (MermaidFlow Transformation Invariance). Let S denote the declarative workflow space defined in Section 3.2. For any workflow graph G ∈ S and any atomic transformation operator O defined above, the resulting graph G′ = O(G) also belongs to S: ∀ G ∈ S, ∀ O ∈O, O(G) ∈ S (4) where O is the set of constraint-preserving operators over MermaidFlow graphs. That is, S is closed under all valid EP operations."

    S is defined in Eq. (2) as the set of graphs satisfying Cstatic (type compatibility, role-consistent edges, connectivity), while each operator in Section 4.1 is gated by exactly those constraints: Node Addition requires Tout(va)=Tin(v') and Tout(v')=Tin(vb), Edge Rewiring requires Tout(va)=Tin(vc) or Tout(vc)=Tin(vb), and so on. Therefore G'=O(G)∈S follows immediately from the operator's applicability precondition plus the definition of S; Lemma 1 restates the definitions rather than establishing an independent closure property. The later claim that 'the evolution ... remains the safe subspace' inherits this definitional content. This tautology does not drive the empirical benchmark results, which compare against external datasets.

full rationale

The main empirical claim—consistent improvements in solve rates and pass@1 on GSM8K, MATH, HumanEval, and MBPP (Table 1)—is anchored to external benchmarks and does not reduce to the paper's internal definitions or fitted values, so it is not circular. There are no load-bearing self-citations and no fitted parameter is relabeled as a prediction. The only definitional circularity is Lemma 1, whose 'transformation invariance' is built into the operator preconditions and the definition of S; it is a tautology rather than an independent safety guarantee. Separately, the paper's stronger wording that candidates are 'valid by construction' is undercut by its own Section 4.1 admission that LLM-generated Mermaid code 'may sometimes violate predefined safety constraints' and by Appendix A.2, whose checks W1–W5 verify interface presence, connectivity, node types, and ensemble fan-in but not the Tout=Tin interface compatibility that Lemma 1's operators assume; that is a soundness/overclaim issue rather than circularity. Overall, the derivation chain is largely self-contained, and the tautological lemma does not invalidate the externally grounded experimental comparison.

Assumptions & free parameters 5 free parameters · 4 assumptions · 1 invented entities

The central empirical comparison is externally anchored, but the formal-looking safety guarantee is defined by the paper's own checker rather than an independent type system, and several search hyperparameters are set by hand without analysis.

free parameters (5)
  • Softmax sampling temperature alpha = not reported
    Controls how strongly scores influence parent selection in Section 4.2; chosen by hand and not swept.
  • Exploration coefficient lambda = not reported
    Balances uniform versus score-weighted parent sampling in Section 4.2; no sensitivity analysis is given.
  • Crossover probability = 0.10
    Set by hand in Section 5.1 to stabilize search; not ablated.
  • Candidate pool size = 4
    Four candidate workflows per round in Section 5.1; no study of how this affects search quality.
  • Initial hand-authored Mermaid template = hand-crafted per dataset
    Algorithm 1 loads a pre-built Mermaid workflow as the starting point; this template may encode a large share of the final performance and is not measured as a variable.
assumptions (4)
  • domain assumption Mermaid CLI compilation plus soft checks W1-W5 is a sound static verifier of workflow correctness
    Used to define S in Section 3.2 and to claim validity by construction; the checks cover connectivity and ensemble fan-in but not all interface type compatibility.
  • domain assumption Type compatibility between node interfaces is sufficient for semantic correctness of data flow
    Operators such as node addition, rewiring, and deletion require Tout(va)=Tin(vb), but no formal type semantics or runtime verification is provided in Section 4.1.
  • domain assumption LLM-as-judge scores correlate with measured validation accuracy
    The search selects candidates by LLM judgment before execution in Section 4.2; no agreement analysis between judge scores and true solve rates is reported.
  • domain assumption gpt-4o-mini can translate Mermaid graphs to correct Python with high reliability
    The pipeline depends on this translation in Algorithm 1 and Section 3.1; the paper claims over 90 percent success but provides no supporting measurement.
invented entities (1)
  • MermaidFlow typed node classes (Interface, CustomOp, ProgrammerOp, ScEnsembleOp, TestOp, CustomCodeGenerateOp) independent evidence
    purpose: Define the operators from which workflows are composed and enable static checking and Python translation
    Implemented in the released code and observable in generated workflows; their effect is testable on the four benchmarks, though no independent formal semantics is provided.

how reviews work

0 comments
Cite this review

Pith. "Pith review of MermaidFlow: Redefining Agentic Workflow Generation via Safety-Constrained Evolutionary Programming." pith.science (2026). https://pith.science/paper/ST5NVMWL

@misc{pith2026250522967,
  author       = {Pith},
  title        = {Pith review of: MermaidFlow: Redefining Agentic Workflow Generation via Safety-Constrained Evolutionary Programming},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ST5NVMWL}},
  note         = {Machine review of arXiv:2505.22967}
}
read the original abstract

Despite the promise of autonomous agentic reasoning, existing workflow generation methods frequently produce fragile, unexecutable plans due to unconstrained LLM-driven construction. We introduce MermaidFlow, a framework that redefines the agentic search space through safety-constrained graph evolution. At its core, MermaidFlow represent workflows as a verifiable intermediate representation using Mermaid, a structured and human-interpretable graph language. We formulate domain-aware evolutionary operators, i.e., crossover, mutation, insertion, and deletion, to preserve semantic correctness while promoting structural diversity, enabling efficient exploration of a high-quality, statically verifiable workflow space. Without modifying task settings or evaluation protocols, MermaidFlow achieves consistent improvements in success rates and faster convergence to executable plans on the agent reasoning benchmark. The experimental results demonstrate that safety-constrained graph evolution offers a scalable, modular foundation for robust and interpretable agentic reasoning systems.

Figures

Figures reproduced from arXiv: 2505.22967 by the authors.

Figure 2
Figure 2. Overview of the MermaidFlow framework. Left: Comparison between imperative (Python-based) and declarative (Mermaid-based) workflow representations. MermaidFlow models workflows as statically typed, verifiable graphs, enabling interpretable planning and structure-aware code generation. Right: Illustration of the safety-aware evolutionary programming process. Given historical Mermaid workflows, the EP sampler selects … view at source ↗
Figure 5.3
Figure 5.3. MermaidFlow demonstrates a more consis￾tent improvement in workflow quality during training and better generalization to the test set. The core difference between MermaidFlow and AFlow lies in the search space. AFlow operates directly on Python code, applying textual edits with prompts con￾straints. This approach often leads to invalid and non￾functional programs, with only a 50% success rate in generating executabl… view at source ↗
Figure 3
Figure 3. An illustrative figure com￾paring the highest solve rates on the MATH dataset between MermaidFlow and AFlow on the training set (119 problems) and test set (486 problems) across opti￾mization iterations. Evolution Efficiency To evaluate the effectiveness of our approach, we compare the learning curves of Mer￾maidFlow and AFlow on the MATH dataset, as shown in [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: A case study on the HumanEval dataset showcasing how MermaidFlow evolves structured agentic workflows through evolutionary programming (with a detailed example of the crossover operator). The declarative graph representation also enables reliable translation of workflo…
Figure 5
Figure 5. Figure 5: Mermaid diagram for GSM8K. Mermaid Code flowchart TD %% Nodes PROBLEM([Problem]) C["Custom<br/>(role: simple_solver_1)"] P1["Programmer<br/>(analysis: 'Calculate step by step')"] P2["Programmer<br/>(analysis: 'Generate solution with edge cases')"] P3["Programmer<br/>(a…
Figure 6
Figure 6. Figure 6: Mermaid diagram for MATH. Mermaid Code flowchart TD %% Nodes PROBLEM([Problem]) C1["Custom<br/>(role: simple_solver_1)"] C2["Custom<br/>(role: simple_solver_2)"] C3["Custom<br/>(role: alternative_solver)"] C4["Custom<br/>(role: detailed_solution_outline)"] C5["Custom<b…
Figure 7
Figure 7. Figure 7: Mermaid diagram for HumanEval. Mermaid Code flowchart TD %% Nodes PROBLEM([Problem]) ENTRY_POINT([entry_point]) C1["CustomCodeGenerate<br/>(instruction: simple_solver_1)"] C2["CustomCodeGenerate<br/>(instruction: simple_solver_2)"] C3["CustomCodeGenerate<br/>(instructi…
Figure 8
Figure 8. Figure 8: Mermaid diagram for MBPP. Mermaid Code flowchart TD %% Nodes PROBLEM([Problem]) ENTRY_POINT([entry_point]) C1["CustomCodeGenerate<br/>(instruction: simple_solver_1)"] C2["CustomCodeGenerate<br/>(instruction: simple_solver_2)"] C3["CustomCodeGenerate<br/>(instruction: o…

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. FlowEvo: Self-Evolving Agents through the Co-Evolution of Workflows and Executable Skills

    cs.AI 2026-04 conditional novelty 5.0 of 10

    FlowEvo compiles successful agent workflows into executable skill records and reuses them at inference time, reporting the best accuracy-cost tradeoff across ALFWorld, HumanEval, and GSM8K among tested baselines.

Reference graph

Works this paper leans on

94 extracted references · 58 canonical work pages · cited by 1 Pith paper

  1. [1]

    Program synthesis with large language models

    Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al. Program synthesis with large language models. arXiv preprint arXiv:2108.07732, 2021

    arXiv 2021

  2. [2]

    Pan, Shuyi Yang, Lakshya A

    Mert Cemri, Melissa Z. Pan, Shuyi Yang, Lakshya A. Agrawal, Bhavya Chopra, Rishabh Tiwari, Kurt Keutzer, Aditya G. Parameswaran, Dan Klein, Kannan Ramchandran, Matei Zaharia, Joseph E. Gonzalez, and Ion Stoica. Why do multi-agent LLM systems fail? CoRR, abs/2503.13657, 2025

    arXiv 2025

  3. [3]

    Autoagents: A framework for automatic agent generation

    Guangyao Chen, Siwei Dong, Yu Shu, Ge Zhang, Jaward Sesay, Börje F Karlsson, Jie Fu, and Yemin Shi. Autoagents: A framework for automatic agent generation. In Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence, IJCAI 2024, Jeju, South Korea, August 3-9, 2024, pages 22–30, 2024

    2024

  4. [4]

    Evaluating large language models trained on code

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374, 2021

    arXiv 2021

  5. [5]

    Training verifiers to solve math word problems

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021

    arXiv

  6. [6]

    Tenenbaum, and Igor Mordatch

    Yilun Du, Shuang Li, Antonio Torralba, Joshua B. Tenenbaum, and Igor Mordatch. Improving factuality and reasoning in language models through multiagent debate. In Forty-first Inter- national Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024 . OpenReview.net, 2024

    2024

  7. [7]

    Complexity-based prompting for multi-step reasoning

    Yao Fu, Hao Peng, Ashish Sabharwal, Peter Clark, and Tushar Khot. Complexity-based prompting for multi-step reasoning. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023. OpenReview.net, 2023

    2023

  8. [8]

    Flowreasoner: Reinforcing query-level meta-agents

    Hongcheng Gao, Yue Liu, Yufei He, Longxu Dou, Chao Du, Zhijie Deng, Bryan Hooi, Min Lin, and Tianyu Pang. Flowreasoner: Reinforcing query-level meta-agents. arXiv preprint arXiv:2504.15257, 2025

    arXiv 2025

Show all 94 references
  1. [9]

    Chawla, Olaf Wiest, and Xiangliang Zhang

    Taicheng Guo, Xiuying Chen, Yaqi Wang, Ruidi Chang, Shichao Pei, Nitesh V . Chawla, Olaf Wiest, and Xiangliang Zhang. Large language model based multi-agents: A survey of progress and challenges. In Proceedings of the Thirty-Third International Joint Conference on Artificial I...

    2024

  2. [10]

    Measuring mathematical problem solving with the math dataset

    Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. Measuring mathematical problem solving with the math dataset. arXiv preprint arXiv:2103.03874, 2021. 10

    2021 arXiv

  3. [11]

    Metagpt: Meta programming for A multi-agent collaborative framework

    Sirui Hong, Mingchen Zhuge, Jonathan Chen, Xiawu Zheng, Yuheng Cheng, Jinlin Wang, Ceyao Zhang, Zili Wang, Steven Ka Shing Yau, Zijuan Lin, Liyang Zhou, Chenyu Ran, Lingfeng Xiao, Chenglin Wu, and Jürgen Schmidhuber. Metagpt: Meta programming for A multi-agent collaborative fr...

    2024

  4. [12]

    Automated design of agentic systems

    Shengran Hu, Cong Lu, and Jeff Clune. Automated design of agentic systems. CoRR, abs/2408.08435, 2024

    2024 arXiv

  5. [13]

    Llm-blender: Ensembling large language models with pairwise comparison and generative fusion

    Dongfu Jiang, Xiang Ren, and Bill Yuchen Lin. Llm-blender: Ensembling large language models with pairwise comparison and generative fusion. In Proceedings of the 61th Annual Meeting of the Association for Computational Linguistics (ACL 2023), 2023

    2023

  6. [14]

    Large language models are zero-shot reasoners

    Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners. In Advances in Neural Information Processing Systems 35: Annual Conference on Neural Information Processing Systems 2022, NeurIPS 2022, New Orleans...

    2022

  7. [16]

    A review of prominent paradigms for llm-based agents: Tool use, planning (in- cluding rag), and feedback learning

    Xinzhe Li. A review of prominent paradigms for llm-based agents: Tool use, planning (in- cluding rag), and feedback learning. In Proceedings of the 31st International Conference on Computational Linguistics, COLING 2025, Abu Dhabi, UAE, January 19-24, 2025 , pages 9760–9779, 2025

    2025

  8. [17]

    Encouraging divergent thinking in large language models through multi-agent debate

    Tian Liang, Zhiwei He, Wenxiang Jiao, Xing Wang, Yan Wang, Rui Wang, Yujiu Yang, Shuming Shi, and Zhaopeng Tu. Encouraging divergent thinking in large language models through multi-agent debate. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Pro...

    2024

  9. [18]

    A dynamic llm-powered agent network for task-oriented agent collaboration

    Zijun Liu, Yanzhe Zhang, Peng Li, Yang Liu, and Diyi Yang. A dynamic llm-powered agent network for task-oriented agent collaboration. In First Conference on Language Modeling, 2024

    2024

  10. [19]

    Self- refine: Iterative refinement with self-feedback

    Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, Shashank Gupta, Bodhisattwa Prasad Majumder, Katherine Hermann, Sean Welleck, Amir Yazdanbakhsh, and Peter Clark. Self- refine: Iterati...

    2023

  11. [20]

    Scaling large-language-model-based multi-agent collab- oration

    Chen Qian, Zihao Xie, Yifei Wang, Wei Liu, Yufan Dang, Zhuoyun Du, Weize Chen, Cheng Yang, Zhiyuan Liu, and Maosong Sun. Scaling large-language-model-based multi-agent collab- oration. CoRR, abs/2406.07155, 2024

    2024 arXiv

  12. [21]

    Debflow: Automating agent creation via agent debate

    Jinwei Su, Yinghui Xia, Ronghua Shi, Jianhui Wang, Jianuo Huang, Yijin Wang, Tianyu Shi, JingSong Yang, and Lewei He. Debflow: Automating agent creation via agent debate. CoRR, abs/2503.23781, 2025

    2025

  13. [22]

    Le, Ed H

    Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V . Le, Ed H. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali...

    2023

  14. [23]

    Scoreflow: Mastering LLM agent workflows via score-based preference optimization

    Yinjie Wang, Ling Yang, Guohao Li, Mengdi Wang, and Bryon Aragam. Scoreflow: Mastering LLM agent workflows via score-based preference optimization. CoRR, abs/2502.04306, 2025. 11

    2025 arXiv

  15. [24]

    Narasimhan, and Yuan Cao

    Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R. Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5,

    2023

  16. [25]

    MAS-GPT: training llms to build llm-based multi-agent systems

    Rui Ye, Shuo Tang, Rui Ge, Yaxin Du, Zhenfei Yin, Siheng Chen, and Jing Shao. MAS-GPT: training llms to build llm-based multi-agent systems. CoRR, abs/2503.03686, 2025

    2025 arXiv

  17. [26]

    Breaking agents: Compromising autonomous LLM agents through malfunction amplification

    Boyang Zhang, Yicong Tan, Yun Shen, Ahmed Salem, Michael Backes, Savvas Zannettou, and Yang Zhang. Breaking agents: Compromising autonomous LLM agents through malfunction amplification. CoRR, abs/2407.20859, 2024

    2024 arXiv

  18. [27]

    Evoflow: Evolving diverse agentic workflows on the fly

    Guibin Zhang, Kaijie Chen, Guancheng Wan, Heng Chang, Hong Cheng, Kun Wang, Shuyue Hu, and Lei Bai. Evoflow: Evolving diverse agentic workflows on the fly. CoRR, abs/2502.07373, 2025

    2025 arXiv

  19. [28]

    Multi-agent architecture search via agentic supernet

    Guibin Zhang, Luyang Niu, Junfeng Fang, Kun Wang, Lei Bai, and Xiang Wang. Multi-agent architecture search via agentic supernet. CoRR, abs/2502.04180, 2025

    2025 arXiv

  20. [29]

    G-designer: Architecting multi-agent communication topologies via graph neural networks

    Guibin Zhang, Yanwei Yue, Xiangguo Sun, Guancheng Wan, Miao Yu, Junfeng Fang, Kun Wang, and Dawei Cheng. G-designer: Architecting multi-agent communication topologies via graph neural networks. CoRR, abs/2410.11782, 2024

    2024 arXiv

  21. [30]

    Aflow: Automating agentic workflow generation

    Jiayi Zhang, Jinyu Xiang, Zhaoyang Yu, Fengwei Teng, Xionghui Chen, Jiaqi Chen, Mingchen Zhuge, Xin Cheng, Sirui Hong, Jinlin Wang, Bingnan Zheng, Bang Liu, Yuyu Luo, and Chenglin Wu. Aflow: Automating agentic workflow generation. CoRR, abs/2410.10762, 2024

    arXiv

  22. [31]

    Which agent causes task failures and when? on automated failure attribution of llm multi-agent systems

    Shaokun Zhang, Ming Yin, Jieyu Zhang, Jiale Liu, Zhiguang Han, Jingyang Zhang, Beibin Li, Chi Wang, Huazheng Wang, Yiran Chen, et al. Which agent causes task failures and when? on automated failure attribution of llm multi-agent systems. arXiv preprint arXiv:2505.00212, 2025

    2025 arXiv

  23. [32]

    Han Zhou, Xingchen Wan, Ruoxi Sun, Hamid Palangi, Shariq Iqbal, Ivan Vulic, Anna Korhonen, and Sercan Ö. Arik. Multi-agent design: Optimizing agents with better prompts and topologies. CoRR, abs/2502.02533, 2025

    2025

  24. [33]

    Gptswarm: Language agents as optimizable graphs

    Mingchen Zhuge, Wenyi Wang, Louis Kirsch, Francesco Faccio, Dmitrii Khizbullin, and Jürgen Schmidhuber. Gptswarm: Language agents as optimizable graphs. In Forty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024. OpenReview.net, 2...

    2024

  25. [35]

    Custom<br/>(role: validate_1)

    CustomOp - Used for specialized reasoning strategies, executing specific tasks through defined roles. Example: K["Custom<br/>(role: validate_1)"] , styled with blue fill

  26. [36]

    Example: PROBLEM([Problem]), styled with light purple fill

    Interface - Entry and exit nodes for workflows, including problem input points and return output points. Example: PROBLEM([Problem]), styled with light purple fill

  27. [37]

    Programmer<br/>(analysis: ’Calculate step by step’)

    ProgrammerOp - Nodes that execute code generation and computation, particularly suitable for mathematical problems. Example: P["Programmer<br/>(analysis: ’Calculate step by step’)"], styled with red fill. Only math problems use this node type

  28. [38]

    ScEnsemble<br/>

    ScEnsembleOp - Nodes that combine multiple solutions into one result, requiring multiple inputs to function properly. Example: ENSEMBLE["ScEnsemble<br/>"], styled with yellow fill

  29. [39]

    Test<br/>

    TestOp - Nodes for validating solutions, used to verify whether code passes predefined test cases. Example: T["Test<br/>"], styled with green fill. Only code problems use this node type

  30. [40]

    CustomCodeGenerate<br/>(instruction: xxx)

    CustomCodeGenerateOp - Nodes specifically designed to generate code based on prob- lem descriptions. Example: CODE_GEN["CustomCodeGenerate<br/>(instruction: xxx)"], styled with red fill. Only code problems use this node type. Each node type has specific connection rules and us...

  31. [41]

    W1 - Verifies the presence of required PROBLEM and RETURN interface nodes

  32. [42]

    W2 - Ensures each node is properly connected in the workflow with paths from PROBLEM and to RETURN

  33. [43]

    W3 - Validates that PROBLEM and RETURN nodes are correctly classified as Interface types

  34. [44]

    W4 - Checks that all nodes have valid types according to configuration

  35. [45]

    W5 - Ensures ScEnsembleOp nodes have at least two incoming connections We also implement rigorous hard checks by compiling the Mermaid code directly through the Mermaid CLI3, which thoroughly verifies syntactic correctness and catches any compilation errors. This comprehensive...

  36. [48]

    "" UPDATE_MERMAID_WORKFLOW_CODE =

    Modifying existing nodes or their connections or their prompt. Prompt is very important for performance here are some exmaples you can learn from it: {prompt_few_shot} Ensure all necessary prompts are properly prepared and defined. The generated custom node 's role is defined ...

  37. [49]

    Note: adding nodes may require modifying prompts for related nodes, you should also update the corresponding prompts if needed.,→

    Expanding the graph by adding new single (operators). Note: adding nodes may require modifying prompts for related nodes, you should also update the corresponding prompts if needed.,→

  38. [50]

    Deleting unnecessary nodes from the graph

  39. [51]

    solution

    Modifying existing nodes or their connections or their prompt. Prompt is very important for performance here are some exmaples you can learn from it: {prompt_few_shot} Ensure all necessary prompts are properly prepared and defined. The generated custom node 's role is defined ...

  40. [52]

    Analyze both **parent graphs** carefully (structure, nodes, connections, prompts, and purpose)

  41. [53]

    Pay special attention to their structure, specificity, and instruction clarity

    Create a new graph that: - Combines strengths from both parents - Introduces strategic innovations for improved performance - Applies evolutionary operations such as: * Crossover: Merging effective sections(nodes) from both parents, you should separate components in a reasonab...

  42. [54]

    A detailed explanation of your modifications in the <modification> section

  43. [55]

    The complete Mermaid code for the new graph

  44. [56]

    "" MERMAID_GUIDANCE Mermaid Code Guidance MERMAID_CUSTOM_MATH =

    The updated prompts for any custom nodes NOTE: Ensure the new graph is valid Mermaid syntax and represents a complete workflow solution. The following section contains critical rules and guidance for using operators in Mermaid that you MUST follow to create an effective workfl...

  45. [57]

    The connections between each node

  46. [58]

    The role and function of each operator

  47. [59]

    "" LLM_AS_JUDGE LLM As Judge LLM_AS_JUDGER =

    The input/output relationships For each node in the graph, implement the appropriate code and use the corresponding prompts from new_prompt. If a node or operator doesn 't have an explicit prompt provided, DO NOT create one - use empty strings instead. For example, Program ope...

  48. [60]

    **Workflow Coherence**: Assess how well the nodes connect and form a logical workflow

  49. [61]

    **Innovation**: Evaluate how the new graph improves upon the parent graphs

  50. [62]

    **Complexity Balance**: Check if the graph has appropriate complexity (neither too simple nor unnecessarily complex),→

  51. [63]

    **Prompt Quality**: Examine the quality and specificity of the node prompts 22

  52. [64]

    **Modification Rationale**: Consider the thoughtfulness of the explanation provided for the changes,→ Here are some history of previous graphs and their corresponding score: <history> {elites_history} </history> For each candidate graph, provide a score from 1-10 for each crit...

  53. [65]

    Incorrectly connecting nodes (e.g., CustomOp should not directly feed into ProgrammerOp)

  54. [66]

    Not properly using the ensemble node (all solution-generating nodes should feed into it)

  55. [67]

    Missing critical connections between nodes

  56. [68]

    Creating circular dependencies Here are the specific structural rules for this type of workflow: {mermaid_usage} Additionally, consider how well the graph follows established patterns from successful historical examples.,→ Then select the graph with the highest total score as ...

  57. [69]

    Mermaid encodes workflows as statically typed and semantically annotated graphs, allowing prompt semantics and operator roles to be aligned explicitly within the workflow structure

    Typed, Semantically Aligned Representation. Mermaid encodes workflows as statically typed and semantically annotated graphs, allowing prompt semantics and operator roles to be aligned explicitly within the workflow structure

  58. [70]

    Human-Readable and Verifiable Syntax. Unlike imperative representations, Mermaid provides a format that is both visually interpretable and statically verifiable, facilitating intuitive debugging, planning, and workflow validation

  59. [71]

    Custom<br/>(role: simple_solver_1)

    Reliable Code Translation. The structured nature of Mermaid graphs enables seamless compilation into executable Python code, with consistent type and role guarantees that reduce runtime errors and improve reliability. These properties make MermaidFlow not just a search mechani...

  60. [76]

    Verify and validate your results to ensure accuracy

  61. [78]

    "" 27 B.2 MATH Figure 6: Mermaid diagram for MATH. Mermaid Code flowchart TD %% Nodes PROBLEM([Problem]) C1[

    Ensure all mathematical notation is in LaTeX format. Your solution should be thorough, mathematically sound, and easy to understand.""" 27 B.2 MATH Figure 6: Mermaid diagram for MATH. Mermaid Code flowchart TD %% Nodes PROBLEM([Problem]) C1["Custom<br/>(role: simple_solver_1)"...

  62. [79]

    State the problem clearly

  63. [80]

    Outline the approach and any relevant formulas or concepts

  64. [81]

    Provide detailed calculations, using LaTeX notation for mathematical expressions

  65. [82]

    Explain each step of your reasoning

  66. [83]

    Present the final answer enclosed in \\boxed{} LaTeX notation

  67. [84]

    "" ALTERNATIVE_SOLVER =

    Ensure all mathematical notation is in LaTeX format. """ ALTERNATIVE_SOLVER = """Please provide an alternative approach to solving the given mathematical problem. Follow these guidelines:,→

  68. [86]

    Identify any different methods or perspectives that could be applied

  69. [87]

    Provide calculations and reasoning, using LaTeX notation for mathematical expressions

  70. [88]

    Ensure clarity and correctness in your explanation

  71. [89]

    "" DETAILED_SOLUTION_OUTLINE =

    Present the final answer enclosed in \\boxed{} LaTeX notation. """ DETAILED_SOLUTION_OUTLINE = """Please provide a detailed outline for solving the given mathematical problem. Follow these guidelines:,→

  72. [91]

    Identify key concepts and theorems relevant to the problem

  73. [92]

    Outline the steps needed to solve the problem, including any necessary calculations

  74. [93]

    Ensure that the outline is structured logically and is easy to follow

  75. [94]

    "" COMPREHENSIVE_SOLUTION =

    Use LaTeX notation for any mathematical expressions. """ COMPREHENSIVE_SOLUTION = """Please provide a comprehensive solution to the given mathematical problem. Follow these guidelines:,→

  76. [95]

    Clearly restate the problem

  77. [96]

    Explain the mathematical concepts and theorems involved

  78. [97]

    Provide a detailed, logical progression of steps leading to the solution

  79. [98]

    Show all calculations using LaTeX notation for mathematical expressions

  80. [99]

    "" REFINE_SOLUTION =

    Present the final answer clearly marked and enclosed in \\boxed{} LaTeX notation. """ REFINE_SOLUTION = """Given the mathematical problem and the solutions generated, please refine the output to ensure clarity and correctness. Follow these guidelines:,→

  81. [100]

    Review the solutions provided

  82. [101]

    Ensure all calculations are accurate and clearly presented

  83. [102]

    Summarize the findings and present the final answer in a clear format

  84. [103]

    Use LaTeX notation for any mathematical expressions

  85. [104]

    "" 30 B.3 HumanEval Figure 7: Mermaid diagram for HumanEval. Mermaid Code flowchart TD %% Nodes PROBLEM([Problem]) ENTRY_POINT([entry_point]) C1[

    Ensure the final answer is enclosed in \\boxed{} LaTeX notation. """ 30 B.3 HumanEval Figure 7: Mermaid diagram for HumanEval. Mermaid Code flowchart TD %% Nodes PROBLEM([Problem]) ENTRY_POINT([entry_point]) C1["CustomCodeGenerate<br/>(instruction: simple_solver_1)"] C2["Custo...

  86. [2023]

    OpenReview.net, 2023

    2023

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.