REVIEW 4 major objections 5 minor 37 references
CURATE: Leveraging LLM Agents to Compose, Catalog, and Deploy Reproducible Workflows
T0 review · 4 major / 5 minor · reviewed 2026-08-08 · deepseek-v4-flash
Pith's one-line read A human-in-the-loop multi-agent system with a module catalog can carry a natural-language workflow sketch through composition, code generation, deployment, monitoring, and reuse.
desk verdict A promising feasibility prototype for agent-driven workflow generation whose central evidence is undercut by an unresolved contradiction between Table I and the text about Experiment 3. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing mechanism is a module catalog combined with a workflow-management-system-agnostic skeleton representation. A skeleton is a DAG of natural-language task descriptions generated from the user's request; the candidate agent queries the catalog (ranked by BM25 over keyword fields) and labels each node as new, catalog-reused, cached from an earlier revision, or user-supplied, optionally with an 'adapt' flag. Generation then proceeds in topological order, with a coding agent implementing only new or adapted functions in isolated scratch directories and testing them against local stubs before the workflow is deployed. Five human-in-the-loop gates let the user approve the skeleton, the source resolution, the code, the deployment, and the final run, so error correction is routed back either to function-level changes or to a structural re-sketch.
What would settle it
Run the same six workflow prompts through CURATE ten times from scratch on the same infrastructure and count correct, error-free end-to-end runs; if the per-run success rate is not clearly better than the baseline's one-success-in-three criterion, the claimed reliability advantage rests on a single favorable run.
Extended reading notes
Core claim
The paper's central discovery claim is that a multi-agent architecture with a persistent module catalog can manage scientific workflows end to end without the user touching the underlying infrastructure. In the prototype, four agents handle composition, candidate selection, module generation, and deployment; the catalog stores verified modules with descriptions, input/output signatures, dependencies, and provenance, so the candidate agent can resolve each workflow step to an existing module, an adapted module, or fresh code. The experiments show the system faithfully reproducing the structure and outputs of four benchmark-derived workflows, reusing a MapReduce subgraph in a new PDF-extraction workflow, and turning a synthetic 'raw' wastewater dataset with unit errors, gaps, and spikes into a cleaned input for an existing anaerobic digestion simulation, then scaling that simulation to 20 concurrent runs. The paper's own account treats these six successes as evidence that this lifecycle-level automation is feasible for small-to-medium scientific workflows.
Load-bearing premise
The feasibility result assumes that one successful run of CURATE on each workflow is representative of its reliability, even though the comparison baseline was allowed up to three attempts and only one coding agent and one target deployment system were tested.
Editorial extensions
If this is right
- A user who can describe a workflow in plain language can obtain a deployed, monitored serverless workflow without learning the workflow management system's syntax or infrastructure details.
- Catalog reuse turns verified modules into building blocks: a new workflow can inherit previously tested functions, and only genuinely new steps pay the cost of code generation.
- Scaling a workflow can be as simple as changing the prompt: the same anaerobic digestion pipeline was extended to a 20-run parameter sweep that required one post-run revision suggested by the monitoring agent.
- Human-in-the-loop checking at multiple stages can catch errors that pure code generation misses, including a hard-coded model parameter that would have made all 20 sweep runs identical.
- Token usage and oversight can decrease as modules are reused across related workflows, since later runs skip generation of already-verified functions.
Reading between the lines
- A natural extension is to treat the catalog itself as a FAIR substrate: if entries carry provenance, versioning, and usage counts, shared catalogs could let different laboratories assemble and audit each other's verified components.
- The skeleton abstraction suggests the same agents could target other workflow management systems by replacing the deployment agent and I/O stubs, so the core contribution may transfer beyond serverless functions.
- A stress test with repeated runs, different underlying language models, and different target systems would convert the current feasibility result into a reliability claim; until then, the strongest open question is whether one successful run per workflow adequately represents system behavior.
- The post-run agent's ability to propose a substantive model correction hints that monitoring agents could evolve from failure reporters into hypothesis proposers, which would enlarge the role of human approval.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces CURATE, a human-in-the-loop multi-agent system that composes, generates, deploys, and reuses scientific workflows, with a module catalog as a core mechanism. The prototype uses LangGraph and Claude Opus 4.8 to implement a nine-stage pipeline with five human-in-the-loop gates, targeting FaaSr on GitHub Actions. Feasibility is evaluated through six experiments: four SeBS-Flow-derived workflows and two PyADM1 anaerobic-digestion workflows. The authors report that CURATE recreated three SeBS-Flow workflows end-to-end, reused a MapReduce subgraph to build a new workflow, automated a preprocessing-plus-simulation pipeline, and compared token usage and success rates against a single coding-agent baseline.
Significance. If the reported results hold, the paper makes a useful empirical contribution: it demonstrates that an agentic workflow system can go beyond code generation to handle deployment, monitoring, and reusable module catalogs, and it provides an open-source prototype and a grounded, benchmark-anchored evaluation. The reuse of a MapReduce subgraph in Experiment 4 and the WDA-initiated revision in Experiment 6 are concrete, falsifiable demonstrations of the claimed lifecycle support. The main value is as a feasibility proof rather than a rigorous comparative study; the central claim is not a theoretical derivation, so the evaluation's correctness and internal consistency are what matter most.
major comments (4)
- [V-A.3 and Table I] The paper's headline claim that CURATE 'faithfully recreated 3 workflows in the SeBS-Flow serverless benchmark suite' (Section VII) is contradicted by its own results table. Section V-A.3 states that the video-analysis workflow 'correctly identified the cars and did not produce any runtime errors,' but Table I lists Experiment 3's CURATE Success as 'GT ✗', which per the table legend means the output did not match ground truth. This is directly load-bearing: if the table is correct, only two SeBS-Flow workflows were recreated; if the text is correct, the table contains an error that casts doubt on the other reported success labels. The manuscript must reconcile these statements, and the repository artifact should be checked to determine which version is accurate. At a minimum, the abstract, Section V introduction, and Section VII all need to be adjusted to match the corrected outcome.
- [V, Baseline paragraph] The baseline comparison is not apples-to-apples, and the claim that the baseline has 'no room for HITL revision' is undercut by the procedure described. The paper states that each baseline experiment was run three times and that 'in all such cases, we manually corrected the issue' for FaaSr-related errors. Those manual corrections are human interventions applied after the coding agent's own attempts, so the baseline's reported success counts in Table I do not reflect an unaided agent. The paper should quantify the manual corrections per run, report whether they were applied before or after deciding the success of a run, and explicitly acknowledge that CURATE was executed once per experiment while the baseline received up to three attempts plus manual fixes. This is important because the central feasibility claim is partly supported by the contrast with the baseline.
- [V, Experiments 5 and 6] The 'EV ✓' success labels for Experiments 5 and 6 rest on an unspecified expert-verification protocol. The text says the authors 'rely on domain expert knowledge to verify the behavior of data reconciliation and that model outputs respond to SRT in the expected directions,' but it does not state who the expert was, what exact criteria and thresholds were used, whether the expert saw only final outputs or also intermediate artifacts, or how disagreements were resolved. Without this information, the E5/E6 success claims cannot be independently checked, and the statement in Section VI that the WDA revision 'produced the correct results' is an assertion rather than a documented evaluation. Please provide a reproducible verification protocol, even a brief one.
- [V-A.3] The sentence 'In SeBS-Flow, the results were flattened to include only the outputs of a single frame; we fixed this manually' is ambiguous about what was fixed and by whom. If the SeBS-Flow baseline output was restructured manually before comparison, the paper must clarify whether this manual step affected the ground-truth comparison for Experiment 3, especially given the conflicting GT ✗ label in Table I.
minor comments (5)
- [V-A.5] The text 'T is hard-coded in PyADM1' appears to be a typo: Section VI and Experiment 6 describe the hard-coded parameter as q_ad, the digester feed flow. Please correct the parameter name for consistency.
- [Table I] The 'Rev.' column mixes dashes and checkmarks in a way that is hard to read; use a uniform notation, such as '—' for no revision and 'G_k ✓' for a revision at a gate, and make clear whether the checkmark indicates that the revision was applied successfully.
- [V, Introduction] The abstract says the experiments include 'reproducing and adapting 4 workflows derived from the SeBS-Flow benchmark suite,' while the text says 'We first recreate three SeBS-Flow workflows fully end-to-end' and then describes Experiment 4 as a reuse-and-extension exercise. Clarify whether Experiment 4 is counted as a reproduction or an adaptation so the counts are consistent.
- [IV-A and V] The claim that 'it is trivial to target other platforms using FaaSr' is not supported by evidence in this paper; either add a citation to FaaSr documentation or soften the claim.
- [Figure 3] The baseline token-usage bars appear to be means over three runs, but no variance is shown; adding error bars or reporting individual run values would make the comparison more informative, especially given the baseline's reported nondeterminism.
Circularity Check
No significant circularity: CURATE's feasibility claims are evaluated against external ground truth and expert verification, with no fitted parameter or self-citation chain doing load-bearing work.
full rationale
The paper makes no numerical derivation or statistical fit whose output could reduce to its input by construction. The central feasibility claims are anchored to external benchmarks: SeBS-Flow ground truth for experiments 1-3, a hand-computed word count from the PDF for experiment 4, and domain-expert verification for experiments 5-6. The catalog reuse in experiment 4 is evaluated against the manually counted word frequencies of the PDF, not against the reused modules' prior outputs, so the success criterion is independent of the reused code. The WDA revision suggestion in experiment 6 is a demonstrated system feature, not a fitted parameter: the workflow's success is checked against whether the 20 SRT-ranked runs produce distinct steady states in the expected direction, and the suggestion itself is part of the proposed human-in-the-loop design rather than a hidden input to the claimed result. The only self-references are to the open-source prototype repository [33], which is an artifact of the work and not cited as proof of any mathematical or empirical claim. The manuscript does contain a substantive internal inconsistency: Table I marks experiment 3 as 'GT ✗' while Section V-A.3 states the CURATE workflow correctly identified the cars, and the conclusion claims faithful recreation of 3 SeBS-Flow workflows. This is a correctness and reporting risk, not circularity, because neither statement is derived from the other. Similarly, the acknowledged limitations about SeBS-Flow being a performance benchmark, the ADM1 workflow covering only initial data preparation, and the single-run generalization issue are concerns about scope and robustness, not about circular reasoning. No step in the paper's argument is equivalent to its own inputs by definition.
Assumptions & free parameters
assumptions (4)
- domain assumption Claude Opus 4.8 coding-agent reliability
- domain assumption FaaSr function-granularity DAG workflows are a sufficient proxy for scientific workflows
- domain assumption BM25 retrieval is adequate for catalog source resolution
- domain assumption Domain expert verification of PyADM1 outputs is sufficient
Cite this review
Pith. "Pith review of CURATE: Leveraging LLM Agents to Compose, Catalog, and Deploy Reproducible Workflows." pith.science (2026). https://pith.science/paper/QADN3Z4F
@misc{pith2026260804270,
author = {Pith},
title = {Pith review of: CURATE: Leveraging LLM Agents to Compose, Catalog, and Deploy Reproducible Workflows},
year = {2026},
howpublished = {\url{https://pith.science/paper/QADN3Z4F}},
note = {Machine review of arXiv:2608.04270}
}
read the original abstract
Agentic code generation has shown promise in automating and accelerating software development by utilizing Large Language Models (LLMs) to generate, test, and deploy code. For engineers and scientists, such systems have the potential to accelerate the development of applied and scientific workflows while reducing barriers to entry in domains that have yet to fully realize their benefits. However, a key gap remains: existing coding agents primarily focus on code generation and do not address the entire workflow lifecycle, including deployment and sharing. As a result, users develop and stitch modules independently while managing deployment on their own. To address this gap, we propose CURATE - Composition, User-in-the-loop, Reuse, and Automated Task Execution - a novel human-in-the-loop multi-agent system that uses LLM agents to manage and develop composable workflows across their entire lifecycle. A key feature of the system is a catalog that allows for the storage and reuse of modules across workflows. Module catalogs provide a foundation that can be expanded to support FAIR principles by facilitating the sharing and reuse of curated modules and subgraphs. We demonstrate the feasibility of our system with an initial prototype using Claude Opus 4.8, comprising 6 experiments: reproducing and adapting 4 workflows derived from the SeBS-Flow benchmark suite, and automating the development and scaling of a workflow that leverages a complex mechanistic model in environmental engineering used to simulate anaerobic digestion.
Figures
Reference graph
Works this paper leans on
-
[1]
Scientific workflow management and the kepler system,
B. Lud ¨ascheret al., “Scientific workflow management and the kepler system,”Concurr. Comput. : Pract. Exper., vol. 18, no. 10, p. 1039–1065, Aug. 2006
work page 2006
-
[2]
Workflows and e-science: An overview of workflow system features and capabilities,
E. Deelman, D. Gannon, M. Shields, and I. Taylor, “Workflows and e-science: An overview of workflow system features and capabilities,” Future Gener. Comput. Syst., vol. 25, no. 5, p. 528–540, May 2009
work page 2009
-
[3]
A community roadmap for scientific workflows research and development,
R. F. da Silvaet al., “A community roadmap for scientific workflows research and development,” inProc. IEEE Workshop on Workflows in Support of Large-Scale Science (WORKS), Nov. 2021, p. 81–90
work page 2021
-
[4]
Llms for science: Usage for code generation and data analysis,
M. Nejjar, L. Zacharias, F. Stiehle, and I. Weber, “Llms for science: Usage for code generation and data analysis,”Journal of Software: Evolution and Process, vol. 37, no. 1, p. e2723, 2025
work page 2025
-
[5]
A. Dhruv and A. Dubey, “Leveraging large language models for code translation and software development in scientific computing,” inProc. Platform for Advanced Scientific Computing Conf. (PASC), 2025, p. 1–9
work page 2025
-
[6]
Llm4workflow: An llm-based automated workflow model generation tool,
J. Xu, W. Du, X. Liu, and X. Li, “Llm4workflow: An llm-based automated workflow model generation tool,” inProc. IEEE/ACM Int. Conf. on Automated Software Engineering (ASE), 2024, p. 2394–2398
work page 2024
-
[7]
Toolformer: Language models can teach themselves to use tools,
T. Schick, J. Dwivedi-Yu, R. Dess `ı, R. Raileanu, M. Lomeli, L. Zettlemoyer, N. Cancedda, and T. Scialom, “Toolformer: Language models can teach themselves to use tools,” 2023. [Online]. Available: https://arxiv.org/abs/2302.04761
arXiv 2023
-
[8]
SWE-agent: Agent-computer interfaces enable automated software engineering,
J. Yanget al., “SWE-agent: Agent-computer interfaces enable automated software engineering,” inProc. NeurIPS, 2024
work page 2024
Show all 37 references
-
[9]
React: Synergizing reasoning and acting in language models,
S. Yaoet al., “React: Synergizing reasoning and acting in language models,” inProc. Int. Conf. on Learning Representations (ICLR), 2023
2023
-
[10]
Reflexion: language agents with verbal reinforcement learning,
N. Shinn, F. Cassano, A. Gopinath, K. Narasimhan, and S. Yao, “Reflexion: language agents with verbal reinforcement learning,” in Proc. NeurIPS, 2023
2023
-
[11]
Human- in-the-loop software development agents,
W. Takerngsaksiri, J. Pasuksmit, P. Thongtanunam, C. Tantithamthavorn, R. Zhang, F. Jiang, J. Li, E. Cook, K. Chen, and M. Wu, “Human- in-the-loop software development agents,” 2025. [Online]. Available: https://arxiv.org/abs/2411.12924
2025 arXiv
-
[12]
Claude code,
Anthropic, “Claude code,” 2025, [Online]. Available: https://www.anthropic.com/claude-code. [Accessed: Jul. 2026]
2025
-
[13]
Available: https://openai.com/codex/
OpenAI, “Codex,” 2025, [Online]. Available: https://openai.com/codex/. [Accessed: Jul. 2026]
2025
-
[14]
Workflowhub: a registry for computational workflows,
O. J. R. Gustafssonet al., “Workflowhub: a registry for computational workflows,”Scientific Data, vol. 12, no. 1, May 2025
2025
-
[15]
Methods included: standardizing computational reuse and portability with the common workflow language,
M. R. Crusoeet al., “Methods included: standardizing computational reuse and portability with the common workflow language,”Communi- cations of the ACM, vol. 65, no. 6, 2022
2022
-
[16]
Applying the fair principles to computational workflows,
S. R. Wilkinsonet al., “Applying the fair principles to computational workflows,”Scientific Data, vol. 12, no. 1, Feb. 2025
2025
-
[17]
A terminology for scientific workflow systems,
F. Suteret al., “A terminology for scientific workflow systems,”Future Generation Computer Systems, vol. 174, p. 107974, 2026
2026
-
[18]
Agent skills for large language models: Architecture, acquisition, security, and the path forward,
R. Xu and Y . Yan, “Agent skills for large language models: Architecture, acquisition, security, and the path forward,” 2026. [Online]. Available: https://arxiv.org/abs/2602.12430
2026 arXiv
-
[19]
From specification to execution: Ai assisted scientific workflow management,
K. Thareja, H. Safri, R. Mayani, A. Mandal, and E. Deelman, “From specification to execution: Ai assisted scientific workflow management,”
-
[20]
Pegasus, a workflow management system for science automation,
E. Deelmanet al., “Pegasus, a workflow management system for science automation,”Future Gener. Comput. Syst., vol. 46, no. C, p. 17–35, May 2015
2015
-
[21]
Can an llm find its way around a spreadsheet?
C.-T. Leeet al., “Can an llm find its way around a spreadsheet?” in Proc. IEEE/ACM Int. Conf. on Software Engineering (ICSE), 2025, p. 294–306
2025
-
[22]
Similarity search for scientific workflows,
J. Starlinger, B. Brancotte, S. Cohen-Boulakia, and U. Leser, “Similarity search for scientific workflows,”Proc. VLDB Endow., vol. 7, no. 12, p. 1143–1154, Aug. 2014
2014
-
[23]
Accelerating scientific discovery with co-scientist,
J. Gottweiset al., “Accelerating scientific discovery with co-scientist,” Nature, vol. 655, no. 8122, p. 487–496, May 2026
2026
-
[24]
Claude science: An AI workbench for scientists,
Anthropic, “Claude science: An AI workbench for scientists,” https: //www.anthropic.com/news/claude-science, Jun. 2026, accessed: 2026- 07-17
2026
-
[25]
Langchain-parsl: Connect large language model agents to high per- formance computing resource,
H. Ma, A. Brace, C. Siebenschuh, I. Foster, and A. Ramanathan, “Langchain-parsl: Connect large language model agents to high per- formance computing resource,” inProc. SC ’25 Workshops, 2025, p. 78–85
2025
-
[26]
Parsl: Pervasive parallel programming in python,
Y . Babujiet al., “Parsl: Pervasive parallel programming in python,” inProc. Int. Symp. on High-Performance Parallel and Distributed Computing (HPDC), Jun. 2019, p. 25–36
2019
-
[27]
Do large language models speak scientific workflows?
O. Yildiz and T. Peterka, “Do large language models speak scientific workflows?” inProc. SC ’25 Workshops, 2025, p. 2225–2233
2025
-
[28]
The (r)evolution of scientific workflows in the agentic ai era: Towards autonomous science,
W. Shinet al., “The (r)evolution of scientific workflows in the agentic ai era: Towards autonomous science,” inProc. SC ’25 Workshops, 2025, p. 2305–2316
2025
-
[29]
A-mem: Agentic memory for llm agents,
W. Xu, Z. Liang, K. Mei, H. Gao, J. Tan, and Y . Zhang, “A-mem: Agentic memory for llm agents,” inProc. NeurIPS, 2025
2025
-
[30]
Model context protocol specifi- cation,
Model Context Protocol Contributors, “Model context protocol specifi- cation,” https://modelcontextprotocol.io/specification/2025-11-25, 2025, revision 2025-11-25. Accessed: 2026-07-18
2025
-
[31]
Empowering scientific workflows with federated agents,
A. Kamataret al., “Empowering scientific workflows with federated agents,” inProc. IEEE Int. Parallel and Distributed Processing Sympo- sium (IPDPS), 2026
2026
-
[32]
Characterizing serverless platforms with ServerlessBench,
T. Yuet al., “Characterizing serverless platforms with ServerlessBench,” inProc. 11th ACM Symp. Cloud Computing (SoCC), 2020, pp. 30–44
2020
-
[33]
CURATE-prototype,
N. Cutler, “CURATE-prototype,” https://github.com/nolcut/CURATE-p rototype, 2026, accessed: Aug. 3, 2026
2026
-
[34]
Faasr: Cross-platform function-as-a-service serverless scientific workflows in r,
S. Parket al., “Faasr: Cross-platform function-as-a-service serverless scientific workflows in r,” inProc. IEEE Int. Conf. on e-Science, 2024
2024
-
[35]
The probabilistic relevance framework: Bm25 and beyond,
S. Robertson and H. Zaragoza, “The probabilistic relevance framework: Bm25 and beyond,”Found. Trends Inf. Retr., vol. 3, no. 4, p. 333–389, Apr. 2009
2009
-
[36]
Sebs-flow: Benchmarking serverless cloud function work- flows,
L. Schmid, M. Copik, A. Calotoiu, L. Brandner, A. Koziolek, and T. Hoefler, “Sebs-flow: Benchmarking serverless cloud function work- flows,” inProc. European Conf. on Computer Systems (EuroSys), 2025, p. 902–920
2025
-
[2026]
Available: https://arxiv.org/abs/2606.18425
[Online]. Available: https://arxiv.org/abs/2606.18425
Reviewed August 8, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.