Pith. sign in

REVIEW 3 major objections 5 minor 67 references

The paper claims that a system is a prompt graph if and only if it meets four conditions — explicit structure, separation of structure from content, executable semantics, and first-class artifact status — and operationalizes them as a four-

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 05:17 UTC pith:ALEAAYBH

load-bearing objection A careful, honest definitional paper that mostly delivers: the four-condition test is new and useful, but the 'iff' is a stipulation and the mixed-mode verdicts need an explicit aggregation rule. the 3 major comments →

arxiv 2607.27578 v1 pith:ALEAAYBH submitted 2026-07-30 cs.AI

What makes prompts a graph: necessary and sufficient conditions for prompt graph engineering

Sandeco Macedo This is my paper
classification cs.AI
keywords prompt graph engineeringprompt engineeringlarge language modelsLLM pipelinesagentic workflowsgraph of thoughtsoperational definitionmulti-agent systems
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper argues that a distinct engineering discipline is forming around structured, multi-call language-model systems: prompts sit as nodes in an explicit graph, and the field needs a reference definition it currently lacks. The author proposes a constitutive definition: a practice or system belongs to 'prompt graph engineering' if and only if it instantiates four conditions — explicit structure, separation of structure from prompt content, executable semantics, and first-class artifact status. These conditions are turned into a four-question inclusion and exclusion test that classifies real systems consistently. If the definition holds, engineers gain a shared vocabulary for comparing frameworks and researchers gain a way to ask when the structure, not the wording, is what improves a system.

Core claim

The central claim is the constitutive definition: prompt graph engineering is the discipline that represents, composes, and executes prompt-mediated language model computation as an explicit graph, and a practice or system belongs to it if and only if it instantiates four elements. G1 requires an explicit structure with authored units as nodes and dependencies as edges, enumerable without running the system. G2 requires separation of structure from prompt content, so either can change without rewriting the other. G3 requires executable semantics, a runtime that schedules nodes, routes outputs, and manages shared state including branching, parallelism, and cycles. G4 requires the graph to be

What carries the argument

The carrying mechanism is the four-condition constitutive definition (G1–G4) paired with its operationalized decision procedure (T1–T4). G1 demands an explicit, enumerable node-and-edge representation; G2 demands that structure and prompt text vary independently; G3 demands a runtime that executes the graph rather than hand-written control flow; G4 demands that the graph exist as an object beyond any single run, consumable by type checkers, visualizers, diffs, or optimizers. The test classifies candidates by asking four yes/no questions, with each 'no' naming the neighboring category into which the candidate falls. The definition separates the genus — the dataflow tradition of explicit, exec

Load-bearing premise

The definition's 'necessary and sufficient' claim rests on the assumption that the four conditions fixed in advance from the cited lineages are genuinely necessary — especially G2 (structure separates from prompt content) and G4 (the graph is a first-class artifact) — which the paper supports with removal thought experiments rather than a proof or an agreed benchmark.

What would settle it

A widely used system that the community unambiguously calls prompt graph engineering yet that fails one of the four tests — for example, a framework where structural wiring and prompt text are edited together in one declaration, or one where the graph exists only as an ephemeral runtime trace but is still optimized in practice — would falsify the necessity of G2 or G4. A simpler empirical check: an inter-rater study in which independent analysts applying T1–T4 to the same systems reach different verdicts on any of them would show the test is not operational as stated.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Any candidate practice or system can be classified by answering four yes/no questions, and failing any one places it in a specific neighboring category rather than leaving it undefined.
  • The boundary against six neighboring concepts is drawn by named failing conditions: classic prompt engineering fails on explicit structure, thought topologies fail on separation and authorship, emergent agent conversations fail on explicit structure and artifact status, RAG pipelines typically fail on artifact status or separation, and classic workflow engines lack the prompt-parameterized node.
  • Applied to six contemporary systems, the test includes three, splits two by operating mode, and excludes one, showing it can discriminate within a small sample chosen to span the practice.
  • The entire lineage of automatic prompt and workflow optimization presupposes G4: a structure can only be searched, compiled, or refactored if it exists as an object, which gives the definition a direct connection to optimization research.
  • Membership is binary but maturity is gradual: a small retrieve-generate-verify graph serialized as a file is a legitimate instance, while a compiled, optimizer-armed program differs in degree of maturity, not in membership.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The paper leaves open whether emergence can serve as a discovery mode for structure: if a system records an emergent delegation flow, lifts it into an explicit graph, and replays or refines it, the explicit/emergent distinction becomes a pipeline rather than a boundary. This extends the paper's hint about dynamic DAG construction.
  • A direct empirical follow-up suggested by the paper's own caveats is an inter-rater study: independent analysts applying T1–T4 to the same systems should reach the same verdicts, and disagreement on any system would test the test's operational claim.
  • If the definition gains traction, it becomes a design checklist for framework builders: a framework can be evaluated by whether it offers enumerable structure, separable content, a runtime, and an optimizable artifact, turning a terminological proposal into an engineering specification.
  • The author's removal thought experiments for G2 and G4 could be complemented by adversarial case collection — seeking out systems that practitioners call prompt graph engineering despite interleaved structure/content or ephemeral-but-optimizable graphs — to pressure-test the claimed necessity.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes a constitutive definition of 'prompt graph engineering': the discipline that represents, composes, and executes prompt-mediated language model computation as an explicit graph, with four necessary conditions (G1 explicit structure, G2 separation of structure and prompt content, G3 executable semantics, G4 first-class artifact status). This is stated as an iff claim and operationalized as a T1–T4 inclusion/exclusion test. The paper reconstructs the genealogy of the idea from dataflow systems through chaining and thought topologies to optimizable workflow graphs, delimits the concept from six neighboring notions, applies the test to six real systems (LangGraph, DSPy, Prompt Flow included; AutoGen and CrewAI split by operating mode; Claude Code subagents excluded), and closes with a research agenda organized along four design tension axes. The paper is explicitly definitional and acknowledges several limitations, including the thought-experiment status of necessity and the single-analyst application of the test.

Significance. If accepted, the definition would give a fast-moving area a shared, checkable vocabulary: it separates membership from quality, attaches a named failing condition to each neighboring concept, and makes a falsifiable classification claim, e.g., that Claude Code subagent delegation is excluded. The paper is honest about its evidential basis: it gives verifiable criteria for each test item, states that the four conditions were fixed before examining current frameworks, and openly reports that no second rater has repeated the classification. These strengths make the core contribution genuinely useful. The central 'necessary and sufficient' claim, however, currently outruns the evidence offered, and the decision procedure is incomplete for mixed-mode systems, which is a load-bearing gap for an operational definition.

major comments (3)
  1. [§4, Table 2] The central claim is a system-level iff: 'A practice, or a system, belongs to prompt graph engineering if and only if it instantiates the four elements.' Yet the test output for AutoGen is 'Included in GraphFlow mode; excluded in emergent conversation' and for CrewAI is 'partial.' No aggregation rule is given for systems with multiple operating modes: is membership existential over modes, universal, or determined by a primary mode? Without such a rule, 'belongs iff' is undefined for the very systems Table 2 labels 'partial,' and the abstract's statement that the test 'includes and excludes consistently' across six real systems overreaches. This is an internal gap in the decision procedure, not a scope caveat. Please either restrict the iff to a specific practice or configuration, or add an explicit system-level aggregation rule.
  2. [§4] The necessity of G1–G4 is supported by remove-one-condition thought experiments. These illustrate that each condition is useful — that removing it forfeits inspection, reuse, executability, or optimizability — but they do not show that no practice lacking the condition could count as prompt graph engineering. In particular, G2 and G4 are stipulative in ways that a reader could contest: a system with interleaved structure and content, or an ephemeral-but-optimizable graph, is not obviously outside the discipline. Since the 'necessary and sufficient' claim is the paper's central contribution, the necessity step needs either a tighter conceptual argument against plausible counterexamples or an explicit reframing as a stipulative constitutive definition whose criteria are accepted by convention rather than by proof.
  3. [§6] The paper states that 'no second rater has yet repeated it, and inter-rater agreement is the obvious first empirical follow-up.' As a consequence, the claim that the test 'includes and excludes consistently' is currently unverified: Table 2 is a single-analyst classification. The verifiable criteria reduce subjectivity, but they do not establish inter-rater reliability, and the application section is the only empirical evidence for the consistency claim in RQ4. Either report an inter-rater exercise, provide a fuller audit trail of how each T1–T4 judgment was reached, or soften the wording to 'classifies these six systems as follows under the stated criteria.'
minor comments (5)
  1. [Abstract] The first paragraph has spacing errors: 'Graphnames' and 'prompt engineeringstill' should be 'Graph names' and 'prompt engineering still.'
  2. [§4] 'There is no fifth condition to add' is asserted without argument. A paragraph explaining why observability, caching, cost control, and human-in-the-loop nodes are specializations rather than additional conditions would strengthen the definition.
  3. [§6] The sentence 'StateFlow makes the same movement inside the same ecosystem' is unclear; specify that 'movement' refers to reifying an emergent conversational flow into an explicit state machine.
  4. [Figure 4 / Table 2] Figure 4 uses half-filled cells for partial conditions while Table 2 uses 'partial' with textual qualifications. Ensure the visual legend explicitly defines half-filled cells as 'passes in one operating mode, fails in another' so the two presentations are unambiguous.
  5. [§7] 'Three transversal problems' is unusual in English; 'cross-cutting problems' or 'transverse problems' would be clearer.

Circularity Check

0 steps flagged

No significant circularity: the constitutive definition is stipulative, and the application test has independent discriminative content.

full rationale

The paper does not derive a prediction from fitted data. Its central claim is a constitutive definition ('A practice, or a system, belongs to prompt graph engineering if and only if it instantiates the four elements'), which is stipulative: G1–G4 and the corresponding T1–T4 are the definition, and §6 is an application of that instrument, not a test that could falsify it. The genealogy in §3 provides motivation, not evidence that makes the classification circular. The application has independent discriminative content: AutoGen and CrewAI split across modes and Claude Code subagents are excluded (Table 2), which a test simply fit to the graph-first frameworks would not produce. The stated limitation in §6 that classification was single-rater and 'no second rater has yet repeated it' is an empirical validation gap, not a circular step. Self-citations [7, 32] appear only in related-work positioning and are not load-bearing for the necessity/sufficiency argument. No equation or parameter reduces to its own input; no prediction is forced by construction.

Axiom & Free-Parameter Ledger

0 free parameters · 4 axioms · 1 invented entities

The paper rests on the stipulation that G1–G4 are necessary and sufficient. It introduces one central invented entity — the named discipline itself — but gives it an operational, independently applicable test. There are no fitted free parameters; the paper is conceptual, not empirical.

axioms (4)
  • ad hoc to paper The four conditions (G1–G4) are necessary and sufficient for prompt graph engineering.
    The paper's central stipulation; argued by removal thought experiments in §4, not formal proof.
  • domain assumption Graph is the correct representation for prompt-mediated computation.
    Motivates the definition; asserted in §1 as the convergence of practice.
  • domain assumption Conceptual analysis over DOI-verified sources and grey literature is a valid method for grounding the definition.
    Method stated in §1 and §2; the paper is deliberately definitional rather than empirical.
  • ad hoc to paper Membership is binary (T1–T4 all yes) while quality is gradual.
    Distinction stated in §4 caveats; a design choice that makes the test decisive.
invented entities (1)
  • prompt graph engineering (the named discipline) independent evidence
    purpose: Provides a shared vocabulary and a membership test for prompt-mediated computation structured as graphs.
    The concept is introduced as the paper's contribution and operationalized through the T1–T4 test, which can be applied to any system; thus it has a checkable handle outside the paper.

pith-pipeline@v1.3.0-daily-deepseek · 16900 in / 10843 out tokens · 92227 ms · 2026-08-01T05:17:40.682243+00:00 · methodology

0 comments
read the original abstract

Prompts stopped being isolated strings some time ago. In real systems, one model call feeds another, retrieval interleaves with generation, routers branch, and aggregators merge parallel results. Practice converged on a single structure to hold this together: the graph. Frameworks such as LangGraph, DSPy, and Prompt Flow expose it openly, and research systems already optimize it automatically. The vocabulary, however, lags behind. Graph names, variously, a reasoning topology inside one sampling strategy, a multi-agent conversation, or an orchestration artifact, while prompt engineering still evokes writing one good string. What is missing is a reference definition treating prompts as nodes of an explicit, executable, improvable graph. We build that definition through conceptual analysis over sources with persistent identifiers, complemented by primary grey literature. We reconstruct the genealogy of the idea, from dataflow graphs and build systems, through prompt chaining and the thought topologies (chain, tree, graph), to graphs compiled and optimized as artifacts. We then propose a constitutive definition of prompt graph engineering, state its four conditions (explicit structure, separation between structure and prompt content, executable semantics, and the graph as a first-class engineering artifact), and operationalize them as an inclusion and exclusion test. We draw the boundary against six neighboring concepts and apply the test to six real systems (LangGraph, DSPy, Prompt Flow, AutoGen, CrewAI, and Claude Code subagents); it includes and excludes consistently. We close with a research agenda organized along four design tension axes. The contribution is an operational definition and a shared vocabulary for a practice that industry already exercises daily without naming precisely.

Figures

Figures reproduced from arXiv: 2607.27578 by Sandeco Macedo.

Figure 1
Figure 1. Figure 1: Genealogy of the prompt graph. The graph as model of computation (dataflow, build [PITH_FULL_IMAGE:figures/full_fig_p005_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: From string to graph. The four shapes of prompt-mediated computation: the single [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Anatomy of prompt graph engineering. At the center, the graph: authored prompt [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: The inclusion and exclusion test applied to six real systems. Filled cells indicate that [PITH_FULL_IMAGE:figures/full_fig_p012_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: The six analyzed systems positioned on the four design tension axes: explicit versus [PITH_FULL_IMAGE:figures/full_fig_p014_5.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

67 extracted references · 2 canonical work pages · 2 internal anchors

  1. [1]

    Graph of Thoughts: Solving Elaborate Problems with Large Language Models

    Maciej Besta, Nils Blach, Ales Kubicek, Robert Gerstenberger, Michal Podstawski, Lukas Gianinazzi, Joanna Gajda, Tomasz Lehmann, Hubert Niewiadomski, Piotr Nyczyk, and Torsten Hoefler. Graph of Thoughts: Solving Elaborate Problems with Large Language Models. InProceedings of the AAAI Conference on Artificial Intelligence, pages 17682–17690,

  2. [2]

    Demystifying Chains, Trees, and Graphs of Thoughts, 2024

    Maciej Besta, Florim Memedi, Zhenyu Zhang, Robert Gerstenberger, Guangyuan Piao, Nils Blach, Piotr Nyczyk, Marcin Copik, Grzegorz Kwaśniewski, Jürgen Müller, Lukas Gianinazzi, Ales Kubicek, Hubert Niewiadomski, Aidan O’Mahony, Onur Mutlu, and Torsten Hoefler. Demystifying Chains, Trees, and Graphs of Thoughts, 2024. Preprint, arXiv. doi:10.48550/arXiv.2401.14295

    doi:10.48550/arxiv.2401.14295 2024

  3. [3]

    Prompting Is Programming: A Query Language for Large Language Models.Proceedings of the ACM on Programming Languages, 7:1946–1969, 2023

    Luca Beurer-Kellner, Marc Fischer, and Martin Vechev. Prompting Is Programming: A Query Language for Large Language Models.Proceedings of the ACM on Programming Languages, 7:1946–1969, 2023. doi:10.1145/3591300

    doi:10.1145/3591300 1946

  4. [4]

    Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin...

    Pith/arXiv arXiv doi:10.48550/arxiv.2005.14165 2020

  5. [5]

    Wenhu Chen, Xueguang Ma, Xinyi Wang, and William W. Cohen. Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Tasks,

  6. [6]

    Data Flow Program Graphs.Computer, 15:26–41, 1982

    Davis and Keller. Data Flow Program Graphs.Computer, 15:26–41, 1982. doi:10.1109/MC.1982.1653939

    arXiv 1982

  7. [7]

    What makes a harness a harness: necessary and sufficient conditions for an agent harness

    Sanderson Oliveira de Macedo. What makes a harness a harness: necessary and sufficient conditions for an agent harness, 2026. Preprint, arXiv. doi:10.48550/arXiv.2606.10106

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2606.10106 2026

  8. [8]

    Workflows and e-Science: An overview of workflow system features and capabilities.Future Generation Computer Systems, 25:528–540, 2009

    Ewa Deelman, Dennis Gannon, Matthew Shields, and Ian Taylor. Workflows and e-Science: An overview of workflow system features and capabilities.Future Generation Computer Systems, 25:528–540, 2009. doi:10.1016/j.future.2008.06.012

    doi:10.1016/j.future.2008.06.012 2009

  9. [9]

    Jack B. Dennis. First version of a data flow procedure language. InLecture Notes in Computer Science, pages 362–376, 1974. doi:10.1007/3-540-06859-7_145

    doi:10.1007/3-540-06859-7_145 1974

  10. [10]

    Everything of Thoughts: De- fying the Law of Penrose Triangle for Thought Generation, 2023

    Ruomeng Ding, Chaoyun Zhang, Lu Wang, Yong Xu, Minghua Ma, Wei Zhang, Si Qin, Saravan Rajmohan, Qingwei Lin, and Dongmei Zhang. Everything of Thoughts: De- fying the Law of Penrose Triangle for Thought Generation, 2023. Preprint, arXiv. doi:10.48550/arXiv.2311.04254. 15

    Pith/arXiv arXiv doi:10.48550/arxiv.2311.04254 2023

  11. [11]

    Saurous, Jascha Sohl-dickstein, Kevin Murphy, and Charles Sutton

    David Dohan, Winnie Xu, Aitor Lewkowycz, Jacob Austin, David Bieber, Raphael Gon- tijo Lopes, Yuhuai Wu, Henryk Michalewski, Rif A. Saurous, Jascha Sohl-dickstein, Kevin Murphy, and Charles Sutton. Language Model Cascades, 2022. Preprint, arXiv. doi:10.48550/arXiv.2207.10342

    Pith/arXiv arXiv doi:10.48550/arxiv.2207.10342 2022

  12. [12]

    Stuart I. Feldman. Make, a program for maintaining computer programs.Software: Practice and Experience, 9:255–265, 1979. doi:10.1002/spe.4380090402

    doi:10.1002/spe.4380090402 1979

  13. [13]

    Promptbreeder: Self-Referential Self-Improvement Via Prompt Evolution,

    Chrisantha Fernando, Dylan Banarse, Henryk Michalewski, Simon Osindero, and Tim Rocktäschel. Promptbreeder: Self-Referential Self-Improvement Via Prompt Evolution,

  14. [14]

    PAL: Program-aided Language Models, 2022

    Luyu Gao, Aman Madaan, Shuyan Zhou, Uri Alon, Pengfei Liu, Yiming Yang, Jamie Callan, and Graham Neubig. PAL: Program-aided Language Models, 2022. Preprint, arXiv. doi:10.48550/arXiv.2211.10435

    Pith/arXiv arXiv doi:10.48550/arxiv.2211.10435 2022

  15. [15]

    Retrieval-Augmented Generation for Large Language Models: A Survey, 2023

    Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yi Dai, Jiawei Sun, Meng Wang, and Haofen Wang. Retrieval-Augmented Generation for Large Language Models: A Survey, 2023. Preprint, arXiv. doi:10.48550/arXiv.2312.10997

    Pith/arXiv arXiv doi:10.48550/arxiv.2312.10997 2023

  16. [16]

    Thomas R. Gruber. Toward principles for the design of ontologies used for knowl- edge sharing?International Journal of Human-Computer Studies, 43:907–928, 1995. doi:10.1006/ijhc.1995.1081

    arXiv 1995

  17. [17]

    Chawla, Olaf Wiest, and Xiangliang Zhang

    Taicheng Guo, Xiuying Chen, Yaqi Wang, Ruidi Chang, Shichao Pei, Nitesh V. Chawla, Olaf Wiest, and Xiangliang Zhang. Large Language Model based Multi-Agents: A Survey of Progress and Challenges, 2024. Preprint, arXiv. doi:10.48550/arXiv.2402.01680

    Pith/arXiv arXiv doi:10.48550/arxiv.2402.01680 2024

  18. [18]

    MetaGPT: Meta Programming for A Multi-Agent Collaborative Framework, 2023

    Sirui Hong, Mingchen Zhuge, Jiaqi Chen, Xiawu Zheng, Yuheng Cheng, Ceyao Zhang, Jinlin Wang, ZiliWang, StevenKaShing Yau, Zijuan Lin, LiyangZhou, ChenyuRan, Lingfeng Xiao, Chenglin Wu, and Jürgen Schmidhuber. MetaGPT: Meta Programming for A Multi-Agent Collaborative Framework, 2023. Preprint, arXiv. doi:10.48550/arXiv.2308.00352

    Pith/arXiv arXiv doi:10.48550/arxiv.2308.00352 2023

  19. [19]

    Automated Design of Agentic Systems, 2024

    Shengran Hu, Cong Lu, and Jeff Clune. Automated Design of Agentic Systems, 2024. Preprint, arXiv. doi:10.48550/arXiv.2408.08435

    Pith/arXiv arXiv doi:10.48550/arxiv.2408.08435 2024

  20. [20]

    Graph Chain-of-Thought: Augmenting Large Language Models by Reasoning on Graphs, 2024

    Bowen Jin, Chulin Xie, Jiawei Zhang, Kashob Kumar Roy, Yu Zhang, Zheng Li, Ruirui Li, Xianfeng Tang, Suhang Wang, Yu Meng, and Jiawei Han. Graph Chain-of-Thought: Augmenting Large Language Models by Reasoning on Graphs, 2024. Preprint, arXiv. doi:10.48550/arXiv.2404.07103

    Pith/arXiv arXiv doi:10.48550/arxiv.2404.07103 2024

  21. [21]

    Flows: Building Blocks of Reasoning and Collaborating AI, 2023

    Martin Josifoski, Lars Klein, Maxime Peyrard, Nicolas Baldwin, Yifei Li, Saibo Geng, Julian Paul Schnitzler, Yuxing Yao, Jiheng Wei, Debjit Paul, and Robert West. Flows: Building Blocks of Reasoning and Collaborating AI, 2023. Preprint, arXiv. doi:10.48550/arXiv.2308.01285

    Pith/arXiv arXiv doi:10.48550/arxiv.2308.01285 2023

  22. [22]

    Demonstrate-Search-Predict: Composing retrieval and language models for knowledge-intensive NLP, 2022

    Omar Khattab, Keshav Santhanam, Xiang Lisa Li, David Hall, Percy Liang, Christopher Potts, and Matei Zaharia. Demonstrate-Search-Predict: Composing retrieval and language models for knowledge-intensive NLP, 2022. Preprint, arXiv. doi:10.48550/arXiv.2212.14024

    Pith/arXiv arXiv doi:10.48550/arxiv.2212.14024 2022

  23. [23]

    Joshi, Hanna Moazam, Heather Miller, Matei Zaharia, and Christopher Potts

    Omar Khattab, Arnav Singhvi, Paridhi Maheshwari, Zhiyuan Zhang, Keshav Santhanam, Sri Vardhamanan, Saiful Haq, Ashutosh Sharma, Thomas T. Joshi, Hanna Moazam, Heather Miller, Matei Zaharia, and Christopher Potts. DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelines, 2023. Preprint, arXiv. doi:10.48550/arXiv.2310.03714

    Pith/arXiv arXiv doi:10.48550/arxiv.2310.03714 2023

  24. [24]

    Decomposed Prompting: A Modular Approach for Solving Complex Tasks, 2022

    Tushar Khot, Harsh Trivedi, Matthew Finlayson, Yao Fu, Kyle Richardson, Peter Clark, and Ashish Sabharwal. Decomposed Prompting: A Modular Approach for Solving Complex Tasks, 2022. Preprint, arXiv. doi:10.48550/arXiv.2210.02406. 16

    Pith/arXiv arXiv doi:10.48550/arxiv.2210.02406 2022

  25. [25]

    Mahoney, Kurt Keutzer, and Amir Gholami

    Sehoon Kim, Suhong Moon, Ryan Tabrizi, Nicholas Lee, Michael W. Mahoney, Kurt Keutzer, and Amir Gholami. An LLM Compiler for Parallel Function Calling, 2023. Preprint, arXiv. doi:10.48550/arXiv.2312.04511

    Pith/arXiv arXiv doi:10.48550/arxiv.2312.04511 2023

  26. [26]

    Large Language Models are Zero-Shot Reasoners, 2022

    Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwa- sawa. Large Language Models are Zero-Shot Reasoners, 2022. Preprint, arXiv. doi:10.48550/arXiv.2205.11916

    Pith/arXiv arXiv doi:10.48550/arxiv.2205.11916 2022

  27. [27]

    Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks, 2020

    Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks, 2020. Preprint, arXiv. doi:10.48550/arXiv.2005.11401

    Pith/arXiv arXiv doi:10.48550/arxiv.2005.11401 2020

  28. [28]

    CAMEL: Communicative Agents for "Mind" Exploration of Large Language Model Society, 2023

    Guohao Li, Hasan Abed Al Kader Hammoud, Hani Itani, Dmitrii Khizbullin, and Bernard Ghanem. CAMEL: Communicative Agents for "Mind" Exploration of Large Language Model Society, 2023. Preprint, arXiv. doi:10.48550/arXiv.2303.17760

    Pith/arXiv arXiv doi:10.48550/arxiv.2303.17760 2023

  29. [29]

    Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang

    Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. Lost in the Middle: How Language Models Use Long Con- texts.Transactions of the Association for Computational Linguistics, 12:157–173, 2024. doi:10.1162/tacl_a_00638

    doi:10.1162/tacl_a_00638 2024

  30. [30]

    Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing.ACM Computing Surveys, 55:1–35, 2023

    Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing.ACM Computing Surveys, 55:1–35, 2023. doi:10.1145/3560815

    doi:10.1145/3560815 2023

  31. [31]

    Agent Design Pattern Catalogue: A Collection of Architectural Patterns for Foundation Model based Agents, 2024

    Yue Liu, Sin Kit Lo, Qinghua Lu, Liming Zhu, Dehai Zhao, Xiwei Xu, Stefan Harrer, and Jon Whittle. Agent Design Pattern Catalogue: A Collection of Architectural Patterns for Foundation Model based Agents, 2024. Preprint, arXiv. doi:10.48550/arXiv.2405.10467

    Pith/arXiv arXiv doi:10.48550/arxiv.2405.10467 2024

  32. [32]

    Stop Hand-Holding Your Coding Agent: Engineering the Loops that Replace Step-by-Step Prompting, 2026

    Sandeco Macedo. Stop Hand-Holding Your Coding Agent: Engineering the Loops that Replace Step-by-Step Prompting, 2026. Preprint, arXiv. doi:10.48550/arXiv.2607.00038

    Pith/arXiv arXiv doi:10.48550/arxiv.2607.00038 2026

  33. [33]

    Self-Refine: Iterative Refinement with Self-Feedback, 2023

    Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegr- effe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, Shashank Gupta, Bod- hisattwa Prasad Majumder, Katherine Hermann, Sean Welleck, Amir Yazdanbakhsh, and Peter Clark. Self-Refine: Iterative Refinement with Self-Feedback, 2023. Preprint, arXiv. doi:10.48550/arXiv.2...

    Pith/arXiv arXiv doi:10.48550/arxiv.2303.17651 2023

  34. [34]

    The Landscape of Emerging AI Agent Architectures for Reasoning, Planning, and Tool Calling: A Survey, 2024

    Tula Masterman, Sandi Besen, Mason Sawtell, and Alex Chao. The Landscape of Emerging AI Agent Architectures for Reasoning, Planning, and Tool Calling: A Survey, 2024. Preprint, arXiv. doi:10.48550/arXiv.2404.11584

    Pith/arXiv arXiv doi:10.48550/arxiv.2404.11584 2024

  35. [35]

    Skeleton- of-Thought: Prompting LLMs for Efficient Parallel Generation, 2023

    Xuefei Ning, Zinan Lin, Zixuan Zhou, Zifu Wang, Huazhong Yang, and Yu Wang. Skeleton- of-Thought: Prompting LLMs for Efficient Parallel Generation, 2023. Preprint, arXiv. doi:10.48550/arXiv.2307.15337

    Pith/arXiv arXiv doi:10.48550/arxiv.2307.15337 2023

  36. [36]

    Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. Training language models to follow instructions with human feedback,...

    Pith/arXiv arXiv doi:10.48550/arxiv.2203.02155 2022

  37. [37]

    ChatDev: Communicative Agents for Software Development, 2023

    Chen Qian, Wei Liu, Hongzhang Liu, Nuo Chen, Yufan Dang, Jiahao Li, Cheng Yang, Weize Chen, Yusheng Su, Xin Cong, Juyuan Xu, Dahai Li, Zhiyuan Liu, and Maosong Sun. ChatDev: Communicative Agents for Software Development, 2023. Preprint, arXiv. doi:10.48550/arXiv.2307.07924. 17

    Pith/arXiv arXiv doi:10.48550/arxiv.2307.07924 2023

  38. [38]

    Code Generation with Alpha- Codium: From Prompt Engineering to Flow Engineering, 2024

    Tal Ridnik, Dedy Kredo, and Itamar Friedman. Code Generation with Alpha- Codium: From Prompt Engineering to Flow Engineering, 2024. Preprint, arXiv. doi:10.48550/arXiv.2401.08500

    Pith/arXiv arXiv doi:10.48550/arxiv.2401.08500 2024

  39. [39]

    A Systematic Survey of Prompt Engineering in Large Language Models: Techniques and Applications, 2024

    Pranab Sahoo, Ayush Kumar Singh, Sriparna Saha, Vinija Jain, Samrat Mondal, and Aman Chadha. A Systematic Survey of Prompt Engineering in Large Language Models: Techniques and Applications, 2024. Preprint, arXiv. doi:10.48550/arXiv.2402.07927

    Pith/arXiv arXiv doi:10.48550/arxiv.2402.07927 2024

  40. [40]

    Toolformer: Language Models Can Teach Themselves to Use Tools, 2023

    Timo Schick, Jane Dwivedi-Yu, Roberto Dessì, Roberta Raileanu, Maria Lomeli, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. Toolformer: Language Models Can Teach Themselves to Use Tools, 2023. Preprint, arXiv. doi:10.48550/arXiv.2302.04761

    Pith/arXiv arXiv doi:10.48550/arxiv.2302.04761 2023

  41. [41]

    Rogers, Inna Goncearenco, Giuseppe Sarli, Igor Galynker, Denis Peskoff, Marine Carpuat, Jules White, Shyamal Anadkat, Alexander Hoyle, and Philip Resnik

    Sander Schulhoff, Michael Ilie, Nishant Balepur, Konstantine Kahadze, Amanda Liu, Chen- glei Si, Yinheng Li, Aayush Gupta, HyoJung Han, Sevien Schulhoff, Pranav Sandeep Dulepet, Saurav Vidyadhara, Dayeon Ki, Sweta Agrawal, Chau Pham, Gerson Kroiz, Feileen Li, Hudson Tao, Ashay Srivastava, Hevander Da Costa, Saloni Gupta, Megan L. Rogers, Inna Goncearenco,...

    Pith/arXiv arXiv doi:10.48550/arxiv.2406.06608 2024

  42. [42]

    Reflexion: Language Agents with Verbal Reinforcement Learning, 2023

    Noah Shinn, Federico Cassano, Edward Berman, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Language Agents with Verbal Reinforcement Learning, 2023. Preprint, arXiv. doi:10.48550/arXiv.2303.11366

    Pith/arXiv arXiv doi:10.48550/arxiv.2303.11366 2023

  43. [43]

    PDL: A Declarative Prompt Programming Language, 2024

    Mandana Vaziri, Louis Mandel, Claudio Spiess, and Martin Hirzel. PDL: A Declarative Prompt Programming Language, 2024. Preprint, arXiv. doi:10.48550/arXiv.2410.19135

    Pith/arXiv arXiv doi:10.48550/arxiv.2410.19135 2024

  44. [44]

    A Survey on Large Language Model based Autonomous Agents, 2023

    Lei Wang, Chen Ma, Xueyang Feng, Zeyu Zhang, Hao Yang, Jingsen Zhang, Zhiyuan Chen, Jiakai Tang, Xu Chen, Yankai Lin, Wayne Xin Zhao, Zhewei Wei, and Ji-Rong Wen. A Survey on Large Language Model based Autonomous Agents, 2023. Preprint, arXiv. doi:10.48550/arXiv.2308.11432

    Pith/arXiv arXiv doi:10.48550/arxiv.2308.11432 2023

  45. [45]

    Self-Consistency Improves Chain of Thought Reasoning in Language Models, 2022

    Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-Consistency Improves Chain of Thought Reasoning in Language Models, 2022. Preprint, arXiv. doi:10.48550/arXiv.2203.11171

    Pith/arXiv arXiv doi:10.48550/arxiv.2203.11171 2022

  46. [46]

    Chain-of-Thought Prompting Elicits Reasoning in Large Language Models, 2022

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. Chain-of-Thought Prompting Elicits Reasoning in Large Language Models, 2022. Preprint, arXiv. doi:10.48550/arXiv.2201.11903

    Pith/arXiv arXiv doi:10.48550/arxiv.2201.11903 2022

  47. [47]

    APromptPatternCatalogtoEnhance Prompt Engineering with ChatGPT, 2023

    JulesWhite, QuchenFu, SamHays, MichaelSandborn, CarlosOlea, HenryGilbert, AshrafEl- nashar, JesseSpencer-Smith, andDouglasC.Schmidt. APromptPatternCatalogtoEnhance Prompt Engineering with ChatGPT, 2023. Preprint, arXiv. doi:10.48550/arXiv.2302.11382

    Pith/arXiv arXiv doi:10.48550/arxiv.2302.11382 2023

  48. [48]

    AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversation, 2023

    Qingyun Wu, Gagan Bansal, Jieyu Zhang, Yiran Wu, Beibin Li, Erkang Zhu, Li Jiang, Xiaoyun Zhang, Shaokun Zhang, Jiale Liu, Ahmed Hassan Awadallah, Ryen W White, Doug Burger, and Chi Wang. AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversation, 2023. Preprint, arXiv. doi:10.48550/arXiv.2308.08155

    Pith/arXiv arXiv doi:10.48550/arxiv.2308.08155 2023

  49. [49]

    PromptChainer: Chaining Large Language Model Prompts through Visual Programming

    Tongshuang Wu, Ellen Jiang, Aaron Donsbach, Jeff Gray, Alejandra Molina, Michael Terry, and Carrie J Cai. PromptChainer: Chaining Large Language Model Prompts through Visual Programming. InCHI Conference on Human Factors in Computing Systems Extended Abstracts, pages 1–10, 2022. doi:10.1145/3491101.3519729. 18

    arXiv 2022

  50. [50]

    AI Chains: Transparent and Control- lable Human-AI Interaction by Chaining Large Language Model Prompts

    Tongshuang Wu, Michael Terry, and Carrie Jun Cai. AI Chains: Transparent and Control- lable Human-AI Interaction by Chaining Large Language Model Prompts. InCHI Conference on Human Factors in Computing Systems, pages 1–22, 2022. doi:10.1145/3491102.3517582

    arXiv 2022

  51. [51]

    StateFlow: Enhancing LLM Task-Solving through State-Driven Workflows, 2024

    Yiran Wu, Tianwei Yue, Shaokun Zhang, Chi Wang, and Qingyun Wu. StateFlow: Enhancing LLM Task-Solving through State-Driven Workflows, 2024. Preprint, arXiv. doi:10.48550/arXiv.2403.11322

    Pith/arXiv arXiv doi:10.48550/arxiv.2403.11322 2024

  52. [52]

    The Rise and Potential of Large Language Model Based Agents: A Survey, 2023

    Zhiheng Xi, Wenxiang Chen, Xin Guo, Wei He, Yiwen Ding, Boyang Hong, Ming Zhang, Junzhe Wang, Senjie Jin, Enyu Zhou, Rui Zheng, Xiaoran Fan, Xiao Wang, Limao Xiong, Yuhao Zhou, Weiran Wang, Changhao Jiang, Yicheng Zou, Xiangyang Liu, Zhangyue Yin, Shihan Dou, Rongxiang Weng, Wensen Cheng, Qi Zhang, Wenjuan Qin, Yongyan Zheng, Xipeng Qiu, Xuanjing Huang, a...

    Pith/arXiv arXiv doi:10.48550/arxiv.2309.07864 2023

  53. [53]

    Beyond Chain-of-Thought: A Survey of Chain-of-X Paradigms for LLMs

    Yu Xia, Rui Wang, Xu Liu, Mingyan Li, Tong Yu, Xiang Chen, Julian McAuley, and Shuai Li. Beyond Chain-of-Thought: A Survey of Chain-of-X Paradigms for LLMs, 2024. Preprint, arXiv. doi:10.48550/arXiv.2404.15676

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2404.15676 2024

  54. [54]

    Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press

    John Yang, Carlos E. Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press. SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering, 2024. Preprint, arXiv. doi:10.48550/arXiv.2405.15793

    Pith/arXiv arXiv doi:10.48550/arxiv.2405.15793 2024

  55. [55]

    Griffiths, Yuan Cao, and Karthik Narasimhan

    Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L. Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of Thoughts: Deliberate Problem Solving with Large Language Models, 2023. Preprint, arXiv. doi:10.48550/arXiv.2305.10601

    Pith/arXiv arXiv doi:10.48550/arxiv.2305.10601 2023

  56. [56]

    ReAct: Synergizing Reasoning and Acting in Language Models, 2022

    Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. ReAct: Synergizing Reasoning and Acting in Language Models, 2022. Preprint, arXiv. doi:10.48550/arXiv.2210.03629

    Pith/arXiv arXiv doi:10.48550/arxiv.2210.03629 2022

  57. [57]

    Beyond Chain-of-Thought, Effective Graph-of-Thought Reasoning in Language Models, 2023

    Yao Yao, Zuchao Li, and Hai Zhao. Beyond Chain-of-Thought, Effective Graph-of-Thought Reasoning in Language Models, 2023. Preprint, arXiv. doi:10.48550/arXiv.2305.16582

    Pith/arXiv arXiv doi:10.48550/arxiv.2305.16582 2023

  58. [58]

    Differentiation

    Mert Yuksekgonul, Federico Bianchi, Joseph Boen, Sheng Liu, Zhi Huang, Carlos Guestrin, and James Zou. TextGrad: Automatic "Differentiation" via Text, 2024. Preprint, arXiv. doi:10.48550/arXiv.2406.07496

    Pith/arXiv arXiv doi:10.48550/arxiv.2406.07496 2024

  59. [59]

    AFlow: Automating Agentic Workflow Generation, 2024

    Jiayi Zhang, Jinyu Xiang, Zhaoyang Yu, Fengwei Teng, Xionghui Chen, Jiaqi Chen, Mingchen Zhuge, Xin Cheng, Sirui Hong, Jinlin Wang, Bingnan Zheng, Bang Liu, Yuyu Luo, and Chenglin Wu. AFlow: Automating Agentic Workflow Generation, 2024. Preprint, arXiv. doi:10.48550/arXiv.2410.10762

    Pith/arXiv arXiv doi:10.48550/arxiv.2410.10762 2024

  60. [60]

    A Survey of Large Language Models, 2023

    Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, Yifan Du, Chen Yang, Yushuo Chen, Zhipeng Chen, Jinhao Jiang, Ruiyang Ren, Yifan Li, Xinyu Tang, Zikang Liu, Peiyu Liu, Jian-Yun Nie, and Ji-Rong Wen. A Survey of Large Language Models, 2023. Preprint, arXiv. doi:10.48550/arXiv...

    Pith/arXiv arXiv doi:10.48550/arxiv.2303.18223 2023

  61. [61]

    Gonzalez, Clark Barrett, and Ying Sheng

    Lianmin Zheng, Liangsheng Yin, Zhiqiang Xie, Chuyue Sun, Jeff Huang, Cody Hao Yu, Shiyi Cao, Christos Kozyrakis, Ion Stoica, Joseph E. Gonzalez, Clark Barrett, and Ying Sheng. SGLang: Efficient Execution of Structured Language Model Programs, 2023. Preprint, arXiv. doi:10.48550/arXiv.2312.07104

    Pith/arXiv arXiv doi:10.48550/arxiv.2312.07104 2023

  62. [62]

    Least-to-Most Prompt- 19 ing Enables Complex Reasoning in Large Language Models, 2022

    Denny Zhou, Nathanael Schärli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang, Dale Schuurmans, Claire Cui, Olivier Bousquet, Quoc Le, and Ed Chi. Least-to-Most Prompt- 19 ing Enables Complex Reasoning in Large Language Models, 2022. Preprint, arXiv. doi:10.48550/arXiv.2205.10625

    Pith/arXiv arXiv doi:10.48550/arxiv.2205.10625 2022

  63. [63]

    Large Language Models Are Human-Level Prompt Engineers, 2022

    Yongchao Zhou, Andrei Ioan Muresanu, Ziwen Han, Keiran Paster, Silviu Pitis, Harris Chan, and Jimmy Ba. Large Language Models Are Human-Level Prompt Engineers, 2022. Preprint, arXiv. doi:10.48550/arXiv.2211.01910

    Pith/arXiv arXiv doi:10.48550/arxiv.2211.01910 2022

  64. [64]

    Language Agents as Optimizable Graphs, 2024

    Mingchen Zhuge, Wenyi Wang, Louis Kirsch, Francesco Faccio, Dmitrii Khizbullin, and Jürgen Schmidhuber. Language Agents as Optimizable Graphs, 2024. Preprint, arXiv. doi:10.48550/arXiv.2402.16823. 20

    Pith/arXiv arXiv doi:10.48550/arxiv.2402.16823 2024

  65. [2022]

    doi:10.48550/arXiv.2211.12588

    Preprint, arXiv. doi:10.48550/arXiv.2211.12588

    Pith/arXiv arXiv doi:10.48550/arxiv.2211.12588

  66. [2023]

    doi:10.48550/arXiv.2309.16797

    Preprint, arXiv. doi:10.48550/arXiv.2309.16797

    Pith/arXiv arXiv doi:10.48550/arxiv.2309.16797

  67. [2024]

    doi:10.1609/aaai.v38i16.29720

    doi:10.1609/aaai.v38i16.29720