Pith. sign in

REVIEW 4 major objections 5 minor 4 cited by

Cognify: Supercharging Gen-AI Workflows With Hierarchical Autotuning

T0 review · 4 major / 5 minor · reviewed 2026-08-08 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that gen-AI workflows—multi-step programs that call models, tools, and code—can be autotuned by an adaptive hierarchical search, AdaSeek, which allocates a limited budget across architecture, step, and prompt changes…

desk verdict Real system, honest about most limitations, but the headline DSPy/Trace comparison is a system-level comparison with an extra cog, not a clean test of the search algorithm. read the letter →

arxiv 2502.08056 v1 pith:RLAPPPT2 submitted 2025-02-12 cs.LG cs.AIcs.MA

Zijian He , Reyna Abhyankar , Vikranth Srivatsa , Yiying Zhang This is my paper · ORCID
classification cs.LGcs.AIcs.MA
keywords gen-AIworkflowsworkflowautotuninghierarchicalsearchBayesianoptimizationTPEsuccessivehalvingmulti-objectiveprompt
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that tuning a multi-step generative-AI workflow is a systematic search problem rather than a manual art. It models a workflow as a graph of steps and edges, classifies every possible improvement as an architecture change, a step change, or a prompt-level weight change, and searches this space with a new algorithm, AdaSeek. AdaSeek adaptively chooses one, two, or three search layers based on the user's total budget, redirects budget from poor to promising configurations with successive halving, and stops early when results converge. The authors report that this approach beats the original expert-written workflows and two earlier optimizers on six workflows, with up to 2.8x higher quality, up to 10x lower cost, and up to 2.7x lower latency. If these results hold, developers can substitute a bounded automated search for hours of ad-hoc prompt and model hand-tuning.

What carries the argument

The load-bearing mechanism is the recursive LayerSearch procedure inside AdaSeek, operating on a cog hierarchy. A cog is one tunable dimension of the workflow (architecture, step, or weight), and the hierarchy decides how many levels of cogs are searched together. For a one-layer configuration, AdaSeek merges all cogs into a flat search space so the optimizer can converge locally; for two or three layers, architecture cogs occupy the outermost layer and weight cogs the innermost, so each sampled architecture is explored through step and prompt choices before evaluation. Budgets are assigned to layers from an estimated search size $S_i = N_i^\alpha$, and each non-innermost layer uses chunked TPE sampling with successive halving—keeping only the top $\lfloor |\Theta|/\eta \rfloor$ configurations per round and multiplying the per-config budget by $\eta$—plus early stopping when recent evaluation results converge. This combination is what lets a small optimization budget cover a high-dimensional cog space without getting stuck in a local optimum.

What would settle it

Run AdaSeek with an evaluator that returns deliberately noisy scores, such as an LLM judge sampled at nonzero temperature with no repeated averaging, on any supported workflow with a 64-iteration budget, and compare the final selected configurations against random search. If the selected configurations are no better than random search, or if the reported quality, cost, and latency gains disappear, then the stability assumption behind successive-halving pruning fails. A cheaper check: for a tuned workflow, re-score every configuration that was pruned in the first successive-halving round on a held-out set; if any pruned configuration would have beaten the final selection, the pruning signal was unreliable.

Watch

Extended reading notes

Core claim

The central claim, stated on the paper's own terms, is that gen-AI workflow autotuning should be treated as discrete search over three kinds of cogs and that an adaptive hierarchy is the right way to run that search on a small budget. Cognify's search space is built from a workflow graph W=(S,E) with edge weights w_ij, and each tuning method is a cog: architecture cogs add, remove, or move steps and edges; step cogs swap the operation at a node, such as the language model or the code implementation; weight cogs modify edge messages, such as appending chain-of-thought reasoning or few-shot examples to a prompt. AdaSeek starts with a flat one-layer search when the budget is small and, as the budget grows, splits cogs into two or three layers, assigning each layer a budget proportional to its estimated search-size complexity. Within each layer, the search samples configurations in chunks with the TPE Bayesian sampler, evaluates each candidate through recursive calls down to the innermost layer, prunes the worst fraction of a chunk with successive halving, and stops a chunk or layer early when recent evaluations stop improving. The paper claims that this design, implemented in Cognify with six cogs, pushes the Pareto frontier on quality, cost, and latency beyond both the original workflows and non-hierarchical optimizers, and its ablations attribute the gains to the adaptive hierarchy, chunked successive halving, and early exit.

Load-bearing premise

The reported gains assume the user-supplied evaluator and training set produce stable, low-noise scores, because AdaSeek prunes configurations based on single evaluation results; if scores are noisy or the training set has only a handful of examples, the pruning can discard good configurations and the headline improvements will not reproduce.

Editorial extensions

If this is right

  • Workflow developers can hand the optimizer a budget, a small training set, and an evaluator, and receive multiple tuned versions that trade off quality, cost, and latency.
  • Small budgets around 16 iterations favor a flat one-layer search, while budgets of 64 to 128 iterations benefit from two- or three-layer hierarchical search, so the same algorithm adapts to users who cannot specify a sufficient budget in advance.
  • Combining prompt-level cogs with model-selection and code-rewriting cogs yields improvements that prompt-only optimizers miss, such as picking a faster small model for some steps while adding reasoning to earlier steps.
  • The multi-objective treatment means optimization can target cost and latency with a quality constraint rather than maximizing quality alone, which is how text-to-SQL became 10x cheaper and FinRobot 2.5x faster.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The same budget-to-hierarchy principle should transfer to other expensive black-box searches whose dimensions come in natural layers, such as database query plan tuning or compiler flag tuning, where small budgets should search flat and larger budgets should search hierarchically.
  • The 10x cost savings partly depend on the current API price gap between small and large models; if that gap narrows, cost gains would shrink unless the reasoning and few-shot cogs also reduce expensive retries.
  • The paper's sensitivity result, where quality degrades only when training has about three examples, suggests a testable floor: varying training size between one and ten examples on a new workflow should reveal the minimum dataset size at which AdaSeek's pruning decisions become trustworthy.
  • Because the evaluator is assumed stable, a natural extension is noise-aware pruning, evaluating each candidate more than once when scores are noisy, which the paper does not model but which its successive-halving machinery could incorporate.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper formalizes the problem of autotuning generative-AI workflows, categorizes tuning actions into architecture, step, and weight cogs, and proposes AdaSeek, an adaptive hierarchical search algorithm that assigns search budgets across layers and prunes unpromising configurations via chunked successive halving and early stopping. The authors implement AdaSeek in the Cognify framework and evaluate it on six workflow types, reporting up to 2.8x quality improvement, 10x cost reduction, and 2.7x latency reduction over original workflows, and claiming superior quality/cost/latency relative to DSPy and Trace. The paper also contributes an ablation study, a grid-search comparison, a training-set-size sensitivity analysis, and an analysis of layering under different budgets.

Significance. If the central empirical claims held, the paper would be a useful contribution: it addresses a practical problem, the formalization of workflow cogs is clear, and the framework is open-sourced. The paper also ships an ablation study that isolates the adaptive-hierarchy, chunked successive-halving, and early-exit components, and it shows a favorable comparison against exhaustive grid search on HotpotQA. However, the headline comparison against DSPy and Trace is currently confounded by an unequal allocation of capabilities, and the absence of variance reporting makes the quantitative claims difficult to assess. The core AdaSeek idea and the layered-budget analysis are plausible and worth pursuing, but the evidence as presented does not yet support the stated superiority over prior optimizers.

major comments (4)
  1. [Section 5, Figure 2 and surrounding text] The comparison against DSPy and Trace is not controlled: Cognify is given a per-step model-selection cog (Section 4.2), while the text states that DSPy and Trace 'do not support model selection and are given GPT-4o-mini for all steps.' The paper itself attributes FinRobot's 2.5x latency reduction to the model-selection cog choosing Llama-8B for certain steps. Consequently, the reported quality/cost/latency advantages over DSPy and Trace cannot be attributed to AdaSeek; they may largely reflect the extra freedom to choose a cheaper/faster model. A matched comparison, e.g., giving DSPy and Trace a fixed per-step model choice with the same two-model pool, or ablating Cognify's model-selection cog, is needed before the headline 'outperforms DSPy and Trace' claim is supported.
  2. [Section 5, all reported results] The evaluation reports single-point numbers for quality, cost, and latency on stochastic LLM-based workflows, with no standard deviations, confidence intervals, or repeated-seed runs. Since the evaluators include LLM-as-a-Judge and pass rates on small test sets, the differences claimed (e.g., 91% vs 87% on code generation in Table 1) may be within run-to-run noise. The paper should report at least 3-5 repeated optimization runs with variance, or otherwise justify why the reported differences are stable.
  3. [Section 3, Algorithms 1 and 2] Several algorithm constants are fixed without sensitivity analysis: alpha (budget estimation exponent), eta (successive halving divisor), W (chunk size), the early-stop threshold, the K% important-step fraction, and the top-k few-shot examples. Since the adaptive hierarchy's behavior depends directly on these choices, the paper should either derive them from a principle, provide a sensitivity study, or show that the main conclusions are robust across reasonable ranges of these constants.
  4. [Section 5.2.2, Figure 6] The ablation study only reports quality on Text-to-SQL and does not report cost or latency for the ablated variants. Since the paper's contribution includes multi-objective optimization and cost/latency reductions, the ablation should at least confirm that the adaptive-hierarchy gains do not come at a hidden cost/latency penalty, or the scope of the claim should be narrowed.
minor comments (5)
  1. [Section 2, formalization] The notation for workflow weights is inconsistent: the text defines w_ij as edge weights but later uses p* in Figure 1 and w25/w45 in the caption without clearly connecting these to the formal definition. Please align the notation.
  2. [Appendix A.1.1] The TPE description cites 'TPE [? ]' with a missing reference key; it should cite the original TPE paper or the referenced survey [39] consistently.
  3. [Section 5.1, Trace results] The sentence 'While code rewriting may be effective to generate Trace is especially ineffective for the Text-2-SQL and FinRobot workloads' is grammatically incomplete and should be rewritten.
  4. [Section 5.1, Table 1 caption] Table 1 lists accuracy numbers but the caption does not state the test-set sizes or whether these are exact-match or pass@k metrics; please clarify.
  5. [Figure 2] The figure does not show error bars or markers for repeated runs, and the Trace points for Text-2-SQL and FinRobot are omitted due to zero quality; please indicate this directly in the legend or caption.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the central claims are empirical comparisons against external baselines, and AdaSeek's search procedures are not fitted to the reported outcomes.

full rationale

Walking the derivation chain, the paper's central claims are empirical: Cognify is evaluated on six external benchmark workflows (HotpotQA, BIRD, MatPlotAgent, FinRobot, HumanEval, BigBench) with defined quality metrics (F1, pass rate, exact match, LLM-as-a-judge), and compared against original workflows, DSPy, and Trace. AdaSeek is a search algorithm (TPE sampling, successive halving, hierarchical layering, early stopping) that optimizes a user-supplied evaluator on a training split; Appendix A.3 shows a train/validation/test split, so reporting held-out quality is standard evaluation, not a prediction that reduces to the optimization objective. The constants alpha=1.1, eta, W, chunk sizes, and the K% importance thresholds are hand-chosen defaults stated in the algorithm, not fitted to the specific results claimed as improvements, so there is no fitted-input-called-prediction step. No load-bearing self-citation or imported uniqueness theorem appears; TPE is cited to Bergstra et al., and DSPy/Trace are external baselines. The reviewer-flagged concern that Cognify alone is allowed a model-selection cog in the DSPy/Trace comparison is a comparison-fairness threat, not a circularity of the claimed derivation, and does not affect the circularity score.

Assumptions & free parameters 6 free parameters · 4 assumptions · 0 invented entities

No new physical entities are introduced. The central claim rests on several hand-set search hyperparameters that are not fully reported, plus standard assumptions about evaluator fidelity and representativeness of benchmarks.

free parameters (6)
  • alpha (budget estimation exponent) = 1.1 (default)
    Controls the estimated search size S_i = N_i^alpha in Algorithm 1. Hand chosen, with no sensitivity analysis reported.
  • eta (successive halving divisor) = not specified
    Controls chunk reduction and budget multiplication in Algorithm 2. Never given a default value, despite being load-bearing for budget allocation.
  • W (chunk size) = not specified
    Number of TPE samples per chunk in the non-innermost layers of Algorithm 2. No default or sensitivity analysis is reported.
  • early-stop threshold = not specified
    Stops a layer or chunk when recent evaluations improve by less than a threshold. The threshold is never defined in the paper.
  • K% important-step fraction for model selection = not specified
    Selects which LLM steps TPE can tune in Section 4.2. The fraction is hand chosen and not varied in the evaluation.
  • top-k few-shot examples = not specified
    Number of best-performing training examples appended as few-shot prompts in Section 4.3. No value or sensitivity analysis is given.
assumptions (4)
  • domain assumption User-provided evaluator and training set are faithful, low-noise proxies for deployment quality.
    Section 2 defines E(M) as user-defined; Algorithm 2 prunes and early-stops based on evaluation scores. Noisy evaluators would undermine the search.
  • domain assumption TPE can model the discrete, non-differentiable workflow cog search space with tens to hundreds of evaluations.
    Section 3 assumes Bayesian optimization with TPE is efficient for workflow cogs. No theoretical guarantee or high-dimensional noisy-space comparison is provided.
  • domain assumption LLM-as-a-judge complexity ratings and dynamic few-shot selection give stable signals for pruning.
    Section 4.1 uses LLM ratings to pick decomposition targets, and Section 4.3 evolves few-shot sets from evaluation feedback. The stability of these signals is assumed.
  • domain assumption The six benchmark workflows and their metrics are representative of production gen-AI workflows.
    Section 5 evaluates only six tasks, yet the paper claims a generic, extensible autotuning platform. Generality is an extrapolation beyond the evidence.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Cognify: Supercharging Gen-AI Workflows With Hierarchical Autotuning." pith.science (2026). https://pith.science/paper/RLAPPPT2

@misc{pith2026250208056,
  author       = {Pith},
  title        = {Pith review of: Cognify: Supercharging Gen-AI Workflows With Hierarchical Autotuning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/RLAPPPT2}},
  note         = {Machine review of arXiv:2502.08056}
}
read the original abstract

Today's gen-AI workflows that involve multiple ML model calls, tool/API calls, data retrieval, or generic code execution are often tuned manually in an ad-hoc way that is both time-consuming and error-prone. In this paper, we propose a systematic approach for automatically tuning gen-AI workflows. Our key insight is that gen-AI workflows can benefit from structure, operator, and prompt changes, but unique properties of gen-AI workflows require new optimization techniques. We propose AdaSeek, an adaptive hierarchical search algorithm for autotuning gen-AI workflows. AdaSeek organizes workflow tuning methods into different layers based on the user-specified total search budget and distributes the budget across different layers based on the complexity of each layer. During its hierarchical search, AdaSeek redistributes the search budget from less useful to more promising tuning configurations based on workflow-level evaluation results. We implement AdaSeek in a workflow autotuning framework called Cognify and evaluate Cognify using six types of workflows such as RAG-based QA and text-to-SQL transformation. Overall, Cognify improves these workflows' generation quality by up to 2.8x, reduces execution monetary cost by up to 10x, and reduces end-to-end latency by 2.7x.

Figures

Figures reproduced from arXiv: 2502.08056 by the authors.

Figure 1
Figure 1. Gen-AI Workflows Tuning Methods. SLM and LLM represent different language models (e.g., small and large). Code, Tool, and Data represent code blocks, tool calls, and data retrieval. Dash curved lines represent loops or control flow changes. p* represents prompt optimizations or additional information added for the downstream step. To confront these challenges, we propose an adaptive, hierar￾chical search-based algor… view at source ↗
Figure 2
Figure 2. Generation Quality vs Cost/Latency. Dashed lines show the Pareto frontier (upper left is better). Cost shown as model API dollar cost for every 1000 requests. Cognify selects models from GPT-4o-mini and Llama-8B. DSPy and Trace do not support model selection and are given GPT-4o-mini for all steps. Trace results for Text-2-SQL and FinRobot have 0 quality and are not included. Original DSPy Trace Cognify Code Gen 70%… view at source ↗
Figure 3
Figure 3. Effectiveness of Layering over Budget. The quality (higher is better), cost (lower is better), and latency (lower is better) achieved on Text-to-SQL by Cognify when using different number of search layers under different budget. 0.2 0.3 0.4 0.5 0.6 0.7 0.8 Cost ($ per 1000 executions) 0.2 0.3 0.4 0.5 0.6 Quality (F1 Score) Grid Search Cognify 0 20 40 60 80 100 Cognify Optimization Iteration [PITH_FULL_IMAGE:figures… view at source ↗
Figures from the paper (3 more)
Figure 5
Figure 5. Figure 5: Sensitivity to Training Set Size. Conducted on FinRobot. X-axis represents number of training data points, Y-axis represents quality. As indicated by the heatmap, Cognify quickly moves towards the highest quality and lowest cost in approximately 20 iterations. With jus…
Figure 7
Figure 7. Figure 7: Search Time (left) and Cost (right). Comparison between DSPy and Cognify (Trace omitted due to poor local convergence) OpenAI Swarm [4], CrewAI [10], and Dify [12], Vellum [38], and Coze [9]. These programming frameworks allow programmers to more easily develop and tes…
Figure 8
Figure 8. Figure 8: Data Visualization Workflow Input is natural language description of a plotting task along with a CSV file containing relevant data. Expected output is a figure. Workflow contains 4 agents. An example optimization for this workflow is as follows: • Query expansion: Few…

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 4 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. HedraRAG: Coordinating LLM Generation and Database Retrieval in Heterogeneous RAG Serving

    cs.DB 2025-07 conditional novelty 6.0 of 10

    HedraRAG uses a graph abstraction and dynamic transformations to pipeline generation and retrieval stages, achieving 1.5x to 5x speedups in heterogeneous RAG serving.

  2. KVFlow: Efficient Prefix Caching for Accelerating LLM-Based Multi-Agent Workflows

    cs.DC 2025-07 conditional novelty 6.0 of 10

    KVFlow uses workflow-aware eviction priorities and overlapped KV prefetching to cut cache-miss latency in LLM multi-agent serving.

  3. Single-agent or Multi-agent Systems? Why Not Both?

    cs.MA 2025-05 conditional novelty 6.0 of 10

    On 15 agentic benchmarks, the accuracy advantage of multi-agent LLM systems over single-agent systems mostly disappears with stronger base models, and a hybrid single/multi-agent cascade improves accuracy and cuts cost.

  4. A Survey on Agent Workflow -- Status and Future

    cs.AI 2025-08 conditional novelty 3.0 of 10

    A review that classifies 24 agent workflow systems along functional and architectural axes and argues for standardization, optimization, and security work.

Reference graph

Works this paper leans on

52 extracted references · 31 canonical work pages · cited by 4 Pith papers

  1. [1]

    Fireworks AI. 2024. API Reference - Fireworks AI. https://docs.fireworks.ai/api- reference/introduction. Accessed: 2025-02-10

    work page 2024

  2. [2]

    BIG bench authors. 2023. Beyond the Imitation Game: Quantifying and extrap- olating the capabilities of language models. Transactions on Machine Learning Research (2023). https://openreview.net/forum?id=uyTL5Bvosj

    work page 2023

  3. [3]

    James Bergstra, Rémi Bardenet, Yoshua Bengio, and Balázs Kégl. 2011. Algo- rithms for Hyper-Parameter Optimization. In Proceedings of the 25th International Conference on Neural Information Processing Systems (NeurIPS) . 2546–2554

    work page 2011

  4. [4]

    Ilan Bigio, James Hills, Shyamal Anadkat, Charu Jaiswal, Colin Jarvis, and Katia Gil Guzman. [n. d.]. Swarm by OpenAI. https://github.com/openai/swarm

    work page

  5. [5]

    Lingjiao Chen, Matei Zaharia, and James Zou. 2024. FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performance.Transactions on Machine Learning Research (2024)

    work page 2024

  6. [6]

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, Alex Ray, Raul Puri, Gretchen Krueger, Michael Petrov, Heidy Khlaaf, Girish Sastry, Pamela Mishkin, Brooke Chan, Scott Gray, Nick Ryder, Mikhail Pavlov, Alethea Power, Lukasz Kaiser, Mohammad Bavarian...

    arXiv 2021

  7. [7]

    Ching-An Cheng, Allen Nie, and Adith Swaminathan. 2024. Trace is the Next AutoDiff: Generative Optimization with Rich Feedback, Execution Traces, and LLMs. In Advances in Neural Information Processing Systems . https://neurips.cc/ virtual/2024/poster/93431

    work page 2024

  8. [8]

    Cognition AI. [n. d.]. Devin AI: Autonomous AI Software Engineer. https: //devin.ai/

    work page

Show all 52 references
  1. [9]

    Coze. [n. d.]. Coze: Intelligent Communication Solutions. https://www.coze.com/

  2. [10]

    CrewAI. [n. d.]. CrewAI: The Leading Multi-Agent Platform. https://www.crewai. com/. Website

  3. [11]

    Databricks. [n. d.]. Compound AI Systems: Integrating Multiple AI Models and Tools. https://www.databricks.com/glossary/compound-ai-systems

  4. [12]

    Dify. [n. d.]. Dify: Empowering AI Applications. https://dify.ai/

  5. [13]

    Stefan Falkner, Aaron Klein, and Frank Hutter. 2018. BOHB: Robust and Efficient Hyperparameter Optimization at Scale. In Proceedings of the 35th International Conference on Machine Learning (ICML) . 1437–1446. arXiv:1807.01774

    2018 arXiv

  6. [14]

    Dawei Gao, Haibin Wang, Yaliang Li, Xiuyu Sun, Yichen Qian, Bolin Ding, and Jingren Zhou. 2023. Text-to-SQL Empowered by Large Language Models: A Benchmark Evaluation. Proceedings of the VLDB Endowment (2023). https: //doi.org/10.48550/arXiv.2308.15363

    2023 doi arXiv

  7. [15]

    Google Cloud. [n. d.]. Vertex AI Search: Build Google-quality search applications. https://cloud.google.com/enterprise-search. Accessed: 2025-01-20

    2025

  8. [16]

    Wenlong Huang, Pieter Abbeel, Deepak Pathak, and Igor Mordatch. 2022. Lan- guage Models as Zero-Shot Planners: Extracting Actionable Knowledge for Em- bodied Agents. arXiv preprint arXiv:2201.07207 (2022)

    2022 arXiv

  9. [17]

    Frank Hutter, Lars Kotthoff, and Joaquin Vanschoren. 2019. Automated machine learning: methods, systems, challenges . Springer Nature

    2019

  10. [18]

    Jones, Matthias Schonlau, and William J

    Donald R. Jones, Matthias Schonlau, and William J. Welch. 1998. Efficient Global Optimization of Expensive Black-Box Functions. Journal of Global Optimization 13 (1998), 455–492. https://api.semanticscholar.org/CorpusID:263864014

    1998

  11. [19]

    Kirthevasan Kandasamy, Willie Neiswanger, Jeff Schneider, Barnabas Poczos, and Eric P. Xing. 2018. Neural Architecture Search with Bayesian Optimisation and Optimal Transport. In Advances in Neural Information Processing Systems (NeurIPS). 2016–2025. arXiv:1802.07191

    2018 arXiv

  12. [20]

    Omar Khattab, Keshav Santhanam, Xiang Lisa Li, David Hall, Percy Liang, Christo- pher Potts, and Matei Zaharia. 2022. Demonstrate-Search-Predict: Composing Retrieval and Language Models for Knowledge-Intensive NLP. arXiv preprint arXiv:2212.14024 (2022)

    2022 arXiv

  13. [21]

    Joshi, Hanna Moazam, Heather Miller, Matei Zaharia, and Christopher Potts

    Omar Khattab, Arnav Singhvi, Paridhi Maheshwari, Zhiyuan Zhang, Keshav Santhanam, Sri Vardhamanan, Saiful Haq, Ashutosh Sharma, Thomas T. Joshi, Hanna Moazam, Heather Miller, Matei Zaharia, and Christopher Potts. 2024. DSPy: Compiling Declarative Language Model Calls into Self...

    2024

  14. [22]

    Tushar Khot, Harsh Trivedi, Matthew Finlayson, Yao Fu, Kyle Richardson, Peter Clark, and Ashish Sabharwal. 2023. Decomposed Prompting: A Modular Ap- proach for Solving Complex Tasks. In The Eleventh International Conference on Learning Representations. https://openreview.net/f...

    2023

  15. [23]

    Sandip Kulkarni and Alexandra Savelieva. 2025. The Path to a Golden Dataset, or How to Evaluate Your RAG. https://medium.com/data-science-at-microsoft/the- path-to-a-golden-dataset-or-how-to-evaluate-your-rag-045e23d1f13f Accessed: 2025-02-10

    2025

  16. [24]

    LangChain. [n. d.]. LangChain: Building Applications with LLMs through Com- posability. https://github.com/langchain-ai/langchain

  17. [25]

    Orr, and Klaus-Robert Müller

    Yann LeCun, Léon Bottou, Genevieve B. Orr, and Klaus-Robert Müller. 1998. Efficient BackProp. In Neural Networks: Tricks of the Trade . Springer, 9–50. https://doi.org/10.1007/3-540-49430-8_2

    1998 doi

  18. [26]

    Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. 2020. Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. In Advances in ...

    2020

  19. [27]

    LlamaIndex. [n. d.]. LlamaIndex: Connecting LLMs with your Data. https://www. llamaindex.ai/

  20. [28]

    Meta. 2024. Introducing Llama 3.1: Our most capable models to date. https: //ai.meta.com/blog/meta-llama-3-1/

    2024

  21. [29]

    Andrew Ng. [n. d.]. What’s Next for AI Agentic Workflows ft. Andrew Ng of AI Fund. https://www.youtube.com/watch?v=sal78ACtGTc&t=740s

  22. [30]

    Gonzalez, M Waleed Kadous, and Ion Stoica

    Isaac Ong, Amjad Almahairi, Vincent Wu, Wei-Lin Chiang, Tianhao Wu, Joseph E. Gonzalez, M Waleed Kadous, and Ion Stoica. 2024. RouteLLM: Learning to Route Cognify: Supercharging Gen-AI Workflows With Hierarchical Autotuning He*, Abhyankar*, Srivatsa, Zhang LLMs with Preference...

    2024 arXiv

  23. [31]

    OpenAI. 2024. GPT-4o mini: Advancing Cost-Efficient Intelligence. https:// openai.com/index/gpt-4o-mini-advancing-cost-efficient-intelligence/ Accessed: 2025-02-10

    2024

  24. [32]

    OpenAI. 2024. openai-python. https://github.com/openai/openai-python. Ac- cessed: 2025-02-10

    2024

  25. [33]

    Krista Opsahl-Ong, Michael J Ryan, Josh Purtell, David Broman, Christo- pher Potts, Matei Zaharia, and Omar Khattab. 2024. Optimizing In- structions and Demonstrations for Multi-Stage Language Model Programs. arXiv:2406.11695 [cs.CL] https://arxiv.org/abs/2406.11695

    2024 arXiv

  26. [34]

    Guan, Barret Zoph, Quoc V

    Hieu Pham, Melody Y. Guan, Barret Zoph, Quoc V. Le, and Jeff Dean. 2018. Effi- cient Neural Architecture Search via Parameter Sharing. arXiv:1802.03268 [cs.LG] https://arxiv.org/abs/1802.03268

    2018 arXiv

  27. [35]

    David E Rumelhart, Geoffrey E Hinton, and Ronald J Williams. 1986. Learning representations by back-propagating errors. nature 323, 6088 (1986), 533–536

    1986

  28. [36]

    Salesforce. [n. d.]. Agentforce: Create Powerful AI Agents. https://www.salesforce. com/agentforce/. Accessed: 2025-01-20

    2025

  29. [37]

    Dimitris Stripelis, Zijian Hu, Jipeng Zhang, Zhaozhuo Xu, Alay Dilipbhai Shah, Han Jin, Yuhang Yao, Salman Avestimehr, and Chaoyang He. 2024. Tensor- Opera Router: A Multi-Model Router for Efficient LLM Inference. arXiv preprint arXiv:2408.12320 (2024)

    2024 arXiv

  30. [38]

    Vellum AI. [n. d.]. Vellum: Streamlining Large Language Model Operations. https://www.vellum.ai/

  31. [39]

    Shuhei Watanabe. 2023. Tree-structured Parzen estimator: Understanding its algorithm components and their roles for better empirical performance. ArXiv abs/2304.11127 (2023). https://api.semanticscholar.org/CorpusID:258291728

    2023 arXiv

  32. [40]

    Chi, Quoc V

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoc V. Le, and Denny Zhou. 2024. Chain-of-thought prompting elicits reasoning in large language models. InProceedings of the 36th International Conference on Neural Information Processin...

    2024

  33. [41]

    Williams

    Ronald J. Williams. 1992. Simple Statistical Gradient-Following Algorithms for Connectionist Reinforcement Learning. In Proceedings of the 8th International Conference on Neural Information Processing Systems (NIPS) . MIT Press, 877–884

    1992

  34. [42]

    Chengrun Yang, Xuezhi Wang, Yifeng Lu, Hanxiao Liu, Quoc V Le, Denny Zhou, and Xinyun Chen. 2024. Large Language Models as Optimizers. In The Twelfth International Conference on Learning Representations . https://openreview.net/ forum?id=Bb4VGOWELI

    2024

  35. [43]

    Hongyang Yang, Boyu Zhang, Neng Wang, Cheng Guo, Xiaoli Zhang, Likun Lin, Junlin Wang, Tianyu Zhou, Mao Guan, Runjia Zhang, et al. 2024. FinRobot: An Open-Source AI Agent Platform for Financial Applications using Large Language Models. arXiv preprint arXiv:2405.14767 (2024)

    2024 arXiv

  36. [44]

    Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W Cohen, Ruslan Salakhutdinov, and Christopher D Manning. 2018. HotpotQA: A dataset for di- verse, explainable multi-hop question answering. arXiv preprint arXiv:1809.09600 (2018)

    2018 arXiv

  37. [45]

    Zhiyu Yang, Zihan Zhou, Shuo Wang, Xin Cong, Xu Han, Yukun Yan, Zhenghao Liu, Zhixing Tan, Pengyuan Liu, Dong Yu, Zhiyuan Liu, Xiaodong Shi, and Maosong Sun. 2024. MatPlotAgent: Method and Evaluation for LLM-Based Agentic Scientific Data Visualization. arXiv:2402.11453 [cs.CL]

    2024 arXiv

  38. [46]

    Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. 2023. Tree of Thoughts: Deliberate Problem Solving with Large Language Models. In Thirty-seventh Con- ference on Neural Information Processing Systems , A. Oh, T. Naumann, A. Glo...

    2023

  39. [47]

    Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R Narasimhan, and Yuan Cao. 2023. ReAct: Synergizing Reasoning and Acting in Language Models. In The Eleventh International Conference on Learning Representations . Kigali, Rwanda. https://openreview.net/forum?i...

    2023

  40. [48]

    Differentiation

    Mert Yuksekgonul, Federico Bianchi, Joseph Boen, Sheng Liu, Zhi Huang, Carlos Guestrin, and James Zou. 2024. TextGrad: Automatic "Differentiation" via Text. (2024). arXiv:2406.07496

    2024 arXiv

  41. [49]

    Gonzalez, and Ion Stoica

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. 2023. Judging LLM-as-a-Judge with MT- Bench and Chatbot Arena. In Thirty-seventh Conference on Neura...

    2023

  42. [50]

    Wangchunshu Zhou, Yixin Ou, Shengwei Ding, Long Li, Jialong Wu, Tiannan Wang, Jiamin Chen, Shuai Wang, Xiaohua Xu, Ningyu Zhang, et al. 2024. Symbolic learning enables self-evolving agents. arXiv preprint arXiv:2406.18532 (2024)

    2024 arXiv

  43. [51]

    Mingchen Zhuge, Wenyi Wang, Louis Kirsch, Francesco Faccio, Dmitrii Khizbullin, and Jürgen Schmidhuber. 2024. GPTSwarm: Language Agents as Optimizable Graphs. In Proceedings of the 41st International Conference on Machine Learning (Proceedings of Machine Learning Research, Vol...

    2024

  44. [52]

    m a t h _ s o l v e r

    Barrett Zoph and Quoc Le. 2017. Neural Architecture Search With Reinforcement Learning. In Proceedings of International Conference on Learning Representations . A Appendix A.1 Bayesian Optimization Algorithm 3 Bayesian Optimization 1: Input: Evaluator function𝑓 Search Budget𝐵 ...

    2017

Pith tools

Reviewed August 8, 2026 · model on record in the stance chip above.