Pith. sign in

REVIEW 3 major objections 5 minor 56 references

Agent workflows need a serving layer that jointly picks each node’s model and verifier, then revises only unfinished work under load.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-12 05:55 UTC pith:2NZEFPXU

load-bearing objection Solid systems paper that names a real missing layer and backs joint model-verifier compilation plus a pre-solved load ladder with multi-workload and multi-tenant numbers; main limits are fleet scale and profile-transfer assumptions, not a broken core argument. the 3 major comments →

arxiv 2607.02942 v1 pith:2NZEFPXU submitted 2026-07-03 cs.DC cs.MA

A Workflow-Aware Serving Layer for Agentic Applications

Jiayi Qian , Zishen Wan , Hanchen Yang , Chun Tao , Souvik Kundu , Tushar Krishna This is my paper
classification cs.DC cs.MA
keywords agentic AI servingworkflow-aware schedulingmodel-verifier joint compilationinteger linear programstrategy ladderskill-conditioned profilingmulti-tenant SLO goodputtool-failure recovery
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Agentic applications turn each request into a directed acyclic graph of model and tool calls. Existing stacks split ownership: orchestrators know the graph but not live backend cost, while serving engines optimize single calls without workflow structure or optional quality operators. The paper argues that the physical plan—per-node model, verifier, and backend—is therefore unowned, even though accuracy is end-to-end and different nodes warrant different spending. Dyserve fills that gap. At admission it compiles a joint integer program over skill-conditioned offline profiles, topology reach, and measured node vulnerability so quality work concentrates where errors propagate furthest. Because no fixed latency–quality weight fits every mix, it pre-solves the same program at several pressure levels and, under load, swaps only the uncommitted suffix among those plans without running a solver on the hot path; a failed tool triggers a one-time residual re-solve that keeps finished work. Across four agentic benchmarks the compiled plans raise accuracy while cutting latency, and under multi-tenant bursts the ladder restores goodput for oversubscribed preferences without touching stable ones.

Core claim

The paper establishes that a dedicated workflow-aware serving layer can own the per-node physical plan throughout execution: jointly compiling model–verifier–backend assignments from transferable skill profiles and structure-aware weights, then adapting only uncommitted work via pre-solved pressure strategies under load and a residual re-solve on tool failure. On LiveCodeBench, GAIA, ComplexFuncBench, and SWE-bench those compiled strategies achieve the highest accuracy on every workload—three to ten points above the strongest baseline—at 1.1 to 6.8× lower latency; under bursts the precomputed ladder restores an oversubscribing plan’s SLO goodput from 18% to 67%, within 6.5 points of the best

What carries the argument

Admission-time ILP compilation over skill-conditioned, hardware-portable profiles of (model, verifier) pairs, weighted by topological reach and measured vulnerability, coupled with a precomputed strategy ladder that installs pressure-rung restrictions on the uncommitted suffix by pointer swap and a one-time residual re-solve on tool failure.

Load-bearing premise

The load-bearing premise is that offline skill-tagged profiles and a residual-makespan approximation transfer across workflows well enough for the ILP’s surrogate score to pick good physical plans without per-application re-profiling or fully modeling still-running committed predecessors.

What would settle it

Re-run the four benchmarks with held-out workflows whose dominant-skill tags are systematically wrong or whose residual branches are gated by long in-flight committed nodes; if the compiled plans then lose the accuracy and goodput margins over joint-axis and static baselines, the transfer and residual claims fail.

Watch this falsifier — get emailed when new claim-graph text bears on it.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes Dyserve, a serving layer between agent orchestrators and heterogeneous LLM backends that treats each request as a known DAG of LLM and tool nodes. At admission it jointly assigns per-node (model, verifier, backend) choices via a single ILP whose coefficients come from skill-conditioned offline profiles, topology reach weights, and counterfactual vulnerability; under load it swaps the uncommitted suffix among strategies pre-solved at several pressure rungs (keeping the solver off the load-shift path), and on tool failure it performs a one-time residual re-solve. On LiveCodeBench, GAIA, ComplexFuncBench, and SWE-bench the compiled plans report the highest accuracy of compared systems (3–10 points above the best baseline) at 1.1–6.8× lower latency; under multi-tenant bursts the ladder restores an oversubscribing plan’s SLO goodput from 18% to 67%, within 6.5 points of the best static plan, with control-path compile under 60 ms at p95.

Significance. If the results hold under broader fleets and noisier skill tags, the paper fills a real ownership gap between orchestrators (which see structure but not fleet cost) and engines (which see load but not workflow quality operators). The joint model–verifier compilation, explicit call-stage accounting for verification, and the precomputed strategy ladder (motivated by a measured online-solver failure mode) are concrete systems contributions. Strengths include held-out skill probes, exhaustive joint-vs-single-axis ablations (Table 3), bootstrap CIs and paired permutation tests, three arrival seeds for the burst headline, and an honest residual-makespan caveat in §E. The work is complementary to workflow synthesis, routing, and Sherlock-style placement rather than a replacement for them.

major comments (3)
  1. [§5.1, Eq. (1); §7.5 Limitations] §5.1 and abstract claim that skill-conditioned profiles Φ(s_n, m, p) “transfer across workflows” via dominant-skill tags with no re-profiling. Plan quality is stated to rest on those tags (§7.5). The four-workload evaluation uses held-out probes and shows gains, but there is no sensitivity study to tag noise or misclassification (e.g., systematically flipping code↔reasoning on mid-graph nodes). Because the ILP’s ranking surrogate (Eq. 5) is driven by these coefficients, a modest tag-error rate could erase the reported 3–10 point accuracy edge. A controlled tag-perturbation experiment, or a clearer scoping that transfer is demonstrated only under oracle/author tags on these four templates, is needed for the transfer claim to be load-bearing rather than aspirational.
  2. [§E Residual Formulation; Table 2; §6.2] §E states that residual makespan “can understate the suffix’s completion time when a long in-flight branch gates it,” because waiting on still-executing committed predecessors is not modeled. Load adaptation and recovery both install or re-solve over this residual (Eq. 6–7, §6.2–6.3). Under the double-pulse bursts and concurrent multi-tenant traces that produce the 18%→67% goodput result (Table 2, Fig. 9), this bias is unquantified. If understatement systematically mis-ranks pressure rungs or recovery plans as concurrency deepens, the ladder’s measured rescue may not generalize. Please quantify residual vs. true suffix completion time on the burst traces (or bound the error) and discuss whether rung selection remains stable under that bias.
  3. [Table 2; §7.4; §6.2] Table 2 and §7.4 show that restoring goodput still requires a mix-dependent rung set (aggressive L∈{0,1.5,3} on the balanced mix; gentle L∈{0,0.5,1} on the code-heavy mix). The paper’s motivation is that “no single latency-quality preference fits every workload mix,” yet the ladder reintroduces an operator choice of comparable sensitivity. Without an automatic rung-selection rule or a demonstration that one fixed ladder works across mixes, the claim that pre-solving removes preference fragility is only partially supported. Clarify how rungs should be chosen in deployment, or show a single ladder that matches the best-static band on both mixes.
minor comments (5)
  1. [Figure 7] Figure 7’s dual-axis stars/bars are dense; adding numerical accuracy labels next to stars (as in the text) would improve readability without relying on the caption alone.
  2. [§7.1 Baselines] The Sherlock-style baseline is a “training-free approximation” of the released system (§7.1). State explicitly which components of Sherlock are omitted so readers do not over-interpret the comparison as against the full trained system.
  3. [Eq. (3); §7.5] Eq. (3) defines ρ(n) with effort knob e; §7.5 sweeps e but does not report interaction with λ_ℓ under the burst setting. A short note or appendix plot would help operators who must set both.
  4. [Table 1] Table 1 is useful; a one-line row for Dyserve itself would make the “missing control boundary” claim self-contained.
  5. [Abstract; §1] Typo/consistency: abstract and intro use “Dyserve” and “1 .1to6 .8×” with odd spacing; normalize numeric formatting throughout.

Circularity Check

0 steps flagged

No significant circularity: empirical systems paper with held-out profiles, surrogate ILP validated by exhaustive measurement, and end-to-end evaluation on external benchmarks.

full rationale

Dyserve’s load-bearing chain is measurement → ILP ranking surrogate → held-out end-to-end accuracy/latency, not a derivation that reduces to its inputs by construction. Skill-conditioned profiles Φ(s_n, m, p) are built once from held-out probe datasets (HumanEval+/MBPP+, MATH-500/AIME, GPQA/MMLU-Pro) and are explicitly disjoint from reported requests; hardware enters only via per-model throughput sweeps. The objective (Eq. 5) is stated as a ranking surrogate, not a claim that success decomposes into independent node accuracies, and is validated by exhaustive per-node assignment on a LiveCodeBench subset (Table 3) where the full objective’s argmax recovers the highest measured accuracy. Weights and the effort knob are selected on held-out workflows and reused; the strategy ladder pre-solves pressure rungs and is audited under bursts against static plans and round-robin baselines. Self-citations (authors’ prior neuro-symbolic/agent work) are peripheral and not used to force uniqueness or forbid alternatives. Residual-makespan understatement and skill-tag transfer are limitations of generalization, not circular reductions. No self-definitional loop, no fitted parameter renamed as prediction of the same quantity, and no uniqueness theorem imported from the authors.

Axiom & Free-Parameter Ledger

4 free parameters · 5 axioms · 3 invented entities

The central claims rest on a systems model of predefined agent DAGs, transferable skill profiles, and a weighted ILP surrogate—not on new physics. Free parameters are operator weights and ladder rungs; axioms are domain assumptions about workflow materialization and error impact; invented entities are the Dyserve control plane constructs themselves.

free parameters (4)
  • Objective weights λ_ℓ, λ_c, λ_f, λ_ld
    Latency, cost, failure-risk, and load-demand weights selected on held-out workflows and reused; they place plans on the quality-latency frontier and define pressure rungs.
  • Topology emphasis β and effort knob e
    Hand/operator parameters that scale how strongly quality concentrates on high-reach nodes; e is swept as a one-dimensional family of operating points.
  • Pressure rung set L and hysteresis thresholds
    Discrete load levels and up-shift/flip-back/dwell rules chosen per mix (aggressive vs gentle rungs); they determine when the ladder fires.
  • Vulnerability coefficients v_n from fault injection
    Measured terminal-correctness drops conditioned on role, skill, and fan-in; used as risk prices in the objective rather than pure topology.
axioms (5)
  • domain assumption The logical workflow is known at admission as a finite concrete DAG after the orchestrator resolves branches and unrolls bounded loops.
    Stated in §1 and §4.1; dynamic graph expansion starts a new compilation instance and is outside the design.
  • domain assumption A node’s quality and residual error for a (model, verifier) pair are well approximated by its dominant skill class via offline probes that transfer across workflows.
    Core of §5.1 skill-conditioned profiling; tool_use substitutes reasoning probes for quality lookup.
  • domain assumption Hardware effects enter only through per-model decode throughput at a calibrated operating point, so profiles stay device-independent.
    Equation (2) and §5.1; adding a GPU requires only a throughput sweep.
  • ad hoc to paper The weighted sum of topology-scaled quality, cost, risk, and critical-path latency is a useful ranking surrogate for end-to-end workflow success.
    Objective (5); paper explicitly says it is not a claim that success decomposes into independent node accuracies and validates via exhaustive subset ranking.
  • ad hoc to paper Strategy changes only at node boundaries; in-progress calls are never migrated, and residual makespan can ignore waiting on still-executing committed predecessors.
    §6.1 consistency rule and residual formulation §E; can understate suffix completion time.
invented entities (3)
  • Dyserve workflow-aware serving layer no independent evidence
    purpose: Own the per-node physical plan (model, verifier, backend) between orchestrator and engines throughout execution.
    The paper’s primary system; independent evidence is the implemented adapters and evaluation, not an external physical discovery.
  • Precomputed strategy ladder with pressure-augmented ILP rungs no independent evidence
    purpose: Adapt uncommitted suffixes under load without placing a solver on the load-shift path.
    Introduced in §6.2 after online re-solving was measured to interfere; validated by burst experiments.
  • Skill-conditioned model-verifier profile table Φ no independent evidence
    purpose: Price quality, residual error, and token demand portably across open-ended workflows.
    Built from held-out skill probes; transfer is a design claim of the paper.

pith-pipeline@v1.1.0-grok45 · 27679 in / 3670 out tokens · 33181 ms · 2026-07-12T05:55:27.096237+00:00 · methodology

0 comments
read the original abstract

Agentic AI applications form an emerging serving workload in which a request creates a workflow: a directed acyclic graph of LLM and tool calls that exposes per-node model choices and optional quality operators such as verifiers. This workload falls between two existing layers. Model-serving engines execute individual calls efficiently but cannot see workflow structure, while agent frameworks fix the workflow but cannot see backend load, so neither jointly chooses each node's model, verifier, and backend under serving-time conditions. We present Dyserve, a workflow-aware serving layer that fills this gap. Dyserve compiles each workflow's per-node model and verifier choices in one integer linear program (ILP) over a heterogeneous backend pool, priced by skill-conditioned offline profiles that transfer across workflows. This couples with hardware entering only through per-model throughput sweeps, and is weighted to concentrate strong models and verification on the nodes whose errors propagate the furthest. Because no single latency-quality preference fits every workload mix, Dyserve pre-solves the program at several pressure levels at admission and shifts a workflow's uncommitted suffix among these strategies under load, keeping the solver off the load-shift path; a failed tool call triggers a one-time residual re-solve that preserves committed work.

Figures

Figures reproduced from arXiv: 2607.02942 by Chun Tao, Hanchen Yang, Jiayi Qian, Souvik Kundu, Tushar Krishna, Zishen Wan.

Figure 1
Figure 1. Figure 1: Agentic AI serving stack with Dyserve: It lies be￾tween the orchestration framework that submits ready work￾flow nodes, and the model-serving engines. At admission, Dyserve compiles each workflow into a per-node serving strategy; at runtime, it shifts the uncommitted suffix among pre-solved strategies under load and re-solves it once on a tool failure. No component in the conventional stack owns this deci￾… view at source ↗
Figure 2
Figure 2. Figure 2: Agent workflows expose serving headroom. On LiveCodeBench (hard), base strategies pin the model with no verification and verify-all strategies attach each node’s accuracy-maximizing verifier for its skill; the oracle selects, for every problem, the best verified outcome among the strategies executed on it. It exceeds 27B verify-all’s ac￾curacy by 12 points at 1.7× lower mean latency (244.1 s vs. 414.5 s). … view at source ↗
Figure 3
Figure 3. Figure 3: Node vulnerability depends on more than topology. Each point is the terminal-correctness drop after fault injec￾tion, conditioned on a correct unperturbed execution; color denotes required skill, marker shape fan-in. Cells within the same topological role differ sharply across semantic node types. the profile predicted. The second is workload drift. The pref￾erence weights that place a compiled plan on its… view at source ↗
Figure 5
Figure 5. Figure 5: The compiled strategy for the running coding ex￾ample. The high-reach planner gets the strong model without a verifier; the code generator substitutes a 9B model plus a verifier for a stronger model; the test generator takes the strong model unverified, since the test-execution tool al￾ready exercises its output; the answer node takes the cheap￾est model with a light check. No single global routing or veri… view at source ↗
Figure 6
Figure 6. Figure 6: The strategy ladder. The pressure-augmented pro￾gram is pre-solved over the full graph at 𝐾 rungs at admis￾sion; pressure triggers stepwise up-shifts and a direct flip￾back to the lowest drained rung. A shift installs the selected rung’s restriction to the uncommitted suffix; committed work never moves, no solver runs on this path. At 𝐿 = 0 this is the admission program restricted to the suffix; as 𝐿 grows… view at source ↗
Figure 7
Figure 7. Figure 7: End-to-end accuracy and latency on the four workloads. Bars report mean wall-clock latency (right axis), stars the official task accuracy (left axis). Dyserve has the highest accuracy on every workload at substantially lower latency than uniform verification and, where supported, MAS-Zero, which lacks the tool interfaces of GAIA and ComplexFuncBench and exceeds the per-task budget on SWE-bench. 68 70 72 74… view at source ↗
Figure 8
Figure 8. Figure 8: Steady-state multi-tenant serving at 1.0 and 2.5 requests/s (mixed trace, overall accuracy; up and left is bet￾ter). Base and Sherlock run on the (4+0) fleet. Heterogeneity moves the compiled policy left at equal or better accuracy, while it makes base worse on both axes; Sherlock-style place￾ment pays for unpriced verification in tail latency. Sherlock-style placement cannot substitute model strength for … view at source ↗
Figure 9
Figure 9. Figure 9: The strategy ladder under the double-pulse burst on the (2+2) fleet. Admission only executes the compiled quality plan unchanged; + ladder is that plan with pre-solved rungs; best static is the static sweep’s best preference on this trace (𝜆ℓ=0.1). Top: rolling p95 over 60-second arrival win￾dows (log scale); annotations: burst-window p95. Bottom: the fleet’s active rung, with rung 0 the preferred 𝜋 (0) . … view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

56 extracted references · 21 linked inside Pith

  1. [1]

    Pranjal Aggarwal, Aman Madaan, Ankit Anand, Srividya Pranavi Potharaju, Swaroop Mishra, Pei Zhou, Aditya Gupta, Dheeraj Ra- jagopal, Karthik Kappaganthu, Yiming Yang, Shyam Upadhyay, Man- aal Faruqui, and Mausam. 2025. AutoMix: Automatically Mixing Language Models. arXiv:2310.12963 [cs.CL]https://arxiv.org/abs/ 2310.12963

    Pith/arXiv arXiv 2025

  2. [2]

    LangChain AI. 2024. LangGraph: Building stateful, multi-actor appli- cations with LLMs.https://github.com/langchain-ai/langgraph

    2024

  3. [3]

    Gohar Irfan Chaudhry, Esha Choukse, Haoran Qiu, Íñigo Goiri, Ro- drigo Fonseca, Adam Belay, and Ricardo Bianchini. 2025. Murakkab: Resource-Efficient Agentic Workflow Orchestration in Cloud Plat- forms. arXiv:2508.18298 [cs.MA] doi:10.48550/arXiv.2508.18298

    Pith/arXiv arXiv doi:10.48550/arxiv.2508.18298 2025

  4. [4]

    Lingjiao Chen, Matei Zaharia, and James Zou. 2024. FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performance.Transactions on Machine Learning Research(2024).https: //openreview.net/forum?id=cSimKw5p6R

    2024

  5. [5]

    Jasper Dekoninck, Nikola Jovanović, Tim Gehrunger, Kári Rögnvalds- son, Ivo Petrov, Chenhao Sun, and Martin Vechev. 2026. Beyond Benchmarks: MathArena as an Evaluation Platform for Mathematics with LLMs. (2026). arXiv:2605.00674 [cs.CL]https://arxiv.org/abs/ 2605.00674

    Pith/arXiv arXiv 2026

  6. [6]

    Dujian Ding, Ankur Mallick, Chi Wang, Robert Sim, Subhabrata Mukherjee, Victor Ruhle, Laks V. S. Lakshmanan, and Ahmed Has- san Awadallah. 2024. Hybrid LLM: Cost-Efficient and Quality-Aware Query Routing. arXiv:2404.14618 [cs.LG]https://arxiv.org/abs/2404. 14618

    Pith/arXiv arXiv 2024

  7. [7]

    Tenenbaum, and Igor Mordatch

    Yilun Du, Shuang Li, Antonio Torralba, Joshua B. Tenenbaum, and Igor Mordatch. 2024. Improving Factuality and Reasoning in Lan- guage Models through Multiagent Debate. InProceedings of the 41st International Conference on Machine Learning (Proceedings of Machine Learning Research, Vol. 235), Ruslan Salakhutdinov, Zico Kolter, Kather- ine Heller, Adrian We...

    2024

  8. [8]

    2011.PuLP: A Linear Programming Toolkit for Python

    Iain Dunning, Stuart Mitchell, and Michael O’Sullivan. 2011.PuLP: A Linear Programming Toolkit for Python. Technical Report. Depart- ment of Engineering Science, The University of Auckland.https: //optimization-online.org/2011/09/3178/

    2011

  9. [9]

    Hongcheng Gao, Yue Liu, Yufei He, Longxu Dou, Chao Du, Zhijie Deng, Bryan Hooi, Min Lin, and Tianyu Pang. 2025. FlowReasoner: Reinforcing Query-Level Meta-Agents. arXiv:2504.15257 [cs.AI]https: //arxiv.org/abs/2504.15257

    Pith/arXiv arXiv 2025

  10. [10]

    Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac’h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. 2024. The L...

    doi:10.5281/zenodo.12608602 2024

  11. [11]

    Sirui Hong, Mingchen Zhuge, Jonathan Chen, Xiawu Zheng, Yuheng Cheng, Jinlin Wang, Ceyao Zhang, Zili Wang, Steven Ka Shing Yau, Zi- juan Lin, Liyang Zhou, Chenyu Ran, Lingfeng Xiao, Chenglin Wu, and Jürgen Schmidhuber. 2024. MetaGPT: Meta Programming for A Multi- Agent Collaborative Framework. InInternational Conference on Learn- ing Representations.https...

    2024

  12. [12]

    Shengran Hu, Cong Lu, and Jeff Clune. 2025. Automated Design of Agentic Systems. arXiv:2408.08435 [cs.AI]https://arxiv.org/abs/2408. 08435

    Pith/arXiv arXiv 2025

  13. [13]

    Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, and Ion Stoica. 2024. LiveCodeBench: Holistic and Contamination Free Evalu- ation of Large Language Models for Code. arXiv:2403.07974 [cs.SE] https://arxiv.org/abs/2403.07974

    Pith/arXiv arXiv 2024

  14. [14]

    Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik R

    Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik R. Narasimhan. 2024. SWE-bench: Can Lan- guage Models Resolve Real-World GitHub Issues?. InInternational Con- ference on Learning Representations.https://openreview.net/forum? id=VTF8yNQM66

    2024

  15. [15]

    Zixuan Ke, Austin Xu, Yifei Ming, Xuan-Phi Nguyen, Ryan Chin, Caiming Xiong, and Shafiq Joty. 2026. MAS-ZERO: Designing Multi- Agent Systems with Zero Supervision. arXiv:2505.14996 [cs.CL]https: //arxiv.org/abs/2505.14996

    arXiv 2026

  16. [16]

    Joshi, Hanna Moazam, Heather Miller, Matei Zaharia, and Christopher Potts

    Omar Khattab, Arnav Singhvi, Paridhi Maheshwari, Zhiyuan Zhang, Keshav Santhanam, Sri Vardhamanan, Saiful Haq, Ashutosh Sharma, Thomas T. Joshi, Hanna Moazam, Heather Miller, Matei Zaharia, and Christopher Potts. 2023. DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelines. arXiv:2310.03714 [cs.CL] https://arxiv.org/abs/2310.03714

    Pith/arXiv arXiv 2023

  17. [17]

    Gonzalez, Hao Zhang, and Ion Stoica

    Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica

  18. [18]

    InProceedings of the 29th Symposium on Operating Systems Principles

    Efficient Memory Management for Large Language Model Serving with PagedAttention. InProceedings of the 29th Symposium on Operating Systems Principles. Association for Computing Machinery, 611–626. doi:10.1145/3600006.3613165

    doi:10.1145/3600006.3613165

  19. [19]

    Hunter Lightman, Vineet Kosaraju, Yura Burda, Harri Edwards, Bowen Baker, Teddy Lee, Jan Leike, John Schulman, Ilya Sutskever, and Karl Cobbe. 2023. Let’s Verify Step by Step.arXiv preprint arXiv:2305.20050 (2023)

    Pith/arXiv arXiv 2023

  20. [20]

    Chaofan Lin, Zhenhua Han, Chengruidong Zhang, Yuqing Yang, Fan Yang, Chen Chen, and Lili Qiu. 2024. Parrot: Efficient Serving of LLM- based Applications with Semantic Variable. arXiv:2405.19888 [cs.LG] https://arxiv.org/abs/2405.19888

    Pith/arXiv arXiv 2024

  21. [21]

    Jiawei Liu, Chunqiu Steven Xia, Yuyao Wang, and Lingming Zhang

  22. [22]

    arXiv:2305.01210 [cs.SE]https://arxiv.org/abs/2305.01210

    Is Your Code Generated by ChatGPT Really Correct? Rig- orous Evaluation of Large Language Models for Code Generation. arXiv:2305.01210 [cs.SE]https://arxiv.org/abs/2305.01210

    Pith/arXiv arXiv

  23. [23]

    13 Gonzalez, and Ion Stoica

    Michael Luo, Xiaoxiang Shi, Colin Cai, Tianjun Zhang, Justin Wong, Yichuan Wang, Chi Wang, Yanping Huang, Zhifeng Chen, Joseph E. 13 Gonzalez, and Ion Stoica. 2025. Autellix: An Efficient Serving Engine for LLM Agents as General Programs. arXiv:2502.13965 [cs.LG]https: //arxiv.org/abs/2502.13965

    Pith/arXiv arXiv 2025

  24. [24]

    Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, Shashank Gupta, Bodhisattwa Prasad Majumder, Kather- ine Hermann, Sean Welleck, Amir Yazdanbakhsh, and Peter Clark

  25. [25]

    InAd- vances in Neural Information Processing Systems, Vol

    Self-Refine: Iterative Refinement with Self-Feedback. InAd- vances in Neural Information Processing Systems, Vol. 36. 46534–46594

  26. [26]

    Grégoire Mialon, Clémentine Fourrier, Thomas Wolf, Yann LeCun, and Thomas Scialom. 2024. GAIA: A Benchmark for General AI Assistants. InInternational Conference on Learning Representations. https://openreview.net/forum?id=fibxvahvs3

    2024

  27. [27]

    Boye Niu, Yiliao Song, Kai Lian, Yifan Shen, Yu Yao, Kun Zhang, and Tongliang Liu. 2025. Flow: Modularized Agentic Workflow Au- tomation. InInternational Conference on Learning Representations. https://openreview.net/forum?id=sLKDbuyq99

    2025

  28. [28]

    Gonzalez, M

    Isaac Ong, Amjad Almahairi, Vincent Wu, Wei-Lin Chiang, Tianhao Wu, Joseph E. Gonzalez, M. Waleed Kadous, and Ion Stoica. 2025. RouteLLM: Learning to Route LLMs from Preference Data. InInter- national Conference on Learning Representations.https://openreview. net/forum?id=8sSqNntaMr

    2025

  29. [29]

    David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R. Bowman. 2023. GPQA: A Graduate-Level Google-Proof Q&A Bench- mark. arXiv:2311.12022 [cs.AI]https://arxiv.org/abs/2311.12022

    Pith/arXiv arXiv 2023

  30. [30]

    Yeonju Ro, Haoran Qiu, Íñigo Goiri, Rodrigo Fonseca, Ricardo Bian- chini, Aditya Akella, Zhangyang Wang, Mattan Erez, and Esha Choukse. 2025. Sherlock: Reliable and Efficient Agentic Workflow Execution. arXiv:2511.00330 [cs.MA] doi:10.48550/arXiv.2511.00330

    doi:10.48550/arxiv.2511.00330 2025

  31. [31]

    Jon Saad-Falcon, Adrian Gamarra Lafuente, Shlok Natarajan, Nahum Maru, Hristo Todorov, Etash Guha, E Kelly Buchanan, Mayee Chen, Neel Guha, Christopher Ré, et al . 2024. Archon: An architecture search framework for inference-time techniques.arXiv preprint arXiv:2409.15254(2024)

    Pith/arXiv arXiv 2024

  32. [32]

    Rana Shahout, Hayder Tirmazi, Minlan Yu, and Michael Mitzenmacher

  33. [33]

    arXiv preprint arXiv:2603.13605(2026)

    Orla: A Library for Serving LLM-Based Multi-Agent Systems. arXiv preprint arXiv:2603.13605(2026)

    arXiv 2026

  34. [34]

    Significant Gravitas. 2023. AutoGPT: An Autonomous GPT-4 Experi- ment.https://github.com/Significant-Gravitas/AutoGPT. Software

    2023

  35. [35]

    Biao Sun, Ziming Huang, Hanyu Zhao, Wencong Xiao, Xinyi Zhang, Yong Li, and Wei Lin. 2024. Llumnix: Dynamic Scheduling for Large Language Model Serving. arXiv:2406.03243 [cs.AR]https://arxiv.org/ abs/2406.03243

    Pith/arXiv arXiv 2024

  36. [36]

    Xin Tan, Yimin Jiang, Yitao Yang, and Hong Xu. 2025. Towards End-to- End Optimization of LLM-based Applications with Ayo. InProceedings of the 30th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 2 (ASPLOS ’25). ACM, 1302–1316. doi:10.1145/3676641.3716278

    doi:10.1145/3676641.3716278 2025

  37. [37]

    Zishen Wan, Che-Kai Liu, Jiayi Qian, Hanchen Yang, Arijit Raychowd- hury, and Tushar Krishna. 2026. Reason: Accelerating probabilistic log- ical reasoning for scalable neuro-symbolic intelligence.arXiv preprint arXiv:2601.20784(2026)

    arXiv 2026

  38. [38]

    Zishen Wan, Hanchen Yang, Jiayi Qian, Ritik Raj, Joongun Park, Chenyu Wang, Arijit Raychowdhury, and Tushar Krishna. 2025. Com- positional AI Beyond LLMs: System Implications of Neuro-Symbolic- Probabilistic Architectures. InProceedings of the 31st ACM Interna- tional Conference on Architectural Support for Programming Languages and Operating Systems, Vol...

    doi:10.1145/3760250.3762235 2025

  39. [39]

    Chenyu Wang, Zishen Wan, Hao Kang, Emma Chen, Zhiqiang Xie, Tushar Krishna, Vijay Janapa Reddi, and Yilun Du. 2025. Slm-mux: Orchestrating small language models for reasoning.arXiv preprint arXiv:2510.05077(2025)

    arXiv 2025

  40. [40]

    Le, Ed H

    Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc V. Le, Ed H. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. 2023. Self- Consistency Improves Chain of Thought Reasoning in Language Mod- els. InInternational Conference on Learning Representations.https: //openreview.net/forum?id=1PL1NIMMrw

    2023

  41. [41]

    Yiding Wang, Kai Chen, Haisheng Tan, and Kun Guo. 2023. Tabi: An Efficient Multi-Level Inference System for Large Language Models. InProceedings of the Eighteenth European Conference on Computer Systems. Association for Computing Machinery, 233–248. doi:10.1145/ 3552326.3587438

    arXiv 2023

  42. [42]

    Yingxu Wang, Siwei Liu, Jinyuan Fang, and Zaiqiao Meng. 2025. EvoA- gentX: An Automated Framework for Evolving Agentic Workflows. InProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, Ivan Habernal, Peter Schulam, and Jörg Tiedemann (Eds.). Association for Computational Linguistics, Suzhou, China,...

    doi:10.18653/v1/2025.emnlp- 2025

  43. [43]

    Yubo Wang, Xueguang Ma, Ge Zhang, Yuansheng Ni, Abhranil Chan- dra, Shiguang Guo, Weiming Ren, Aaran Arulraj, Xuan He, Ziyan Jiang, Tianle Li, Max Ku, Kai Wang, Alex Zhuang, Rongqi Fan, Xiang Yue, and Wenhu Chen. 2024. MMLU-Pro: A More Robust and Challenging Multi- Task Language Understanding Benchmark. arXiv:2406.01574 [cs.CL] https://arxiv.org/abs/2406.01574

    Pith/arXiv arXiv 2024

  44. [44]

    Yinjie Wang, Ling Yang, Guohao Li, Mengdi Wang, and Bryon Aragam

  45. [45]

    arXiv:2502.04306 [cs.CL]https://arxiv.org/ abs/2502.04306

    ScoreFlow: Mastering LLM Agent Workflows via Score-based Preference Optimization. arXiv:2502.04306 [cs.CL]https://arxiv.org/ abs/2502.04306

    Pith/arXiv arXiv

  46. [46]

    White, Doug Burger, and Chi Wang

    Qingyun Wu, Gagan Bansal, Jieyu Zhang, Yiran Wu, Beibin Li, Erkang Zhu, Li Jiang, Xiaoyun Zhang, Shaokun Zhang, Ahmed Awadallah, Ryen W. White, Doug Burger, and Chi Wang

  47. [47]

    InConference on Language Modeling

    AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversation. InConference on Language Modeling. https://www.microsoft.com/en-us/research/publication/autogen- enabling-next-gen-llm-applications-via-multi-agent-conversation- framework/

  48. [48]

    Narasimhan, and Yuan Cao

    Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik R. Narasimhan, and Yuan Cao. 2023. ReAct: Synergizing Reasoning and Acting in Language Models. InInternational Conference on Learning Representations.https://openreview.net/forum?id=WE_vluYUL-X

    2023

  49. [49]

    Naimeng Ye, Arnav Ahuja, Georgios Liargkovas, Yunan Lu, Kostis Kaffes, and Tianyi Peng. 2025. Speculative Actions: A Lossless Framework for Faster Agentic Systems. arXiv:2510.04371 [cs.AI] https://arxiv.org/abs/2510.04371

    Pith/arXiv arXiv 2025

  50. [50]

    Guibin Zhang, Luyang Niu, Junfeng Fang, Kun Wang, Lei Bai, and Xiang Wang. 2025. Multi-agent Architecture Search via Agentic Supernet. InInternational Conference on Machine Learning.https: //openreview.net/forum?id=imcyVlzpXh

    2025

  51. [51]

    Jiayi Zhang, Jinyu Xiang, Zhaoyang Yu, Fengwei Teng, Xiong-Hui Chen, Jiaqi Chen, Mingchen Zhuge, Xin Cheng, Sirui Hong, Jin- lin Wang, Bingnan Zheng, Bang Liu, Yuyu Luo, and Chenglin Wu

  52. [52]

    InInterna- tional Conference on Learning Representations.https://openreview.net/ forum?id=z5uVAKwmjf

    AFlow: Automating Agentic Workflow Generation. InInterna- tional Conference on Learning Representations.https://openreview.net/ forum?id=z5uVAKwmjf

  53. [53]

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhang- hao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. 2023. Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in neural information processing systems36 (2023), 46595– 46623

    2023

  54. [54]

    Gonzalez, Clark Barrett, and Ying Sheng

    Lianmin Zheng, Liangsheng Yin, Zhiqiang Xie, Chuyue Sun, Jeff Huang, Cody Hao Yu, Shiyi Cao, Christos Kozyrakis, Ion Sto- ica, Joseph E. Gonzalez, Clark Barrett, and Ying Sheng. 2024. SGLang: Efficient Execution of Structured Language Model Programs. arXiv:2312.07104 [cs.AI]https://arxiv.org/abs/2312.07104 14 Policy Behavior Calls nonepass-through 1 self_...

    Pith/arXiv arXiv 2024

  55. [55]

    Lucen Zhong, Zhengxiao Du, Xiaohan Zhang, Haiyi Hu, and Jie Tang. 2025. ComplexFuncBench: Exploring Multi-Step and Constrained Function Calling under Long-Context Scenario. arXiv:2501.10132 [cs.CL]https://arxiv.org/abs/2501.10132

    Pith/arXiv arXiv 2025

  56. [56]

    Mingchen Zhuge, Wenyi Wang, Louis Kirsch, Francesco Faccio, Dmitrii Khizbullin, and Jürgen Schmidhuber. 2024. Language Agents as Optimizable Graphs. arXiv:2402.16823 [cs.AI]https://arxiv.org/ abs/2402.16823 A Verification Policy Call Structures Table 5 lists the eight verification policies exposed at every applicable LLM node with their call structures. E...

    Pith/arXiv arXiv 2024