Pith. sign in

REVIEW 4 major objections 6 minor 30 references

Composable Effect Handling for Programming LLM-integrated Scripts

T0 review · 4 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Treating LLM calls, I/O, and concurrency as abstract operations discharged by composable effect handlers lets developers write one sequential script and reparameterize it for parallelism, testing, or tracing—measured here as a 10.88×…

desk verdict A clean new application of effect handlers to LLM scripting that deserves refereeing; the 10.88x speedup is the load-bearing claim and the sync baseline is never shown. read the letter →

arxiv 2507.22048 v1 pith:4ADEQWQA submitted 2025-07-29 cs.PL

Di Wang This is my paper · ORCID
classification cs.PL
keywords effecthandlersalgebraiceffectsLLMscriptingmodularityparallelismTree-of-Thoughtsasynchronousprogrammingprogramcomposition
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that composable effect handling—the programming technique that decouples abstract operations from their implementations—is a natural fit for writing LLM-integrated scripts. It proposes treating LLM calls, I/O, and concurrency primitives as abstract operations that are discharged by interchangeable effect handlers, so the workflow script itself never mentions async, awaits, or a specific LLM API. The central evidence is a Tree-of-Thoughts case study in which the same sequential workflow, wrapped only with parallel handlers, runs about 10.88× faster than a synchronous version while preserving modularity. If the approach holds, developers can write one clean script and reparameterize it for speed, testing, or tracing by swapping handlers.

What carries the argument

The central objects are abstract operations and effect handlers. An abstract operation such as complete(prompt) is a callable placeholder with no implementation; an effect handler is a manager that registers a set of operations and can be composed by stacking, so that the runtime dispatches each operation to the topmost handler that discharges it. The paper defines four abstract operations—async_, await_, complete, and parse—and builds handlers that discharge them with asyncio, the OpenAI async client, and a sequencing policy that preserves the order of side-effecting callbacks. The same composition mechanism separates the Tree-of-Thoughts workflow from the Game-of-24 specifics, allowing parallelism to be added purely by changing handlers.

What would settle it

Reproduce the Tree-of-Thoughts case study with the synchronous baseline implemented as a plain sequential loop that awaits each LLM response before issuing the next request; if the speedup drops below roughly 2×, or if the async version's per-request latency is not lower than the synchronous version's, the central performance claim would be an artifact of the chosen baseline.

Watch

Extended reading notes

Core claim

The paper claims that the four abstract operations async_, await_, complete, and parse, combined with one-shot effect handlers that discharge them, let a developer write an LLM workflow as an ordinary sequential Python program and then obtain parallel, traced, or mocked executions without modifying the workflow. The composition rule is that handlers form a stack; an operation is discharged by the topmost handler that knows it, and a handler may in turn invoke other operations that are discharged by handlers below it. The Tree-of-Thoughts case study uses this to implement beam search sequentially and to discharge expand and score through AsyncGame24Handler, reusing generic AsyncHandler, AsyncLLMHandler, and AsyncSeqHandler, yielding an average 10.88× speedup over a synchronous handler-based baseline. The paper also sketches an operational semantics showing the intended behavior of a core calculus with handler stacks.

Load-bearing premise

The measured speedup assumes the synchronous Game24Handler baseline is a fair, typical sequential implementation; the paper gives no code for that handler, so if the baseline includes avoidable serialization or batching inefficiencies, the 10.88× figure overstates the benefit.

Editorial extensions

If this is right

  • A developer can write an LLM-integrated script once in a clean sequential style and later wrap it in different handler combinations to obtain parallel execution, a replay mock for testing, or a logged trace, without editing the workflow.
  • The 10.88× average speedup in the Tree-of-Thoughts case study suggests that the main cost of parallelism—synchronization and scheduling—can be pushed entirely into reusable handlers, so individual scripts need not be rewritten for performance.
  • Since handlers compose, generic infrastructure such as event loops, LLM clients, and sequencing policies can be built once and shared across many scripts, with application-specific operations like expand and score handled separately.
  • The modularity claim implies that swapping an LLM backend only requires providing a new handler for complete and parse, rather than modifying every call site in the script.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the approach generalizes, it offers a middle path between monolithic agent frameworks and hand-rolled async code: the workflow stays library-agnostic while performance strategies are selected externally, which could make benchmarking and testing of LLM pipelines substantially cheaper.
  • The four abstract operations are a deliberately small vocabulary; one could extend them with operations for streaming, tool calls, retries, or rate limiting, and the same handler-stack discipline would keep those concerns orthogonal to workflow logic.
  • The appendix's observation that multi-shot handlers, needed for backtracking search, require runtime support suggests that a Python implementation has an inherent ceiling; a language with native algebraic effects could carry the same style further, for example by implementing search strategies like Tree-of-Thoughts purely as handlers.
  • The speedup number depends on the synchronous baseline's fairness; the paper does not show the synchronous handler's code, so a natural next step is to release both handlers and reproduce the comparison on a fixed prompt set.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes using composable algebraic effect handlers to structure LLM-integrated scripts, separating workflow logic from effectful operations such as LLM calls, I/O, and concurrency. It presents a Python-based implementation sketch with abstract operations (async_, await_, complete, parse) and handlers (AsyncHandler, AsyncLLMHandler, AsyncSeqHandler), and a Tree-of-Thoughts case study on Game of 24, reporting an average 10.88× speedup over a synchronous baseline. The paper argues that this style preserves modularity while enabling parallelism, and it includes an operational semantics for the handler calculus in the appendix.

Significance. If the central claims are established, the paper offers a genuinely useful programming style for LLM scripting: it shows how standard PL techniques (algebraic effects and handlers) can decouple workflow logic from concrete LLM and concurrency implementations, allowing the same script to be re-parameterized for parallelism, mocking, or tracing. The core idea is appealing and clearly explained, and the paper makes a credible conceptual connection between effect handlers and LLM-agent programming. The appendix's operational semantics is a valuable formal grounding, even if it is non-integral to the main empirical claim. However, the quantitative evidence is thin: the baseline is not actually shown, the evaluation rests on three inputs with no variance reporting, and no source code is provided. The modularity claim is also only demonstrated through one compositional example. These gaps currently prevent the paper from fully supporting its headline speedup and general-purpose claims.

major comments (4)
  1. [§3, Table 1] The synchronous baseline Game24Handler is never shown; the paper only states that the synchronous version is obtained by replacing AsyncGame24Handler with Game24Handler. No code, prompts, or batching details are given for Game24Handler, making it impossible to rule out avoidable serialization or per-call overhead in the baseline. Since the 10.88× speedup is the central quantitative claim, this missing baseline is a load-bearing reproducibility and fairness gap. Provide the complete baseline handler implementation, the prompt templates, the number of LLM calls per input for both versions, and a reason to believe the baseline matches typical production synchronous execution.
  2. [§3, paragraph 3] The paper states that the number of steps is set to 4 because "we add a final step to ask the LLM to validate the solution and extract a single expression," but it never describes how the synchronous version treats this final step. If the baseline omits the validation call or makes a different number of LLM requests, the speedup conflates an algorithmic difference with the effect-handling benefit. Specify the exact workflow and the number of LLM calls for both the async and sync versions, and confirm that the only intended difference is the concurrency strategy.
  3. [§3, Table 1] The evaluation reports a single run per input for three inputs, with no variance, no confidence intervals, and no acknowledgment of remote LLM latency variability. Given that wall-clock time against a remote service (Qwen-Turbo) is highly sensitive to network conditions and service load, the reported average speedup is not statistically substantiated. Report multiple runs per input (at least 3–5), include per-run raw timings, and state any concurrency-relevant parameters (e.g., connection limits, timeouts) that could affect the comparison.
  4. [§2 and §3, modularity claim] The abstract and introduction claim the approach achieves speedups "without compromising modularity," but the only concrete evidence is the single tree_of_thoughts composition with AsyncGame24Handler, plus the short research_topics example in §2. No evaluation demonstrates that handlers can be swapped arbitrarily (e.g., switching to AsyncReplayLLMHandler for mocking, or adding tracing handlers) while preserving correct behavior. The modularity claim is plausible but under-supported; qualify it to the demonstrated scope or add an experiment showing the same workflow running under multiple handler configurations with equivalent outputs.
minor comments (6)
  1. [§2, 'Abstract Operations'] The paper defines abstract operations as "callable objects" but does not specify the public interface of Operation (e.g., how operations are constructed and whether they are invoked with positional arguments). Clarify the API so that the reader can relate the code snippets to the operational semantics.
  2. [§3, 'Case Study'] The phrase "conceptually similar to the ToT's official implementation" is too vague; please give the exact repository commit or version of [27] and state which parts of that implementation were reused or adapted for Game24Handler.
  3. [§3, Table 1] The header "Async" and "Sync" could be made clearer by indicating the units (seconds, as stated in the caption) and by noting the number of runs per input, if multiple runs are eventually reported.
  4. [References] Reference [22] is the Qwen2.5 Technical Report, but the text uses the Qwen-Turbo model. Cite the specific model card or API documentation for Qwen-Turbo, and state the sampling parameters (temperature, max tokens) used in the evaluation.
  5. [§4, Related Work] The paragraph on Epic (reference [14]) is useful but terse; a sentence describing the escalation of 'opportunistic evaluation' and how it differs from the handler-based approach would help the reader assess the claimed orthogonality.
  6. [Throughout] Minor typos and formatting issues: in the code comment in §3, "inital" should be "initial"; the appendix typesetting for the (OpForward) rule is difficult to parse, and the variables in Figure 1 could use a fresh set of metavariables to avoid confusion between handler names and the stack H.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the speedup is an empirical measurement, no fitted parameter is relabeled as a prediction, and the paper contains no load-bearing self-citation chain.

full rationale

The paper's central claim is that composing effect handlers yields modular LLM-integrated scripts that can nevertheless achieve parallel speedups. The 10.88x speedup in Section 3 (Table 1) is a measured running-time comparison between an asynchronous handler composition and a synchronous handler composition, not a quantity derived from the approach's own assumptions. No parameter is fitted to data and then reported as a prediction; no equation defines one claimed result in terms of another; and the paper contains no self-citations, so there is no self-citation loop carrying the argument. The background on algebraic effects and handlers is cited to prior external literature (Plotkin and Power, Plotkin and Pretnar, Kammar et al.), and those citations supply independent, established foundations rather than an unverified premise that forces the paper's conclusion. The reader-identified concern that the synchronous baseline Game24Handler is not shown in detail is a legitimate evaluation-fairness or soundness question, but it is not circular reasoning: an unfair baseline would weaken the empirical comparison without making the derivation equivalent to its inputs. The modularity benefit is demonstrated concretely by running the same workflow script under different with-block handler compositions, which is a direct construction rather than a re-labeling of a known result. Accordingly, there is no circular step to report, and the appropriate score is 0.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The central performance claim rests on hand-chosen ToT hyperparameters, the correctness of effect-handler semantics, the concurrency behavior of the LLM API, and the representativeness of the sync baseline. No fitted coefficients or new physical entities appear; the speedup is measured, not derived.

free parameters (3)
  • n_steps = 4
    Number of Tree-of-Thoughts search steps; hand-chosen in §3 to match the ToT setup plus a final validation step. Affects the number of LLM calls and absolute runtimes.
  • n_select = 5
    Beam width for the frontier; hand-selected based on the ToT configuration. Determines how many states are expanded each step.
  • n_eval = 3
    Number of evaluations per candidate; hand-selected. Increases the LLM call count and thus the wall-clock time.
assumptions (4)
  • domain assumption The algebraic theory of abstract operations and effect handlers is implementable in Python with with-block handler stacks.
    The paper's mechanism depends on this implementation assumption, stated in §2 but not proven.
  • domain assumption OpenAI's async client supports overlapping concurrent LLM requests without serialization or throttling that would negate the speedup.
    AsyncLLMHandler relies on awaiting chat.completions.create inside coroutines; the speedup assumes the service can handle parallel calls.
  • ad hoc to paper The operational semantics in Appendix B correctly models the intended runtime and is sound.
    The small-step rules are presented without proof of correctness or correspondence to the Python implementation.
  • domain assumption Game24Handler is conceptually similar to the official synchronous ToT implementation.
    This is asserted in §3 without showing the code, and it underpins the fairness of the baseline used to compute speedups.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Composable Effect Handling for Programming LLM-integrated Scripts." pith.science (2026). https://pith.science/paper/4ADEQWQA

@misc{pith2026250722048,
  author       = {Pith},
  title        = {Pith review of: Composable Effect Handling for Programming LLM-integrated Scripts},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/4ADEQWQA}},
  note         = {Machine review of arXiv:2507.22048}
}
abstract

Implementing LLM-integrated scripts introduces challenges in modularity and performance, as scripts are often coupled to specific LLM implementations and fail to exploit parallelization opportunities. This paper proposes using composable effect handling to separate workflow logic from effectful operations, such as LLM calls, I/O, and concurrency, enabling modularity without sacrificing the opportunity for performance optimization. By treating these operations as abstract interfaces and discharging them via effect handlers, this paper shows that scripts can achieve significant speedups (e.g., 10$\times$ in a Tree-of-Thoughts case study) without compromising modularity. This paper aims to promote composable effect handling as a programming style for LLM scripting.

Figures

Figures reproduced from arXiv: 2507.22048 by the authors.

Figure 1
Figure 1. Rules of the operational semantics. to handle them. For simplicity, we assume every operation receives exactly one argument. In with ℎ do 𝑐, we push ℎ onto the handler stack, proceed to execute 𝑐, and pop out ℎ after 𝑐 finishes. We use ∅ to denote an empty stack and ℎ : H to denote a stack formed by pushing ℎ onto H. 𝑣 F 𝑥 | true | false | · · · ℎ F handler {op1 (𝑥) ↦→ 𝑐1; · · · ; op𝑛 (𝑥) ↦→ 𝑐𝑛} 𝑐 F return 𝑣 | let 𝑥… view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

30 extracted references · 24 canonical work pages

  1. [1]

    Agno, Inc. 2025. agno-ai/agno: Full-stack framework for building Multi- Agent Systems with memory, knowledge and reasoning. Available on https://github.com/agno-agi/agno

    work page 2025

  2. [2]

    Anthropic. 2024. Building effective agents. Available on https://www. anthropic.com/engineering/building-effective-agents

    work page 2024

  3. [3]

    Andrej Bauer and Matija Pretnar. 2015. Programming with algebraic effects and handlers. J. Logical and Algebraic Methods in Programming 84 (January 2015), 108–123. Issue 1. doi:10.1016/j.jlamp.2014.02.001

    doi:10.1016/j.jlamp.2014.02.001 2015

  4. [4]

    Luca Beurer-Kellner, Marc Fischer, and Martin Vechev. 2023. Prompting Is Programming: A Query Language for Large Language Models. Proc. ACM Program. Lang. 7, 186 (June 2023), 1946–1969. Issue PLDI. doi: 10. 1145/3591300

    work page 2023

  5. [5]

    Chen, Martin Jankowiak, Fritz Obermeyer, Neeraj Pradhan, Theofanis Karaletsos, Rishabh Singh, Paul Szerlip, Paul Horsfall, and Noah D

    Eli Bingham, Jonathan P. Chen, Martin Jankowiak, Fritz Obermeyer, Neeraj Pradhan, Theofanis Karaletsos, Rishabh Singh, Paul Szerlip, Paul Horsfall, and Noah D. Goodman. 2018. Pyro: Deep Universal Probabilistic Programming. J. Machine Learning Research 20 (January 2018). Issue 1. https://dl.acm.org/doi/10.5555/3322706.3322734

    work page arXiv 2018

  6. [6]

    Hu, Moksh Jain, Eric Elmoznino, Younesse Kaddar, Guil- laume Lajoie, Yoshua Bengio, and Nikolay Malkin

    Edward J. Hu, Moksh Jain, Eric Elmoznino, Younesse Kaddar, Guil- laume Lajoie, Yoshua Bengio, and Nikolay Malkin. 2024. Amortizing intractable inference in large language models. InInt. Conf. on Learning Representations (ICLR’24)

    work page 2024

  7. [7]

    Ohad Kammar, Sam Lindley, and Nicolas Oury. 2013. Handlers in Action. In Int. Conf. on Functional Programming (ICFP’13) . 145–158. doi:10.1145/2500365.2500590

    work page arXiv 2013

  8. [8]

    Satoru Kawahara and Yukiyoshi Kameyama. 2020. One-shot Algebraic Effects as Coroutines. In Trends in Functional Programming (TFP’20) . 159–179. doi:10.1007/978-3-030-57761-2_8

    work page doi:10.1007/978-3-030-57761-2_8 2020

Show all 30 references
  1. [9]

    Joshi, Hanna Moazam, Heather Miller, Matei Zaharia, and Christopher Potts

    Omar Khattab, Arnav Singhvi, Paridhi Maheshwari, Zhiyuan Zhang, Keshav Santhanam, Sri Vardhamanan, Saiful Haq, Ashutosh Sharma, Thomas T. Joshi, Hanna Moazam, Heather Miller, Matei Zaharia, and Christopher Potts. 2024. DSPy: Compiling Declarative Language Model Calls into Self...

    2024

  2. [10]

    LangChain, Inc. 2025. langchain-ai/langchain: Build context-aware reasoning applications. Available on https://github.com/langchain- ai/langchain

    2025

  3. [11]

    Lew, Tan Zhi-Xuan, Gabriel Grand, and Vikash K

    Alexander K. Lew, Tan Zhi-Xuan, Gabriel Grand, and Vikash K. Mans- inghka. 2024. Sequential Monte Carlo Steering of Large Language Models using Probabilistic Programs. doi:abs/2306.03081

    2024 arXiv

  4. [12]

    Ziyang Li, Jiani Huang, Jason Liu, Felix Zhu, Eric Zhao, William Dodds, Neelay Velingker, Rajeev Alur, and Mayur Naik. 2024. Relational Pro- gramming with Foundation Models. InAAAI Conf. on Artif. Intelligence (AAAI’24). 10635–10644. doi:10.1609/aaai.v38i9.28934

    2024 doi

  5. [13]

    Jerry Liu. 2025. run-llama/llama_index: LlamaIndex is the leading framework for building LLM-powered agents over your data. Available on https://github.com/run-llama/llama_index

    2025

  6. [14]

    Stephen Mell, Konstantinos Kallas, Steve Zdancewic, and Osbert Bastani. 2025. Opportunistically Parallel Lambda Calculus. https: //arxiv.org/abs/2405.11361

    2025

  7. [15]

    Gorinova

    Dave Moore and Maria I. Gorinova. 2018. Effect Handling for Com- posable Program Transformations in Edward2. https://arxiv.org/abs/ 1811.06150

    2018 arXiv

  8. [16]

    Minh Nguyen, Roly Perera, Meng Wang, and Steven Ramsay. 2023. Effect Handlers for Programmable Inference. In Symp. on Haskell (Haskell’23). 44–58. doi:10.1145/3609026.3609729

    2023

  9. [17]

    OpenAI. 2025. openai/openai-agents-python: A lightweight, powerful framework for multi-agent workflows. Available on https://github. com/openai/openai-agents-python

    2025

  10. [18]

    OpenAI. 2025. openai/openai-python: The official Python library for the OpenAI API. Available on https://github.com/openai/openai- python

    2025

  11. [19]

    Plotkin and John Power

    Gordon D. Plotkin and John Power. 2001. Adequacy for Algebraic Effects. In Foundations of Software Science and Computation Structures (FoSSaCS’01). 1–24. doi:10.1007/3-540-45315-6_1

    2001 doi

  12. [20]

    Plotkin and Matija Pretnar

    Gordon D. Plotkin and Matija Pretnar. 2009. Handlers of Algebraic Effects. In European Symp. on Programming (ESOP’09) . 80–94. doi:10. 1007/978-3-642-00590-9_7

    2009

  13. [21]

    Matija Pretnar. 2015. An Introduction to Algebraic Effects and Handlers (Invited tutorial paper). Electr. Notes Theor. Comp. Sci. 319 (December 2015), 19–35. doi: 10.1016/j.entcs.2015.12.003 The 31st Conference on the Mathematical Foundations of Programming Semantics (MFPS XXXI)

    2015 doi

  14. [22]

    Qwen Team. 2025. Qwen2.5 Technical Report. doi:10.48550/arXiv.2412. 15115

    2025 doi

  15. [23]

    Adam Ścibior and Ohad Kammar. 2015. Effects in Bayesian inference. In Workshop on Higher-Order Programming with Effects (HOPE’15)

    2015

  16. [24]

    The Guidance Contributors. 2025. guidance-ai/guidance: A guidance language for controlling large language models. Available on https: //github.com/guidance-ai/guidance

    2025

  17. [25]

    Hoffman, Rif A

    Dustin Tran, Matthew D. Hoffman, Rif A. Saurous, Eugene Brevdo, Kevin Murphy, and David M. Blei. 2017. Deep Probabilistic Program- ming. In Int. Conf. on Learning Representations (ICLR’17)

    2017

  18. [26]

    Griffiths, Yuan Cao, and Karthik Narasimhan

    Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L. Griffiths, Yuan Cao, and Karthik Narasimhan. 2023. Tree of Thoughts: Delib- erate Problem Solving with Large Language Models. In Neural Info. Processing Syst. (NeurIPS’23). 11809–11822. https://dl.acm.org/doi/abs/ 10....

    2023

  19. [27]

    Grif- fiths, Yuan Cao, and Karthik Narasimhan

    Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L. Grif- fiths, Yuan Cao, and Karthik Narasimhan. 2025. princeton-nlp/tree-of- thoughts-llm: [NeurIPS 2023] Tree of Thoughts: Deliberate Problem Solving with Large Language Models. Available onhttps://github.com/ princet...

    2025

  20. [28]

    Stephen Zhao, Rob Brekelmans, Alireza Makhzani, and Roger Grosse

  21. [29]

    Gonzalez, Clark Barrett, and Ying Sheng

    Lianmin Zheng, Liangsheng Yin, Zhiqiang Xie, Chuyue Sun, Jeff Huang, Cody Hao Yu, Shiyi Cao, Christos Kozyrakis, Ion Stoica, Joseph E. Gonzalez, Clark Barrett, and Ying Sheng. 2024. SGLang: Efficient Execution of Structured Language Model Programs. https: //arxiv.org/abs/2312....

    2024 arXiv

  22. [2024]

    Probabilistic Inference in Language Models via Twisted Se- quential Monte Carlo. In Int. Conf. on Machine Learning (ICML’24) . 60704–60748. https://dl.acm.org/doi/10.5555/3692070.3694582

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.