LACUNA: Safe Agents as Recursive Program Holes

Cao Nguyen Pham; Frank Zhengqing Wu; Martin Odersky; Oliver Bra\v{c}evac; Yaoyu Zhao; Yichen Xu

LACUNA lets LLM agents fill typed program holes with code that shapes runtime while whole-action type checks enforce safety.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.3

2026-06-29 11:41 UTC pith:FGL563RZ

load-bearing objection LACUNA frames agent actions as typed recursive holes filled by the LLM and checked before running, which is a clean new primitive, but the safety argument hinges on type-checking being enough to block runtime failures. the 2 major comments →

arxiv 2605.28617 v1 pith:FGL563RZ submitted 2026-05-27 cs.AI cs.PL

LACUNA: Safe Agents as Recursive Program Holes

Yaoyu Zhao , Yichen Xu , Oliver Bra\v{c}evac , Cao Nguyen Pham , Frank Zhengqing Wu , Martin Odersky This is my paper

classification cs.AI cs.PL

keywords LLM agentssafe code generationprogram holestype checkingagent programming modelReAct loopsruntime safety

verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper introduces LACUNA as a programming model that removes the boundary between an agent's runtime loop and the code an LLM writes for each action. Each action appears in the source as a typed call agent[T](task); when execution reaches it the model supplies the body, which must type-check against the surrounding program before it runs. Because acceptance or rejection happens for the entire action at once, a rejected fill leaves the environment unchanged and its diagnostics trigger a retry. The same static check limits which tools and data the action may touch. This single primitive is shown to express ReAct loops, sub-agents, parallel decomposition and multi-model planning simply as ordinary control flow rather than special runtime machinery.

Core claim

LACUNA closes the split between runtime and model-written code while preserving safety; each action is accepted or rejected as a whole so a rejected one leaves the environment untouched, and the same check bounds tool and data use. The central mechanism is the typed call agent[T](task) that the LLM fills when execution reaches it; the filled code is type-checked against the enclosing program before any side effects occur.

What carries the argument

The agent[T](task) primitive: a typed call site that the LLM fills with code, which is statically type-checked against the surrounding program before execution.

Load-bearing premise

Static type-checking of the LLM-filled code is sufficient to prevent runtime safety failures such as prompt injection, wrong tool calls, or inconsistent state.

What would settle it

An accepted code fill that still produces a prompt injection, an unauthorized tool call, or an inconsistent environment state at runtime.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

ReAct-style loops, sub-agents, and skills become ordinary control-flow constructs rather than runtime primitives.
Parallel decomposition and multi-model planning are expressed directly in the source language.
Rejections occur before any environment change, so retries cost only the generation step.
The same type discipline limits data and tool exposure for every action.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

If the type system is extended with effect or capability annotations, the same hole mechanism could enforce finer-grained resource policies without new runtime checks.
The approach could be applied to non-agent code-generation settings where partial programs must be safely inserted into larger typed contexts.
Empirical work could measure whether the observed 0.7 retries per query remains stable as task complexity grows.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Desk Editor's Note

LACUNA frames agent actions as typed recursive holes filled by the LLM and checked before running, which is a clean new primitive, but the safety argument hinges on type-checking being enough to block runtime failures.

read the letter

LACUNA's core move is to replace the usual split between the agent's runtime loop and the code an LLM writes with a single primitive: agent[T](task). When execution hits one of these, the model supplies code that must type-check against the surrounding program; only then does it run. Rejected actions are dropped without side effects and the diagnostics feed a retry. The paper shows this can express ReAct-style loops, sub-agents, parallel decomposition, and multi-model planning as ordinary control flow rather than special-cased tool calls.

That framing is the actual novelty. It turns the model-written fragment into a first-class part of the program's structure while keeping the whole-action accept/reject boundary. The evaluation numbers are modest but concrete: 8.6% rejections on BrowseComp-Plus with 0.7 retries per query, and 76% task solve rate on tau2-bench. Those figures at least show the mechanism is usable.

The soft spot is exactly the one the stress-test flags. Static type-checking can bound which tools and data are visible, but it does not obviously stop a prompt-injection string that stays inside the allowed types, nor does it prevent code that type-checks yet fails partway through and leaves inconsistent state. The abstract does not describe an effect system, totality checker, or runtime monitor that would close those gaps, so the safety claim rests on an assumption that has not yet been shown. The benchmark results do not include error analysis or strong baselines that would let a reader judge how often those failure modes actually occur.

The work is aimed at people designing the next layer of agent runtimes and programming models. It is coherent enough on its own terms to deserve a serious referee, even if the safety argument will need more evidence in revision.

Referee Report

2 major / 0 minor

Summary. The paper introduces LACUNA, a programming model in which LLM agents execute actions via the primitive agent[T](task). The LLM fills each typed hole with code that is statically type-checked against the surrounding program before any execution occurs. Whole-action accept/reject semantics ensure that a rejected action leaves the environment unchanged, while the same check is claimed to bound tool use, data flow, and control-flow effects. The model is shown to express ReAct loops, sub-agents, parallel decomposition, and multi-model planning as ordinary control flow. Empirical results are given on BrowseComp-Plus (8.6% generations rejected, 0.7 retries per query, 27.1% accuracy) and τ²-bench (76.0% solve rate across 392 tasks, on par with a baseline agent).

Significance. If the type-checking discipline and whole-action rejection actually deliver the claimed safety invariants, LACUNA would provide a concrete route to letting model-generated code shape runtime control flow without enlarging the attack surface. The ability to encode standard agent idioms inside ordinary typed programs is a clear engineering contribution, and the reported rejection overhead is low enough to be practically relevant. The benchmark numbers supply a falsifiable starting point for further measurement.

major comments (2)

[Abstract] Abstract (safety-problems paragraph): the claim that type-checking the filled hole plus whole-action rejection suffices to block prompt injection, wrong tool calls, and inconsistent state from partial execution is load-bearing for the central safety argument, yet the manuscript supplies no description of an effect system, totality guarantee, or runtime monitor that would close the gap between static bounds on tools/data and dynamic failures that remain inside the allowed types.
[Abstract] Abstract (evaluation paragraph): the reported 27.1% accuracy and 76.0% solve rate are presented without error analysis, per-task rejection counts, or a breakdown of which failures were prevented by the type check versus which were merely retried; this makes it impossible to assess whether the safety mechanism contributes measurably to the observed performance.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the careful reading and constructive major comments. We address each point below and indicate planned revisions where appropriate.

read point-by-point responses

Referee: [Abstract] Abstract (safety-problems paragraph): the claim that type-checking the filled hole plus whole-action rejection suffices to block prompt injection, wrong tool calls, and inconsistent state from partial execution is load-bearing for the central safety argument, yet the manuscript supplies no description of an effect system, totality guarantee, or runtime monitor that would close the gap between static bounds on tools/data and dynamic failures that remain inside the allowed types.

Authors: The safety invariants follow directly from the type discipline: LLM-generated code is only executed if it type-checks against the surrounding program, which statically restricts the tools, data, and control-flow constructs that may appear. Whole-action rejection ensures that any code failing the check is discarded before it can affect the environment, so partial execution cannot occur. Prompt injection succeeds only if the injected content produces type-correct code; otherwise it is rejected. The manuscript does not introduce an effect system or runtime monitor beyond the type checker and the accept/reject semantics of the primitive; the type system itself is the mechanism that bounds behavior. We will revise the abstract to state these invariants more precisely and avoid any implication of additional dynamic enforcement. revision: partial
Referee: [Abstract] Abstract (evaluation paragraph): the reported 27.1% accuracy and 76.0% solve rate are presented without error analysis, per-task rejection counts, or a breakdown of which failures were prevented by the type check versus which were merely retried; this makes it impossible to assess whether the safety mechanism contributes measurably to the observed performance.

Authors: We agree that the current evaluation lacks the requested granularity. The revised manuscript will add an error-analysis subsection that reports per-task rejection rates, classifies failures (type errors caught before execution versus semantic or runtime errors after acceptance), and quantifies how many queries were resolved only after a type-check-driven retry. This will allow readers to evaluate the practical impact of the safety mechanism on the reported accuracy and solve rates. revision: yes

Circularity Check

0 steps flagged

No circularity: design claims rest on type system and benchmark results, not self-referential reductions

full rationale

The paper introduces a typed agent primitive and evaluates it on BrowseComp-Plus and τ²-bench. No equations, fitted parameters, or predictions appear; safety and expressiveness claims are justified by the static type-checking mechanism and whole-action acceptance/rejection, while performance numbers are direct empirical measurements. No self-citations are load-bearing for the central argument, and the derivation chain contains no self-definitional steps or ansatzes smuggled via prior work. The system is therefore self-contained against external benchmarks.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 1 invented entities

The central claim rests on the effectiveness of type-checking for bounding LLM-generated code behavior; the paper introduces one new primitive but does not list explicit free parameters or additional axioms beyond standard type-system assumptions.

axioms (1)

domain assumption Type checking of LLM-filled code suffices to bound tool use, data flow, and prevent inconsistent states when code shapes runtime control flow.
Invoked in the safety argument of the abstract.

invented entities (1)

agent[T](task) primitive no independent evidence
purpose: Typed hole that the LLM fills with code to shape the agent runtime.
New construct introduced to close the runtime-model split.

pith-pipeline@v0.9.1-grok · 5865 in / 1342 out tokens · 30034 ms · 2026-06-29T11:41:37.719573+00:00 · methodology

0 comments

read the original abstract

LLM agents increasingly act by writing code, yet a split persists between the runtime that drives the agent and the code the model writes. The runtime owns the loop, context, and control flow, and the model has little say over any of them. Letting model-written code shape the runtime itself would make agents more expressive, but it would also sharpen safety problems. A model can be diverted by a prompt injection, call the wrong tool, or fail partway and leave an inconsistent state, and each such failure reaches further when the code shapes the runtime than when it expresses a single action. We present LACUNA, a programming model for agents that closes this split while preserving safety. Each agent action is a typed call $\texttt{agent[T](task)}$ that the LLM fills with code when execution reaches it, and the code is type-checked against the surrounding program before it runs. Because each action is accepted or rejected as a whole, a rejected one leaves the environment untouched, and its compiler diagnostics drive a retry. The same check also bounds which tools and data an action may use and how they flow. Our primitive expresses ReAct loops, sub-agents, skills, parallel decomposition, and multi-model planning as ordinary control flow. We evaluate LACUNA on a collection of test cases, BrowseComp-Plus, and $\tau^2$-bench. On BrowseComp-Plus, $8.6\%$ of generations are rejected before execution, with 0.7 retries per query on average, and the agent reaches $27.1\%$ accuracy. On $\tau^2$-bench, LACUNA solves $76.0\%$ of $392$ tasks across four domains with a capable model, on par with the baseline agent.

discussion (0)

Reference graph

Works this paper leans on

3 extracted references · 3 canonical work pages · 2 internal anchors

[1]

The Attacker Moves Second: Stronger Adaptive Attacks Bypass Defenses Against Llm Jailbreaks and Prompt Injections

System FR: formalized foundations for the stainless verifier.Proc. ACM Program. Lang., 3(OOPSLA):166:1–166:30. Omar Khattab, Arnav Singhvi, Paridhi Maheshwari, Zhiyuan Zhang, Keshav Santhanam, Sri Vard- hamanan, Saiful Haq, Ashutosh Sharma, Thomas T. Joshi, Hanna Moazam, Heather Miller, Matei Za- haria, and Christopher Potts. 2024. DSPy: Com- piling decla...

work page internal anchor Pith review Pith/arXiv arXiv 2024
[2]

MCP Safety Audit: LLMs with the Model Context Protocol Allow Major Security Exploits

Accessed: 2026-05-25. Brandon Radosevich and John Halloran. 2025. MCP safety audit: LLMs with the Model Context Protocol allow major security exploits.CoRR, abs/2504.03767. Patrick Maxim Rondon, Ming Kawaguchi, and Ranjit Jhala. 2008. Liquid types. InProceedings of the ACM SIGPLAN 2008 Conference on Programming Language Design and Implementation, Tucson, ...

work page Pith review arXiv 2026
[3]

Recursive Language Models

smolagents: A barebones library for agents that think in code. https://github.com/ huggingface/smolagents. Accessed: 2026-05-15. Scala. 2024a. Scala 3: Capture checker. Source: https: //github.com/scala/scala3. Accessed: 2026-05- 25. Scala. 2024b. Scala 3: Explicit nulls. Source: https: //github.com/scala/scala3. Accessed: 2026-05- 25. Scala. 2024c. Scala...

work page internal anchor Pith review Pith/arXiv arXiv 2026

[1] [1]

The Attacker Moves Second: Stronger Adaptive Attacks Bypass Defenses Against Llm Jailbreaks and Prompt Injections

System FR: formalized foundations for the stainless verifier.Proc. ACM Program. Lang., 3(OOPSLA):166:1–166:30. Omar Khattab, Arnav Singhvi, Paridhi Maheshwari, Zhiyuan Zhang, Keshav Santhanam, Sri Vard- hamanan, Saiful Haq, Ashutosh Sharma, Thomas T. Joshi, Hanna Moazam, Heather Miller, Matei Za- haria, and Christopher Potts. 2024. DSPy: Com- piling decla...

work page internal anchor Pith review Pith/arXiv arXiv 2024

[2] [2]

MCP Safety Audit: LLMs with the Model Context Protocol Allow Major Security Exploits

Accessed: 2026-05-25. Brandon Radosevich and John Halloran. 2025. MCP safety audit: LLMs with the Model Context Protocol allow major security exploits.CoRR, abs/2504.03767. Patrick Maxim Rondon, Ming Kawaguchi, and Ranjit Jhala. 2008. Liquid types. InProceedings of the ACM SIGPLAN 2008 Conference on Programming Language Design and Implementation, Tucson, ...

work page Pith review arXiv 2026

[3] [3]

Recursive Language Models

smolagents: A barebones library for agents that think in code. https://github.com/ huggingface/smolagents. Accessed: 2026-05-15. Scala. 2024a. Scala 3: Capture checker. Source: https: //github.com/scala/scala3. Accessed: 2026-05- 25. Scala. 2024b. Scala 3: Explicit nulls. Source: https: //github.com/scala/scala3. Accessed: 2026-05- 25. Scala. 2024c. Scala...

work page internal anchor Pith review Pith/arXiv arXiv 2026