ASPERA: A Simulated Environment to Evaluate Planning for Complex Action Execution

Alexandru Coca; Anders Johannsen; Bo-Hsiang Tseng; Diarmuid \'O S\'eaghdha; H\'ector Martinez Alonso; Jianpeng Cheng; Mark Gaynor; Pete Boothroyd; Zhenxing Zhang

REVIEW 3 major objections 4 minor 14 references

ASPERA: A Simulated Environment to Evaluate Planning for Complex Action Execution

T0 review · 3 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that composing programs from a custom assistant library's primitives is a separate, harder challenge for LLMs than standard code generation, and measures that gap with a 250-task executable benchmark.

desk verdict A genuinely useful benchmark for LLM agents, but the LLM-generated evaluators need a validity audit before the headline numbers carry weight. read the letter →

arxiv 2507.15501 v1 pith:D5UYDJR6 submitted 2025-07-21 cs.CL cs.AIcs.LG

Alexandru Coca , Mark Gaynor , Zhenxing Zhang , Jianpeng Cheng , Bo-Hsiang Tseng , Pete Boothroyd , Héctor Martinez Alonso , Diarmuid Ó Séaghdha

show 1 more author

Anders Johannsen

This is my paper

classification cs.CLcs.AIcs.LG

keywords ASPERAAsper-BenchcomplexactionexecutionprogramsynthesisLLMevaluationdigitalassistantsprimitiveselectionfunctionalcorrectness

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

ASPERA constructs a simulated assistant library and couples it to a human-assisted LLM data-generation loop, so digital-assistant requests can be scored by executing the assistant's program and checking whether the user's goal was met. The paper's central finding is that composing an action-execution program from a custom, project-specific library is a distinct and significant challenge for LLMs, even when the full documentation is supplied in the prompt. On the 250-task Asper-Bench set, o1-preview completes 80.13% of tasks while GPT-4o completes 45.33%, despite both performing well on standard code benchmarks. When the model must first read module documentation and issue imports to choose primitives before writing the program, o1-preview's success falls to 28.4%, exposing primitive selection as an additional bottleneck for practical assistants.

What carries the argument

The central object is the three-program task representation: an action execution program (AEP) that satisfies the user query, a state initialisation program (SIP) that builds the database state the query presupposes, and an evaluation program (EP) that runs the SIP, runs the agent's AEP, and asserts that the user goal was achieved without side effects. The EP is the load-bearing piece because it turns 'did the assistant satisfy the user?' into executable assertions, including for information-seeking queries and queries with multiple allowed outcomes. Task success is then defined operationally: the generated AEP runs without error and passes all assertions in the reference EP. A second mechanism, the primitive-selection loop, iterates over module documentation and asks the model to issue import statements before generating the AEP, isolating the cost of not knowing which primitives exist.

What would settle it

Run independent programmers on a sample of Asper-Bench tasks and have them write alternative correct solutions that reach the user goal through different primitive combinations, orderings, or branch structures, then execute those alternatives under the reference evaluation programs; if many correct alternatives fail assertions, the reported task-success rates overstate difficulty.

Watch

Extended reading notes

Core claim

The paper claims that the bottleneck for LLM-powered digital assistants is not writing syntactically correct code but composing the right primitives into a multi-step program that satisfies a user's goal under a custom library's type and documentation constraints. To test this, the authors create Asper-Bench, 250 tasks grounded in a simulated corporate calendar and room-booking assistant, and run a range of proprietary and open-weight models in a setting where the full library documentation is in the prompt. The best model, o1-preview, succeeds on 80.13% of tasks; GPT-4o succeeds on 45.33%; and when the model is forced to select primitives from module documentation before generation, o1-preview's success drops to 28.4%. The error analysis shows that stronger models fail in a distinct pattern: their programs run, but take the wrong action or hand control back to the user, which are unrecoverable in a single trial. The paper's conclusion is that program generation grounded in custom assistant libraries is a significant, under-measured challenge compared to dependency-free code generation.

Load-bearing premise

The benchmark's difficulty numbers rest on the assumption that the evaluation programs, after human editing, accept every genuinely correct solution to a task and reject every genuinely wrong one.

Editorial extensions

If this is right

Benchmarking assistants against full library documentation is now feasible: any agent can be scored by whether its generated program executes and passes the reference evaluation program.
Task success on Asper-Bench separates models that merely generate plausible programs from those that satisfy constraints: o1-preview reaches 80.13% while GPT-4o reaches 45.33%.
Primitive selection is a distinct challenge: forcing o1-preview to select imports from module documentation before generation drops its success from 80.13% to 28.4%, so practical assistants must solve retrieval before composition.
Error profiles shift with capability: stronger models produce more task-completion and handback-control errors, which are irreversible in a single trial, so progress markers should watch those categories rather than execution errors alone.
The ASPERA engine can generate new tasks with executable evaluation programs in new domains, reducing the human cost of building evaluation data.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

If evaluation-program generation generalises, the same AEP/SIP/EP pattern could be applied to other grounded-action domains where capabilities are defined as libraries, making functional correctness measurement reusable beyond calendars.
The primitive-selection bottleneck suggests a concrete architectural prediction: an assistant that adds an explicit retrieval or exploration loop over library documentation before planning should close much of the gap between the full-documentation and selection settings.
A natural stress test of the benchmark is adversarial EP generation: deliberately crafting correct solutions with different primitive combinations, orderings, or branch structures and checking whether the assertions reject them.
Because query diversity is LLM-generated under focus instructions, scaling ASPERA to other domains will likely depend on how much new variety human focus instructions can inject, and measuring that marginal diversity would be a direct extension.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Dear colleague,

This is a solid benchmark paper with a real contribution: ASPERA lets an LLM and a developer co-generate not just queries and reference programs but also state-initialization and evaluation programs, producing a dynamic per-query environment. That genuinely moves past the templated queries in WorkBench and the hand-written checkers in AppWorld. The resulting Asper-Bench tasks look hard, the error analysis is careful, and they ship the code and data. If you work on LLM agents or code generation, this is a dataset you'll want to know about.

The headline numbers—o1 at 80%, GPT-4o at 45%—are plausible in direction, but they rest on a measurement premise that needs checking. The EPs are generated by an LLM prompted with the reference AEP and SIP, then human-supervised. That supervision verifies the EP passes the reference program and runs, but it doesn't verify the EP accepts all valid alternative solutions. If some EPs encode a particular composition order or assumption, they can reject semantically correct AEPs and understate success rates, and since o1 writes longer, more compositional programs than GPT-4o, the gap between models could be distorted. The paper does show some awareness of this—the repetition_schedule tool and the multi-outcome assertions in Appendix A.5 are aimed at exactly this problem—so it's not a fatal flaw, but the authors need to audit EPs with independent correct solutions before the numbers get cited as fact.

Two smaller issues. First, the abstract says custom-library generation is a challenge 'compared to dependency-free code generation,' but there's no controlled comparison under the same prompting protocol; they cite standard benchmarks as context. That claim should be softened or tested directly. Second, variance is underreported. Proprietary models are averaged over three runs but there are no error bars, so we don't know how stable the 80 vs 45 gap is.

Bottom line: this deserves a serious referee. The benchmark is useful, the framework is novel in a concrete way, and the limitations section is unusually honest. I'd ask for the EP audit and the variance reporting, but the core idea and dataset are worth publishing. I'd bring it to a reading group on LLM agents and would cite it in that context.

Recommendation: send to peer review, with the EP validity and controlled-comparison requests as major comments.

Referee Report

3 major / 4 minor

Summary. This paper presents ASPERA, an environment and data-generation pipeline for evaluating LLM-based digital assistants on complex action execution. A developer and an LLM interact to produce, for each task, a user query, a reference action execution program (AEP), a state initialisation program (SIP), and an evaluation program (EP) that runs the candidate AEP in the simulated environment and asserts goal satisfaction and absence of side effects. The authors release Asper-Bench, 250 tasks built with GPT-4o, and evaluate 13 LLMs under a complete-knowledge (CCK) setting and a primitive-selection (PS) setting, reporting task success rates from 80.13% (o1-preview) down to 2.40% (CodeGemma). They conclude that program generation grounded in custom assistant libraries is a significant challenge compared with dependency-free code generation and that primitive selection is a bottleneck.

Significance. If the evaluation is trustworthy, ASPERA is a useful addition to the agent and code-generation benchmark landscape: it targets execution-based evaluation of complex action execution, uses dynamic state initialisation rather than hand-authored template states, and ships explicit evaluation programs, a detailed error taxonomy, and a public code and data release. The paper also has strengths that are easy to undervalue: the task definitions are explicit programs rather than fitted model parameters, and the evaluation is execution-based rather than LLM-judged. The main caveat is that the empirical claims are only as strong as the EPs' accept/reject decisions, and the paper does not currently validate those decisions against independently written solutions. The headline comparison to dependency-free code generation is also not directly supported by a controlled experiment.

major comments (3)

[§2.3.3 and App. A.4] The central measurement premise is not directly validated. EPs are generated by an LLM prompted with the reference AEP and SIP (App. A.4) and are only human-supervised to the extent that they run and pass the reference AEP (§2.3.3). This establishes that the EP accepts the reference solution, but not that it accepts all semantically valid alternative programs (false rejections) or rejects all invalid programs (false acceptances). Because o1 produces longer, more compositional programs than GPT-4o (Table 5), any implementation-specific strictness in the assertions is likely to affect models differentially, so the headline gap (80.13% vs 45.33%) could be distorted. The mechanisms in App. A.5 (repetition_schedule, multiple-outcome assertions) address specific known cases, not the general problem. I ask for a validation study: sample or independently write valid alternative AEPs (different primitive composition, iteration order, date-time handling) and invalid ones, run them through the EPs, and report false-rejection and false-acceptance rates, ideally broken down by model family and program style.
[Abstract and §5] The claim that custom-library program generation is "a significant challenge to LLMs compared to dependency-free code generation" is not supported by a controlled comparison. Table 3 reports absolute task success on Asper-Bench, and Table 21 reports rank correlations with EvalPlus and BigCode, but neither establishes that the same models would perform better on dependency-free versions of comparable tasks. The comparison with standard benchmarks in §G.3 is qualitative. To keep the claim, add a matched baseline in which the same queries are implemented with an equivalent standard-library API (or a dependency-free control task set) and report per-model success; otherwise soften the abstract to "low absolute task success" or "challenging relative to typical code-generation benchmarks."
[§3 and §9] The human supervision described in §3 and the limitation stated in §9 together imply that data quality is currently concentrated in the lead author plus two annotators who "confirmed data quality while carrying out the error analysis." There is no independent annotation of EP correctness or inter-annotator agreement. Given that the EPs are the evaluation instruments, a second, independent human pass that specifically validates accept/reject decisions on a sample of agent outputs (not just reference AEPs) should be reported; this is needed even if the validation study in Comment 1 is added.

minor comments (4)

[Table 1 vs App. B] Table 1 reports 69 primitives (49 functions + 20 classes), while Appendix B, Table 10 states that the library defines 62 primitives; these counts should be reconciled.
[§7 and App. A.3] Section 7 contains the typo "WorkBebnch" (should be "WorkBench"), and the caption of Figure 10 in App. A.3 uses "replacted" where "replaced" is intended.
[Table 6] Table 6's columns "Programs debugged" and "Programs analysed" are not defined in the caption or surrounding text; please clarify what distinguishes these two subsets.
[Table 3] The o1 row in Table 3 has no entry in the syntax-error column; state explicitly whether syntax errors were not observed or were not recorded for that model.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the benchmark results are empirical measurements against fixed executable checks, with no fitted parameters or self-citation chain doing load-bearing work.

full rationale

The paper's central claim is an empirical benchmark result, not a quantity derived from its own inputs. Task success is measured by executing candidate AEPs against reference EPs in a simulated library, and no parameter is fitted to model outputs; likewise, no 'prediction' is computed from a quantity that already contains it. The reference AEPs, SIPs and EPs are human-supervised artifacts (§2.3.3, §3), and the paper explicitly acknowledges in §9 that supervision is required and that dataset bias remains a limitation. The fact that GPT-4o assisted in dataset construction is a data-generation bias, not a logical circularity: a benchmark built with one model can still evaluate other models, and even GPT-4o itself, against fixed executable checks. The paper also reports error analyses and correlations with external code benchmarks (Table 21), which provide independent grounding. The only substantive concern, whether LLM-generated EPs accept all and only valid programs, is an evaluation-validity and robustness issue, not a self-definitional reduction: the paper does not define 'task success' in terms of any fitted value, and it does not claim that the EPs are guaranteed to accept all semantically equivalent alternatives. No load-bearing self-citation appears; the citation to Cheng et al. (2020) is background context on intent APIs. Therefore the derivation chain is self-contained and no circular step is exhibited.

Assumptions & free parameters 0 free parameters · 4 assumptions · 2 invented entities

The central claim rests on the validity of the simulated environment and the evaluation programs, plus the hand-written policy assumptions that shape task difficulty. None of these are fitted to data; they are design choices. The main risk is that the evaluation programs may not be solution-agnostic.

assumptions (4)

domain assumption The assistant library simulation is a faithful implementation of the documented APIs.
All task success evaluation executes AEPs against this simulation; if the simulation deviates from its documentation, scores do not measure the intended capability. Appears in Sections 2.1 and 4.
domain assumption The evaluation programs (EPs) correctly identify all and only the solutions that satisfy the user goal.
EPs are generated by an LLM conditioned on the reference AEP and then human-supervised; they may reject valid alternative solutions (e.g., different but equivalent compositions). Appears in Section 2.3.3 and App. A.4.
ad hoc to paper Scheduling policy assumptions (weekends, work hours, unique employee names) are reasonable for a digital assistant benchmark and do not trivialize or distort difficulty.
These hand-written guidelines shape the dataset: for example, 'meetings should not be scheduled on weekends unless explicitly stated' adds a policy layer. They are design choices specific to ASPERA. Appears in Figure 8 and Figure 19b.
domain assumption Human supervision of generated data ensures quality.
The lead author edited programs; two annotators confirmed quality during error analysis. This is reported but not measured with inter-annotator agreement. Appears in Section 3, Quality control.

invented entities (2)

Simulated assistant library (Employee, Event, ConferenceRoom, TimeInterval, RepetitionSpec, etc.)
purpose: Provides the custom API surface against which AEPs are generated and executed.
These are simulation constructs for benchmarking, not claims about real-world entities. They have no falsifiable handle outside the paper.
ASPERA simulation and evaluation tools (simulate_org_structure, repetition_schedule, etc.)
purpose: Aid LLM generation of SIPs and EPs.
Developer-implemented tools embedded in the benchmark environment; no external evidence.

how reviews work

0 comments

Cite this review

Pith. "Pith review of ASPERA: A Simulated Environment to Evaluate Planning for Complex Action Execution." pith.science (2026). https://pith.science/paper/D5UYDJR6

@misc{pith2026250715501,
  author       = {Pith},
  title        = {Pith review of: ASPERA: A Simulated Environment to Evaluate Planning for Complex Action Execution},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/D5UYDJR6}},
  note         = {Machine review of arXiv:2507.15501}
}

read the original abstract

This work evaluates the potential of large language models (LLMs) to power digital assistants capable of complex action execution. These assistants rely on pre-trained programming knowledge to execute multi-step goals by composing objects and functions defined in assistant libraries into action execution programs. To achieve this, we develop ASPERA, a framework comprising an assistant library simulation and a human-assisted LLM data generation engine. Our engine allows developers to guide LLM generation of high-quality tasks consisting of complex user queries, simulation state and corresponding validation programs, tackling data availability and evaluation robustness challenges. Alongside the framework we release Asper-Bench, an evaluation dataset of 250 challenging tasks generated using ASPERA, which we use to show that program generation grounded in custom assistant libraries is a significant challenge to LLMs compared to dependency-free code generation.

Figures

Figures reproduced from arXiv: 2507.15501 by the authors.

**Figure 2.** Sample ASPERA task, depicting action execution (A), state initialisation (B) and evaluation (C) programs. [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗

**Figure 3.** Distributions of key complexity measures in the [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗

Figures from the paper (17 more)

**Figure 4.** Figure 4: Assistant error types for OpenAI and Gemini [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]

**Figure 5.** Figure 5: Task success as a function of reference AEP [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]

**Figure 6.** Figure 6: System turn. In the above the field code is replaced with the documentation of the assistant library and query_solution_examples is replaced with 5 AEP examples. See [PITH_FULL_IMAGE:figures/full_fig_p016_6.png]

**Figure 7.** Figure 7: User turn. To encourage diversity, we optionally include the history of the queries generated in the prompt, [PITH_FULL_IMAGE:figures/full_fig_p017_7.png]

**Figure 8.** Figure 8: Guidelines iterated over to populate {{instruction}} fields in the loop in [PITH_FULL_IMAGE:figures/full_fig_p017_8.png]

**Figure 9.** Figure 9: Prompt template used for AEP generation given a human-authored request. (Section [PITH_FULL_IMAGE:figures/full_fig_p018_9.png]

**Figure 10.** Figure 10: Prompt template used for runtime setup program generation (Figure [PITH_FULL_IMAGE:figures/full_fig_p020_10.png]

**Figure 11.** Figure 11: Prompt template used for evaluation program generation (Figure [PITH_FULL_IMAGE:figures/full_fig_p021_11.png]

**Figure 12.** Figure 12: Definition of RepetitionSpec, an object used for generating recurring event instances. Documentation omitted for brevity. Evaluation tools EP generation prompts include evaluation tools to support robust evaluation and access to environment state that is not possible…

**Figure 13.** Figure 13: Challenging queries from lines 3 -5 of Table [PITH_FULL_IMAGE:figures/full_fig_p024_13.png]

**Figure 15.** Figure 15: Asper-Bench AEP query length vs number of unique primitives. 10 20 30 40 50 Query length (words) 0 2 4 6 8 10 12 14 16 18 20 Cyclomatic complexity [PITH_FULL_IMAGE:figures/full_fig_p026_15.png]

**Figure 17.** Figure 17: Asper-Bench AEP length distribution [PITH_FULL_IMAGE:figures/full_fig_p026_17.png]

**Figure 18.** Figure 18: a. (a) LLM-generated policy for error handling and disambiguation. (b) RequiresUserInput documentation [PITH_FULL_IMAGE:figures/full_fig_p027_18.png]

**Figure 19.** Figure 19: ASPERA AEP generation prompt template. D.1 Primitive selection prompt You are a programmer using a Python library of personal assistant tools in order to write a program that executes a user query. You will be shown signatures from a Python module and a query, and wil…

**Figure 20.** Figure 20: Primitive selection prompt [PITH_FULL_IMAGE:figures/full_fig_p028_20.png]

**Figure 21.** Figure 21: Execution error classification for the first [PITH_FULL_IMAGE:figures/full_fig_p029_21.png]

**Figure 22.** Figure 22: Comparison of query length and action distribution between ToolAlpaca ( [PITH_FULL_IMAGE:figures/full_fig_p035_22.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

14 extracted references · 5 canonical work pages

[1]

Please limit yourself to generating programs involving complex combinations of the members of our codebase. It is not helpful to assume scenarios that our application cannot implement or assume unknown details about method implementations - focus on the interfaces and read our documentation carefully

work page
[2]

In Proceedings of the 2020 Confer- ence on Empirical Methods in Natural Language Processing, EMNLP 2020, Online, November 16-20, 2020, pages 8107–8117

Conversational semantic parsing for dialog state tracking. In Proceedings of the 2020 Confer- ence on Empirical Methods in Natural Language Processing, EMNLP 2020, Online, November 16-20, 2020, pages 8107–8117. Association for Computa- tional Linguistics. Xiang Deng, Yu Gu, Boyuan Zheng, Shijie Chen, Samual Stevens, Boshi Wang, Huan Sun, and Yu Su

work page 2020
[3]

```python

To reiterate, diversity (2) should not come at the expense of imagining scenarios our codebase cannot support (1). We will discuss how to improve our codebase in the future. ### Program structure guidelines ### The examples above follow {{ guidelines.generation_labelling | length }} structure guidelines listed below. Do the same, clearly stating when you ...

work page 2024
[4]

In Findings of the Association for Computational Linguistics: EMNLP 2020, Online Event, 16-20 November 2020, volume EMNLP 2020 of Findings of ACL, pages 1307–1323

Evaluating models’ local decision boundaries via contrast sets. In Findings of the Association for Computational Linguistics: EMNLP 2020, Online Event, 16-20 November 2020, volume EMNLP 2020 of Findings of ACL, pages 1307–1323. Association for Computational Linguistics. Zhicheng Guo, Sijie Cheng, Hao Wang, Shihao Liang, Yujia Qin, Peng Li, Zhiyuan Liu, Ma...

arXiv 2020
[9]

Appworld: A controllable world of apps and people for benchmarking interactive coding agents. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2024, Bangkok, Thailand, August 11-16, 2024, pages 16022–16076. Association for Computational Linguistics. Boshi Wang, Hao Fang, Jason Eisner, ...

arXiv 2024
[10]

OpenReview.net. Terry Yue Zhuo, Minh Chien Vu, Jenny Chim, Han Hu, Wenhao Yu, Ratnadira Widyasari, Imam Nur Bani Yusuf, Haolan Zhan, Junda He, Indraneil Paul, Simon Brunner, Chen Gong, Thong Hoang, Armel Randy Zebaze, Xiaoheng Hong, Wen-Ding Li, Jean Kad- dour, Ming Xu, Zhihan Zhang, Prateek Yadav, Na- man Jain, Alex Gu, Zhoujun Cheng, Jiawei Liu, Qian Li...

arXiv 2024
[12]

Focus on user requests that can be parsed to a fairly complex program implemented with the codebase above

Diversity is key. Focus on user requests that can be parsed to a fairly complex program implemented with the codebase above. Just put yourself in the shoes of the user wanting to get a lot done with our application. Some ways to achieve diversity may be: - imagine scenarios using for loops - imagine scenarios based on user conditions - imagine scenarios r...

work page
[14]

Consequently, Asper- Bench complements existing benchmarks by as- sessing program generation under custom depen- dencies

requires only standard library dependencies, whereas more general software capability bench- marks (Zhuo et al., 2024) assess program gener- ation based on widely used dependencies seen in training (e.g., numpy apis). Consequently, Asper- Bench complements existing benchmarks by as- sessing program generation under custom depen- dencies. While prior bench...

work page 2024

Show all 14 references

[2018]

In 2018 IEEE Conference on Com- puter Vision and Pattern Recognition, CVPR 2018, Salt Lake City, UT, USA, June 18-22, 2018 , pages 8494–8502

Virtualhome: Simulating household activities via programs. In 2018 IEEE Conference on Com- puter Vision and Pattern Recognition, CVPR 2018, Salt Lake City, UT, USA, June 18-22, 2018 , pages 8494–8502. Computer Vision Foundation / IEEE Computer Society. Yujia Qin, Shihao Liang,...

2018 arXiv
[2020]

Task-oriented dialogue as dataflow synthesis. Trans. Assoc. Comput. Linguistics, 8:556–571. Jacob Austin, Augustus Odena, Maxwell I. Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie J. Cai, Michael Terry, Quoc V . Le, and Charles Sutton. 2021a. Program ...

2024 arXiv
[2021]

CoRR, abs/2105.13231

Androidenv: A reinforcement learning plat- form for android. CoRR, abs/2105.13231. Harsh Trivedi, Tushar Khot, Mareike Hartmann, Ruskin Manku, Vinty Dong, Edward Li, Shashank Gupta, Ashish Sabharwal, and Niranjan Balasubramanian

arXiv
[2023]

Mind2web: Towards a generalist agent for the web. In Advances in Neural Information Processing Systems 36: Annual Conference on Neural Informa- tion Processing Systems 2023, NeurIPS 2023, New Orleans, LA, USA, December 10 - 16, 2023. Shengda Fan, Xin Cong, Yuepeng Fu, Zhong Zh...

2023 arXiv
[2024]

CoRR, abs/2403.05530

Gemini 1.5: Unlocking multimodal under- standing across millions of tokens of context. CoRR, abs/2403.05530. Yongliang Shen, Kaitao Song, Xu Tan, Wenqi Zhang, Kan Ren, Siyu Yuan, Weiming Lu, Dongsheng Li, and Yueting Zhuang. 2023. Taskbench: Bench- marking large language model...

2023 arXiv
[4976]

Harsh Jhamtani, Hao Fang, Patrick Xia, Eran Levy, Ja- cob Andreas, and Ben Van Durme

Association for Computational Linguistics. Harsh Jhamtani, Hao Fang, Patrick Xia, Eran Levy, Ja- cob Andreas, and Ben Van Durme. 2024. Natural language decomposition and interpretation of com- plex utterances. Yuhang Lai, Chengxi Li, Yiming Wang, Tianyi Zhang, Ruiqi Zhong, Luk...

2024 arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.

[1] [1]

Please limit yourself to generating programs involving complex combinations of the members of our codebase. It is not helpful to assume scenarios that our application cannot implement or assume unknown details about method implementations - focus on the interfaces and read our documentation carefully

work page

[2] [2]

In Proceedings of the 2020 Confer- ence on Empirical Methods in Natural Language Processing, EMNLP 2020, Online, November 16-20, 2020, pages 8107–8117

Conversational semantic parsing for dialog state tracking. In Proceedings of the 2020 Confer- ence on Empirical Methods in Natural Language Processing, EMNLP 2020, Online, November 16-20, 2020, pages 8107–8117. Association for Computa- tional Linguistics. Xiang Deng, Yu Gu, Boyuan Zheng, Shijie Chen, Samual Stevens, Boshi Wang, Huan Sun, and Yu Su

work page 2020

[3] [3]

```python

To reiterate, diversity (2) should not come at the expense of imagining scenarios our codebase cannot support (1). We will discuss how to improve our codebase in the future. ### Program structure guidelines ### The examples above follow {{ guidelines.generation_labelling | length }} structure guidelines listed below. Do the same, clearly stating when you ...

work page 2024

[4] [4]

In Findings of the Association for Computational Linguistics: EMNLP 2020, Online Event, 16-20 November 2020, volume EMNLP 2020 of Findings of ACL, pages 1307–1323

Evaluating models’ local decision boundaries via contrast sets. In Findings of the Association for Computational Linguistics: EMNLP 2020, Online Event, 16-20 November 2020, volume EMNLP 2020 of Findings of ACL, pages 1307–1323. Association for Computational Linguistics. Zhicheng Guo, Sijie Cheng, Hao Wang, Shihao Liang, Yujia Qin, Peng Li, Zhiyuan Liu, Ma...

arXiv 2020

[5] [9]

Appworld: A controllable world of apps and people for benchmarking interactive coding agents. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2024, Bangkok, Thailand, August 11-16, 2024, pages 16022–16076. Association for Computational Linguistics. Boshi Wang, Hao Fang, Jason Eisner, ...

arXiv 2024

[6] [10]

OpenReview.net. Terry Yue Zhuo, Minh Chien Vu, Jenny Chim, Han Hu, Wenhao Yu, Ratnadira Widyasari, Imam Nur Bani Yusuf, Haolan Zhan, Junda He, Indraneil Paul, Simon Brunner, Chen Gong, Thong Hoang, Armel Randy Zebaze, Xiaoheng Hong, Wen-Ding Li, Jean Kad- dour, Ming Xu, Zhihan Zhang, Prateek Yadav, Na- man Jain, Alex Gu, Zhoujun Cheng, Jiawei Liu, Qian Li...

arXiv 2024

[7] [12]

Focus on user requests that can be parsed to a fairly complex program implemented with the codebase above

Diversity is key. Focus on user requests that can be parsed to a fairly complex program implemented with the codebase above. Just put yourself in the shoes of the user wanting to get a lot done with our application. Some ways to achieve diversity may be: - imagine scenarios using for loops - imagine scenarios based on user conditions - imagine scenarios r...

work page

[8] [14]

Consequently, Asper- Bench complements existing benchmarks by as- sessing program generation under custom depen- dencies

requires only standard library dependencies, whereas more general software capability bench- marks (Zhuo et al., 2024) assess program gener- ation based on widely used dependencies seen in training (e.g., numpy apis). Consequently, Asper- Bench complements existing benchmarks by as- sessing program generation under custom depen- dencies. While prior bench...

work page 2024