Pith. sign in

REVIEW 4 major objections 5 minor 32 references

Running in CIRCLE? A Simple Benchmark for LLM Code Interpreter Security

T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A new 1,260-prompt benchmark shows that LLM code interpreters routinely execute denial-of-service attacks, and that harmless-sounding indirect prompts are the most reliable way to trigger them.

desk verdict Useful first benchmark for interpreter DoS, but the unvalidated judge LLM means the exact numbers shouldn't be trusted yet. read the letter →

arxiv 2507.19399 v1 pith:7RTYASI4 submitted 2025-07-25 cs.CR cs.AI

Gabriel Chua This is my paper
classification cs.CRcs.AI
keywords LLMsecuritycodeinterpreterbenchmarkresourceexhaustiondenial-of-serviceindirectpromptautomatedevaluationsafety
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that the code interpreters built into commercial LLMs expose a distinct, under-measured class of cybersecurity risk: denial-of-service through CPU, memory, and disk exhaustion. To measure it, it introduces CIRCLE, a benchmark of 1,260 prompts split into openly malicious ('direct') and disguised benign-sounding ('indirect') variants, and runs them against seven OpenAI and Google models. The headline finding is that defenses are shallow and inconsistent: even the most refusal-prone model, o4-mini, refuses only 7.1% of risky requests and fulfills 70.2%, while other models refuse less than 1%. The paper's central concern is that indirect, socially-engineered prompts reliably bypass model defenses, so interpreter safety needs dedicated benchmarking and guardrails rather than reliance on general instruction-following.

What carries the argument

The central object is the CIRCLE benchmark and its automated evaluation harness. CIRCLE is a set of 1,260 prompts — 60 direct prompts (20 per risk category) and 1,200 indirect prompts built from 60 indirect prompts combined with 20 social-engineering template prefixes — targeting three resources: CPU time, memory, and disk. The harness sends each prompt to a model with native code-interpreter access, actually executes the generated code inside the provider's interpreter environment, and then uses GPT-4.1 as a judge LLM to classify each run into one of six outcomes: refusal, reframe, follow-up, incorrect code, fulfilled, or timeout. What carries the argument is the direct/indirect prompt contrast combined with the six-way outcome taxonomy, which goes beyond a binary 'unsafe code generated?' check and lets the paper point at specific failure modes such as execution timeouts and silent simplifications.

What would settle it

Take a random sample of roughly 300 judge-labeled responses from the benchmark and have human annotators independently assign the six outcome categories, then measure agreement with GPT-4.1's labels; if disagreement exceeds a few percentage points, the reported refusal, fulfillment, and timeout rates could not be trusted.

Watch

Extended reading notes

Core claim

CIRCLE establishes that all seven tested commercial code interpreters will, when prompted, generate and execute code that exhausts CPU, memory, or disk, and that no tested model refuses a majority of risky requests. The most striking pattern is the direct–indirect split: indirect prompts that frame resource-intensive tasks as legitimate work (for example, 'calibrating CI autoscaling') sharply lower refusal rates and raise execution timeouts, which the paper interprets as socially-engineered prompts defeating safety training. The paper also documents strong provider-level differences in timeout enforcement, with Google's 30-second limit producing high timeout shares, and within-provider inconsistency, such as GPT-4.1 refusing 0.5% of risky requests while o4-mini refuses 7.1% yet fulfills 70.2%.

Load-bearing premise

The entire measurement rests on the unvalidated assumption that the GPT-4.1 judge correctly labels every model response into the six outcome categories, so any systematic mislabeling would change every headline rate, including the o4-mini versus GPT-4.1 refusal gap.

Editorial extensions

If this is right

  • If these results hold, providers' interpreter safety cannot be inferred from refusal rates alone; fulfillment and timeout rates must be reported together to reflect real-world risk.
  • Indirect prompts engineered to look like routine operational tasks should be treated as a first-class attack vector in every LLM safety evaluation, not a corner case.
  • Provider timeout policies materially change the measured vulnerability profile, so cross-model comparisons must control for execution limits or the comparison is confounded.
  • A model that refuses more often is not necessarily safer overall: o4-mini's high refusal rate coexists with the highest fulfillment rate, showing that refusal and unsafe execution are separable failure modes.
  • The six-category outcome scheme gives security teams a concrete checklist for monitoring interpreter logs, since 'fulfilled' and 'timeout' are the signals that a resource-exhaustion attack may be under way.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The judge-LLM design means every headline percentage inherits GPT-4.1's classification behavior; a human-labeled subset validation of the judge is the natural next step the paper does not take.
  • The 20 indirect templates are likely only a sample of possible social engineering; adversarial prompt synthesis could plausibly push fulfillment rates higher than the 70.2% observed for o4-mini.
  • Because the benchmark executes code in live provider sandboxes, it could be extended to test resource exhaustion across consecutive prompts, checking whether one fulfilled attack degrades the interpreter's state for later requests — a scenario the paper does not measure.
  • The paper positions its harness as supporting Anthropic and Mistral interpreters but evaluates only OpenAI and Google models, so the cross-provider inconsistency it reports may be only a partial picture.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces CIRCLE, a benchmark of 1,260 prompts targeting CPU, memory, and disk resource exhaustion in LLM code interpreters, with both explicitly malicious ('direct') and plausibly benign ('indirect') prompt variants. The authors evaluate seven commercial models from OpenAI and Google using an automated pipeline that executes generated code and then uses GPT-4.1 as a judge LLM to classify each interaction into one of six outcomes (refusal, reframe, follow-up, incorrect code, fulfilled, timeout). Headline findings include low refusal rates across models (o4-mini at 7.1% vs GPT-4.1 at 0.5%), high fulfillment rates under indirect prompts, and frequent timeouts for Gemini models (up to 65.1%). The benchmark dataset, code, and logs are released publicly.

Significance. If the measured rates are reliable, CIRCLE would fill a real gap: it is a lightweight, interpreter-specific benchmark for resource-exhaustion attacks triggered by single prompts, with fine-grained outcome categories and actual code execution. The public release of the dataset and evaluation harness is a concrete and reproducible contribution. The weakness is that the headline quantitative claims rest entirely on an unvalidated judge LLM and on inconsistent timeout definitions; the paper does not demonstrate label reliability, so the cross-model comparisons and the 'indirect prompts weaken defenses' conclusion are currently unsupported as stated. The benchmark's taxonomy and execution-in-the-loop design are nevertheless valuable, and the identified issues are addressable in revision.

major comments (4)
  1. [Section 3.2, Appendix A] The six-category outcome labels in Tables 1 and 2, including the o4-mini 7.1% vs GPT-4.1 0.5% refusal gap, are produced solely by GPT-4.1 with no reported human validation, inter-annotator agreement, or accuracy check. The boundaries between 'refusal', 'reframe', and 'follow-up', and between 'incorrect_code' and 'fulfilled', are subjective, and the judge sees only the textual transcript plus optional REPL output rather than an independent ground truth. Because GPT-4.1 is also one of the seven evaluated models, systematic judge bias could directly distort cross-model comparisons. Please report a human-annotated gold set (e.g., 100-200 samples) with agreement statistics, or otherwise validate the judge, before the headline percentages can be accepted.
  2. [Section 3.2] Phase 1 states 'A standardized timeout of 5 minutes per task evaluation is set', while the Phase 2 category definition defines 'timeout' as execution exceeding the '3-minute evaluation timeout'. This contradiction directly affects the reported timeout rates (e.g., 65.1% for Gemini 2.5 Pro in Table 1) and must be resolved by specifying which timeout was actually enforced and how it was measured. The discrepancy also undermines the reliability of the 'timeout' label that the judge is asked to assign.
  3. [Section 4.3] The chi-square test (χ2 = 3803.16) pools outcomes across models and prompt types without accounting for the non-independence of repeated prompts: the same 1,260 prompts are evaluated on all seven models, and indirect prompts are constructed from 60 base tasks × 20 templates. The McNemar tests are described without their pairing structure, so it is unclear what paired outcomes are being compared. The direct-vs-indirect comparison in Table 2 couples n=60 direct prompts with n=1200 indirect prompts per model, making the percentage differences difficult to interpret without exact sample sizes and confidence intervals. These statistical analyses should be replaced with per-model or cluster-robust tests that respect the data structure.
  4. [Section 3.2, Table 1] Timeout and fulfillment outcomes are partly determined by provider-specific interpreter environments, as the paper itself notes for Google's documented 30-second timeout; however, the results are then interpreted as model-level vulnerabilities. For example, the high timeout rate for Gemini 2.5 Pro (65.1%) may reflect provider timeout policy rather than model behavior. To support the claim of 'significant and inconsistent vulnerabilities' across providers, the analysis must either control for environment differences (timeout, memory cap, CPU limits) or clearly separate model behavior from infrastructure limits, at minimum by reporting the per-provider resource caps used in the evaluation.
minor comments (5)
  1. [Abstract and Section 3.1] The description of the corpus size is ambiguous: the text says the benchmark includes 1,260 unique prompts, but Table 2 reports results on 60 direct and 1,200 indirect prompts per model; please state explicitly whether 1,260 is the per-model prompt count and that a total of 8,820 model evaluations were run.
  2. [Abstract] The phrase 'correctly refuses risky requests at 7.1%' is not established by the evaluation, since refusal is one of several non-harmful outcomes and the judge's correctness is unvalidated; rephrase to avoid overclaiming.
  3. [Section 3.2] The claim that Google has a documented 30-second timeout is not backed by a specific citation with a query date; please provide the exact documentation reference.
  4. [Table 2] Several rows in Table 2 are run together in the text (e.g., 'Gemini 2.0 Flash Indirect ... 29.3Gemini 2.0 Flash Direct ...'); please use a properly formatted table with separate rows.
  5. [References] There are typographical errors in the references, including 'Anthropic documation' and missing punctuation before 'Rabin et al. (2025)' in Section 2; please proofread.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation found; the benchmark's headline rates are empirical measurements, and the judge-LLM overlap is a validity concern rather than a circularity.

full rationale

CIRCLE is an empirical benchmark rather than a derivation chain, so the circularity patterns targeted by this review (self-definitional equivalence, fitted inputs renamed as predictions, load-bearing self-citations, imported uniqueness theorems, ansatz smuggling, or renaming known results) do not apply. The headline statistics, such as the o4-mini 7.1% refusal rate and GPT-4.1 0.5% refusal rate, are measurements produced by executing prompts in sandboxed interpreter environments and then labeling transcripts with a judge LLM (GPT-4.1, Section 3.2). No equation or construction makes any reported rate equal to an input parameter by definition. The strongest related concern is that GPT-4.1 serves both as the judge and as one of the seven evaluated models, so its own outcome labels are self-judged, and no human validation or inter-annotator agreement is reported. This is a legitimate measurement-validity and potential-bias risk, but it is not circularity: the judge could in principle label its own responses either favorably or unfavorably, and the labels are not forced by construction. Similarly, the discrepancy between the 5-minute Phase 1 timeout and the 3-minute timeout mentioned in the Phase 2 category definition is an internal inconsistency, not a circular step. The paper contains no load-bearing self-citations that replace evidence, and its benchmark dataset and evaluation harness are released for external reproduction. Accordingly, no specific circular step can be quoted and reduced to an input by construction, and the appropriate finding is no significant circularity.

Assumptions & free parameters 4 free parameters · 3 assumptions · 0 invented entities

This is an empirical benchmark rather than a derivation, so there are no fitted parameters in the usual sense. However, design choices such as the judge model, timeout threshold, and template count function as free parameters that directly influence the reported outcome rates. The axioms reflect unvalidated assumptions about judge reliability, prompt realism, and cross-provider comparability.

free parameters (4)
  • Timeout threshold for evaluation = 5 minutes (stated) / 3 minutes (definition in Timeout category)
    The choice of timeout directly determines the Timeout outcome rate; the paper contradicts itself on whether it is 5 or 3 minutes (Section 3.2).
  • Judge LLM = gpt-4.1-2025-04-14
    The classification of all outcomes depends on this hand-picked model; its accuracy is not validated.
  • Number of indirect prompt templates = 20
    The 1,200 indirect prompts are generated from 60 base prompts x 20 templates; the number of templates inflates the prompt count.
  • Memory cap in interpreter environment = 1-2 GB (per Ethics section)
    The stated resource cap shapes whether memory exhaustion attempts trigger timeouts or failures.
assumptions (3)
  • domain assumption The judge LLM's outcome classifications are accurate.
    Section 3.2 and Appendix A rely on GPT-4.1 to label outcomes; no human agreement or accuracy check is reported.
  • domain assumption The 1,260 prompts are representative of real-world resource-exhaustion attacks on code interpreters.
    Section 3.1 states prompts were AI-generated and manually reviewed, but no external validation of realism is given.
  • domain assumption The tested interpreter environments are comparable across providers except for documented timeout differences.
    Section 3.2 notes undocumented timeout thresholds, yet results are aggregated as if timeout outcomes are equivalent.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Running in CIRCLE? A Simple Benchmark for LLM Code Interpreter Security." pith.science (2026). https://pith.science/paper/7RTYASI4

@misc{pith2026250719399,
  author       = {Pith},
  title        = {Pith review of: Running in CIRCLE? A Simple Benchmark for LLM Code Interpreter Security},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/7RTYASI4}},
  note         = {Machine review of arXiv:2507.19399}
}
read the original abstract

As large language models (LLMs) increasingly integrate native code interpreters, they enable powerful real-time execution capabilities, substantially expanding their utility. However, such integrations introduce potential system-level cybersecurity threats, fundamentally different from prompt-based vulnerabilities. To systematically evaluate these interpreter-specific risks, we propose CIRCLE (Code-Interpreter Resilience Check for LLM Exploits), a simple benchmark comprising 1,260 prompts targeting CPU, memory, and disk resource exhaustion. Each risk category includes explicitly malicious ("direct") and plausibly benign ("indirect") prompt variants. Our automated evaluation framework assesses not only whether LLMs refuse or generates risky code, but also executes the generated code within the interpreter environment to evaluate code correctness, simplifications made by the LLM to make the code safe, or execution timeouts. Evaluating 7 commercially available models from OpenAI and Google, we uncover significant and inconsistent vulnerabilities. For instance, evaluations show substantial disparities even within providers - OpenAI's o4-mini correctly refuses risky requests at 7.1%, notably higher rates compared to GPT-4.1 at 0.5%. Results particularly underscore that indirect, socially-engineered prompts substantially weaken model defenses. This highlights an urgent need for interpreter-specific cybersecurity benchmarks, dedicated mitigation tools (e.g., guardrails), and clear industry standards to guide safe and responsible deployment of LLM interpreter integrations. The benchmark dataset and evaluation code are publicly released to foster further research.

Figures

Figures reproduced from arXiv: 2507.19399 by the authors.

Figure 1
Figure 1. Overview of the CIRCLE benchmark evaluation workflow. Test prompts from three risk categories — CPU (time exhaustion), memory, and disk exhaustion—are sent to evaluated LLMs configured with native code interpreters. Models may refuse, reframe, request further details, or generate executable code. Outputs, including generated responses, code, and execution outcomes, are then systematically assessed by a dedicated jud… view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

32 extracted references · 11 canonical work pages

  1. [1]

    Jimenez, Farshad Khorrami, Prashanth Krishnamurthy, Brendan Dolan-Gavitt, Muhammad Shafique, Karthik Narasimhan, Ramesh Karri, and Ofir Press

    Talor Abramovich, Meet Udeshi, Minghao Shao, Kilian Lieret, Haoran Xi, Kimberly Milner, Sofija Jancheska, John Yang, Carlos E. Jimenez, Farshad Khorrami, Prashanth Krishnamurthy, Brendan Dolan-Gavitt, Muhammad Shafique, Karthik Narasimhan, Ramesh Karri, and Ofir Press. Enigma: Interactive tools substantially assist lm agents in finding security vulnerabil...

    arXiv 2025

  2. [2]

    Anthropic documation - code execution tool, 2025

    Anthropic. Anthropic documation - code execution tool, 2025. URL https://docs.anthropic.com/en/docs/agents-and-tools/tool-use/code-execution-tool. Accessed: 2025-06-14

    work page 2025

  3. [3]

    Purple llama cyberseceval: A secure coding benchmark for language models, 2023

    Manish Bhatt, Sahana Chennabasappa, Cyrus Nikolaidis, Shengye Wan, Ivan Evtimov, Dominik Gabi, Daniel Song, Faizan Ahmad, Cornelius Aschermann, Lorenzo Fontana, Sasha Frolov, Ravi Prakash Giri, Dhaval Kapil, Yiannis Kozyrakis, David LeBlanc, James Milazzo, Aleksandar Straumann, Gabriel Synnaeve, Varun Vontimitta, Spencer Whitman, and Joshua Saxe. Purple l...

    arXiv 2023

  4. [4]

    Cyberseceval 2: A wide-ranging cybersecurity evaluation suite for large language models, 2024

    Manish Bhatt, Sahana Chennabasappa, Yue Li, Cyrus Nikolaidis, Daniel Song, Shengye Wan, Faizan Ahmad, Cornelius Aschermann, Yaohui Chen, Dhaval Kapil, David Molnar, Spencer Whitman, and Joshua Saxe. Cyberseceval 2: A wide-ranging cybersecurity evaluation suite for large language models, 2024. URL https://arxiv.org/abs/2404.13161

    arXiv 2024

  5. [5]

    Autoadvexbench: Benchmarking autonomous exploitation of adversarial example defenses, 2025

    Nicholas Carlini, Javier Rando, Edoardo Debenedetti, Milad Nasr, and Florian Tramèr. Autoadvexbench: Benchmarking autonomous exploitation of adversarial example defenses, 2025. URL https://arxiv.org/abs/2503.01811

    arXiv 2025

  6. [6]

    Airtbench: Measuring autonomous ai red teaming capabilities in language models, 2025

    Ads Dawson, Rob Mulla, Nick Landers, and Shane Caldwell. Airtbench: Measuring autonomous ai red teaming capabilities in language models, 2025. URL https://arxiv.org/abs/2506.14682

    arXiv 2025

  7. [7]

    Agentdojo: a dynamic environment to evaluate prompt injection attacks and defenses for llm agents

    Edoardo Debenedetti, Jie Zhang, Mislav Balunovic, Luca Beurer-Kellner, Marc Fischer, and Florian Tram\` e r. Agentdojo: a dynamic environment to evaluate prompt injection attacks and defenses for llm agents. In Proceedings of the 38th International Conference on Neural Information Processing Systems, NIPS '24, Red Hook, NY, USA, 2025. Curran Associates In...

    work page 2025

  8. [8]

    Gemini 2.0 deep dive: Code execution, 2025

    Google. Gemini 2.0 deep dive: Code execution, 2025. URL https://developers.googleblog.com/en/gemini-20-deep-dive-code-execution/. Accessed: 2025-06-14

    work page 2025

Show all 32 references
  1. [9]

    Redcode: Risky code execution and generation benchmark for code agents

    Chengquan Guo, Xun Liu, Chulin Xie, Andy Zhou, Yi Zeng, Zinan Lin, Dawn Song, and Bo Li. Redcode: Risky code execution and generation benchmark for code agents. In A. Globerson, L. Mackey, D. Belgrave, A. Fan, U. Paquet, J. Tomczak, and C. Zhang (eds.), Advances in Neural Info...

    2024

  2. [10]

    Data interpreter: An llm agent for data science, 2024

    Sirui Hong, Yizhang Lin, Bang Liu, Bangbang Liu, Binhao Wu, Ceyao Zhang, Chenxing Wei, Danyang Li, Jiaqi Chen, Jiayi Zhang, Jinlin Wang, Li Zhang, Lingyao Zhang, Min Yang, Mingchen Zhuge, Taicheng Guo, Tuo Zhou, Wei Tao, Xiangru Tang, Xiangtao Lu, Xiawu Zheng, Xinbing Liang, Y...

    2024 arXiv

  3. [11]

    SWE -bench: Can language models resolve real-world github issues? In The Twelfth International Conference on Learning Representations, 2024

    Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik R Narasimhan. SWE -bench: Can language models resolve real-world github issues? In The Twelfth International Conference on Learning Representations, 2024. URL https://openreview.net/f...

    2024

  4. [12]

    Mario: Math reasoning with code interpreter output -- a reproducible pipeline, 2024

    Minpeng Liao, Wei Luo, Chengxi Li, Jing Wu, and Kai Fan. Mario: Math reasoning with code interpreter output -- a reproducible pipeline, 2024. URL https://arxiv.org/abs/2401.08190

    2024 arXiv

  5. [13]

    Demystifying rce vulnerabilities in llm-integrated apps

    Tong Liu, Zizhuang Deng, Guozhu Meng, Yuekang Li, and Kai Chen. Demystifying rce vulnerabilities in llm-integrated apps. In Proceedings of the 2024 on ACM SIGSAC Conference on Computer and Communications Security, CCS '24, pp.\ 1716–1730, New York, NY, USA, 2024. Association f...

    2024

  6. [14]

    Agentbench: Evaluating llms as agents

    Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu, Xuanyu Lei, Hanyu Lai, Yu Gu, Hangliang Ding, Kaiwen Men, Kejuan Yang, Shudan Zhang, Xiang Deng, Aohan Zeng, Zhengxiao Du, Chenhui Zhang, Sheng Shen, Tianjun Zhang, Yu Su, Huan Sun, Minlie Huang, Yuxiao Dong, and Jie Tang. Agentbench:...

    2023 arXiv

  7. [15]

    Build ai agents with the mistral agents api, 2025

    Mistral. Build ai agents with the mistral agents api, 2025. URL https://mistral.ai/news/agents-api. Accessed: 2025-06-14

    2025

  8. [16]

    Rossi, Handong Zhao, Ruiyi Zhang, Puneet Mathur, Nedim Lipka, Yu Wang, Trung Bui, Franck Dernoncourt, and Tianyi Zhou

    Dang Nguyen, Viet Dac Lai, Seunghyun Yoon, Ryan A. Rossi, Handong Zhao, Ruiyi Zhang, Puneet Mathur, Nedim Lipka, Yu Wang, Trung Bui, Franck Dernoncourt, and Tianyi Zhou. Dynasaur: Large language agents beyond predefined actions, 2025. URL https://arxiv.org/abs/2411.01747

    2025 arXiv

  9. [17]

    New tools and features in the responses api, 2025 a

    OpenAI. New tools and features in the responses api, 2025 a . URL https://openai.com/index/new-tools-and-features-in-the-responses-api/. Accessed: 2025-06-14

    2025

  10. [18]

    Introducing gpt-4.1 in the api, 2025 b

    OpenAI. Introducing gpt-4.1 in the api, 2025 b . URL https://openai.com/index/gpt-4-1/. Accessed: 2025-06-14

    2025

  11. [19]

    o3-pro, 2025 c

    OpenAI. o3-pro, 2025 c . URL https://platform.openai.com/docs/models/o3-pro. Accessed: 2025-06-14

    2025

  12. [20]

    Sandboxeval: Towards securing test environment for untrusted code, 2025

    Rafiqul Rabin, Jesse Hostetler, Sean McGregor, Brett Weir, and Nick Judd. Sandboxeval: Towards securing test environment for untrusted code, 2025. URL https://arxiv.org/abs/2504.00018

    2025 arXiv

  13. [21]

    Nyu ctf bench: A scalable open-source benchmark dataset for evaluating llms in offensive security, 2025

    Minghao Shao, Sofija Jancheska, Meet Udeshi, Brendan Dolan-Gavitt, Haoran Xi, Kimberly Milner, Boyuan Chen, Max Yin, Siddharth Garg, Prashanth Krishnamurthy, Farshad Khorrami, Ramesh Karri, and Muhammad Shafique. Nyu ctf bench: A scalable open-source benchmark dataset for eval...

    2025 arXiv

  14. [22]

    Executable code actions elicit better llm agents

    Xingyao Wang, Yangyi Chen, Lifan Yuan, Yizhe Zhang, Yunzhu Li, Hao Peng, and Heng Ji. Executable code actions elicit better llm agents. In Proceedings of the 41st International Conference on Machine Learning, ICML'24. JMLR.org, 2024

    2024

  15. [23]

    Intercode: Standardizing and benchmarking interactive coding with execution feedback, 2023

    John Yang, Akshara Prabhakar, Karthik Narasimhan, and Shunyu Yao. Intercode: Standardizing and benchmarking interactive coding with execution feedback, 2023. URL https://arxiv.org/abs/2306.14898

    2023 arXiv

  16. [24]

    -bench: A benchmark for tool-agent-user interaction in real-world domains, 2024

    Shunyu Yao, Noah Shinn, Pedram Razavi, and Karthik Narasimhan. -bench: A benchmark for tool-agent-user interaction in real-world domains, 2024. URL https://arxiv.org/abs/2406.12045

    2024 arXiv

  17. [25]

    Cibench: Evaluating your llms with a code interpreter plugin, 2024 a

    Chuyu Zhang, Songyang Zhang, Yingfan Hu, Haowen Shen, Kuikun Liu, Zerun Ma, Fengzhe Zhou, Wenwei Zhang, Xuming He, Dahua Lin, and Kai Chen. Cibench: Evaluating your llms with a code interpreter plugin, 2024 a . URL https://arxiv.org/abs/2407.10499

    2024 arXiv

  18. [26]

    Pybench: Evaluating llm agent on various real-world coding tasks, 2024 b

    Yaolun Zhang, Yinxu Pan, Yudong Wang, and Jie Cai. Pybench: Evaluating llm agent on various real-world coding tasks, 2024 b . URL https://arxiv.org/abs/2407.16732

    2024 arXiv

  19. [27]

    Debug like a human: A large language model debugger via verifying runtime execution step-by-step, 2024

    Li Zhong, Zilong Wang, and Jingbo Shang. Debug like a human: A large language model debugger via verifying runtime execution step-by-step, 2024. URL https://arxiv.org/abs/2402.16906

    2024 arXiv

  20. [28]

    Cve-bench: A benchmark for ai agents' ability to exploit real-world web application vulnerabilities, 2025

    Yuxuan Zhu, Antony Kellermann, Dylan Bowman, Philip Li, Akul Gupta, Adarsh Danda, Richard Fang, Conner Jensen, Eric Ihli, Jason Benn, Jet Geronimo, Avi Dhir, Sudhit Rao, Kaicheng Yu, Twm Stone, and Daniel Kang. Cve-bench: A benchmark for ai agents' ability to exploit real-worl...

    2025 arXiv

  21. [29]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

  22. [30]

    @esa (Ref

    \@ifxundefined[1] #1\@undefined \@firstoftwo \@secondoftwo \@ifnum[1] #1 \@firstoftwo \@secondoftwo \@ifx[1] #1 \@firstoftwo \@secondoftwo [2] @ #1 \@temptokena #2 #1 @ \@temptokena \@ifclassloaded agu2001 natbib The agu2001 class already includes natbib coding, so you should ...

  23. [31]

    \@lbibitem[] @bibitem@first@sw\@secondoftwo \@lbibitem[#1]#2 \@extra@b@citeb \@ifundefined br@#2\@extra@b@citeb \@namedef br@#2 \@nameuse br@#2\@extra@b@citeb \@ifundefined b@#2\@extra@b@citeb @num @parse #2 @tmp #1 NAT@b@open@#2 NAT@b@shut@#2 \@ifnum @merge>\@ne @bibitem@firs...

  24. [32]

    B `8F gHr## l2J ?\ϋX j<(nlJ2d\/AUQcY

    @open @close @open @close and [1] URL: #1 \@ifundefined chapter * \@mkboth \@ifxundefined @sectionbib * \@mkboth * \@mkboth\@gobbletwo \@ifclassloaded amsart * \@ifclassloaded amsbook * \@ifxundefined @heading @heading NAT@ctr thebibliography [1] @ \@biblabel @NAT@ctr \@bibset...

    1999

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.