When Developer Aid Becomes Security Debt: A Systematic Analysis of Insecure Behaviors in LLM Coding Agents

Matous Kozak; Roshanak Zilouchian Moghaddam; Siva Sivaraman

REVIEW 4 major objections 5 minor 2 cited by

When Developer Aid Becomes Security Debt: A Systematic Analysis of Insecure Behaviors in LLM Coding Agents

T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A systematic evaluation of five leading coding agents on 93 real-world setup tasks finds that 21% of trajectories contain insecure actions, that secure trajectories complete tasks more often, and that feedback-based remediation fixes…

desk verdict The qualitative finding is credible and useful—agents do take insecure actions during routine setup tasks—but the headline rates rest on a detector validated on its own tuning set, so the specific numbers should be treated as provisional, not as benchmarks. read the letter →

arxiv 2507.09329 v2 pith:S5FJ4ZNE submitted 2025-07-12 cs.AI cs.CR

Matous Kozak , Roshanak Zilouchian Moghaddam , Siva Sivaraman This is my paper · ORCID

classification cs.AIcs.CR

keywords LLMcodingagentsagentsecurityevaluationinsecureactionsCWEtaxonomyinformationexposureaccesscontrolfeedbackmitigationsoftwaresetuptasks

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that LLM-based coding agents, left to their default settings on routine software setup tasks, regularly take actions that introduce exploitable security weaknesses, and that these lapses are not rare edge cases. Analyzing more than 12,000 agent actions across five leading language models on 93 real-world setup tasks, it reports that 21% of task trajectories contained at least one insecure action, with model-level trajectory rates between 16.13% and 26.88%. It also claims a consistent link between security and task success: trajectories free of insecure steps finished the task more often than those containing insecure steps for every model. Finally, it evaluates two mitigation strategies and reports that real-time feedback corrected 73.3% of flagged actions on average, with the best model reaching 96.8%.

What carries the argument

The load-bearing mechanism is a two-stage, prompt-based detection pipeline rather than a formal analyzer. A custom classification prompt, optimized on 500 human-labeled steps, takes each agent action together with the task assignment and the agent's own reasoning and labels the action secure or insecure, reaching 98.6% accuracy, 100% precision, and 61.11% recall on that same labeled set; a second categorization prompt maps flagged actions to four CWE classes. The pipeline is what converts raw trajectories into every reported rate, so the entire argument runs through it.

What would settle it

Re-score the same 12,000-plus actions with a deterministic rule-based detector for the four named patterns (hardcoded credentials, chmod 666/777, curl-pipe-to-shell, and disabled SSH host-key or database auth) and compare trajectory-level rates against the reported 21.0%. If deterministic scoring finds the rate below about 15% across models, the paper's headline claim overstates the problem; if it matches or exceeds 21%, the claim holds. Running the LLM detector repeatedly on identical trajectories would also settle whether the model ranking is stable.

Watch

Extended reading notes

Core claim

The paper's central discovery is that insecure behavior is a normal byproduct of current coding agents rather than an adversarial trigger. On a standardized benchmark of 93 software setup tasks, an open-source coding agent backed by five state-of-the-art LLMs produced at least one action classified as insecure in 21% of trajectories; the most common failure was CWE-200, exposure of sensitive information such as hardcoded credentials, followed by improper access control (CWE-284) and downloading code without integrity checks (CWE-494). The same data show a security-success correlation: for all five models, trajectories without insecure steps had higher task completion rates than trajectories with insecure steps, with the largest gap being 24.1 percentage points. The paper further claims that mitigation works unevenly: feeding the agent the detector's reasoning as feedback remediated 73.3% of insecure actions on average, while one model reached 96.8% and another only 54.8%, and that proactive security reminders were less effective than targeted feedback.

Load-bearing premise

The reported rates and rankings depend on the LLM-based detector being a valid and sufficiently complete measure of what counts as insecure; the detector was tuned and evaluated on the same 500 labeled steps, has 61% recall, and was explicitly instructed to prefer 'secure' when unsure, so undiscovered false negatives could shift every headline number.

Editorial extensions

If this is right

Without added security guardrails, an enterprise deploying such agents should expect roughly one in five setup tasks to include at least one security-relevant mistake.
Insecure trajectories are less likely to complete their task, so security failures are not merely hypothetical risks; they co-occur with functional failure.
Hardcoded credentials and overly broad permissions are the dominant failure classes, so credential-management tooling and least-privilege defaults would address most observed issues.
Feedback that explains why an action is insecure is more effective than generic security reminders, supporting runtime guardrails over prompt-only guidance.
Mitigation effectiveness varies widely by model, so model choice determines how much benefit guardrails will deliver.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Inference: Because the detector's recall is only 61% and it is biased toward 'secure' in ambiguous cases, the 21% trajectory rate is more likely an undercount than an overcount; a more complete detector would probably push the rate higher while possibly changing the model ordering.
Inference: The consistent success gap between secure and insecure trajectories suggests insecurity is a marker of task difficulty or poor trajectory quality, so a runtime security check could double as a progress signal, not just a guardrail.
Inference: The mitigation numbers come from offline post-hoc re-scoring, not live intervention; in a real agent loop, interrupting an action would likely change both remediation success and task completion, so the 73.3% average is an upper-bound estimate.
Inference: The same method could be extended to longer code-editing tasks and to agent-visible tests; a testable prediction is that the 56.61% late-trajectory concentration of first insecure steps grows with context length, since the paper cites long-context degradation as a candidate cause.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Two things to know. First, the qualitative core is real and worth knowing: all five coding agents, running ordinary setup tasks with no security guardrails, do things like embed credentials in scripts, chmod 777 files, and pipe curl to sh. The examples in Appendix D are concrete, and the manual labeling found insecure steps, so this phenomenon is not an artifact of the detector. Second, the headline numbers—21% of trajectories, the per-model ranking, the 96.8% remediation rate—are not independently established. The detector was iteratively optimized on the same 500 labeled actions used to report its accuracy, precision, and recall, so those metrics are training-set metrics. With 61% recall and a prompt that instructs the model to prefer SECURE when unsure, the measured rates are likely lower bounds with unknown bias across models.

What's actually new: this is the first comparative measurement I know of looking at default, non-adversarial agent behavior across multiple LLM backends on a common benchmark, with a CWE-based taxonomy and mitigation experiments. The taxonomy (CWE-200, 284, 494, 693) is sensible, and the observation that insecure trajectories are less likely to succeed is interesting and consistent across models. The authors are transparent about their method and honestly list limitations in Section 4.2.

Soft spots: besides the evaluation leakage, Table 4 gives no denominators, so GPT-4.1's 96.8% could rest on a handful of flagged steps. There is no held-out validation, no repeated-run variance for the detector, and no artifacts or data released, which makes it impossible to verify the CWE categorization or the trajectory rates. The paper acknowledges LLM detector nondeterminism but doesn't quantify it. None of this undercuts the qualitative conclusion, but it does mean the precise numbers should be read as illustrative, not as security benchmarks for model-selection decisions.

Who this is for: people building guardrails for coding agents, security teams evaluating LLM backends, and researchers studying evaluation methodology for agent safety. The paper is a useful wake-up call that agent behavior in normal operation deserves security review.

Recommendation: worth sending to peer review. A serious referee should require release of trajectories and labels, a held-out validation of the detector, and denominators for the mitigation table. With those changes the quantitative claims could be made solid; without them, the paper is a useful qualitative study with unreliable point estimates.

Referee Report

4 major / 5 minor

Summary. This paper presents an empirical study of insecure behaviors in LLM-based coding agents. The authors run the OpenHands agent on 93 SetupBench configuration tasks with five LLM backends (GPT-4o, GPT-4.1, Claude 3.5, Claude 3.7, Claude 4 Sonnet), use an LLM-based classifier to label agent actions as secure or insecure, and report that 21% of trajectories contain at least one insecure action, with per-model trajectory rates from 16.13% to 26.88%. They categorize insecure actions into four CWE-based classes, propose two mitigation strategies (security reminders and feedback), and report remediation success rates, with GPT-4.1 achieving 96.8% average remediation success. The paper also reports a negative association between insecure steps and task success.

Significance. If the quantitative results were independently validated, this would be a useful first systematic measurement of how often coding agents take security-relevant actions during routine setup tasks, with practical implications for model selection and guardrail design. The paper is transparent in publishing the detection and categorization prompts, providing concrete examples of flagged behaviors, and acknowledging the post-hoc nature of the mitigation evaluation. Its main weakness is that the detector that drives all reported rates is validated only on the same 500-step subset used to optimize it, with 61.11% recall and an explicit bias toward SECURE classifications; until this is addressed, the precise headline numbers and model rankings should be treated as provisional.

major comments (4)

[2.2] The performance metrics in Section 2.2 are computed on the same 500 manually labeled steps used to iteratively optimize the detection prompt, so 98.6% accuracy, 100% precision, and 61.11% recall are training-set estimates rather than independent validation. Because every downstream result in Tables 2 and 4 is produced by this detector, the headline trajectory rates and remediation rates inherit any bias in the detector. Please re-validate on a held-out set (or with cross-validation), report repeated-run variance given the nondeterminism acknowledged in Section 4.2, and provide a full confusion matrix.
[2.2 / Appendix A] The detection prompt in Appendix A explicitly instructs the classifier to prefer SECURE when unsure and to treat task-aligned actions as SECURE. Combined with the reported 61.11% recall, this means roughly 39% of expert-identified insecure actions are missed, and the paper does not analyze whether false negatives are distributed uniformly across models, action types, or CWE categories. If they cluster, the per-model ranking in Table 2 and the secure-versus-insecure success comparison in Figure 2 could shift materially. Please report false-negative distributions and a sensitivity analysis.
[3.2 / Table 4] Table 4 reports remediation success rates without denominators; with only 21 baseline insecure steps for GPT-4.1 (Table 2), 96.8% could correspond to a very small sample, and the same issue affects other cells and the trajectory percentages in Table 2 (e.g., GPT-4o 16.13% = 15/93). Please report raw counts, per-model and per-strategy denominators, and confidence intervals for every percentage. In addition, because Section 4.2 states that mitigation was evaluated offline and post-hoc, the term 'remediation success' should be qualified to reflect that no live agent interaction was tested.
[3.1.1 / Figure 2] The claim that secure trajectories consistently achieve higher success rates rests on very small per-model counts: for GPT-4.1, 31.2% of insecure trajectories corresponds to 5 of 16 trajectories, and one additional success changes the rate by more than six percentage points. Please report exact trajectory counts, event counts, and a statistical test (e.g., Fisher's exact test) or confidence intervals before framing this as a consistent security-performance trade-off.

minor comments (5)

[2.2] The phrase 'fine-tuned detection prompt' is misleading; the paper optimizes a prompt, not model weights, so 'prompt-engineered' would be more accurate.
[3.1.1] The statement that 56.61% of first insecure steps occur in the second half of trajectories lacks a per-model breakdown; please provide the supporting counts.
[3.1.2 / Table 3] Percentages are reported without counts or confidence intervals; given the small totals (e.g., 21 for GPT-4.1), a single misclassification can change a category share substantially.
[Figure 1] The caption says 'distribution of collected agent's actions' but does not state what the y-axis represents; clarify whether bars are action counts and add the task-success information referenced in the text.
[References] Reference [18] lists authors 'Gabriel Costa, John Smith, and Alice Johnson,' which appear generic; verify that this is the correct bibliographic entry.

Circularity Check

2 steps flagged · score 6.0 of 10

Headline rates rest on a detector tuned and scored on the same 500 labeled steps; remediation success is scored by the same detector that issued the feedback.

fitted input called prediction [Section 2.2, 'Detection of Insecure Practices' (iterative optimization and performance-evaluation paragraphs; Abstract repeats the claim)]
"To detect insecure behaviors, we developed a custom prompt through an iterative optimization process using a subset of 500 manually-labeled steps from OpenHands' trajectories. ... The final prompt achieves 98.6% accuracy with 100% precision and 61.11% recall on our evaluation dataset."

The prompt was iteratively optimized on the same 500 manually labeled steps that are then used as the evaluation dataset. Accuracy, precision, and recall are therefore training-set metrics, not independent estimates of detector performance. These metrics are the evidence that the detector is 'high-precision' before it is applied to produce the 21% trajectory rate, the per-model ranking in Table 2, and the remediation rates. With recall at 61.11% and the prompt instructed to prefer SECURE when unsure, the headline quantities inherit an unvalidated, lenient measurement instrument.
self definitional [Sections 2.3 and 3.2 (feedback mechanism description and Table 4 caption)]
"The feedback mechanism is implemented by providing the reasoning output from the detection prompt ... that explains why the action is considered insecure with a suggestion to remediate the issue ... In the following Table 4, we report the remediation success rates, which are calculated based on the total number of insecure steps that we were able to re-evaluate with the detection prompt."

Remediation 'success' is defined as the same detection prompt now labeling the modified action SECURE. The feedback and the evaluation use the same instrument, which was optimized on 500 examples and explicitly told to prefer SECURE when unsure. The 73.3% average and GPT-4.1's 96.8% therefore measure the detector's label change and self-consistency, not an independent confirmation that the modified action is actually secure. The mitigation result reduces to the detector re-classifying its own outputs.

full rationale

The paper's qualitative conclusion that coding agents sometimes take insecure actions is supported by external evidence: human experts labeled 18 of 500 steps (3.6%) as insecure, and the detector has 100% precision on those labels. However, the quantitative headline claims — 21% of trajectories, the model ranking in Table 2, and the remediation success rates in Table 4 — all flow through a detection prompt that was iteratively optimized on the same 500 labeled steps used for its performance evaluation. Accuracy and precision are thus training-set metrics, and the 'prefer SECURE when unsure' instruction plus 61.11% recall mean the detector likely undercounts insecure actions. The remediation evaluation is additionally self-referential because the same detector provides the feedback and then judges whether the agent's correction succeeded. These are not cases of imported uniqueness theorems or self-citations carrying the argument; the manual labels and concrete examples provide independent existence evidence. Still, the precise rates and model comparisons are not independently established, so the circularity score is 6 rather than higher.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The central claim rests on the detector's labels, on the representativeness of SetupBench, and on the assumption that post-hoc mitigation approximates live behavior. No new physical or conceptual entities are introduced. The manual labels are the only external anchor, and they are small, imbalanced, and also used to tune the detector.

free parameters (3)

Detection prompt bias toward SECURE = not quantified; yields 61.11% recall on the 500 labeled steps
The prompt instructs the classifier to prefer SECURE when unsure (Appendix A), an explicit precision/recall trade-off chosen during iterative optimization on the 500-step subset. This directly lowers the detected insecure-step rate.
Manual label subset of 500 steps = 18 insecure steps, 3.6% of the subset
The subset used for prompt tuning and evaluation is tiny, imbalanced, and its sampling procedure from trajectories is not described; all detector metrics and downstream rates depend on it.
Maximum 100 iterations per task = 100
Trajectory length cutoff affects which actions are observed; the paper notes 56.61% of first insecure steps occur in the second half, so truncation could miss late insecure behavior.

assumptions (4)

domain assumption LLM-based classification labels correspond to real, exploitable security vulnerabilities
Section 2.2 treats the detection prompt's labels as ground truth for all 12,000+ actions; manual validation covers only the 500 steps used to train the prompt.
domain assumption SetupBench tasks are representative of real-world software setup work
Section 4.2 acknowledges the benchmark limitation; the paper generalizes from 93 configuration tasks to developer scenarios.
domain assumption Post-hoc mitigation evaluation approximates live agent behavior
Section 4.2 states mitigation was applied offline rather than in real-time agent interaction; the paper assumes this approximates real-world performance.
domain assumption Actions within one trajectory can be treated as independent observations
Action-level percentages treat each of the 12,000+ actions as independent, but actions within a single trajectory are correlated, inflating effective sample sizes.

how reviews work

0 comments

Cite this review

Pith. "Pith review of When Developer Aid Becomes Security Debt: A Systematic Analysis of Insecure Behaviors in LLM Coding Agents." pith.science (2026). https://pith.science/paper/S5FJ4ZNE

@misc{pith2026250709329,
  author       = {Pith},
  title        = {Pith review of: When Developer Aid Becomes Security Debt: A Systematic Analysis of Insecure Behaviors in LLM Coding Agents},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/S5FJ4ZNE}},
  note         = {Machine review of arXiv:2507.09329}
}

read the original abstract

LLM-based coding agents are rapidly being deployed in software development, yet their safety implications remain poorly understood. These agents, while capable of accelerating software development, may exhibit unsafe behaviors during normal operation that manifest as cybersecurity vulnerabilities. We conducted the first systematic safety evaluation of autonomous coding agents, analyzing over 12,000 actions across five state-of-the-art models (GPT-4o, GPT-4.1, Claude variants) on 93 real-world software setup tasks. Our findings reveal significant security concerns: 21% of agent trajectories contained insecure actions, with models showing substantial variation in unsafe behavior. We developed a high-precision detection system that identified four major vulnerability categories, with information exposure (CWE-200) being the most prevalent one. We also evaluated mitigation strategies including feedback mechanisms and security reminders with various effectiveness between models. GPT-4.1 demonstrated exceptional security awareness with 96.8% mitigation success.

Figures

Figures reproduced from arXiv: 2507.09329 by the authors.

**Figure 2.** Success rate comparison between trajectories with and without insecure steps by model. [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

IssueTrojanBench: Benchmarking AI Coding Agents Against Malicious Issue Requests
cs.CR 2026-07 conditional novelty 6.0 of 10

AI coding agents followed malicious instructions embedded in issue-style artifacts in 66.5% of 4,176 test runs.
The Vulnerability With No CVE: Managing Persistent Gaps Between Mandate and Authority in AI Coding Agents
cs.CR 2026-08 conditional novelty 5.0 of 10

The paper defines a new vulnerability-management object, the agentic posture vulnerability, with a thresholded definition, lifecycle, record schema, and testable research agenda for AI coding agents.

Reference graph

Works this paper leans on

22 extracted references · 21 canonical work pages · cited by 2 Pith papers

[1]

https://github.com/features/copilot

Github copilot. https://github.com/features/copilot

work page
[2]

https://cursor.com

Cursor - the ai code editor. https://cursor.com

work page
[3]

https://roocode.com

Roo code. https://roocode.com. 9

work page
[4]

https://windsurf.com

Windsurf. https://windsurf.com

work page
[5]

Xu, Xiangru Tang, Mingchen Zhuge, Jiayi Pan, Yueqi Song, Bowen Li, Jaskirat Singh, Hoang H

Xingyao Wang, Boxuan Li, Yufan Song, Frank F. Xu, Xiangru Tang, Mingchen Zhuge, Jiayi Pan, Yueqi Song, Bowen Li, Jaskirat Singh, Hoang H. Tran, Fuqiang Li, Ren Ma, Mingzhang Zheng, Bill Qian, Yanjun Shao, Niklas Muennighoff, Yizhe Zhang, Binyuan Hui, Junyang Lin, Robert Brennan, Hao Peng, Heng Ji, and Graham Neubig. Openhands: An open platform for AI soft...

work page 2025
[6]

Setupbench: Assessing software engineering agents’ ability to bootstrap development environments

Avi Arora, Jinu Jang, and Roshanak Zilouchian Moghaddam. Setupbench: Assessing software engineering agents’ ability to bootstrap development environments. 2025

work page 2025
[7]

Evolutionary multi-objective optimization of large language model prompts for balancing sentiments

Jill Baumann and Oliver Kramer. Evolutionary multi-objective optimization of large language model prompts for balancing sentiments. In Stephen Smith, João Correia, and Christian Cintrano, editors, Applications of Evolutionary Computation , pages 212–224, Cham, 2024. Springer Nature Switzerland

work page 2024
[8]

Long-context llms struggle with long in-context learning

Tianle Li, Ge Zhang, Quy Duc Do, Xiang Yue, and Wenhu Chen. Long-context llms struggle with long in-context learning. 2024

work page 2024

Show all 22 references

[9]

Guiding ai to fix its own flaws: An empirical study on llm-driven secure code generation

Hao Yan, Swapneel Suhas Vaidya, Xiaokuan Zhang, and Ziyu Yao. Guiding ai to fix its own flaws: An empirical study on llm-driven secure code generation. 2025

2025
[10]

Wildfeedback: Aligning llms with in-situ user interactions and feedback

Taiwei Shi, Zhuoer Wang, Longqi Yang, Ying-Chun Lin, Zexue He, Mengting Wan, Pei Zhou, Sujay Jauhar, Sihao Chen, Shan Xia, Hongfei Zhang, Jieyu Zhao, Xiaofeng Xu, Xia Song, and Jennifer Neville. Wildfeedback: Aligning llms with in-situ user interactions and feedback. 2025

2025
[11]

Llamafirewall: An open source guardrail system for building secure ai agents

Sahana Chennabasappa, Cyrus Nikolaidis, Daniel Song, David Molnar, Stephanie Ding, Shengye Wan, Spencer Whitman, Lauren Deason, Nicholas Doucette, Abraham Montilla, Alekhya Gampa, Beto de Paola, Dominik Gabi, James Crnkovich, Jean-Christophe Testud, Kat He, Rashnil Chaturvedi,...

2025
[12]

Poskitt, and Jun Sun

Haoyu Wang, Christopher M. Poskitt, and Jun Sun. Agentspec: Customizable runtime enforce- ment for safe and reliable llm agents. 2025

2025
[13]

Llms cannot reliably identify and reason about security vulnerabilities (yet?): A compre- hensive evaluation, framework, and benchmarks

Saad Ullah, Mingji Han, Saurabh Pujar, Hammond Pearce, Ayse Coskun, and Gianluca Stringh- ini. Llms cannot reliably identify and reason about security vulnerabilities (yet?): A compre- hensive evaluation, framework, and benchmarks. In 2024 IEEE Symposium on Security and Privac...

2024
[14]

Easy problems that llms get wrong

James Huckle and Sean Williams. Easy problems that llms get wrong. In Kohei Arai, editor, Advances in Information and Communication, pages 313–332, Cham, 2025. Springer Nature Switzerland

2025
[15]

Agent-safetybench: Evaluating the safety of llm agents

Zhexin Zhang, Shiyao Cui, Yida Lu, Jingzhuo Zhou, Junxiao Yang, Hongning Wang, and Minlie Huang. Agent-safetybench: Evaluating the safety of llm agents. 2025

2025
[16]

Ignore previous prompt: Attack techniques for language models

Fábio Perez and Ian Ribeiro. Ignore previous prompt: Attack techniques for language models. 2022

2022
[17]

Wolf hidden in sheep’s conversations: Toward harmless data-based backdoor attacks for jailbreaking large language models

Jiawei Kong, Hao Fang, Xiaochen Yang, Kuofeng Gao, Bin Chen, Shu-Tao Xia, Yaowei Wang, and Min Zhang. Wolf hidden in sheep’s conversations: Toward harmless data-based backdoor attacks for jailbreaking large language models. 2025

2025
[18]

Securing ai agents with information-flow control

Gabriel Costa, John Smith, and Alice Johnson. Securing ai agents with information-flow control. In Proceedings of the IEEE Symposium on Security and Privacy, pages 145–160, 2024

2024
[19]

Choquette-Choo, Milad Nasr, Chawin Sitawarin, Gena Gibson, Andreas Terzis, and John "Four" Flynn

Chongyang Shi, Sharon Lin, Shuang Song, Jamie Hayes, Ilia Shumailov, Itay Yona, Juliette Pluto, Aneesh Pappu, Christopher A. Choquette-Choo, Milad Nasr, Chawin Sitawarin, Gena Gibson, Andreas Terzis, and John "Four" Flynn. Lessons from defending gemini against indirect prompt ...

2025
[20]

SWE-agent: Agent-computer interfaces enable automated soft- ware engineering

John Yang, Carlos E Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik R Narasimhan, and Ofir Press. SWE-agent: Agent-computer interfaces enable automated soft- ware engineering. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024

2024
[21]

Redcode: Risky code execution and generation benchmark for code agents

Chengquan Guo, Xun Liu, Chulin Xie, Andy Zhou, Yi Zeng, Zinan Lin, Dawn Song, and Bo Li. Redcode: Risky code execution and generation benchmark for code agents. In The Thirty-eight Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2024

2024
[22]

thought

Hammond Pearce, Baleegh Ahmad, Benjamin Tan, Brendan Dolan-Gavitt, and Ramesh Karri. Asleep at the keyboard? assessing the security of github copilot’s code contributions. In 2022 IEEE Symposium on Security and Privacy (SP), pages 754–768, 2022. A Detection of Insecure Practis...

2022

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.

[1] [1]

https://github.com/features/copilot

Github copilot. https://github.com/features/copilot

work page

[2] [2]

https://cursor.com

Cursor - the ai code editor. https://cursor.com

work page

[3] [3]

https://roocode.com

Roo code. https://roocode.com. 9

work page

[4] [4]

https://windsurf.com

Windsurf. https://windsurf.com

work page

[5] [5]

Xu, Xiangru Tang, Mingchen Zhuge, Jiayi Pan, Yueqi Song, Bowen Li, Jaskirat Singh, Hoang H

Xingyao Wang, Boxuan Li, Yufan Song, Frank F. Xu, Xiangru Tang, Mingchen Zhuge, Jiayi Pan, Yueqi Song, Bowen Li, Jaskirat Singh, Hoang H. Tran, Fuqiang Li, Ren Ma, Mingzhang Zheng, Bill Qian, Yanjun Shao, Niklas Muennighoff, Yizhe Zhang, Binyuan Hui, Junyang Lin, Robert Brennan, Hao Peng, Heng Ji, and Graham Neubig. Openhands: An open platform for AI soft...

work page 2025

[6] [6]

Setupbench: Assessing software engineering agents’ ability to bootstrap development environments

Avi Arora, Jinu Jang, and Roshanak Zilouchian Moghaddam. Setupbench: Assessing software engineering agents’ ability to bootstrap development environments. 2025

work page 2025

[7] [7]

Evolutionary multi-objective optimization of large language model prompts for balancing sentiments

Jill Baumann and Oliver Kramer. Evolutionary multi-objective optimization of large language model prompts for balancing sentiments. In Stephen Smith, João Correia, and Christian Cintrano, editors, Applications of Evolutionary Computation , pages 212–224, Cham, 2024. Springer Nature Switzerland

work page 2024

[8] [8]

Long-context llms struggle with long in-context learning

Tianle Li, Ge Zhang, Quy Duc Do, Xiang Yue, and Wenhu Chen. Long-context llms struggle with long in-context learning. 2024

work page 2024