Pith. sign in

REVIEW 3 major objections 6 minor 36 references

Agent Skills Can Be Harmful: An Empirical Study of Skill-Induced Failures in LLM Agents

T0 review · 3 major / 6 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Agent skills can be harmful: even topically relevant skills frequently induce failures and cost regressions in LLM agents.

desk verdict A thoughtful empirical study of skill-induced agent failures with a plausible taxonomy, but the headline 307 cases rest on single-run paired comparisons, so the causal counts are not yet established. read the letter →

arxiv 2608.11888 v1 pith:ZXGSYV6Z submitted 2026-08-12 cs.AI

Gen Dong , Yanjie Gao , Liqun Li , Tianyin Xu , Yu Hua , Fan Yang This is my paper · ORCID
classification cs.AI
keywords agentskillsskill-inducedfailuresLLMagentsdifferentialtestingroot-causetaxonomyfunctionalefficiencyregressionscost-awareskillreuse
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that loading a reusable agent skill into an LLM agent's context can cause the agent to fail tasks or become substantially more expensive, even when the skill is topically relevant to the task. It establishes this with paired runs that keep the task, verifier, model, and environment fixed and vary only the skill setup, yielding 307 confirmed skill-induced failures: 125 functional failures and 182 efficiency regressions. The dominant functional failure mechanism is Task-Implementation Fault, where a seemingly relevant skill leads the agent to fill a required element incorrectly (46 cases) or omit it (36 cases), together 82 of 125 cases. The dominant cost mechanism is Excessive Procedure, where skills turn optional exploration, construction, and verification into mandatory work, accounting for 114 of 182 regressions, with excessive verification alone at 67 cases. If these attributions hold, skill reuse needs compatibility checks, cost prediction, and guarded task constraints rather than blind trust in topical relevance.

What carries the argument

The mechanism is differential paired execution, adapted from differential testing. For every audited skill, the paper runs the same task under the skill and under a reference setup, either no skill or a semantically matched public skill, while holding task, verifier, agent framework, model, and repository state fixed. A functional failure is recorded when the skill run fails and the reference run passes; an efficiency regression is recorded when both pass but token use and execution time both increase and at least one more than doubles. This contrast turns the loaded skill into the only plausible causal variable. The companion machinery is a pair of root-cause taxonomies: functional failures split into Applicability Mismatch, Environment Mismatch, Task-Implementation Fault, and Artifact Misplacement; efficiency regressions split into Context Bloat, Excessive Procedure, and Dependency Resolution. The triage tool operationalizes these taxonomies as differential-evidence checklists for automated attribution.

What would settle it

Take a random sample of the 125 functional-failure pairs and re-run each condition, say 20 times, with the same model, verifier, and skill setup. If the no-skill condition and the skill condition each produce a mix of PASS and FAIL across seeds, or if the FAIL/PASS contrast reverses in a substantial share of seeds, the skill-induced attribution for those cases is not stable and the 307-count conclusion would need revision.

Watch

Extended reading notes

Core claim

The central discovery is that skills harm agents mainly by being too trusted in the wrong way, not by being obviously irrelevant. In the functional-failure taxonomy only 2 of 125 cases are Applicability Mismatch; 86 cases are Task-Implementation Fault, where the agent correctly recognizes the topic but implements a task-required element with the wrong API, value, scaling, or format, or leaves that element absent. A further 24 cases are Artifact Misplacement, where the agent writes a plausible artifact at a repository-convention location rather than the task-specified path, and 13 are Environment Mismatch. On the efficiency side, the paper shows regression cost is not explained by prompt length alone: when context overhead appears it is almost always the mandatory skill-body text (43 of 46), but the larger class is Excessive Procedure (114 of 182), dominated by excessive verification (67) and heavy implementation pipelines (30). The paper also demonstrates that a taxonomy-guided triage tool can reproduce the manual category for 117/125 functional failures and 145/182 efficiency regressions, showing the classification is operationalizable.

Load-bearing premise

The attribution assumes that a single PASS in the reference run and a single FAIL or single costly run in the skill run reflect the skill's causal effect rather than ordinary sampling variation in the language model, because the paper does not report repeated runs or variance estimates.

Editorial extensions

If this is right

  • Agent platforms should treat task-specified paths, integration points, and environment state as guarded constraints, because Artifact Misplacement and Environment Mismatch together account for 37 of 125 functional failures.
  • Skill authors should separate mandatory task requirements from reusable examples and defaults, and lazy-load long skill bodies, since mandatory body text causes 43 of 46 context-overhead regressions.
  • Agents need budget-aware verification policies: excessive verification is the single largest regression source, so validation checklists become mandatory work in 67 of 182 cases.
  • Skill selection should estimate induced procedural cost, not just prompt length, because Excessive Procedure accounts for 114 of 182 regressions, more than double Context Bloat.
  • Automated triage at 88.8% exact-subcategory agreement for functional failures and 72.5% for efficiency regressions makes continuous screening of skill updates feasible.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The single-run design means the reported counts are likely an upper bound; re-running with multiple seeds would probably shrink the 307 total, though the rough shape of the taxonomies could survive.
  • The same differential-pair method could be applied to other context injections, such as retrieved documentation, system prompts, or example packs, to test whether implementation distortion is unique to skills or generic to added guidance.
  • A cheap screening experiment suggested by the paper: take a skill with Excessive Verification and replace its exhaustive checklist with a conditional 'verify only if the change is risky' instruction; if cost drops without new failures, it confirms the mandatory-work mechanism.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. This paper presents an empirical study of skill-induced failures in LLM agents. The authors propose a differential analysis framework that attributes task failures and efficiency regressions to loaded skills by comparing a target skill-guided run against a no-skill or semantically matched reference run on the same task. They instantiate the framework on SkillsBench and SWE-Skills-Bench, reporting 307 confirmed skill-induced failures, of which 125 are functional failures and 182 are efficiency regressions. From these cases, the authors derive root-cause taxonomies, finding that Task-Implementation Fault accounts for 68.8% of functional failures and Excessive Procedure accounts for 62.6% of efficiency regressions. They also build SKILLTRIAGE, an LLM-based tool that normalizes paired cases, extracts differential evidence, and produces attribution reports, reporting 88.8% subcategory agreement for functional failures and 72.5% for efficiency regressions against manual labels.

Significance. If the causal attribution is sound, this study makes a valuable contribution to the emerging literature on LLM agent skills. It moves beyond aggregate pass-rate comparisons to a mechanism-level analysis of how skill contents and induced trajectories cause failures and cost regressions. The differential design is a logical and appropriate way to isolate skill effects, and the manual labeling with group consensus is a strength. The taxonomy is detailed and the tooling direction is timely. However, the significance is currently tempered by the lack of repeated runs, the absence of a released dataset or code, and the internal-consistency nature of the SKILLTRIAGE evaluation. The central claim that skills can cause failures is plausible and the qualitative examples are suggestive, but the quantitative headline counts are not yet separated from sampling variation.

major comments (3)
  1. [III-A and III-D] The causal attribution in this paper rests on paired target/reference runs in which only the skill setup is varied, yet each condition appears to be executed exactly once: Section III-A states that "PASS and FAIL denote the deterministic verifier outcome of a single run," and no repeated seeds or variance estimates are reported anywhere. Because LLM agent trajectories are stochastic, a single no-skill PASS and a single skill FAIL do not by themselves establish that the skill caused the failure. The problem is amplified by the search over 20,664 potential paired comparisons (Table I) and the later selection of extreme cases where the target fails while the reference passes or cost ratios exceed 2.0; under sampling noise, such extreme pairs are expected in a large comparison space. The efficiency regression threshold in Section III-D (min(r_tok,r_time)>1.0 and max(r_tok,r_time)>2.0) can also be triggered by token/time noise. The threats-to-validity paragraph in Section VII-B acknowledges "ordinary agent variance" but does not measure it and does not include a negative control such as no-skill versus no-skill or same-skill versus same-skill repeated runs. Without repeated runs or a noise baseline, the headline counts (125 functional failures, 182 regressions) and the percentages in Findings 1 and 4 cannot be separated from sampling variation.
  2. [VI-C and VII-B] The evaluation of SKILLTRIAGE is an internal consistency check rather than an external validation. The taxonomy used as ground truth (Tables III and IV) was induced from the same 125 functional failures and 182 efficiency regressions that are later used to test the tool, and SKILLTRIAGE's evidence extraction is explicitly built from those taxonomy definitions (Section VI-A). The paper's claim in Section VII-B that SKILLTRIAGE "provides an independent consistency check" is therefore overstated: the reported 88.8% subcategory agreement for functional failures and 72.5% for efficiency regressions measure how well the tool reproduces the human labels on the very data used to construct the taxonomy. A held-out set of newly collected cases, or at least a cross-validation split, is needed to support the claim that the tool generalizes.
  3. [III-D and Table II] The dataset refinement step that reduces 665 labeled candidates to 307 analysis cases is not sufficiently specified for reproducibility. Section III-D says only that "We remove candidates with insufficient evidence, likely verifier-induced false positives, and duplicate same-task/same-skill effects," without operational definitions, per-reason counts, or examples. Because the final counts are central to every finding, the paper should report how many candidates were excluded for each reason, with representative examples, and describe the consensus procedure in enough detail that the filtering could be re-run.
minor comments (6)
  1. [Abstract] The abstract contains a typo: "thede factomechanism" should read "the de facto mechanism."
  2. [III-C] The semantic similarity threshold (0.7), the use of all-MiniLM-L6-v2, and the top-5 candidate selection are free parameters; please report a sensitivity analysis or at least justify these exact choices.
  3. [III-A and Figure 2] Figure 2's color coding (orange and yellow boxes) is mentioned in the text but not explained in the caption; please add a legend or caption description.
  4. [III-C] The sentence in Section III-C explaining that the self-generated-skill condition is "included in the original comparison space but not treated as a variant in the expanded public-skill comparison space" is confusing; please clarify what this means for the reported counts.
  5. [V-A] The rule for assigning a case to Context Bloat versus Excessive Procedure when both effects are present ("we assign the subcategory according to the larger cost driver observed in the trajectory") is subjective; please provide a more operational decision rule or inter-rater agreement for this boundary.
  6. [General] For an empirical study of this kind, releasing the paired trajectories, skill files, verifier outputs, and labeling code would greatly aid reproducibility and allow independent verification of the 307 attributed failures.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the 307 failure attributions are contrastive empirical outcomes, not fitted predictions, and the SkillTriage evaluation is explicitly an internal consistency check that does not feed into the central findings.

full rationale

The paper's central derivation chain is an empirical contrastive measurement, not a mathematical reduction. The 307 skill-induced failures are defined by paired target/reference runs in which the task, verifier, agent framework, model, repository or container state, and input data are fixed and only the skill setup varies; the resulting FAIL/PASS or PASS/PASS-with-high-cost differences are observed outcomes, not fitted parameters renamed as predictions. No equation is shown to reduce to its own inputs, and no fitted parameter is later relabeled as a prediction. The root-cause taxonomy was induced from manual audits of these cases, and SkillTriage is evaluated against the same manual labels; however, the paper states explicitly that 'the tool is not used to create the taxonomy or replace manual validation,' so this evaluation is an internal consistency check rather than a load-bearing justification for the headline findings. The central claims about Task-Implementation Fault and Excessive Procedure are based on the manual audit, not on SkillTriage. The paper also acknowledges the relevant threat that a cost change may reflect 'ordinary agent variance' in its Threats to Validity section, which is a statistical validity concern rather than a circularity. References to prior benchmarks and the agent-skills abstraction are external support, and no load-bearing self-citation or imported uniqueness theorem appears. Overall, the derivation chain is self-contained for the claims it actually makes, and no specific circular reduction can be exhibited.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

The paper introduces a new taxonomy and a triage tool, but no physical or ontological entities. The main epistemic load is carried by hand-set thresholds and the pseudo-oracle assumption, which are listed above.

free parameters (3)
  • T = 2.0
    Efficiency regression threshold: max token or time ratio must exceed 2.0 while both exceed 1.0. Chosen by hand to focus on large regressions (Section III-D).
  • semantic similarity threshold = 0.7
    Minimum cosine similarity for retaining public skill candidates retrieved via all-MiniLM-L6-v2; chosen to filter off-topic matches (Section III-C).
  • top-k candidates = 5
    Maximum number of semantically matched public skills retained per curated skill, limiting evaluation cost (Section III-C).
assumptions (5)
  • domain assumption The paired-run setup isolates the skill as the only varying factor, so outcome differences are attributable to the skill.
    Assumes LLM sampling variance is negligible or controlled; no repeated runs are reported (Section III-A).
  • domain assumption Deterministic programmatic verifiers correctly measure task correctness.
    PASS/FAIL outcomes are taken from verifier outputs without independent validation (Section III-A).
  • domain assumption The no-skill reference run passing is a pseudo-oracle showing the task is solvable by the base agent.
    The paper explicitly calls it a pseudo-oracle, not ground truth (Section III-A).
  • domain assumption Retrieved public skills from smithery.ai and skillsmp.com are plausible skill alternatives for the audited tasks.
    Relevance is based on embedding similarity of metadata, not task-level validation (Section III-C).
  • domain assumption Manual root-cause labels assigned by group consensus are the ground truth for SkillTriage evaluation.
    The tool is evaluated against these labels, so their correctness is assumed (Section VI-B).

how reviews work

0 comments
Cite this review

Pith. "Pith review of Agent Skills Can Be Harmful: An Empirical Study of Skill-Induced Failures in LLM Agents." pith.science (2026). https://pith.science/paper/ZXGSYV6Z

@misc{pith2026260811888,
  author       = {Pith},
  title        = {Pith review of: Agent Skills Can Be Harmful: An Empirical Study of Skill-Induced Failures in LLM Agents},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ZXGSYV6Z}},
  note         = {Machine review of arXiv:2608.11888}
}
read the original abstract

Agent skills are the de facto mechanism for extending LLM agents with reusable guidance. A skill can shape the agent's task execution, including planning, tool use, problem-solving, and validation. Prior work reported mixed results of agent skills: some skills improve task success rates, while others have no effect, increase token use and execution time, and even reduce success rates. This paper presents a comprehensive analysis of skill-induced agent failures by attributing task failures and cost regressions to specific loaded skills. We introduce a differential analysis framework that attributes a failure or regression to a skill by comparing a target skill-guided run against a no-skill or semantically matched skill reference run that solves the same task, or solves it more cheaply. We instantiate this framework on SkillsBench and SWE-Skills-Bench, yielding 307 skill-induced failures, including 125 functional failures and 182 efficiency regressions. We also build SkillTriage, a taxonomy-guided attribution tool that normalizes paired cases, extracts differential evidence, and produces triage reports. Our major findings include: (1) Skill induced functional failures are rarely caused by obviously irrelevant skills; instead, seemingly relevant skills often make the agent incorrectly implement or omit task-required implementation elements. (2) Skill-induced efficiency regressions are not explained by prompt length alone. (3) The largest sources within Excessive Procedure are excessive verification and heavy implementation pipelines, contributing 67 and 30 cases, respectively. This shows that skills often turn validation checklists and construction recipes into mandatory work. Based on our findings, we propose research topics and tooling improvements for safer and more cost-aware skill reuse.

Figures

Figures reproduced from arXiv: 2608.11888 by the authors.

Figure 2
Figure 2. Differential evaluation across skill setups. [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Overview of the study methodology. cost, but they often do not provide an alternative successful skill for the same task, which is needed to localize the failure to differences in skill content and induced trajectories. C. Data Collection To broaden the skill setups available for paired evalu￾ation beyond the original benchmark settings, we search two public skill-sharing sites that host reusable agent skills: smith… view at source ↗
Figure 4
Figure 4. Workflow of SKILLTRIAGE. ity vote. Both evaluations use manually assigned attribution labels derived from the corresponding functional-failure and efficiency-regression audits. Majority voting is a standard en￾semble strategy for reducing prediction variance [12]; in LLM settings, related self-consistency methods similarly aggregate multiple sampled outputs to improve reliability [13]. We use a 2-of-3 vote because t… view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

36 extracted references · 22 canonical work pages

  1. [1]

    Skillsbench: Benchmarking how well agent skills work across diverse tasks,

    X. Li, W. Chen, Y . Liu, S. Zheng, X. Chen, Y . He, Y . Li, B. You, H. Shen, J. Sunet al., “Skillsbench: Benchmarking how well agent skills work across diverse tasks,”arXiv preprint arXiv:2602.12670, 2026

    arXiv 2026

  2. [2]

    Swe-skills- bench: Do agent skills actually help in real-world software engineering?

    T. Han, Y . Zhang, W. Song, C. Fang, Z. Chen, Y . Sun, and L. Hu, “Swe-skills- bench: Do agent skills actually help in real-world software engineering?”arXiv preprint arXiv:2603.15401, 2026

    arXiv 2026

  3. [3]

    Lost in the middle: How language models use long contexts,

    N. F. Liu, K. Lin, J. Hewitt, A. Paranjape, M. Bevilacqua, F. Petroni, and P. Liang, “Lost in the middle: How language models use long contexts,” Transactions of the association for computational linguistics, vol. 12, pp. 157– 173, 2024

    work page 2024

  4. [4]

    Large language models can be easily distracted by irrelevant context,

    F. Shi, X. Chen, K. Misra, N. Scales, D. Dohan, E. H. Chi, N. Sch ¨arli, and D. Zhou, “Large language models can be easily distracted by irrelevant context,” inInternational Conference on Machine Learning. PMLR, 2023, pp. 31 210–31 227

    2023

  5. [5]

    Quantifying language models’ sensitivity to spurious features in prompt design or: How i learned to start worrying about prompt formatting,

    M. Sclar, Y . Choi, Y . Tsvetkov, and A. Suhr, “Quantifying language models’ sensitivity to spurious features in prompt design or: How i learned to start worrying about prompt formatting,” inInternational Conference on Learning Representations, vol. 2024, 2024, pp. 25 055–25 083

    work page 2024

  6. [6]

    More doc- uments, same length: Isolating the challenge of multiple documents in rag,

    S. Levy, N. Mazor, L. Shalmon, M. Hassid, and G. Stanovsky, “More doc- uments, same length: Isolating the challenge of multiple documents in rag,” arXiv preprint arXiv:2503.04388, 2025

    arXiv 2025

  7. [7]

    Differential testing for software,

    W. M. McKeeman, “Differential testing for software,”Digital Technical Jour- nal, vol. 10, no. 1, pp. 100–107, 1998

    work page 1998

  8. [8]

    Equipping agents for the real world with agent skills,

    B. Zhang, K. Lazuka, and M. Murag, “Equipping agents for the real world with agent skills,”Anthropic Engineering Blog, 2025

    work page 2025

Show all 36 references
  1. [9]

    OpenCode: The open source coding agent,

    Anomaly, “OpenCode: The open source coding agent,” https://github.com/anomalyco/opencode, 2025

    2025

  2. [10]

    Introducing claude opus 4.6,

    Anthropic, “Introducing claude opus 4.6,” https://www.anthropic.com/news/claude-opus-4-6, 2026

    2026

  3. [11]

    Introducing gpt-5.5,

    OpenAI, “Introducing gpt-5.5,” https://openai.com/index/introducing-gpt-5- 5/, 2026

    2026

  4. [12]

    Ensemble methods in machine learning,

    T. G. Dietterich, “Ensemble methods in machine learning,” inInternational workshop on multiple classifier systems. Springer, 2000, pp. 1–15

    2000

  5. [13]

    Self-consistency improves chain of thought reasoning in language models,

    X. Wang, J. Wei, D. Schuurmans, Q. Le, E. Chi, S. Narang, A. Chowdhery, and D. Zhou, “Self-consistency improves chain of thought reasoning in language models,”arXiv preprint arXiv:2203.11171, 2022

    2022 arXiv

  6. [14]

    V oyager: An open-ended embodied agent with large lan- guage models,

    G. Wang, Y . Xie, Y . Jiang, A. Mandlekar, C. Xiao, Y . Zhu, L. Fan, and A. Anandkumar, “V oyager: An open-ended embodied agent with large lan- guage models,”arXiv preprint arXiv:2305.16291, 2023

    2023 arXiv

  7. [15]

    Executable code actions elicit better llm agents,

    X. Wang, Y . Chen, L. Yuan, Y . Zhang, Y . Li, H. Peng, and H. Ji, “Executable code actions elicit better llm agents,” inForty-first International Conference on Machine Learning, 2024

    2024

  8. [16]

    Gpt4tools: Teaching large language model to use tools via self-instruction,

    R. Yang, L. Song, Y . Li, S. Zhao, Y . Ge, X. Li, and Y . Shan, “Gpt4tools: Teaching large language model to use tools via self-instruction,”Advances in Neural Information Processing Systems, vol. 36, pp. 71 995–72 007, 2023

    2023

  9. [17]

    React: Synergizing reasoning and acting in language models,

    S. Yao, J. Zhao, D. Yu, N. Du, I. Shafran, K. Narasimhan, and Y . Cao, “React: Synergizing reasoning and acting in language models,”arXiv preprint arXiv:2210.03629, 2022

    2022 arXiv

  10. [18]

    Same task, more tokens: the impact of input length on the reasoning performance of large language models,

    M. Levy, A. Jacoby, and Y . Goldberg, “Same task, more tokens: the impact of input length on the reasoning performance of large language models,” in Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2024, pp. 15 339–15 353

    2024

  11. [19]

    Why do multi-agent llm systems fail?

    M. Cemri, M. Z. Pan, S. Yang, L. A. Agrawal, B. Chopra, R. Tiwari, K. Keutzer, A. Parameswaran, D. Klein, K. Ramchandranet al., “Why do multi-agent llm systems fail?”Advances in Neural Information Processing Systems, vol. 38, 2026

    2026

  12. [20]

    Agentless: Demystifying llm- based software engineering agents,

    C. S. Xia, Y . Deng, S. Dunn, and L. Zhang, “Agentless: Demystifying llm- based software engineering agents,”arXiv preprint arXiv:2407.01489, 2024

    2024 arXiv

  13. [21]

    Swe-agent: Agent-computer interfaces enable automated software engineering,

    J. Yang, C. Jimenez, A. Wettig, K. Lieret, S. Yao, K. Narasimhan, and O. Press, “Swe-agent: Agent-computer interfaces enable automated software engineering,”Advances in Neural Information Processing Systems, vol. 37, pp. 50 528–50 652, 2024

    2024

  14. [22]

    Swe-bench: Can language models resolve real-world github issues?

    C. E. Jimenez, J. Yang, A. Wettig, S. Yao, K. Pei, O. Press, and K. Narasimhan, “Swe-bench: Can language models resolve real-world github issues?” in International Conference on Learning Representations, vol. 2024, 2024, pp. 54 107–54 157

    2024

  15. [23]

    Openhands: An open platform for ai software developers as generalist agents, 2024,

    X. Wang, B. Li, Y . Song, F. F. Xu, X. Tang, M. Zhuge, J. Pan, Y . Song, B. Li, J. Singhet al., “Openhands: An open platform for ai software developers as generalist agents, 2024,”URL https://arxiv. org/abs/2407.16741, vol. 2, no. 4, p. 9, 2024

    2024 arXiv

  16. [24]

    Multi-swe-bench: A multilingual benchmark for issue resolving,

    D. Zan, Z. Huang, W. Liu, H. Chen, S. Xin, L. Zhang, Q. Liu, L. Aoyan, L. Chen, X. Zhonget al., “Multi-swe-bench: A multilingual benchmark for issue resolving,”Advances in Neural Information Processing Systems, vol. 38, 2026

    2026

  17. [25]

    Swe-lancer: Can frontier llms earn $1 million from real-world freelance software engineering?

    S. Miserendino, M. Wang, T. Patwardhan, and J. Heidecke, “Swe-lancer: Can frontier llms earn $1 million from real-world freelance software engineering?” arXiv preprint arXiv:2502.12115, 2025

    2025 arXiv

  18. [26]

    Measuring ai ability to complete long software tasks,

    T. Kwa, B. West, J. Becker, A. Deng, K. Garcia, M. Hasin, S. Jawhar, M. Kinniment, N. Rush, S. V on Arxet al., “Measuring ai ability to complete long software tasks,”Advances in Neural Information Processing Systems, vol. 38, pp. 92 213–92 266, 2026

    2026

  19. [27]

    Understanding and detecting real-world performance bugs,

    G. Jin, L. Song, X. Shi, J. Scherpelz, and S. Lu, “Understanding and detecting real-world performance bugs,”ACM SIGPLAN Notices, vol. 47, no. 6, pp. 77– 88, 2012

    2012

  20. [28]

    Discovering, reporting, and fixing perfor- mance bugs,

    A. Nistor, T. Jiang, and L. Tan, “Discovering, reporting, and fixing perfor- mance bugs,” in2013 10th working conference on mining software repositories (MSR). IEEE, 2013, pp. 237–246

    2013

  21. [29]

    Performance issues and optimizations in javascript: an empirical study,

    M. Selakovic and M. Pradel, “Performance issues and optimizations in javascript: an empirical study,” inProceedings of the 38th international conference on software engineering, 2016, pp. 61–72

    2016

  22. [30]

    Understanding performance problems in deep learning systems,

    J. Cao, B. Chen, C. Sun, L. Hu, S. Wu, and X. Peng, “Understanding performance problems in deep learning systems,” inProceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, 2022, pp. 357–369

    2022

  23. [31]

    A comprehensive study on deep learning bug characteristics,

    M. J. Islam, G. Nguyen, R. Pan, and H. Rajan, “A comprehensive study on deep learning bug characteristics,” inProceedings of the 2019 27th ACM joint meeting on european software engineering conference and symposium on the foundations of software engineering, 2019, pp. 510–520

    2019

  24. [32]

    Taxonomy of real faults in deep learning systems,

    N. Humbatova, G. Jahangirova, G. Bavota, V . Riccio, A. Stocco, and P. Tonella, “Taxonomy of real faults in deep learning systems,” inProceedings of the ACM/IEEE 42nd international conference on software engineering, 2020, pp. 1110–1121

    2020

  25. [33]

    An empirical study on tensorflow program bugs,

    Y . Zhang, Y . Chen, S.-C. Cheung, Y . Xiong, and L. Zhang, “An empirical study on tensorflow program bugs,” inProceedings of the 27th ACM SIGSOFT international symposium on software testing and analysis, 2018, pp. 129–140

    2018

  26. [34]

    An empirical study on configuration errors in commercial and open source systems,

    Z. Yin, X. Ma, J. Zheng, Y . Zhou, L. N. Bairavasundaram, and S. Pasupathy, “An empirical study on configuration errors in commercial and open source systems,” inProceedings of the Twenty-Third ACM Symposium on Operating Systems Principles, 2011, pp. 159–172

    2011

  27. [35]

    Hey, you have given me too many knobs!: Understanding and dealing with over-designed configuration in system software,

    T. Xu, L. Jin, X. Fan, Y . Zhou, S. Pasupathy, and R. Talwadker, “Hey, you have given me too many knobs!: Understanding and dealing with over-designed configuration in system software,” inProceedings of the 2015 10th Joint Meeting on Foundations of Software Engineering, 2015, ...

    2015

  28. [36]

    An empirical study on performance bugs for highly config- urable software systems,

    X. Han and T. Yu, “An empirical study on performance bugs for highly config- urable software systems,” inProceedings of the 10th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement, 2016, pp. 1–10

    2016

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.