Pith. sign in

REVIEW 3 major objections 6 minor 6 cited by

When Prompts Go Wrong: Evaluating Code Model Robustness to Ambiguous, Contradictory, and Incomplete Task Descriptions

T0 review · 3 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Code-generation models lose 20–40 percent of their accuracy when task descriptions are unclear.

desk verdict The contradictory-condition Pass@1 drop is largely a test-oracle artifact, but the dataset, the mutation taxonomy, and the RQ1 classification result are worth a serious referee. read the letter →

arxiv 2507.20439 v1 pith:MPTWST6J submitted 2025-07-27 cs.SE cs.AI

Maya Larbi , Amal Akli , Mike Papadakis , Rihab Bouyousfi , Maxime Cordy , Federica Sarro , Yves Le Traon This is my paper
classification cs.SEcs.AI
keywords codegenerationLLMrobustnessambiguouspromptscontradictoryrequirementsincompletespecificationspromptmutationnaturallanguagePass@1
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Code generation language models are usually evaluated on clean, expert-written task descriptions, but real developer instructions are often ambiguous, incomplete, or self-contradictory. This paper tries to establish how much those imperfections matter by systematically mutating the task descriptions of two standard benchmarks and measuring how well seven models of different sizes still produce correct code. It finds that even small phrasing changes cut Pass@1 by 20 to 40 percentage points, that no model reliably detects when a description is unclear, and that different defect types push models into different error classes. The upshot is that current benchmarks overstate how well code LLMs would perform in real software workflows, and robustness to unclear requirements needs to be a first-class evaluation target.

What carries the argument

The load-bearing object is a mutation pipeline that turns clear benchmark descriptions into unclear ones: a large language model is prompted with a taxonomy of defect rules covering ambiguity, contradiction, and incompleteness to rewrite each original task description, and the results pass a two-step expert validation with manual refinement. The evaluation then reuses the original test suites as an oracle and measures degradation with Pass@1, Successful Execution Rate, and Runnable-but-Incorrect Rate, plus a classification task for detecting description quality. This design lets the paper attribute performance differences to description quality while keeping task intent fixed, at least for the non-contradictory mutations.

What would settle it

Score code generated from contradictory descriptions against tests written to match the mutated description instead of the original; if Pass@1 returns to near the clear-description level, the reported degradation is a test-specification mismatch rather than a model robustness failure.

Watch

Extended reading notes

Core claim

On its own terms, the paper's central discovery is that code generation LLMs are not robust to realistic task-description defects: across both benchmarks, unclear descriptions lower Pass@1 by 20–40 percentage points, and the share of runnable-but-incorrect outputs rises to 60–90% under defect conditions. Contradictory descriptions are the most damaging, with the best-performing model dropping from about 74% to below 7% Pass@1 on one benchmark. The paper also finds that models cannot reliably classify descriptions as clear or unclear (MCC between -0.1 and 0.55), and that the three defect types produce distinct failure signatures: incomplete descriptions trigger structural errors such as TypeErrors and SyntaxErrors, ambiguous descriptions cause semantically wrong but executable code, and contradictory descriptions yield logically inconsistent behavior. Larger models are somewhat more resilient, but scale does not remove the vulnerability.

Load-bearing premise

The study assumes that the original HumanEval and MBPP test suites are the right yardstick for code generated from mutated descriptions, even though a contradictory description may describe a task that no correct program can satisfy.

Editorial extensions

If this is right

  • Current code-generation benchmarks that use only clean, expert-written descriptions overestimate the correctness users will get from informal prompts.
  • In real development, a large share of code generated from imperfect instructions will be runnable but semantically wrong, so execution-based checks are not a sufficient safety net.
  • Models need a clarification or description-verification step before generating code, since they currently cannot reliably tell a clear request from an unclear one.
  • Scaling model size helps robustness on some benchmarks but does not remove the vulnerability, so training on noisy or under-specified data will be needed.
  • Different defect types have different failure signatures, so debugging tools could be tailored to the type of description flaw.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • For contradictory descriptions, no program can satisfy both the mutated wording and the original test suite; the reported Pass@1 drop therefore partly measures an unsatisfiable specification, not pure model failure.
  • A natural extension would be to let models ask for clarification before generating code and score whether clarification restores accuracy; the paper's RQ1 result suggests most models would fail this test today.
  • The same mutation pipeline could be applied to longer, multi-file, or multilingual tasks, where the paper's own validity discussion predicts even larger robustness gaps.
  • The distinct error signatures by defect type suggest a cheap diagnostic: classify the prompt's defect category and predict likely exception classes before running the code.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper presents an empirical study of code LLM robustness to ambiguous, contradictory, and incomplete task descriptions. It mutates HumanEval and MBPP prompts with GPT-4 under three defect categories, validates mutations with human experts, and evaluates seven model variants (GPT-4, CodeLlama, DeepSeek-Coder, and Qwen2.5) on Pass@1, SER, and RIR, plus an RQ1 classification task and an RQ3 error-type analysis. The main claims are that code LLMs cannot reliably detect unclear descriptions, that Pass@1 drops by 20-40 percentage points under unclear descriptions, that contradictory descriptions cause logical errors, and that larger models are somewhat more robust. The dataset and code are publicly available.

Significance. If the results hold, this would be a useful step beyond surface-level robustness benchmarks like ReCode: it targets semantic requirement defects and provides a published dataset and open repository. The study is also commendable for human validation of the mutations and for separating syntax, structural, and logical failure modes. However, the contradictory-condition oracle is conceptually mismatched, and the RQ1 sampling is not reproducible as written; both are load-bearing for the main claims. With a re-analysis that separates prompt-test mismatch from model deficiency, the paper could make a solid contribution.

major comments (3)
  1. [Section 5.2, Tables 2-3, Table 5] The Pass@1 oracle for the contradictory condition is mismatched. Section 5.2 defines Pass@1 as passing the original HumanEval/MBPP test suite, while Section 4 defines contradictory descriptions as containing incompatible requirements. For the contradictory mutations, the original tests encode only one side of the conflict: MBPP/2 asks to 'include elements not in both', and HumanEval/28 asks to 'return an integer' from a list-of-strings function, so any faithful implementation of the mutated prompt is guaranteed to fail the original tests. Consequently, the reported 20-40 percentage-point drops (Tables 2 and 3) and the RQ3 'logical inconsistency' failure mode partially measure prompt-test mismatch rather than model deficiency. The authors acknowledge memorization as a confound but not this oracle mismatch. I recommend re-analyzing the contradictory condition with a specification-appropriate oracle, e.g., human-annotated expected outputs for the mutated descriptions, or at minimum reporting the proportion of failures that are due to unsatisfiable original tests.
  2. [Section 6.1] The RQ1 sampling protocol is impossible as described: 'we randomly selected 300 tasks from the original benchmarks' cannot be drawn from HumanEval alone, which has 164 tasks. If the 300 come from the combined pool or from MBPP, the proportions and per-benchmark sample sizes must be reported. As written, the MCC results for RQ1 cannot be reproduced and may not be balanced across datasets.
  3. [Tables 2 and 3, Section 6.2] No confidence intervals or significance tests are provided for any Pass@1, SER, or RIR comparison. Since HumanEval contains only 164 tasks and MBPP contains 974 (with a possibly unbalanced subset), the claims that 'larger models tend to be more resilient' and that specific families underperform should be backed by per-task paired comparisons or bootstrap intervals; otherwise the differences could reflect prompt-sample variance. At minimum, report the number of tasks per condition and a paired test between original and mutated conditions.
minor comments (6)
  1. [Section 6.2] The sentence '54%, 65% and 89% in the documents. cases of Incomplete, Ambiguous and contradictory descriptions' contains a typo; it should read 'in the Incomplete, Ambiguous, and Contradictory cases.'
  2. [Section 8] In the concluding section, 'menas' should be corrected to 'means.'
  3. [Section 5.2] Reference [6] (Chai and Draxler on RMSE/MAE) does not support the metric definitions in Section 5.2; please cite a code-generation evaluation reference instead.
  4. [Section 5.1] Reference [10] is cited to justify the benchmark selection but points to a GitHub Copilot security paper; the intended benchmark reference appears to be [8] or [9].
  5. [Table 4] Table 4 uses 'NotFoundError'; the standard Python exception is FileNotFoundError. Please clarify whether this is a distinct error category or a typo.
  6. [Section 4.2] The text says 'All task descriptions failing the two validations were flagged to undergo the second validation step'; please clarify whether failing either or both validations triggers refinement, and report how many mutations were manually refined.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: the study is an empirical benchmark evaluation whose ground truth comes from human validation and original test suites, and no fitted parameter is relabeled as a prediction.

full rationale

The paper does not derive results from an input that already contains those results. The mutation procedure uses GPT-4 to generate unclear descriptions, but the correctness ground truth is the original HumanEval/MBPP test suite, and the mutations are human-validated (85% naturalness, 93% valid defect presence) and manually refined. Using GPT-4 both as mutation engine and as one evaluated model is a design choice, not a circular derivation, because the models are scored against external tests rather than against GPT-4's own judgment. The contradictory-condition evaluation has a genuine construct-validity concern: Pass@1 is defined as passing the original test suite (Section 5.2), while contradictory descriptions are defined as conflicting with the original specification (Section 4), so a model that faithfully follows the mutated prompt is expected to fail the original tests. However, this is not a circularity in the derivation-chain sense: models sometimes ignore the mutated clause and still pass original tests (e.g., GPT-4 still achieves 6.7% on contradictory HumanEval tasks), so the reported drop is an empirical observation about model behavior rather than a tautology forced by the definitions. The paper also explicitly acknowledges memorization as a confound in Section 7. No self-citation chain, no fitted-input-as-prediction loop, and no uniqueness theorem imported from the authors' own prior work is present. The central claims are therefore not equivalent to their inputs by construction.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

The paper introduces no fitted parameters or invented entities. Its conclusions rest on domain assumptions about the validity of the mutation procedure and the use of original tests as ground truth.

assumptions (3)
  • domain assumption Mutated descriptions preserve the original task intent, so original test suites are valid oracles.
    The paper uses original HumanEval and MBPP tests to score code generated from mutated descriptions (Sections 5.2, 5.3). For contradictory descriptions, satisfying both the description and the oracle is impossible by construction.
  • domain assumption The three requirement defects (ambiguity, contradiction, incompleteness) are the most prevalent and are faithfully operationalized by GPT-4 mutations plus expert validation.
    Section 4 relies on expert review (85% naturalness, 93% defect presence) rather than an external validated corpus; GPT-4's generation choices may not span the full space of real-world unclear descriptions.
  • domain assumption Performance degradation is attributed to reasoning failure rather than to breaking memorized training-data patterns.
    Authors acknowledge in Section 7 that mutations may break exact overlaps with training data, so the drop may partly stem from reduced familiarity advantage.

how reviews work

0 comments
Cite this review

Pith. "Pith review of When Prompts Go Wrong: Evaluating Code Model Robustness to Ambiguous, Contradictory, and Incomplete Task Descriptions." pith.science (2026). https://pith.science/paper/MPTWST6J

@misc{pith2026250720439,
  author       = {Pith},
  title        = {Pith review of: When Prompts Go Wrong: Evaluating Code Model Robustness to Ambiguous, Contradictory, and Incomplete Task Descriptions},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/MPTWST6J}},
  note         = {Machine review of arXiv:2507.20439}
}
read the original abstract

Large Language Models (LLMs) have demonstrated impressive performance in code generation tasks under idealized conditions, where task descriptions are clear and precise. However, in practice, task descriptions frequently exhibit ambiguity, incompleteness, or internal contradictions. In this paper, we present the first empirical study examining the robustness of state-of-the-art code generation models when faced with such unclear task descriptions. We extend the HumanEval and MBPP benchmarks by systematically introducing realistic task descriptions flaws through guided mutation strategies, producing a dataset that mirrors the messiness of informal developer instructions. We evaluate multiple LLMs of varying sizes and architectures, analyzing their functional correctness and failure modes across task descriptions categories. Our findings reveal that even minor imperfections in task description phrasing can cause significant performance degradation, with contradictory task descriptions resulting in numerous logical errors. Moreover, while larger models tend to be more resilient than smaller variants, they are not immune to the challenges posed by unclear requirements. We further analyze semantic error patterns and identify correlations between description clarity, model behavior, and error types. Our results underscore the critical need for developing LLMs that are not only powerful but also robust to the imperfections inherent in natural user tasks, highlighting important considerations for improving model training strategies, designing more realistic evaluation benchmarks, and ensuring reliable deployment in practical software development environments.

Figures

Figures reproduced from arXiv: 2507.20439 by the authors.

Figure 1
Figure 1. Examples of mutated task descriptions (original [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Overview of the process followed to create the de [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 4
Figure 4. Classification accuracy (%) of LLMs on identifying task description issues in two benchmarks. The top row shows [PITH_FULL_IMAGE:figures/full_fig_p006_4.png] view at source ↗
Figures from the paper (2 more)
Figure 5
Figure 5. Figure 5: Mean Pass@1 accuracy (%) of small (6–7B) versus large (32–34B) model variants on HumanEval (left) and MBPP (right) [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 6
Figure 6. Figure 6: Runnable but incorrect rates (%) of seven LLM vari [PITH_FULL_IMAGE:figures/full_fig_p009_6.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 6 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. AssumptionMiner: Extracting, Tracing, and Revising Implicit Assumptions in LLM Code Generation

    cs.SE 2026-07 conditional novelty 7.0 of 10

    AssumptionMiner extracts implicit assumptions from LLM-generated code with a cross-model ensemble F1 of 0.816 on a new 180-task benchmark, and enables assumption-guided targeted regeneration.

  2. From Failing to Passing: Evolving Natural Language Prompt Optimization Rules for LLM Code Generation

    cs.SE 2026-07 conditional novelty 6.5 of 10

    DualFix recovers up to 30% of LLM code-generation failures by combining evolved, error-agnostic prompt rewrite rules with execution feedback, fixing 3–5× more cases than Self-Fix and transferring zero-shot across models.

  3. Obey, Diverge, Collapse: Blind Obedience to Incorrect Instructions Drives Code LLMs to Irrecoverable Code Semantic Collapse

    cs.SE 2026-07 conditional novelty 6.5 of 10

    Code LLMs correctly label incorrect repair instructions as wrong, then follow them anyway, creating compounding Ghost Errors that self-guided iterative repair usually cannot reverse.

  4. VClare: Resolving Imperfect Specifications in LLM-Based Verilog Generation

    cs.AR 2026-07 conditional novelty 6.0 of 10

    Repairing imperfect Verilog specs with LLM inconsistency mining and simulation clustering lifts generation pass rates ~13%, with sim-level repair remaining robust as designs grow complex.

  5. On the risk of coding before testing: An empirical study on LLM-based test generation workflow

    cs.SE 2026-07 conditional novelty 6.0 of 10

    Generating LLM tests after faulty LLM code cuts fault detection to about 14% versus about 25% when tests are generated from the task description alone, across models and prompting strategies.

  6. Automatically Evolving Prompt Guidelines for Task-Specific Optimization

    cs.CL 2026-05 conditional novelty 6.0 of 10

    AGOPS automatically evolves task-specific prompt guidelines from reference answers and reports recovering 15.5–81.7% of the performance lost to underspecified prompts.

Reference graph

Works this paper leans on

43 extracted references · 26 canonical work pages · cited by 6 Pith papers

  1. [1]

    IEEE Recommended Practice for Software Requirements Specifications

    1998. IEEE Recommended Practice for Software Requirements Specifications. IEEE Std 830-1998 (1998), 1–40. doi:10.1109/IEEESTD.1998.88286

    work page arXiv 1998

  2. [2]

    DeepSeek AI. 2024. DeepSeek-Coder: Unlocking Code Generation Abilities with Large Language Models. arXiv preprint arXiv:2401.14196 (2024). https: //arxiv.org/abs/2401.14196

    arXiv 2024

  3. [3]

    Maalem Abdulsattar Atiyaa and Marwa Adeeb Al-Jawaherry. 2025. Exploring deep learning for accurate software cost estimation: A long short-term memory (LSTM) approach. In AIP Conference Proceedings, Vol. 3264. AIP Publishing LLC, 040019

    work page 2025

  4. [5]

    Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, and Charles Sutton. 2021. Program Synthesis with Large Language Models. arXiv preprint arXiv:2108.07732 (2021)

    arXiv 2021

  5. [6]

    Tianfeng Chai and Roland R Draxler. 2014. Root mean square error (RMSE) or mean absolute error (MAE)?–Arguments against avoiding RMSE in the literature. Geoscientific model development 7, 3 (2014), 1247–1250

    work page 2014

  6. [7]

    Haoyang Chen, Botong Xu, Louis Zhong Rui Wong, and Kaiyang Zhong. 2025. Enhancing software effort estimation through reinforcement learning-based project management-oriented feature selection. International Journal of Manag- ing Projects in Business (2025)

    work page 2025

  7. [9]

    Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde De Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. 2021. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374 (2021)

    arXiv 2021

  8. [10]

    Arghavan Moradi Dakhel, Vahid Majdinasab, Amin Nikanjam, Foutse Khomh, Michel C Desmarais, and Zhen Ming Jack Jiang. 2023. Github copilot ai pair programmer: Asset or liability? Journal of Systems and Software 203 (2023), 111734

    2023

Show all 43 references
  1. [12]

    Shihan Dou, Haoxiang Jia, Shenxi Wu, Huiyuan Zheng, Weikang Zhou, Muling Wu, Mingxu Chai, Jessica Fan, Caishuang Huang, Yunbo Tao, Yan Liu, Enyu Zhou, Ming Zhang, Yuhao Zhou, Yueming Wu, Rui Zheng, Ming Wen, Rongxiang Weng, Jingang Wang, Xunliang Cai, Tao Gui, Xipeng Qiu, Qi Z...

    2024

  2. [13]

    Adithya V Ganesan, Yash Kumar Lal, August Håkan Nilsson, and H Andrew Schwartz. 2023. Systematic evaluation of GPT-3 for zero-shot personality esti- mation. arXiv preprint arXiv:2306.01183 (2023)

    2023 arXiv

  3. [14]

    Dan Hendrycks, Steven Basart, Saurav Kadavath, Mantas Mazeika, Akul Arora, Ethan Guo, Collin Burns, Samir Puranik, Horace He, Dawn Song, and Jacob Steinhardt. 2021. Measuring Coding Challenge Competence With APPS. arXiv preprint arXiv:2105.09938 (2021). https://arxiv.org/abs/2...

    2021 arXiv

  4. [15]

    Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory.Neural computation 9, 8 (1997), 1735–1780

    1997

  5. [16]

    Xinyi Hou, Yanjie Zhao, Yue Liu, Zhou Yang, Kailong Wang, Li Li, Xiapu Luo, David Lo, John Grundy, and Haoyu Wang. 2024. Large Language Models for Software Engineering: A Systematic Literature Review. ACM Trans. Softw. Eng. Methodol. 33, 8, Article 220 (Dec. 2024), 79 pages. d...

    2024 doi

  6. [17]

    Anca-Elena Iordan. 2024. An optimized LSTM neural network for accurate estimation of software development effort. Mathematics 12, 2 (2024), 200

    2024

  7. [18]

    Magne Jorgensen and Martin Shepperd. 2006. A systematic review of software development cost estimation studies. IEEE Transactions on software engineering 33, 1 (2006), 33–53

    2006

  8. [19]

    Yuhang Lai, Chengxi Li, Yiming Wang, Tianyi Zhang, Ruiqi Zhong, Luke Zettle- moyer, Wen-tau Yih, Daniel Fried, Sida Wang, and Tao Yu. 2023. DS-1000: a natural and reliable benchmark for data science code generation. In Proceedings of the 40th International Conference on Machin...

    2023

  9. [20]

    Yasir Mahmood, Nazri Kama, Azri Azmi, Ahmad Salman Khan, and Mazlan Ali

  10. [21]

    Ruchika Malhotra. 2015. A systematic review of machine learning techniques for software fault prediction. Applied Soft Computing 27 (2015), 504–518

    2015

  11. [22]

    Bertrand Meyer. 1985. On Formalism in Specifications. IEEE Softw. 2, 1 (1985), 6–26. doi:10.1109/MS.1985.229776

    1985

  12. [23]

    Lloyd Montgomery, Davide Fucci, Abir Bouraffa, Lisa Scholz, and Walid Maalej

  13. [24]

    OpenAI. 2023. GPT-4 Technical Report. https://cdn.openai.com/papers/gpt- 4.pdf

    2023

  14. [25]

    Empirical research on requirements quality: a systematic mapping study. Requir. Eng. 27, 2 (2022), 183–209. doi:10.1007/S00766-021-00367-Z

    2022 doi

  15. [26]

    Zepeng Qin, Xiangyang Li, Zhongzhi Huang, et al. 2024. Qwen-Code: A Pow- erful Language Model for Code Understanding and Generation. arXiv preprint arXiv:2403.05530 (2024). https://arxiv.org/abs/2403.05530

    2024 arXiv

  16. [27]

    Hammond Pearce, Baleegh Ahmad, Benjamin Tan, Brendan Dolan-Gavitt, and Ramesh Karri. 2025. Asleep at the keyboard? assessing the security of github copilot’s code contributions. Commun. ACM 68, 2 (2025), 96–105

    2025

  17. [28]

    Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. 2018. Semantically equivalent adversarial rules for debugging NLP models. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (volume 1: long papers). 856–865

    2018

  18. [29]

    Mizanur Rahman, Teresa Goncalves, and Hasan Sarwar. 2023. Review of existing datasets used for software effort estimation. Int. J. Adv. Comput. Sci. Appl. 14, 7 (2023)

    2023

  19. [30]

    Bruno Budel Rossi and Lisandra Manzoni Fontoura. [n. d.]. AI-Based Approaches for Software Tasks Effort Estimation: A Systematic Review of Methods and Trends. ([n. d.])

  20. [31]

    Ritu and Pankaj Bhambri. 2025. Enhancing software development effort estima- tion with a cloud-based data framework using use case points, fuzzy logic, and machine learning. Discover Computing 28, 1 (2025), 143

    2025

  21. [32]

    Ángel J Sánchez-García, María Saarayim González-Hernández, Karen Cortés- Verdín, and Juan Carlos Pérez-Arriaga. 2024. Software Estimation in the Design Stage with Statistical Models and Machine Learning: An Empirical Study. Math- ematics 12, 7 (2024), 1058

    2024

  22. [33]

    Baptiste Roziere, Loubna Allal, Lewis Tunstall, et al. 2023. Code Llama: Open Foundation Models for Code. arXiv preprint arXiv:2308.12950 (2023). https: //arxiv.org/abs/2308.12950

    2023 arXiv

  23. [34]

    Wilamis KN Silva, Bernan R Nascimento, Péricles Miranda, and Emanuel P Vicente. [n. d.]. Predictive Regression Models of Machine Learning for Effort Estimation in Software Teams: An Experimental Study. ([n. d.])

  24. [35]

    Md Mahfuzul Islam Shamim, Abu Bakar bin Abdul Hamid, Tadiwa Elisha Nya- masvisva, and Najmus Saqib Bin Rafi. 2025. Advancement of Artificial Intelligence in Cost Estimation for Project Management Success: A Systematic Review of Machine Learning, Deep Learning, Regression, and ...

    2025

  25. [36]

    Karthik Valmeekam, Alberto Olmo, Sarath Sreedharan, and Subbarao Kambham- pati. 2022. Large language models still can’t plan (a benchmark for LLMs on planning and reasoning about change). In NeurIPS 2022 Foundation Models for Decision Making Workshop

    2022

  26. [37]

    Victor Uc-Cetina. 2023. Recent advances in software effort estimation using machine learning. arXiv preprint arXiv:2303.03482 (2023)

    2023 arXiv

  27. [38]

    Jiexin Wang, Xitong Luo, Liuwen Cao, Hongkui He, Hailin Huang, Jiayuan Xie, Adam Jatowt, and Yi Cai. 2024. Is your ai-generated code really safe? evaluating large language models on secure code generation with codeseceval.arXiv preprint arXiv:2407.02395 (2024)

    2024 arXiv

  28. [39]

    Axel van Lamsweerde. 2009. Requirements Engineering - From System Goals to UML Models to Software Specifications . Wiley. http://eu.wiley.com/WileyCDA/ WileyTitle/productCd-EHEP000863.html

    2009

  29. [40]

    Fangzhou Wu, Xiaogeng Liu, and Chaowei Xiao. 2023. Deceptprompt: Exploiting llm-driven code generation via adversarial natural language instructions. arXiv preprint arXiv:2312.04730 (2023)

    2023 arXiv

  30. [41]

    Shiqi Wang, Zheng Li, Haifeng Qian, Chenghao Yang, Zijian Wang, Mingyue Shang, Varun Kumar, Samson Tan, Baishakhi Ray, Parminder Bhatia, Ramesh Nal- lapati, Murali Krishna Ramanathan, Dan Roth, and Bing Xiang. 2022. ReCode: Ro- bustness Evaluation of Code Generation Models. ar...

    2022 arXiv

  31. [42]

    Zihan Zheng, Shahad Hardan, Darya Taratynova, Abdelmajid Essofi, Karthik Nandakumar, and Mohammad Yaqub. 2025. LiveCodeBench Pro: How Do Olympiad Medalists Judge LLMs in Competitive Programming? arXiv preprint arXiv:2506.11928 (2025). https://arxiv.org/abs/2506.11928 Evaluatin...

    2025 arXiv

  32. [43]

    Burak Yetistiren, Isik Ozsoy, and Eray Tuzun. 2022. Assessing the quality of GitHub copilot’s code generation. In Proceedings of the 18th international confer- ence on predictive models and data analytics in software engineering . 62–71

    2022

  33. [44]

    Kaijie Zhu, Qinlin Zhao, Hao Chen, Jindong Wang, and Xing Xie. 2024. Prompt- bench: A unified library for evaluation of large language models. Journal of Machine Learning Research 25, 254 (2024), 1–22

    2024

  34. [45]

    Zibin Zheng, Kaiwen Ning, Qingyuan Zhong, Jiachi Chen, Wenqing Chen, Lianghong Guo, Weicheng Wang, and Yanlin Wang. 2025. Towards an un- derstanding of large language models in software engineering tasks. Empirical Software Engineering 30, 2 (2025), 50

    2025

  35. [2022]

    Software: Practice and experience 52, 1 (2022), 39–65

    Software effort estimation accuracy prediction of machine learning tech- niques: A systematic performance evaluation. Software: Practice and experience 52, 1 (2022), 39–65

    2022

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.