Why you shouldn't fully trust ChatGPT: A synthesis of this AI tool's error rates across disciplines and the software engineering lifecycle

Vahid Garousi

REVIEW 5 major objections 5 minor 26 references

Why you shouldn't fully trust ChatGPT: A synthesis of this AI tool's error rates across disciplines and the software engineering lifecycle

T0 review · 5 major / 5 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read ChatGPT's error rates are non-negligible and task-dependent, ranging from about 5% in structured drafting to over 80% in rare-disease diagnosis, so full automation without human oversight is not yet safe.

desk verdict A timely review with a sensible qualitative warning, but the quantitative error-rate synthesis is unverifiable as presented and needs heavy revision before it can be trusted. read the letter →

arxiv 2504.18858 v1 pith:ODETKOGR submitted 2025-04-26 cs.SE cs.AI

Vahid Garousi This is my paper · ORCID

classification cs.SEcs.AI

keywords ChatGPTerrorrateslargelanguagemodelsmultivocalliteraturereviewsoftwaredevelopmentlifecyclereliabilityhumanoversight

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper is a synthesis of published and grey-literature measurements of ChatGPT's error rates. It aggregates studies across healthcare, business, economics, engineering, computer science, and software engineering, converting reported success rates into error rates and grouping them by domain and by software development phase. The central claim is that ChatGPT's errors are non-negligible and unevenly distributed: typical ranges run from roughly 8% to 83% in healthcare, about 5% to 20% in early design work, and 10% to 50% in coding, testing, and maintenance, with newer model versions consistently lowering but not eliminating errors. A fair reader should come away with a concrete map of where ChatGPT output can serve as a draft and where it must be checked by a human.

What carries the argument

The carrying mechanism is the multivocal literature review itself, specifically its synthesis rule: each included study's reported accuracy, pass rate, or success rate is converted to an error rate by computing $(100\% - \text{accuracy})$, and the resulting data points are grouped into boxplots per domain and per software development lifecycle phase. Those boxplots carry the argument: they let the paper claim visible patterns in the spread and center of reported errors—narrow, lower boxes for requirements and design versus wider, higher boxes for coding, testing, and maintenance—rather than relying on any single benchmark.

What would settle it

Re-run the synthesis from the raw scored outputs of the cited studies, recording each study's metric type (exact-match accuracy, pass rate, user acceptance) and recomputing errors without the uniform 100%-minus-accuracy shortcut; if the recomputed domain ranges differ materially from the reported 8% to 83% healthcare span or the 5% to 20% versus 10% to 50% lifecycle split, the headline figures collapse.

Watch

Extended reading notes

Core claim

The paper's central discovery, stated as a synthesized empirical result, is that ChatGPT's reliability varies sharply with task, domain, and model version, and that no domain or software development lifecycle phase is error-free. The concrete numbers are: 28% error in GPT-3.5 clinical decision-making and 23% in GPT-4 final diagnosis, rising to 83% in rare-disease diagnosis; accounting exam errors falling from roughly 47% with GPT-3.5 to 15% with GPT-4; an economics midterm error rate dropping from 69% to 27%; coding success up to 87.5% yet 52% incorrect answers on real-world programming questions; and software engineering requirements and design phases at roughly 5% to 20% errors versus 10% to 50% for implementation, testing, and maintenance. The paper concludes that full reliance on ChatGPT without human oversight remains risky, especially in high-stakes settings.

Load-bearing premise

The synthesis assumes that every reported accuracy, pass rate, or success rate can be read as a direct error rate by subtracting it from 100%, and that the phase-level ranges labeled 'estimated' rest on the cited studies rather than the paper's own interpolation; if that conversion and those estimates are wrong, the boxplot ranges are not reliable measurements.

Editorial extensions

If this is right

Treat every ChatGPT output as a draft that enters a validation pipeline, with review intensity scaled to the observed error range of the task.
Prioritize human review in healthcare diagnosis and in coding, testing, and maintenance, where the reported error ceilings are highest.
Version upgrades from GPT-3.5 to GPT-4 lower error rates but do not make any phase safe, so upgrade decisions should not replace oversight.
Structured, context-rich prompts are a measurable lever for reducing errors and should be treated as part of the engineering process.
Because fluent wrong answers are sometimes rated as high quality, user confidence in ChatGPT output is not a reliability signal.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

A natural extension the paper does not spell out is a risk-tiering rule: allocate the heaviest human review to debugging, merge, and refactoring tasks, whose observed error ranges are roughly twice those of requirements drafting.
The conversion rule used here probably understates true errors on multiple-choice tasks where guessing or partial credit inflates accuracy, so the reported lower bounds should be read as optimistic in those settings.
If the version-upgrade trend continues, the same benchmarks should show GPT-5-era models pushing common-condition diagnostic errors below 20% while rare-disease and open-ended debugging errors stay above 30%; that is a testable prediction.
The divergence between objective correctness and user-rated quality reported in the underlying studies implies that error-rate tracking and user-trust tracking should be run side by side, not separately.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

What should I tell you about arXiv:2504.18858? It's a review paper that aggregates reported ChatGPT error rates across domains and software engineering lifecycle phases. The qualitative core is sensible and the high-level warning—don't fully trust ChatGPT without oversight—holds up. The quantitative narrative, though, is wobbly. Several headline ranges are labelled 'estimated' with no derivation, and the synthesis converts heterogeneous metrics into a single error-rate scale without enough justification. The limitations section honestly admits that error rates were inferred and model versions guessed, but the paper doesn't give you the underlying data to check any of it.

What's genuinely useful: it collects a scattered set of evaluations into one place, groups them by domain and SDLC phase, and draws a distinction between early phases (requirements, design, with lower reported errors) and later phases (implementation, testing, maintenance, with wider variance). That framing is a reasonable organizer for practitioners trying to decide where to put human review. The boxplots are a decent visual summary, although the underlying numbers are shaky. The paper also correctly notes that model upgrades matter and that fluent output can mask incorrect answers.

Where it's soft: Section 2.2 says error = 100% - accuracy, but the source studies use pass rates, grades, test-case success, and user acceptance. Those aren't interchangeable. Sections 4.2, 4.4, and 4.5 give ranges like 5–20%, 10–30%, 30–50% with the word 'estimated' but no explanation of how those estimates were computed from the cited references. There's no data table, no search string, no screening count. The reference list also has misattributions: [14] is a multitask prompting paper, not evidence about ChatGPT oversimplifying designs, and [24] is about secure software development, not general Python test-case success. Those look like genuine citation errors, and they undercut confidence in the other attributions.

Net: the message is right, but the numbers are not load-bearing in their current form. A reader who wants the qualitative takeaway can get it from the abstract. A reader who wants to rely on the specific ranges can't verify them. If the author were to resubmit with the extraction table, explicit mapping of metrics, and defense of the estimated ranges, the paper would be much stronger. As it stands, it's a useful map with some unreliable coordinates.

My recommendation: this deserves a serious referee, not a desk reject, but the referee should be told to focus on reproducibility. I would not cite it in its current form, but I'd bring it to the reading group to talk about what counts as evidence in LLM reviews.

Referee Report

5 major / 5 minor

Summary. This manuscript reports a multivocal literature review (MLR) of ChatGPT error rates across broad domains and across software engineering lifecycle (SDLC) phases. The author compiles reported accuracy/error figures from academic and grey sources, converts accuracy to error rates where needed, groups results by domain (healthcare, business, economics, engineering, computer science) and SDLC phase (requirements, design, implementation, testing, maintenance), and visualizes the distributions in boxplots. The headline findings are that error rates are substantial and variable, for example 8–83% in healthcare, 5–20% in requirements/design, and 10–50% in implementation/testing/maintenance, with GPT-4 generally outperforming GPT-3.5. The conclusion recommends human oversight and validation before relying on ChatGPT outputs in professional settings.

Significance. The question addressed is timely and practically important, and the qualitative observation that LLM error rates vary by domain and task and that human oversight is needed is plausible and broadly consistent with the wider literature. If the quantitative synthesis were reliable, the paper could serve as a useful reference for practitioners deciding where to invest in human-in-the-loop review. However, the paper's contribution rests on a quantitative synthesis whose method is not reproducible and whose key ranges are partly based on unstated author estimates; the quantitative claims are not currently supported. The paper explicitly acknowledges several threats to validity in Section 5.2, which is commendable, but those limitations directly undermine the central numeric results rather than merely qualifying them.

major comments (5)

[§2.2] The rule 'Where accuracy rather than error rates were reported, we inferred error rates as (100% – accuracy %)' is applied to studies whose metrics are not commensurable: exam pass rates, grade rubrics, code test-case pass rates, user acceptance ratings, and diagnostic accuracy. A 47% CPA-exam error rate is not the same kind of quantity as an 18% test-case failure rate or a 52% 'incorrect or partially incorrect' Stack Overflow answer rate, so pooling them into Figures 1 and 2 produces distributions without a well-defined interpretation. This is load-bearing because the headline domain- and phase-specific ranges are drawn from these pooled boxplots.
[§2.1–2.2] The review is not reproducible as reported. The text does not provide the search strings, the databases and grey-literature sources with search dates, the number of records retrieved, screened, and included, the inclusion/exclusion decisions, or a data extraction table linking each data point to its source and metric. Without these, a reader cannot verify that the boxplots reflect the stated evidence base. This falls below the reporting standard expected for an MLR and for any quantitative synthesis.
[§4.2, §4.4, §4.5] The central SDLC-phase ranges are introduced as 'estimated' without derivation: for example, 'Estimated error rates in preliminary design suggestions ranged between 5% and 20%' (§4.2), 'estimated error rates of around 10–30%' for testing (§4.4), and a code-review miss rate of 'approximately 30–50%' (§4.5). The cited paragraphs describe qualitative observations about oversimplification, missing boundary cases, and missed subtle flaws, but no calculation, mapping, or source-specific numbers are given in the text or in any supplementary table. Since these estimates are the quantitative basis of Figure 2 and of the conclusion that requirements/design are safer than implementation/testing/maintenance, the claim is not supported by the cited evidence as presented.
[§3.5, §4.2, references] Several reference assignments are not credible. Reference [14], cited in §2.2, §4.2, and §5.2, is titled 'Multitask Prompted Training Enables Zero-Shot Task Generalization' and does not appear to be a study of ChatGPT oversimplifying software designs; reference [24], titled 'Evaluating LLMs for Secure Software Development,' is cited in §3.5 and §4.3 for Python test-case success rates (81.96% and 87.5%). If these data points cannot be located in the cited sources, the quantitative claims drawn from them are unverifiable. The author should provide exact locations (table or figure numbers) or replace the citations.
[§5.2] The limitations section concedes that 'we had to infer consistent error rates in some cases' and that model versions were sometimes inferred from publication dates or context, but no sensitivity analysis or robustness check is provided. For a paper whose main output is numeric error ranges, the effect of these inferences and of the acknowledged reporting inconsistencies must be quantified or at least bounded; otherwise the ranges cannot be distinguished from the author's priors.

minor comments (5)

[References] The reference list contains duplicate entries: [1] and [16], [2] and [18], [3] and [19], [4] and [20], [8] and [24], and [9] and [25] are the same works. This will confuse readers and suggests the list was not carefully curated.
[References] Reference [15] (Tukey, Exploratory Data Analysis) is listed but not cited in the text; please cite it where boxplots are discussed or remove it.
[Figures 1 and 2] The figures are not reproducible from the text alone: no data points, sample sizes, or per-study values are given. Even if the figures appear correctly in the compiled PDF, the paper should include a data table or online appendix listing the underlying values.
[Abstract and §3.4] The abstract states 'Engineering tasks averaged 20–30%' while §3.4 reports 25% for GPT-4 on environmental engineering and 40–50% for GPT-3.5 on mechanical engineering; please reconcile or clarify which model and task set the 'average' covers.
[§2.2] The section heading appears as '2.2 2.2 Data Extraction and Synthesis' in the manuscript text; the duplicated number should be removed.

Circularity Check

2 steps flagged · score 5.0 of 10

Partial circularity in SDLC-phase ranges: author-'estimated' design (5–20%) and testing (10–30%) values are inserted into the synthesis and then reported as findings; domain-level numbers remain externally grounded.

fitted input called prediction [Section 4.2 (Design); reused in Abstract and Section 6]
"Estimated error rates in preliminary design suggestions ranged between 5% and 20%, depending on system complexity."

The 5–20% design range is introduced with the word 'estimated' and is not derived from a reported measurement in the cited studies; the cited reference [14] is a multitask prompting paper, not a design-evaluation study. The paper later reports 'requirements and design phases showed the lowest error rates (~5–20%)' in the Abstract and 'relatively low error rates during requirements and design phases (~5–20%)' in Section 6. The design portion of that finding is the same author-supplied estimate recycled as a synthesized result: the output equals the input by construction, with no independent primary-study number behind it.
fitted input called prediction [Section 4.4 (Testing); feeds Abstract and Section 5.1]
"ChatGPT occasionally assumed incorrect function behaviors or omitted key boundary cases, leading to estimated error rates of around 10–30%."

The 10–30% 'estimated' range is not shown to be computed from the cited study [6]; it is the author's interpolation. This estimate is then included in the Testing-phase boxplot and contributes to the Abstract's claim that 'coding, testing, and maintenance phases had higher variability (10–50%)' and to Section 5.1's 'implementation, testing, and maintenance activities showed higher variability and risk (~10–50%)'. Thus part of the phase-specific quantitative claim is composed of the author's own estimate, making that sub-claim partially circular.

full rationale

This is a literature-review synthesis, so there is no formal first-principles derivation, and most domain-level results (e.g., healthcare 8–83%, accounting ~47%→~15%, economics 69%→27%, Python test-case ~18% and ~12.5%) are taken from cited primary studies and are externally grounded. The central claim that ChatGPT makes non-negligible errors is supported by many independent studies, and there is no load-bearing self-citation, no imported uniqueness theorem, and no ansatz smuggled by citation. However, the SDLC-phase 'findings' are partly circular: Sections 4.2 and 4.4 introduce author-'estimated' ranges (5–20% for design, 10–30% for testing) that are then reported in the abstract and conclusion as empirical synthesized error-rate ranges. Section 5.2 explicitly concedes 'we had to infer consistent error rates in some cases, which introduces interpretation bias,' confirming that the aggregate contains the author's own inference. Because these specific numbers reduce to the author's estimate by construction, the paper is not fully clean; score 5 reflects partial circularity in a subset of the quantitative claims while the broader reliability message remains independently supported. It is not scored 8–10 because the primary domain-level findings do not reduce to the author's inputs and no self-citation chain forces the result.

Assumptions & free parameters 5 free parameters · 3 assumptions · 0 invented entities

The paper contributes no primary data; its quantitative ranges are either lifted from individual studies or are the author's 'estimated' ranges. The latter are free parameters in the synthesis. The main structural assumption is that heterogeneous error metrics can be merged into one scale.

free parameters (5)

design-phase error range = 5-20%
Section 4.2 states 'Estimated error rates in preliminary design suggestions ranged between 5% and 20%' with no source calculation; the range is supplied by the author.
testing-phase error range = 10-30%
Section 4.4 states 'estimated error rates of around 10-30%' for ChatGPT testing assistance, without a reproducible computation.
code-review miss rate = 30-50%
Section 4.5 states ChatGPT 'missed subtle logic flaws approximately 30-50% of the time' in code review; no underlying data is shown.
refactoring error range = 10-20%
Section 4.5 states refactoring had 'minor risk (~10-20%) of inadvertently changing functionality'.
documentation error ceiling = below 10%
Section 4.5 states documentation updates 'showed low error rates (often below 10%)'.

assumptions (3)

domain assumption Error rate equals 100% minus reported accuracy or pass rate
Section 2.2: 'Where accuracy rather than error rates were reported, we inferred error rates as (100% - accuracy %).' Assumes a single error scale across heterogeneous metrics.
domain assumption Studies grouped into five domains and five SDLC phases are comparable
Section 2.3 groups results by domain and SDLC phase despite different tasks, models, and evaluation formats.
domain assumption Cited primary studies are accurately represented
The synthesis relies on the accuracy of the cited studies and the fidelity of the author's extraction; citation mismatches in [14], [29], and [30] weaken this assumption.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Why you shouldn't fully trust ChatGPT: A synthesis of this AI tool's error rates across disciplines and the software engineering lifecycle." pith.science (2026). https://pith.science/paper/ODETKOGR

@misc{pith2026250418858,
  author       = {Pith},
  title        = {Pith review of: Why you shouldn't fully trust ChatGPT: A synthesis of this AI tool's error rates across disciplines and the software engineering lifecycle},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ODETKOGR}},
  note         = {Machine review of arXiv:2504.18858}
}

read the original abstract

Context: ChatGPT and other large language models (LLMs) are widely used across healthcare, business, economics, engineering, and software engineering (SE). Despite their popularity, concerns persist about their reliability, especially their error rates across domains and the software development lifecycle (SDLC). Objective: This study synthesizes and quantifies ChatGPT's reported error rates across major domains and SE tasks aligned with SDLC phases. It provides an evidence-based view of where ChatGPT excels, where it fails, and how reliability varies by task, domain, and model version (GPT-3.5, GPT-4, GPT-4-turbo, GPT-4o). Method: A Multivocal Literature Review (MLR) was conducted, gathering data from academic studies, reports, benchmarks, and grey literature up to 2025. Factual, reasoning, coding, and interpretive errors were considered. Data were grouped by domain and SE phase and visualized using boxplots to show error distributions. Results: Error rates vary across domains and versions. In healthcare, rates ranged from 8% to 83%. Business and economics saw error rates drop from ~50% with GPT-3.5 to 15-20% with GPT-4. Engineering tasks averaged 20-30%. Programming success reached 87.5%, though complex debugging still showed over 50% errors. In SE, requirements and design phases showed lower error rates (~5-20%), while coding, testing, and maintenance phases had higher variability (10-50%). Upgrades from GPT-3.5 to GPT-4 improved reliability. Conclusion: Despite improvements, ChatGPT still exhibits non-negligible error rates varying by domain, task, and SDLC phase. Full reliance without human oversight remains risky, especially in critical settings. Continuous evaluation and critical validation are essential to ensure reliability and trustworthiness.

Figures

Figures reproduced from arXiv: 2504.18858 by the authors.

**Figure 2.** Boxplot of reported ChatGPT error rates by SDLC phase [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

26 extracted references · 22 canonical work pages

[14]

Multitask Prompted Training Enables Zero-Shot Task Generalization,

S. Bang, et al., "Multitask Prompted Training Enables Zero-Shot Task Generalization," arXiv preprint arXiv:2301.04704, 2023

arXiv 2023
[24]

Evaluating LLMs for Secure Software Development,

M. Alam, et al., "Evaluating LLMs for Secure Software Development," ACM Digital Threats, 2024

work page 2024
[5]

Exploring LLMs for Software Requirements Engineering,

H. Marques, et al., "Exploring LLMs for Software Requirements Engineering," IEEE Software, 2023

work page 2023
[6]

Text-Blueprint: An Interactive Platform for Plan-based Conditional Generation

M. Zhang, et al., "ChatGPT for Software Testing: Promise and Challenges," arXiv preprint arXiv:2305.00034, 2023

work page Pith review arXiv 2023
[7]

AI in Software Architecture: Opportunities and Risks,

M. Ozkaya, "AI in Software Architecture: Opportunities and Risks," Software Architecture Workshop, 2023

work page 2023
[10]

Guidelines for performing systematic literature reviews in software engineering,

B. Kitchenham and S. Charters, "Guidelines for performing systematic literature reviews in software engineering," EBSE Technical Report, 2007

work page 2007
[11]

Software Engineering Research Using Multivocal Literature Reviews,

B. Carver, et al., "Software Engineering Research Using Multivocal Literature Reviews," Empirical Software Engineering, vol. 21, no. 5, pp. 1913–1966, 2016

work page 1913
[12]

Empirical Studies of Large Language Models: A Systematic Literature Review,

N. Hosseini, et al., "Empirical Studies of Large Language Models: A Systematic Literature Review," arXiv preprint arXiv:2305.07998, 2023

arXiv 2023

Show all 26 references

[13]

Evaluating ChatGPT: Current Research and Future Directions,

E. Chen, et al., "Evaluating ChatGPT: Current Research and Future Directions," arXiv preprint arXiv:2305.00548, 2023

2023 arXiv
[15]

Tukey, Exploratory Data Analysis, Addison-Wesley, 1977

J. Tukey, Exploratory Data Analysis, Addison-Wesley, 1977

1977
[16]

Assessing the accuracy of GPT-3.5 and GPT-4 in clinical decision-making,

J. S. Patel, et al., "Assessing the accuracy of GPT-3.5 and GPT-4 in clinical decision-making," Journal of Medical AI, 2023

2023
[17]

Evaluation of LLMs for Lab Test Interpretations,

M. B. Goyal, et al., "Evaluation of LLMs for Lab Test Interpretations," JAMA Open Network, 2023

2023
[18]

Would ChatGPT get a Wharton MBA?

C. Terwiesch, "Would ChatGPT get a Wharton MBA?" University of Pennsylvania Working Paper, 2023

2023
[19]

ChatGPT’s accounting exam results reveal striking improvement,

D. Curtis, "ChatGPT’s accounting exam results reveal striking improvement," CFO Dive, 2023

2023
[20]

GPT-3.5 versus GPT-4 on my midterm exam,

B. Caplan, "GPT-3.5 versus GPT-4 on my midterm exam," EconLog, 2023

2023
[21]

The Rise of the Machine: How Well Can ChatGPT Perform on Undergraduate Economics Exams?

G. Geerling, "The Rise of the Machine: How Well Can ChatGPT Perform on Undergraduate Economics Exams?", SSRN Electronic Journal, 2023

2023
[22]

Evaluating ChatGPT’s performance on FE Environmental Engineering exams,

S. Guo and J. Crittenden, "Evaluating ChatGPT’s performance on FE Environmental Engineering exams," Engineering Education Review, 2024. 9

2024
[23]

GPT-4 and Engineering Exams: An Early Study,

R. Rahimi, "GPT-4 and Engineering Exams: An Early Study," Mechanical Engineering Research, 2024

2024
[25]

Do Users Trust AI on Stack Overflow? An Empirical Study,

S. Ye, et al., "Do Users Trust AI on Stack Overflow? An Empirical Study," arXiv preprint arXiv:2308.00064, 2023

2023 arXiv
[26]

ChatGPT for Requirements Validation: Opportunities and Risks,

A. Ahmed, et al., "ChatGPT for Requirements Validation: Opportunities and Risks," Requirements Engineering Journal, 2024

2024
[27]

LLMs for Software Development: Promise and Peril,

J. Gao, et al., "LLMs for Software Development: Promise and Peril," arXiv preprint arXiv:2305.09876, 2023

2023 arXiv
[28]

Assessing ChatGPT for Software Maintenance Tasks,

E. Nguyen, et al., "Assessing ChatGPT for Software Maintenance Tasks," IEEE Transactions on Software Engineering, 2024

2024
[29]

Prompting Large Language Models: Survey and Challenges,

S. Min, et al., "Prompting Large Language Models: Survey and Challenges," arXiv preprint arXiv:2307.04652, 2023

2023 arXiv
[30]

Assessing User Trust in AI-Generated Answers: A Case Study on Stack Overflow,

D. Hohensee, et al., "Assessing User Trust in AI-Generated Answers: A Case Study on Stack Overflow," arXiv preprint arXiv:2306.00800, 2023

2023 arXiv
[31]

Prompt Optimization for LLMs: Techniques and Benchmarks,

S. Zhou, et al., "Prompt Optimization for LLMs: Techniques and Benchmarks," arXiv preprint arXiv:2310.08458, 2023

2023 arXiv
[32]

How Is ChatGPT’s Behavior Changing Over Time?

S. Chen, et al., "How Is ChatGPT’s Behavior Changing Over Time?", Stanford HAI Working Paper, 2023

2023

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.

[1] [14]

Multitask Prompted Training Enables Zero-Shot Task Generalization,

S. Bang, et al., "Multitask Prompted Training Enables Zero-Shot Task Generalization," arXiv preprint arXiv:2301.04704, 2023

arXiv 2023

[2] [24]

Evaluating LLMs for Secure Software Development,

M. Alam, et al., "Evaluating LLMs for Secure Software Development," ACM Digital Threats, 2024

work page 2024

[3] [5]

Exploring LLMs for Software Requirements Engineering,

H. Marques, et al., "Exploring LLMs for Software Requirements Engineering," IEEE Software, 2023

work page 2023

[4] [6]

Text-Blueprint: An Interactive Platform for Plan-based Conditional Generation

M. Zhang, et al., "ChatGPT for Software Testing: Promise and Challenges," arXiv preprint arXiv:2305.00034, 2023

work page Pith review arXiv 2023

[5] [7]

AI in Software Architecture: Opportunities and Risks,

M. Ozkaya, "AI in Software Architecture: Opportunities and Risks," Software Architecture Workshop, 2023

work page 2023

[6] [10]

Guidelines for performing systematic literature reviews in software engineering,

B. Kitchenham and S. Charters, "Guidelines for performing systematic literature reviews in software engineering," EBSE Technical Report, 2007

work page 2007

[7] [11]

Software Engineering Research Using Multivocal Literature Reviews,

B. Carver, et al., "Software Engineering Research Using Multivocal Literature Reviews," Empirical Software Engineering, vol. 21, no. 5, pp. 1913–1966, 2016

work page 1913

[8] [12]

Empirical Studies of Large Language Models: A Systematic Literature Review,

N. Hosseini, et al., "Empirical Studies of Large Language Models: A Systematic Literature Review," arXiv preprint arXiv:2305.07998, 2023

arXiv 2023