Pith. sign in

REVIEW 3 major objections 5 minor 1 cited by

Can LLMs Generate User Stories and Assess Their Quality?

T0 review · 3 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Given clear evaluation criteria, large language models can judge the semantic quality of user stories as reliably as human experts, even though their own stories match human coverage but not human diversity or rationale.

desk verdict Worth a careful look: a big empirical study of LLMs for user-story generation and evaluation, but the headline kappa claim compares apples to oranges. read the letter →

arxiv 2507.15157 v1 pith:FW37R3DX submitted 2025-07-20 cs.SE cs.AI

Giovanni Quattrocchi , Liliana Pasquale , Paola Spoletini , Luciano Baresi This is my paper
classification cs.SEcs.AI
keywords LargeLanguageModelsRequirementEngineeringElicitationQualityUserStoriesAgileMethodssemanticassessmentmulti-agent
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Requirements elicitation — extracting what a system should do from stakeholder interviews — and the semantic quality review of the resulting user stories are largely manual, slow steps in agile software development. This paper asks whether large language models can do either step, and its answer is a qualified yes: generation succeeds on coverage and style, while assessment succeeds when the model is handed an explicit codebook. With the paper's full codebook, the best model agreed with human labels at $\kappa = 0.87$, above the $0.74$–$0.78$ agreement the human annotators reached with each other. For generation, the ten tested models covered the reference requirements better than the thirty students did in the replicated experiment, but their output was far less diverse and they produced fewer stories meeting all acceptance criteria. The practical upshot is a division of labor: LLMs as semi-automated drafters and first-pass quality screeners, with humans supplying diversity, rationale, and final judgment.

What carries the argument

The argument rests on a multi-agent replication of the Interview-Based Learning (IBL) elicitation process, in which 30 instances of each of 10 LLMs act as interviewing analysts and a single instance plays the customer, holding the 53 ground-truth user stories. Three measuring instruments carry the evaluation. Coverage and diversity come from embedding-based cosine similarity between story sets, with thresholds $c = 0.8$ and $c = 0.85$ calibrated by human agreement on story pairs. The assessment results come from a codebook: the rules and examples that three human annotators distilled while labeling 153 stories on five semantic quality metrics (feature specificity, rationale clarity, problem orientation, language clarity, internal consistency), which the LLMs receive in three prompt conditions. Agreement with human labels is measured by a conflict-aware Cohen's kappa that counts only disagreements where one side rated a story non-acceptable; syntactic quality is measured separately by the AQUSA tool, a rule-based checker of user-story structure.

What would settle it

Run the same multi-agent interview protocol on a private, never-published set of user stories for a new domain and compare coverage. If coverage on the private set falls far below the reported $96\%$, the headline coverage result is an artifact of training-data overlap rather than a capability for interview-based elicitation.

Watch

Extended reading notes

Core claim

The paper's central claim is that LLMs can reliably assess the semantic quality of user stories when they are given clear evaluation criteria, and can generate user stories that rival human ones in coverage and stylistic quality. The evidence: in a replicated interview-based elicitation experiment, all ten LLMs covered the ground-truth requirements at higher rates than the human students (up to $96.23\%$ at cosine threshold $c = 0.8$, versus the students' $52.83\%$), while no LLM approached the students' diversity ($98.58\%$ average pairwise diversity versus a $44.23\%$–$74.74\%$ range across models). For assessment, Claude 3 Opus equipped with the complete codebook reached a conflict-aware Cohen's kappa of $0.87$ against human labels, exceeding the average human-human agreement of $0.74$–$0.78$; even without a codebook its agreement stayed substantial. The paper also establishes a quality gap: LLM-generated stories passed acceptance criteria less often than human ones, with rejection rates ranging from roughly $16\%$ for the best generators to $77\%$ for GPT-3.5, driven mainly by excessive conjunctions and weak rationale clarity. The authors conclude that LLMs can support, but not replace, human analysts: they are strongest on structurally grounded criteria (language clarity, internal consistency) and weakest where the 'why' behind a requirement must be made explicit.

Load-bearing premise

The load-bearing premise is that the 53 ground-truth stories given to the customer model had not already been memorized by the analyst models during training; if those stories are in the training data, the high coverage scores would measure retrieval from memory rather than successful interview-based elicitation.

Editorial extensions

If this is right

  • Agile teams can hand the first elicitation pass to an LLM: interview a simulated customer, get a draft set of user stories, and spend human effort on diversifying and sharpening them.
  • Large backlog assessments can be semi-automated: a codebook-equipped LLM flags stories that fail acceptability criteria, reducing the manual labeling effort that today dominates large-scale quality review.
  • Syntactic screening with existing automated tools plus LLM-based semantic screening covers both quality layers, and the common defects the paper found (conjunction overuse, missing role or means) can be detected before stories reach developers.
  • Model selection should be task-specific: scale is not a reliable predictor, since smaller models sometimes beat larger ones on both diversity and defect rates, and the best evaluator was not the newest model.
  • The strongest LLM evaluator matched or exceeded human-human agreement, suggesting LLMs can act as an additional consistent rater in multi-annotator quality studies.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The coverage headline (up to $96\%$ at $c = 0.8$) is only interpretable if the analyst models had not memorized the 53 public ground-truth stories used to brief the customer; the paper's defense is a general citation on memorization, so a leak-free replication on a private dataset is the natural next check.
  • The codebook effect suggests a practical grading pipeline: an LLM rater with a codebook can serve as a first-pass screener, leaving humans to review only the stories the model marks non-acceptable, where the conflict-aware kappa is most informative.
  • Because rationale clarity is the weakest dimension across models, prompting that explicitly forces a separate 'so that' justification sentence may close much of the gap without needing larger models.
  • The diversity metric, being the complement of pairwise coverage, measures semantic redundancy; a lexical or structural diversity measure could change the ranking of models and would be a cheap addition to the published replication package.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper replicates Ferrari et al.'s interview-based requirements elicitation experiment using 10 LLMs as analysts and one LLM instance as a customer, producing 13,958 user stories. It evaluates these stories against a ground-truth set and against student-generated stories using embedding-based coverage, pairwise diversity, and GPTZero detectability (RQ1). It then asks the same LLMs to assess the semantic quality of 153 user stories under three prompt conditions (no codebook, partial codebook, full codebook), comparing their labels with human annotations via conflict-aware Cohen's kappa (RQ2). Finally, it evaluates the syntactic quality of generated stories with AQUSA and the semantic quality with the best-performing LLM, Claude 3 Opus (RQ3). The paper concludes that LLMs can generate user stories with high coverage and stylistic similarity but lower diversity and creativity, that they can reliably assess semantic quality when given clear criteria, and that their stories meet acceptance criteria less consistently than human-written ones.

Significance. If the claims hold, this is a useful large-scale empirical contribution to the emerging literature on LLM-supported requirements engineering. The study has notable strengths: a large generated corpus (13,958 stories), a direct replication of an established elicitation protocol, blind human annotation with a multi-stage reconciliation procedure, a publicly released codebook and replication package, and the use of an external syntactic tool (AQUSA). The finding that LLM-generated stories tend to have lower diversity and weaker rationale clarity is valuable and actionable. However, the paper's most prominent claim—that LLMs can assess semantic quality more reliably than human annotators—rests on a comparison between two differently constructed kappa statistics, and the coverage result is partly threatened by the way the ground truth is fed into the Customer LLM. These issues need to be addressed before the central conclusions can be accepted as stated.

major comments (3)
  1. [Section VI-B, Eq. (1), Tables V and VI] The claim that Claude 3 Opus's kappa of 0.87 'exceeds the average inter-human reliability' (0.74–0.78) compares two non-comparable metrics. Equation (1) defines a conflict only when one rater assigns 1 and the other assigns 2 or 3, so 2-vs-3 disagreements are ignored; the LLM-human kappa in Figure 3 and Table VI is computed in this conflict-aware, effectively binarized setting. Table V, however, reports human-human agreement on the original 3-point scale before conflict resolution, where 2-vs-3 mismatches count against agreement. Additionally, the LLM is compared against the reconciled consensus labels, whereas human-human kappa is pairwise between independent annotators. Both asymmetries inflate the apparent LLM advantage. The authors should recompute human-human kappa using the same conflict-aware binarization and, ideally, also compute LLM-human kappa on the original 3-point scale against individual annotators. Without this re-derivation, the central RQ2 conclusion is not supported as stated.
  2. [Section IV-A, Table II, Section VIII] The coverage result, which is the first headline finding, is partly by construction: the 53 ground-truth user stories are placed directly in the Customer LLM's context, and the Analyst LLM is asked to elicit requirements from that customer. High coverage of the GT set may therefore reflect propagation of the input through the interview, not independent elicitation skill. The threat that the Analyst LLMs may have memorized the GT stories during pretraining is addressed in Section VIII only with a general citation [23] and no leakage test. The authors should provide a concrete analysis, such as measuring n-gram or embedding-level overlap between the generated stories and the GT, or running an ablation in which the Customer LLM is not given the GT set, to demonstrate that the high coverage is not an artifact of memorization or direct information transfer.
  3. [Section VII-B, Section VIII] The semantic-quality evaluation for RQ3 uses Claude 3 Opus, which is also one of the story generators, as the sole evaluator for all sources including its own output. The paper acknowledges this bias risk in Section VIII but does not quantify it. Because one of the main conclusions is that LLM-generated stories meet acceptance criteria less frequently than human-written ones, the authors should provide a robustness check, for example by having a second LLM or a manually annotated subset evaluate the same stories, and show that the relative ordering of sources is stable under the choice of evaluator.
minor comments (5)
  1. [Section IV-A] There is a typo in the sentence 'each LLM instance operated in an independent context configured witt the default sampling temperature'; 'witt' should be 'with'.
  2. [Table I] The columns 'Bin', '% Match', '# Matches', and '# Count' are confusing because the table appears to show cumulative bins while the text says 'for each similarity bin ... 20 pairs'; please clarify how the pairs were sampled and how the percentages were computed.
  3. [Section VI] The text and Table VI refer to 'Overall κ' as the average across the five quality dimensions; this is an average of per-criterion kappas, not a single kappa computed over all labels. Please state this explicitly in the text and caption to avoid misinterpretation.
  4. [Section VI-B] The conflict-aware kappa is described verbally but the exact formula used to compute kappa from conflict/non-conflict outcomes is not given; please add the formula or a precise reference.
  5. [Table II] The row 'Total LLM Coverage' pools all LLM-generated stories; since coverage is existential ('at least one similar story'), pooling trivially increases coverage. This should be interpreted as an upper bound and the text should say so explicitly.

Circularity Check

3 steps flagged · score 4.0 of 10

RQ2's claim that LLMs beat inter-human agreement compares a conflict-aware (binarized) LLM-human kappa to a full-scale human-human kappa, and RQ3 lets Claude 3 Opus judge its own stories; RQ1 coverage is also scored against GT stories that were placed in the Customer LLM's context. These are partial circularities, not a fully self-referential derivation.

  1. other [Section VI (intro, Eq. (1)), Section VI-A Table V, Section VI-B Table VI]
    "To focus on the most meaningful cases of disagreement, we adopted a conflict-aware formulation of κ. Specifically, a disagreement between an LLM and human annotation was only counted when there was a conflict (see Equation 1). ... In some cases, such as Claude 3 Opus (κ = 0.87) and Claude 3.5 Sonnet (κ = 0.84), the agreement even exceeds the average inter-human reliability observed in our manual annotation phase."

    Equation (1) counts a disagreement only when one rater gives 1 and the other gives 2 or 3, collapsing 2 and 3 into one non-conflict category. The LLM-human kappa is computed in this conflict-aware space (Figure 3, Table VI), while the human-human baseline in Table V is 'Before Conflict Resolution' on the original 1/2/3 scale. The same human labels are thus scored under two different definitions, and the LLM side is definitionally relieved of 2-vs-3 disagreements. The claim that Claude 3 Opus (κ=0.87) exceeds human-human κ (0.74-0.78) is an artifact of applying Eq. (1) to only one side, so RQ2's headline comparison is not supported by the numbers as presented.

  2. other [Section VII-B / Table VIII; Section VIII (Internal Validity)]
    "We automated the annotation process using Claude 3 Opus, the best performing model from our earlier evaluation. This choice could introduce a bias in favor of Claude 3 Opus when it evaluates its own generated user stories."

    In RQ3, Claude 3 Opus is both the generator whose stories are in the evaluated set and the LLM used to score all stories. Its high self-scores in Table VIII are not independent measurements; they are the same model's opinion of its own output. The authors explicitly acknowledge this bias in Section VIII and still report the self-scores as comparative evidence. The ranking of Claude 3 Opus relative to other generators is therefore partly circular, with the same model on both sides of the evaluation. The broader RQ3 conclusions about relative weaknesses (e.g., Rationale Clarity) are less affected, but this specific result is self-referential.

1 more flagged steps
  1. self definitional [Section IV-A and Section V-A / Eq. (2), Table II]
    "Meanwhile, the additional LLM instance acting as the customer (Customer LLM) received the same GT US that were originally given to the human expert. ... Claude 3 Sonnet and Claude 3 Opus achieve a top coverage of 96.23% at the 0.8 threshold."

    The coverage target in Eq. (2) is the GT US set, and the same GT US set was placed verbatim in the Customer LLM's context before the interview. The coverage score therefore measures how much of the input context propagated through the dialogue, not an independent elicitation capability. Section VIII addresses only pretraining memorization via citation [23], not this in-context path by which the target stories were supplied to the pipeline. High coverage is thus partially guaranteed by construction, and interpreting it as evidence that LLMs are 'highly capable of capturing the core requirements' overstates the independence of the result.

full rationale

The paper is not a pure circularity case: the codebook was built from the annotators' reconciliation rather than from LLM outputs; AQUSA provides an external syntactic check; diversity and detectability are not definitionally tied to the generation inputs; and the only self-citation ([43]) is descriptive and not load-bearing. However, three steps undermine the independence of headline claims. First, RQ2's 'exceeds inter-human reliability' compares a conflict-aware (1-vs-2/3 binarized) LLM-human kappa to a 3-point-scale human-human kappa computed before conflict resolution, so the comparison is not on equal terms and the conclusion is an artifact of the asymmetric metric definition. Second, RQ3's semantic evaluation lets Claude 3 Opus judge its own stories; the authors acknowledge this bias. Third, RQ1's high coverage is measured against GT US that were placed in the Customer LLM's context, so the target of the metric is also an input to the pipeline; the memorization discussion does not test in-context propagation. These are partial, not total, circularities: the underlying kappa values and syntactic measurements may still be informative if re-derived with comparable metrics and an independent evaluator.

Assumptions & free parameters 1 free parameters · 4 assumptions · 0 invented entities

The study is empirical, so the ledger lists measurement assumptions and calibrated thresholds rather than mathematical axioms. The coverage and diversity numbers depend on the cosine threshold, which was fitted to a 120-pair human calibration study. The validity of the semantic evaluation rests on the authors' manual labels and on the assumption that Claude 3 Opus can evaluate stories without self-preference. No new entities are posited.

free parameters (1)
  • cosine similarity threshold c for semantic coverage = 0.8 (primary) and 0.85 (conservative)
    Table I: threshold chosen after 3 annotators assessed 120 story pairs in similarity bins; at 0.8, 72.5% of pairs were judged similar, at 0.85, 95%. This threshold sets every coverage and diversity percentage in Tables II and III.
assumptions (4)
  • domain assumption The 53 CamperPlus stories are a valid ground truth for the summer camp system, and the Customer LLM, prompted with these stories, emulates a human customer closely enough for comparison with student interviews.
    Sections II and IV.A: both the original experiment and the LLM replication give the customer the GT stories; if the LLM customer answers more completely or literally than the human customer, LLM coverage is inflated relative to students.
  • domain assumption OpenAI text-embedding-3-small cosine similarity is a valid proxy for semantic equivalence of user stories, and GPTZero's AI probability is a valid proxy for human-likeness.
    Section V: coverage, diversity, and detectability rest entirely on these external tools; no validation beyond the small threshold calibration is provided for the embedding, and GPTZero is used without version or date information.
  • domain assumption The three authors' reconciled labels on 153 stories form a valid gold standard for the five QUS semantic criteria.
    Section IV.B: LLM-human agreement in RQ2 is measured against these labels, and the codebook given to LLMs was written by the same annotators, so agreement may partly reflect the LLM learning the annotators' rubric.
  • domain assumption The LLMs under test were not trained on the GT dataset in a way that inflates coverage.
    Section VIII: defended only by a general citation on memorization (Carlini et al. 2023); no membership or leakage test is performed for CamperPlus.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Can LLMs Generate User Stories and Assess Their Quality?." pith.science (2026). https://pith.science/paper/FW37R3DX

@misc{pith2026250715157,
  author       = {Pith},
  title        = {Pith review of: Can LLMs Generate User Stories and Assess Their Quality?},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/FW37R3DX}},
  note         = {Machine review of arXiv:2507.15157}
}
read the original abstract

Requirements elicitation is still one of the most challenging activities of the requirements engineering process due to the difficulty requirements analysts face in understanding and translating complex needs into concrete requirements. In addition, specifying high-quality requirements is crucial, as it can directly impact the quality of the software to be developed. Although automated tools allow for assessing the syntactic quality of requirements, evaluating semantic metrics (e.g., language clarity, internal consistency) remains a manual and time-consuming activity. This paper explores how LLMs can help automate requirements elicitation within agile frameworks, where requirements are defined as user stories (US). We used 10 state-of-the-art LLMs to investigate their ability to generate US automatically by emulating customer interviews. We evaluated the quality of US generated by LLMs, comparing it with the quality of US generated by humans (domain experts and students). We also explored whether and how LLMs can be used to automatically evaluate the semantic quality of US. Our results indicate that LLMs can generate US similar to humans in terms of coverage and stylistic quality, but exhibit lower diversity and creativity. Although LLM-generated US are generally comparable in quality to those created by humans, they tend to meet the acceptance quality criteria less frequently, regardless of the scale of the LLM model. Finally, LLMs can reliably assess the semantic quality of US when provided with clear evaluation criteria and have the potential to reduce human effort in large-scale assessments.

Figures

Figures reproduced from arXiv: 2507.15157 by the authors.

Figure 1
Figure 1. An overview of our study. using AQUSA and the semantic metrics using the LLM that exhibits the best performance when answering RQ2. IV. USER STORY GENERATION AND ASSESSMENT A. Multi-Agent US Generation We generated US using LLMs by replicating the study described in Section III (step 1 ). We implemented the IBL process using LLM as agents acting as analysts (instead of students) and customers (instead of human exper… view at source ↗
Figure 2
Figure 2. Diversity across different LLMs and Students [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 3
Figure 3. Cohen’s Kappa between each LLM and the human-annotated ground truth (across all quality criteria). [PITH_FULL_IMAGE:figures/full_fig_p010_3.png] view at source ↗

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Automated Alignment between Elicitation Interviews and Requirements

    cs.CL 2025-10 conditional novelty 5.0 of 10

    TEXT2STORIES formalizes chunk–story alignment with correctness and completeness metrics; an LLM judge reaches 0.86 macro-F1 on manually labeled pairs.

Reference graph

Works this paper leans on

55 extracted references · 48 canonical work pages · cited by 1 Pith paper

  1. [23]

    Quantifying memorization across neural language models,

    N. Carlini, D. Ippolito, M. Jagielski, K. Lee, F. Tram `er, and C. Zhang, “Quantifying memorization across neural language models,” in Proceedings of the 11th International Conference on Learning Representations (ICLR), 2023. [Online]. Available: https://arxiv.org/abs/ 2202.07646

    arXiv 2023

  2. [1]

    Detecting Terminological Ambiguity in User Stories: Tool and Experimentation,

    F. Dalpiaz, I. Van Der Schalk, S. Brinkkemper, F. B. Aydemir, and G. Lucassen, “Detecting Terminological Ambiguity in User Stories: Tool and Experimentation,” Information and Software Technology , vol. 110, pp. 3–16, 2019

    work page 2019

  3. [2]

    Improving Agile Requirements: The Quality User Story Framework and Tool,

    G. Lucassen, F. Dalpiaz, J. M. E. M. van der Werf, and S. Brinkkemper, “Improving Agile Requirements: The Quality User Story Framework and Tool,” Requirements Engineering, vol. 21, no. 3, pp. 383–403, 2016

    work page 2016

  4. [3]

    Systematic Literature Mapping of User Story Research,

    A. R. Amna and G. Poels, “Systematic Literature Mapping of User Story Research,” IEEE access, vol. 10, pp. 51 723–51 746, 2022

    work page 2022

  5. [4]

    A Systematic Literature Review on Agile Requirements Engineering Practices and Challenges,

    I. Inayat, S. S. Salim, S. Marczak, M. Daneva, and S. Shamshirband, “A Systematic Literature Review on Agile Requirements Engineering Practices and Challenges,” Computers in human behavior , vol. 51, pp. 915–929, 2015

    work page 2015

  6. [5]

    Requirements Engineering Challenges and Practices in Large- Scale Agile System Development,

    R. Kasauli, E. Knauss, J. Horkoff, G. Liebel, and F. G. de Oliveira Neto, “Requirements Engineering Challenges and Practices in Large- Scale Agile System Development,” Journal of Systems and Software , vol. 172, p. 110851, 2021

    work page 2021

  7. [6]

    Forging High-Quality User Stories: Towards a Discipline for Agile Re- quirements,

    G. Lucassen, F. Dalpiaz, J. M. E. Van Der Werf, and S. Brinkkemper, “Forging High-Quality User Stories: Towards a Discipline for Agile Re- quirements,” in 2015 IEEE 23rd International Requirements Engineering Conference (RE). IEEE, 2015, pp. 126–135

    work page 2015

  8. [7]

    Bert: Pre-training of Deep Bidirectional Transformers for Language Understanding,

    J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “Bert: Pre-training of Deep Bidirectional Transformers for Language Understanding,” arXiv preprint arXiv:1810.04805, 2018

    arXiv 2018

Show all 55 references
  1. [8]

    Language Models Are Unsupervised Multitask Learners,

    A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, I. Sutskever et al., “Language Models Are Unsupervised Multitask Learners,”OpenAI Blog, vol. 1, no. 8, p. 9, 2019

    2019

  2. [9]

    Language Models Are Few-Shot Learners,

    T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askellet al., “Language Models Are Few-Shot Learners,” Advances in Neural Information Processing Systems, vol. 33, pp. 1877–1901, 2020

    1901

  3. [10]

    Large Language Models for Software Engineer- ing: Survey and Open Problems ,

    A. Fan, B. Gokkaya, M. Harman, M. Lyubarskiy, S. Sengupta, S. Yoo, and J. M. Zhang, “ Large Language Models for Software Engineer- ing: Survey and Open Problems ,” in 2023 IEEE/ACM International Conference on Software Engineering: Future of Software Engineering (ICSE-FoSE). IE...

    2023

  4. [11]

    Evaluating Large Language Models in Class-Level Code Generation,

    X. Du, M. Liu, K. Wang, H. Wang, J. Liu, Y . Chen, J. Feng, C. Sha, X. Peng, and Y . Lou, “Evaluating Large Language Models in Class-Level Code Generation,” in 2024 IEEE/ACM 46th International Conference on Software Engineering (ICSE). IEEE Computer Society, 2024, pp. 865– 865

    2024

  5. [12]

    Studying LLM Performance on Closed-and Open-source Data,

    Ahmed, T. and Bird, C. and Devanbu, P. and Chakraborty, S., “Studying LLM Performance on Closed-and Open-source Data,” arXiv preprint arXiv:2402.15100, 2024

    2024 arXiv

  6. [13]

    Inferfix: End-to-end Program Repair with LLMs,

    M. Jin, S. Shahriar, M. Tufano, X. Shi, S. Lu, N. Sundaresan, and A. Svyatkovskiy, “Inferfix: End-to-end Program Repair with LLMs,” in Proceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (ESEC/FSE)...

    2023

  7. [14]

    Automated Program Repair in the Era of Large Pre-Trained Language Models,

    C. S. Xia, Y . Wei, and L. Zhang, “Automated Program Repair in the Era of Large Pre-Trained Language Models,” in 2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE). IEEE, 2023, pp. 1482–1494

    2023

  8. [15]

    Copiloting the Copilots: Fusing Large Language Models with Completion Engines for Automated Program Repair,

    Y . Wei, C. S. Xia, and L. Zhang, “Copiloting the Copilots: Fusing Large Language Models with Completion Engines for Automated Program Repair,” in Proceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineerin...

    2023

  9. [16]

    Using an LLM to Help With Code Understanding,

    D. Nam, A. Macvean, V . Hellendoorn, B. Vasilescu, and B. Myers, “Using an LLM to Help With Code Understanding,” in 2024 IEEE/ACM 46th International Conference on Software Engineering (ICSE) . IEEE Computer Society, 2024, pp. 881–881

    2024

  10. [17]

    Advancing Requirements Engineering through Generative AI: Assessing the Role of LLMs,

    C. Arora, J. Grundy, and M. Abdelrazek, “Advancing Requirements Engineering through Generative AI: Assessing the Role of LLMs,” in Generative AI for Effective Software Development . Springer, 2024, pp. 129–148

    2024

  11. [18]

    How do requirements evolve during elicitation? an empirical study combining interviews and app store analysis,

    A. Ferrari, P. Spoletini, and S. Debnath, “How do requirements evolve during elicitation? an empirical study combining interviews and app store analysis,” Requirements Engineering , vol. 27, no. 4, pp. 489–519, Dec 2022. [Online]. Available: https://doi.org/10.1007/ s00766-022-00383-7

    2022

  12. [19]

    Interrater reliability: the kappa statistic,

    M. L. McHugh, “Interrater reliability: the kappa statistic,” Biochemia medica, vol. 22, no. 3, pp. 276–282, 2012

    2012

  13. [20]

    The effect of sampling temperature on problem solving in large language models,

    M. Renze, “The effect of sampling temperature on problem solving in large language models,” in Findings of the Association for Computa- tional Linguistics: EMNLP 2024 , 2024, pp. 7346–7356

    2024

  14. [21]

    Improving agile requirements: the quality user story framework and tool,

    G. Lucassen, F. Dalpiaz, J. M. E. van der Werf, and S. Brinkkemper, “Improving agile requirements: the quality user story framework and tool,” Requirements engineering, vol. 21, pp. 383–403, 2016

    2016

  15. [22]

    Empirical research methods in web and software engineering,

    C. Wohlin, M. H ¨ost, and K. Henningsson, “Empirical research methods in web and software engineering,” in Web Engineering. Springer, 2006, pp. 409–430

    2006

  16. [24]

    Application of Large Language Models to Software Engi- neering Tasks: Opportunities, Risks, and Implications,

    I. Ozkaya, “Application of Large Language Models to Software Engi- neering Tasks: Opportunities, Risks, and Implications,” IEEE Software, vol. 40, no. 3, pp. 4–8, 2023

    2023

  17. [25]

    On the use of GPT-4 for creating goal models: An exploratory study,

    B. Chen, K. Chen, S. Hassani, Y . Yang, D. Amyot, L. Lessard, G. Mussbacher, M. Sabetzadeh, and D. Varr ´o, “On the use of GPT-4 for creating goal models: An exploratory study,” in 2023 IEEE 31st IEEE TRANSACTIONS ON SOFTW ARE ENGINEERING 16 International Requirements Engineer...

    2023

  18. [26]

    Automated domain modeling with large language models: A comparative study,

    K. Chen, Y . Yang, B. Chen, J. A. H. L ´opez, G. Mussbacher, and D. Varr´o, “Automated domain modeling with large language models: A comparative study,” in 2023 ACM/IEEE 26th International Conference on Model Driven Engineering Languages and Systems (MODELS) . IEEE, 2023, pp. 162–172

    2023

  19. [27]

    Towards taming large language models with prompt templates for legal GRL modeling,

    S. de Kinderen and K. Winter, “Towards taming large language models with prompt templates for legal GRL modeling,” in Business Process Modeling, Development, and Support (BPMDS 2024) . Springer, 2024, pp. 213–228

    2024

  20. [28]

    On the assessment of generative ai in modeling tasks: an experience report with chatgpt and uml,

    J. C ´amara, J. Troya, L. Burgue ˜no, and A. Vallecillo, “On the assessment of generative ai in modeling tasks: an experience report with chatgpt and uml,” Software and Systems Modeling, vol. 22, no. 3, pp. 781–793, 2023

    2023

  21. [29]

    Prompts matter: Insights and strategies for prompt engineering in automated software traceability,

    A. Rodriguez et al., “Prompts matter: Insights and strategies for prompt engineering in automated software traceability,” in Proc. of RE’23 Workshops, 2023

    2023

  22. [30]

    Code Gradients: Towards Automated Traceability of LLM-Generated Code,

    M. North, A. Atapour-Abarghouei, and N. Bencomo, “Code Gradients: Towards Automated Traceability of LLM-Generated Code,” in Proc. of RE’24, 2024

    2024

  23. [31]

    Requirements are all you need: From requirements to code with llms,

    B. Wei, “Requirements are all you need: From requirements to code with llms,” arXiv preprint https://arxiv.org/abs/2406.10101, 2024

    2024 arXiv

  24. [32]

    Research directions for using llm in software requirement engineering: a systematic review,

    A. Hemmat, M. Sharbaf, S. Kolahdouz-Rahimi, K. Lano, and S. Y . Tehrani, “Research directions for using llm in software requirement engineering: a systematic review,”Frontiers in Computer Science, vol. 7, p. 1519437, 2025

    2025

  25. [33]

    Generative ai for requirements engineering: A systematic literature review,

    H. Cheng, J. H. Husen, Y . Lu, T. Racharak, N. Yoshioka, N. Ubayashi, and H. Washizaki, “Generative ai for requirements engineering: A systematic literature review,” 2025. [Online]. Available: https://arxiv.org/abs/2409.06741

    2025

  26. [34]

    Using chatgpt in software requirements engineering: A comprehensive review,

    N. Marques, R. Rocha Silva, and J. Bernardino, “Using chatgpt in software requirements engineering: A comprehensive review,” Future Internet , vol. 16, no. 6, p. 180, 2024. [Online]. Available: https://www.mdpi.com/1999-5903/16/6/180

    2024

  27. [35]

    Improving requirements completeness: Automated assistance through large language models,

    D. Luitel, S. Hassani, and M. Sabetzadeh, “Improving requirements completeness: Automated assistance through large language models,” Requirements Engineering, vol. 29, no. 1, pp. 73–95, 2024

    2024

  28. [36]

    Bert: Pre- training of deep bidirectional transformers for language understanding,

    J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “Bert: Pre- training of deep bidirectional transformers for language understanding,” arXiv, vol. arXiv, no. 1810.04805, May 2019. [Online]. Available: https://doi.org/10.48550/arXiv.1810.04805

    2019 doi arXiv

  29. [37]

    Inconsistency Detec- tion in Natural Language Requirements Using Chatgpt: A Preliminary Evaluation,

    A. Fantechi, S. Gnesi, L. Passaro, and L. Semini, “Inconsistency Detec- tion in Natural Language Requirements Using Chatgpt: A Preliminary Evaluation,” in 2023 IEEE 31st International Requirements Engineering Conference (RE). IEEE, 2023, pp. 335–340

    2023

  30. [38]

    Chatgpt Prompt Patterns for Improving Code Quality, Refactoring, Requirements Elicitation, and Software Design,

    J. White, S. Hays, Q. Fu, J. Spencer-Smith, and D. C. Schmidt, “Chatgpt Prompt Patterns for Improving Code Quality, Refactoring, Requirements Elicitation, and Software Design,” in Generative AI for Effective Software Development . Springer, 2024, pp. 71–108

    2024

  31. [40]

    Generating Requirements Elicitation Interview Scripts with Large Language Models,

    B. G ¨orer and F. B. Aydemir, “Generating Requirements Elicitation Interview Scripts with Large Language Models,” in 2023 IEEE 31st International Requirements Engineering Conference Workshops (REW) . IEEE, 2023, pp. 44–51

    2023

  32. [41]

    Teaching Requirements Elicitation Interviews: An Empirical Study of Learning from Mistakes,

    M. Bano, D. Zowghi, A. Ferrari, P. Spoletini, and B. Donati, “Teaching Requirements Elicitation Interviews: An Empirical Study of Learning from Mistakes,” Requirements Engineering, vol. 24, pp. 259–289, 2019

    2019

  33. [42]

    Elicitron: An LLM Agent-Based Simulation Framework for Design Requirements Elicitation,

    M. Ataei, H. Cheong, D. Grandi, Y . Wang, N. Morris, and A. Tessier, “Elicitron: An LLM Agent-Based Simulation Framework for Design Requirements Elicitation,” arXiv preprint arXiv:2404.16045 , 2024

    2024 arXiv

  34. [43]

    Strategies, Benefits and Challenges of App Store- inspired Requirements Elicitation,

    F. A. and S. P., “Strategies, Benefits and Challenges of App Store- inspired Requirements Elicitation,” in 45th IEEE/ACM International Conference on Software Engineering, ICSE 2023, Melbourne, Australia, May 14-20, 2023 . IEEE, 2023, pp. 1290–1302

    2023

  35. [44]

    Translating requirements in property specification patterns using llms,

    D. Guidotti, L. Pandolfo, T. Fanni, K. Zedda, and L. Pulina, “Translating requirements in property specification patterns using llms,” in Proceedings of the International Workshop on Artificial Intelligence for Climate Change, Italian Workshop on Planning and Scheduling, RCRA ...

    2024

  36. [45]

    nl2spec: Interactively translating unstructured natural language to temporal logics with large language models,

    M. Cosler, C. Hahn, D. Mendoza, F. Schmitt, and C. Trippel, “nl2spec: Interactively translating unstructured natural language to temporal logics with large language models,” in Computer Aided Verification , C. Enea and A. Lal, Eds. Cham: Springer Nature Switzerland, 2023, pp. 383– 396

    2023

  37. [46]

    Exploring llms for verifying technical system specifications against requirements,

    L. M. Reinpold, M. Schieseck, L. P. Wagner, F. Gehlhoff, and A. Fay, “Exploring llms for verifying technical system specifications against requirements,” arXiv preprint arXiv:2411.11582, 2024. [Online]. Available: https://arxiv.org/abs/2411.11582

    2024 arXiv

  38. [47]

    Formal requirements engineering and large language models: A two-way roadmap,

    A. Ferrari and P. Spoletini, “Formal requirements engineering and large language models: A two-way roadmap,” Information and Software Technology, vol. 181, p. 107697, 2025

    2025

  39. [48]

    Prompting Creative Requirements via Traceable and Adversarial Examples in Deep Learning,

    H. Gudaparthi, N. Niu, B. Wang, T. Bhowmik, H. Liu, J. Zhang, J. Savolainen, G. Horton, S. Crowe, T. Scherz et al. , “Prompting Creative Requirements via Traceable and Adversarial Examples in Deep Learning,” in 2023 IEEE 31st International Requirements Engineering Conference (...

    2023

  40. [49]

    Exploring the Efficacy of ChatGPT in Generating Requirements: An Experimental Study,

    L. Bencheikh and N. Hoglund, “Exploring the Efficacy of ChatGPT in Generating Requirements: An Experimental Study,” Master’s thesis, University of Gothenburg/Department of Computer Science and Engi- neering, 2023

    2023

  41. [50]

    Investigating ChatGPT’s Po- tential to Assist in Requirements Elicitation Processes,

    K. Ronanki, C. Berger, and J. Horkoff, “Investigating ChatGPT’s Po- tential to Assist in Requirements Elicitation Processes,” in Proc. of the 49th Euromicro Conference on Software Engineering and Advanced Applications (SEAA), 2023, pp. 354–361

    2023

  42. [51]

    Using llms in software requirements specifications: An empirical evaluation,

    M. Krishna, B. Gaur, A. Verma, and P. Jalote, “Using llms in software requirements specifications: An empirical evaluation,” in 2024 IEEE 32nd International Requirements Engineering Conference (RE) , Jun. 2024, pp. 475–483. [Online]. Available: https://ieeexplore.ieee.org/ doc...

    2024

  43. [52]

    Engineering safety requirements for autonomous driving with large language models,

    A. Nouri, B. Cabrero-Daniel, F. T ¨orner, H. Sivencrona, and C. Berger, “Engineering safety requirements for autonomous driving with large language models,” in 2024 IEEE 32nd International Requirements Engineering Conference (RE) . IEEE, 2024, p. to appear

    2024

  44. [53]

    Improving User Story Practice with the Grimm Method: A Multiple Case Study in the Software Industry,

    G. Lucassen, F. Dalpiaz, J. M. E. van der Werf, and S. Brinkkemper, “Improving User Story Practice with the Grimm Method: A Multiple Case Study in the Software Industry,” in Requirements Engineering: Foundation for Software Quality: 23rd International Working Confer- ence, REF...

    2017

  45. [54]

    Evaluating the Impact of User Stories Quality on the Ability to Understand and Structure Requirements,

    Y . Wautelet, D. Gielis, S. Poelmans, and S. Heng, “Evaluating the Impact of User Stories Quality on the Ability to Understand and Structure Requirements,” in The Practice of Enterprise Modeling: 12th IFIP Working Conference, PoEM 2019, Luxembourg, Luxembourg, November 27–29, ...

    2019

  46. [55]

    Crowd- Based Requirements Elicitation via Pull Feedback: Method and Case Studies,

    J. Wouters, A. Menkveld, S. Brinkkemper, and F. Dalpiaz, “Crowd- Based Requirements Elicitation via Pull Feedback: Method and Case Studies,” Requirements Engineering, vol. 27, no. 4, pp. 429–455, Dec

  47. [2022]

    Available: https://doi.org/10.1007/s00766-022-00384-6

    [Online]. Available: https://doi.org/10.1007/s00766-022-00384-6

    doi

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.