Pith. sign in

REVIEW 3 major objections 6 minor 1 cited by

Prompt Engineering Guidelines for Using Large Language Models in Requirements Engineering

T0 review · 3 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Prompt engineering guidelines can be mapped to the five requirements engineering activities, giving RE practitioners a structured starting point for LLM prompts.

desk verdict A transparent qualitative synthesis whose proposed theme-to-RE mapping outruns the interview evidence, but a useful starting point that deserves a serious referee. read the letter →

arxiv 2507.03405 v1 pith:L2WEPFZF submitted 2025-07-04 cs.SE

Krishna Ronanki , Simon Arvidsson , Johan Axell This is my paper
classification cs.SE
keywords promptengineeringlargelanguagemodelsrequirementsgenerativeAIguidelinespatternssystematicreviewexpertinterviews
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper sets out to show that prompt engineering guidelines developed for large language models in other fields can be repurposed for requirements engineering (RE). It extracts 36 guidelines from a systematic review of 28 studies, groups them into nine recurring themes—context, persona, templates, disambiguation, reasoning, analysis, keywords, wording, and few-shot prompts—and then asks three RE experts which themes suit which RE activity. The result is a proposed mapping: context, template, and keyword themes for elicitation; template, analysis, and reasoning for analysis; persona and disambiguation for specification; persona and template for validation; and keyword and reasoning for management. A sympathetic reader would care because the mapping offers RE practitioners an explicit, literature-grounded place to start when choosing prompts, rather than rediscovering prompt guidance for each task.

What carries the argument

The core mechanism is the theme-to-activity mapping shown in Table 4: a table that pairs nine prompt guideline themes with the five RE activities based on expert interview judgments. It sits on top of a second machinery, the nine-theme taxonomy of 36 guidelines (context, persona, templates, disambiguation, reasoning, analysis, keywords, wording, few-shot prompts). The taxonomy converts a heterogeneous collection of prompts from many domains into comparable categories, and the mapping then tells which categories apply where, turning a general prompt-guideline list into RE-specific guidance.

What would settle it

A controlled study would settle the claim: have RE practitioners build prompts for elicitation, analysis, specification, validation, and management tasks using the mapped themes (e.g., context/template/keyword for elicitation), and compare output quality against prompts built from unmapped guidelines or from generic advice. If the mapped prompts do not produce higher-quality outputs than the alternatives across these tasks, the central claim that the mapping provides effective RE-specific guidance would be undercut.

Watch

Extended reading notes

Core claim

This paper's central discovery is a thematic mapping between prompt engineering guidelines and the five standard requirements engineering (RE) activities: elicitation, analysis, specification, validation, and management. The authors compiled 36 guidelines from 28 primary studies covering domains such as text-to-image generation, legal judgment, translation, mental health analysis, and biomedical reasoning, and organized them into nine themes. They then conducted semi-structured interviews with three RE experts and used thematic synthesis to judge which themes are relevant for each RE phase. The mapping assigns context, template, and keyword guidelines to elicitation; template, analysis, and reasoning to analysis; persona and disambiguation to specification; persona and template to validation; and keyword and reasoning to management. The paper argues that this mapping helps fill the shortage of domain-specific prompt guidance for RE by giving users a structured basis for selecting prompts per activity.

Load-bearing premise

The mapping's load-bearing premise is that prompt guidelines gathered from other domains, such as text-to-image generation, legal judgment, translation, mental health, and biomedical reasoning, transfer meaningfully to requirements engineering when three experts judge them relevant.

Editorial extensions

If this is right

  • RE practitioners get a concrete starting point for prompt selection: use context, template, and keyword guidelines for elicitation; template, analysis, and reasoning for analysis; persona and disambiguation for specification; persona and template for validation; and keyword and reasoning for management.
  • If the mapping holds, guidelines in the few-shot and wording themes are likely less useful for RE as currently formulated, pointing to a need for RE-specific versions of those techniques.
  • The mapping suggests that template guidelines are broadly applicable across RE activities, while context and disambiguation guidelines, though mapped to fewer activities, may bring higher value where they apply.
  • Adoption of the mapping could shift LLM use in RE from ad-hoc prompting toward a more structured, repeatable process, improving consistency and traceability of LLM output.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the mapping is correct, one testable extension is that prompts built from mapped themes will outperform unmapped prompts in a benchmark covering all five RE activities; an LLM-output-quality experiment could verify this.
  • The same interview-driven mapping method could produce analogous theme-to-task maps for other software engineering domains, such as testing, debugging, or code review.
  • Because the mapping rests on only three experts, a larger practitioner survey might reveal that certain theme assignments (e.g., disambiguation for specification) depend on the specific RE context or stakeholder assumptions.
  • The paper's finding that few-shot and wording guidelines were deemed irrelevant suggests a gap: RE may need its own few-shot prompt conventions, for instance pairing requirement samples with rationale in a format specific to requirement specifications.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper reports a systematic literature review of prompt engineering guidelines for large language models (LLMs), resulting in 36 guidelines grouped into nine themes, followed by semi-structured interviews with three academic requirements engineering (RE) experts. Based on the expert insights, the authors propose a mapping of guideline themes (Context, Persona, Template, Disambiguation, Reasoning, Analysis, Keywords, Wording, Few-shot) to five RE activities (Elicitation, Analysis, Specification, Validation, Management), presented in Table 4. The stated contribution is a structured, RE-specific starting point for prompt engineering, addressing a perceived gap in the literature. The paper also discusses advantages and limitations of applying such guidelines in RE and identifies future research directions.

Significance. The paper has a clear and useful aim: consolidating dispersed prompt-engineering guidance and relating it to RE activities. Its strengths include a recognizable Kitchenham-style review structure, transparent inclusion/exclusion criteria, a documented data extraction form, use of established qualitative methods (content analysis, thematic synthesis, member checking), and explicit discussion of validity threats. If the proposed mapping were empirically grounded, it would give practitioners a structured starting point for choosing prompts per RE activity. However, the mapping is the central contribution and it is not validated; it rests on a small expert sample and on an unexamined transfer assumption from non-RE domains. The paper is valuable as a synthesis and a source of hypotheses, but it does not yet deliver validated RE-specific prompt guidelines.

major comments (3)
  1. [Section 4.3, Table 4] The mapping of themes to RE activities is not supported by the expert interview data as reported. In Section 4.2, Expert 2 is quoted as saying, "I think it's a bad idea to use the prompt engineering for validation," and the same expert expresses skepticism about LLM capabilities for requirements analysis, while Expert 1 cautions against over-reliance on templates in requirements analysis. Yet Table 4 maps Persona and Template to Validation, and Template to Analysis. The justifications in Section 4.3 (e.g., "template guidelines can help in having some predefined approaches") do not address these explicit reservations. Since the mapping is the paper's central claim (answering RQ3), the authors must either re-analyze the interview data to show how the mapping follows from the experts' statements, or re-frame the mapping as an untested proposal with clear caveats.
  2. [Section 3.2 and Table 3] The transferability of guidelines from non-RE domains to RE is asserted rather than demonstrated. Table 3 includes guidelines whose originating contexts are far from RE: T2 is a text-to-image template ("[Medium] [Subject] [Artist(s)] [Details] [Image repository support]"), C4 concerns pre-appending context tokens in vision-language models, W1 is about newlines in translation tasks, and A3 is an emotion-enhanced prompting method for mental health analysis. The paper assumes that three academic experts' judgments are sufficient to transfer these guidelines to RE, but no evidence or argument is given that the underlying prompt mechanisms remain effective for RE tasks. This is load-bearing because the mapping in Table 4 is built from these themes. The authors should either provide per-theme justification for transferability or restrict the mapping to guidelines with a clear RE interpretation.
  3. [Section 5.1 and Section 6] The paper overstates the evidential basis of its claims relative to the data. Only three experts were interviewed, all from academia, and the paper itself acknowledges limited generalizability and possible confirmation bias in the threats to validity. Nevertheless, the conclusion states that "Our interviews highlighted the usefulness of the identified guidelines" for various activities, which conflicts with the dissenting views documented in Section 4.2 (e.g., Expert 2 on validation, Expert 1 on templates, and all experts on the complexity of requirements analysis). The claims should be scaled back to reflect the exploratory nature of the study, and the mapping should be presented as a set of hypotheses to be tested rather than as established guidelines.
minor comments (6)
  1. [Section 1, paragraph 2] The statement "there are no guidelines or strategies that RE practitioners can leverage" is too absolute given that Section 2 cites related work, e.g., White et al. and Sasaki et al. Consider softening to "few" or "no consolidated" guidelines.
  2. [Section 4.2, Requirements Analysis] There is a typo: "They 2 also suggested that templates..." should read "Expert 2 also suggested that templates...". Also, "for for requirement elicitation" appears in Section 4.2, Requirements Elicitation and should be corrected.
  3. [Section 4.3, first paragraph] The sentence "The primary reason is eliminate and reduce ambiguities" is grammatically incomplete; it should read "The primary reason is to eliminate and reduce ambiguities."
  4. [Section 5, second paragraph] The phrase "traceability and and reasoning" contains a duplicated "and"; it should be "traceability and reasoning."
  5. [Table 3, Analysis theme] The theme name "Analysis" is easy to confuse with the RE activity "Requirements Analysis" used in Table 4 and Section 4.2. Consider renaming the theme to something like "Answer Validation" or "Output Checking" to avoid ambiguity.
  6. [Bibliography] Some bibliographic entries use nonstandard formatting (e.g., author names in the form "Brown, Tom and Mann, Benjamin and..." followed by "and others"), while others use standard style. Please normalize the reference list for consistency.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: the central mapping is an expert-informed synthesis from external literature and fresh interview data, not a fitted parameter renamed as a prediction or a result forced by self-citation.

full rationale

The paper's derivation chain is qualitative and self-contained: a systematic literature review yields 36 prompt engineering guidelines, which are grouped into nine themes (Section 4.1, Table 3); semi-structured interviews with three RE experts then provide advantages and limitations of applying these guidelines to RE activities (Section 4.2); and the proposed mapping in Table 4 is presented as an integration of the experts' insights (Section 4.3). No equation is fitted, no quantity is predicted from a fitted parameter, and no theorem is imported to forbid alternatives. The mapping's basis is explicitly the interview data: 'This mapping was done based on the experts' insights presented in detail within subsection 4.2.' The two self-citations (Ronanki et al. 2024a and 2024b) are minor: 2024b is background on prior RE prompt-pattern evaluation, and 2024a is cited as the source of one guideline (A2, 'best of three strategy') among many guidelines from unrelated external studies. Neither citation forces the central theme-to-RE-activity mapping, which retains independent content from the nine themes and the expert interviews. The skeptic's concern that the mapping is asserted at theme level and may overstate applicability (e.g., text-to-image template T2 mapped indirectly through the 'Template' theme; Expert 2's skepticism about validation) is a validity and evidence-quality critique, not a circularity critique: the mapping does not reduce by construction to its inputs. Therefore the circularity score is low.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The paper has no fitted numerical parameters; the central analysis is qualitative. The key unstated premises are the canonical five-activity RE structure, the 2018 literature cutoff, the cross-domain transferability of prompt guidelines, and the sufficiency of three academic expert judgments.

assumptions (4)
  • domain assumption The five RE activities (elicitation, analysis, specification, validation, management) are the canonical decomposition for mapping prompt guidelines.
    Invoked in Section 2 and used to structure Table 4, relying on Pressman and Sommerville rather than on evidence collected in this study.
  • domain assumption A 2018 publication cutoff is sufficient to capture all relevant prompt engineering literature for the 'shortage of RE-specific guidelines' conclusion.
    Section 3.1 sets the boundary based on the transformer era; the finding of shortage depends on this scope decision.
  • ad hoc to paper Prompt engineering guidelines from non-RE domains (text-to-image, legal, translation, biomedical) transfer meaningfully to RE when experts judge them applicable.
    Table 3 includes image-generation templates, legal prompts, translation wording, and mental-health emotion prompting; the paper assumes expert opinion is enough to map them onto RE activities.
  • ad hoc to paper Three academic RE experts, selected by publication history and academic background, provide a sufficient evidential basis for the proposed mapping.
    Section 3.2 describes n=3; Section 5.1 acknowledges external validity threats; no industrial practitioners or broader survey are used.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Prompt Engineering Guidelines for Using Large Language Models in Requirements Engineering." pith.science (2026). https://pith.science/paper/L2WEPFZF

@misc{pith2026250703405,
  author       = {Pith},
  title        = {Pith review of: Prompt Engineering Guidelines for Using Large Language Models in Requirements Engineering},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/L2WEPFZF}},
  note         = {Machine review of arXiv:2507.03405}
}
read the original abstract

The rapid emergence of generative AI models like Large Language Models (LLMs) has demonstrated its utility across various activities, including within Requirements Engineering (RE). Ensuring the quality and accuracy of LLM-generated output is critical, with prompt engineering serving as a key technique to guide model responses. However, existing literature provides limited guidance on how prompt engineering can be leveraged, specifically for RE activities. The objective of this study is to explore the applicability of existing prompt engineering guidelines for the effective usage of LLMs within RE. To achieve this goal, we began by conducting a systematic review of primary literature to compile a non-exhaustive list of prompt engineering guidelines. Then, we conducted interviews with RE experts to present the extracted guidelines and gain insights on the advantages and limitations of their application within RE. Our literature review indicates a shortage of prompt engineering guidelines for domain-specific activities, specifically for RE. Our proposed mapping contributes to addressing this shortage. We conclude our study by identifying an important future line of research within this field.

Figures

Figures reproduced from arXiv: 2507.03405 by the authors.

Figure 1
Figure 1. Theme distribution of the extracted guidelines from the studies in the review. Few-shot Prompts The theme “Few-shot Prompts” categorises guidelines that are intended for any form of few-shot prompting. Each guideline was assigned to one of the nine themes and given a unique identifier, as listed in [PITH_FULL_IMAGE:figures/full_fig_p008_1.png] view at source ↗

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Prompts Blend Requirements and Solutions: From Intent to Implementation

    cs.SE 2026-03 conditional novelty 4.0 of 10

    Prompts in AI-assisted development can be decomposed into functionality/quality, general solutions, and specific solutions — the 'Prompt Triangle' framework.

Reference graph

Works this paper leans on

23 extracted references · 8 canonical work pages · cited by 1 Pith paper

  1. [6]

    Bert: Pre- training of deep bidirectional transformers for language understanding.arXiv preprint arXiv:1810.04805,

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre- training of deep bidirectional transformers for language understanding.arXiv preprint arXiv:1810.04805,

    arXiv

  2. [7]

    Optimizing prompts for text-to- image generation

    Yaru Hao, Zewen Chi, Li Dong, and Furu Wei. Optimizing prompts for text-to- image generation. arXiv preprint arXiv:2212.09611,

    arXiv

  3. [8]

    I think this is the most dis- ruptive technology

    Haque, Mubin Ul and Dharmadasa, Isuru and Sworna, Zarrin Tasnim and Ra- japakse, Roshan Namal and Ahmad, Hussain. "I think this is the most dis- ruptive technology": Exploring Sentiments of ChatGPT Early Adopters using Twitter Data.arXiv preprint arXiv:2212.05856,

    arXiv

  4. [10]

    Design Guidelines for Prompt Engineering Text-to-Image Generative Models

    Vivian Liu and Lydia B Chilton. Design Guidelines for Prompt Engineering Text-to-Image Generative Models. InProceedings of the 2022 CHI Conference on Human Factors in Computing Systems, CHI ’22, New York, NY, USA, 2022a. Association for Computing Machinery. ISBN 9781450391573.https: //doi.org/10.1145/3491102.3501825. URL https://doi.org/10.1145/ 3491102.3...

    arXiv 2022

  5. [11]

    Chatgpt as a factual inconsistency evaluator for abstractive text summarization

    Zheheng Luo, Qianqian Xie, and Sophia Ananiadou. Chatgpt as a factual inconsistency evaluator for abstractive text summarization. arXiv preprint arXiv:2303.15621,

    arXiv

  6. [12]

    Prompting AI Art: An Investigation into the Creative Skill of Prompt Engineering

    Jonas Oppenlaender, Rhema Linder, and Johanna Silvennoinen. Prompting ai art: An investigation into the creative skill of prompt engineering. arXiv preprint arXiv:2303.13534,

    work page Pith review arXiv

  7. [15]

    Krishna Ronanki, Beatriz Cabrero-Daniel, and Christian Berger

    https: //doi.org/10.1109/REW57809.2023.00087. Krishna Ronanki, Beatriz Cabrero-Daniel, and Christian Berger. Chatgpt as a tool for user story quality evaluation: Trustworthy out of the box? InAgile Processes in Software Engineering and Extreme Programming – Workshops, pages 173–181. Springer Nature Switzerland, 2024a. ISBN 978-3-031-48550-3. Krishna Ronan...

    arXiv 2023

  8. [19]

    Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou

    https://doi.org/10.1109/MS.2024.3410712. Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. Self-consistency improves chain of thought reasoning in language models.arXiv preprint arXiv:2203.11171, 2022a. Title Suppressed Due to Excessive Length 19 Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, ...

    work page arXiv 2024

Show all 23 references
  1. [20]

    Symbolicknowledge distillation: from general language models to commonsense models

    Peter West, Chandra Bhagavatula, Jack Hessel, Jena D Hwang, Liwei Jiang, RonanLeBras,XimingLu,SeanWelleck,andYejinChoi. Symbolicknowledge distillation: from general language models to commonsense models. arXiv preprint arXiv:2110.07178,

    arXiv

  2. [22]

    Legal prompting: Teaching a lan- guage model to think like a lawyer.arXiv preprint arXiv:2212.01326,

    Fangyi Yu, Lee Quartey, and Frank Schilder. Legal prompting: Teaching a lan- guage model to think like a lawyer.arXiv preprint arXiv:2212.01326,

    arXiv

  3. [23]

    Learning to promptforvision-languagemodels

    Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu. Learning to promptforvision-languagemodels. International Journal of Computer Vision, 130(9):2337–2348, 2022a. Yongchao Zhou, Andrei Ioan Muresanu, Ziwen Han, Keiran Paster, Silviu Pitis, Harris Chan, and Jimmy Ba. L...

    arXiv

  4. [1993]

    Alberto D

    URL https://cir.nii.ac.jp/ crid/1130282271686871424. Alberto D. Rodriguez, Katherine R. Dearstyne, and Jane Cleland-Huang. Prompts matter: Insights and strategies for prompt engineering in auto- mated software traceability. In2023 IEEE 31st International Requirements Engineeri...

  5. [1997]

    Strandberg, Per Erik

    ISBN 0471974447. Strandberg, Per Erik. Ethical Interviews in Software Engineering. In 2019 ACM/IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM), pages 1–11, 2019.https://doi.org/10.1109/ESEM. 2019.8870192. Dietrich Trautmann, Alina Petrova,...

    2019

  6. [1999]

    Ian Sommerville and Pete Sawyer.Requirements Engineering: A Good Practice Guide

    https: //doi.org/10.1109/32.799955. Ian Sommerville and Pete Sawyer.Requirements Engineering: A Good Practice Guide. John Wiley & Sons, Inc., USA, 1st edition,

    doi

  7. [2007]

    Humans in humans out: On gpt converging toward common sense in both success and failure.arXiv preprint arXiv:2303.17276,

    Philipp Koralus and Vincent Wang-Maścianica. Humans in humans out: On gpt converging toward common sense in both success and failure.arXiv preprint arXiv:2303.17276,

    arXiv

  8. [2011]

    https://doi.org/https://doi.org/10.1016/j.infsof

    ISSN 0950-5849. https://doi.org/https://doi.org/10.1016/j.infsof. 2011.01.004. URL https://www.sciencedirect.com/science/article/ pii/S095058491100005X. Special Section on Best Papers from XP2010. Paul Denny, Viraj Kumar, and Nasser Giacaman. Conversing with copilot: Ex- plori...

    2011 doi

  9. [2018]

    Prompt programming for large language models: Beyond the few-shot paradigm

    Laria Reynolds and Kyle McDonell. Prompt programming for large language models: Beyond the few-shot paradigm. In Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems, pages 1–7,

    2021

  10. [2020]

    Evaluation of chatgpt family of models for biomedical reasoning and classification.arXiv preprint arXiv:2304.02496,

    Shan Chen, Yingya Li, Sheng Lu, Hoang Van, Hugo JWL Aerts, Guergana K Savova, and Danielle S Bitterman. Evaluation of chatgpt family of models for biomedical reasoning and classification.arXiv preprint arXiv:2304.02496,

    arXiv

  11. [2021]

    A prompt pattern catalog to enhance prompt engineering with chatgpt.arXiv preprint arXiv:2302.11382, 2023a

    Jules White, Quchen Fu, Sam Hays, Michael Sandborn, Carlos Olea, Henry Gilbert, Ashraf Elnashar, Jesse Spencer-Smith, and Douglas C Schmidt. A prompt pattern catalog to enhance prompt engineering with chatgpt.arXiv preprint arXiv:2302.11382, 2023a. Jules White, Quchen Fu, Sam ...

    arXiv

  12. [2022]

    Language Models are Few-shot Learners.Advances in Neural Information Processing Systems, 33:1877–1901,

    Brown, Tom and Mann, Benjamin and Ryder, Nick and Subbiah, Melanie and Kaplan, Jared D and Dhariwal, Prafulla and Neelakantan, Arvind and Shyam, Pranav and Sastry, Girish and Askell, Amanda and others. Language Models are Few-shot Learners.Advances in Neural Information Proces...

    1901

  13. [2023]

    Large language models in the workplace: A case study on prompt engineering for job type classification

    Benjamin Clavié, Alexandru Ciceu, Frederick Naylor, Guillaume Soulié, and Thomas Brightwell. Large language models in the workplace: A case study on prompt engineering for job type classification. arXiv preprint arXiv:2303.07142,

    arXiv

  14. [2024]

    Seaman,C.B

    https://doi.org/10.1109/COMPSAC61105.2024.00096. Seaman,C.B. QualitativeMethodsinEmpiricalStudiesofSoftwareEngineering. IEEE Transactions on Software Engineering, 25(4):557–572,

    2024 arXiv

  15. [2025]

    Accessed: 19/02/2025

    URL https: //agpt.co/. Accessed: 19/02/2025. Bhavya Bhavya, Jinjun Xiong, and Chengxiang Zhai. Analogy generation by prompting large language models: A case study of instructgpt.arXiv preprint arXiv:2210.04186,

    2025 arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.