REVIEW 3 major objections 4 minor 44 references
From Requirements to Code: Understanding Developer Practices in LLM-Assisted Software Engineering
T0 review · 3 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash
Pith's one-line read Documented requirements are too abstract to feed directly into code-generating LLMs; developers must first decompose them into programming tasks and add design and architectural context before prompting.
desk verdict A useful and honest interview study of how developers actually get from requirements to LLM-based code, with one 'must' claim that outruns the self-reported evidence. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is the programming task, the intermediate artifact that sits between a documented requirement and an LLM prompt. It carries the theory because it is both the output of the manual decomposition step and the core component of every code-generation prompt. The two models are the machinery: the process model traces a requirements artifact through decomposition into programming tasks, context construction (ad-hoc or elaborate), code generation, checking, and adjustment, with three interaction patterns; the content model specifies the five context categories—language and libraries, interface and data format, infrastructure and deployment, business logic and algorithms, and unit tests—plus the code context that practitioners add so the generated code can be integrated.
What would settle it
A log-based field study recording the actual prompts, accepted completions, and code diffs of developers over several weeks would settle the claim: if raw requirement texts are regularly pasted verbatim into prompts and the resulting code is integrated without any prior decomposition into programming tasks, the central claim is contradicted; if every code-generating prompt is preceded by a hand-written task breakdown, the theory is supported.
Extended reading notes
Core claim
The study's central claim is that LLM-assisted implementation does not bypass requirements engineering; it depends on it. Practitioners reported that pasting a user story or a catalog requirement into a chat or IDE assistant yields code that cannot be integrated into the existing code base. Instead, they derive programming tasks—smaller units that specify how a requirement is to be realized in code—and then construct prompts around those tasks, adding information about the programming language and libraries, interfaces and data formats, infrastructure and deployment constraints, unit tests, and the relevant code context. The paper organizes this into a process model (requirements artifact to programming tasks, then technical exploration, code generation, or manual coding, with prompt templates and reused chat histories to lower context-construction effort) and a content model listing the context categories that make generated code usable. The authors conclude that any scientific approach claiming to automate a requirements-centric software engineering task should explain how it represents requirements and how it handles the manual decomposition and enrichment this study observed.
Load-bearing premise
The theory stands on what 18 practitioners said about their own workflows in interviews, rather than on logs or direct observation of their prompts, keystrokes, or code diffs; if those accounts are inaccurate, the paper describes what developers say they do, not what they actually do.
Editorial extensions
If this is right
- Benchmarks that evaluate code generation from short coding descriptions should not be treated as evidence about requirements-to-code practice; the paper draws a sharp line between the two.
- Fully automated software engineering from raw requirements remains distant for complex software, because decomposition and context selection require requirements and software engineering expertise.
- Prompts are currently treated as transient artifacts rather than documented, reviewed engineering artifacts, so traceability and accountability of prompt content become pressing concerns.
- Tooling that supports context reuse—prompt templates, chat histories, and pre-filled agent contexts—directly addresses the effort bottleneck the practitioners described.
- The optimal granularity of programming tasks is an open research question, and future studies should compare the granularity of benchmark inputs with the granularity of tasks derived from real requirements.
Reading between the lines
- The authors leave implicit that if decomposition is the bottleneck, then measuring and supporting the decomposition step—for example, with semi-automated task-splitting tools—may matter more than further improving code generation quality.
- A testable extension would be a log-based field study that records actual prompts, accepted completions, and code diffs; unlike retrospective interviews, such logs could quantify how much of the effort is prompt construction versus post-generation code adjustment.
- The content model suggests a structured prompt schema; one could test whether systematically supplying the five context categories reduces integration failures, such as compilation errors or failing tests, compared with unstructured prompts.
- The theory predicts the programming-task intermediate step will shrink but not disappear as models gain repository grounding, because the abstraction gap is partly project-specific knowledge the model cannot infer from a requirement alone.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This paper reports a semi-structured interview study with 18 practitioners from 14 companies to understand how software engineers incorporate requirements and design information when using large language models for code generation. Based on inductive in-vivo coding of 179 quotes, the authors propose a process model (requirements artifacts → programming tasks → prompt construction → code generation, technical exploration, or manual coding) and a content model (what context information is added to prompts). The central claim is that traditional requirements artifacts are too abstract for direct use as LLM input, and practitioners must first manually decompose them into programming tasks, which are then enriched with design decisions and architectural constraints. The paper also identifies three interaction patterns: incremental code generation, manual coding with intelligent auto-completion, and extensive code generation. The authors discuss implications for requirements engineering research and for the feasibility of fully automated requirements-to-code systems.
Significance. If the theory is accepted, it directly challenges the premise of many NL2Code benchmarks and automated requirements-to-code proposals, which treat raw requirements or task descriptions as sufficient prompt input. The study is one of the first to examine real-world requirements artifacts in LLM-assisted implementation, and its process/content models provide a useful vocabulary for future research. Strengths include a transparent qualitative method, a replication package with interview guide and codebook, explicit tracing from in-vivo codes to themes, multi-company and multi-domain sampling, and acknowledged threats to validity. The paper does not claim statistical generalization and is careful to position its contribution as theory building. The main weakness is that the headline normative claim rests entirely on retrospective self-reports, with inconsistencies in the reported participant support and no artifact-based evidence in the findings.
major comments (3)
- [Section IV-A, Takeaway 1 and abstract] The universal normative claim is not fully supported by the reported evidence. The text states that practitioners 'unanimously stated' they first derive smaller units from requirements artifacts, but the list of confirming participants contains only 12 of 18 (P01, P04, P05, P06, P07, P08, P09, P10, P11, P13, P15, P17). The three participants who used unstructured Issues with IDE integration (P02, P03, P18 in Table I) are absent from that list, and their interaction pattern is described as 'manual coding with intelligent auto-completion,' in which suggestions are accepted while writing code as usual. These cases may not involve a separate decomposition step at all, so the process model's single 'Requirements artifact' entry node and the abstract's 'must first be manually decomposed' overstate universality. Please either supply direct evidence (prompts, artifacts, or observed sessions) for these cases, or revise the process model and wording to present decomposition as a dominant reported pattern rather than a universal necessity.
- [Section III-B and Section IV-A] The central empirical support is retrospective self-report, and the stated attempt to collect artifacts is not reported in the Findings. Section III-B says participants were asked to 'show us intermediate artifacts such as user stories, prompt templates, and prompts, if possible,' but the Findings contain no such artifact-based data; no logs, prompt snapshots, or observed interaction data are presented. The claims about what practitioners 'do not use' and 'must' do are therefore based entirely on what participants said they do, not on what they demonstrably did. I am not asking for quantitative validation, but the manuscript should either report whatever artifact evidence was collected or explicitly restrict the theory's scope to stated practices and soften the modal language in Takeaway 1 and the abstract.
- [Section IV-A, 'Deriving Programming Tasks From Requirements Artifacts'] There is an internal inconsistency in the evidence reporting. The first paragraph of this subsection says practitioners 'unanimously stated' that requirements artifacts must be decomposed, while two paragraphs later the text says 'Almost all interviewees confirmed this notion' and lists 12 of 18 participants. 'Unanimously' and 'almost all' cannot both describe the same finding. Please use a single, accurate formulation and, if necessary, explain how the non-listed participants were coded.
minor comments (4)
- [Throughout] The author name appears as 'V ogelsang' in the header and in references [27] and [44]; this is presumably a typesetting error and should be corrected to 'Vogelsang'.
- [Section III-B] Please clarify whether the DeepL-translated German transcripts were reviewed against the original recordings or German text by a second researcher, as the fidelity of the quoted statements is otherwise dependent on a single machine translation.
- [Figure 1] The loop from 'Code adjustment' back to 'Refine programming task' and the optional paths around 'Use auto-completion' are visually dense; consider labeling the loop-back condition explicitly or adding a note in the caption.
- [Section IV-B] The parenthesized participant IDs after each content category are useful, but a small table mapping each content category to the confirming participants would make the evidence easier to verify.
Circularity Check
No circularity: the interview-based theory is built inductively from participant statements; prior self-citations are background only.
full rationale
The paper makes no formal derivation, fits no parameters, and predicts no quantitative quantity that could reduce to an input by construction. Its central claim—that requirements, as typically documented, are too abstract for direct LLM input and must be manually decomposed into programming tasks (Section IV-A, Takeaway 1)—is an inductive generalization from 18 semi-structured interviews, supported with participant quotes and a replication package that traces in-vivo codes to themes to model entities. The process model (Fig. 1) and content model (Fig. 2) are descriptive codifications of what participants reported, not results forced by definitions or by an imported uniqueness theorem. The authors' own prior work (e.g., Vogelsang [27], [29], [44]) appears only as background, related work, or discussion context and is not used to justify the empirical findings; the identified interaction patterns are explicitly compared with, rather than derived from, Barke et al. [32]. The principal threat to the study is evidential—retrospective self-reports, convenience sampling, and DeepL translation (Section III-B)—which bears on correctness and generalizability, not on circularity. No step reduces by construction to its own inputs, and no self-citation chain is load-bearing. The findings would stand or fail on the quality and representativeness of the interview evidence, not on any circular dependency.
Assumptions & free parameters
assumptions (4)
- domain assumption Practitioner self-reports in interviews accurately reflect actual work practices.
- domain assumption Convenience sampling from the authors' professional network yields a representative range of practitioners.
- domain assumption Saturation was reached after interviews P15-P18.
- domain assumption DeepL machine translation preserves the meaning of German interview transcripts.
Cite this review
Pith. "Pith review of From Requirements to Code: Understanding Developer Practices in LLM-Assisted Software Engineering." pith.science (2026). https://pith.science/paper/TNMU64JD
@misc{pith2026250707548,
author = {Pith},
title = {Pith review of: From Requirements to Code: Understanding Developer Practices in LLM-Assisted Software Engineering},
year = {2026},
howpublished = {\url{https://pith.science/paper/TNMU64JD}},
note = {Machine review of arXiv:2507.07548}
}
read the original abstract
With the advent of generative LLMs and their advanced code generation capabilities, some people already envision the end of traditional software engineering, as LLMs may be able to produce high-quality code based solely on the requirements a domain expert feeds into the system. The feasibility of this vision can be assessed by understanding how developers currently incorporate requirements when using LLMs for code generation-a topic that remains largely unexplored. We interviewed 18 practitioners from 14 companies to understand how they (re)use information from requirements and other design artifacts to feed LLMs when generating code. Based on our findings, we propose a theory that explains the processes developers employ and the artifacts they rely on. Our theory suggests that requirements, as typically documented, are too abstract for direct input into LLMs. Instead, they must first be manually decomposed into programming tasks, which are then enriched with design decisions and architectural constraints before being used in prompts. Our study highlights that fundamental RE work is still necessary when LLMs are used to generate code. Our theory is important for contextualizing scientific approaches to automating requirements-centric SE tasks.
Figures
Reference graph
Works this paper leans on
-
[1]
Sommerville, Software Engineering, 10th ed
I. Sommerville, Software Engineering, 10th ed. Pearson, 2015
work page 2015
-
[2]
On the naturalness of software,
A. Hindle, E. T. Barr, M. Gabel, Z. Su, and P. Devanbu, “On the naturalness of software,” Communications of the ACM , vol. 59, no. 5, pp. 122–131, 2016
2016
-
[3]
A survey of neural code intelligence: Paradigms, advances and beyond,
Q. Sun, Z. Chen, F. Xu, K. Cheng, C. Ma, Z. Yin, J. Wang, C. Han, R. Zhu, S. Yuan et al., “A survey of neural code intelligence: Paradigms, advances and beyond,” arXiv preprint arXiv:2403.14734 , 2024
arXiv 2024
-
[4]
Evaluating large language models trained on code,
M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. D. O. Pinto, J. Kaplan, H. Edwards, Y . Burda, N. Joseph, G. Brockman et al., “Evaluating large language models trained on code,” arXiv preprint arXiv:2107.03374 , 2021
arXiv 2021
-
[5]
Code llama: Open foundation models for code,
B. Roziere, J. Gehring, F. Gloeckle, S. Sootla, I. Gat, X. E. Tan, Y . Adi, J. Liu, R. Sauvestre, T. Remez et al. , “Code llama: Open foundation models for code,” arXiv preprint arXiv:2308.12950 , 2023
arXiv 2023
-
[6]
DeepSeek-Coder: When the large language model meets programming–the rise of code intelligence,
D. Guo, Q. Zhu, D. Yang, Z. Xie, K. Dong, W. Zhang, G. Chen, X. Bi, Y . Wu, Y . Liet al. , “DeepSeek-Coder: When the large language model meets programming–the rise of code intelligence,” arXiv preprint arXiv:2401.14196, 2024
arXiv 2024
-
[7]
Deep learning for code intelligence: Survey, benchmark and toolkit,
Y . Wan, Z. Bi, Y . He, J. Zhang, H. Zhang, Y . Sui, G. Xu, H. Jin, and P. Yu, “Deep learning for code intelligence: Survey, benchmark and toolkit,” ACM Computing Surveys , 2024
work page 2024
-
[8]
Navigating the complexity of generative AI adoption in software engineering,
D. Russo, “Navigating the complexity of generative AI adoption in software engineering,” ACM Transactions on Software Engineering and Methodology (TOSEM), 2024
work page 2024
Show all 44 references
-
[9]
Large language models for software engi- neering: A systematic literature review,
X. Hou, Y . Zhao, Y . Liu, Z. Yang, K. Wang, L. Li, X. Luo, D. Lo, J. Grundy, and H. Wang, “Large language models for software engi- neering: A systematic literature review,” ACM Transactions on Software Engineering and Methodology , vol. 33, no. 8, pp. 1–79, 2024
2024
-
[10]
Deep learning based program generation from requirements text: Are we there yet?
H. Liu, M. Shen, J. Zhu, N. Niu, G. Li, and L. Zhang, “Deep learning based program generation from requirements text: Are we there yet?” IEEE Transactions on Software Engineering , vol. 48, no. 4, pp. 1268– 1289, 2020
2020
-
[11]
Clarifygpt: A framework for enhancing llm-based code generation via requirements clarification,
F. Mu, L. Shi, S. Wang, Z. Yu, B. Zhang, C. Wang, S. Liu, and Q. Wang, “Clarifygpt: A framework for enhancing llm-based code generation via requirements clarification,” Proceedings of the ACM on Software Engineering, vol. 1, no. FSE, pp. 2332–2354, 2024
2024
-
[12]
Understanding neural code intelligence through program simplification,
M. R. I. Rabin, V . J. Hellendoorn, and M. A. Alipour, “Understanding neural code intelligence through program simplification,” in 29th ACM Joint Meeting on European Software Engineering Conference and Sym- posium on the F oundations of Software Engineering (ESEC/FSE) , 2021, ...
2021
-
[13]
code2vec: learning distributed representations of code,
U. Alon, M. Zilberstein, O. Levy, and E. Yahav, “code2vec: learning distributed representations of code,” Proceedings of the ACM on Pro- gramming Languages (POPL) , vol. 3, pp. 1–29, 2019
2019
-
[14]
Codebert: A pre-trained model for programming and natural languages,
Z. Feng, D. Guo, D. Tang, N. Duan, X. Feng, M. Gong, L. Shou, B. Qin, T. Liu, D. Jiang et al., “Codebert: A pre-trained model for programming and natural languages,” arXiv preprint arXiv:2002.08155 , 2020
2002 arXiv
-
[15]
Codet5: Identifier-aware unified pre-trained encoder-decoder models for code understanding and generation,
Y . Wang, W. Wang, S. Joty, and S. C. Hoi, “Codet5: Identifier-aware unified pre-trained encoder-decoder models for code understanding and generation,” arXiv preprint arXiv:2109.00859 , 2021
2021 arXiv
-
[16]
Language mod- els are few-shot learners,
T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell et al., “Language mod- els are few-shot learners,” Advances in neural information processing systems, vol. 33, pp. 1877–1901, 2020
1901
-
[17]
A systematic evaluation of large language models of code,
F. F. Xu, U. Alon, G. Neubig, and V . J. Hellendoorn, “A systematic evaluation of large language models of code,” in Proceedings of the 6th ACM SIGPLAN International Symposium on Machine Programming , 2022, pp. 1–10
2022
-
[18]
Codegen: An open large language model for code with multi-turn program synthesis,
E. Nijkamp, B. Pang, H. Hayashi, L. Tu, H. Wang, Y . Zhou, S. Savarese, and C. Xiong, “Codegen: An open large language model for code with multi-turn program synthesis,” arXiv preprint arXiv:2203.13474 , 2022
2022 arXiv
-
[19]
Starcoder: may the source be with you!
R. Li, L. B. Allal, Y . Zi, N. Muennighoff, D. Kocetkov, C. Mou, M. Marone, C. Akiki, J. Li, J. Chim et al. , “Starcoder: may the source be with you!” arXiv preprint arXiv:2305.06161 , 2023
2023 arXiv
-
[20]
Natural language generation and understanding of big code for AI-assisted programming: A review,
M.-F. Wong, S. Guo, C.-N. Hang, S.-W. Ho, and C.-W. Tan, “Natural language generation and understanding of big code for AI-assisted programming: A review,” Entropy, vol. 25, no. 6, p. 888, 2023
2023
-
[21]
Program synthesis with large language models,
J. Austin, A. Odena, M. Nye, M. Bosma, H. Michalewski, D. Dohan, E. Jiang, C. Cai, M. Terry, Q. Le et al. , “Program synthesis with large language models,” arXiv preprint arXiv:2108.07732 , 2021
2021 arXiv
-
[22]
Swe-bench: Can language models resolve real-world github issues?
C. E. Jimenez, J. Yang, A. Wettig, S. Yao, K. Pei, O. Press, and K. Narasimhan, “Swe-bench: Can language models resolve real-world github issues?” arXiv preprint arXiv:2310.06770 , 2023
2023 arXiv
-
[23]
An empirical comparison of pre-trained models of source code,
C. Niu, C. Li, V . Ng, D. Chen, J. Ge, and B. Luo, “An empirical comparison of pre-trained models of source code,” in IEEE/ACM 45th International Conference on Software Engineering (ICSE) . IEEE, 2023, pp. 2136–2148
2023
-
[24]
Mapping language to code in programmatic context,
S. Iyer, I. Konstas, A. Cheung, and L. Zettlemoyer, “Mapping language to code in programmatic context,” arXiv preprint arXiv:1808.09588 , 2018
2018 arXiv
-
[25]
Natural language to code: How far are we?
S. Wang, M. Geng, B. Lin, Z. Sun, M. Wen, Y . Liu, L. Li, T. F. Bissyandé, and X. Mao, “Natural language to code: How far are we?” in Proceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the F oundations of Software Engineering, 2023, pp. 375–387
2023
-
[26]
Codesearchnet challenge: Evaluating the state of semantic code search,
H. Husain, H.-H. Wu, T. Gazit, M. Allamanis, and M. Brockschmidt, “Codesearchnet challenge: Evaluating the state of semantic code search,” arXiv preprint arXiv:1909.09436 , 2019
1909 arXiv
-
[27]
From specifications to prompts: On the future of genera- tive large language models in requirements engineering,
A. V ogelsang, “From specifications to prompts: On the future of genera- tive large language models in requirements engineering,” IEEE Software, vol. 41, no. 5, pp. 9–13, 2024
2024
-
[28]
IEEE standard glossary of software engineering terminology,
“IEEE standard glossary of software engineering terminology,” IEEE Std 610.12-1990 , pp. 1–84, 1990
1990
-
[29]
On the impact of requirements smells in prompts: The case of automated traceability,
A. V ogelsang, A. Korn, G. Broccia, A. Ferrari, J. Fischbach, and C. Arora, “On the impact of requirements smells in prompts: The case of automated traceability,” in 47th IEEE/ACM International Conference on Software Engineering (ICSE-NIER) , 2025
2025
-
[30]
Productivity assessment of neural code completion,
A. Ziegler, E. Kalliamvakou, X. A. Li, A. Rice, D. Rifkin, S. Simister, G. Sittampalam, and E. Aftandilian, “Productivity assessment of neural code completion,” in Proceedings of the 6th ACM SIGPLAN Interna- tional Symposium on Machine Programming , 2022, pp. 21–29
2022
-
[31]
Expectation vs. experi- ence: Evaluating the usability of code generation tools powered by large language models,
P. Vaithilingam, T. Zhang, and E. L. Glassman, “Expectation vs. experi- ence: Evaluating the usability of code generation tools powered by large language models,” in Chi conference on human factors in computing systems extended abstracts , 2022, pp. 1–7
2022
-
[32]
Grounded copilot: How programmers interact with code-generating models,
S. Barke, M. B. James, and N. Polikarpova, “Grounded copilot: How programmers interact with code-generating models,” Proceedings of the ACM on Programming Languages (OOPSLA) , vol. 7, pp. 85–111, 2023
2023
-
[33]
Discovering the syntax and strategies of natural lan- guage programming with generative language models,
E. Jiang, E. Toh, A. Molina, K. Olson, C. Kayacik, A. Donsbach, C. J. Cai, and M. Terry, “Discovering the syntax and strategies of natural lan- guage programming with generative language models,” in Proceedings of the 2022 CHI Conference on Human Factors in Computing Systems ,...
2022
-
[34]
H. J. Rubin and I. S. Rubin, Qualitative interviewing: The art of hearing data. sage, 2011
2011
-
[35]
J. W. Creswell and J. D. Creswell, Research design: Qualitative, quantitative, and mixed methods approaches . Sage publications, 2017
2017
-
[36]
Characterizing architecturally significant requirements,
L. Chen, M. Ali Babar, and B. Nuseibeh, “Characterizing architecturally significant requirements,” IEEE Software , vol. 30, no. 2, pp. 38–45, 2013
2013
-
[37]
In-IDE code generation from natural language: Promise and challenges,
F. F. Xu, B. Vasilescu, and G. Neubig, “In-IDE code generation from natural language: Promise and challenges,” ACM Transactions on Software Engineering and Methodology (TOSEM) , vol. 31, no. 2, pp. 1–47, 2022
2022
-
[38]
A qualitative study on the implementation design decisions of developers,
J. T. Liang, M. Arab, M. Ko, A. J. Ko, and T. D. LaToza, “A qualitative study on the implementation design decisions of developers,” in IEEE/ACM 45th International Conference on Software Engineering (ICSE). IEEE, 2023, pp. 435–447
2023
-
[39]
Requirements are all you need: From requirements to code with LLMs,
B. Wei, “Requirements are all you need: From requirements to code with LLMs,” in IEEE 32nd International Requirements Engineering Conference (RE). IEEE, 2024, pp. 416–422
2024
-
[40]
Generating test scenarios from NL requirements using retrieval-augmented LLMs: An industrial study,
C. Arora, T. Herda, and V . Homm, “Generating test scenarios from NL requirements using retrieval-augmented LLMs: An industrial study,” in IEEE 32nd International Requirements Engineering Conference (RE) , 2024, pp. 240–251
2024
-
[41]
Model generation with LLMs: From requirements to UML sequence diagrams,
A. Ferrari, S. Abualhaija, and C. Arora, “Model generation with LLMs: From requirements to UML sequence diagrams,” in IEEE 32nd Interna- tional Requirements Engineering Conference Workshops (REW) , vol. 24. IEEE, 2024, p. 291–300
2024
-
[42]
Handshaking: Negotiate to provoke the right understanding of requirements,
S. Fricker, T. Gorschek, C. Byman, and A. Schmidle, “Handshaking: Negotiate to provoke the right understanding of requirements,” IEEE Software, 2019
2019
-
[43]
Measuring the fitness-for-purpose of requirements: An initial model of activities and attributes,
J. Frattini, J. Fischbach, D. Fucci, M. Unterkalmsteiner, and D. Mendez, “Measuring the fitness-for-purpose of requirements: An initial model of activities and attributes,” in 32nd International Requirements Engineer- ing Conference (RE) . IEEE, 2024, pp. 398–406
2024
-
[44]
Requirements quality is quality in use,
H. Femmer and A. V ogelsang, “Requirements quality is quality in use,” IEEE Software , vol. 36, no. 3, p. 83–91, May 2019. [Online]. Available: http://dx.doi.org/10.1109/MS.2018.110161823
2019
Reviewed August 6, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.