Pith. sign in

REVIEW 3 major objections 4 minor 73 references

ReqBrain: Task-Specific Instruction Tuning of LLMs for AI-Assisted Requirements Generation

T0 review · 3 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper's central claim is that ReqBrain—a LoRA fine-tuned Zephyr-7b-beta—generates software requirements that four experienced evaluators cannot reliably tell apart from human-authored ones (50.7% precision, p = 0.903), and that the…

desk verdict A careful, honest fine-tuning study with a real dataset and above-average statistics, but the headline 'indistinguishable from humans' outruns the evidence: a non-significant chi-square with four raters is not equivalence. read the letter →

arxiv 2505.17632 v1 pith:HOY7ZSPR submitted 2025-05-23 cs.SE cs.AIcs.LG

Mohammad Kasra Habib , Daniel Graziotin , Stefan Wagner This is my paper
classification cs.SEcs.AIcs.LG
keywords requirementsengineeringelicitationspecificationlargelanguagemodelsinstructiontuningfine-tuningISO/IEC/IEEE29148AI-assistedsoftware
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper proposes ReqBrain, a 7-billion-parameter language model fine-tuned to generate software requirements in chat sessions. The central claim is that fine-tuning makes a small, locally hostable model produce requirements that are hard to tell apart from human-written ones: four experienced evaluators judged 47.8% of ReqBrain outputs as human-authored versus 8.8% for the untuned baseline, and in a direct human-versus-ReqBrain comparison the overall classification precision was 50.7%, statistically no better than chance (chi-square p = 0.903). The same evaluators rated the generated requirements as consistent with, missing from, and enhancing a given specification, with median 4 on a 1-5 scale and p < .001, and the model scored 89.30% BERT F1 and 91.20 FRUGAL, above untuned ChatGPT-4o. If correct, this would let requirements engineers draft and refine specifications with a local open model instead of sending data to a large commercial API.

What carries the argument

The machinery is a small, task-specific instruction-tuning dataset plus a LoRA fine-tune. The dataset contains 166 (instruction, completion) pairs spanning three task types: How-to? INST teaches ISO 29148 syntax and signaling keywords (shall, should, may, will); RE-types INST teaches requirement classes such as functional, usability, and security; Missing INST presents a partial requirement set and asks for missing or additional requirements. The argument is carried by the low-rank adaptation (LoRA) fine-tune with rank r = 64 applied to Zephyr-7b-beta, and the evaluation chain runs through BERTScore and FRUGALScore automatic alignment followed by human ratings analyzed with non-parametric tests.

What would settle it

Open the released instruction dataset and check whether any source project contributes requirements to both the training and evaluation splits, then recompute the BERT and FRUGAL scores on a project-disjoint subset. If the 89.30% BERT F1 and 91.20 FRUGAL score drop materially, the automatic authenticity claim rests on training/evaluation overlap rather than on generation quality.

Watch

Extended reading notes

Core claim

ReqBrain, produced by fine-tuning Zephyr-7b-beta with LoRA on 166 instruction instances built from 242 ISO/IEC/IEEE 29148-compliant requirements, generates authentic and adequate requirements. Authenticity is established by human evaluation: evaluators classified ReqBrain outputs as human-authored 47.8% of the time versus 8.8% for the untuned baseline, and in a direct human-versus-ReqBrain comparison the overall classification precision was 50.7%, statistically indistinguishable from chance (chi-square p = 0.903). Adequacy is established by median ratings of 4 on 1-5 Likert scales for consistency with, missing from, and enhancing the completeness of a given specification, all significantly above the neutral midpoint (p < .001), and by comparable ISO 29148 syntax and keyword compliance with human authors. Automatic metrics support the selection of Zephyr-7b-beta as the best base model: BERT F1 89.30% and FRUGAL 91.20, above untuned ChatGPT-4o.

Load-bearing premise

The results assume that no software project's requirements appear in both the training and evaluation halves of the instruction dataset; if a project does appear in both, the reported similarity scores could partly reflect the model having seen sibling requirements during training rather than its ability to generate fresh ones.

Editorial extensions

If this is right

  • Requirements engineers could draft an initial specification from a chat prompt, with the model producing ISO 29148-compliant statements in real time instead of raw elicitation notes.
  • A 7-billion-parameter open model hosted on local infrastructure could offer this capability without sending proprietary project data to a commercial API.
  • The same fine-tuned model could be pointed at existing tickets, bug reports, or legacy documents to extract requirements and generate missing ones, which the paper supports through its Missing INST task.
  • Teams still need human review; the paper positions ReqBrain as a drafting support that merges elicitation and specification, not as a replacement for analysis and validation.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the reported effects generalize, the same 166-instruction recipe should transfer to adjacent tasks the paper lists as future work—defect identification, test-case generation, and user-story creation—since those tasks share the underlying requirement-language distribution; the released dataset makes that directly testable.
  • A project-disjoint evaluation would clarify whether the 'missing requirements' capability generalizes across projects or mostly matches within-project style, since the Missing INST prompts and completions are drawn from the same project.
  • The 50.7% human-classification precision comes from four raters; an independent replication with more raters, or with raters who are adversarial and know they are being tested for AI authorship, would stress-test the indistinguishability claim.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper introduces ReqBrain, a chat-based tool built on a LoRA fine-tuned 7B-parameter LLM for generating authentic and adequate software requirements. The authors curate a 166-instance instruct dataset from the SwaRD corpus, filter it for ISO/IEC/IEEE 29148 compliance, create three instruction task categories (How-to? INST, RE-types INST, Missing INST), and fine-tune five open LLMs. Zephyr-7b-beta is selected as the best model based on BERT and FRUGAL scores (F1 89.30, FRUGAL 91.20) and is then compared against its untuned baseline and against human-authored requirements in human evaluations with four raters. The human study finds that ReqBrain outputs are judged human-authored at 47.8% versus 8.8% for the baseline (Fisher p<.001), and that human raters cannot significantly distinguish ReqBrain from human-written requirements (chi-square p=0.90331, precision 50.7%). Additional adequacy ratings (consistency, missing requirements, completeness) have medians of 4 on a 1-5 scale with significant Wilcoxon tests. The paper concludes that fine-tuned LLMs can generate requirements indistinguishable from human-written ones and adequate for elicitation and specification support.

Significance. If the central claims were fully supported, the contribution would be valuable: an open instruct dataset for requirements generation, open-source fine-tuned models, a systematic comparison of five LLMs, and a human evaluation design with non-parametric tests, effect sizes, and Holm-Bonferroni adjustments. The dataset and models are positioned as reusable artifacts, and the study targets a real gap in AI-assisted requirements engineering. However, the load-bearing authenticity claim currently rests on a non-significant chi-square test rather than an equivalence test, and the evaluation protocol has a potential project-level leakage risk. These issues make the headline conclusions stronger than the evidence supports, but they are in principle addressable with additional analysis or re-splitting of the data.

major comments (3)
  1. [Section VI-A4, Table VIII] The claim that ReqBrain-generated requirements are 'indistinguishable' from human-authored ones is supported only by failing to reject H0,2 in a chi-square test (p=0.90331). A non-significant result does not establish equivalence, particularly when the human identification precision is 50.7% with a 95% CI extending to 56%, which is compatible with a modest but real human ability to detect AI-generated text. To support the indistinguishability claim, the authors need a pre-specified equivalence margin and an appropriate equivalence test (e.g., TOST or confidence-interval comparison against the margin), and the analysis must account for the clustering of multiple ratings made by the same four evaluators. As written, the abstract, Section VI findings, and the conclusion overstate what the statistical evidence shows.
  2. [Section IV-B3, Section IV-B2] The 80/20 stratified split is described as stratified only by targeted task category, not by the original software project. For the Missing INST task, Section IV-B2 states that requirements are grouped by their original software projects and split into two groups: one used in the instruction and the other used as completion labels. If any project appears in both the training and evaluation sets, the model may have been fine-tuned on sibling requirements of the evaluation items, which would inflate the BERT/FRUGAL similarity scores in RQ1.1 and RQ1.2 and could also affect the human authenticity judgments, since tasks B and C use the same evaluation set. The paper must demonstrate that the training and evaluation splits are project-disjoint (or quantify the overlap and re-analyze with a project-disjoint split).
  3. [Section V-A, Section V-C1, Section VI-A1] The automatic metrics that drive model selection for RQ1.1 and RQ1.2 are computed on an evaluation set of only about 33 instances (20% of the 166-instance dataset), and the best model is selected on that same set before the reported BERT F1 of 89.30 and FRUGAL score of 91.20 are quoted. These numbers are therefore not unbiased held-out performance estimates. The same evaluation set is then reused in the human evaluation tasks (Section V-C2 and V-C3), so selection bias may propagate to the human results. At minimum, the authors should present the sizes of the evaluation sets explicitly, report the variance of the scores, and discuss the optimism introduced by model selection on the evaluation set.
minor comments (4)
  1. [Table XI] Table XI reports n=128 for the three adequacy dimensions, while all other human-evaluation tables report n=136; the discrepancy is never explained, and the paper should clarify why eight observations are missing.
  2. [Section V-B2] The description of Task B says participants 'knowing the set contained a mix of human-authored and AI-generated requirements,' but the task is defined as comparing ReqBrain against its untuned baseline model; this wording is confusing and should be corrected to describe the actual composition of the evaluation set.
  3. [Section VI-A4] The phrase 'indistinguishable from those authored by humans' in the Findings RQ1 box and in the conclusion is too strong given the absence of an equivalence test; a more cautious phrasing such as 'not reliably distinguished by our evaluators' would be more consistent with the evidence.
  4. [Section VIII-C] The external validity threat about the three KIB 3 projects coming from the same domain is acknowledged, but the paper would benefit from a similar caveat in the discussion of the adequacy results in Section VI-B3, where the generality of the median-4 rating is otherwise easy to over-read.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: held-out automatic scores plus independent human evaluations carry the load-bearing claims; minor self-citations (SwaRD, [58]) are not load-bearing, and the real weaknesses (project-level split overlap, null-result-as-equivalence) are validity and inference concerns, not circular reductions.

full rationale

The central claims — that ReqBrain generates authentic and adequate requirements — do not reduce by construction to their inputs. The automatic metrics (BERT/FRUGAL, RQ1.1/RQ1.2) match the fine-tuning objective (similarity to human-authored ground truth), but they are computed on a 20% evaluation split held out from training (Section IV-B3), so they are genuine predictions unless the split is contaminated. Two unaddressed validity risks remain, and both are weighed here as non-circularity concerns. First, the split is stratified by task category, not by source project; for the Missing INST task the dataset construction itself groups requirements by project and splits them into instruction-context and completion-label groups (Section IV-B2). If any project spans the train/eval boundary, the reported F1=89.30 and FRUGAL=91.20 scores would be partly in-family rather than out-of-distribution. The text does not demonstrate project-disjointness, so this is a missing-support issue, but not exhibited circularity. Second, the 'indistinguishable from humans' conclusion (Section VI-A4, Table VIII: chi-square p=0.90331, 50.7% precision, 95% CI [44,56]) interprets a failure to reject H0,2 as evidence of equivalence; with four raters providing clustered judgments and a CI compatible with modest detectability, this is a statistical inference weakness (correctness risk), not a definitional reduction. Self-citations are present but peripheral: [58] (Wagner et al., including a co-author) guides reporting conventions only, and SwaRD — the authors' own, not-yet-released dataset — is assembled from public sources [26]-[36] and serves as training/evaluation material rather than as the evidence for the target claims. The adequacy evaluation on KIB 3 (Section V-C4) uses external, unpublished, out-of-distribution requirements rated by independent participants, supplying genuinely independent content for the load-bearing claim. The manuscript also discloses its own limitations (four-evaluator generalizability, non-replicable manual selection), consistent with a non-circular study design. No derivation step in the paper equals its own input by definition, so the paper is self-contained against external evidence and scores minimally.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The central claim rests on several domain assumptions: that ISO 29148 compliance can be reduced to two syntax templates and four keywords, that BERT/FRUGAL similarity to a single human reference measures authenticity, that a 20% split of 166 instances supports stable model selection, and that failing to reject the null in a four-rater authorship test establishes indistinguishability. No free parameters are fitted to the evaluation data; the LoRA rank and learning rate are taken from prior work. No new theoretical entities are introduced; ReqBrain is a fine-tuned model, not a new natural kind.

assumptions (4)
  • domain assumption ISO 29148-compliant requirements can be operationalized via two recommended syntaxes and four signaling keywords (shall, should, may, will).
    Section II-B limits ISO 29148 compliance to these syntactic markers; the adequacy construct relies on this reduction without validating it against the full standard.
  • domain assumption BERT and FRUGAL scores computed against a single human reference measure authenticity (clarity, coherence, relevance, realism, implementability).
    Section V-D defines Human Alignment via these metrics, but the paper does not validate that similarity to one reference captures all five authenticity dimensions.
  • domain assumption Human raters' inability to distinguish AI from human requirements is evidence of authenticity.
    Sections V-B and VI-A4 use failure to reject H0,2 as support for indistinguishability; this treats absence of evidence as evidence of equivalence.
  • domain assumption A 20% stratified split (approximately 33 instances) is sufficient to rank models and estimate generation quality.
    Section IV-B3 describes the split; no analysis demonstrates that the evaluation set size yields stable automatic metric estimates or reliable model ranking.

how reviews work

0 comments
Cite this review

Pith. "Pith review of ReqBrain: Task-Specific Instruction Tuning of LLMs for AI-Assisted Requirements Generation." pith.science (2026). https://pith.science/paper/HOY7ZSPR

@misc{pith2026250517632,
  author       = {Pith},
  title        = {Pith review of: ReqBrain: Task-Specific Instruction Tuning of LLMs for AI-Assisted Requirements Generation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/HOY7ZSPR}},
  note         = {Machine review of arXiv:2505.17632}
}
read the original abstract

Requirements elicitation and specification remains a labor-intensive, manual process prone to inconsistencies and gaps, presenting a significant challenge in modern software engineering. Emerging studies underscore the potential of employing large language models (LLMs) for automated requirements generation to support requirements elicitation and specification; however, it remains unclear how to implement this effectively. In this work, we introduce ReqBrain, an Al-assisted tool that employs a fine-tuned LLM to generate authentic and adequate software requirements. Software engineers can engage with ReqBrain through chat-based sessions to automatically generate software requirements and categorize them by type. We curated a high-quality dataset of ISO 29148-compliant requirements and fine-tuned five 7B-parameter LLMs to determine the most effective base model for ReqBrain. The top-performing model, Zephyr-7b-beta, achieved 89.30\% Fl using the BERT score and a FRUGAL score of 91.20 in generating authentic and adequate requirements. Human evaluations further confirmed ReqBrain's effectiveness in generating requirements. Our findings suggest that generative Al, when fine-tuned, has the potential to improve requirements elicitation and specification, paving the way for future extensions into areas such as defect identification, test case generation, and agile user story creation.

Figures

Figures reproduced from arXiv: 2505.17632 by the authors.

Figure 1
Figure 1. AI-assisted generation of software requirements using ReqBrain. [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. AI-assisted requirements generation approach overview, integrating ReqBrain. [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Performance metrics across three task categories (see Section IV-B). [PITH_FULL_IMAGE:figures/full_fig_p010_3.png] view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

73 extracted references · 51 canonical work pages

  1. [1]

    Requirements Elicitation: A Survey of Techniques, Approaches, and Tools,

    D. Zowghi and C. Coulin, “Requirements Elicitation: A Survey of Techniques, Approaches, and Tools,” Springer-Verlag, pp. 19–46, 2005

    work page 2005

  2. [2]

    Ambiguity as a resource to dis- close tacit knowledge,

    A. Ferrari, P. Spoletini, and S. Gnesi, “Ambiguity as a resource to dis- close tacit knowledge,” in 2015 IEEE 23rd International Requirements Engineering Conference (RE) , 2015, pp. 26–35

    work page 2015

  3. [3]

    Unpacking Tacit Knowledge for Requirements En- gineering,

    V . e. a. Gervasi, “Unpacking Tacit Knowledge for Requirements En- gineering,” in Managing Requirements Knowledge . Springer Berlin Heidelberg, 2013, pp. 23–47

    work page 2013

  4. [4]

    The en- gagement between knowledge transfer and requirements engineering,

    A. Distanont, H. Haapasalo, M. Vaananen, and J. Lehto, “The en- gagement between knowledge transfer and requirements engineering,” International Journal of Management, Knowledge and Learning , vol. 1, no. 2, pp. 131–156, 2012

    work page 2012

  5. [5]

    Naming the pain in requirements engineering: Contemporary problems, causes, and effects in practice,

    D. M ´endez Fern ´andez, S. Wagner, and et al., “Naming the pain in requirements engineering: Contemporary problems, causes, and effects in practice,” Empirical Software Engineering , vol. 22, pp. 2298–2338, 2017

    work page 2017

  6. [6]

    Requirements elicitation: Towards the unknown unknowns,

    A. Sutcliffe and P. Sawyer, “Requirements elicitation: Towards the unknown unknowns,” in 2013 21st IEEE International Requirements Engineering Conference (RE) , 2013, pp. 92–104

    work page 2013

  7. [7]

    Detecting Requirements Smells With Deep Learning: Experiences, Challenges and Future Work,

    M. K. Habib, S. Wagner, and D. Graziotin, “Detecting Requirements Smells With Deep Learning: Experiences, Challenges and Future Work,” in 2021 IEEE 29th International Requirements Engineering Conference Workshops (REW), 2021, pp. 153–156

    work page 2021

  8. [8]

    Easy Approach to Requirements Syntax (EARS),

    A. Mavin, P. Wilkinson, A. Harwood, and M. Novak, “Easy Approach to Requirements Syntax (EARS),” in 2009 17th IEEE International Requirements Engineering Conference , 2009, pp. 317–322

    work page 2009

Show all 73 references
  1. [9]

    The Importance of Modeling Social Factors of Language: Theory and Practice,

    D. Hovy and D. Yang, “The Importance of Modeling Social Factors of Language: Theory and Practice,” in Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies , 2021, pp. 588–602

    2021

  2. [10]

    ELICA: An Automated Tool for Dynamic Extraction of Requirements Relevant Information,

    Z. Shakeri Hossein Abad, V . Gervasi, D. Zowghi, and K. Barker, “ELICA: An Automated Tool for Dynamic Extraction of Requirements Relevant Information,” in 2018 5th International Workshop on Artificial Intelligence for Requirements Engineering (AIRE) , 2018, pp. 8–14

    2018

  3. [11]

    BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding,

    J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding,” in Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologi...

    2019

  4. [12]

    Improving Language Understanding by Generative Pre-Training,

    A. Radford, K. Narasimhan, T. Salimans, and I. Sutskever, “Improving Language Understanding by Generative Pre-Training,” 2018

    2018

  5. [13]

    ISO/IEC/IEEE 29148:2018,

    “ISO/IEC/IEEE 29148:2018,” https://www.iso.org/standard/72089.html, 2018, [Acessed: Dec. 26, 2023]

    2018

  6. [14]

    Large Language Models for Software Engineering: A Systematic Literature Review,

    X. Hou, Y . Zhao, and et al., “Large Language Models for Software Engineering: A Systematic Literature Review,” ACM Trans. Softw. Eng. Methodol., vol. 33, no. 8, pp. 1–79, Nov. 2024

    2024

  7. [15]

    Prompt-MIL: Boosting Multi-Instance Learning Schemes via Task-specific Prompt Tuning,

    J. Zhang and S. e. a. Kapse, “Prompt-MIL: Boosting Multi-Instance Learning Schemes via Task-specific Prompt Tuning,” arXiv preprint arXiv:2303.12214, 2023, available: http://arxiv.org/abs/2303.12214

    2023 arXiv

  8. [16]

    PPT: Pre-trained Prompt Tuning for Few-shot Learning,

    Y . Gu, X. Han, Z. Liu, and M. Huang, “PPT: Pre-trained Prompt Tuning for Few-shot Learning,” arXiv preprint arXiv:2109.04332, 2022, available: http://arxiv.org/abs/2109.04332

    2022 arXiv

  9. [17]

    Context-faithful Prompting for Large Language Models,

    W. Zhou, S. Zhang, H. Poon, and M. Chen, “Context-faithful Prompting for Large Language Models,” arXiv preprint arXiv:2303.11315, 2023, available: https://arxiv.org/abs/2303.11315

    2023 arXiv

  10. [18]

    Advancing Requirements Engineering through Generative AI: Assessing the Role of LLMs,

    C. Arora, J. Grundy, and M. Abdelrazek, “Advancing Requirements Engineering through Generative AI: Assessing the Role of LLMs,” arXiv preprint arXiv:2310.13976, 2023, available: http://arxiv.org/abs/2310.13976

    2023 arXiv

  11. [19]

    Maybe Only 0.5% Data is Needed: A Preliminary Exploration of Low Training Data Instruction Tuning,

    H. Chen, Y . Zhang, Q. Zhang, H. Yang, X. Hu, X. Ma, Y . Yanggong, and J. Zhao, “Maybe Only 0.5% Data is Needed: A Preliminary Exploration of Low Training Data Instruction Tuning,” arXiv preprint arXiv:2305.09246, 2023, available: http://arxiv.org/abs/2305.09246

    2023 arXiv

  12. [20]

    Prompt Engineering or Fine-Tuning? A Case Study on Phishing Detection with Large Language Models,

    F. Trad and A. Chehab, “Prompt Engineering or Fine-Tuning? A Case Study on Phishing Detection with Large Language Models,” Machine Learning and Knowledge Extraction , vol. 6, no. 1, pp. 367–384, Mar. 2024

    2024

  13. [21]

    Investigating ChatGPT’s Potential to Assist in Requirements Elicitation Processes,

    K. Ronanki, C. Berger, and J. Horkoff, “Investigating ChatGPT’s Potential to Assist in Requirements Elicitation Processes,” in 2023 49th Euromicro Conference on Software Engineering and Advanced Applications (SEAA), 2023, pp. 354–361

    2023

  14. [22]

    Mitigating Large Language Model Hallucination with Faithful Finetuning,

    M. Hu, B. He, Y . Wang, L. Li, C. Ma, and I. King, “Mitigating Large Language Model Hallucination with Faithful Finetuning,” arXiv preprint arXiv:2406.11267, 2024, available: http://arxiv.org/abs/2406.11267

    2024 arXiv

  15. [23]

    Two-stage LLM Fine-tuning with Less Spe- cialization and More Generalization,

    Y . Wang and S. e. a. Si, “Two-stage LLM Fine-tuning with Less Spe- cialization and More Generalization,” arXiv preprint arXiv:2211.00635, 2024, available: http://arxiv.org/abs/2211.00635

    2024 arXiv

  16. [24]

    RE- COVER: Toward the Automatic Requirements Generation from Stake- holders’ Conversations,

    G. V oria, F. Casillo, C. Gravino, G. Catolino, and F. Palomba, “RE- COVER: Toward the Automatic Requirements Generation from Stake- holders’ Conversations,” arXiv preprint arXiv:2411.19552, 2024, avail- able: http://arxiv.org/abs/2411.19552

    2024 arXiv

  17. [25]

    Washizaki, Guide to the Software Engineering Body of Knowledge (SWEBOK Guide), 4th ed

    H. Washizaki, Guide to the Software Engineering Body of Knowledge (SWEBOK Guide), 4th ed. Los Alamitos, CA, USA: IEEE Computer Society, 2024

    2024

  18. [26]

    of Excellence for Software & Systems Traceability, “Coest,” http://sarec.nd.edu/coest/datasets.html, 2002

    C. of Excellence for Software & Systems Traceability, “Coest,” http://sarec.nd.edu/coest/datasets.html, 2002

    2002

  19. [27]

    frieden84/nlp4re-reqeval: Second release,

    S. Abualhaija, “frieden84/nlp4re-reqeval: Second release,” Zenodo,

  20. [28]

    Pure: A dataset of public re- quirements documents,

    A. Ferrari, G. O. Spagnolo, and S. Gnesi, “Pure: A dataset of public re- quirements documents,” in 2017 IEEE 25th International Requirements Engineering Conference (RE) , 2017, pp. 502–505

    2017

  21. [29]

    Extracting Software Requirements from Unstructured Documents,

    V . Ivanov, A. Sadovykh, A. Naumchev, A. Bagnato, and K. Yakovlev, “Extracting Software Requirements from Unstructured Documents,” in Recent Trends in Analysis of Images, Social Networks and Texts: 10th International Conference, AIST 2021, Tbilisi, Georgia, December 16–18, 202...

    2021

  22. [30]

    Knauss, S

    E. Knauss, S. H. Houmb, S. Islam, J. J ¨urjens, and K. Schneider, “Secreq,” Zenodo, 2021. [Online]. Available: https://doi .org/10.5281/ zenodo.4530183

    2021

  23. [31]

    Software requirement risk prediction dataset,

    Z. shaukat, R. Naseem, and M. Zubair, “Software requirement risk prediction dataset,” Zenodo, 2018. [Online]. Available: https: //doi.org/10.5281/zenodo.1209601 15

    2018 doi

  24. [32]

    Rfp attachment #1: Functional and nonfunctional requirements for vhcures 3.0,

    S. o. V . Green Mountain Care Board, “Rfp attachment #1: Functional and nonfunctional requirements for vhcures 3.0,” 2018. [Online]. Available: https://gmcboard .vermont.gov/content/RFP-Attachment1- functional-and-nonfunctional-requirements-vhcures-30-rfp

    2018

  25. [33]

    Requirements Classification with Interpretable Machine Learning and Dependency Parsing,

    F. Dalpiaz, D. Dell’Anna, F. B. Aydemir, and S. C ¸ evikol, “Requirements Classification with Interpretable Machine Learning and Dependency Parsing,” in 2019 IEEE 27th International Requirements Engineering Conference (RE), 2019, pp. 142–152

    2019

  26. [34]

    The requirements tracing on target (retro).net dataset,

    J. H. Hayes, “The requirements tracing on target (retro).net dataset,” Zenodo, 2018. [Online]. Available: https://doi .org/10.5281/ zenodo.1223649

    2018

  27. [35]

    Eye tracking experiments data set - linking use cases and associated requirements: On the impact of linking variants on reading behavior,

    O. Karras, “Eye tracking experiments data set - linking use cases and associated requirements: On the impact of linking variants on reading behavior,” Zenodo, 2021. [Online]. Available: https://doi.org/10.5281/zenodo.4778899

    2021 doi

  28. [36]

    Eclipse public license-v2.0,

    Eclipse Foundation, “Eclipse public license-v2.0,” 2017. [Online]. Available: https://www.eclipse.org/legal/epl-2.0/

    2017

  29. [37]

    Fine- Tuning can Distort Pretrained Features and Underperform Out- of-Distribution,

    A. Kumar, A. Raghunathan, R. Jones, T. Ma, and P. Liang, “Fine- Tuning can Distort Pretrained Features and Underperform Out- of-Distribution,” arXiv preprint arXiv:2202.10054, 2022, available: https://arxiv.org/abs/2202.10054

    2022 arXiv

  30. [38]

    Continual Pre-Training of Large Language Models: How to (re)warm your model?

    K. Gupta, B. Th ´erien, and et al., “Continual Pre-Training of Large Language Models: How to (re)warm your model?” in Interna- tional Conference on Machine Learning , 2023, p. 1

    2023

  31. [39]

    Language Models are Few- Shot Learners,

    T. Brown, B. Mann, Ryder, and et al., “Language Models are Few- Shot Learners,” in Advances in Neural Information Processing Systems , Vancouver, Canada, 2020, pp. 1877–1901

    2020

  32. [40]

    Efficient Few-Shot Learning Without Prompts,

    “Efficient Few-Shot Learning Without Prompts,” Tunstall, Lewis and Reimers, Nils and Jo, Unso Eun Seo and Bates, Luke and Ko- rat, Daniel and Wasserblat, Moshe and Pereg, Oren, 2022, available: http://arxiv.org/abs/2209.11055

    2022 arXiv

  33. [41]

    Pre-Trained Language Models and Their Applications,

    H. Wang, “Pre-Trained Language Models and Their Applications,” Engineering, vol. 25, pp. 51–65, Jun. 2023

    2023

  34. [42]

    Improving language understanding with unsupervised learning,

    A. Radford, K. Narasimhan, T. Salimans, and I. Sutskever, “Improving language understanding with unsupervised learning,” 2018

    2018

  35. [43]

    LoRA: Low-Rank Adaptation of Large Language Models,

    E. J. Hu, Y . Shen, P. Wallis, Z. Allen-Zhu, Y . Li, S. Wang, L. Wang, and W. Chen, “LoRA: Low-Rank Adaptation of Large Language Models,” arXiv preprint arXiv:2106.09685, 2021, available: http://arxiv.org/abs/2106.09685

    2021 arXiv

  36. [44]

    Overcoming catastrophic forgetting in neural networks,

    J. Kirkpatrick and R. e. a. Pascanu, “Overcoming catastrophic forgetting in neural networks,” Proceedings of the National Academy of Sciences , vol. 114, no. 13, pp. 3521–3526, mar 2017

    2017

  37. [45]

    Task Arithmetic with LoRA for Continual Learning,

    R. Chitale, A. Vaidya, A. Kane, and A. Ghotkar, “Task Arithmetic with LoRA for Continual Learning,” arXiv preprint arXiv:2311.02428, 2023, available: http://arxiv.org/abs/2311.02428

    2023 arXiv

  38. [46]

    Investigating the Catastrophic Forgetting in Multimodal Large Lan- guage Models,

    Y . Zhai, S. Tong, X. Li, M. Cai, Q. Qu, Y . J. Lee, and Y . Ma, “Investigating the Catastrophic Forgetting in Multimodal Large Lan- guage Models,” arXiv preprint arXiv:2309.10313, 2023, available: http://arxiv.org/abs/2309.10313

    2023 arXiv

  39. [47]

    Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuning,

    A. Aghajanyan, L. Zettlemoyer, and S. Gupta, “Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuning,” arXiv preprint arXiv:2012.13255, 2020, available: http://arxiv.org/abs/2012.13255

    2012 arXiv

  40. [48]

    Selection-Inference: Exploiting Large Language Models for Interpretable Logical Reasoning,

    A. Creswell, M. Shanahan, and I. Higgins, “Selection-Inference: Exploiting Large Language Models for Interpretable Logical Reasoning,” arXiv preprint arXiv:2205.09712, 2022, available: https://arxiv.org/abs/2205.09712

    2022 arXiv

  41. [49]

    Building Real-World Meeting Summarization Systems using Large Language Models: A Practical Perspective,

    M. T. R. Laskar, X.-Y . Fu, C. Chen, and S. B. TN, “Building Real-World Meeting Summarization Systems using Large Language Models: A Practical Perspective,” arXiv preprint arXiv:2310.19233, 2023, available: https://arxiv.org/abs/2310.19233

    2023 arXiv

  42. [50]

    Llama 2: Open Foundation and Fine- Tuned Chat Models | Research - AI at Meta,

    M. Facebook, “Llama 2: Open Foundation and Fine- Tuned Chat Models | Research - AI at Meta,” 2023. [Online]. Available: https://ai .meta.com/research/publications/llama-2- open-foundation-and-fine-tuned-chat-models/

    2023

  43. [51]

    Mistral 7B,

    A. Q. Jiang, A. Sablayrolles, and A. e. a. Mensch, “Mistral 7B,” arXiv preprint arXiv:2310.06825, 2023, available: http://arxiv.org/abs/2310.06825

    2023 arXiv

  44. [52]

    Zephyr: Direct Distillation of LM Alignment,

    L. Tunstall, E. Beeching, and et al., “Zephyr: Direct Distillation of LM Alignment,” arXiv preprint arXiv:2310.16944, 2023, available: http://arxiv.org/abs/2310.16944

    2023 arXiv

  45. [53]

    Falcon-40B: an open large language model with state-of-the-art performance,

    E. Almazrouei, H. Alobeidli, and et al., “Falcon-40B: an open large language model with state-of-the-art performance,” 2023

    2023

  46. [54]

    Hugging face,

    H. Face, “Hugging face,” 2024. [Online]. Available: https: //huggingface.co/docs

    2024

  47. [55]

    Training Compute-Optimal Large Language Models,

    J. Hoffmann and e. a. Borgeaud, “Training Compute-Optimal Large Language Models,” arXiv preprint arXiv:2203.15556, 2022, available: http://arxiv.org/abs/2203.15556

    2022 arXiv

  48. [56]

    How to do human evaluation: A brief introduction to user studies in NLP,

    H. Schuff, L. Vanderlyn, H. Adel, and N. T. Vu, “How to do human evaluation: A brief introduction to user studies in NLP,” Natural Language Engineering, vol. 29, no. 5, pp. 1199–1222, 2023

    2023

  49. [57]

    Within-subjects designs: To use or not to use?

    A. G. Greenwald, “Within-subjects designs: To use or not to use?” Psychological Bulletin, vol. 83, no. 2, pp. 314–320, 1976

    1976

  50. [58]

    Towards Evalua- tion Guidelines for Empirical Studies involving LLMs,

    S. Wagner, M. M. Baron, D. Falessi, and S. Baltes, “Towards Evalua- tion Guidelines for Empirical Studies involving LLMs,” arXiv preprint arXiv:2411.07668, 2025, available: http://arxiv.org/abs/2411.07668

    2025 arXiv

  51. [59]

    BERTScore: Evaluating Text Generation with BERT,

    T. Zhang, V . Kishore, F. Wu, K. Q. Weinberger, and Y . Artzi, “BERTScore: Evaluating Text Generation with BERT,” arXiv preprint arXiv:1904.09675, 2020, available: http://arxiv.org/abs/1904.09675

    1904 arXiv

  52. [60]

    Fru- galScore: Learning Cheaper, Lighter and Faster Evaluation Metricsfor Automatic Text Generation,

    M. K. Eddine, G. Shang, A. J.-P. Tixier, and M. Vazirgiannis, “Fru- galScore: Learning Cheaper, Lighter and Faster Evaluation Metricsfor Automatic Text Generation,” arXiv preprint arXiv:2110.08559, 2021, available: http://arxiv.org/abs/2110.08559

    2021 arXiv

  53. [61]

    Bleu: a Method for Automatic Evaluation of Machine Translation,

    K. Papineni, S. Roukos, T. Ward, and W.-J. Zhu, “Bleu: a Method for Automatic Evaluation of Machine Translation,” in Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics , 2002, pp. 311–318

    2002

  54. [62]

    ROUGE: A Package for Automatic Evaluation of Sum- maries,

    C.-Y . Lin, “ROUGE: A Package for Automatic Evaluation of Sum- maries,” in Text Summarization Branches Out , 2004, pp. 74–81

    2004

  55. [63]

    A Study of Translation Edit Rate with Targeted Human Annotation,

    M. Snover, B. Dorr, R. Schwartz, L. Micciulla, and J. Makhoul, “A Study of Translation Edit Rate with Targeted Human Annotation,” in Proceedings of the 7th Conference of the Association for Machine Translation in the Americas: Technical Papers , 2006, pp. 223–231

    2006

  56. [64]

    Exploring Correlation Between ROUGE and Hu- man Evaluation on Meeting Summaries,

    F. Liu and Y . Liu, “Exploring Correlation Between ROUGE and Hu- man Evaluation on Meeting Summaries,” IEEE Transactions on Audio, Speech, and Language Processing, vol. 18, no. 1, pp. 187–196, jan 2010

    2010

  57. [65]

    Comparing Automatic Evaluation Measures for Image Description,

    D. Elliott and F. Keller, “Comparing Automatic Evaluation Measures for Image Description,” in Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), 2014, pp. 452–457

    2014

  58. [66]

    Why We Need New Evaluation Metrics for NLG,

    J. Novikova, O. Du ˇsek, A. C. Curry, and V . Rieser, “Why We Need New Evaluation Metrics for NLG,” in Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing , 2017, pp. 2231– 2240

    2017

  59. [67]

    Well-Read Students Learn Better: On the Importance of Pre-training Com- pact Models,

    I. Turc, M.-W. Chang, K. Lee, and K. Toutanova, “Well-Read Students Learn Better: On the Importance of Pre-training Com- pact Models,” arXiv preprint arXiv:1908.08962, 2019, available: http://arxiv.org/abs/1908.08962

    1908 arXiv

  60. [68]

    A call for greater use of nonparametric statistics

    N. L. Leech and A. J. Onwuegbuzie, “A call for greater use of nonparametric statistics.” Annual Meeting of the Mid-South Educational Research Association, 2002

    2002

  61. [69]

    A practical guide for using statistical tests to assess randomized algorithms in software engineering,

    A. Arcuri and L. Briand, “A practical guide for using statistical tests to assess randomized algorithms in software engineering,” in Proceedings of the 33rd International Conference on Software Engineering . ACM, 2011, pp. 1–10

    2011

  62. [70]

    A Critique and Improvement of the CL Common Language Effect Size Statistics of McGraw and Wong,

    A. Vargha and H. D. Delaney, “A Critique and Improvement of the CL Common Language Effect Size Statistics of McGraw and Wong,” Journal of Educational and Behavioral Statistics , vol. 25, no. 2, pp. 101–132, Jun. 2000

    2000

  63. [71]

    A Simple Sequentially Rejective Multiple Test Procedure,

    S. Holm, “A Simple Sequentially Rejective Multiple Test Procedure,” Scandinavian Journal of Statistics , vol. 6, no. 2, pp. 65–70, 1979

    1979

  64. [72]

    A systematic review of statistical power in software engineering experiments,

    T. Dyba, V . B. Kampenes, and D. I. K. Sjoberg, “A systematic review of statistical power in software engineering experiments,” Information and Software Technology, vol. 48, no. 8, pp. 745–755, Aug. 2006

    2006

  65. [2021]

    Available: https://doi .org/10.5281/zenodo.4471411

    [Online]. Available: https://doi .org/10.5281/zenodo.4471411

    doi

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.