Pith. sign in

REVIEW 4 major objections 6 minor 3 cited by

Multi-Modal Requirements Data-based Acceptance Criteria Generation using LLMs

T0 review · 4 major / 6 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that adding UI screenshots and domain documents as retrieved context yields acceptance criteria that are more relevant, correct, and understandable, with practitioners reporting less manual effort.

desk verdict A plausible multi-modal RAG pipeline for acceptance criteria generation, undermined by in-sample configuration tuning and a small expert panel; worth refereeing but not as-is. read the letter →

arxiv 2508.06888 v1 pith:QNGQEP5B submitted 2025-08-09 cs.SE

Fanyu Wang , Chetan Arora , Yonghui Liu , Kaicheng Huang , Chakkrit Tantithamthavorn , Aldeida Aleti , Dishan Sambathkumar , David Lo This is my paper
classification cs.SE
keywords acceptancecriteriagenerationretrieval-augmentedmulti-modalLLMsrequirementsengineeringuserstoriesUIscreenshotsrewardmodelsindustrialcasestudy
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Acceptance criteria decide when a feature is accepted, but writing them well requires domain knowledge and visual context that user stories usually omit. The paper claims that a retrieval-augmented generation pipeline fed both project documents and UI screenshots produces criteria that are significantly more relevant, correct, and understandable than criteria generated without that context. In an industrial evaluation of a learning management system used by roughly 100,000 users, QA practitioners rated the generated criteria near 4 out of 5 on relevance, correctness, understandability, and coverage, and said the approach would save effort and catch criteria they would have missed. A reward-based polishing step scores the generated set and rewrites the weakest criterion, which the paper reports improves correctness and alignment. If this holds, teams can generate trustworthy acceptance tests from user stories plus artifacts they already maintain.

What carries the argument

The central object is RAGcceptance M2RE, a three-stage generation pipeline. First, a textual retriever ranks paragraph-level chunks of domain documentation (using sentence-embedding similarity or an in-context retrieval-augmented language model), and a visual retriever ranks UI screenshots (via image-to-HTML conversion or direct screenshot embeddings). Second, a prompt is built with the user story, the retrieved text, and the retrieved visual content, and an LLM with Chain-of-Thought prompting emits atomic GIVEN/WHEN/THEN criteria. Third, a two-level reward model checks the whole set against a five-level quality rubric; if the set scores below threshold, a local reward model identifies the w

What would settle it

Run a preregistered blind study on a second industrial system with 30+ fresh user stories: have an independent panel rate criteria from the full multi-modal pipeline against criteria from the same LLM with only textual retrieval, presented in random order. If the multi-modal criteria do not receive higher mean relevance and coverage ratings, the paper's claim that integrating UI screenshots significantly enhances acceptance-criteria quality is falsified.

Watch

Extended reading notes

Core claim

The central claim is that RAGcceptance M2RE—a pipeline that retrieves relevant textual domain passages and UI screenshots, feeds them to an LLM alongside the user story, and then polishes the output with reward models—generates acceptance criteria that are more relevant, correct, and comprehensible than text-only or no-retrieval generation. The paper's stepwise evaluation reports that the best configuration, selected across retrieval, generation, and polishing stages, improves LLM-judged correctness and hit rates at the criterion level, while a practitioner survey on 17 user stories and 81 generated criteria rates the output close to 4 out of 5 on relevance, correctness, understandability, a

Load-bearing premise

The load-bearing premise is that consensus ratings from three practitioners in one two-hour session, averaged over 81 generated criteria, and the best configuration selected on that same data, form a reliable measure of acceptance-criteria quality; if the panel is too small, unrepresentative, or biased, or the tuning overfits, the claimed enhancement is not established.

Editorial extensions

If this is right

  • Analysts new to a project can produce acceptance criteria at a level close to experienced staff because the retrieval step supplies the domain context they lack.
  • UI-intensive features benefit most from the visual retrieval branch, so teams can collect and index screenshots for stories where layout, labels, or visual states matter.
  • Reward-based polishing gives QA teams a quality gate before human review: whole sets are scored, weak criteria are flagged and rewritten automatically.
  • The same retrieval-plus-polish structure is not specific to acceptance criteria and can be extended to other requirements-driven artifacts such as test scenarios and validation checklists, as the paper's future-work agenda anticipates.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A blind head-to-head trial on a second industrial product—where the same LLM gets either full multi-modal context or text-only context, and practitioners do not know which is which—would isolate the contribution of UI screenshots; the paper's stepwise design compares configurations incrementally, so such a trial is the natural next check.
  • The three visual retrieval variants encode screenshots as full HTML, pruned HTML, or raw embeddings, so the released dataset could be mined for a mapping between UI element types and the best encoding; the paper does not draw that mapping.
  • Because the reward loop emits interpretable quality scores, the architecture could accept practitioner corrections as the reward signal itself, letting a project's terminology and standards be absorbed without prompt rewriting.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes RAGcceptance M2RE, a multi-modal retrieval-augmented generation (RAG) pipeline for automatically generating acceptance criteria (ACs) from user stories, optionally augmented with textual domain documents and UI screenshots. The pipeline has three stages: text and visual retrieval (T-RAG and V-RAG), generation with prompt-construction variants and several LLMs, and reward-based post-processing that polishes low-quality ACs. The authors evaluate the approach on two feature sets from the UniLearn system at eSolutions, using ranking metrics for retrieval, LLM-based quality metrics for generation, and a practitioner survey with three experts on 17 user stories and 81 ACs. They report that integrating multi-modal information significantly enhances relevance, correctness, and comprehensibility, reduces manual effort, and captures stakeholder intent. The implementation and an anonymized dataset are made available.

Significance. If the central claim were established, the approach would be a practically valuable contribution to requirements-driven QA automation, especially in UI-intensive industrial settings. The paper's strengths include an industrial case study with real practitioner involvement, an explicit multi-modal RAG architecture, and the release of implementation artifacts and a dataset. These are concrete assets that support replication and follow-up work. However, the current evidence does not support the strength of the stated claim: the configuration search is performed on the same ground truth used to report final results, the expert sample is very small and not blinded to conditions, and no statistical significance tests are provided. The approach is novel and plausible, but the empirical support needs substantial strengthening before the central claims can be accepted.

major comments (4)
  1. [§IV-D, Table II] The stepwise configuration search is conducted on the same ground truth data on which the final results are reported. The text says: 'Given the ground truth from UniLearn data, we conducted a stepwise evaluation in RQ1 and RQ2 to determine the best configuration.' With 2×4×3×4×2×3×2×3 = 3,456 possible combinations, a greedy stepwise selection on the same data can overfit, meaning the reported gains of the multi-modal configuration over the w/o-RAG and w/o-VRAG baselines may be an in-sample optimum rather than an unbiased estimate. A held-out split, cross-validation, or at least a clear separation between configuration-selection data and evaluation data is required. This issue is load-bearing because the central claim of 'significant enhancement' rests on the comparison to baselines under the selected configuration.
  2. [§IV-C, §V] The expert evaluation is too limited to support the claim that multi-modal RAG 'significantly enhances' relevance, correctness, and comprehensibility. The study uses three eSolutions experts, 17 user stories, and 81 ACs, rated in a single two-hour consensus session. There is no inter-rater reliability metric, no report of individual ratings before consensus, and no statistical significance test (the reported Likert averages near 4/5 are descriptive only). More importantly, the experts rate only the final selected configuration; they are not shown text-only or no-RAG outputs in a blinded or counterbalanced design. Thus the practitioner ratings cannot validate the incremental contribution of RAG or of the visual modality. The threats-to-validity section does not acknowledge this selection-bias and blinding limitation.
  3. [§III-C, Table V] There is a circularity risk between the reward model used for polishing and the LLM-based evaluation metrics. The global reward model Prometheus uses a rubric 'designed based on the dimensions in our expert evaluation (see Sec. IV-C)', and the evaluation in RQ2 uses LLM-annotated Hit/Correct and pairwise preference metrics. If the reward model and the judge share the same criteria or preferences, the polishing step may be rewarded by the evaluator for making outputs conform to the reward model's own notion of quality, rather than to an independent standard. The paper should clarify how the evaluator LLMs differ from the reward model, and should report human-judge agreement on a held-out sample to break this circularity.
  4. [Algorithm 1, Table II] The global reward threshold θ and the six-dimensional rubric are free parameters of the pipeline, but they are not listed in the configuration choices of Table II, and their values are not reported. The sentence 'we maintain global reward model as an integral part of our approach' does not explain how θ is chosen. If θ was tuned on the same ground truth, it is another source of selection bias; if it was set arbitrarily, the robustness of the polishing step to θ is unknown. A sensitivity analysis over θ and a clear statement of which parameters were optimized and on which data are necessary for reproducibility and for interpreting the polishing results.
minor comments (6)
  1. [§III-B, Fig. 4] The APEER prompt template in Fig. 4 is rendered as a garbled placeholder in the PDF; readers cannot inspect the actual prompt. Please include a readable prompt in the appendix or repository.
  2. [Table V] The row labels such as 'Claude_Reward' and the 'NA' entry under Compare are not defined. Please explain the naming convention and the meaning of each baseline and metric.
  3. [§IV-D] The sentence 'while the generation ability is strongly determined by the generative methods, such as LLMs, Pretrained LMs, etc, they separately evaluate the information retrieval process...' is grammatically unclear. Please rewrite to describe precisely the stepwise procedure and the order in which variables are selected.
  4. [§V] The threats-to-validity section should explicitly discuss the risk of overfitting from the stepwise configuration search, the small size of the expert panel, and the lack of blind comparisons to baselines. Currently these are not mentioned.
  5. [Algorithm 1] The algorithm polishes only the single worst AC per invocation and returns without an iterative loop; the text does not clarify whether one iteration is intended or whether multiple low-quality ACs are handled in practice.
  6. [§III-A] Minor typo: 'I n the first alternative' should be 'In the first alternative'.

Circularity Check

1 steps flagged · score 6.0 of 10

Multi-modal enhancement claim rests on an in-sample configuration fit: the best configuration is selected on the same ground truth used to report the improvements.

  1. fitted input called prediction [Section IV-D, 'Evaluation Procedure and Metrics' (Table II, Table V); abstract claim]
    "Given the ground truth from UniLearn data, we conducted a stepwise evaluation in RQ1 and RQ2 to determine the best configuration for RAGcceptance M2RE in three steps, including 1) information retrieval, 2) AC generation, and 3) AC polishing."

    The stepwise search selects the best configuration (including whether to use T-RAG, V-RAG, or neither) by maximizing metrics computed against the same UniLearn ground truth that is later used to report the 'significant enhancement' of multi-modal RAG over the w/o-RAG and w/o-VRAG baselines. Since those baselines are explicitly among the eight choices in the search space (Table II), the best multi-modal configuration is, by construction, at least as good as the baselines on that same data. The reported improvement is therefore an in-sample optimum, not an unbiased prediction, and the abstract's claim that 'integrating multi-modal information significantly enhances' relevance/correctness/comprehensibility is not independently established. The expert survey only rates the final selected confi

full rationale

The paper's central claim—that multi-modal RAG significantly improves acceptance-criteria quality—is evaluated by first choosing the best configuration on the ground-truth data (Section IV-D) and then reporting that same configuration's metrics as the evidence. Because the 'w/o RAG' and 'w/o VRAG' baselines are part of the configuration space, the selected best configuration is guaranteed to match or beat them on the selection data. No held-out split is reported, and the threats-to-validity section does not acknowledge this selection-bias/overfitting issue. The expert survey provides independent evidence that the final output is useful to practitioners, but it was conducted only on the already-selected configuration and without modality-blinded comparisons, so it cannot validate the incremental benefit of visual information. No load-bearing self-citation chain or reward-model self-evaluation was established from the text alone: the LLM-as-judge identity used in the RQ2 tables is not specified. The primary circularity is therefore the in-sample fit of the configuration that supports the headline comparison, placing the paper at partial circularity rather than full equivalence.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The paper introduces no new physical or conceptual entities; it composes existing RAG, embedding, and reward-model components. The main unobserved supports are the validity of retrieval similarity, the correctness of the industrial ground truth, the reliability of a three-expert panel, and the alignment of LLM judges with human quality.

free parameters (4)
  • k for T-RAG (number of textual passages retrieved) = selected from {1, 5, 10, 20} via stepwise evaluation
    The paper tunes this on the evaluation data for each research question; the choice affects retrieval quality and thus the generated ACs.
  • k for V-RAG (number of visual documents retrieved) = selected from {1, 5, 10, 20}
    Similarly tuned per configuration; controls how many UI screenshots or HTML representations are fed to the LLM.
  • Global reward threshold theta = not specified in the text
    Algorithm 1 uses a threshold to decide whether to polish the AC set; the value is chosen by the authors and no sensitivity analysis is given.
  • Six-dimensional rubric levels for Prometheus global reward model = 5 quality levels
    Hand-crafted from the expert evaluation dimensions; used to score overall AC quality and guide polishing, but not independently validated.
assumptions (4)
  • domain assumption Embedding-based cosine similarity between user story and domain text/images is a valid proxy for relevance
    Used throughout Section III-A to retrieve context; if retrieval is poor, RAG can introduce noise rather than help.
  • domain assumption The ground-truth ACs from UniLearn used in RQ1 and RQ2 are correct and complete references
    Section IV-D evaluates retrieval and generation against these ground-truth ACs; no independent verification of their quality is provided.
  • domain assumption Expert Likert ratings and consensus from three practitioners in a two-hour session reliably measure AC quality
    Section IV-C; small sample, no inter-rater reliability or statistical testing reported.
  • ad hoc to paper LLM-as-judge outputs (Prometheus, etc.) reflect human quality judgments
    Used as global and local reward models and as evaluation annotators; potential for self-preference bias since the generator and judge are both LLMs.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Multi-Modal Requirements Data-based Acceptance Criteria Generation using LLMs." pith.science (2026). https://pith.science/paper/QNGQEP5B

@misc{pith2026250806888,
  author       = {Pith},
  title        = {Pith review of: Multi-Modal Requirements Data-based Acceptance Criteria Generation using LLMs},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/QNGQEP5B}},
  note         = {Machine review of arXiv:2508.06888}
}
read the original abstract

Acceptance criteria (ACs) play a critical role in software development by clearly defining the conditions under which a software feature satisfies stakeholder expectations. However, manually creating accurate, comprehensive, and unambiguous acceptance criteria is challenging, particularly in user interface-intensive applications, due to the reliance on domain-specific knowledge and visual context that is not always captured by textual requirements alone. To address these challenges, we propose RAGcceptance M2RE, a novel approach that leverages Retrieval-Augmented Generation (RAG) to generate acceptance criteria from multi-modal requirements data, including both textual documentation and visual UI information. We systematically evaluated our approach in an industrial case study involving an education-focused software system used by approximately 100,000 users. The results indicate that integrating multi-modal information significantly enhances the relevance, correctness, and comprehensibility of the generated ACs. Moreover, practitioner evaluations confirm that our approach effectively reduces manual effort, captures nuanced stakeholder intent, and provides valuable criteria that domain experts may overlook, demonstrating practical utility and significant potential for industry adoption. This research underscores the potential of multi-modal RAG techniques in streamlining software validation processes and improving development efficiency. We also make our implementation and a dataset available.

Discussion (0). Sign in to comment.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Guidelines for Empirical Studies in Software Engineering involving Large Language Models

    cs.SE 2025-08 accept novelty 7.0 of 10

    The paper delivers a taxonomy of seven LLM study types in software engineering along with eight guidelines that separate mandatory requirements from recommended practices to address reproducibility challenges.

  2. LLMCFG-TGen: Using LLM-Generated Control Flow Graphs to Automatically Create Test Cases from Use Cases

    cs.SE 2025-12 conditional novelty 6.0 of 10

    An LLM-generated control-flow graph intermediate representation improves completeness and reduces redundancy in test cases generated from natural-language use cases.

  3. Guidelines for Empirical Studies in Software Engineering involving Large Language Models

    cs.SE 2025-08 accept novelty 6.0 of 10

    A group of 22 researchers proposes seven study types and eight guidelines for empirical software engineering studies involving LLMs to enhance reproducibility and replicability.

Reference graph

Works this paper leans on

89 extracted references · 51 canonical work pages · cited by 2 Pith papers

  1. [1]

    Automatic creation of acceptance tests by extracting conditionals from requirements: Nlp approach and case study,

    J. Fischbach, J. Frattini, A. V ogelsang, D. Mendez, M. Unterkalmsteiner, A. Wehrle, P. R. Henao, P. Yousefi, T. Juricic, J. Radduenz et al. , “Automatic creation of acceptance tests by extracting conditionals from requirements: Nlp approach and case study,” Journal of Systems and Software, vol. 197, p. 111549, 2023

    2023

  2. [2]

    Test case generation for agent-based models: A systematic literature review,

    A. G. Clark, N. Walkinshaw, and R. M. Hierons, “Test case generation for agent-based models: A systematic literature review,”Information and Software Technology, vol. 135, p. 106567, 2021

    2021

  3. [3]

    A review on test automation for test cases generation using nlp techniques,

    S. Perala and A. Roy, “A review on test automation for test cases generation using nlp techniques,” Turkish Journal of Computer and Mathematics Education, vol. 12, no. 6, pp. 1488–1491, 2021

    2021

  4. [4]

    What makes agile test artifacts useful? an activity-based quality model from a practitioners’ perspective,

    J. Fischbach, H. Femmer, D. Mendez, D. Fucci, and A. V ogelsang, “What makes agile test artifacts useful? an activity-based quality model from a practitioners’ perspective,” in Proceedings of the 14th ACM/IEEE International Symposium on Empirical Software Engineering and Mea- surement (ESEM), 2020, pp. 1–10

    2020

  5. [5]

    Smells in system user interactive tests,

    R. Rwemalika, S. Habchi, M. Papadakis, Y . Le Traon, and M.-C. Brasseur, “Smells in system user interactive tests,” Empirical Software Engineering, vol. 28, no. 1, p. 20, 2023

    2023

  6. [6]

    Automated acceptance tests as software requirements: An experiment to compare the applicability of fit tables and gherkin language,

    E. C. dos Santos and P. Vilain, “Automated acceptance tests as software requirements: An experiment to compare the applicability of fit tables and gherkin language,” in International conference on agile software development. Springer, 2018, pp. 104–119

    2018

  7. [7]

    Comprehensive evaluation and insights into the use of large language models in the automation of behavior-driven development acceptance test formulation,

    S. Karpurapu, S. Myneni, U. Nettur, L. S. Gajja, D. Burke, T. Stiehm, and J. Payne, “Comprehensive evaluation and insights into the use of large language models in the automation of behavior-driven development acceptance test formulation,” IEEE Access, 2024

    2024

  8. [8]

    Requirements-driven automated software testing: A systematic review,

    F. Wang, C. Arora, C. Tantithamthavorn, K. Huang, and A. Aleti, “Requirements-driven automated software testing: A systematic review,”

Show all 89 references
  1. [9]

    Large language models for software engineering: Sur- vey and open problems,

    A. Fan, B. Gokkaya, M. Harman, M. Lyubarskiy, S. Sengupta, S. Yoo, and J. M. Zhang, “Large language models for software engineering: Sur- vey and open problems,” in 2023 IEEE/ACM International Conference on Software Engineering: Future of Software Engineering (ICSE-FoSE). IEEE...

    2023

  2. [10]

    Generative artificial intelligence for software engineering–a research agenda,

    A. Nguyen-Duc, B. Cabrero-Daniel, A. Przybylek, C. Arora, D. Khanna, T. Herda, U. Rafiq, J. Melegati, E. Guerra, K.-K. Kemell et al. , “Generative artificial intelligence for software engineering–a research agenda,” arXiv preprint arXiv:2310.18648 , 2023

    2023 arXiv

  3. [11]

    Domain knowledge is all you need: A field deployment of llm-powered test case generation in fintech domain,

    Z. Xue, L. Li, S. Tian, X. Chen, P. Li, L. Chen, T. Jiang, and M. Zhang, “Domain knowledge is all you need: A field deployment of llm-powered test case generation in fintech domain,” in Proceedings of the 2024 IEEE/ACM 46th International Conference on Software Engineering: Com...

    2024

  4. [12]

    On the effectiveness of large language models in domain- specific code generation,

    X. Gu, M. Chen, Y . Lin, Y . Hu, H. Zhang, C. Wan, Z. Wei, Y . Xu, and J. Wang, “On the effectiveness of large language models in domain- specific code generation,” ACM Transactions on Software Engineering and Methodology, vol. 34, no. 3, pp. 1–22, 2025

    2025

  5. [13]

    Enhancing large language models through external domain knowledge,

    L. Welz and C. Lanquillon, “Enhancing large language models through external domain knowledge,” in International Conference on Human- Computer Interaction. Springer, 2024, pp. 135–146

    2024

  6. [14]

    Retrieval- augmented generation for knowledge-intensive nlp tasks,

    P. Lewis, E. Perez, A. Piktus, F. Petroni, V . Karpukhin, N. Goyal, H. K ¨uttler, M. Lewis, W.-t. Yih, T. Rockt ¨aschel et al. , “Retrieval- augmented generation for knowledge-intensive nlp tasks,” Advances in neural information processing systems , vol. 33, pp. 9459–9474, 2020

    2020

  7. [15]

    Generating test scenarios from nl requirements using retrieval-augmented llms: An industrial study,

    C. Arora, T. Herda, and V . Homm, “Generating test scenarios from nl requirements using retrieval-augmented llms: An industrial study,” in 2024 IEEE 32nd International Requirements Engineering Conference (RE). IEEE, 2024, pp. 240–251

    2024

  8. [16]

    Cohn, User stories applied: For agile software development

    M. Cohn, User stories applied: For agile software development . Addison-Wesley Professional, 2004

    2004

  9. [17]

    Artefact Repository: Multi- Modal Requirements Data based Acceptance Criteria Generation using LLMs

    Anonymous Author, “Artefact Repository: Multi- Modal Requirements Data based Acceptance Criteria Generation using LLMs.” online; accessed 15 Mar

  10. [18]

    Evaluation of retrieval-augmented generation: A survey,

    H. Yu, A. Gan, K. Zhang, S. Tong, Q. Liu, and Z. Liu, “Evaluation of retrieval-augmented generation: A survey,” in CCF Conference on Big Data. Springer, 2024, pp. 102–120

    2024

  11. [19]

    Available: https://anonymous.4open.science/r/ Multi-Modal-Requirements-Data-based-Acceptance-Criteria-Generation-using-LLMs-1279/

    [Online]. Available: https://anonymous.4open.science/r/ Multi-Modal-Requirements-Data-based-Acceptance-Criteria-Generation-using-LLMs-1279/

  12. [20]

    Vul-rag: Enhancing llm-based vulnerability detection via knowledge-level rag,

    X. Du, G. Zheng, K. Wang, J. Feng, W. Deng, M. Liu, B. Chen, X. Peng, T. Ma, and Y . Lou, “Vul-rag: Enhancing llm-based vulnerability detection via knowledge-level rag,” arXiv preprint arXiv:2406.11147 , 2024

    2024 arXiv

  13. [21]

    A comprehensive survey of retrieval-augmented generation (rag): Evolution, current landscape and future directions,

    S. Gupta, R. Ranjan, and S. N. Singh, “A comprehensive survey of retrieval-augmented generation (rag): Evolution, current landscape and future directions,” arXiv preprint arXiv:2410.12837 , 2024

    2024 arXiv

  14. [22]

    Reveal: Retrieval-augmented visual-language pre-training with multi-source multimodal knowledge memory,

    Z. Hu, A. Iscen, C. Sun, Z. Wang, K.-W. Chang, Y . Sun, C. Schmid, D. A. Ross, and A. Fathi, “Reveal: Retrieval-augmented visual-language pre-training with multi-source multimodal knowledge memory,” in Pro- ceedings of the IEEE/CVF conference on computer vision and pattern rec...

    2023

  15. [23]

    A survey on rag meeting llms: Towards retrieval-augmented large language models,

    W. Fan, Y . Ding, L. Ning, S. Wang, H. Li, D. Yin, T.-S. Chua, and Q. Li, “A survey on rag meeting llms: Towards retrieval-augmented large language models,” in Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining , ser. KDD ’24. New York, NY , U...

    2024

  16. [24]

    Improvements to bm25 and language models examined,

    A. Trotman, A. Puurula, and B. Burgess, “Improvements to bm25 and language models examined,” in Proceedings of the 19th Australasian Document Computing Symposium , 2014, pp. 58–65

    2014

  17. [25]

    Using tf-idf to determine word relevance in document queries,

    J. Ramos et al., “Using tf-idf to determine word relevance in document queries,” in Proceedings of the first instructional conference on machine learning, vol. 242, no. 1. Citeseer, 2003, pp. 29–48

    2003

  18. [26]

    Maximizing rag efficiency: A comparative analysis of rag methods,

    T. S ¸akar and H. Emekci, “Maximizing rag efficiency: A comparative analysis of rag methods,” Natural Language Processing, vol. 31, no. 1, pp. 1–25, 2025

    2025

  19. [27]

    Cos-mix: cosine similarity and dis- tance fusion for improved information retrieval,

    K. Juvekar and A. Purwar, “Cos-mix: cosine similarity and dis- tance fusion for improved information retrieval,” arXiv preprint arXiv:2406.00638, 2024

    2024 arXiv

  20. [28]

    In-context retrieval-augmented language mod- els,

    O. Ram, Y . Levine, I. Dalmedigos, D. Muhlgay, A. Shashua, K. Leyton- Brown, and Y . Shoham, “In-context retrieval-augmented language mod- els,” Transactions of the Association for Computational Linguistics , vol. 11, pp. 1316–1331, 2023

    2023

  21. [29]

    Sentence-bert: Sentence embeddings using siamese bert-networks,

    N. Reimers and I. Gurevych, “Sentence-bert: Sentence embeddings using siamese bert-networks,” arXiv preprint arXiv:1908.10084 , 2019

    1908 arXiv

  22. [30]

    Retrieval-augmented multi- modal language modeling,

    M. Yasunaga, A. Aghajanyan, W. Shi, R. James, J. Leskovec, P. Liang, M. Lewis, L. Zettlemoyer, and W.-t. Yih, “Retrieval-augmented multi- modal language modeling,” arXiv preprint arXiv:2211.12561 , 2022

    2022 arXiv

  23. [31]

    Murag: Multimodal retrieval-augmented generator for open question answering over images and text,

    W. Chen, H. Hu, X. Chen, P. Verga, and W. W. Cohen, “Murag: Multimodal retrieval-augmented generator for open question answering over images and text,” arXiv preprint arXiv:2210.02928 , 2022

    2022 arXiv

  24. [32]

    Rate: Score reward models with imperfect rewrites of rewrites,

    D. Reber, S. Richardson, T. Nief, C. Garbacea, and V . Veitch, “Rate: Score reward models with imperfect rewrites of rewrites,” arXiv preprint arXiv:2410.11348, 2024

    2024 arXiv

  25. [33]

    Mastering the game of go without human knowledge,

    D. Silver, J. Schrittwieser, K. Simonyan, I. Antonoglou, A. Huang, A. Guez, T. Hubert, L. Baker, M. Lai, A. Bolton et al. , “Mastering the game of go without human knowledge,” nature, vol. 550, no. 7676, pp. 354–359, 2017

    2017

  26. [34]

    Muse: Modularizing unsupervised sense embeddings,

    G.-H. Lee and Y .-N. Chen, “Muse: Modularizing unsupervised sense embeddings,” arXiv preprint arXiv:1704.04601 , 2017

    2017 arXiv

  27. [35]

    Rm-bench: Benchmarking reward models of language models with subtlety and style,

    Y . Liu, Z. Yao, R. Min, Y . Cao, L. Hou, and J. Li, “Rm-bench: Benchmarking reward models of language models with subtlety and style,” arXiv preprint arXiv:2410.16184 , 2024

    2024 arXiv

  28. [36]

    Balancing the scales: Reinforce- ment learning for fair classification,

    L. Eshuijs, S. Wang, and A. Fokkens, “Balancing the scales: Reinforce- ment learning for fair classification,” arXiv preprint arXiv:2407.10629 , 2024

    2024 arXiv

  29. [37]

    Inferring lexicographically-ordered rewards from preferences,

    A. H ¨uy¨uk, W. R. Zame, and M. van der Schaar, “Inferring lexicographically-ordered rewards from preferences,” in Proceedings of the AAAI Conference on Artificial Intelligence , vol. 36, no. 5, 2022, pp. 5737–5745

    2022

  30. [38]

    Helpsteer2-preference: Complementing ratings with preferences,

    Z. Wang, A. Bukharin, O. Delalleau, D. Egert, G. Shen, J. Zeng, O. Kuchaiev, and Y . Dong, “Helpsteer2-preference: Complementing ratings with preferences,” arXiv preprint arXiv:2410.01257 , 2024

    2024 arXiv

  31. [39]

    Prometheus 2: An open source language model specialized in evaluating other language models,

    S. Kim, J. Suk, S. Longpre, B. Y . Lin, J. Shin, S. Welleck, G. Neubig, M. Lee, K. Lee, and M. Seo, “Prometheus 2: An open source language model specialized in evaluating other language models,” arXiv preprint arXiv:2405.01535, 2024

    2024 arXiv

  32. [40]

    Llms- as-judges: a comprehensive survey on llm-based evaluation methods,

    H. Li, Q. Dong, J. Chen, H. Su, Y . Zhou, Q. Ai, Z. Ye, and Y . Liu, “Llms- as-judges: a comprehensive survey on llm-based evaluation methods,” arXiv preprint arXiv:2412.05579 , 2024

    2024 arXiv

  33. [41]

    Judging the judges: Evaluating alignment and vulnerabili- ties in llms-as-judges,

    A. S. Thakur, K. Choudhary, V . S. Ramayapally, S. Vaidyanathan, and D. Hupkes, “Judging the judges: Evaluating alignment and vulnerabili- ties in llms-as-judges,” arXiv preprint arXiv:2406.12624 , 2024

    2024 arXiv

  34. [42]

    Improving zero-shot LLM re-ranker with risk minimization,

    X. Yuan, Z. Yang, Y . Wang, J. Zhao, and K. Liu, “Improving zero-shot LLM re-ranker with risk minimization,” in Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, Y . Al-Onaizan, M. Bansal, and Y .-N. Chen, Eds. Miami, Florida, USA: Associa...

    2024

  35. [43]

    Generative verifiers: Reward modeling as next-token prediction,

    L. Zhang, A. Hosseini, H. Bansal, M. Kazemi, A. Kumar, and R. Agar- wal, “Generative verifiers: Reward modeling as next-token prediction,” arXiv preprint arXiv:2408.15240 , 2024

    2024 arXiv

  36. [44]

    Reducing requirements ambiguity via gamification: comparison with traditional techniques,

    H. S. Dar, S. Imtiaz, and M. I. Lali, “Reducing requirements ambiguity via gamification: comparison with traditional techniques,” Computa- tional Intelligence and Neuroscience, vol. 2022, no. 1, p. 3183411, 2022

    2022

  37. [45]

    Automated test case generation from requirements: A systematic literature review,

    A. Mustafa, W. M. Wan-Kadir, N. Ibrahim, M. A. Shah, M. Younas, A. Khan, M. Zareei, and F. Alanazi, “Automated test case generation from requirements: A systematic literature review,” Computers, Materi- als and Continua , vol. 67, no. 2, pp. 1819–1833, 2021

    2021

  38. [46]

    Regression test selection on system requirements,

    P. K. Chittimalli and M. J. Harrold, “Regression test selection on system requirements,” in Proceedings of the 1st India software engineering conference, 2008, pp. 87–96

    2008

  39. [47]

    Gam- ify4lexamb: a gamification-based approach to address lexical ambiguity in natural language requirements,

    H. Dar, R. Aziz, J. A. Khan, M. I. Lali, and N. A. Almujally, “Gam- ify4lexamb: a gamification-based approach to address lexical ambiguity in natural language requirements,” PeerJ Computer Science, vol. 10, p. e2229, 2024

    2024

  40. [48]

    Representation of knowledge from software requirements expressed in natural language,

    R. P. Verma and M. R. Beg, “Representation of knowledge from software requirements expressed in natural language,” in 2013 6th International Conference on Emerging Trends in Engineering and Technology. IEEE, 2013, pp. 154–158

    2013

  41. [49]

    Reqcap: Hierarchical requirements modeling and test generation for industrial control systems,

    A. Almohammad, J. F. Ferreira, A. Mendes, and P. White, “Reqcap: Hierarchical requirements modeling and test generation for industrial control systems,” in 2017 IEEE 25th International Requirements Engi- neering Conference Workshops (REW) . IEEE, 2017, pp. 351–358

    2017

  42. [50]

    A multi- case study of agile requirements engineering and the use of test cases as requirements,

    E. Bjarnason, M. Unterkalmsteiner, M. Borg, and E. Engstr ¨om, “A multi- case study of agile requirements engineering and the use of test cases as requirements,” Information and Software Technology, vol. 77, pp. 61–79, 2016

    2016

  43. [51]

    Aat4irs: automated acceptance testing for industrial robotic systems,

    M. G. Dos Santos, S. Hall ´e, F. Petrillo, and Y .-G. Gu´eh´eneuc, “Aat4irs: automated acceptance testing for industrial robotic systems,” Frontiers in Robotics and AI , vol. 11, p. 1346580, 2024

    2024

  44. [52]

    Torc: test plan optimiza- tion by requirements clustering,

    B. G ¨uldali, H. Funke, S. Sauer, and G. Engels, “Torc: test plan optimiza- tion by requirements clustering,” Software Quality Journal, vol. 19, pp. 771–799, 2011

    2011

  45. [53]

    Exploring llms impact on student-created user stories and acceptance testing in software development,

    A. Brockenbrough, H. Feild, and D. Salinas, “Exploring llms impact on student-created user stories and acceptance testing in software development,” in Proceedings of the 56th ACM Technical Symposium on Computer Science Education V . 2, 2025, pp. 1401–1402

    2025

  46. [54]

    Automating acceptance testing with tool support,

    T. Straszak and M. ´Smialek, “Automating acceptance testing with tool support,” in 2014 Federated Conference on Computer Science and Information Systems. IEEE, 2014, pp. 1569–1574

    2014

  47. [55]

    Automatic generation of acceptance test cases from use case specifications: an nlp-based approach,

    C. Wang, F. Pastore, A. Goknil, and L. C. Briand, “Automatic generation of acceptance test cases from use case specifications: an nlp-based approach,” IEEE Transactions on Software Engineering , vol. 48, no. 2, pp. 585–616, 2020

    2020

  48. [56]

    V ogelsang and J

    A. V ogelsang and J. Fischbach, Using Large Language Models for Natural Language Processing Tasks in Requirements Engineering: A Systematic Guideline . Cham: Springer Nature Switzerland, 2025, pp. 435–456

    2025

  49. [57]

    Advancing requirements engineering through generative ai: Assessing the role of llms,

    C. Arora, J. Grundy, and M. Abdelrazek, “Advancing requirements engineering through generative ai: Assessing the role of llms,” in Generative AI for Effective Software Development . Springer, 2024, pp. 129–148

    2024

  50. [58]

    Prompting large language models with chain-of-thought for few-shot knowledge base question generation,

    Y . Liang, J. Wang, H. Zhu, L. Wang, W. Qian, and Y . Lan, “Prompting large language models with chain-of-thought for few-shot knowledge base question generation,” ArXiv, vol. abs/2310.08395, 2023. [Online]. Available: https://api.semanticscholar.org/CorpusID:263909537

    2023 arXiv

  51. [59]

    Prompt engineering for requirements engineering: A literature review and roadmap,

    K. Huang, F. Wang, Y . Huang, and C. Arora, “Prompt engineering for requirements engineering: A literature review and roadmap,” arXiv preprint arXiv:2507.07682, 2025

    2025 arXiv

  52. [60]

    Xai meets llms: A survey of the relation between explainable ai and large language models,

    E. Cambria, L. Malandri, F. Mercorio, N. Nobani, and A. Seveso, “Xai meets llms: A survey of the relation between explainable ai and large language models,” arXiv preprint arXiv:2407.15248 , 2024

    2024 arXiv

  53. [61]

    Apeer: Automatic prompt en- gineering enhances large language model reranking,

    C. Jin, H. Peng, S. Zhao, Z. Wang, W. Xu, L. Han, J. Zhao, K. Zhong, S. Rajasekaran, and D. N. Metaxas, “Apeer: Automatic prompt en- gineering enhances large language model reranking,” arXiv preprint arXiv:2406.14449, 2024

    2024 arXiv

  54. [62]

    Enhancing noise robustness of retrieval-augmented language models with adaptive adversarial training,

    F. Fang, Y . Bai, S. Ni, M. Yang, X. Chen, and R. Xu, “Enhancing noise robustness of retrieval-augmented language models with adaptive adversarial training,” arXiv preprint arXiv:2405.20978 , 2024

    2024 arXiv

  55. [63]

    Navigating llm ethics: Advancements, challenges, and future directions,

    J. Jiao, S. Afroogh, Y . Xu, and C. Phillips, “Navigating llm ethics: Advancements, challenges, and future directions,” arXiv preprint arXiv:2406.18841, 2024

    2024 arXiv

  56. [64]

    Llama-3.2-3B-Instruct,

    Meta, “Llama-3.2-3B-Instruct,” online; accessed 15 Jan 2025. [Online]. Available: https://huggingface.co/meta-llama/Llama-3.2-3B-Instruct

    2025

  57. [65]

    An information bottleneck perspective for effec- tive noise filtering on retrieval-augmented generation,

    K. Zhu, X. Feng, X. Du, Y . Gu, W. Yu, H. Wang, Q. Chen, Z. Chu, J. Chen, and B. Qin, “An information bottleneck perspective for effec- tive noise filtering on retrieval-augmented generation,” arXiv preprint arXiv:2406.01549, 2024

    2024

  58. [66]

    Htmlrag: Html is better than plain text for modeling retrieved knowledge in rag systems,

    J. Tan, Z. Dou, W. Wang, M. Wang, W. Chen, and J.-R. Wen, “Htmlrag: Html is better than plain text for modeling retrieved knowledge in rag systems,” arXiv preprint arXiv:2411.02959 , 2024

    2024 arXiv

  59. [67]

    Screenshot-to-code,

    abi, “Screenshot-to-code,” online; accessed 15 Jan 2025. [Online]. Available: https://github.com/abi/screenshot-to-code

    2025

  60. [68]

    all-MiniLM-L12-v2 ,

    Sentence Transformers, “all-MiniLM-L12-v2 ,” online; accessed 15 Jan

  61. [69]

    dse-phi3-docmatix-v2,

    Tevatron, “dse-phi3-docmatix-v2,” online; accessed 15 Jan 2025. [On- line]. Available: https://huggingface.co/Tevatron/dse-phi3-docmatix-v2

    2025

  62. [70]

    [Online]

    Atlassian, “Jira,” online; accessed 15 Jan 2025. [Online]. Available: https://www.atlassian.com/software/jira

    2025

  63. [71]

    Available: https://huggingface.co/sentence-transformers/ all-MiniLM-L12-v2

    [Online]. Available: https://huggingface.co/sentence-transformers/ all-MiniLM-L12-v2

  64. [72]

    Llama-3.1-8B-Instruct,

    Meta, “Llama-3.1-8B-Instruct,” online; accessed 15 Jan 2025. [Online]. Available: https://huggingface.co/meta-llama/Llama-3.1-8B-Instruct

    2025

  65. [73]

    Retrieval-augmented generation for large language models: A survey,

    Y . Gao, Y . Xiong, X. Gao, K. Jia, J. Pan, Y . Bi, Y . Dai, J. Sun, H. Wang, and H. Wang, “Retrieval-augmented generation for large language models: A survey,” arXiv preprint arXiv:2312.10997 , vol. 2, 2023

    2023 arXiv

  66. [74]

    A technique for the measurement of attitudes

    R. Likert, “A technique for the measurement of attitudes.” Archives of psychology, 1932

    1932

  67. [75]

    Retrieval-augmented generation for ai-generated content: A survey,

    P. Zhao, H. Zhang, Q. Yu, Z. Wang, Y . Geng, F. Fu, L. Yang, W. Zhang, J. Jiang, and B. Cui, “Retrieval-augmented generation for ai-generated content: A survey,” arXiv preprint arXiv:2402.19473 , 2024

    2024 arXiv

  68. [76]

    Survey of code search based on deep learning,

    Y . Xie, J. Lin, H. Dong, L. Zhang, and Z. Wu, “Survey of code search based on deep learning,” ACM Transactions on Software Engineering and Methodology, vol. 33, no. 2, pp. 1–42, 2023

    2023

  69. [77]

    A survey on retrieval- augmented text generation,

    H. Li, Y . Su, D. Cai, Y . Wang, and L. Liu, “A survey on retrieval- augmented text generation,” arXiv preprint arXiv:2202.01110 , 2022

    2022 arXiv

  70. [78]

    Deep learning-based sequential recommender systems: Concepts, algorithms, and evaluations,

    H. Fang, G. Guo, D. Zhang, and Y . Shu, “Deep learning-based sequential recommender systems: Concepts, algorithms, and evaluations,” in Web Engineering: 19th International Conference, ICWE 2019, Daejeon, South Korea, June 11–14, 2019, Proceedings 19 . Springer, 2019, pp. 574–577

    2019

  71. [79]

    [Online]

    DeepMind, “Gemini,” online; accessed 15 Jan 2025. [Online]. Available: https://deepmind.com/technologies/gemini/flash/

    2025

  72. [80]

    Counterfactual data augmentation via perspective transition for open-domain dialogues,

    J. Ou, J. Zhang, Y . Feng, and J. Zhou, “Counterfactual data augmentation via perspective transition for open-domain dialogues,” arXiv preprint arXiv:2210.16838, 2022

    2022 arXiv

  73. [81]

    [Online]

    Anthropic, “Claude,” 2025, online; accessed 15 Jan 2025. [Online]. Available: https://www.anthropic.com/news/claude-3-5-sonnet

    2025

  74. [82]

    Unveiling llm evaluation focused on metrics: Challenges and solutions,

    T. Hu and X.-H. Zhou, “Unveiling llm evaluation focused on metrics: Challenges and solutions,” arXiv preprint arXiv:2404.09135 , 2024

    2024 arXiv

  75. [83]

    Gpt-4o system card,

    A. Hurst, A. Lerer, A. P. Goucher, A. Perelman, A. Ramesh, A. Clark, A. Ostrow, A. Welihinda, A. Hayes, A. Radford et al., “Gpt-4o system card,” arXiv preprint arXiv:2410.21276 , 2024

    2024 arXiv

  76. [84]

    Combining similarity features and deep representation learning for stance detection in the context of checking fake news,

    L. Borges, B. Martins, and P. Calado, “Combining similarity features and deep representation learning for stance detection in the context of checking fake news,” Journal of Data and Information Quality (JDIQ) , vol. 11, no. 3, pp. 1–26, 2019

    2019

  77. [85]

    Systematic evaluation of llm-as-a-judge in llm alignment tasks: Explainable metrics and diverse prompt templates,

    H. Wei, S. He, T. Xia, F. Liu, A. Wong, J. Lin, and M. Han, “Systematic evaluation of llm-as-a-judge in llm alignment tasks: Explainable metrics and diverse prompt templates,” arXiv preprint arXiv:2408.13006, 2024

    2024 arXiv

  78. [86]

    Open llms are necessary for current private adaptations and outperform their closed alternatives,

    V . Hanke, T. Blanchard, F. Boenisch, I. Olatunji, M. Backes, and A. Dziedzic, “Open llms are necessary for current private adaptations and outperform their closed alternatives,” Advances in Neural Informa- tion Processing Systems , vol. 37, pp. 1220–1250, 2024

    2024

  79. [87]

    Ragas: Automated evaluation of retrieval augmented generation,

    S. Es, J. James, L. E. Anke, and S. Schockaert, “Ragas: Automated evaluation of retrieval augmented generation,” inProceedings of the 18th Conference of the European Chapter of the Association for Computa- tional Linguistics: System Demonstrations , 2024, pp. 150–158

    2024

  80. [89]

    Social desirability bias,

    P. Grimm, “Social desirability bias,” Wiley international encyclopedia of marketing, 2010

    2010

  81. [2025]

    Available: https://arxiv.org/abs/2502.18694

    [Online]. Available: https://arxiv.org/abs/2502.18694

    arXiv

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.