Pith. sign in

REVIEW 3 major objections 6 minor 76 references

Requirements-Augmented Generation for Trustworthy Acceptance Testing of LLM-Based Software

T0 review · 3 major / 6 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Acceptance testing of LLM-based software can be made trustworthy by generating oracles from requirements and personas and calibrating a cascade of LLM judges, reaching 98.8% accuracy and 31.7% lower cost in an industrial case.

desk verdict Worth reading and worth sending out, but the 31.7% cost-efficiency claim is refuted by the paper's own cascade algorithm. read the letter →

arxiv 2608.12970 v1 pith:DSL3EUYJ submitted 2026-08-13 cs.SE

Fanyu Wang , Chetan Arora , Zhenping Xie , Yonghui Liu , Kla Tantithamthavorn , Aldeida Aleti , Siwei Jiang This is my paper · ORCID
classification cs.SE
keywords LLM-basedsoftwareAcceptancetestingTestoracleRequirementsengineeringRetrieval-augmentedgenerationConformalriskcontrolLLM-as-a-judgeCascadejudgment
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

LLM-based software is intent-driven: the same query needs different answers for a diabetic user, an athlete, or a pregnant user, so a fixed expected answer cannot serve as an acceptance-test oracle. This paper claims that the missing oracle can be reconstructed from the software's own requirements, personas, and domain knowledge, and that the reliability of the resulting verdicts can be calibrated statistically. It introduces REAG, which retrieves these software artifacts and reasons over them to produce executable pass/fail criteria, and a confidence-calibrated cascade of LLM judges that accepts, escalates, or abstains based on simulated expert agreement, with thresholds set by conformal risk control. In an industrial case study on a production nutrition advisory application with 346 test scenarios and 46 user profiles, the framework produced oracles rated 3.91/5 on average, reaching qualified or marginal quality in 82% of cases, and the cascade achieved 98.8% verdict accuracy while improving retained oracle quality to 4.30 and cutting cost by 31.7% versus the strongest single judge. The stakes are practical: if this holds, stochastic, personalised LLM systems can be acceptance-tested automatically rather than by human inspection alone.

What carries the argument

The pipeline's load-bearing components are REAG and the cascade judge. REAG uses ICRALM scoring, the average log-probability of the user intention given a candidate document, to rank software artifacts; an adaptive top-k rule that cuts off at the largest similarity gap; and self-reasoning steps that check relevance, cite evidence, and synthesise an interpreted intention. The interpreted intention is then rendered into an oracle with a TestObjective, an ExpectedResultsDescription, and explicit PassFailCriteria. The cascade is a tiered set of LLM judges from different model families whose confidence is estimated by treating N simulated annotators, each conditioned on K in-context examples, and averaging their label probabilities. Calibrated thresholds are chosen by fixed-sequence testing with a binomial upper confidence bound on disagreement risk, and if even the strongest judge falls below threshold the criterion is abstained and sent to human review. This combination turns an uncalibrated 'does the response look right' judgment into a decision rule with a stated risk bound on accepted verdicts.

What would settle it

Collect a fresh set of real, non-synthetic user queries from the same application, run the cascade with the paper's calibrated thresholds, and compare its accepted verdicts against expert labels; if the observed disagreement rate exceeds the calibrated alpha, for instance 0.14, by more than sampling error, the exchangeability assumption fails and the finite-sample guarantee does not hold outside the calibration distribution.

Watch

Extended reading notes

Core claim

The central claim is that acceptance testing for LLM-based software should be reframed as requirements-grounded intent reconstruction rather than output comparison. The paper argues that user persona is a first-class input: a correct response is defined by who is asking and under what constraints, so any oracle that ignores the asker is underspecified. REAG operationalises this by ranking and retrieving requirements, domain knowledge, and persona descriptions, dynamically choosing how many artifacts to retrieve, and generating an oracle as a structured set of pass/fail criteria. The cascade then treats the verdict as a statistical decision: each judge estimates confidence through simulated expert agreement, low-confidence verdicts escalate to a stronger judge, and if no judge is confident enough the system abstains rather than emitting an unreliable verdict. Conformal risk control calibrates the confidence thresholds so that, on the calibration distribution, the disagreement rate of accepted verdicts stays below a chosen risk level with finite-sample guarantees; the paper reports that empirical human agreement met or exceeded the target across all tested risk settings.

Load-bearing premise

The reliability guarantee assumes that the 246 calibration examples, including user profiles manufactured by a simulated user agent, look like the queries and users the system will actually face in production; if the synthetic profiles do not match real usage, the calibrated thresholds will not keep the verdict error rate at the promised level.

Editorial extensions

If this is right

  • Acceptance testing moves from the service layer to the semantic layer: tests can fail because a response is wrong for this user's health context, not just because an API misbehaved.
  • The abstention mechanism doubles as a quality filter: at alpha = 0.14 the cascade keeps 86% of criteria for automated verdicts and raises the average quality of the remaining oracles from 3.91 to 4.30.
  • Reliability guarantees are local: any new deployment must re-run calibration on its own expert-labelled set and obtain its own thresholds; the paper's alpha = 0.14 operating point is not portable.
  • Cost savings follow from price-capability tiering: routing easy cases to a cheap judge and only escalating hard ones beats always paying for the strongest model, here by 31.7% at equal or better accuracy.
  • Retrieval precision, not generation quality, sets the ceiling on oracle correctness; improving the organisation of requirements artifacts should directly raise oracle quality.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If retrieval precision bounds oracle quality as the paper argues, then corpus quality, such as partitioning technical from user-facing requirements, becomes a testable upstream intervention; reorganising artifacts could be measured directly against the 18% unqualified-oracle rate.
  • Because the calibration profiles are partly synthetic, an obvious next experiment is to calibrate only on real production queries and compare accepted-verdict disagreement rates with the reported 98.8%.
  • The cascade's abstain-on-doubt design generalises beyond acceptance testing to any LLM output validation task where a small expert-labelled calibration set is affordable, such as content safety or medical triage checks.
  • The reported 82% qualified-oracle rate suggests that current RAG retrieval will be the limiting factor when this framework is ported to domains with sparser requirements documentation, so proxy metrics for artifact retrievability could predict portability.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper presents an automated acceptance-testing framework for LLM-based software (LBS), with two main contributions. First, Requirements-Augmented Generation (REAG) interprets user intentions by retrieving software requirements, personas, and domain knowledge via adaptive RAG and self-reasoning, producing executable test oracles. Second, a confidence-calibrated cascade of LLM judges uses simulated expert agreement and conformal risk control to accept, escalate, or abstain on verdicts, with the abstention mechanism acting as a quality gate. The evaluation is an industrial case study on a nutrition advisory application with 346 scenarios, a 246-item calibration set, and a 100-item evaluation set. The paper reports that REAG achieves 3.91/5 oracle quality, that the cascade reaches 98.8% selective agreement at alpha=0.14 with 14% abstention, and that it offers a 31.7% cost-efficiency improvement over the strongest single judge.

Significance. If the reported results held, the paper would make a timely and useful contribution: it applies conformal risk control to acceptance testing for LLM-based software, combines oracle generation with reliability-aware judging, and provides an industrial case study. The empirical design is stronger than much of the LLM-as-judge literature: calibration and evaluation sets are separated, calibration labels were frozen before evaluation, expert agreement is reported (Fleiss' kappa 0.76), failure analysis is explicit, and the replication package is public. The framework's core idea is credible and the paper's framing of limitations is unusually candid. However, the cost-efficiency claim in RQ3 is inconsistent with the cascade algorithm described in Section 3.4.1, and the headline selective-agreement number is based on a small accepted set with a mix of selective and full-set metrics. These issues affect the central viability claim and require correction.

major comments (3)
  1. [Section 3.4.1, Table 3, Section 4.7 (RQ3 Answer)] The cost model in Table 3 charges only the final deciding tier. Under the cascade algorithm in Section 3.4.1, every item is first processed by J1, and every item escalated to a higher tier also incurs the cost of all lower tiers. At alpha=0.14, the judge distribution is 0/72.1/27.9, so the actual cost per 100 units is 100*1 + 100*3 + 27.9*4.6 = 528.3, not 344.6. The corrected CPP is 528.3/98.8, approximately 5.35, which is worse than the J3-alone baseline (460/90, approximately 5.11). At alpha=0.16, the corrected cost is 100 + 300 + 17.2*4.6 = 479.1, giving CPP approximately 5.27, again above the J3 baseline. Therefore the claimed 31.7% cost-efficiency advantage over Gemini-2.5-Flash, and the statement that the cascade is more cost-efficient than the J2 baseline, are not supported by the paper's own algorithm. RQ3, the abstract, and the conclusion must be recomputed, or the table must explicitly state that it reports deciding-tier cost only, with a clear justification for why that is the relevant cost metric.
  2. [Section 4.6, Figure 4] The headline 98.8% selective agreement at alpha=0.14 is computed on the accepted subset, which is 86% of the 100 evaluation items, i.e., approximately 86 verdicts, and it corresponds to a single disagreement. This number is reported as a precise percentage and compared directly with full-set accuracies of single judges (90% and 83%), mixing selective agreement with unconditional accuracy. Please report the number of accepted items in the denominator, provide a binomial confidence interval for the selective agreement, and compare all systems on the same basis, either by applying the same acceptance rule to the baselines or by reporting both selective and full-set metrics for every system.
  3. [Section 4.6, Figure 5, Abstract, Conclusion] The statement that the cascade 'improves average oracle quality from 3.91 to 4.30' describes a property of the filtered retained set, not a causal improvement to oracle generation. Because the filter is correlated with the rated quality, the retained-set mean is higher by construction. The paper should state this as the selective quality of the accepted verdicts rather than as an 'improvement' in oracle quality. As written, the abstract and conclusion overstate the result.
minor comments (6)
  1. [Section 4.2.2] Reporting a single Fleiss' kappa of 0.76 'across D3, D4, and D5' is ambiguous because those data categories involve different annotation tasks (Likert-style ratings, binary verdicts, and interviews); please report per-task kappa values.
  2. [Figure 4] The notation '88* is the last observable point' is unexplained; please define the asterisk and state why only 88 of 100 evaluation points are observable at that setting.
  3. [Section 3.4, Eq. (6)] The definition of the upper confidence bound in Eq. (6) is not transparent; specify the Clopper-Pearson construction, the confidence level, and the direction of the supremum so that the formula can be verified independently.
  4. [Section 3.4.1] The symbol K is used for the number of in-context examples in the prompt and also appears in the adaptive top-k retrieval discussion; rename one of these to avoid confusion.
  5. [References] Reference [54] has a DOI that currently resolves to a preprint server rather than the cited ACM TOSEM version, and reference [60] contains duplicated author names; both should be corrected.
  6. [Section 4.5 (RQ1)] The 82% qualified/marginal rate uses a per-dimension flag rule, while the composite 3.91/5 average uses a different threshold; the text explains this, but a small table with both definitions would make it easier to verify.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: cascade thresholds are calibrated on a separate 246-item set and evaluated on 100 held-out items, so the headline claims are empirical, not definitional.

full rationale

Derivation chain check: REAG oracle quality (3.91/5) is measured by external expert Likert ratings; cascade thresholds are fit on D_cal (246 expert-annotated oracle-output pairs) using conformal risk control (Eqs. 5-7), and RQ2 accuracy and selective agreement are computed on a separate 100-item evaluation set. The paper explicitly reports that all calibration labels were frozen before any evaluation-set item was scored or any threshold was fit (Section 4.2.2), so the 98.8% selective agreement is a held-out empirical result, not an identity. The 3.91-to-4.30 quality improvement is a disclosed selection effect of the abstention gate (Figure 5, Section 4.6); it is an observed consequence of filtering, not a quantity predicted from the calibration equations. Self-citations [6, 53, 54] support evaluation-dimension choices and background only; the load-bearing machinery cites external work ([4], [27], [43], [49], [60]) and is not author-specific. No uniqueness theorem is imported from the authors, and no central claim reduces to its own input by definition. Two non-circular caveats should be tracked as correctness and validity risks rather than circularity: (i) Table 3's cascade Total Cost omits the mandatory J1 and J2 calls described in Section 3.4.1, which would alter the CPP numbers and the 31.7% cost-efficiency claim; and (ii) Section 5.1 itself flags that synthetic persona generation may stress the exchangeability assumption behind Eq. 1.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

The framework rests on standard statistical calibration plus domain assumptions about LBS behavior and oracle measurement. No new physical or conceptual entities are introduced; REAG and the cascade are compositions of existing techniques. The main fitted quantities are the confidence thresholds and the post-hoc temperature scaling, which are honestly disclosed.

free parameters (4)
  • Confidence thresholds lambda_i per judge tier = Not reported; calibrated on D_cal via conformal risk control
    Eq. 5-7 in Section 3.4.2 fit a threshold for each judge to keep expected disagreement below alpha on calibration data.
  • Risk tolerance alpha = 0.14 (operating point)
    Chosen after inspecting coverage-accuracy curves in Section 4.6; the authors present alpha=0.14 and 0.16 as sweet spots, but this is a post hoc operating point selection.
  • Selective temperature scaling T_i = T_i > 1, chosen so each judge handles at least tau samples
    Section 3.4.2 applies temperature scaling post-hoc to judges with insufficient calibration samples, then re-calibrates thresholds; this is a fitting procedure that relaxes formal guarantees.
  • Simulated annotator count N and in-context examples K = N=5, K=3
    Section 3.4.1 sets these by hand for confidence estimation; they are not swept or justified by sensitivity analysis.
assumptions (5)
  • standard math Exchangeability of calibration and deployment data for conformal risk control (Eq. 1)
    The conformal guarantee in Section 2.3 and Eq. 1 requires that calibration and deployment data share a distribution; the paper applies this to its cascade.
  • domain assumption Personas are first-class oracle inputs and all queries must be associated with a persona
    Section 3.1 asserts that persona is a first-class input for any LBS, which is a domain assumption about how correct behavior is defined.
  • domain assumption Judges from distinct model families have sufficiently independent errors
    Section 2.3 and 3.4 select Gemini and GPT judges hoping to reduce correlated errors; this is an empirical assumption not formally proven.
  • domain assumption The four Likert dimensions (relevance, coverage, correctness, coherence) measure oracle quality
    Section 4.3.1 defines oracle quality through these dimensions; the validity of this measurement is assumed and acknowledged as perception-based in Section 5.1.
  • ad hoc to paper Qualified or marginal oracle quality means >=3/5 on Likert, with exactly 3 treated as marginal
    Section 4.3.1 and RQ1 results set the threshold for 'qualified or marginal' at >=3/5; this classification boundary is chosen by the authors and drives the 82% figure.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Requirements-Augmented Generation for Trustworthy Acceptance Testing of LLM-Based Software." pith.science (2026). https://pith.science/paper/DSL3EUYJ

@misc{pith2026260812970,
  author       = {Pith},
  title        = {Pith review of: Requirements-Augmented Generation for Trustworthy Acceptance Testing of LLM-Based Software},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/DSL3EUYJ}},
  note         = {Machine review of arXiv:2608.12970}
}
read the original abstract

LLM-based software (LBS) integrates large language models as core components to deliver flexible, personalised responses. Unlike traditional software with deterministic outputs, LBSs exhibit context-dependent, stochastic behaviour that renders classical acceptance testing and test oracles insufficient: the same query may require fundamentally different responses depending on user personas and software context. This gap creates an urgent need for automated acceptance testing frameworks that autonomously interpret user instructions, while reliably inferring user intentions in a changing environment. In this paper, we present an automated acceptance testing framework for LBS with calibrated verdict reliability via two technical contributions. First, we introduce Requirements-Augmented Generation (REAG), which interprets user intentions by retrieving relevant software requirements, domain knowledge, and personas via adaptive RAG and self-reasoning to generate context-aware test oracles. Second, recognising that oracle generation may retrieve irrelevant constraints, misinterpret intent, or hallucinate requirements, we introduce a confidence-calibrated cascade judgment. This method quantifies verdict reliability via simulated expert agreement -- accepting high-confidence verdicts, escalating ambiguous cases, or abstaining when uncertain -- with empirical reliability guarantees backed by conformal risk control. An industrial case study on a production nutrition advisory application demonstrates that REAG achieves a 3.91/5 oracle quality score, reaching qualified or marginal oracle quality in 82% of cases. The confidence-calibrated cascade achieves 98.8% accuracy, improves oracle quality from 3.91 to 4.30 by filtering unqualified outputs, and delivers a 31.7% cost-efficiency improvement over single-judge baselines, validating industrial viability

Figures

Figures reproduced from arXiv: 2608.12970 by the authors.

Figure 1
Figure 1. Acceptance Testing for an LBS (Our case study). [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Overview of Acceptance Testing Framework [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. RQ1 - Frequency of Likert Scale ratings for each evaluation criterion by experts. [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: RQ2 - Selection Coverage (R) & Selective Agreements (L) by [PITH_FULL_IMAGE:figures/full_fig_p010_4.png]
Figure 5
Figure 5. Figure 5: Sample Elimination Trend by 𝛼 Settings (Default 𝛿 = 0.1) assign high confidence to incorrect verdicts [50], and properly calibrated thresholds prevent this from affecting the accepted set. RQ2 Answer Our cascade achieves optimal balance at 𝛼 = 0.14 with 98.8% accuracy …

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

76 extracted references · 28 canonical work pages

  1. [1]

    Replication Package: Requirements-Augmented Genera- tion for Trustworthy Acceptance Testing of LLM-Based Software

    2026. Replication Package: Requirements-Augmented Genera- tion for Trustworthy Acceptance Testing of LLM-Based Software. https://github.com/fanyuuwang/Requirements-Augmented-Generation- for-Trustworthy-Acceptance-Testing-of-LLM-based-Software. Accessed: 2026-07-30

    work page 2026

  2. [2]

    Raiyaan Abdullah, Yogesh Singh Rawat, and Shruti Vyas. 2025. iSafetyBench: A video-language benchmark for safety in industrial environment. InProceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) Workshops. 1444–1453. doi:10.1109/iccvw69036.2025.00154

    arXiv 2025

  3. [3]

    Sandhini Agarwal, Lama Ahmad, Jason Ai, Sam Altman, Andy Applebaum, Edwin Arbus, Rahul K Arora, Yu Bai, Bowen Baker, Haiming Bao, et al. 2025. gpt-oss- 120b & gpt-oss-20b model card.arXiv preprint arXiv:2508.10925(2025). doi:10. 48550/arXiv.2508.10925

    arXiv doi:10.48550/arxiv.2508.10925 2025

  4. [4]

    Anastasios N Angelopoulos, Stephen Bates, Adam Fisch, Lihua Lei, and Tal Schuster. 2022. Conformal risk control.arXiv preprint arXiv:2208.02814(2022). doi:10.48550/arXiv.2208.02814

    arXiv doi:10.48550/arxiv.2208.02814 2022

  5. [5]

    Antonelli, Mariángeles Hozikian, G

    L. Antonelli, Mariángeles Hozikian, G. Camilleri, Alejandro Fernández, J. Grigera, Diego Torres, and P. Zaraté. 2019. Wiki support for automated definition of software test cases.Kybernetes49, 4 (2019), 1305–1324. doi:10.1108/k-10-2018- 0548

    work page doi:10.1108/k-10-2018- 2019

  6. [6]

    Chetan Arora, Tomas Herda, and Verena Homm. 2024. Generating test scenarios from NL requirements using retrieval-augmented LLMs: An industrial study. In 2024 IEEE 32nd International Requirements Engineering Conference (RE). IEEE, 240–251. doi:10.1109/re59067.2024.00031

    arXiv 2024

  7. [7]

    Sher Badshah, Ali Emami, and Hassan Sajjad. 2026. SCOPE: Selective Conformal Optimized Pairwise LLM Judging.arXiv preprint arXiv:2602.13110(2026). doi:10. 48550/arXiv.2602.13110

    arXiv doi:10.48550/arxiv.2602.13110 2026

  8. [8]

    Yejin Bang, Ziwei Ji, Alan Schelten, Anthony Hartshorn, Tara Fowler, Cheng Zhang, Nicola Cancedda, and Pascale Fung. 2025. Hallulens: Llm hallucination benchmark. InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 24128–24156. doi:10.18653/ v1/2025.acl-long.1176

    work page 2025

Show all 76 references
  1. [9]

    Rina Foygel Barber, Emmanuel J Candes, Aaditya Ramdas, and Ryan J Tibshirani

  2. [10]

    Earl T Barr, Mark Harman, Phil McMinn, Muzammil Shahbaz, and Shin Yoo. 2015. The oracle problem in software testing: A survey.IEEE transactions on software engineering41, 5 (2015), 507–525. doi:10.1109/tse.2014.2372785

    2015

  3. [11]

    Peter Bauer. 1991. Multiple testing in clinical trials.Statistics in medicine10, 6 (1991), 871–890. doi:10.1002/sim.4780100609

    1991 doi

  4. [12]

    Bjarnason, M

    E. Bjarnason, M. Unterkalmsteiner, Markus Borg, and Emelie Engström. 2016. A multi-case study of agile requirements engineering and the use of test cases as requirements.Information and Software Technology77 (2016), 61–79. doi:10.1016/ j.infsof.2016.03.008

    2016

  5. [13]

    Weyssow, Junda He, Yu zeng Lyu, and David Lo

    Alessio Bucaioni, M. Weyssow, Junda He, Yu zeng Lyu, and David Lo. 2025. A Functional Software Reference Architecture for LLM-Integrated Systems.2025 IEEE 22nd International Conference on Software Architecture Companion (ICSA-C) (2025), 1–5. doi:10.1109/icsa-c65153.2025.00006

    2025

  6. [14]

    Shihao Cai, Chongming Gao, Yang Zhang, Wentao Shi, Jizhi Zhang, Keqin Bao, Qifan Wang, and Fuli Feng. 2025. K-order Ranking Preference Optimization for Large Language Models.Findings of the Association for Computational Linguistics: ACL 2025(2025), 4844–4859. doi:10.18653/v1/2...

    2025 doi

  7. [15]

    Zhou, Yanjie Zhao, Haoyu Wang, Shuai Wang, Xiao Chen, Tégawendé F

    Daihang Chen, Yonghui Liu, M. Zhou, Yanjie Zhao, Haoyu Wang, Shuai Wang, Xiao Chen, Tégawendé F. Bissyandé, Jacques Klein, and Li Li. 2025. LLM for Mobile: An Initial Roadmap.ACM Transactions on Software Engineering and Methodology34, 5 (2025), 1–29. doi:10.1145/3708528

    2025 doi

  8. [16]

    Mike Cohn. 2024. Is It Dangerous to Calculate the Cost per Point? https://www.mountaingoatsoftware.com/blog/is-it-dangerous-to-calculate- the-cost-per-point. Last updated June 24, 2024

    2024

  9. [17]

    Jean-Pierre Corriveau and Wei Shi. 2013. Traceability in acceptance testing. Journal of Software Engineering and Applications6, 10 (2013), 36–46. doi:10.4236/ jsea.2013.610a005

    2013

  10. [18]

    Lívia Fernanda de Oliveira, Cássio Leonardo Rodrigues, and Renato de Freitas Bulcão-Neto. 2023. Characterizing the software acceptance testing and the inclusion of people with disabilities by means of a systematic mapping.IEEE Latin America Transactions21, 1 (2023), 35–46. doi...

    2023

  11. [19]

    dos Santos, Sylvain Hallé, Fábio Petrillo, and Yann-Gaël Guéhéneuc

    Marcela G. dos Santos, Sylvain Hallé, Fábio Petrillo, and Yann-Gaël Guéhéneuc

  12. [20]

    Yann Dubois, Chen Xuechen Li, Rohan Taori, Tianyi Zhang, Ishaan Gulrajani, Jimmy Ba, Carlos Guestrin, Percy S Liang, and Tatsunori B Hashimoto. 2023. Alpacafarm: A simulation framework for methods that learn from human feed- back.Advances in Neural Information Processing Syste...

    2023 doi

  13. [21]

    Sebastian Farquhar, Jannik Kossen, Lorenz Kuhn, and Yarin Gal. 2024. Detecting hallucinations in large language models using semantic entropy.Nature630, 8017 (2024), 625–630. doi:10.1038/s41586-024-07421-0

    2024 doi

  14. [22]

    Margarida Ferreira, Luis Viegas, João Pascoal Faria, and B. Lima. 2025. Acceptance Test Generation with Large Language Models: An Industrial Case Study.2025 IEEE/ACM International Conference on Automation of Software Test (AST)(2025), 1–11. doi:10.1109/ast66626.2025.00007

    2025

  15. [23]

    2020.The future of software quality assurance

    Stephan Goericke (Ed.). 2020.The future of software quality assurance. Springer Nature. doi:10.1007/978-3-030-29509-7

    2020 doi

  16. [24]

    Shan Jiang, Chenguang Zhu, and Sarfraz Khurshid. 2024. Generating executable oracles to check conformance of client code to requirements of JDK Javadocs using LLMs.ArXivabs/2411.01789 (2024). doi:10.48550/arXiv.2411.01789

    2024 doi arXiv

  17. [25]

    Carlos E Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. 2023. Swe-bench: Can language models resolve real-world github issues?arXiv preprint arXiv:2310.06770(2023). doi:10.48550/ arXiv.2310.06770

    2023 doi arXiv

  18. [26]

    Ankur Joshi, Saket Kale, Satish Chandel, and D Kumar Pal. 2015. Likert scale: Explored and explained.British journal of applied science & technology7, 4 (2015), 396–403. doi:10.9734/bjast/2015/14975

    2015 doi

  19. [27]

    Jaehun Jung, Faeze Brahman, and Yejin Choi. 2024. Trust or Escalate: LLM Judges with Provable Guarantees for Human Agreement.arXiv preprint arXiv:2407.18370 (2024). doi:10.48550/arXiv.2407.18370

    2024 doi arXiv

  20. [28]

    Shanthi Karpurapu, Sravanthy Myneni, Unnati Nettur, Likhit Sagar Gajja, Dave Burke, Tom Stiehm, and Jeffery Payne. 2024. Comprehensive Evaluation and Insights Into the Use of Large Language Models in the Automation of Behavior- Driven Development Acceptance Test Formulation.IE...

    2024

  21. [29]

    Elliot Kim, Avi Garg, Kenny Peng, and Nikhil Garg. 2025. Correlated errors in large language models.arXiv preprint arXiv:2506.07962(2025). doi:10.48550/arXiv. 2506.07962

    2025 doi arXiv

  22. [30]

    Hannah Kim, Kushan Mitra, Rafael Li Chen, Sajjadur Rahman, and Dan Zhang

  23. [31]

    Seungone Kim, Juyoung Suk, Shayne Longpre, Bill Yuchen Lin, Jamin Shin, Sean Welleck, Graham Neubig, Moontae Lee, Kyungjae Lee, and Minjoon Seo. 2024. Prometheus 2: An open source language model specialized in evaluating other language models.arXiv preprint arXiv:2405.01535(20...

    2024 doi arXiv

  24. [32]

    Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, M

    Patrick Lewis, Ethan Perez, Aleksandra Piktus, F. Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, M. Lewis, Wen tau Yih, Tim Rocktäschel, Se- bastian Riedel, and Douwe Kiela. 2020. Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks.Neural Information ...

    2020 doi arXiv

  25. [33]

    doi:10.48550/arXiv.2402.18050

    Meganno+: A human-llm collaborative annotation system.arXiv preprint arXiv:2402.18050(2024). doi:10.48550/arXiv.2402.18050

    2024 doi arXiv

  26. [34]

    Feiyang Li, Peng Fang, Zhan Shi, Arijit Khan, Fang Wang, Dan Feng, Weihao Wang, Xin Zhang, and Yongjian Cui. 2025. Cot-rag: Integrating chain of thought and retrieval-augmented generation to enhance reasoning in large language models.arXiv preprint arXiv:2504.13534(2025), 22. ...

    2025 doi arXiv

  27. [35]

    Haoxuan Li, Wei Song, Aofan Liu, and Peiwu Qin. 2025. AdaDocVQA: Adaptive Framework for Long Document Visual Question Answering in Low-Resource Settings.Proceedings of the 33rd ACM International Conference on Multimedia (2025), 13777–13783. doi:10.1145/3746027.3761996

    2025

  28. [36]

    Dawei Li, Bohan Jiang, Liangjie Huang, Alimohammad Beigi, Chengshuai Zhao, Zhen Tan, Amrita Bhattacharjee, Yuxuan Jiang, Canyu Chen, Tianhao Wu, et al

  29. [37]

    doi:10.48550/arXiv.2411.16594

    From generation to judgment: Opportunities and challenges of llm-as-a- judge.arXiv preprint arXiv:2411.16594(2024). doi:10.48550/arXiv.2411.16594

    2024 doi

  30. [38]

    Daniel Maciel, A. C. Paiva, and A. Silva. 2019. From Requirements to Automated Acceptance Tests of Interactive Apps: An Integrated Model-based Testing Ap- proach.International Conference on Evaluation of Novel Approaches to Software Engineering(2019), 265–272. doi:10.5220/0007...

    2019 doi

  31. [39]

    Muhammad Fraz Malik, MNA Khan, Uzma Bibi, and Muhammad Ayaz Malik

  32. [40]

    Xinyi Li, Sai Wang, Siqi Zeng, Yu Wu, and Yi Yang. 2024. A survey on LLM-based multi-agent systems: workflow, infrastructure, and challenges.Vicinagearth1, 1 (2024), 9. doi:10.1007/s44336-024-00009-2

    2024 doi

  33. [41]

    Gribova, V

    Zhipeng Li, Hailong Su, Yong Wu, Qinhu Zhang, Chang-An Yuan, V. Gribova, V. Filaretov, and De-Shuang Huang. 2024. Hierarchical Multiview Top-k Pooling With Deep-Q-Networks.IEEE Transactions on Artificial Intelligence5, 6 (2024), 2985–2996. doi:10.1109/tai.2023.3334261

    2024

  34. [42]

    I. K. Raharjana, Fadel Harris, and Army Justitia. 2020. Tool for Generating Behavior-Driven Development Test-Cases.Journal of Information Systems Engi- neering and Business Intelligence6, 1 (2020), 27–36. doi:10.20473/jisebi.6.1.27-36

    2020 doi

  35. [43]

    Shashua, Kevin Leyton- Brown, and Y

    Ori Ram, Yoav Levine, Itay Dalmedigos, Dor Muhlgay, A. Shashua, Kevin Leyton- Brown, and Y. Shoham. 2023. In-Context Retrieval-Augmented Language Models. Transactions of the Association for Computational Linguistics11 (2023), 1316–1331. doi:10.1162/tacl_a_00605

    2023 doi

  36. [44]

    Nils Reimers and Iryna Gurevych. 2019. Sentence-bert: Sentence embeddings using siamese bert-networks.arXiv preprint arXiv:1908.10084(2019). doi:10. 48550/arXiv.1908.10084

    2019 doi arXiv

  37. [45]

    Jianing Qiu, Kyle Lam, Guohao Li, Amish Acharya, Tien Yin Wong, Ara Darzi, Wu Yuan, and Eric J Topol. 2024. LLM-based agentic systems in medicine and healthcare.Nature Machine Intelligence6, 12 (2024), 1418–1420. doi:10.1038/ s42256-024-00944-1

    2024

  38. [46]

    Victor Quach, Adam Fisch, Tal Schuster, Adam Yala, Jae Ho Sohn, Tommi S Jaakkola, and Regina Barzilay. 2023. Conformal language modeling.arXiv preprint arXiv:2306.10193(2023). doi:10.48550/arXiv.2306.10193

    2023 doi arXiv

  39. [47]

    Jake C Snell and Thomas L Griffiths. 2025. Conformal prediction as bayesian quadrature.arXiv preprint arXiv:2502.13228(2025). doi:10.48550/arXiv.2502.13228

    2025 doi arXiv

  40. [48]

    Tomasz Straszak and M. Smialek. 2015. Model-driven acceptance test automation based on use cases.Computer Science and Information Systems12, 2 (2015), 707–728. doi:10.2298/csis141217033s

    2015 doi

  41. [49]

    Chihiro Taguchi, Seiji Maekawa, and Nikita Bhutani. 2025. Efficient Context Selec- tion for Long-Context QA: No Tuning, No Iteration, Just Adaptive-k.Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing(2025), 20116–20141. doi:10.18653/v1/2025...

    2025 doi

  42. [50]

    2012.Case study research in software engineering: Guidelines and examples

    Per Runeson, Martin Host, Austen Rainer, and Bjorn Regnell. 2012.Case study research in software engineering: Guidelines and examples. John Wiley & Sons. doi:10.1002/9781118181034

    2012 doi

  43. [51]

    Chaudhary, Kalpa Gunaratna, Srinivasan Parthasarathy, and Manas Gaur

    Yash Saxena, Ankur Padia, Mandar S. Chaudhary, Kalpa Gunaratna, Srinivasan Parthasarathy, and Manas Gaur. 2025. Ranking Free RAG: Replacing Re-ranking with Selection in RAG for Sensitive Domains.arXiv preprint arXiv:2505.16014 (2025). doi:10.48550/arXiv.2505.16014

    2025 doi arXiv

  44. [52]

    June M Verner, Jennifer Sampson, Vladimir Tosic, NA Abu Bakar, and Barbara A Kitchenham. 2009. Guidelines for industrially-based multiple case studies in soft- ware engineering. In2009 Third International Conference on Research Challenges in Information Science. IEEE, 313–324....

    2009

  45. [53]

    Fanyu Wang, Chetan Arora, Yonghui Liu, Kaicheng Huang, Chakkrit Tan- tithamthavorn, Aldeida Aleti, Dishan Sambathkumar, and David Lo. 2025. Multi- Modal Requirements Data-based Acceptance Criteria Generation using LLMs. In 2025 40th IEEE/ACM International Conference on Automat...

    2025

  46. [54]

    Fanyu Wang, Chetan Arora, Chakkrit Tantithamthavorn, Kaicheng Huang, and Aldeida Aleti. 2025. Requirements-driven automated software testing: A system- atic review.ACM Transactions on Software Engineering and Methodology(2025). doi:10.20944/preprints202502.0628.v1 Preprint; DO...

    2025

  47. [55]

    Zailong Tian, Zhuoheng Han, Yanzhe Chen, Haozhe Xu, Xi Yang, Richeng Xuan, Houfeng Wang, and Lizi Liao. 2025. Overconfidence in llm-as-a-judge: Diagnosis and confidence-driven solution.arXiv preprint arXiv:2508.06225(2025). doi:10. 48550/arXiv.2508.06225

    2025 doi arXiv

  48. [56]

    Jack van Heugten Breurkes, Fabian Gilson, and M. Galster. 2022. Overlap be- tween Automated Unit and Acceptance Testing – a Systematic Literature Review. International Conference on Evaluation and Assessment in Software Engineering (2022), 80–89. doi:10.1145/3530019.3530028

    2022

  49. [57]

    Xinru Wang, Hannah Kim, Sajjadur Rahman, Kushan Mitra, and Zhengjie Miao

  50. [58]

    Claes Wohlin. 2021. Case Study Research in Software Engineering—It is a Case, and it is a Study, but is it a Case Study?Information and Software Technology133 (2021), 106514. doi:10.1016/j.infsof.2021.106514

    2021

  51. [59]

    Fangzhou Wu, Ning Zhang, Somesh Jha, Patrick McDaniel, and Chaowei Xiao

  52. [60]

    Fanyu Wang, Xiaoxi Kang, Paul Burgess, Aashish Srivastava, Chetan Arora, Adnan Trakic, Lay-Ki Soon, Md Khalid Hossain, and Lizhen Qu. 2026. LePREC: Reasoning as Classification over Structured Factors for Assessing Relevance of Legal Issues.arXiv preprint arXiv:2604.19464(2026)...

    2026 doi arXiv

  53. [61]

    Junjie Wang, Yuchao Huang, Chunyang Chen, Zhe Liu, Song Wang, and Qing Wang. 2024. Software Testing With Large Language Models: Survey, Landscape, and Vision.IEEE Transactions on Software Engineering50, 4 (2024), 911–936. doi:10.1109/tse.2024.3368208

    2024

  54. [62]

    2018.Case study research and applications(6 ed.)

    Robert K Yin. 2018.Case study research and applications(6 ed.). Sage Thousand Oaks, CA

    2018

  55. [63]

    InProceedings of the 2024 CHI Conference on Human Factors in Computing Systems

    Human-llm collaborative annotation through effective verification of llm labels. InProceedings of the 2024 CHI Conference on Human Factors in Computing Systems. 1–21. doi:10.1145/3613904.3641960

    2024

  56. [64]

    Hao Yu, Aoran Gan, Kai Zhang, Shiwei Tong, Qi Liu, and Zhaofeng Liu. 2025. Evaluation of Retrieval-Augmented Generation: A Survey.Communications in Computer and Information Science(2025), 102–120. doi:10.1007/978-981-96-1024- 2_8

    2025 doi

  57. [65]

    Jinghan Zhang, Xiting Wang, Weijieying Ren, Lu Jiang, Dongjie Wang, and Kunpeng Liu. 2025. Ratt: A thought structure for coherent and correct llm reasoning. InProceedings of the AAAI Conference on Artificial Intelligence, Vol. 39. 26733–26741. doi:10.1609/aaai.v39i25.34876

    2025 doi

  58. [66]

    doi:10.48550/arXiv.2402

    A new era in llm security: Exploring security concerns in real-world llm- based systems.arXiv preprint arXiv:2402.18649(2024). doi:10.48550/arXiv.2402. 18649

    2024 doi arXiv

  59. [67]

    Peng Xia, Peng Xia, Kangyu Zhu, Haoran Li, Haoran Li, Weijia Shi, Sheng Wang, Linjun Zhang, James Zou, and Huaxiu Yao. 2024. MMed-RAG: Versatile Multi- modal RAG System for Medical Vision Language Models.International Conference on Learning Representations(2024). doi:10.48550/...

    2024 doi arXiv

  60. [68]

    Yunpeng Xu, Mufang Ying, Wenge Guo, and Zhi Wei. 2025. Two-stage risk control with application to ranked retrieval. InProceedings of the Thirty-Fourth International Joint Conference on Artificial Intelligence(Montreal, Canada)(IJCAI ’25). 9104–9111. doi:10.24963/ijcai.2025/1012

    2025 doi

  61. [69]

    Xin Zhou, Yiwen Guo, Ruotian Ma, Tao Gui, Qi Zhang, and Xuanjing Huang. 2025. Self-consistency of the internal reward models improves self-rewarding language models.arXiv preprint arXiv:2502.08922(2025). doi:10.48550/arXiv.2502.08922 Received 2026-04-30; accepted 2026-07-01

    2025 doi arXiv

  62. [70]

    Boxi Yu, Qiuyang Mang, Qingshuo Guo, and Pinjia He. 2023. Retromorphic Testing: A New Approach to the Test Oracle Problem.ArXivabs/2310.06433 (2023). doi:10.48550/arXiv.2310.06433

    2023 doi arXiv

  63. [73]

    Quanjun Zhang, Chunrong Fang, Siqi Gu, Ye Shang, Zhenyu Chen, and Liang Xiao. 2025. Large Language Models for Unit Testing: A Systematic Literature Review.arXiv preprint arXiv:2506.15227(2025). doi:10.48550/arXiv.2506.15227

    2025 doi arXiv

  64. [74]

    Jiahui Zhao, Ziyi Meng, Stepan Gordeev, Zijie Pan, Dongjin Song, Sandro Stein- bach, and Caiwen Ding. 2024. Key Information Retrieval to Classify the Un- structured Data Content of Preferential Trade Agreements.arXiv.org(2024). doi:10.48550/arXiv.2401.12520

    2024 doi arXiv

  65. [75]

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. 2023. Judging llm-as-a-judge with mt-bench and chatbot arena.Advances in neural information processing systems36 (2023), 46595–46623. doi:10...

    2023 doi

  66. [2017]

    International Journal of Advanced Computer Science and Applications8, 8 (2017)

    Multi-Agent based Functional Testing in the Distributed Environment. International Journal of Advanced Computer Science and Applications8, 8 (2017). doi:10.14569/IJACSA.2017.080818 Requirements-Augmented Generation for Trustworthy Acceptance Testing of LLM-Based Software ASE ’...

    2017

  67. [2023]

    doi:10.1214/23-aos2276

    Conformal prediction beyond exchangeability.The Annals of Statistics51, 2 (2023), 816–845. doi:10.1214/23-aos2276

    2023 doi

  68. [2024]

    Frontiers in Robotics and AI11 (2024)

    AAT4IRS: automated acceptance testing for industrial robotic systems. Frontiers in Robotics and AI11 (2024). doi:10.3389/frobt.2024.1346580

    2024

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.