REVIEW 4 major objections 6 minor 35 references
Boundary Value Test Input Generation Using Prompt Engineering with LLMs: Fault Detection and Coverage Analysis
T0 review · 4 major / 6 minor · reviewed 2026-08-10 · deepseek-v4-flash
Pith's one-line read Prompt-engineered GPT-4o can generate boundary value test inputs that rival traditional boundary value analysis in fault detection and coverage for selected programs, while struggling with complex inputs.
desk verdict Useful empirical data on GPT-4o for test input generation, but the central boundary-vs-general comparison is not validated by the authors' own admission. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the four-prompt comparison on GPT-4o: Prompt 1 asks for boundary value test inputs, Prompt 2 for general test inputs, Prompt 3 for exactly 50 boundary value test inputs, and Prompt 4 for exactly 50 general test inputs. The evaluation harness injects six fault types into each program, executes every generated input on the original and all mutants, and uses Gcov to extract execution paths; a mutant is killed if any input drives a different path than the original. This path-difference definition is what makes the white-box boundary analysis operational, because it ties fault detection to divergence in branch behavior rather than to output comparison alone.
What would settle it
Check whether the test inputs from the boundary prompts (Prompt 1 and Prompt 3) are actually closer to the comparison-predicate boundaries than the inputs from the general prompts (Prompt 2 and Prompt 4). If, for a program such as triType with conditions like `a>0 && a<=60`, the boundary-prompted inputs are not concentrated at the boundary values (e.g., 0, 1, 60, 61), then the reported boundary-specific advantages would not be tied to boundary generation and the comparison collapses.
Extended reading notes
Core claim
The central claim is that GPT-4o, when guided by boundary-focused prompts, produces test inputs whose fault detection and coverage are comparable to or better than traditional methods in selected cases. The paper establishes this on eight small C programs, using six hand-injected fault types (off-by-one, constant replacement, relational operator replacement, arithmetic operator replacement, scalar variable replacement, logical operator replacement) and a white-box kill criterion: a mutant is killed when at least one input produces an execution path, captured by Gcov, that differs from the original program's path. In the 50-input comparison, GPT-4o's boundary-prompted sets achieved higher kill rates than the machine-learning-based MLBVA baseline in six of eight programs, and matched or exceeded random testing and Klee on several programs. The paper also reports that boundary prompts did not consistently beat general prompts once input count was fixed, and that GPT-4o did not fully analyze boundary values when generating boundary-focused inputs, which the authors interpret as evidence that LLMs are a promising supplement rather than a complete replacement for existing boundary testing techniques.
Load-bearing premise
The argument assumes that the boundary-focused prompts actually change what GPT-4o produces, meaning the resulting inputs really sit at or near the program's boundary conditions; the paper itself notes that GPT-4o did not fully analyze boundary values when generating boundary-focused inputs, and the generated inputs are never validated against the code's predicates.
Editorial extensions
If this is right
- Boundary-focused prompts on GPT-4o can produce higher mutant kill rates than an ML-based boundary analyzer in the majority of the eight studied programs, suggesting LLMs are a competitive source of boundary-directed tests.
- Increasing the number of generated test inputs does not reliably improve fault detection or coverage, so test-set quality and prompt design matter more than raw quantity.
- When the test set is fixed at 50 inputs, boundary prompts do not consistently outperform general prompts, indicating GPT-4o already incorporates some boundary-like values during general generation.
- Fault detection rate and branch coverage are positively correlated, especially for boundary-prompted inputs, supporting branch coverage as a practical proxy when evaluating boundary-focused generation.
Reading between the lines
- Extension: validate the generated inputs against the actual predicate boundaries (for example, for a condition like `a>0 && a<=60`, check that boundary sets concentrate on 0, 1, 60, 61); if Prompt 1 and Prompt 2 outputs are indistinguishable in distance to boundaries, the reported boundary advantage could be an artifact of input quantity rather than boundary targeting.
- Extension: rerun the comparison with output-based or assertion-based oracles instead of path-difference; the currently reported kill rates are upper bounds because path divergence can count coincidental branch changes, so absolute kill rates would likely drop.
- Inference: the paper's observed weakness on interdependent predicates points to a hybrid architecture—use GPT-4o to propose candidate boundary inputs and a lightweight constraint solver to filter or refine them—which could combine LLM breadth with exact boundary computation.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes prompt engineering with GPT-4o to generate boundary-value test inputs for white-box testing, and compares the generated inputs with general-prompt GPT inputs, random testing, concolic testing (Klee), and an ML-based boundary value analysis (MLBVA). For eight C programs with manually injected faults, the authors measure fault detection rate (defined by path differences) and statement/branch coverage. They report mixed findings: boundary-prompts sometimes outperform general prompts and MLBVA on kill rate, but not consistently, and increasing the number of inputs does not reliably improve effectiveness.
Significance. This is a useful empirical exploration of a practically relevant question: whether LLM prompt engineering can approximate or complement classical boundary value analysis. The study has concrete strengths: multiple subject programs, a clear fault-injection procedure, replication of prompt runs, and comparison with several established baselines. However, the central independent variable—the distinction between boundary-focused and general test inputs—is not validated, and the authors themselves concede this in Section IV.C. Because the paper's main claims rest on that distinction, the significance of the results is conditional on additional validation. If the boundary/general manipulation were shown to produce measurably different input distributions, the comparison would provide valuable evidence about LLM-based white-box BVA; as written, the conclusion is not yet supported.
major comments (4)
- [IV.C (RQ1 discussion)] The central comparison in RQ1 is not construct-valid. The authors report in Section IV.C that GPT 'did not fully analyze boundary values when generating boundary-focused test inputs' and 'still considered some boundary values when generating general test inputs.' Yet the paper never measures whether the inputs produced by Prompts 1 and 3 actually lie closer to the boundary regions of the predicates than those from Prompts 2 and 4. Without such a measurement (e.g., for each comparison predicate like a<=60, the frequency of inputs at the boundary value and its immediate neighbors), the observed differences in kill rate and coverage cannot be attributed to boundary-focused generation. Please add a quantitative validation of the manipulation, or reframe the research questions to compare 'prompt wording' rather than 'boundary value analysis'.
- [IV.C (RQ1)] RQ1 asks whether there is a 'significant difference' in fault detection and coverage between boundary and general inputs, but the paper provides no statistical analysis. Figure 2 and Tables III–V show overlapping values across the three runs, and with data from only eight programs it is unclear whether the differences are meaningful. Please add paired statistical tests (e.g., Wilcoxon signed-rank) with effect sizes and confidence intervals, or clearly state that the analysis is descriptive and avoid the word 'significant'.
- [IV.C (RQ4, Figures 3 and 6)] The correlation analysis in RQ4 is not adequately specified. It is unclear how many data points are used in each regression and whether R² values are computed across the three runs per prompt per program, or across programs. Some values in Figure 6, notably R² = -1 for plgndr under genGPT, are not possible for a standard linear regression R² and suggest a different formula or a fit artifact. Moreover, because kill rate is defined by path differences, a correlation between branch coverage and kill rate is partly tautological. Please report the regression setup, the number of points, the actual fitted lines, and significance tests, and discuss the definitional dependency between the two metrics.
- [IV.A and Table III (RQ3)] The comparison with Klee is not fair because the test set sizes differ by orders of magnitude: for bessj and expint, Klee generates only 2 and 4 tests respectively, while the GPT methods generate 50. The low kill rates for Klee on those programs (0.37 and 0.38) may reflect insufficient test count rather than lower fault-detection capability. In contrast, for plgndr Klee generates 14,221 tests. Please either control for test suite size (e.g., using effort-aware measures such as kill rate per test, or adapting Klee to produce a comparable number of tests) or restrict the comparison to configurations with comparable test counts, and explicitly discuss the effect of size on the reported differences.
minor comments (6)
- [Table V] In the row for 50BVAGPT2 under findMiddle, the branch coverage value is written as '0..97'; this appears to be a typo for '0.97'.
- [IV.C and Figure 6] The paper reports R² values that can be negative in Figure 6; if the intent is to show correlation strength, please use Pearson correlation coefficient (r) or explain the regression model and why negative R² arises.
- [IV.B] The exact prompts are summarized but not quoted verbatim in a dedicated appendix or table; for reproducibility, please include the full prompt templates used in the experiments.
- [Table II] The 'English' program is described only as 'English examination program'; a brief description of its logic and input domain would make the subject selection more transparent.
- [III.C] The definition of a killed mutant relies on execution path differences rather than output differences; this is a reasonable white-box choice but should be discussed as a threat to validity, since path changes can occur without observable failures and vice versa.
- [IV.A] The MLBVA baseline is from the authors' prior work (reference [35]); please state explicitly in the text that MLBVA is an internal baseline and describe the configuration used in this study beyond the cited paper.
Circularity Check
No significant circularity: the paper is an empirical comparison study with no derivation chain that reduces to its inputs; the only self-citation (MLBVA) is a comparison baseline, not a load-bearing premise.
full rationale
This is an empirical study. LLM-generated test inputs are obtained from prompts, mutants are manually injected, and fault detection and coverage are measured using Gcov; there is no fitted parameter later renamed as a prediction, no equation defining one result in terms of another, and no uniqueness theorem imported from the authors' prior work. The only self-citation is reference [35] for MLBVA, which is used as a comparison baseline rather than as a premise supporting the paper's conclusions, so it is not load-bearing. The paper's own admission in Section IV.C that GPT 'did not fully analyze boundary values when generating boundary-focused test inputs' is a legitimate construct-validity caveat about whether the prompt manipulation actually produced boundary-concentrated inputs, but it does not make the measured kill rates or coverage values equivalent to the prompt inputs by construction. The central claims are grounded in the reported experimental comparisons and do not reduce to their inputs, so no circularity is present.
Assumptions & free parameters
assumptions (3)
- domain assumption A mutant is killed when its execution path differs from the original on at least one generated input.
- domain assumption The six manually injected fault types are representative of common boundary-related faults.
- domain assumption The Gcov tool's branch and statement coverage counts are an accurate measure of test coverage for these C programs.
Cite this review
Pith. "Pith review of Boundary Value Test Input Generation Using Prompt Engineering with LLMs: Fault Detection and Coverage Analysis." pith.science (2026). https://pith.science/paper/BFJRTAAR
@misc{pith2026250114465,
author = {Pith},
title = {Pith review of: Boundary Value Test Input Generation Using Prompt Engineering with LLMs: Fault Detection and Coverage Analysis},
year = {2026},
howpublished = {\url{https://pith.science/paper/BFJRTAAR}},
note = {Machine review of arXiv:2501.14465}
}
read the original abstract
As software systems grow more complex, automated testing has become essential to ensuring reliability and performance. Traditional methods for boundary value test input generation can be time-consuming and may struggle to address all potential error cases effectively, especially in systems with intricate or highly variable boundaries. This paper presents a framework for assessing the effectiveness of large language models (LLMs) in generating boundary value test inputs for white-box software testing by examining their potential through prompt engineering. Specifically, we evaluate the effectiveness of LLM-based test input generation by analyzing fault detection rates and test coverage, comparing these LLM-generated test sets with those produced using traditional boundary value analysis methods. Our analysis shows the strengths and limitations of LLMs in boundary value generation, particularly in detecting common boundary-related issues. However, they still face challenges in certain areas, especially when handling complex or less common test inputs. This research provides insights into the role of LLMs in boundary value testing, underscoring both their potential and areas for improvement in automated testing methods.
Figures
Figures from the paper (3 more)
Reference graph
Works this paper leans on
- [1]
-
[2]
Analysis and testing using boundary value analysis methods for geographic information system,
T. Puspitasari, A. Kurniasari, and P. Puspitasari, “Analysis and testing using boundary value analysis methods for geographic information system,” in IOP Conference Series: Earth and Environmental Science , vol. 1168, no. 1. IOP Publishing, 2023, p. 012051
work page 2023
-
[3]
I. Forg ´acs and A. Kov´acs, “Domain testing,” in Modern Software Testing Techniques: A Practical Guide for Developers and Testers . Springer, 2023, pp. 119–206
work page 2023
-
[4]
Challenges and barriers to software testing,
S. Ashiq, A. B. Masood, M. H. Fakhar, M. W. Iqbal, Z. Nazir, H. A. B. Muhammad, S. U. Rahman, S. Z. Ahmad et al., “Challenges and barriers to software testing,” Bulletin of Business and Economics (BBE) , vol. 13, no. 1, 2024
work page 2024
-
[5]
GPT-3: Its nature, scope, limits, and consequences,
L. Floridi and M. Chiriatti, “GPT-3: Its nature, scope, limits, and consequences,” Minds and Machines , vol. 30, pp. 681–694, 2020
work page 2020
-
[6]
Bert: Pre-training of deep bidirectional transformers for language understanding,
J. D. M.-W. C. Kenton and L. K. Toutanova, “Bert: Pre-training of deep bidirectional transformers for language understanding,” in Proceedings of naacL-HLT, vol. 1. Minneapolis, Minnesota, 2019, p. 2
2019
-
[7]
Intellicode compose: Code generation using transformer,
A. Svyatkovskiy, S. K. Deng, S. Fu, and N. Sundaresan, “Intellicode compose: Code generation using transformer,” in Proceedings of the 28th ACM joint meeting on European software engineering conference and symposium on the foundations of software engineering , 2020, pp. 1433–1443
2020
-
[8]
S. Thummalapenta, S. Sinha, N. Singhania, and S. Chandra, “Automat- ing test automation,” in 2012 34th international conference on software engineering (ICSE). IEEE, 2012, pp. 881–891
work page 2012
Show all 35 references
-
[9]
Protocol for a systematic literature review on the evaluation of code recommender systems,
D. Borst, S. Sobernig, K. Innerebner, and D. Curation, “Protocol for a systematic literature review on the evaluation of code recommender systems,” 2024
2024
-
[10]
Are we testing or being tested? exploring the practical applications of large language models in software testing,
R. Santos, I. Santos, C. Magalhaes, and R. de Souza Santos, “Are we testing or being tested? exploring the practical applications of large language models in software testing,” in 2024 IEEE Conference on Software Testing, Verification and Validation (ICST) . IEEE, 2024, pp. 353–360
2024
-
[11]
Software testing with large language models: Survey, landscape, and vision,
J. Wang, Y . Huang, C. Chen, Z. Liu, S. Wang, and Q. Wang, “Software testing with large language models: Survey, landscape, and vision,”IEEE Transactions on Software Engineering , vol. 50, pp. 911–936, 2024
2024
-
[12]
Large language models for software engi- neering: A systematic literature review,
X. Hou, Y . Zhao, Y . Liu, Z. Yang, K. Wang, L. Li, X. Luo, D. Lo, J. Grundy, and H. Wang, “Large language models for software engi- neering: A systematic literature review,” ACM Transactions on Software Engineering and Methodology , 2023
2023
-
[13]
Boundary value ex- ploration for software analysis,
F. Dobslaw, F. G. de Oliveira Neto, and R. Feldt, “Boundary value ex- ploration for software analysis,” in 2020 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW) . IEEE, 2020, pp. 346–353
2020
-
[14]
An empirical analysis of equivalence partitioning, boundary value analysis and random testing,
S. C. Reid, “An empirical analysis of equivalence partitioning, boundary value analysis and random testing,” in Proceedings fourth international software metrics symposium . IEEE, 1997, pp. 64–73
1997
-
[15]
Random testing: Theoretical results and practical implications,
A. Arcuri, M. Z. Iqbal, and L. Briand, “Random testing: Theoretical results and practical implications,” IEEE transactions on Software En- gineering, vol. 38, no. 2, pp. 258–277, 2011
2011
-
[16]
EvoMBT: Evolutionary model based testing,
R. Ferdous, C.-k. Hung, F. Kifetew, D. Prandi, and A. Susi, “EvoMBT: Evolutionary model based testing,” Science of Computer Programming , vol. 227, p. 102942, 2023
2023
-
[17]
A comprehensive overview of large language models,
H. Naveed, A. U. Khan, S. Qiu, M. Saqib, S. Anwar, M. Usman, N. Akhtar, N. Barnes, and A. Mian, “A comprehensive overview of large language models,” arXiv preprint arXiv:2307.06435 , 2023
2023 arXiv
-
[18]
TIFA: Accurate and interpretable text-to-image faithfulness evaluation with question answering,
Y . Hu, B. Liu, J. Kasai, Y . Wang, M. Ostendorf, R. Krishna, and N. A. Smith, “TIFA: Accurate and interpretable text-to-image faithfulness evaluation with question answering,” in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2023, pp. 20 406–20 417
2023
-
[19]
TenGAN: Pure transformer encoders make an efficient discrete gan for de novo molecular generation,
L. Chen and Y . Yoshihiro, “TenGAN: Pure transformer encoders make an efficient discrete gan for de novo molecular generation,” in Inter- national Conference on Artificial Intelligence and Statistics . PMLR, 2024, pp. 361–369
2024
-
[20]
GxV AEs: Two joint vaes generate hit molecules from gene expression profiles,
C. Li and Y . Yamanishi, “GxV AEs: Two joint vaes generate hit molecules from gene expression profiles,” in Proceedings of the AAAI Conference on Artificial Intelligence , vol. 38, no. 12, 2024, pp. 13 455–13 463
2024
-
[21]
Large language models as oracles for instantiating ontologies with domain-specific knowledge,
G. Ciatto, A. Agiollo, M. Magnini, and A. Omicini, “Large language models as oracles for instantiating ontologies with domain-specific knowledge,” arXiv preprint arXiv:2404.04108 , 2024
2024 arXiv
-
[22]
Towards an intelligent test case generation framework using llms and prompt engineering,
M. Boukhlif, N. Kharmoum, M. Hanine, M. Kodad, and S. N. Lagmiri, “Towards an intelligent test case generation framework using llms and prompt engineering,” in International Conference on Smart Medical, IoT & Artificial Intelligence . Springer, 2024, pp. 24–31
2024
-
[23]
Chatunitest: A framework for llm-based test generation,
Y . Chen, Z. Hu, C. Zhi, J. Han, S. Deng, and J. Yin, “Chatunitest: A framework for llm-based test generation,” in Companion Proceedings of the 32nd ACM International Conference on the Foundations of Software Engineering, 2024, pp. 572–576
2024
-
[24]
Harnessing the power of llms in practice: A survey on chatgpt and beyond,
J. Yang, H. Jin, R. Tang, X. Han, Q. Feng, H. Jiang, S. Zhong, B. Yin, and X. Hu, “Harnessing the power of llms in practice: A survey on chatgpt and beyond,” ACM Transactions on Knowledge Discovery from Data, vol. 18, no. 6, pp. 1–32, 2024
2024
-
[25]
A review of current trends, techniques, and challenges in large language models (llms),
R. Patil and V . Gudivada, “A review of current trends, techniques, and challenges in large language models (llms),” Applied Sciences, vol. 14, no. 5, p. 2074, 2024
2024
-
[26]
Boundary value analysis in automatic white-box test generation,
Z. Zhang, T. Wu, and J. Zhang, “Boundary value analysis in automatic white-box test generation,” in 2015 IEEE 26th International Symposium on Software Reliability Engineering (ISSRE). IEEE, 2015, pp. 239–249
2015
-
[27]
An analysis and survey of the development of mutation testing,
Y . Jia and M. Harman, “An analysis and survey of the development of mutation testing,” IEEE transactions on software engineering , vol. 37, no. 5, pp. 649–678, 2010
2010
-
[28]
Gcov: https://gcc.gnu.org/onlinedocs/gcc/gcov.html
“Gcov: https://gcc.gnu.org/onlinedocs/gcc/gcov.html.”
-
[29]
Pathcrawler: Automatic generation of path tests by combining static and dynamic analysis,
N. Williams, B. Marre, P. Mouy, and M. Roger, “Pathcrawler: Automatic generation of path tests by combining static and dynamic analysis,” in European Dependable Computing Conference. Springer, 2005, pp. 281– 292
2005
-
[30]
Mc/dc automatic test input data generation,
Z. Awedikian, K. Ayari, and G. Antoniol, “Mc/dc automatic test input data generation,” in Proceedings of the 11th Annual conference on Genetic and evolutionary computation , 2009, pp. 1657–1664
2009
-
[31]
Searching for test data,
K. Ghani, “Searching for test data,” Ph. D Thesis , 2009
2009
-
[32]
Numerical recipes in c,
S. A. Teukolsky, B. P. Flannery, W. Press, and W. Vetterling, “Numerical recipes in c,” SMR, vol. 693, no. 1, pp. 59–70, 1992
1992
-
[33]
Supporting controlled experi- mentation with testing techniques: An infrastructure and its potential impact,
H. Do, S. Elbaum, and G. Rothermel, “Supporting controlled experi- mentation with testing techniques: An infrastructure and its potential impact,” Empirical Software Engineering , vol. 10, pp. 405–435, 2005
2005
-
[34]
Klee: unassisted and automatic generation of high-coverage tests for complex systems programs
C. Cadar, D. Dunbar, D. R. Engler et al., “Klee: unassisted and automatic generation of high-coverage tests for complex systems programs.” in OSDI, vol. 8, 2008, pp. 209–224
2008
-
[35]
Towards high-quality test suite generation with ml-based boundary value analysis,
X. Guo, H. Okamura, and T. Dohi, “Towards high-quality test suite generation with ml-based boundary value analysis,” in 2023 10th In- ternational Conference on Dependable Systems and Their Applications (DSA), 2023, pp. 75–85
2023
Reviewed August 10, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.