{"id":"9579d219-cb6e-4303-a499-9a4e949ed8eb","arxiv_id":"2501.14465","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"GPT-4o with simple boundary prompts catches some injected faults in small C programs, but it does not consistently beat random testing, MLBVA, or Klee, and the boundary prompts are not verified to produce boundary values.","lead":"This paper evaluates whether GPT-4o, guided by simple prompts, can generate boundary value test inputs for C programs and catch injected faults. It finds LLMs useful in some cases, but no consistent advantage over traditional testing methods and little evidence that the prompts actually produce boundary values.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central comparison rests on an unvalidated construct: boundary prompts are assumed to generate boundary-concentrated inputs, yet the paper admits GPT 'did not fully analyze boundary values' in Section IV.C.","rationale":"The reader's weakest assumption is exactly the load-bearing concern I identified: the paper never validates that boundary prompts produce boundary-concentrated inputs. My reading of the full text confirms this and adds the specific quotation from Section IV.C where the authors admit the limitation. This is not an external or exotic concern; it is the core construct validity of the study. The paper's empirical measurements may be internally consistent, and the authors are honest about the limitation, but the central comparison between boundary and general inputs is uninterpretable without validation. Therefore the CONDITIONAL verdict is appropriate: the paper should not be accepted as a reliable benchmark until the authors release the generated test sets and demonstrate that the boundary prompts actually shift the input distribution toward the predicates' boundaries. I do not recommend changing the reader's verdict because the concern is already reflected in their weakest_assumption and in their required revision list. The concrete test above is the single check that would settle whether this concern lands: if the boundary and general prompt distributions are indistinguishable, then the paper's central claim about boundary value generation collapses; if they are distinguishable, the RQ1 and RQ4 results become interpretable and the main risk is resolved.","tokens_in":14683,"tokens_out":4781,"duration_ms":42041,"concrete_test":"For each of the eight programs, extract the atomic comparison predicates from the source (e.g., a>0, a<=60, a+b<=5). For every generated test input across all four prompts and three runs, compute the minimal normalized distance from the input to the set of boundary points of each predicate, and record whether the input lies exactly on a boundary. Then perform a two-sample test (e.g., Mann-Whitney U) comparing the boundary-prompt distributions (Prompts 1 and 3) against the general-prompt distributions (Prompts 2 and 4). If the boundary prompts do not yield significantly smaller distances or a significantly higher fraction of exact-boundary inputs, then the prompt manipulation failed and the RQ1/RQ4 comparisons cannot be interpreted as evaluating boundary value generation.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is that prompt-engineered LLMs generate boundary value test inputs that detect boundary-related faults at rates comparable to or better than traditional methods. The key independent variable is the prompt: Prompts 1 and 3 ask for boundary value inputs, while Prompts 2 and 4 ask for general inputs. The entire RQ1 comparison (boundary vs. general) and the RQ4 correlation analysis hinge on these two prompt families actually producing inputs with different distributions over the input space, specifically with boundary-focused inputs closer to the predicates' boundaries. The paper never validates this. There is no measurement of whether the generated inputs lie at, near, or away from the boundary regions of the conditions in the code. Worse, the authors state in Section IV.C: 'although GPT did not fully analyze boundary values when generating boundary-focused test inputs, it still considered some boundary values when generating general test inputs.' This admission means the manipulation may be weak or absent: if the boundary prompt does not concentrate inputs near boundaries, then the observed differences in kill rate and coverage between boundary and general prompts cannot be attributed to boundary-focused generation, and the comparison to MLBVA and Klee is not a comparison of boundary value testing. The coverage results in Tables IV and V also show that general prompts often match or exceed boundary prompts, consistent with the concern that the prompt manipulation did not change the input distribution in the intended way. Without validating the construct, the paper's headline conclusion about LLMs' strengths and limitations in boundary value generation is not supported by the data, even though the authors are appropriately cautious in the discussion.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes prompt engineering with GPT-4o to generate boundary-value test inputs for white-box testing, and compares the generated inputs with general-prompt GPT inputs, random testing, concolic testing (Klee), and an ML-based boundary value analysis (MLBVA). For eight C programs with manually injected faults, the authors measure fault detection rate (defined by path differences) and statement/branch coverage. They report mixed findings: boundary-prompts sometimes outperform general prompts and MLBVA on kill rate, but not consistently, and increasing the number of inputs does not reliably improve effectiveness.","tokens_in":14888,"tokens_out":2898,"duration_ms":29089,"significance":"This is a useful empirical exploration of a practically relevant question: whether LLM prompt engineering can approximate or complement classical boundary value analysis. The study has concrete strengths: multiple subject programs, a clear fault-injection procedure, replication of prompt runs, and comparison with several established baselines. However, the central independent variable—the distinction between boundary-focused and general test inputs—is not validated, and the authors themselves concede this in Section IV.C. Because the paper's main claims rest on that distinction, the significance of the results is conditional on additional validation. If the boundary/general manipulation were shown to produce measurably different input distributions, the comparison would provide valuable evidence about LLM-based white-box BVA; as written, the conclusion is not yet supported.","major_comments":[{"comment":"The central comparison in RQ1 is not construct-valid. The authors report in Section IV.C that GPT 'did not fully analyze boundary values when generating boundary-focused test inputs' and 'still considered some boundary values when generating general test inputs.' Yet the paper never measures whether the inputs produced by Prompts 1 and 3 actually lie closer to the boundary regions of the predicates than those from Prompts 2 and 4. Without such a measurement (e.g., for each comparison predicate like a<=60, the frequency of inputs at the boundary value and its immediate neighbors), the observed differences in kill rate and coverage cannot be attributed to boundary-focused generation. Please add a quantitative validation of the manipulation, or reframe the research questions to compare 'prompt wording' rather than 'boundary value analysis'.","section":"IV.C (RQ1 discussion)"},{"comment":"RQ1 asks whether there is a 'significant difference' in fault detection and coverage between boundary and general inputs, but the paper provides no statistical analysis. Figure 2 and Tables III–V show overlapping values across the three runs, and with data from only eight programs it is unclear whether the differences are meaningful. Please add paired statistical tests (e.g., Wilcoxon signed-rank) with effect sizes and confidence intervals, or clearly state that the analysis is descriptive and avoid the word 'significant'.","section":"IV.C (RQ1)"},{"comment":"The correlation analysis in RQ4 is not adequately specified. It is unclear how many data points are used in each regression and whether R² values are computed across the three runs per prompt per program, or across programs. Some values in Figure 6, notably R² = -1 for plgndr under genGPT, are not possible for a standard linear regression R² and suggest a different formula or a fit artifact. Moreover, because kill rate is defined by path differences, a correlation between branch coverage and kill rate is partly tautological. Please report the regression setup, the number of points, the actual fitted lines, and significance tests, and discuss the definitional dependency between the two metrics.","section":"IV.C (RQ4, Figures 3 and 6)"},{"comment":"The comparison with Klee is not fair because the test set sizes differ by orders of magnitude: for bessj and expint, Klee generates only 2 and 4 tests respectively, while the GPT methods generate 50. The low kill rates for Klee on those programs (0.37 and 0.38) may reflect insufficient test count rather than lower fault-detection capability. In contrast, for plgndr Klee generates 14,221 tests. Please either control for test suite size (e.g., using effort-aware measures such as kill rate per test, or adapting Klee to produce a comparable number of tests) or restrict the comparison to configurations with comparable test counts, and explicitly discuss the effect of size on the reported differences.","section":"IV.A and Table III (RQ3)"}],"minor_comments":[{"comment":"In the row for 50BVAGPT2 under findMiddle, the branch coverage value is written as '0..97'; this appears to be a typo for '0.97'.","section":"Table V"},{"comment":"The paper reports R² values that can be negative in Figure 6; if the intent is to show correlation strength, please use Pearson correlation coefficient (r) or explain the regression model and why negative R² arises.","section":"IV.C and Figure 6"},{"comment":"The exact prompts are summarized but not quoted verbatim in a dedicated appendix or table; for reproducibility, please include the full prompt templates used in the experiments.","section":"IV.B"},{"comment":"The 'English' program is described only as 'English examination program'; a brief description of its logic and input domain would make the subject selection more transparent.","section":"Table II"},{"comment":"The definition of a killed mutant relies on execution path differences rather than output differences; this is a reasonable white-box choice but should be discussed as a threat to validity, since path changes can occur without observable failures and vice versa.","section":"III.C"},{"comment":"The MLBVA baseline is from the authors' prior work (reference [35]); please state explicitly in the text that MLBVA is an internal baseline and describe the configuration used in this study beyond the cited paper.","section":"IV.A"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is a solid starting point for an empirical study on LLM-based boundary value generation, but the main threat to validity is the unvalidated boundary/general prompt manipulation. The authors' own admission in Section IV.C that the model did not reliably analyze boundaries is the crux: without measuring the input distributions, RQ1 and much of the paper lose meaning. The other issues (lack of significance tests, opaque correlation analysis, and unequal baseline sizes) are fixable. I recommend major revision rather than rejection because the underlying question is worthwhile and the experimental setup can be augmented with the missing validation and analysis."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper is a straightforward empirical study: it asks GPT-4o to generate test inputs with boundary-focused and general prompts, then compares kill rates and coverage against random testing, MLBVA, and Klee on eight small C programs. What is genuinely new is the specific benchmark—this particular set of programs, manual fault injection, and path-based kill criterion has not been published before. The authors are also honest about limitations; in Section IV.C they admit GPT did not fully analyze boundary values when generating boundary-focused inputs and that general prompts sometimes considered boundaries anyway. That honesty deserves credit.\n\nThe protocol is reasonable: using Gcov execution paths to decide whether a mutant is killed is a sound white-box approach, and the choice of programs (tcas, nextDate, etc.) gives the results some external grounding. The comparison against MLBVA is interesting, and the authors do not overstate their own baseline.\n\nThe soft spots are real and central. The paper never validates that the boundary prompts actually produce inputs near the predicates' boundaries. Given the authors' admission, the observed differences between boundary and general prompts cannot be attributed to boundary-focused generation. That weakens RQ1's conclusion and the RQ4 interpretation. The statistics are also thin: the paper uses the language of 'significant difference' without any significance test, and the R2 values in Figure 3 include impossible negative values (e.g., -1 for plgndr), which suggests the metric is mislabeled or miscalculated. No artifacts are released, so the benchmark is not currently reproducible.\n\nDespite these issues, the paper deserves serious referee time. The question is timely, the empirical effort is real, and a revision that validates the generated inputs, fixes the statistics, and releases the artifacts could turn this into a useful reference point. I would not cite it in its current form, but I would engage with it.\n\nRecommendation: send to peer review with major revision required. The flaws are fixable, but the current version does not support its headline claim about boundary value generation.","headline":"Useful empirical data on GPT-4o for test input generation, but the central boundary-vs-general comparison is not validated by the authors' own admission.","tokens_in":15484,"tokens_out":1485,"would_cite":false,"duration_ms":18072,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Prompt-engineered GPT-4o can generate boundary value test inputs that rival traditional boundary value analysis in fault detection and coverage for selected programs, while struggling with complex inputs.","keywords":["software testing","boundary value analysis","large language models","prompt engineering","fault detection","test coverage","mutation testing","GPT-4o"],"falsifier":"Check whether the test inputs from the boundary prompts (Prompt 1 and Prompt 3) are actually closer to the comparison-predicate boundaries than the inputs from the general prompts (Prompt 2 and Prompt 4). If, for a program such as triType with conditions like `a>0 && a<=60`, the boundary-prompted inputs are not concentrated at the boundary values (e.g., 0, 1, 60, 61), then the reported boundary-specific advantages would not be tied to boundary generation and the comparison collapses.","tokens_in":14462,"feed_emoji":"🧪","tokens_out":7600,"duration_ms":59751,"temperature":0.7,"pith_summary":"This paper asks whether a large language model, guided by boundary-focused prompts, can generate test inputs that match traditional boundary value analysis in white-box testing. The authors test GPT-4o on eight small C programs, comparing boundary-prompted and general-prompted inputs against random testing, concolic execution via Klee, and an ML-based boundary value analyzer, measuring kill rates on injected mutants plus statement and branch coverage. They find that boundary-prompted GPT-4o inputs achieve higher fault detection rates than the ML baseline in six of eight programs, while the advantage over general prompts largely disappears when the test set is fixed at 50 inputs. The paper concludes that LLMs are a useful supplement to existing boundary testing methods, with clear limitations on complex or uncommon inputs.","feed_headline":"Prompted GPT-4o beats ML boundary analyzer on 6 of 8 programs","feed_subtitle":"Boundary-focused prompts give GPT-4o fault detection on par with classic methods, but quantity does not boost quality.","key_machinery":"The load-bearing object is the four-prompt comparison on GPT-4o: Prompt 1 asks for boundary value test inputs, Prompt 2 for general test inputs, Prompt 3 for exactly 50 boundary value test inputs, and Prompt 4 for exactly 50 general test inputs. The evaluation harness injects six fault types into each program, executes every generated input on the original and all mutants, and uses Gcov to extract execution paths; a mutant is killed if any input drives a different path than the original. This path-difference definition is what makes the white-box boundary analysis operational, because it ties fault detection to divergence in branch behavior rather than to output comparison alone.","core_discovery":"The central claim is that GPT-4o, when guided by boundary-focused prompts, produces test inputs whose fault detection and coverage are comparable to or better than traditional methods in selected cases. The paper establishes this on eight small C programs, using six hand-injected fault types (off-by-one, constant replacement, relational operator replacement, arithmetic operator replacement, scalar variable replacement, logical operator replacement) and a white-box kill criterion: a mutant is killed when at least one input produces an execution path, captured by Gcov, that differs from the original program's path. In the 50-input comparison, GPT-4o's boundary-prompted sets achieved higher kill rates than the machine-learning-based MLBVA baseline in six of eight programs, and matched or exceeded random testing and Klee on several programs. The paper also reports that boundary prompts did not consistently beat general prompts once input count was fixed, and that GPT-4o did not fully analyze boundary values when generating boundary-focused inputs, which the authors interpret as evidence that LLMs are a promising supplement rather than a complete replacement for existing boundary testing techniques.","pith_inferences":["Extension: validate the generated inputs against the actual predicate boundaries (for example, for a condition like `a>0 && a<=60`, check that boundary sets concentrate on 0, 1, 60, 61); if Prompt 1 and Prompt 2 outputs are indistinguishable in distance to boundaries, the reported boundary advantage could be an artifact of input quantity rather than boundary targeting.","Extension: rerun the comparison with output-based or assertion-based oracles instead of path-difference; the currently reported kill rates are upper bounds because path divergence can count coincidental branch changes, so absolute kill rates would likely drop.","Inference: the paper's observed weakness on interdependent predicates points to a hybrid architecture—use GPT-4o to propose candidate boundary inputs and a lightweight constraint solver to filter or refine them—which could combine LLM breadth with exact boundary computation."],"forward_implications":["Boundary-focused prompts on GPT-4o can produce higher mutant kill rates than an ML-based boundary analyzer in the majority of the eight studied programs, suggesting LLMs are a competitive source of boundary-directed tests.","Increasing the number of generated test inputs does not reliably improve fault detection or coverage, so test-set quality and prompt design matter more than raw quantity.","When the test set is fixed at 50 inputs, boundary prompts do not consistently outperform general prompts, indicating GPT-4o already incorporates some boundary-like values during general generation.","Fault detection rate and branch coverage are positively correlated, especially for boundary-prompted inputs, supporting branch coverage as a practical proxy when evaluating boundary-focused generation."],"supporting_citations":[{"why":"Defines the random testing baseline that GPT-4o's kill rates and coverage are compared against.","marker":"[15]"},{"why":"Supplies the white-box boundary value analysis definition of predicates and branches that the kill criterion relies on.","marker":"[26]"},{"why":"Supplies five of the six mutation operator types used to create faulty program versions.","marker":"[27]"},{"why":"The Gcov tool used to extract execution paths and compute statement and branch coverage.","marker":"[28]"},{"why":"Klee provides the concolic testing baseline for comparison.","marker":"[34]"},{"why":"MLBVA is the machine-learning-based boundary analysis method that GPT-4o's boundary inputs are compared against.","marker":"[35]"}],"fun_headline_variants":["GPT-4o with boundary prompts beats ML on 6 of 8 test programs","Boundary-focused prompts let GPT-4o match or beat classic test generation","LLM test generation: GPT-4o with boundary prompts hits 6/8 vs ML","Prompt engineering helps GPT-4o match classic boundary value testing","Boundary prompts give GPT-4o edge in 6 of 8 test programs"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The argument assumes that the boundary-focused prompts actually change what GPT-4o produces, meaning the resulting inputs really sit at or near the program's boundary conditions; the paper itself notes that GPT-4o did not fully analyze boundary values when generating boundary-focused inputs, and the generated inputs are never validated against the code's predicates.","fun_headline_variants_meta":{"raw":{"variants":["GPT-4o with boundary prompts beats ML on 6 of 8 test programs","Boundary-focused prompts let GPT-4o match or beat classic test generation","LLM test generation: GPT-4o with boundary prompts hits 6/8 vs ML","Prompt engineering helps GPT-4o match classic boundary value testing","Boundary prompts give GPT-4o edge in 6 of 8 test programs"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001069,"raw_usage":{"total_tokens":4479,"prompt_tokens":943,"completion_tokens":3536,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":559,"completion_tokens_details":{"reasoning_tokens":3430}},"tokens_in":559,"tokens_out":3536,"duration_ms":21295,"temperature":1.0,"reasoning_tokens":3430,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T15:07:37.648165+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Check whether the test inputs from the boundary prompts (Prompt 1 and Prompt 3) are actually closer to the comparison-predicate boundaries than the inputs from the general prompts (Prompt 2 and Prompt 4). If, for a program such as triType with conditions like `a>0 && a<=60`, the boundary-prompted inputs are not concentrated at the boundary values (e.g., 0, 1, 60, 61), then the reported boundary-specific advantages would not be tied to boundary generation and the comparison collapses.","supporting_citations":[{"cited_title":"Random testing: Theoretical results and practical implications,","cited_arxiv_id":null,"evidence_quote":"Defines the random testing baseline that GPT-4o's kill rates and coverage are compared against."},{"cited_title":"Boundary value analysis in automatic white-box test generation,","cited_arxiv_id":null,"evidence_quote":"Supplies the white-box boundary value analysis definition of predicates and branches that the kill criterion relies on."},{"cited_title":"An analysis and survey of the development of mutation testing,","cited_arxiv_id":null,"evidence_quote":"Supplies five of the six mutation operator types used to create faulty program versions."},{"cited_title":"Gcov: https://gcc.gnu.org/onlinedocs/gcc/gcov.html","cited_arxiv_id":null,"evidence_quote":"The Gcov tool used to extract execution paths and compute statement and branch coverage."},{"cited_title":"Towards high-quality test suite generation with ml-based boundary value analysis,","cited_arxiv_id":null,"evidence_quote":"MLBVA is the machine-learning-based boundary analysis method that GPT-4o's boundary inputs are compared against."}],"review_version":1}