{"id":"db266a6d-e967-4651-bcba-2c295985fcd8","arxiv_id":"2607.00427","paper_version":1,"verdict":"UNVERDICTED","confidence":"LOW","novelty_score":6.0,"correctness_risk":"unknown","formal_verification":"none","parameter_count":0,"one_line_summary":"BT-APE automates prompt engineering for requirements classification using backtracking search and dynamic examples, matching PE2 accuracy while using 72% fewer tokens and 66% less time than that baseline.","lead":"The paper introduces BT-APE, a backtracking search method that automatically designs prompts for LLM-based classification of software requirements. A smart generalist might read it to see how to reduce the manual effort and compute cost of making LLMs reliable for engineering tasks.","discovery_kind":"new_method","skeptic_critique":{"model":"grok-4.3","headline":"Generalization of BT-APE beyond the three datasets and five LLMs is untested","rationale":"The reader's weakest assumption directly identifies the same load-bearing point for the central claim. Because the paper presents BT-APE as a general lightweight framework for RE tasks, the narrow evaluation scope is the place where the argument is least secure; the specific numerical claims on the three datasets are not internally contradicted by the given information.","tokens_in":1765,"tokens_out":322,"duration_ms":19512,"concrete_test":"Select one additional public requirements-classification dataset (e.g., a held-out PROMISE or new industrial set) and one extra instruction-tuned LLM not used in the original study; run BT-APE, PE2, and the four classical baselines for the same number of trials; compare accuracy, input-token count, and wall-clock time. If the accuracy advantage or efficiency margin shrinks below the reported effect sizes, the generalization concern is confirmed.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The strongest claim asserts near-identical accuracy to PE2 plus large efficiency gains (72% fewer tokens, 66% less time) on requirements classification. This rests entirely on results from three specific benchmarks and five LLMs; the backtracking + dynamic example selection procedure could locate prompts that overfit those exact distributions and model behaviors. No evidence is supplied that the same search reliably produces generalizable prompts on unseen RE datasets or additional LLMs, so the practical advantage claimed for resource-constrained deployment cannot be separated from the narrow evaluation scope.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.3","summary":"The paper proposes BT-APE, a lightweight backtracking automatic prompt engineering method for requirements classification. It models prompt design as an iterative optimization using LLM-generated candidates, backtracking search, and dynamic example selection. On three benchmark datasets and five instruction-tuned LLMs, BT-APE matches the accuracy of the resource-intensive PE2 baseline while using ~72% fewer input tokens and ~66% less wall-clock time, and substantially outperforms classical baselines (zero-shot, few-shot, CoT, CoT+few-shot) with large effect sizes. Contributions include an open tool and replication package, the first systematic APE vs. classical prompting comparison in RE, and insights on class definitions and prompt evolution.","tokens_in":1871,"tokens_out":486,"duration_ms":25035,"significance":"If the efficiency claims hold under broader testing, the work offers a practical advance for requirements engineering by enabling effective prompt optimization in resource-constrained environments, where full APE methods like PE2 are impractical. The open replication package and tool are positive for reproducibility.","major_comments":[{"comment":"§4 Evaluation and Table 3 (results): The reported accuracy parity with PE2 and large effect sizes over baselines are presented without statistical tests, variance across LLM stochastic runs, or controls for prompt variability, leaving the robustness of the central performance claims unverified.","section":"§4 Evaluation and Table 3"},{"comment":"§5 Discussion and §6 Conclusion: The claim that BT-APE is 'better suited to resource-constrained deployment' rests entirely on results from three specific datasets and five LLMs; no cross-dataset or cross-LLM generalization experiments are reported, which is load-bearing for separating the efficiency advantage from potential overfitting to the evaluation scope.","section":"§5 Discussion and §6 Conclusion"}],"minor_comments":[{"comment":"The abstract and method description should clarify the exact backtracking termination criteria and dynamic example selection heuristic with pseudocode or equations for reproducibility.","section":"§3 Method"},{"comment":"Figure 2 (prompt evolution example) would benefit from explicit annotation of backtracking steps and token counts to directly illustrate the claimed efficiency.","section":"Figure 2"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":{"model":"grok-4.3","summary":"We thank the referee for the constructive comments. We address each major comment below and note the planned revisions.","responses":[{"response":"We agree that the absence of statistical tests and variance reporting weakens the robustness claims. In the revised manuscript we will rerun all experiments with five random seeds per configuration, report means and standard deviations, and apply paired statistical tests (t-tests or Wilcoxon signed-rank) to the accuracy differences. These additions will be incorporated into §4 and Table 3.","revision_made":"yes","referee_comment":"[§4 Evaluation and Table 3] The reported accuracy parity with PE2 and large effect sizes over baselines are presented without statistical tests, variance across LLM stochastic runs, or controls for prompt variability, leaving the robustness of the central performance claims unverified."},{"response":"The current evaluation already spans three datasets and five LLMs with consistent efficiency gains. We acknowledge that additional cross-validation experiments would provide stronger separation from overfitting. In revision we will qualify the deployment claim in §5 and §6, explicitly state the evaluation scope as a limitation, and temper language to avoid implying broad generalization beyond the tested setting.","revision_made":"partial","referee_comment":"[§5 Discussion and §6 Conclusion] The claim that BT-APE is 'better suited to resource-constrained deployment' rests entirely on results from three specific datasets and five LLMs; no cross-dataset or cross-LLM generalization experiments are reported, which is load-bearing for separating the efficiency advantage from potential overfitting to the evaluation scope."}],"tokens_in":1430,"tokens_out":344,"duration_ms":21632,"standing_objections":[]},"desk_editor":{"model":"grok-4.3","letter":"BT-APE is a backtracking search method for automatic prompt engineering that reaches nearly the same accuracy as PE2 on requirements classification tasks while using roughly 72% fewer input tokens and 66% less time.\n\nThe paper sets up prompt design as an iterative optimization problem using LLM-generated candidates, backtracking, and dynamic example selection. It supplies an open interactive tool plus a replication package, which makes the work immediately usable. The direct head-to-head against zero-shot, few-shot, chain-of-thought, and CoT+few-shot baselines, plus the PE2 comparison, on three benchmark datasets and five LLMs is the first systematic check of this kind inside requirements engineering. Those efficiency numbers are the clearest practical result.\n\nThe evaluation is limited to exactly those three datasets and five models. No information appears on run-to-run variance, statistical tests, or prompt stability, so the reported large effect sizes need checking against the full numbers. The backtracking procedure could be locating prompts that fit the tested distributions rather than generalizing; nothing in the abstract or stress-test note shows results on held-out RE datasets or additional LLMs.\n\nThis is for requirements engineers who already use LLMs for classification and want a lighter alternative to full-scale APE. A reader focused on practical prompt methods in software engineering would extract the efficiency comparison and the tool.\n\nSend it for peer review. The core method and the efficiency claim are concrete enough to justify referee time, even with the narrow scope.","headline":"BT-APE matches a heavier APE method on accuracy for requirements classification while cutting tokens and time, but the gains rest on a narrow set of three datasets and five LLMs.","tokens_in":2348,"tokens_out":381,"would_cite":false,"duration_ms":18528,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.3","headline":"BT-APE produces prompts for requirements classification that reach the accuracy of heavy automatic methods while using 72 percent fewer input tokens and 66 percent less time.","keywords":["automatic prompt engineering","requirements classification","backtracking search","large language models","prompt optimization","software requirements engineering","computational efficiency"],"falsifier":"Running BT-APE on a fourth independent requirements classification dataset or with a sixth LLM and finding that its accuracy falls materially below PE2 or fails to exceed the classical baselines.","tokens_in":2683,"feed_emoji":"⚙️","tokens_out":704,"duration_ms":21577,"temperature":0.7,"pith_summary":"The paper frames prompt design for large language models as an optimization problem and introduces BT-APE, which generates candidate prompts, applies backtracking search to explore them, and uses dynamic example selection to refine choices. On three benchmark datasets and five instruction-tuned models, this approach reaches accuracy levels nearly identical to the more expensive PE2 baseline and well above classical methods such as zero-shot, few-shot, and chain-of-thought prompting. The decisive difference is the reduced computational load, which the authors quantify through token counts and wall-clock measurements. If the claim holds, automated prompt construction becomes practical in settings where repeated calls to large models carry high cost or latency constraints. The work also supplies an interactive tool and replication package to support further use.","feed_headline":"BT-APE matches heavy prompt engineering accuracy at 72% lower token cost","feed_subtitle":"Backtracking search plus dynamic examples delivers PE2-level results for requirements classification while cutting input tokens by 72 percen","key_machinery":"Backtracking search over LLM-generated prompt candidates combined with dynamic example selection.","core_discovery":"BT-APE iteratively refines prompts for requirements classification by having an LLM propose candidates, then applying backtracking search and dynamic example selection to identify high-performing prompts. Across three datasets and five LLMs it matches the accuracy of the resource-intensive PE2 method while consuming roughly 72 percent fewer input tokens and 66 percent less wall-clock time, and it substantially outperforms four classical prompting baselines with large effect sizes.","pith_inferences":["The same backtracking structure could be tested on related requirements engineering tasks such as traceability link recovery or ambiguity detection.","Lower token consumption might allow the optimization loop to run repeatedly inside interactive development environments without noticeable delay.","The observed interaction between class definitions and prompt evolution supplies a concrete starting point for studying how initial human-written constraints shape automated search outcomes."],"forward_implications":["Requirements classification tasks can shift from manual trial-and-error prompt writing to an automated search process.","Automated prompt engineering becomes viable under tighter compute budgets than those required by prior heavy methods.","Performance advantages over zero-shot, few-shot, chain-of-thought, and combined baselines hold across multiple datasets and models.","Class definitions supplied in the initial prompt influence how the search evolves later candidates.","An open tool and replication package make the method immediately usable for other requirements engineering classification problems."],"fun_headline_variants":["BT-APE matches PE2 accuracy with 72% fewer tokens","Backtracking delivers PE2 results saving 72% input tokens","BT-APE rivals heavy APE using 66% less wall-clock time","Lightweight BT-APE equals PE2 on requirements classification"],"cache_read_input_tokens":2112,"weakest_assumption_plain":"The combination of backtracking search and dynamic example selection will locate prompts that continue to perform well on datasets and models outside the three benchmarks and five LLMs examined.","fun_headline_variants_meta":{"raw":{"variants":["BT-APE matches PE2 accuracy with 72% fewer tokens","Backtracking delivers PE2 results saving 72% input tokens","BT-APE rivals heavy APE using 66% less wall-clock time","Lightweight BT-APE equals PE2 on requirements classification"]},"model":"grok-4.3","cost_usd":0.006135,"raw_usage":{"total_tokens":2921,"prompt_tokens":719,"num_sources_used":0,"completion_tokens":71,"cost_in_usd_ticks":61349500,"prompt_tokens_details":{"text_tokens":719,"audio_tokens":0,"image_tokens":0,"cached_tokens":256},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":2131,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":719,"tokens_out":71,"duration_ms":16462,"temperature":1.0,"reasoning_tokens":2131,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-07-02T09:10:47.501523+00:00","model_set":{"reader":"grok-4.3"},"falsifier":"Running BT-APE on a fourth independent requirements classification dataset or with a sixth LLM and finding that its accuracy falls materially below PE2 or fails to exceed the classical baselines.","supporting_citations":[],"review_version":1}