{"id":"0491b10d-17ce-4d6b-a2cc-e79dc94dafe0","arxiv_id":"2411.17404","paper_version":4,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"BPP-Search combines beam search, a process reward model, and pairwise preference ranking to improve tree-of-thought selection for LLM mathematical modeling, beating CoT and ToT baselines on the new StructuredOR and two existing OR datasets.","lead":"The authors release StructuredOR, a dataset of 124 operations-research word problems annotated with complete mathematical modeling steps, and propose BPP-Search, a tree-search method that uses a process reward model and pairwise preference ranking to select final answers. The method reports higher accuracy and fewer LLM calls than chain-of-thought and tree-of-thought baselines on three OR benchmarks, but evaluation is limited to a solvable subset and no code is released.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Headline accuracy is computed only on the 30/143/72 problems where GPT-4o's full ToT traversal found a correct leaf, excluding over half of NL4OPT and ComplexLP; dataset-level superiority is not established.","rationale":"The reader's weakest assumption correctly identifies the solvable-subset evaluation as the load-bearing premise. The set of problems included in Table 6 is not the full test set, and the paper's abstract claims dataset-level superiority without reporting the excluded cases. The proposed full-set recomputation, with verifier retraining on training splits only, would settle whether the selection changes the conclusion. The reader's CONDITIONAL verdict remains appropriate: if the full-set check preserves the margin, the concern is resolved; if not, the claim must be narrowed.","tokens_in":14994,"tokens_out":12164,"duration_ms":114435,"concrete_test":"Run a pre-registered full-test-set evaluation: retrain PRM and Preference Model on the declared training splits only (structurally excluding all 38 StructuredOR test problems), then evaluate BPP-Search, CoT, SC, and the ToT variants on all 38/289/211 test problems, counting any problem with no correct leaf generated as a failure. Report overall correct rates with the original denominators and separately on the complement of the Table 5 subset. If BPP-Search's margin over CoT shrinks or reverses, restrict the claims to the solvable subset.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Table 6's correct rates are computed on the subset of test problems 'confirmed solvable by the policy model in prior experiments' (Table 5 caption; Section 5.1). This filter keeps 30/38 StructuredOR, 143/289 NL4OPT, and 72/211 ComplexLP problems, so on NL4OPT and ComplexLP more than half the test set is removed. The abstract and conclusion claim dataset-level superiority for BPP-Search, but the reported evidence supports only the solvable subset. Because the filter is defined by GPT-4o's full ToT traversal containing at least one correct leaf, the comparison is conditional on a search oracle; single-path baselines such as CoT are evaluated only on problems where the full tree already contains a correct answer. The paper never reports behavior on the excluded hard cases, so if BPP-Search's margin is concentrated in easy or fully-searchable problems, the headline claim fails. A related unresolved point is that Section 3.3 uses StructuredOR ground truth as positive training data for the PRM without specifying whether the 38 test problems were excluded; this should be documented and tested.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces StructuredOR, a dataset of operations-research word problems annotated with the full mathematical modeling process (sets, parameters, variables, objectives, constraints) in addition to objective values, and proposes BPP-Search, a tree-of-thought variant that combines beam search with a process reward model (PRM) and a pairwise preference model. The PRM scores intermediate reasoning steps, and the preference model ranks the final candidates. Experiments on StructuredOR, NL4OPT, and MAMO-ComplexLP report that BPP-Search achieves higher correct rates and uses fewer reasoning steps than CoT, self-consistency, and several ToT baselines with an unfine-tuned GPT-4o policy model.","tokens_in":15281,"tokens_out":6057,"duration_ms":56963,"significance":"If the reported gains hold, the StructuredOR dataset is a potentially valuable community resource for process-supervision and RL research in OR modeling, and the pairwise-preference-over-PRM mechanism is a plausible answer to the known problem of poorly calibrated process-reward scores at leaf selection. The verifier training uses solver-validated objective values as labels, which is standard supervised learning rather than circular reasoning, and the dataset release URLs are provided. However, the evaluation currently supports the accuracy and efficiency claims only on a solvable subset of the test sets, under test-set-selected hyperparameters, and without uncertainty quantification. The headline 'significantly outperforms state-of-the-art' claim is therefore not yet established at dataset level.","major_comments":[{"comment":"The headline accuracy comparison is computed only on the subset of problems for which GPT-4o's full ToT traversal contains at least one correct leaf: 30/38 StructuredOR, 143/289 NL4OPT, and 72/211 ComplexLP. More than half of the NL4OPT and ComplexLP test sets are discarded, and no results are reported on the excluded problems. Because the filter is defined by an outcome of the search process, the comparison is conditional on that oracle, and the abstract's dataset-level claim of superiority is not supported. Please report correct rates on the full test sets or explicitly redefine the claim, and provide some analysis of the excluded problems.","section":"Section 5.1, Table 5, Table 6"},{"comment":"The 'Greedy Search Variant' row in Table 6 does not correspond to a single algorithm: it takes Random Greedy Search + PRM on StructuredOR (0.833), Greedy Search + PRM on MAMO-ComplexLP (0.555), and Epsilon Greedy Search + PRM on NL4OPT (0.713), as shown in Table 7. Similarly, the BPP-Search row uses beam width 2 on StructuredOR and NL4OPT but width 3 on MAMO-ComplexLP, and the Beam Search row uses the width that is best in Table 7 for each dataset. This is a test-set oracle selection of hyperparameters, which overstates the performance of a single method. Please fix hyperparameters on a validation split or report all configurations separately.","section":"Table 6 versus Table 7"},{"comment":"No variance estimates, confidence intervals, repeated runs, or significance tests are reported. On StructuredOR (n=30), BPP-Search's 0.933 correct rate is roughly 28/30, versus 23/30 for ToT-Rethink; on NL4OPT (n=143), the 0.804 versus 0.783 difference is roughly 3 problems out of 143. The word 'significantly' in the abstract and conclusion is therefore unsupported even within the solvable subset. Please add uncertainty quantification or substantially soften the significance claim.","section":"Table 6 and Abstract"},{"comment":"The PRM and Preference Model training procedures do not state whether the StructuredOR test set, or the NL4OPT and MAMO-ComplexLP test problems, were excluded from verifier training. Section 3.3 says StructuredOR annotations are used to augment PRM training data, and Section 4.2 says preference data are generated 'for each problem' using ToT paths classified by objective values, with no split described. If any test problem appears in verifier training, the BPP-Search results are directly inflated. Please specify the exact train/test split for both verifiers and confirm that no test problem is used in training.","section":"Section 3.3 and Section 4.2"},{"comment":"The comparison in Table 6 includes CoT, SC, ToT variants, and the authors' own search variants, all with an unfine-tuned GPT-4o policy. It does not include fine-tuned modeling systems such as ORLM or Chain-of-Experts, which are discussed in Related Work. If the claim is 'state-of-the-art' for the question-to-model task, the comparison is incomplete; if the claim is restricted to tree-search variants without policy fine-tuning, the abstract and conclusion should state that restriction.","section":"Table 6 and Related Work"}],"minor_comments":[{"comment":"The text says BPP-Search reduces computational costs 'exponentially,' but the reported step reductions (39/40 to 15/21) do not support exponential scaling; please rephrase.","section":"Section 5.2"},{"comment":"The threshold in the Random Greedy algorithm is never given a value, and no sensitivity analysis is provided; please report the chosen threshold and how it was selected.","section":"Equation (7), Section 4.3"},{"comment":"The reference list contains apparent duplicates and inconsistencies: Tang et al. 2024a and 2024b are identical entries, and the 'Qwen Team' entry is formatted inconsistently with the other entries; please clean this up.","section":"References"},{"comment":"The table caption uses 'ComplexLP' while the text uses 'MAMO-ComplexLP'; please use one consistent name throughout.","section":"Table 5"},{"comment":"Section 7.1 explicitly concedes that computational resources limited the depth and width of the ToT tree; since the tree shape is a shared input to all compared methods, a sentence on how this constraint affects the generality of the accuracy claims would be helpful.","section":"Section 7.1"}],"recommendation":"major_revision","confidential_remarks":"The StructuredOR dataset is a genuine contribution and the overall direction is sound, but the experimental evidence for the central claim is currently too selective. I would encourage a revision that reports full-dataset numbers, fixes hyperparameters on a validation split, and documents verifier train/test separation. The paper is not ready for acceptance in its present form."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The short version: this paper has a genuinely useful new dataset and a sensible-sounding algorithm, but the experimental claims outrun the evidence because the evaluation only runs on the subset of problems where GPT-4o's full tree search already found a correct answer. That's a load-bearing selection, and the abstract doesn't qualify it.\n\nWhat's new: StructuredOR, a small (124 examples) OR dataset with structured labels for the whole modeling process (sets, parameters, variables, objectives, constraints), not just objective values. That's a real gap-filler for process supervision work. The BPP-Search idea—using beam search during tree expansion, a learned PRM to prune, and a pairwise preference model to pick the best leaf—is a reasonable combination; none of the parts are new, but the pairwise ranking over final candidates is a fresh way to handle the PRM's poor score calibration.\n\nWhat's solid: the dataset pipeline is described carefully, with validation via Gurobi and manual review; the ablations show the pairwise model does help over plain beam search; and the limitations section is honest about computational trade-offs and the small size of the dataset.\n\nSoft spots, in order of severity:\n\n1. The evaluation filter. Table 5 says only 30/38, 143/289, 72/211 problems are \"confirmed solvable\" via full traversal. All accuracy numbers in Tables 6-7 are computed on that subset. On NL4OPT and ComplexLP, more than half the test set is dropped. The abstract and conclusion claim dataset-level superiority, but the data only supports a claim about problems the policy model can already solve inside the tree. This doesn't sink the method, but it means the headline numbers can't be taken at face value.\n\n2. Hyperparameter selection. The beam width that gets reported for each dataset appears to be the one that did best in the ablation (Table 7 vs Table 6). That's test-set fitting. The paper should either fix beam width ahead of time or report results across all widths.\n\n3. Missing baselines. \"State-of-the-art\" here means CoT, SC, and ToT variants. Fine-tuned OR models like ORLM are mentioned in the intro but not compared in the main table. That's a relevant omission.\n\n4. Potential PRM leakage. Section 3.3 says StructuredOR ground truth is used as positive training data for the PRM; the paper doesn't say how the 38 test problems were kept out. This needs to be clarified.\n\n5. Minor: no error bars or significance tests, and no code release for the algorithm.\n\nBottom line: the dataset is worth having, and the method is worth a serious referee. I'd send it to review with a request for full-test-set reporting, fixed hyperparameters, and a statement on PRM train/test separation.","headline":"A useful new OR dataset and a plausible search method, but the evaluation's solvable-subset filter means the headline accuracy claims are not yet supported.","tokens_in":15765,"tokens_out":2818,"would_cite":true,"duration_ms":26478,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"BPP-Search claims that beam-search pruning with a process reward model plus a pairwise preference ranking beats chain-of-thought and full tree traversal on math modeling problems while using far fewer steps, and releases the…","keywords":["BPP-Search","Tree of Thought","Process Reward Model","Pairwise Preference","Beam Search","Mathematical modeling","Operations research","StructuredOR dataset"],"falsifier":"Evaluate BPP-Search against CoT/SC/ToT on every test instance in NL4OPT and MAMO-ComplexLP, including the 146 and 139 problems that full traversal could not solve, and compare correct rate and policy-model call counts on the full sets; if the advantage shrinks or disappears there, the general claim of faster, more accurate retrieval would be refuted.","tokens_in":117,"feed_emoji":"🧮","tokens_out":11061,"duration_ms":219685,"temperature":0.7,"pith_summary":"Turning natural-language optimization problems into LP/MIP models is a realistic but error-prone application of large language models, and this paper targets a specific failure point: deciding which candidate model is correct. The authors propose BPP-Search, which prunes a tree-of-thought search with a process reward model and then resolves final-layer ties with a pairwise preference model, and they report that it beats chain-of-thought, self-consistency, and full tree traversal on three operations-research datasets while using 15-21 reasoning steps instead of 39-40. On the evaluation subset, correct rates reach 93 percent on StructuredOR, 72 percent on MAMO-ComplexLP, and 80 percent on NL4OPT, versus 63 percent, 49 percent, and 57 percent for chain-of-thought. The paper also releases StructuredOR, a dataset whose labels record the complete modeling process, making process-level supervision possible for a domain where existing benchmarks only give objective values.","feed_headline":"Tree search plus pairwise ranking: 93% accuracy in half the steps","feed_subtitle":"A small reward model prunes the reasoning tree and pairwise preference picks the final answer, without retraining the policy model.","key_machinery":"BPP-Search is a tree search over the four modeling layers Q -> SP -> V -> OC, where each node is a partially written mathematical model. A fine-tuned binary process reward model scores each expanded node as $\\sigma(\\ell)$ for the logit $\\ell$ of the correct class; beam search keeps the top $k$ nodes at each layer and prunes the rest. At the final layer, a separately fine-tuned pairwise preference model scores every ordered pair $(A, B)$ as $S(A \\succ B)$, averages these scores over all other candidates to produce one score per candidate, and returns the top candidate as the answer. The PRM handles coarse pruning; the pairwise preference aggregation handles the final selection where correct and incorrect candidates differ only subtly.","core_discovery":"The paper's central claim is that a small, fine-tuned process reward model can do the cheap work of pruning a reasoning tree, and a pairwise preference model can do the delicate work of choosing among near-identical final candidates, so the policy model is not the bottleneck. The empirical claim is an accuracy gain plus a step reduction on the subset of problems that full tree traversal could solve: structured OR modeling accuracy rises from 0.633 with chain-of-thought to 0.933 with BPP-Search on StructuredOR, from 0.486 to 0.722 on MAMO-ComplexLP, and from 0.566 to 0.804 on NL4OPT, with reasoning steps of 15-21 rather than 39-40. This works without fine-tuning the policy model, and the paper argues that the reason is that PRM scores are too coarse at the final layer, which is exactly where pairwise preference aggregation intervenes.","pith_inferences":["Averaged pairwise preference is a portable selection trick: code repair, theorem proving, and planning pipelines that keep several near-tie candidates could use the same aggregation instead of trusting a single verifier score.","The experimental tables cover only the solvable subset of each test set (30/38, 72/211, 143/289), so generalization to the full benchmark is untested; re-running on all examples would be the natural next check.","StructuredOR's annotations could support process-supervised training of the policy model itself, not just the verifier, which may push accuracy beyond what search alone achieves.","The non-monotonic effect of beam width suggests the PRM's scores are miscalibrated; training it with regression or ranking targets rather than binary classification might remove the need for the extra preference model."],"forward_implications":["A 1.5B-parameter reward/preference model can steer a much larger policy model to correct mathematical models, so search accuracy does not depend on fine-tuning the generator.","The reported 15-21 policy-model calls are substantially fewer than the 39-40 required for full tree traversal, while accuracy improves on all three datasets.","StructuredOR's process-level labels fill a gap that previously blocked reinforcement-learning and process-supervision methods for operations research.","Because plain beam search gets worse as width increases, the paper's own ablation implies that PRM scores alone are not enough; the pairwise preference layer is what makes wider search usable."],"supporting_citations":[{"why":"Defines the Tree-of-Thought framework whose leaf-selection problem BPP-Search is designed to solve.","marker":"Yao et al., 2023"},{"why":"Defines chain-of-thought, the primary baseline that BPP-Search must beat.","marker":"Wei et al., 2022"},{"why":"Shows step-by-step process supervision improves reasoning, motivating the process reward model.","marker":"Lightman et al., 2023"},{"why":"Provides the process-versus-outcome feedback comparison that grounds PRM design and labeling.","marker":"Uesato et al., 2022"},{"why":"Supplies the beam search mechanism used to keep only the top-k candidates.","marker":"Lowerre and Reddy, 1976"},{"why":"Supplies the NL4OPT dataset used as one of the three evaluation benchmarks.","marker":"Ramamonjison et al., 2022"},{"why":"Supplies the MAMO-ComplexLP dataset used as another evaluation benchmark.","marker":"Huang et al., 2024"},{"why":"Provides the abstract-model generation framework that StructuredOR extends.","marker":"Xiao et al., 2024"},{"why":"Provides the Qwen2.5-Math checkpoint fine-tuned into the PRM and preference model.","marker":"Yang et al., 2024b"}],"fun_headline_variants":["BPP-Search: Beam + PRM + pairwise ranks cut steps in half","Reinforcement learning tree search boosts OR modeling to 93%","Small reward model prunes tree, pairwise rank picks answer","93% accuracy with 15 steps instead of 40 in OR modeling","Tree-of-thought with beam and PRM: faster, better math models"],"cache_read_input_tokens":17920,"weakest_assumption_plain":"The headline accuracy and efficiency numbers are computed only on the examples where full tree traversal already produced at least one correct answer (30 of 38 on StructuredOR, 72 of 211 on MAMO-ComplexLP, 143 of 289 on NL4OPT), and the comparison assumes those gains carry over to the excluded problems.","fun_headline_variants_meta":{"raw":{"variants":["BPP-Search: Beam + PRM + pairwise ranks cut steps in half","Reinforcement learning tree search boosts OR modeling to 93%","Small reward model prunes tree, pairwise rank picks answer","93% accuracy with 15 steps instead of 40 in OR modeling","Tree-of-thought with beam and PRM: faster, better math models"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000199,"raw_usage":{"total_tokens":1371,"prompt_tokens":942,"completion_tokens":429,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":558,"completion_tokens_details":{"reasoning_tokens":334}},"tokens_in":558,"tokens_out":429,"duration_ms":10484,"temperature":1.0,"reasoning_tokens":334,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T12:08:10.009561+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Evaluate BPP-Search against CoT/SC/ToT on every test instance in NL4OPT and MAMO-ComplexLP, including the 146 and 139 problems that full traversal could not solve, and compare correct rate and policy-model call counts on the full sets; if the advantage shrinks or disappears there, the general claim of faster, more accurate retrieval would be refuted.","supporting_citations":[],"review_version":1}