{"id":"b974381f-7e83-4e6d-8426-745f72105b2d","arxiv_id":"2506.17637","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Step-Opt, a LLaMA-3-8B model fine-tuned on iteratively evolved and stepwise-validated data, reports state-of-the-art accuracy on NL4OPT, MAMO, and IndustryOR.","lead":"Researchers created a system that makes AI models better at turning word problems into mathematical optimization models by generating extra training examples and checking them. The final model beats earlier methods on several operations-research benchmarks, though the authors also corrected benchmark labels by hand.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Unverified manual relabeling of test benchmarks threatens the headline SOTA claim; baseline parity on corrected instances is not established.","rationale":"The reader's weakest assumption pinpoints the same load-bearing issue: the manual relabeling of evaluation benchmarks and the implicit assumption that all baselines were measured on the corrected instances. This is the single most consequential threat to the paper's central empirical claim, because the headline result is a comparison across methods on these benchmarks. If the comparison is unfair, the quantitative contribution (especially the 17.01% improvement on complex problems) cannot be trusted, even though the proposed framework and pipeline are methodologically plausible. The paper provides examples of corrections in Appendix A.2 that look legitimate, but that does not resolve the fairness question. The concrete test I propose would settle it directly: re-run all methods on both the original and corrected label sets. Until then, the appropriate verdict remains CONDITIONAL, as the reader concluded. Secondary concerns such as the absence of error bars and the LLM-based (rather than solver-based) validation are real but less likely to change the qualitative conclusion; they affect confidence in the magnitude of the gains, not the existence of a gain. Therefore, I agree with the reader's verdict and see no need to adjust it.","tokens_in":14352,"tokens_out":3719,"duration_ms":34447,"concrete_test":"Run the released Step-Opt model and each baseline (GPT-4 CoE, ORLM, GPT-4 Reflexion) on both the original uncorrected test instances and the corrected instances described in Sec. 4.1, scoring all outputs with Eq. 1. Report per-benchmark accuracy under each labeling. If Step-Opt's advantage over the strongest baseline is at least as large on the original labels as on the corrected labels, the relabeling concern is resolved. If the margin shrinks or reverses on the original labels, the headline SOTA and the 17.01% improvement are evaluation artifacts.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that Step-Opt achieves state-of-the-art results, with a 17.01% micro-average improvement on complex problems. This claim depends on two conditions: (i) the manual corrections to NL4OPT, MAMO, and IndustryOR test labels (16, 78, and 50 plus 23 removals, Sec. 4.1) reflect true ground truth, and (ii) every baseline in Table 1 was evaluated on the same corrected test set. Neither condition is explicitly established. The paper asterisks only tag-BART as copied from the original paper, implying the other baselines were run by the authors, but it never states that GPT-3.5/GPT-4 prompt-engineering methods and ORLM were re-evaluated after the corrections. If those baselines were scored against the original labels while Step-Opt was scored against the corrected labels, the reported gains—especially the 21.33% absolute lead on MAMO ComplexLP (61.61% vs. 40.28%)—would be inflated by a change in the evaluation set, not by modeling capability. The corrections themselves are plausible (e.g., integer rounding, missing absolute value, missing subtour elimination), but plausibility does not establish that they are the unique objective fixes, nor that applying them consistently favors Step-Opt over baselines. Without the corrected instances released and an explicit statement of re-evaluation for all baselines, the SOTA claim is not verifiable from the paper.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces Step-Opt-Instruct, a framework for synthesizing fine-tuning data for optimization modeling with LLMs. The framework combines iterative problem generation (complexity-evolving and scope-evolving prompts) with a stepwise validation mechanism (description, variable, constraint, and program checkers) to produce a dataset of 4,464 verified examples. The authors fine-tune LLaMA-3-8B and Mistral-7B with LoRA on this data, yielding Step-Opt models that reportedly achieve state-of-the-art accuracy on NL4OPT, MAMO EasyLP/ComplexLP, and IndustryOR. The paper also reports manual correction of benchmark labels (16 in NL4OPT, 78 in MAMO, 50 corrections plus 23 removals in IndustryOR) and provides ablations of the evolving-method components and of the inclusion of the mathematical-model component in training data.","tokens_in":14548,"tokens_out":4605,"duration_ms":42918,"significance":"If the reported results are valid, this is a useful contribution to automated OR modeling: it demonstrates a fully automated data synthesis pipeline that avoids manual post-processing, and it improves open-source 7B/8B models to the point of outperforming GPT-4-based prompting on several benchmarks. The strengths include a clear framework description, release of code and data, and multiple ablations (evolving methods, model vs. program only, comparison with ORLM). The central risk is evaluation integrity: the manual relabeling of test sets by the authors, without independent annotation, and the reliance on LLM-based validation for generated data make the headline SOTA gains difficult to interpret as objective model-capability improvements. The paper also omits an ablation of the validation mechanism in the data-generation pipeline, which is central to its claimed contribution.","major_comments":[{"comment":"The reported state-of-the-art results depend critically on the manual correction of test-set labels (16 in NL4OPT, 78 in MAMO, 50 corrections plus 23 removals in IndustryOR). The paper never explicitly states that every baseline in Table 1 was re-evaluated on the corrected instances; only tag-BART is marked as copied from the original paper. If the prompt-engineering baselines and ORLM were scored on the original labels while Step-Opt was scored on the corrected ones, the reported gains—for example, the 21.33% lead on MAMO ComplexLP and the derived 17.01% micro-average improvement on complex problems—would be inflated by an evaluation-set change rather than by model capability. The authors should release the corrected instances and state clearly that all methods were scored on the same corrected test set, preferably with independent annotation or a detailed per-instance changelog.","section":"§4.1, Table 1"},{"comment":"The stepwise validation mechanism uses LLM-based checkers to assess whether the generated problem description, variables, constraints, and program are mutually consistent and consistent with the problem text. This verifies internal consistency, not correctness against an external solver or ground truth; the program checker only executes the code to detect runtime errors, not modeling errors. Because the same model family (GPT-4) both generates and validates the data, the claim that the pipeline produces 'rigorously verified' data is overstated. This circularity weakens the inference that the observed performance gains come from higher data quality rather than from distributional alignment between the generation/validation model and the downstream fine-tuning objective. The authors should temper this claim and, if possible, incorporate a solver-based verification step (e.g., comparing objective values or feasibility certificates) for at least a sample of generated instances.","section":"§3.3"},{"comment":"The ablation study on evolving methods trains on a randomly sampled 2,000-example subset, whereas the main Step-Opt model uses 4,464 examples. Therefore the ablation numbers in Table 2 are not directly comparable to Table 1, and the measured drop when removing a method could be partially a data-subsampling effect. In addition, the paper never ablates the stepwise validation mechanism itself from the data-generation pipeline: there is no comparison of models trained on data generated with versus without the checkers. Since the validation mechanism is advertised as a core contribution, its absence from the ablations leaves a gap in the evidence for its claimed role in preventing error propagation.","section":"§4.3, Table 2"}],"minor_comments":[{"comment":"The phrase 'complex problem' should be 'complex problems'.","section":"Abstract"},{"comment":"The sentence 'Answers were manually revised when needed' is vague; the subsequent paragraph gives counts, but it would help to state explicitly that all baselines were re-run on the corrected versions (or, if not, to list which were not).","section":"§4.1"},{"comment":"There is a duplicated phrase 'the following: the following:' in the domain-transformation prompt.","section":"Appendix A.4"},{"comment":"The odd accent in 'Érror´' appears to be a typographical artifact; it should be 'Error'.","section":"Appendix A.5"},{"comment":"The phrase 'we randomly sample 2,000 examples without specific methods' is ambiguous; it could mean the sample is not stratified by evolving method, or that no evolving method is applied. Please clarify.","section":"§4.3"},{"comment":"The denominator is written as 'g+ϵ'; if g can be zero or negative, the comparison rule should use |g|+ϵ and should be stated explicitly.","section":"Eq. (1)"}],"recommendation":"major_revision","confidential_remarks":"The paper's evaluation integrity is the central concern. The manual relabeling of all three benchmarks by the authors themselves, without independent annotation, combined with the lack of an explicit statement that all baselines were re-scored on the corrected instances, makes the headline SOTA claim vulnerable to a benchmarking artifact. The authors should be asked to release the corrected test instances and a per-instance diff, and to re-run or clearly document the re-scoring of all baselines. A secondary concern is the absence of an ablation of the validation mechanism in data generation. Otherwise, the framework and empirical findings are of interest to the NLP/OR community."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe thing to know: this is a real contribution to the LLM-for-OR data synthesis line, but the headline SOTA margins rest on a manual re-labeling of test sets that the paper does not fully nail down. The core idea—iterative complexity/scope evolution of seed OR problems plus a stepwise LLM-based validation filter before anything enters the training set—is sensible and clearly described, and the ablations mostly support it. I'd send it to review, but I'd push for transparency fixes before accepting.\n\nWhat's new: they adapt Evol-Instruct to OR modeling by separating complexity-evolving (constraint/objective/parameter) from scope-evolving (domain transform/combination), and they add a four-checker validation (description, variables, constraints, program) that filters generated data before it is used for the next iteration. The 8,400 iterations → 4,464 examples, with ~47% discarded, is concrete evidence the validation is doing something. The comparison of Step-Opt-Instruct vs OR-Instruct on equal 3K data is the right kind of apples-to-apples. The token-matched ablation with vs without the math model is also a fair check of the scaffolding role of the model.\n\nThe soft spots: (1) the manual corrections to NL4OPT (16), MAMO (78), IndustryOR (50 + 23 removals) are reasonable—the appendix examples make sense—but there is no independent annotation, and the paper never states explicitly that every baseline in Table 1 was re-scored on the corrected instances. If GPT-3.5/GPT-4 and ORLM were scored on original labels, the 17.01% complex-problem gain is partly an eval-set artifact. (2) The validation checkers are LLM-based consistency checks; they are not solver-verified. That limits the claim of 'rigorous' data quality. (3) Ablations are on 2,000 examples while the main model uses 4,464; the surviving rates vary, so the ablation drops are not cleanly attributable. (4) No error bars or seed variance anywhere.\n\nThe stress-test note is mostly on target. The relabeling issue is real, but I don't think it sinks the paper—the corrected labels are documented with examples, and the method's value does not hinge on the exact SOTA margin. Still, the authors should release the corrected benchmarks and either re-run baselines on them or state clearly which numbers used which labels.\n\nWho this is for: anyone working on synthetic data generation for optimization modeling or on LLM fine-tuning for structured tasks. It deserves a serious referee; with those transparency fixes, I'd be happy to see it published.","headline":"Real contribution to LLM-for-OR data synthesis, but the headline SOTA margins rest on manually relabeled test sets that need clearer baseline alignment.","tokens_in":15131,"tokens_out":2436,"would_cite":true,"duration_ms":23935,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A fine-tuned 8B model trained on validated synthetic data beats GPT-4 on complex optimization-modeling benchmarks.","keywords":["optimization modeling","operations research","large language models","synthetic data generation","iterative problem generation","stepwise validation","fine-tuning","mathematical programming"],"falsifier":"Re-run every baseline (ORLM, GPT-4 CoE, GPT-4 Reflexion, and the rest) on the exact corrected instances used for the reported numbers and compare against scores on the original uncorrected benchmarks; if the corrections move ORLM or GPT-4 accuracy by more than the claimed gaps, the state-of-the-art claim is an artifact of relabeling. Also train LLaMA-3-8B on an equal-sized unfiltered subset of the same generated data; if accuracy on MAMO ComplexLP stays near 61.61% without the stepwise validation filter, the filtering mechanism is not the cause of the gain.","tokens_in":14087,"feed_emoji":"🧮","tokens_out":7932,"duration_ms":72331,"temperature":0.7,"pith_summary":"The paper argues that the bottleneck in LLM-based optimization modeling is not model size but the quality control of synthetic training data. It introduces Step-Opt-Instruct, which grows a small seed set of 260 operations-research problems into 4,464 training examples by iteratively making problems harder or shifting them to new domains, and by running each candidate through a four-stage validation pipeline before accepting it. Fine-tuning LLaMA-3-8B and Mistral-7B on this filtered data yields Step-Opt, reported as state-of-the-art on NL4OPT, MAMO, and IndustryOR, with a 17.01% micro-average improvement on the most difficult problems over the strongest baselines. A sympathetic reader would take the central claim to be that structured validation plus gradual problem refinement can make small open-source models competitive with, or better than, much larger proprietary models on complex OR tasks.","feed_headline":"A fine-tuned 8B model beats GPT-4 on hard optimization tasks","feed_subtitle":"Iterative problem generation plus stepwise validation lifts accuracy by 17.01% on difficult operations-research problems.","key_machinery":"The load-bearing mechanism is the Step-Opt-Instruct data-generation and filtering loop. Iterative Problem Generation uses five prompt-based evolvers: constraint modification, objective alteration, and parameter adjustment raise difficulty one small step at a time while preserving the seed problem's logical structure, while domain transformation and combination add breadth by moving a problem to a new application area or merging two problems. Stepwise Validation then runs four LLM-based checkers—description completeness, variable definition, constraint correctness (including dedicated Big-M and K-way selection checks), and program execution—and feeds identified errors back to the generator for revision, discarding any item that fails after the retry limit. The paper claims this loop is what keeps the synthetic dataset free of inherited errors, and that this data quality is what allows a 7B-to-8B model to surpass much larger prompt-driven systems on complex problems.","core_discovery":"On the paper's own terms, the central discovery is that carefully filtered synthetic data, not prompt engineering, is what unlocks complex optimization modeling in open-source LLMs. Step-Opt-Instruct combines iterative problem generation—Complexity-Evolving (constraint modification, objective alteration, parameter adjustment) and Scope-Evolving (domain transformation, problem combination)—with a stepwise validation mechanism of four checkers covering description completeness, variable definitions, constraint implementation (with special prompts for Big-M and K-way selection), and executable program output. Only problems that pass every check enter the training set, preventing errors from propagating through later iterations. The resulting Step-Opt-LLaMA-3-8B reaches 84.49% on NL4OPT, 85.28% on MAMO EasyLP, 61.61% on MAMO ComplexLP, and 36.36% on IndustryOR, which the paper reports as state-of-the-art, with a 17.01% micro-average gain over baselines on complex problems. The paper also reports ablations showing that each evolving method contributes, that discarding the mathematical model from training data hurts even when token counts are equalized, and that Step-Opt-Instruct outperforms ORLM's OR-Instruct data when both are used to fine-tune the same backbone with the same number of examples.","pith_inferences":["We infer that the same generate-check-retry-discard pattern could transfer to other structured-generation tasks such as SQL, formal proofs, or scheduling rules, where synthetic-data errors compound across iterations and the checker set would need to be re-specified per domain.","The reported manual benchmark corrections (16 in NL4OPT, 78 in MAMO, 50 corrections plus 23 removals in IndustryOR) change the evaluation set, so the state-of-the-art comparison is only meaningful if all baselines were re-run on the corrected instances; an independent re-scoring would be the decisive test.","Because 46.86% of generated samples were discarded by validation, the improvement could come from filtering, from the extra generation attempts, or both; training on an unfiltered equal-size sample would isolate the validation mechanism's causal contribution.","The authors' stated limitations—uncovered OR modeling techniques and untested other fields—suggest the checkers, not the generators, are the current ceiling; adding checkers for new technique families is the natural way to broaden applicability."],"forward_implications":["Because generation and validation are automated, the pipeline removes the need for manual post-processing of training data, so scaling to more OR problem types should mainly cost API compute rather than expert labor.","If the benchmark results hold, organizations with sensitive operations data can fine-tune small local models instead of sending problem descriptions to proprietary APIs.","The ablation results imply that the composition of the synthetic data matters more than its raw size: dropping any one evolution method degrades performance, with domain transformation the most broadly damaging.","Keeping the mathematical model as an intermediate artifact in training data improves the generated solver code even when token counts are held fixed, indicating that the model is acting as a structured reasoning bridge.","The largest gains appear precisely on the hardest instances, so the method's value is concentrated where prompt-based methods currently fail."],"supporting_citations":[{"why":"ORLM supplies the fine-tuned baseline, the OR-Instruct comparison data, and the IndustryOR benchmark that Step-Opt must beat.","marker":"Tang et al., 2024"},{"why":"NL4OPT is one of the three evaluation benchmarks; 16 instances were manually corrected in this paper.","marker":"Ramamonjison et al., 2023"},{"why":"MAMO provides the EasyLP and ComplexLP benchmarks whose complex split produces the headline 17.01% gain.","marker":"Huang et al., 2024"},{"why":"Evol-Instruct is the methodological precedent for iterative breadth/depth evolution that Step-Opt-Instruct adapts to OR.","marker":"Xu et al., 2024"},{"why":"Chain-of-Experts is the strongest prompt-engineering baseline on MAMO ComplexLP that Step-Opt surpasses.","marker":"Xiao et al., 2023"},{"why":"The COPT solver is used to execute generated programs and compute the objective values that determine correctness.","marker":"Ge et al., 2022"},{"why":"LLaMA-3-8B is the backbone model for the best-performing Step-Opt variant.","marker":"AI@Meta, 2024"},{"why":"Mistral-7B is the second backbone used to show the framework transfers across open-source base models.","marker":"Jiang et al., 2023"}],"fun_headline_variants":["Step-Opt: 8B model beats GPT-4 on tough optimization tasks","Iterative synthetic data + stepwise checks lift OR accuracy 17%","Validated synthetic data powers 8B LLaMA to SOTA on NL4OPT","Self-improving data generation helps small LLMs win on OR"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The reported results assume that the manual corrections to the benchmark labels—16 in NL4OPT, 78 in MAMO, and 50 corrections plus 23 removals in IndustryOR—are genuine fixes to ground truth rather than the authors' own modeling conventions, and that every baseline was scored under the same corrected set.","fun_headline_variants_meta":{"raw":{"variants":["Step-Opt: 8B model beats GPT-4 on tough optimization tasks","Iterative synthetic data + stepwise checks lift OR accuracy 17%","Validated synthetic data powers 8B LLaMA to SOTA on NL4OPT","Self-improving data generation helps small LLMs win on OR"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001196,"raw_usage":{"total_tokens":4975,"prompt_tokens":1034,"completion_tokens":3941,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":650,"completion_tokens_details":{"reasoning_tokens":3858}},"tokens_in":650,"tokens_out":3941,"duration_ms":27343,"temperature":1.0,"reasoning_tokens":3858,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T19:04:38.533741+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run every baseline (ORLM, GPT-4 CoE, GPT-4 Reflexion, and the rest) on the exact corrected instances used for the reported numbers and compare against scores on the original uncorrected benchmarks; if the corrections move ORLM or GPT-4 accuracy by more than the claimed gaps, the state-of-the-art claim is an artifact of relabeling. Also train LLaMA-3-8B on an equal-sized unfiltered subset of the same generated data; if accuracy on MAMO ComplexLP stays near 61.61% without the stepwise validation filter, the filtering mechanism is not the cause of the gain.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"NL4OPT is one of the three evaluation benchmarks; 16 instances were manually corrected in this paper."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Chain-of-Experts is the strongest prompt-engineering baseline on MAMO ComplexLP that Step-Opt surpasses."}],"review_version":1}