{"id":"3dd36b7a-2395-443b-a62a-76b601a65116","arxiv_id":"2412.17874","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"ORQA is a new 1,513-question multiple-choice benchmark showing that open-source LLMs score up to 77% on operations research modeling questions, well below a 93% expert baseline.","lead":"The authors introduce ORQA, a set of 1,513 expert-written multiple-choice questions that test whether large language models can turn real-world operations research problems into mathematical optimization models. Open-source LLMs score far below OR experts, with the best model reaching 77% accuracy versus 93% for a human expert, showing a clear gap in specialized technical reasoning.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Gold-answer reliability is the unverified load-bearing assumption: the reported LLM-human gap (0.772 vs 0.93) depends on expert labeling that was never checked for inter-annotator agreement or external audit.","rationale":"The reader identified gold-answer correctness as the weakest assumption, and this is indeed the single most load-bearing concern for the paper's central claim. The entire benchmark value proposition rests on the gold labels being correct and unambiguous; without that, the reported LLM accuracies and the comparison to a human baseline are not interpretable. The paper describes a two-expert verification process but reports no reliability statistics, and the step-2 handling of multiple modeling approaches is a subjective guarantee rather than a demonstrated property. My proposed test (independent external re-annotation of a random sample) directly settles whether the concern lands: if external experts agree with the gold labels at a high rate, the benchmark is likely sound; if not, the reported numbers are unreliable. This does not change the reader's CONDITIONAL verdict — the concern is real but addressable, and the benchmark may still be a useful contribution if the test passes. I agree with the reader's identification and see no other assumption that is more load-bearing: model evaluation details (single runs, prompt engineering) affect result precision but not the validity of the benchmark itself, and the claim that OR is niche and reduces contamination is plausible but secondary. Therefore the verdict remains CONDITIONAL, unchanged from the reader's assessment.","tokens_in":14505,"tokens_out":2964,"duration_ms":25710,"concrete_test":"Sample 150 ORQA test instances (about 10% of the test set) and have two external OR experts, who are not authors and have no access to gold labels, independently answer them using the same question taxonomy and option format. Compute Cohen's kappa between the two external experts and their agreement rate with the published gold labels. If external-expert agreement with gold labels is below 95% or pairwise kappa is below 0.8, the benchmark's gold-answer reliability is not established; the reported accuracies and the LLM-human gap would need to be re-estimated on the subset of instances with consensus labels.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim of ORQA as a trustworthy benchmark hinges on the correctness and unambiguity of its gold answers. The dataset creation process (Section 'Dataset creation', Steps 2 and 3) has two experts create each question and two different experts verify it, but the verification is split: one expert checks completeness and ambiguity, the other checks correctness and multi-step reasoning. No inter-annotator reliability metric (e.g., Cohen's kappa) is reported, and no independent audit is described. Step 2 states that 'for cases where multiple modeling approaches are possible, OR experts ensured that incorrect options were truly incorrect considering all different models,' but this is a subjective judgment with no validation. If even a small fraction (say 5–10%) of the 1,468 test gold labels are arguable or wrong under a legitimate alternative model formulation, then reported accuracies (best 0.772 vs human 0.93 on 100 instances) become noisy or systematically biased: an LLM choosing a 'wrong' option might actually pick a valid modeling choice, deflating LLM scores, while the single-expert human baseline could be inflated by familiarity with the intended model. The paper's own reasoning-error analysis (Section 'Results and Discussion') shows that models sometimes reach correct answers via flawed reasoning (20% in 0-shot CoT), so answer extraction is not the bottleneck; label quality is. Without independent evidence of label reliability, the benchmark's central quantitative claims are not firmly grounded.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces ORQA, a multiple-choice question-answering benchmark for operations research (OR) modeling, comprising 1,513 expert-created instances across 20 application domains. Each instance provides a natural-language optimization problem description, a question about the underlying mathematical model (e.g., objective, constraints, decision variables), four options, and a gold answer verified by OR experts. The authors evaluate a range of open-source LLMs (Llama 2/3/3.1, Mistral/Mixtral, DeepSeek, Falcon, FLAN-T5, NuminaMath) under standard and chain-of-thought prompting with zero-, one-, and three-shot in-context learning. The main empirical findings are that the best model (Llama3.1-405B-Instruct, 3-shot standard) reaches 0.772 accuracy, well below a preliminary expert baseline of 0.93 on a 100-instance sample, and that CoT prompting generally reduces accuracy. The dataset and evaluation code are released publicly.","tokens_in":14784,"tokens_out":6833,"duration_ms":61855,"significance":"If the benchmark is reliable, ORQA is a valuable new resource for studying LLM reasoning in a specialized technical domain. The paper makes a concrete contribution by focusing on optimization-model component identification, a task that requires multistep reasoning and is under-represented in existing QA benchmarks. The evaluation is broad, covering many open-source model families and sizes, and the authors make a reasonable choice to exclude closed-source APIs for reproducibility. The finding that open LLMs score well below expert-level accuracy is consistent across models and prompting settings. The public release of the dataset and evaluation code is a strength, as is the inclusion of expert-written reasoning steps in the validation set. However, the benchmark's trustworthiness depends on the reliability of the gold answers and the human baseline, both of which currently lack quantitative validation.","major_comments":[{"comment":"The correctness of the 1,468 test answers is the load-bearing assumption for every accuracy number in Table 2, yet no inter-annotator agreement or independent audit is reported. Step 2 states that for cases with multiple modeling approaches \"OR experts ensured that incorrect options were truly incorrect considering all different models,\" but this is an untested subjective judgment. Although Step 3 assigns two experts per instance, they check different aspects (one checks completeness/ambiguity, the other checks reasoning and correctness), so there is no independent double-labeling of the final answer. A small fraction of arguable or incorrect gold labels could systematically shift the reported accuracies and the human-LLM gap. Please add a label-reliability study: have two or more independent OR experts re-answer a random sample of at least 100-200 instances, report agreement (e.g., Cohen's kappa or percentage agreement), and publicly document or resolve ambiguous cases.","section":"Dataset creation, Steps 2-3"},{"comment":"The human baseline is a single expert on a random set of 100 instances, achieving 93% accuracy. This is a very thin basis for the headline claim of a gap between LLMs and human experts. The 95% confidence interval for 93/100 is roughly 86-97%, and no information is given on whether the expert was blind to the gold answers or had any role in dataset creation. Please expand the human evaluation to multiple OR experts (ideally blind to the labels), report the number of experts and their agreement, and provide confidence intervals. Without this, the 0.93 baseline should be described as an anecdotal estimate rather than a reliable reference point.","section":"Experiment Setup for Evaluation (human baseline)"},{"comment":"The few-shot prompts are created by \"randomly sampling instances with the same question type from the validation split,\" but no random seeds, number of runs, or variance is reported. The validation set contains only 45 instances across 11 question types, so the pool for same-type sampling is tiny (roughly 4 per type), and 3-shot settings likely reuse examples across test items. Table 5 shows that ICL example selection can swing accuracy by about 6 points on one model (0.300 vs 0.362), yet Table 2 reports single-run few-shot numbers to three decimal places. Please run each few-shot configuration with at least 3-5 seeds and report mean±standard deviation (or the full range), especially for the 3-shot standard condition that produces the headline number of 0.772.","section":"Prompting strategies and Table 2"},{"comment":"The dramatic drop in 1-shot CoT accuracy relative to 0-shot CoT (e.g., Llama3.1-405B-Instruct: 0.360 vs 0.695) is never explained, and it runs counter to the paper's general observation that ICL helps standard prompting. The text attributes the CoT failures to models ignoring instructions, hallucinating options, and generating incorrect reasoning, but this is supported only by a few illustrative examples (Figures 9-10), not by a systematic error analysis. Because the 1-shot CoT results use a single random ICL example, the drop could be an artifact of the sampled prompt or of the two-step protocol rather than a property of CoT. Please add a controlled comparison (e.g., identical ICL examples in standard and CoT, multiple seeds, and a quantitative breakdown of error types on the full test set) before concluding that \"ICL examples benefit standard but not CoT prompting.\"","section":"Results and Discussion, Table 2 (1-shot CoT column)"}],"minor_comments":[{"comment":"The standard deviations for trigger prompts are reported to three decimal places and are extremely small (e.g., 0.001) despite the stated temperature of 0.7; this seems inconsistent with sampling from a stochastic decoder. Please clarify the decoding settings (e.g., top-p, temperature implementation, or whether sampling was effectively greedy) or re-examine the variance computation.","section":"Table 3"},{"comment":"The bullet \"Model size contributes to reasoning performance\" is immediately qualified by exceptions (Mistral-7B and FLAN-T5 outperform Llama2-13B); the wording is contradictory. Suggest rephrasing to \"model size generally helps within a model family, but other architectural/training factors also matter.\"","section":"Results and Discussion, first bullet"},{"comment":"The terms \"multi-choice\" and \"multiple-choice\" are used interchangeably; please standardize on one form.","section":"Throughout"},{"comment":"The paper does not specify the decoding temperature, top-p, max token length, or number of repeated runs for the main experiments in Table 2. Please state these for reproducibility.","section":"Experiment Setup for Evaluation"}],"recommendation":"major_revision","confidential_remarks":"The core resource—an expert-built OR QA benchmark with a broad open-source LLM evaluation—is a useful contribution, and the main qualitative finding (open LLMs fall well short of expert performance) is likely robust despite the issues above. However, the paper as currently written makes quantitative claims (0.772 vs 0.93, CoT harms performance) that rest on unvalidated gold labels, a single-human baseline, and single-run few-shot evaluations. These are fixable with additional experiments and reliability analyses. I also note that the validation set is very small for in-context example sampling, which amplifies variance; the authors should consider enlarging it or using a more controlled ICL selection protocol. The paper fits the audience of an NLP/ML venue; the release of the dataset and code is a plus."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nBottom line: ORQA is the first expert-curated multiple-choice QA benchmark for operations research, and it looks like a solid contribution. The dataset construction is careful, the evaluation covers many open LLMs, and the main result — open models sit around 0.6–0.77 accuracy while a human expert gets 0.93 — is consistent across families, sizes, and prompting styles. I'd send this to review.\n\nWhat's new: unlike NLP4LP or ComplexOR, ORQA uses MCQ format, so evaluation doesn't depend on code generation or solver output. At 1,513 instances, it's much larger than those predecessors, and the descriptions are context-aware and jargon-free. Eleven question types mapped to three skills gives a useful breakdown for error analysis. Releasing the data and code is the right move.\n\nWhere it's soft: the gold answers are the load-bearing part. Two experts verify each instance, but there's no inter-annotator agreement number and no external audit. In OR, multiple model formulations are often defensible, so a small percentage of disputed labels is plausible. That said, even if 5–10% of labels were arguable, the broad finding would likely survive because the gap is large and even the easiest reading-comprehension questions don't hit ceiling. Still, a kappa or a second-expert sample would remove the doubt. Also, the human baseline is one expert on 100 instances — fine as preliminary, but it should be labeled as such, and it is. Few-shot results come from randomly sampled ICL examples without variance; given their own Table 5 showing sensitivity to ICL selection, I'd want seed-averaged numbers. Minor issue: the 1-shot CoT numbers for Llama3.1 models are implausibly low (0.292, 0.360), and the explanation (models ignore instructions) is plausible but worth a check.\n\nWho this is for: anyone working on LLM reasoning evaluation, and the OR+NLP community. The paper deserves a serious referee. In revision, I'd ask for label-reliability stats and variance reporting, not a redesign.","headline":"ORQA is a genuinely new expert-curated MCQ benchmark for operations research with a consistent LLM-vs-expert gap; missing label-reliability stats and a one-person human baseline are the main soft spots, but the central finding holds up.","tokens_in":15362,"tokens_out":2163,"would_cite":true,"duration_ms":20488,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper introduces ORQA, a 1,513-question expert-built benchmark showing that open-source LLMs still lag human experts on operations-research modeling questions.","keywords":["operations research","multiple-choice question answering","LLM benchmarking","optimization modeling","chain-of-thought prompting","reasoning evaluation","expert-verified dataset","generalization"],"falsifier":"Have a second, independent team of operations-research experts answer a random sample of 100 to 200 test questions blind; if their agreement with the gold answers is far below the 93 percent reported for the first expert, the benchmark's accuracy numbers rest on disputed labels rather than model skill.","tokens_in":14344,"feed_emoji":"🧮","tokens_out":6950,"duration_ms":51664,"temperature":0.7,"pith_summary":"The paper claims that large language models cannot yet generalize to operations research, a technical domain built on multi-step optimization modeling. To test that, it builds ORQA, a benchmark of 1,513 expert-verified multiple-choice questions drawn from 20 real-world application domains; each question forces a model to identify components of a mathematical optimization model, such as objectives, variables, constraints, data, and their relationships, from a plain-language story. Across open-source models and prompting strategies, the best accuracy is 77.2 percent, versus 93 percent for a human expert on a 100-question sample, and chain-of-thought prompting usually makes scores worse. If the benchmark's labels are right, the result matters because it quantifies how far general-purpose LLMs are from automating an expert task that is underrepresented in training data.","feed_headline":"Open LLMs top out at 77% on expert OR modeling quiz","feed_subtitle":"Best open-source model scores 77% on a new OR quiz, below a 93% human baseline; chain-of-thought often hurts.","key_machinery":"The load-bearing object is ORQA itself: 1,513 instances, each with a context averaging 231 words, a question, four options, and a verified target answer, split into 1,468 test and 45 validation instances across 20 domains. The questions are organized into 11 types grouped into three skills: understanding problem specifications, identifying model entities such as sets, parameters, variables, and constraints, and identifying relationships among those entities. This design separates ORQA from earlier optimization datasets because evaluation is a single multiple-choice answer rather than running a solver on generated model code, so it isolates formulation reasoning from coding errors and from solver behavior.","core_discovery":"ORQA asks a model to read a plain-language description of a real optimization problem and choose the right answer about the problem's model components, its objective, variables, constraints, sets, or parameters, from four expert-written options. On 1,468 test questions, the best open-source model reached 77.2 percent accuracy with three-shot standard prompting, while one OR expert scored 93 percent on a random 100-question sample. The paper also reports that chain-of-thought prompting generally lowers accuracy, that larger models within the same family tend to do better, and that questions requiring operations-research knowledge rather than pure reading comprehension are the main failure point.","pith_inferences":["A consequence the paper leaves implicit is that some reported errors may be models choosing a valid alternative optimization model that the fixed gold answer does not accept; a future version of the benchmark could allow multiple correct answers and re-measure the gap.","The sharp chain-of-thought drop suggests ORQA could serve as a stress test for reasoning-faithfulness methods: a technique that reduces this drop while preserving accuracy would be a meaningful signal.","Because the benchmark defines 11 question types, it could support a per-model diagnostic profile of which modeling skill fails first, but the paper only reports aggregated heatmaps; that finer-grained analysis is a natural next step.","The human baseline rests on one expert answering 100 instances, so recruiting several experts and reporting inter-annotator agreement would turn the headline human-versus-LLM gap into a stable number."],"forward_implications":["On the questions each saw, the reported gap between the best open-source model (77.2 percent on 1,468 test items) and a human expert (93 percent on a 100-item sample) is roughly 16 points, though the sample sizes differ and the comparison is preliminary.","Chain-of-thought prompting is not a reliable way to improve performance on this benchmark; in most evaluated configurations it lowers accuracy, so better reasoning-elicitation techniques are needed.","Model size helps within a family: for Llama 3.1, accuracy rises with scale from an 8B to a 405B model under both standard and chain-of-thought prompting.","Questions that require operations-research knowledge or model-building knowledge are the main failure point, while questions answerable by reading comprehension are handled more fairly.","Few-shot in-context examples improve standard prompting, and for the tested model the length of the example affects accuracy more than matching the question type."],"supporting_citations":[{"why":"Prior end-to-end optimization benchmark whose coding-error findings motivate ORQA's solver-free multiple-choice design.","marker":"AhmadiTeshnizi, Gao, and Udell 2024"},{"why":"Earlier work on auto-formulating optimization models from natural language, the line of work ORQA extends by testing component identification at scale.","marker":"Ramamonjison et al. 2022a"},{"why":"Prior small ComplexOR dataset of 37 problems; ORQA's 1,513-instance scope and diversity are positioned against it.","marker":"Xiao et al. 2023"},{"why":"Chain-of-thought prompting method that the paper evaluates and finds generally unhelpful on ORQA.","marker":"Wei et al. 2022"},{"why":"Zero-shot chain-of-thought technique used in the two-step CoT evaluation.","marker":"Kojima et al. 2022"},{"why":"Mistral model family and its sliding-window attention, cited to explain why a 7B model can beat larger Llama2 models.","marker":"Jiang et al. 2023"},{"why":"Supports the claim that scarce long-tail training data makes operations research a hard generalization test for LLMs.","marker":"Kandpal et al. 2023"},{"why":"Supplies the symbol-binding evaluation protocol ORQA uses to score generated answers.","marker":"Robinson and Wingate 2023"}],"fun_headline_variants":["LLMs stumble on OR quiz: best open model hits 77%","Open LLMs fail to match OR experts: 77% vs 93%","ORQA benchmark exposes LLM gaps in optimization reasoning","Chain-of-thought backfires on new OR reasoning test","OR experts beat best open LLM by 16 points on new quiz"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The benchmark's gold answers are correct and unambiguous: each instance was checked by two experts, but no inter-annotator agreement or external audit is reported.","fun_headline_variants_meta":{"raw":{"variants":["LLMs stumble on OR quiz: best open model hits 77%","Open LLMs fail to match OR experts: 77% vs 93%","ORQA benchmark exposes LLM gaps in optimization reasoning","Chain-of-thought backfires on new OR reasoning test","OR experts beat best open LLM by 16 points on new quiz"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001345,"raw_usage":{"total_tokens":5406,"prompt_tokens":827,"completion_tokens":4579,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":443,"completion_tokens_details":{"reasoning_tokens":4488}},"tokens_in":443,"tokens_out":4579,"duration_ms":31882,"temperature":1.0,"reasoning_tokens":4488,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T05:57:21.682684+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Have a second, independent team of operations-research experts answer a random sample of 100 to 200 test questions blind; if their agreement with the gold answers is far below the 93 percent reported for the first expert, the benchmark's accuracy numbers rest on disputed labels rather than model skill.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Prior end-to-end optimization benchmark whose coding-error findings motivate ORQA's solver-free multiple-choice design."},{"cited_title":"J.; Han, X.; Fu, X.; Zhong, T.; Zeng, J.; Song, M.; et al","cited_arxiv_id":null,"evidence_quote":"Prior small ComplexOR dataset of 37 problems; ORQA's 1,513-instance scope and diversity are positioned against it."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the symbol-binding evaluation protocol ORQA uses to score generated answers."}],"review_version":1}