Pith. sign in

REVIEW 4 major objections 4 minor 1 cited by

Evaluating LLM Reasoning in the Operations Research Domain with ORQA

T0 review · 4 major / 4 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper introduces ORQA, a 1,513-question expert-built benchmark showing that open-source LLMs still lag human experts on operations-research modeling questions.

desk verdict ORQA is a genuinely new expert-curated MCQ benchmark for operations research with a consistent LLM-vs-expert gap; missing label-reliability stats and a one-person human baseline are the main soft spots, but the central finding holds up. read the letter →

arxiv 2412.17874 v2 pith:OKMVPIPE submitted 2024-12-22 cs.CL cs.AI

Mahdi Mostajabdaveh , Timothy T. Yu , Samarendra Chandan Bindu Dash , Rindranirina Ramamonjison , Jabo Serge Byusa , Giuseppe Carenini , Zirui Zhou , Yong Zhang This is my paper · ORCID
classification cs.CLcs.AI
keywords operationsresearchmultiple-choicequestionansweringLLMbenchmarkingoptimizationmodelingchain-of-thoughtpromptingreasoningevaluationexpert-verifieddatasetgeneralization
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that large language models cannot yet generalize to operations research, a technical domain built on multi-step optimization modeling. To test that, it builds ORQA, a benchmark of 1,513 expert-verified multiple-choice questions drawn from 20 real-world application domains; each question forces a model to identify components of a mathematical optimization model, such as objectives, variables, constraints, data, and their relationships, from a plain-language story. Across open-source models and prompting strategies, the best accuracy is 77.2 percent, versus 93 percent for a human expert on a 100-question sample, and chain-of-thought prompting usually makes scores worse. If the benchmark's labels are right, the result matters because it quantifies how far general-purpose LLMs are from automating an expert task that is underrepresented in training data.

What carries the argument

The load-bearing object is ORQA itself: 1,513 instances, each with a context averaging 231 words, a question, four options, and a verified target answer, split into 1,468 test and 45 validation instances across 20 domains. The questions are organized into 11 types grouped into three skills: understanding problem specifications, identifying model entities such as sets, parameters, variables, and constraints, and identifying relationships among those entities. This design separates ORQA from earlier optimization datasets because evaluation is a single multiple-choice answer rather than running a solver on generated model code, so it isolates formulation reasoning from coding errors and from solver behavior.

What would settle it

Have a second, independent team of operations-research experts answer a random sample of 100 to 200 test questions blind; if their agreement with the gold answers is far below the 93 percent reported for the first expert, the benchmark's accuracy numbers rest on disputed labels rather than model skill.

Watch

Extended reading notes

Core claim

ORQA asks a model to read a plain-language description of a real optimization problem and choose the right answer about the problem's model components, its objective, variables, constraints, sets, or parameters, from four expert-written options. On 1,468 test questions, the best open-source model reached 77.2 percent accuracy with three-shot standard prompting, while one OR expert scored 93 percent on a random 100-question sample. The paper also reports that chain-of-thought prompting generally lowers accuracy, that larger models within the same family tend to do better, and that questions requiring operations-research knowledge rather than pure reading comprehension are the main failure point.

Load-bearing premise

The benchmark's gold answers are correct and unambiguous: each instance was checked by two experts, but no inter-annotator agreement or external audit is reported.

Editorial extensions

If this is right

  • On the questions each saw, the reported gap between the best open-source model (77.2 percent on 1,468 test items) and a human expert (93 percent on a 100-item sample) is roughly 16 points, though the sample sizes differ and the comparison is preliminary.
  • Chain-of-thought prompting is not a reliable way to improve performance on this benchmark; in most evaluated configurations it lowers accuracy, so better reasoning-elicitation techniques are needed.
  • Model size helps within a family: for Llama 3.1, accuracy rises with scale from an 8B to a 405B model under both standard and chain-of-thought prompting.
  • Questions that require operations-research knowledge or model-building knowledge are the main failure point, while questions answerable by reading comprehension are handled more fairly.
  • Few-shot in-context examples improve standard prompting, and for the tested model the length of the example affects accuracy more than matching the question type.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A consequence the paper leaves implicit is that some reported errors may be models choosing a valid alternative optimization model that the fixed gold answer does not accept; a future version of the benchmark could allow multiple correct answers and re-measure the gap.
  • The sharp chain-of-thought drop suggests ORQA could serve as a stress test for reasoning-faithfulness methods: a technique that reduces this drop while preserving accuracy would be a meaningful signal.
  • Because the benchmark defines 11 question types, it could support a per-model diagnostic profile of which modeling skill fails first, but the paper only reports aggregated heatmaps; that finer-grained analysis is a natural next step.
  • The human baseline rests on one expert answering 100 instances, so recruiting several experts and reporting inter-annotator agreement would turn the headline human-versus-LLM gap into a stable number.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper introduces ORQA, a multiple-choice question-answering benchmark for operations research (OR) modeling, comprising 1,513 expert-created instances across 20 application domains. Each instance provides a natural-language optimization problem description, a question about the underlying mathematical model (e.g., objective, constraints, decision variables), four options, and a gold answer verified by OR experts. The authors evaluate a range of open-source LLMs (Llama 2/3/3.1, Mistral/Mixtral, DeepSeek, Falcon, FLAN-T5, NuminaMath) under standard and chain-of-thought prompting with zero-, one-, and three-shot in-context learning. The main empirical findings are that the best model (Llama3.1-405B-Instruct, 3-shot standard) reaches 0.772 accuracy, well below a preliminary expert baseline of 0.93 on a 100-instance sample, and that CoT prompting generally reduces accuracy. The dataset and evaluation code are released publicly.

Significance. If the benchmark is reliable, ORQA is a valuable new resource for studying LLM reasoning in a specialized technical domain. The paper makes a concrete contribution by focusing on optimization-model component identification, a task that requires multistep reasoning and is under-represented in existing QA benchmarks. The evaluation is broad, covering many open-source model families and sizes, and the authors make a reasonable choice to exclude closed-source APIs for reproducibility. The finding that open LLMs score well below expert-level accuracy is consistent across models and prompting settings. The public release of the dataset and evaluation code is a strength, as is the inclusion of expert-written reasoning steps in the validation set. However, the benchmark's trustworthiness depends on the reliability of the gold answers and the human baseline, both of which currently lack quantitative validation.

major comments (4)
  1. [Dataset creation, Steps 2-3] The correctness of the 1,468 test answers is the load-bearing assumption for every accuracy number in Table 2, yet no inter-annotator agreement or independent audit is reported. Step 2 states that for cases with multiple modeling approaches "OR experts ensured that incorrect options were truly incorrect considering all different models," but this is an untested subjective judgment. Although Step 3 assigns two experts per instance, they check different aspects (one checks completeness/ambiguity, the other checks reasoning and correctness), so there is no independent double-labeling of the final answer. A small fraction of arguable or incorrect gold labels could systematically shift the reported accuracies and the human-LLM gap. Please add a label-reliability study: have two or more independent OR experts re-answer a random sample of at least 100-200 instances, report agreement (e.g., Cohen's kappa or percentage agreement), and publicly document or resolve ambiguous cases.
  2. [Experiment Setup for Evaluation (human baseline)] The human baseline is a single expert on a random set of 100 instances, achieving 93% accuracy. This is a very thin basis for the headline claim of a gap between LLMs and human experts. The 95% confidence interval for 93/100 is roughly 86-97%, and no information is given on whether the expert was blind to the gold answers or had any role in dataset creation. Please expand the human evaluation to multiple OR experts (ideally blind to the labels), report the number of experts and their agreement, and provide confidence intervals. Without this, the 0.93 baseline should be described as an anecdotal estimate rather than a reliable reference point.
  3. [Prompting strategies and Table 2] The few-shot prompts are created by "randomly sampling instances with the same question type from the validation split," but no random seeds, number of runs, or variance is reported. The validation set contains only 45 instances across 11 question types, so the pool for same-type sampling is tiny (roughly 4 per type), and 3-shot settings likely reuse examples across test items. Table 5 shows that ICL example selection can swing accuracy by about 6 points on one model (0.300 vs 0.362), yet Table 2 reports single-run few-shot numbers to three decimal places. Please run each few-shot configuration with at least 3-5 seeds and report mean±standard deviation (or the full range), especially for the 3-shot standard condition that produces the headline number of 0.772.
  4. [Results and Discussion, Table 2 (1-shot CoT column)] The dramatic drop in 1-shot CoT accuracy relative to 0-shot CoT (e.g., Llama3.1-405B-Instruct: 0.360 vs 0.695) is never explained, and it runs counter to the paper's general observation that ICL helps standard prompting. The text attributes the CoT failures to models ignoring instructions, hallucinating options, and generating incorrect reasoning, but this is supported only by a few illustrative examples (Figures 9-10), not by a systematic error analysis. Because the 1-shot CoT results use a single random ICL example, the drop could be an artifact of the sampled prompt or of the two-step protocol rather than a property of CoT. Please add a controlled comparison (e.g., identical ICL examples in standard and CoT, multiple seeds, and a quantitative breakdown of error types on the full test set) before concluding that "ICL examples benefit standard but not CoT prompting."
minor comments (4)
  1. [Table 3] The standard deviations for trigger prompts are reported to three decimal places and are extremely small (e.g., 0.001) despite the stated temperature of 0.7; this seems inconsistent with sampling from a stochastic decoder. Please clarify the decoding settings (e.g., top-p, temperature implementation, or whether sampling was effectively greedy) or re-examine the variance computation.
  2. [Results and Discussion, first bullet] The bullet "Model size contributes to reasoning performance" is immediately qualified by exceptions (Mistral-7B and FLAN-T5 outperform Llama2-13B); the wording is contradictory. Suggest rephrasing to "model size generally helps within a model family, but other architectural/training factors also matter."
  3. [Throughout] The terms "multi-choice" and "multiple-choice" are used interchangeably; please standardize on one form.
  4. [Experiment Setup for Evaluation] The paper does not specify the decoding temperature, top-p, max token length, or number of repeated runs for the main experiments in Table 2. Please state these for reproducibility.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: ORQA's LLM accuracies are measured against externally constructed labels, with self-citations appearing only as related-work context.

full rationale

ORQA is a benchmark-construction paper rather than a derivation from fitted quantities: the 1,513 instances and gold answers are created and verified by OR experts, and model accuracies (e.g., 0.772 for Llama-3.1-405B-Instruct under 3-shot standard prompting, Table 2) are measured by exact-match against those externally fixed labels. No parameter is fitted to the test answers and then renamed as a prediction, and no claimed result is defined in terms of the quantity it is supposed to predict. The self-citations (Ramamonjison et al. 2022a/b; Mostajabdaveh et al. 2024; Mostajabdaveh, Salman, and Tahmasbi 2022) appear only as background and related-work context; they are not used to justify the benchmark's correctness or the LLM results. The legitimate weakness flagged in the dataset-creation section (no overlapping annotators, reliance on a training session and two-expert verification, no reported inter-annotator agreement) bears on label reliability and benchmark validity, but it does not make the measured accuracies circular: the labels are not a function of the model outputs, and the human baseline of 0.93 on 100 instances is a separate measurement, not an input to the LLM evaluation. I therefore find no circular step.

Assumptions & free parameters 1 free parameters · 4 assumptions · 0 invented entities

The benchmark rests on four domain assumptions: the component ontology, the validity of multiple-choice accuracy as a reasoning measure, the scarcity of OR data (no contamination), and the correctness of expert annotations. No free parameters are fitted to data; the only hand-chosen design parameters are complexity ceilings that calibrate difficulty.

free parameters (1)
  • Complexity ceilings for model components = variables 2-7; parameters mean 4.08 (std 2.19); constraints mean 4.60 (std 3.21)
    Hand-chosen thresholds in Step 1 of dataset creation regulate problem difficulty; all reported accuracies are contingent on this calibration. Not fitted to data but chosen by the authors.
assumptions (4)
  • domain assumption An optimization model can be decomposed into the component taxonomy of Sanchez et al. (2021): elements, decision activities, data attributes, calculations, objective criteria, and specifications.
    The task definition and question taxonomy (Table 8) are built on this ontology; if it is not the right decomposition, the questions lose construct validity.
  • domain assumption Multiple-choice answer accuracy on these questions tracks multistep reasoning and OR knowledge.
    The benchmark's validity claim rests on this; no independent construct validation is provided beyond a single expert baseline.
  • domain assumption Open-source OR corpora are scarce enough that LLM performance on ORQA reflects generalization rather than memorization.
    Used to justify the benchmark as measuring generalization; plausible but not directly tested, since no contamination analysis is run.
  • domain assumption The expert-verified target answers are correct and unambiguous.
    Dataset creation relies on expert verification by two reviewers per instance, but no inter-annotator agreement or external audit is reported.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Evaluating LLM Reasoning in the Operations Research Domain with ORQA." pith.science (2026). https://pith.science/paper/OKMVPIPE

@misc{pith2026241217874,
  author       = {Pith},
  title        = {Pith review of: Evaluating LLM Reasoning in the Operations Research Domain with ORQA},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/OKMVPIPE}},
  note         = {Machine review of arXiv:2412.17874}
}
read the original abstract

In this paper, we introduce and apply Operations Research Question Answering (ORQA), a new benchmark designed to assess the generalization capabilities of Large Language Models (LLMs) in the specialized technical domain of Operations Research (OR). This benchmark evaluates whether LLMs can emulate the knowledge and reasoning skills of OR experts when confronted with diverse and complex optimization problems. The dataset, developed by OR experts, features real-world optimization problems that demand multistep reasoning to construct their mathematical models. Our evaluations of various open source LLMs, such as LLaMA 3.1, DeepSeek, and Mixtral, reveal their modest performance, highlighting a gap in their ability to generalize to specialized technical domains. This work contributes to the ongoing discourse on LLMs generalization capabilities, offering valuable insights for future research in this area. The dataset and evaluation code are publicly available.

Figures

Figures reproduced from arXiv: 2412.17874 by the authors.

Figure 1
Figure 1. Left: Dataset instance containing the description, question, options, and answer (bold). Right: Example reasoning steps needed to answer the simple question. We choose to focus on translating textual problem de￾scriptions into mathematical optimization models rather than directly solving optimization problems. Even special￾ized AI models struggle with scalability and generaliza￾tion when solving simple OR problems (… view at source ↗
Figure 2
Figure 2. An example of optimization problem components, their relationships, and corresponding mathematical formulations. [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Selection, creation, and verification process for the [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (6 more)
Figure 5
Figure 5. Figure 5: Heatmap of LLM performance on different ques [PITH_FULL_IMAGE:figures/full_fig_p006_5.png]
Figure 6
Figure 6. Figure 6: Performance comparison of Standard and CoT [PITH_FULL_IMAGE:figures/full_fig_p006_6.png]
Figure 7
Figure 7. Figure 7: The components of an optimization problem and [PITH_FULL_IMAGE:figures/full_fig_p010_7.png]
Figure 8
Figure 8. Figure 8: Green boxes show how a dataset instance is generated in our three-step dataset generation process. [PITH_FULL_IMAGE:figures/full_fig_p012_8.png]
Figure 9
Figure 9. Figure 9: Annotation of CoT reasoning for two illustrative examples. Right: Although there is one incorrect reasoning step, [PITH_FULL_IMAGE:figures/full_fig_p013_9.png]
Figure 10
Figure 10. Figure 10: Annotation of CoT reasoning for an illustrative example from ORQA validation set, where we have ground truth [PITH_FULL_IMAGE:figures/full_fig_p013_10.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. PEARL: Solver-in-the-Loop Interactive Optimization Modeling from Natural Language

    cs.AI 2026-05 reject novelty 7.0 of 10

    Training an LLM as a multi-turn agent that runs and repairs solver code raises verified optimization solve rates, with the 4B PEARL model outperforming DeepSeek-V3.2-685B in aggregate.

Reference graph

Works this paper leans on

55 extracted references · 23 canonical work pages · cited by 1 Pith paper

  1. [1]

    AhmadiTeshnizi, A.; Gao, W.; and Udell, M. 2024. OptiMUS: Scalable Optimization Modeling with (MI) LP Solvers and Large Language Models. In Forty-first International Conference on Machine Learning

    work page 2024

  2. [2]

    Almazrouei, E.; Alobeidli, H.; Alshamsi, A.; Cappelli, A.; Cojocaru, R.; Debbah, M.; Goffinet, E.; Heslow, D.; Launay, J.; Malartic, Q.; Noune, B.; Pannier, B.; and Penedo, G. 2023. The Falcon Series of Open Language Models. arXiv:2311.16867

    arXiv 2023

  3. [3]

    A.; Alnumay, Y.; Alrashed, S.; Alsubaie, S.; Almushaykeh, Y.; Mirza, F.; Alotaibi, N.; Altwairesh, N.; Alowisheq, A.; et al

    Alzahrani, N.; Alyahya, H. A.; Alnumay, Y.; Alrashed, S.; Alsubaie, S.; Almushaykeh, Y.; Mirza, F.; Alotaibi, N.; Altwairesh, N.; Alowisheq, A.; et al. 2024. When benchmarks are targets: Revealing the sensitivity of large language model leaderboards. arXiv preprint arXiv:2402.01781

    arXiv 2024

  4. [4]

    Arkoudas, K. 2023. GPT-4 can't reason. arXiv preprint arXiv:2308.03762

    arXiv 2023

  5. [5]

    A.; and Dawodi, M

    Baktash, J. A.; and Dawodi, M. 2023. Gpt-4: A review on advancements and opportunities in natural language processing. arXiv preprint arXiv:2305.03195

    arXiv 2023

  6. [6]

    W.; Hou, L.; Longpre, S.; Zoph, B.; Tay, Y.; Fedus, W.; Li, Y.; Wang, X.; Dehghani, M.; Brahma, S.; et al

    Chung, H. W.; Hou, L.; Longpre, S.; Zoph, B.; Tay, Y.; Fedus, W.; Li, Y.; Wang, X.; Dehghani, M.; Brahma, S.; et al. 2024. Scaling instruction-finetuned language models. Journal of Machine Learning Research, 25(70): 1--53

    2024

  7. [7]

    H.; Choi, E.; Collins, M.; Garrette, D.; Kwiatkowski, T.; Nikolaev, V.; and Palomaki, J

    Clark, J. H.; Choi, E.; Collins, M.; Garrette, D.; Kwiatkowski, T.; Nikolaev, V.; and Palomaki, J. 2020. TyDi QA: A Benchmark for Information-Seeking Question Answering in Typologically Diverse Languages. Transactions of the Association for Computational Linguistics, 8: 454--470

    work page 2020

  8. [8]

    Dalvi, B.; Jansen, P.; Tafjord, O.; Xie, Z.; Smith, H.; Pipatanangkura, L.; and Clark, P. 2021. Explaining Answers with Entailment Trees. In Moens, M.-F.; Huang, X.; Specia, L.; and Yih, S. W.-t., eds., Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, 7358--7370. Online and Punta Cana, Dominican Republic: Association...

    work page 2021

Show all 55 references
  1. [9]

    Dubey, A.; Jauhri, A.; Pandey, A.; Kadian, A.; Al-Dahle, A.; Letman, A.; Mathur, A.; Schelten, A.; Yang, A.; Fan, A.; et al. 2024. The Llama 3 Herd of Models. arXiv preprint arXiv:2407.21783

    2024 arXiv

  2. [10]

    L.; Jiang, L.; Lin, B

    Dziri, N.; Lu, X.; Sclar, M.; Li, X. L.; Jiang, L.; Lin, B. Y.; Welleck, S.; West, P.; Bhagavatula, C.; Le Bras, R.; et al. 2024. Faith and fate: Limits of transformers on compositionality. Advances in Neural Information Processing Systems, 36

    2024

  3. [11]

    Fan, Z.; Ghaddar, B.; Wang, X.; Xing, L.; Zhang, Y.; and Zhou, Z. 2024. Artificial Intelligence for Operations Research: Revolutionizing the Operations Research Process. arXiv preprint arXiv:2401.03244

    2024 arXiv

  4. [12]

    Gao, Y.; Xiong, Y.; Gao, X.; Jia, K.; Pan, J.; Bi, Y.; Dai, Y.; Sun, J.; and Wang, H. 2023. Retrieval-augmented generation for large language models: A survey. arXiv preprint arXiv:2312.10997

    2023 arXiv

  5. [13]

    Golovneva, O.; Chen, M.; Poff, S.; Corredor, M.; Zettlemoyer, L.; Fazel-Zarandi, M.; and Celikyilmaz, A. 2023. ROSCOE: A Suite of Metrics for Scoring Step-by-Step Reasoning. arXiv:2212.07919

    2023 arXiv

  6. [14]

    S.; and Lieberman, G

    Hillier, F. S.; and Lieberman, G. J. 2015. Introduction to operations research. McGraw-Hill

    2015

  7. [15]

    Q.; Sablayrolles, A.; Mensch, A.; Bamford, C.; Chaplot, D

    Jiang, A. Q.; Sablayrolles, A.; Mensch, A.; Bamford, C.; Chaplot, D. S.; Casas, D. d. l.; Bressand, F.; Lengyel, G.; Lample, G.; Saulnier, L.; et al. 2023. Mistral 7B. arXiv preprint arXiv:2310.06825

    2023 arXiv

  8. [16]

    Q.; Sablayrolles, A.; Roux, A.; Mensch, A.; Savary, B.; Bamford, C.; Chaplot, D

    Jiang, A. Q.; Sablayrolles, A.; Roux, A.; Mensch, A.; Savary, B.; Bamford, C.; Chaplot, D. S.; Casas, D. d. l.; Hanna, E. B.; Bressand, F.; et al. 2024. Mixtral of experts. arXiv preprint arXiv:2401.04088

    2024 arXiv

  9. [17]

    K.; Cappart, Q.; Rousseau, L.-M.; and Laurent, T

    Joshi, C. K.; Cappart, Q.; Rousseau, L.-M.; and Laurent, T. 2022. Learning the travelling salesperson problem requires rethinking generalization. Constraints, 27(1): 70--98

    2022

  10. [18]

    Kandpal, N.; Deng, H.; Roberts, A.; Wallace, E.; and Raffel, C. 2023. Large language models struggle to learn long-tail knowledge. In International Conference on Machine Learning, 15696--15707. PMLR

    2023

  11. [20]

    Khatun, A.; and Brown, D. G. 2024 b . A Study on Large Language Models' Limitations in Multiple-Choice Question Answering. arXiv:2401.07955

    2024 arXiv

  12. [21]

    S.; Reid, M.; Matsuo, Y.; and Iwasawa, Y

    Kojima, T.; Gu, S. S.; Reid, M.; Matsuo, Y.; and Iwasawa, Y. 2022. Large language models are zero-shot reasoners. Advances in neural information processing systems, 35: 22199--22213

    2022

  13. [22]

    Kweon, S.; Kim, J.; Kwak, H.; Cha, D.; Yoon, H.; Kim, K.; Yang, J.; Won, S.; and Choi, E. 2024. EHRNoteQA: An LLM Benchmark for Real-World Clinical Practice Using Discharge Summaries. arXiv:2402.16040

    2024 arXiv

  14. [23]

    R.; Xiong, C.; Joty, S.; and Wu, C.-S

    Laban, P.; Kry \'s ci \'n ski, W.; Agarwal, D.; Fabbri, A. R.; Xiong, C.; Joty, S.; and Wu, C.-S. 2023. Llms as factual reasoners: Insights from existing benchmarks and beyond. arXiv preprint arXiv:2305.14540

    2023 arXiv

  15. [24]

    Lanham, T.; Chen, A.; Radhakrishnan, A.; Steiner, B.; Denison, C.; Hernandez, D.; Li, D.; Durmus, E.; Hubinger, E.; Kernion, J.; et al. 2023. Measuring faithfulness in chain-of-thought reasoning. arXiv preprint arXiv:2307.13702

    2023 arXiv

  16. [25]

    E.; Motzfeldt, A

    Li \'e vin, V.; Hother, C. E.; Motzfeldt, A. G.; and Winther, O. 2024. Can large language models reason about medical questions? Patterns, 5(3)

    2024

  17. [26]

    Lyu, Q.; Havaldar, S.; Stein, A.; Zhang, L.; Rao, D.; Wong, E.; Apidianaki, M.; and Callison-Burch, C. 2023. Faithful chain-of-thought reasoning. In Park, J. C.; Arase, Y.; Hu, B.; Lu, W.; Wijaya, D.; Purwarianti, A.; and Krisnadhi, A. A., eds., Proceedings of the 13th Interna...

    2023

  18. [27]

    Mallen, A.; Asai, A.; Zhong, V.; Das, R.; Khashabi, D.; and Hajishirzi, H. 2023. When not to trust language models: Investigating effectiveness of parametric and non-parametric memories. In Rogers, A.; Boyd-Graber, J.; and Okazaki, N., eds., Proceedings of the 61st Annual Meet...

    2023

  19. [28]

    Mihaylov, T.; Clark, P.; Khot, T.; and Sabharwal, A. 2018. Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answering. arXiv:1809.02789

    2018 arXiv

  20. [29]

    S.; and Tahmasbi, N

    Mostajabdaveh, M.; Salman, F. S.; and Tahmasbi, N. 2022. Two dimensional guillotine cutting stock and scheduling problem in printing industry. Computers & Operations Research, 148: 106014

    2022

  21. [30]

    T.; Ramamonjison, R.; Carenini, G.; Zhou, Z.; and Zhang, Y

    Mostajabdaveh, M.; Yu, T. T.; Ramamonjison, R.; Carenini, G.; Zhou, Z.; and Zhang, Y. 2024. Optimization modeling and verification from problem specifications using a multi-agent multi-stage LLM framework. INFOR: Information Systems and Operational Research, 62(4): 599--617

    2024

  22. [31]

    A.; Archetti, C.; Ayhan, H.; Battarra, M.; Bennell, J

    Petropoulos, F.; Laporte, G.; Aktas, E.; Alumur, S. A.; Archetti, C.; Ayhan, H.; Battarra, M.; Bennell, J. A.; Bourjolly, J.-M.; Boylan, J. E.; et al. 2024. Operational Research: methods and applications. Journal of the Operational Research Society, 75(3): 423--617

    2024

  23. [32]

    Poli, M.; Wang, J.; Massaroli, S.; Quesnelle, J.; Nguyen, E.; and Thomas, A. 2023. StripedHyena: Moving Beyond Transformers with Hybrid Signal Processing Models. https://github.com/togethercomputer/stripedhyena. Online; accessed August 15, 2024

    2023

  24. [33]

    Ramamonjison, R.; Li, H.; Yu, T.; He, S.; Rengan, V.; Banitalebi-Dehkordi, A.; Zhou, Z.; and Zhang, Y. 2022 a . Augmenting Operations Research with Auto-Formulation of Optimization Models From Problem Descriptions. In Proceedings of the 2022 Conference on Empirical Methods in ...

    2022

  25. [34]

    Ramamonjison, R.; Yu, T.; Li, R.; Li, H.; Carenini, G.; Ghaddar, B.; He, S.; Mostajabdaveh, M.; Banitalebi-Dehkordi, A.; Zhou, Z.; et al. 2022 b . Nl4opt competition: Formulating optimization problems based on their natural language descriptions. In Ciccone, M.; Stolovitzky, G...

    2022

  26. [35]

    Ribeiro, D.; Wang, S.; Ma, X.; Zhu, H.; Dong, R.; Kong, D.; Burger, J.; Ramos, A.; Wang, W.; Huang, Z.; Karypis, G.; Xiang, B.; and Roth, D. 2023. STREET: A Multi-Task Structured Reasoning and Explanation Benchmark. arXiv:2302.06729

    2023 arXiv

  27. [36]

    Robinson, J.; and Wingate, D. 2023. Leveraging large language models for multiple choice question answering. In The Eleventh International Conference on Learning Representations, ICLR

    2023

  28. [37]

    S \'a nchez, J. M. G.; et al. 2021. Modelling in Mathematical Programming. International Series in Operations Research and Management Science, Springer, (978-3): 030--57250

    2021

  29. [38]

    J.; Gupta, K.; and Komatsuzaki, A

    Sawada, T.; Paleka, D.; Havrilla, A.; Tadepalli, P.; Vidas, P.; Kranias, A.; Nay, J. J.; Gupta, K.; and Komatsuzaki, A. 2023. ARB: Advanced Reasoning Benchmark for Large Language Models. arXiv:2307.13692

    2023 arXiv

  30. [39]

    Shen, K.; and Kejriwal, M. 2023. An experimental study measuring the generalization of fine-tuned language representation models across commonsense reasoning benchmarks. Expert Systems, 40(5): e13243

    2023

  31. [40]

    Talmor, A.; Herzig, J.; Lourie, N.; and Berant, J. 2019. CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledge. arXiv:1811.00937

    2019 arXiv

  32. [41]

    Touvron, H.; Martin, L.; Stone, K.; Albert, P.; Almahairi, A.; Babaei, Y.; Bashlykov, N.; Batra, S.; Bhargava, P.; Bhosale, S.; et al. 2023. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288

    2023 arXiv

  33. [42]

    H.; Baldwin, T.; Verspoor, K.; and Cohn, T

    Truong, T. H.; Baldwin, T.; Verspoor, K.; and Cohn, T. 2023. Language models are not naysayers: an analysis of language models on negation benchmarks. In Proceedings of the 12th Joint Conference on Lexical and Computational Semantics (* SEM 2023), 101--114

    2023

  34. [43]

    Valmeekam, K.; Marquez, M.; Olmo, A.; Sreedharan, S.; and Kambhampati, S. 2023. PlanBench: An Extensible Benchmark for Evaluating Large Language Models on Planning and Reasoning about Change. arXiv:2206.10498

    2023 arXiv

  35. [44]

    Valmeekam, K.; Olmo, A.; Sreedharan, S.; and Kambhampati, S. 2022. Large language models still can't plan (a benchmark for LLMs on planning and reasoning about change). In NeurIPS 2022 Foundation Models for Decision Making Workshop

    2022

  36. [45]

    Vidal, T.; Laporte, G.; and Matl, P. 2020. A concise guide to existing and emerging vehicle routing problem variants. European Journal of Operational Research, 286(2): 401--416

    2020

  37. [46]

    Wei, J.; Wang, X.; Schuurmans, D.; Bosma, M.; Ichter, B.; Xia, F.; Chi, E.; Le, Q.; and Zhou, D. 2023. Chain-of-Thought Prompting Elicits Reasoning in Large Language Models. arXiv:2201.11903

    2023 arXiv

  38. [47]

    V.; Zhou, D.; et al

    Wei, J.; Wang, X.; Schuurmans, D.; Bosma, M.; Xia, F.; Chi, E.; Le, Q. V.; Zhou, D.; et al. 2022. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35: 24824--24837

    2022

  39. [48]

    J.; Han, X.; Fu, X.; Zhong, T.; Zeng, J.; Song, M.; et al

    Xiao, Z.; Zhang, D.; Wu, Y.; Xu, L.; Wang, Y. J.; Han, X.; Fu, X.; Zhong, T.; Zeng, J.; Song, M.; et al. 2023. Chain-of-Experts: When LLMs Meet Complex Operations Research Problems. In The Twelfth International Conference on Learning Representations

    2023

  40. [49]

    Yao, S.; Yu, D.; Zhao, J.; Shafran, I.; Griffiths, T.; Cao, Y.; and Narasimhan, K. 2024. Tree of thoughts: Deliberate problem solving with large language models. Advances in Neural Information Processing Systems, 36

    2024

  41. [50]

    R.; and Cao, Y

    Yao, S.; Zhao, J.; Yu, D.; Du, N.; Shafran, I.; Narasimhan, K. R.; and Cao, Y. 2023. ReAct: Synergizing Reasoning and Acting in Language Models. In The Eleventh International Conference on Learning Representations

    2023

  42. [51]

    Yoran, O.; Wolfson, T.; Bogin, B.; Katz, U.; Deutch, D.; and Berant, J. 2023. Answering Questions by Meta-Reasoning over Multiple Chains of Thought. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, 5942--5966

    2023

  43. [52]

    Zhao, R.; Li, X.; Joty, S.; Qin, C.; and Bing, L. 2023. Verify-and-Edit: A Knowledge-Enhanced Chain-of-Thought Framework. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 5823--5840

    2023

  44. [53]

    S.; Zhou, P.; Liu, J.; Hua, Y.; Mao, C.; Wu, X.; et al

    Zhou, H.; Gu, B.; Zou, X.; Li, Y.; Chen, S. S.; Zhou, P.; Liu, J.; Hua, Y.; Mao, C.; Wu, X.; et al. 2023. A survey of large language models in medicine: Progress, application, and challenge. arXiv preprint arXiv:2311.05112

    2023 arXiv

  45. [54]

    V.; Chi, E

    Zhou, P.; Pujara, J.; Ren, X.; Chen, X.; Cheng, H.-T.; Le, Q. V.; Chi, E. H.; Zhou, D.; Mishra, S.; and Zheng, H. S. 2024. Self-discover: Large language models self-compose reasoning structures. arXiv preprint arXiv:2402.03620

    2024 arXiv

  46. [55]

    , " * write output.state after.block = add.period write newline

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint howpublished institution isbn journal key month note number organization pages publisher school series title type volume year label extra.label sort.label short.list INTEGERS output.state before.all...

  47. [56]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.