Large Language Models for Supply Chain Decisions

David Simchi-Levi; Ishai Menache; Jeevan Pathuri; Konstantina Mellou

REVIEW 3 major objections 4 minor 2 cited by

Large Language Models for Supply Chain Decisions

T0 review · 3 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper claims that a large language model can translate planners' natural-language questions into small edits to existing optimization models, letting supply chain decisions be made in minutes instead of days.

desk verdict A candid, well-scoped industry case study that adds deployment detail to a known LLM-to-optimization pipeline, with honest limitations but thin evaluation metrics. read the letter →

arxiv 2507.21502 v1 pith:DB7VQ4AJ submitted 2025-07-29 cs.AI

David Simchi-Levi , Konstantina Mellou , Ishai Menache , Jeevan Pathuri This is my paper · ORCID

classification cs.AI

keywords largelanguagemodelssupplychainplanningwhat-ifanalysisin-contextlearningoptimizationdemanddriftprocurementinteractive

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that large language models can act as a translator between business users and the mathematical optimization tools that supply chain planners already rely on. Instead of asking data scientists to explain a recommendation, adjust a model, or run a scenario, planners ask the LLM a question in plain language; the LLM turns the question into a small modification of the optimization model, runs the supply chain tool, and converts the result back into language. The paper reports this design in production for a public cloud provider's hardware fulfillment problem, where demand-drift analysis that took about a week now takes minutes, what-if questions are answered in minutes instead of days, and the deployed system reaches around 90% accuracy on its test questions. If this holds, the time-to-decision bottleneck in supply chain planning is no longer the optimization solve but the interface around it.

What carries the argument

The paper's key object is a three-stage translation loop: a question handler that appends curated question-answer examples to the user's query, an LLM that outputs application code, often a small constraint or objective change to the existing optimization model, and an interpreter that renders the solver output as plain language. The what-if questions are deliberately framed as small changes to an already-solved model, so the LLM's task is narrow and the mathematical guarantees of the underlying optimizer carry over. In-context learning, rather than fine-tuning, supplies the examples; the paper's production deployment uses a large commercial model and achieves around 90% accuracy on its test suite, with smaller open-source models under evaluation.

What would settle it

Compile a set of what-if questions with known optimal answers, such as small but non-trivial network examples, run the deployed LLM-to-optimization pipeline on them, and compare the solver outputs to the true optima under the modified constraints; if accuracy on multi-constraint questions falls well below the reported 90%, or if the generated code is infeasible on a material fraction of cases, the central claim fails.

Watch

Extended reading notes

Core claim

The central claim is that the 'language' of optimization, mathematical programs with constraints and objectives, can be the interface point for LLMs. The LLM is deliberately not asked to solve the optimization problem or to replace the mathematical model; it is only asked to produce application code that represents a small change to an existing model, such as adding a constraint that a retailer can only buy from a particular factory. The planning tool then solves the modified model, and a second LLM pass interprets the output. The paper reports that this separation of responsibilities is what makes what-if analysis accurate enough to deploy: the optimization guarantee comes from the solver, while the LLM only handles translation. In production, the what-if system answers fulfillment questions for a cloud supply chain, and an automated demand-drift report identifies root causes of plan changes; both were previously manual, multi-day processes.

Load-bearing premise

The whole design assumes the LLM translates a natural-language question into correct mathematical code, and that any translation error can be caught before the resulting plan is acted on; the paper explicitly states that tools to validate or correct an LLM-generated mathematical model do not yet exist, and the 90% accuracy figure rests on an undisclosed test set.

Editorial extensions

If this is right

Demand-drift analysis can be fully automated, with the LLM identifying root causes such as new hardware generations or reduced customer requirements, cutting a multi-day process to minutes.
Planners, not data scientists, can explore what-if scenarios such as shutting down a factory or changing tariff levels, because the LLM translates the scenario into a small constraint change and the solver keeps its optimality guarantees.
In procurement, the same data-insight layer can surface contract clauses, such as volume discounts, that humans missed, with cited examples saving millions of dollars.
If smaller fine-tuned language models approach the accuracy of the current large model on these narrow translation tasks, the cost per query can drop without losing the productivity gains.
The reported productivity impact includes roughly 23% savings in fulfillment investigation time at the cloud provider, implying that the interface bottleneck, not the solver, was the binding constraint.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

An unstated test is how the 90% accuracy degrades on out-of-distribution or ambiguous phrasing: the paper's gradual deployment and fallback mechanisms suggest the production team already expects some queries to fail, so the headline accuracy may overstate performance on novel questions.
The same translate-then-solve architecture should transfer to any optimization-over-conversation setting, such as network design, pricing, or workforce scheduling, since the LLM only needs to emit small model edits; this is a corollary the paper only hints at in its procurement and manufacturing examples.
A direct extension would be to log every query and code pair from production and periodically score the LLM's translations against the solver's feasibility and optimality, using the stored pairs to grow the in-context example repository; this would give the paper's accuracy claim a measurable reliability curve.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Colleague,

Read the Simchi-Levi et al. chapter. It's a production deployment case study, not a new method: the core idea of using an LLM to translate natural-language queries into small modifications of an existing optimization model was in their 2023 arXiv paper [1]. What this adds is a real Microsoft cloud supply chain deployment, procurement examples, and a reasonably detailed description of the system architecture (question handler, LLM-as-translator, optimization solver, interpreter). The architecture is sensible: keep the mathematical model intact and the proprietary data out of the LLM prompt.

The paper does a few things well. It is honest about the unsolved verification problem, admitting in Section 5 that they cannot yet validate a complete LLM-generated model, and in Section 8 that even with verification, wrong outputs will occur. The gradual rollout with monitoring and fallback mechanisms is a responsible engineering practice. The narrative is clear and the examples are concrete.

The soft spots are the same ones the stress-test flags. The central claim of 'without human-in-the-loop' is too strong given their own caveats. The 90% accuracy figure in Section 6 is a single number with no disclosed test set, error taxonomy, or severity breakdown. 'Around 90%' could mean anything from harmless formatting issues to silently wrong plans. The 23% investigation time saved is similarly unsupported. For a book chapter these gaps are common, but they do limit the scientific value. Also, the what-if reduction from days to minutes is plausible but not measured with a baseline; it's a testimonial.

The citation pattern is fine – they cite their own prior work because it is the foundation, and they reference the HBR article. The paper is internally consistent and the limitations are stated up front.

If this crossed my desk as a journal submission, I would send it to review only with the expectation that the evaluation section be substantially expanded. As a chapter in a practitioner-oriented volume, it is acceptable as is. It will be useful to readers who want to see a real LLM-optimization integration and its operational challenges, but it is not a scientific contribution in the strict sense.

My recommendation: if you are considering this for peer review, treat it as a case study with valuable honest caveats, but require more rigorous evaluation metrics before publication in a research venue. Otherwise, it's a fine book chapter.

Referee Report

3 major / 4 minor

Summary. The paper reports on the integration of large language models with supply chain planning tools to address three use cases: data discovery and insights, what-if questions, and interactive planning. The authors describe an architecture in which the LLM translates natural-language queries into application code that invokes optimization solvers and data repositories, while keeping proprietary data out of the LLM prompt. They report a production deployment at Microsoft's cloud supply chain, claiming roughly 90% accuracy for GPT-4, a 23% reduction in fulfillment investigation time, and a reduction in decision time from days or weeks to minutes or hours. The paper also includes procurement examples and a discussion of challenges such as ambiguity, verification, and data quality.

Significance. If the central claims are substantiated, this work is significant because it demonstrates a practical path toward democratizing optimization-based supply chain tools, allowing business planners without data science backgrounds to obtain answers and explore scenarios directly. The architectural choice of keeping proprietary data out of the LLM prompt is a valuable and transferable design principle, and the paper names concrete, falsifiable deployment outcomes. However, the paper's quantitative claims are presented without a disclosed evaluation methodology, test set, or baselines, and the paper itself acknowledges that automatic validation of LLM-generated model changes remains unsolved. The significance therefore hinges on whether these claims can be supported with reproducible evidence.

major comments (3)

[Section 6, 'Evaluation and Deployment'] The central quantitative claims—'around 90% accuracy,' '23% of fulfillment investigation time saved,' and 'response times of few minutes instead of days'—are stated without describing the evaluation methodology, the composition of the test set, the definition of accuracy, error bars, or comparison baselines. Because the paper's thesis is that LLM-based technology can operate 'without human-in-the-loop,' it is essential to know whether the 90% figure counts only benign formatting errors or also includes silently wrong plans, and how many and what kinds of questions were tested. The authors should provide a reproducible evaluation protocol, define the accuracy metric, report confidence intervals, and compare against a baseline such as human performance or a rule-based system.
[Sections 5 and 8, 'Interactive Planning' and 'Discussion'] The manuscript contains an internal tension that is load-bearing for the paper's main claim. Section 5 states, 'We still need tools that will either validate the model generated by LLM-based technology or identify what is missing and how to correct the initial model,' and Section 8 states, 'Even with proper verification, we expect LLMs to occasionally produce wrong outputs.' These statements are hard to reconcile with the abstract's claim of interaction 'without human-in-the-loop,' especially given that Section 6 describes a gradual deployment with 'monitor user interactions, accuracy, and fallback mechanisms.' The authors should clarify whether human oversight is present in the deployed system and, if so, how the 'without human-in-the-loop' claim is qualified. Without this clarification, the reader cannot determine the extent to which the reported productivity gains depend on human monitoring and correction.
[Section 4, 'Answering What-If Questions'] The key technical premise is that what-if questions can be translated into 'a small change to the original mathematical model' by the LLM. The manuscript does not describe how the LLM is constrained to produce valid small changes, how syntax or semantic errors are detected, or what happens when the generated code is correct in form but incorrect in business meaning. The in-context learning approach is mentioned with a pointer to reference [1], but since the reliability of this translation step is the crux of the what-if capability and directly underpins the no-human-in-the-loop claim, the manuscript should include at least a summary of the error-handling mechanism and the observed failure modes.

minor comments (4)

[Section 3, first paragraph] There is a typo: 'propriety data' should be 'proprietary data.'
[Section 6, Example 2] The word 'disactivate' should be 'deactivate.'
[Section 4, Figure 1] The text says 'see the figure below' at the end of Section 4, but Figure 1 is placed on the next page; consider referring to it by figure number to avoid ambiguity.
[Section 1, final paragraph] The phrase 'how state-of-the-art LLMs can be applied for reasoning about supply chain data' would read more naturally as 'applied to reasoning about supply chain data.'

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation: the what-if pipeline delegates computation to an optimizer and the reported claims rest on deployment evidence, not on self-referential definitions.

full rationale

The paper does not present a mathematical derivation chain whose outputs are defined in terms of its inputs. The core what-if mechanism (Section 4) is an implementation architecture: the LLM translates a user query into code representing a 'small change' to an existing mathematical model, the optimization solver produces the revised plan, and the LLM only renders the result in natural language. Thus the quantities reported to the planner (cost changes, lost demand, service levels) are computed by the supply chain optimizer, not by the LLM, so no prediction is forced by construction. The paper's heavy use of self-citations ([1]-[5] are all authored or co-authored by the present authors) is notable, but it is not load-bearing circularity: [1] is pointed to for additional detail on the question-answer repository and in-context learning, while the central deployment claims are supported by in-paper evidence from Microsoft's cloud supply chain, including user-reported timelines and the estimate of 23% investigation time saved. No uniqueness theorem or unverified ansatz is imported from prior work by the same authors. The 90% accuracy figure in Section 6 is an internal evaluation whose test set, error taxonomy, and failure severity are not disclosed; this is an evidentiary transparency limitation, not a circular reduction. Similarly, Sections 5 and 8 explicitly state that validation of LLM-generated models is unsolved and that wrong outputs are expected, which undercuts any strong 'without human-in-the-loop' reading but again is a correctness risk, not a case of the paper's claim being equivalent to its own definition. No equation, fitted parameter, or constructed quantity is shown to reduce to its own input, so the appropriate finding is no significant circularity.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The paper introduces no free parameters or invented entities. It relies on domain assumptions about LLM reliability, tool interface maturity, and user query clarity, all of which are identified in the paper's own discussion.

assumptions (4)

domain assumption LLMs can reliably translate natural-language queries into mathematical program modifications given a sufficient set of in-context examples.
The what-if and interactive planning use cases depend on this reliability. The paper reports about 90% accuracy but provides no evaluation details (Sections 4 and 6).
domain assumption The underlying supply chain planning tools expose mature, well-defined APIs that the LLM can call automatically.
Section 8 states that answering what-if questions requires optimization tools to be mature and reliable with well-defined interfaces.
domain assumption Planners can phrase unambiguous questions, or the LLM can insert clarification steps when questions are ambiguous.
Section 8 discusses the ambiguity of terms like "better" and the need to train users or add clarifying-question generation.
domain assumption The company's internal data is clean and accessible enough for query generation and plan retrieval.
Section 8 identifies clean, well-structured data as a prerequisite for the approach.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Large Language Models for Supply Chain Decisions." pith.science (2026). https://pith.science/paper/DB7VQ4AJ

@misc{pith2026250721502,
  author       = {Pith},
  title        = {Pith review of: Large Language Models for Supply Chain Decisions},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/DB7VQ4AJ}},
  note         = {Machine review of arXiv:2507.21502}
}

read the original abstract

Supply Chain Management requires addressing a variety of complex decision-making challenges, from sourcing strategies to planning and execution. Over the last few decades, advances in computation and information technologies have enabled the transition from manual, intuition and experience-based decision-making, into more automated and data-driven decisions using a variety of tools that apply optimization techniques. These techniques use mathematical methods to improve decision-making. Unfortunately, business planners and executives still need to spend considerable time and effort to (i) understand and explain the recommendations coming out of these technologies; (ii) analyze various scenarios and answer what-if questions; and (iii) update the mathematical models used in these tools to reflect current business environments. Addressing these challenges requires involving data science teams and/or the technology providers to explain results or make the necessary changes in the technology and hence significantly slows down decision making. Motivated by the recent advances in Large Language Models (LLMs), we report how this disruptive technology can democratize supply chain technology - namely, facilitate the understanding of tools' outcomes, as well as the interaction with supply chain tools without human-in-the-loop. Specifically, we report how we apply LLMs to address the three challenges described above, thus substantially reducing the time to decision from days and weeks to minutes and hours as well as dramatically increasing planners' and executives' productivity and impact.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

Strategic Buying Agents
econ.TH 2026-07 accept novelty 6.5 of 10

Optimal purchase-timing policies for delegated shopping agents are dynamic or randomized thresholds under stationary, Bayesian, and robust price models, and they compete with simple baselines on real Amazon data while...
Validating Generative Agent-Based Models for Logistics and Supply Chain Management Research
cs.MA 2025-08 conditional novelty 6.0 of 10

Six LLMs show an equivalence-versus-process paradox: some match human surface behaviors but few replicate human decision pathways, so GABMs need dual-level validation before use in logistics research.

Reference graph

Works this paper leans on

13 extracted references · 10 canonical work pages · cited by 2 Pith papers

[1]

Motivation Modern supply chains are complex, containing multiple tiers of suppliers, customers, and service providers. Optimization and a variety of related data science tools have been widely utilized for decision making in supply chains including, for example , tools for supply chain planning, network design, procurement and supply chain resiliency . Th...

work page
[2]

On Large Language Models A large language model (LLM) is a machine learning model that is trained on extensive data and can be applied for a variety of use cases . In the training phase, an LLM learns statistical patterns, word relationships, and contextual information from diverse sources, such as books, articles, websites, and code repositories. At a hi...

work page
[3]

How much r aw material of type T does supplier S have today?

Data Discovery and Insights The most immediate use of LLMs is in assisting business operators understand the current situation of the supply chain by facilitating data discovery and providing insights. Consider a classic supply chain with a certain number of suppliers of raw material , factories for prod ucing certain products and retailers that sell thes...

work page
[4]

What would be the additional cost if the overall product demand increases by 15%?

Answering What-If Questions 4 To get a full picture of the underlying decision processes, operators may require more than just explaining the current supply chain plan. In this section we describe an additional level of sophistication: using LLMs to answer what -if questions. We will discuss the technical innovation that enables answering such questions a...

work page
[5]

For example, a planner may receive real -time information that a specific manufacturing facility is down for seven days due to a winter storm in its area

Interactive Planning Going beyond answering what-if questions, we observe that LLM technology can be used more broadly for interactive planning, where planners would like to update the model to reflect the current business environment or the LLM provides updates to the planner on a change in business conditions. For example, a planner may receive real -ti...

work page
[6]

What is the cost increase if we dock a certain demand a week earlier?

Production Deployment – Microsoft’s Cloud Supply Chain Background Cloud computing is a multi -billion-dollar business that draws substantial capital investments from large companies such as Amazon, Microsoft and Google. Large cloud providers need to accommodate the growing demand for computing resources while avoiding over-provisioning of hardware and ope...

work page
[7]

In the chemical industry, supply contracts include not only pricing, quality, lead time, and risk mitigation information but also detailed regulatory requirements

Procurement Examples Beyond the implementation of LLM -based technology for supply chain decisions at Microsoft, in this section we report implementations to improve the management and execution of procurement decisions. In the chemical industry, supply contracts include not only pricing, quality, lead time, and risk mitigation information but also detail...

work page
[8]

language

Discussion LLMs is a disruptive technology that will shape supply chain management in years to come. The examples reported in this paper demonstrate a substantial increase in productivity and planner satisfaction . We conclude this chapter by discussing how companies can get started with incorporating LLM based technology for supply chain management and h...

work page

Show all 13 references

[9]

Large language models for supply chain optimization

Beibin Li, Konstantina Mellou, Bo Zhang, Jeevan Pathuri, and Ishai Menache. "Large language models for supply chain optimization." arXiv preprint arXiv:2307.03875 (2023). 11

2023 arXiv
[10]

Efficient Cloud Server Deployment Under Demand Uncertainty

Rui Peng Liu, Konstantina Mellou, Evelyn Xiao -Yue Gong, Beibin Li, Thomas Coffee, Jeevan Pathuri, David Simchi-Levi, and Ishai Menache. "Efficient Cloud Server Deployment Under Demand Uncertainty." Manufacturing & Service Operations Management (2025)

2025
[11]

How generative AI improves supply chain management

Ishai Menache, Jeevan Pathuri, David Simchi-Levi, and Tom Linton, “How generative AI improves supply chain management”, Harvard Business Review 104 (1-2), 86-95
[12]

Prediction-Guided Active Experiments

Ruicheng Ao, Hongyu Chen, and David Simchi-Levi, “Prediction-Guided Active Experiments”, (2024), [2411.12036] Prediction-Guided Active Experiments

2024 arXiv
[13]

Small language models for application interactions: A case study

Beibin Li, Yi Zhang, Sébastien Bubeck, Jeevan Pathuri, and Ishai Menache. "Small language models for application interactions: A case study." arXiv preprint arXiv:2405.20347 (2024)

2024 arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.

[1] [1]

Motivation Modern supply chains are complex, containing multiple tiers of suppliers, customers, and service providers. Optimization and a variety of related data science tools have been widely utilized for decision making in supply chains including, for example , tools for supply chain planning, network design, procurement and supply chain resiliency . Th...

work page

[2] [2]

On Large Language Models A large language model (LLM) is a machine learning model that is trained on extensive data and can be applied for a variety of use cases . In the training phase, an LLM learns statistical patterns, word relationships, and contextual information from diverse sources, such as books, articles, websites, and code repositories. At a hi...

work page

[3] [3]

How much r aw material of type T does supplier S have today?

Data Discovery and Insights The most immediate use of LLMs is in assisting business operators understand the current situation of the supply chain by facilitating data discovery and providing insights. Consider a classic supply chain with a certain number of suppliers of raw material , factories for prod ucing certain products and retailers that sell thes...

work page

[4] [4]

What would be the additional cost if the overall product demand increases by 15%?

Answering What-If Questions 4 To get a full picture of the underlying decision processes, operators may require more than just explaining the current supply chain plan. In this section we describe an additional level of sophistication: using LLMs to answer what -if questions. We will discuss the technical innovation that enables answering such questions a...

work page

[5] [5]

For example, a planner may receive real -time information that a specific manufacturing facility is down for seven days due to a winter storm in its area

Interactive Planning Going beyond answering what-if questions, we observe that LLM technology can be used more broadly for interactive planning, where planners would like to update the model to reflect the current business environment or the LLM provides updates to the planner on a change in business conditions. For example, a planner may receive real -ti...

work page

[6] [6]

What is the cost increase if we dock a certain demand a week earlier?

Production Deployment – Microsoft’s Cloud Supply Chain Background Cloud computing is a multi -billion-dollar business that draws substantial capital investments from large companies such as Amazon, Microsoft and Google. Large cloud providers need to accommodate the growing demand for computing resources while avoiding over-provisioning of hardware and ope...

work page

[7] [7]

In the chemical industry, supply contracts include not only pricing, quality, lead time, and risk mitigation information but also detailed regulatory requirements

Procurement Examples Beyond the implementation of LLM -based technology for supply chain decisions at Microsoft, in this section we report implementations to improve the management and execution of procurement decisions. In the chemical industry, supply contracts include not only pricing, quality, lead time, and risk mitigation information but also detail...

work page

[8] [8]

language

Discussion LLMs is a disruptive technology that will shape supply chain management in years to come. The examples reported in this paper demonstrate a substantial increase in productivity and planner satisfaction . We conclude this chapter by discussing how companies can get started with incorporating LLM based technology for supply chain management and h...

work page