REVIEW 4 major objections 5 minor 35 references
State-Inference-Based Prompting for Natural Language Trading with Game NPCs
T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash
Pith's one-line read Natural-language NPC trading stays rule-compliant when the LLM must infer and report its previous dialogue state before answering.
desk verdict SIBP is a useful prompt-engineering recipe with a sharp ablation, but the headline compliance numbers rely on self-reported state labels and a filtered 88-dialogue denominator. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing mechanism is the "identify and report the previous state" directive embedded in a unified prompt, together with the state-specific rules stacked on it. In each JSON response the model must fill last_trade_context, context_type, context_subtype, and an items array, which makes the model's state inference inspectable and usable by the game system. The same field drives conditional transitions—for example, CONFIRM_SELL is permitted only when last_trade_context is CHECK_CONFIRMATION and the player gives positive confirmation. Two auxiliary mechanisms carry the reliability results: state-specific data referencing, which uses the full game item list for general chat but only the merchant inventory for trade, and placeholder-based price post-processing, where the LLM writes __PRICE__ and the system computes the arithmetic.
What would settle it
Run the same SIBP prompt with 100 human players instead of the virtual player and have an independent judge or the game server record whether every purchase reaches CONFIRM_SELL only after a CHECK_CONFIRMATION turn. If human dialogues show a skip rate meaningfully above the 2.3 percent failure seen in simulation, or if a server-side state replay disagrees with the self-reported last_trade_context, the central deployment claim is weakened.
Extended reading notes
Core claim
The central discovery is that reliable rule adherence does not require an external state manager: the LLM can be made to act as its own state tracker. SIBP's prompt tells the model to identify the most recent trading sub-context from dialogue history before responding, and to write that identification into the JSON field last_trade_context. The response format then decides whether a player's confirmation counts as accepting an item description (so the NPC offers a price) or accepting a price (so the NPC asks for final confirmation). With all four prompt elements active, 97.73 percent of dialogues in both scenarios follow the mandatory CHECK_CONFIRMATION-to-CONFIRM_SELL sequence, and the reporting element alone lifts compliance to 94.32 percent when the identification directive is absent. Inside TRADE state, item references draw only from the NPC's inventory, while the OFFER_SELL stage emits a __PRICE__ placeholder that the system later replaces with a computed total, yielding 100.0 percent price accuracy at the offer and 99.7 percent across later states.
Load-bearing premise
The whole evaluation assumes that dialogues generated by a simulated player LLM represent real player behavior, and that the NPC's own self-reported state fields are a trustworthy measure of rule compliance; if either assumption fails, the headline percentages may not transfer to actual players.
Editorial extensions
If this is right
- If SIBP works as reported, game studios can deploy LLM merchant NPCs without an external dialogue-state manager or tool-use pipeline; the prompt itself supplies the control.
- The previous-state reporting element is the main lever: enabling it alone nearly closes the compliance gap, and combining it with explicit transition explanations gives the best result.
- Placeholder post-processing removes arithmetic from the model, so accuracy gains appear even with smaller models: SIBP+PPP on 2.0-flash reaches 100.0 percent price accuracy at the offer stage versus 49.4 percent for basic SIBP on the same model.
- State-specific referencing keeps NPCs able to discuss any game-world item in general conversation while restricting trade proposals to sellable inventory, which is the paper's proposed cure for item hallucination.
- The same prompt recipe can be lifted to other rule-governed LLM interactions such as interactive kiosks and customer-support workflows, where the model must infer which workflow step is active.
Reading between the lines
- A natural next test is to score every pairwise state transition rather than only the CHECK_CONFIRMATION-to-CONFIRM_SELL gate, since the paper's one-gate metric leaves the full transition graph unmeasured.
- Recruiting human players or a second player-model family would test whether the simulated-player results hold under noisier inputs; the current evaluation uses one model for both roles.
- Because placeholder pricing replaces arithmetic with deterministic post-processing, the accuracy gain should transfer across models and languages as long as the placeholder token stays short; the paper's smaller-model result is the first hint.
- A production variant could persist last_trade_context as game state, allowing the NPC to resume a mid-transaction after disconnects or context-window truncation.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes State-Inference-Based Prompting (SIBP), a prompt-engineering framework for LLM-driven NPCs in game trading. The method decomposes trading into five sub-states (SHOW_INVENTORY, OFFER_SELL, NEGOTIATE_PRICE, CHECK_CONFIRMATION, CONFIRM_SELL) under a TRADE context, adds a directive for the LLM to identify and report the previous trading state, and uses state-specific referencing plus placeholder-based price post-processing. The authors evaluate SIBP against several baselines using a virtual player LLM across 100 dialogues per condition, reporting >97% state transition compliance (STCR), >95% sellable-item response rate (SIRR), and 99.7% price accuracy. They also report computational-efficiency comparisons across model sizes and structured-output variants.
Significance. If the reported results were trustworthy, SIBP would be a practically useful prompt-design recipe for rule-governed LLM dialogue, with plausible extensions to other transactional domains. The prompt design ideas—especially the explicit previous-state identification and state-specific post-processing—are reasonable and worth studying. However, the evaluation as presented does not support the headline claims: the main metrics are computed from the LLM's own JSON state labels and item arrays, on a selectively chosen subset of dialogues, and the price-accuracy metric for the post-processed condition is circular by construction. The paper would be significantly strengthened by externally verified metrics and a clear statement of the effective sample size.
major comments (4)
- [Section 4.2, Table 1] STCR is not computed across 100 dialogues as the abstract states. The text says STCR was calculated based on the first 88 dialogues for each method where the CONFIRM_SELL state was reached, and this 88 is the minimum number of such occurrences across methods. This outcome-dependent selection excludes dialogues that never reached the final state, which can inflate compliance rates and makes the comparison unfair across methods with different rates of reaching CONFIRM_SELL. Please report the number of excluded dialogues per method and re-run the analysis with a fixed denominator (e.g., all 100 dialogues, treating failure to reach CONFIRM_SELL as non-compliance when a purchase was intended), or provide a clear statistical justification for conditioning on reaching the final state.
- [Section 4.2, STCR definition] STCR is scored from the NPC LLM's self-reported JSON state fields (context_subtype and last_trade_context), with no verification against the actual natural-language utterance. A dialogue is counted as compliant if the JSON label sequence ends with CHECK_CONFIRMATION followed by CONFIRM_SELL, regardless of whether the npc_dialogue text actually contained a confirmation question. Thus the metric measures the LLM's consistency in labeling states, not its adherence to the rule that an explicit confirmation must be obtained before finalizing a purchase. Please add a manual or programmatic check of the dialogue text (e.g., detecting a confirmation question before the final transition) or re-define the metric accordingly.
- [Section 4.3, SIRR] The Sellable Item Response Rate is computed from the items array in the JSON response, not from the natural-language npc_dialogue. A response that narratively offers a hallucinated item (e.g., mentioning a non-inventoried item in the spoken text) would be scored as correct if the items field contains only sellable items. Since the paper's stated goal is to prevent item hallucination in the player-visible dialogue, SIRR should be evaluated on the full response, including the free-form npc_dialogue, or a separate hallucination rate on the natural-language output should be reported.
- [Section 4.4, Table 2] The 100% price accuracy in OFFER_SELL for SIBP+PPP is by construction: the system computes the total from the item details and substitutes it for the __PRICE__ placeholder, so the metric only checks that the placeholder was emitted, not that the LLM performed the calculation. Moreover, the price-accuracy metric compares the stated total to the item details within the same LLM response, not to the ground-truth inventory prices; a response that lists a wrong unit price but a consistent total would still be counted as accurate. Please report agreement with the actual inventory price list, and separately report (i) the placeholder-emission rate and (ii) the arithmetic-consistency rate after post-processing.
minor comments (5)
- [Abstract and Section 4.2] The abstract claims evaluation across 100 trading dialogues, but STCR uses only 88 per method; please align the wording or the analysis.
- [Section 4.2, Figure 2] The state-transition heatmaps are presented without confidence intervals or statistical tests; the difference between baseline4 (94.32%) and SIBP (97.73%) may not be significant given the sample size. Please report uncertainty or perform a significance test.
- [Section 3.2.1] The paper refers to 'six states' in several places but lists three main contexts (NONE, TRADE, END_CONVERSATION) with five trade subcontexts; the taxonomy should be stated consistently.
- [Section 4.1] The use of a single virtual-player LLM (gemini-2.5-flash, temperature 0.7) is a limitation for external validity; the paper should explicitly discuss potential differences from real player behavior and consider human evaluation or additional backbone models.
- [Appendix and Table 1] The PDF contains rendering artifacts in Table 1 and its caption (e.g., '/uni00000036/...' strings) that obscure the figure; also, reference [16] contains an unusual citation format 'arXiv preprint [6, 7]'. Please fix these formatting issues.
Circularity Check
Two headline metrics are self-referential: OFFER_SELL price accuracy is forced by placeholder post-processing, and STCR is scored from the LLM's own self-reported state labels.
-
fitted input called prediction
[Section 3.3.2 (State-specific Post-processing), Section 4.4 (Evaluation Metric, Table 2)]
"The system calculates the correct price based on responded item information and replaces the placeholder (Sec. 3.3.2); 'Price accuracy is the percentage of NPC responses where the total price stated by the LLM matches the sum calculated from the item details (i.e., prices and quantities) also provided within the same LLM response' (Sec. 4.4)."
In SIBP+PPP, the final total is not written by the LLM: the LLM emits __PRICE__ and the system fills in the sum computed from the same item details that the metric uses as its reference. Hence for OFFER_SELL the condition 'total equals sum of item details' is satisfied by construction whenever placeholder substitution succeeds; the reported 100.0% (and the OFFER_SELL share of the 99.7% headline) measures the deterministic post-processor, not LLM calculation. The metric does not validate prices or quantities against the inventory or the player request, so the headline 'calculation precision' is a self-consistency check whose reference value is generated by the very component being credited.
-
self definitional
[Section 3.2.2 (Element 4), Section 4.2 (Evaluation Metric), Appendix A (Response Format)]
"The JSON response must include a last_trade_context field where the LLM records the identified previous state. This documentation enables system monitoring of the LLM's reasoning process (Sec. 3.2.2); 'STCR is the proportion of conversations that correctly followed the CHECK_CONFIRMATION→ CONFIRM_SELL sequence' (Sec. 4.2)."
The state sequence used to score STCR is the same LLM's self-reported JSON state fields (context_subtype and last_trade_context); the paper defines no independent scorer over npc_dialogue. A model that labels the prior state as CHECK_CONFIRMATION before CONFIRM_SELL passes even if its spoken dialogue never asks the required confirmation question. Compliance is thus self-definitional: the model's own state labels serve as the ground truth for whether it followed the state rule, so STCR can be high while the safeguard behavior is absent. The central >97% state-compliance claim therefore measures self-consistent reporting rather than externally verified rule adherence.
full rationale
There is no self-citation chain or imported uniqueness theorem in this paper; the prompting design itself is an independent contribution, and the SIRR metric, while narrow (it checks only the LLM's items array against the inventory input), is not by-construction circular. The virtual-player setup is a generalizability concern rather than circularity. However, two headline evaluation claims are not externally grounded. First, OFFER_SELL price accuracy under placeholder post-processing is tautological because the system computes the total from the same item details the metric uses as ground truth. Second, STCR as described relies on the LLM's own state labels because the only state-monitoring channel defined is the JSON field the LLM writes; no independent check of the natural-language dialogue is reported. Because the state-compliance claim and at least part of the 99.7% calculation-precision claim reduce to self-consistency, the evaluation is partially circular.
Assumptions & free parameters
free parameters (1)
- STCR evaluation subset size =
88 dialogues
assumptions (4)
- domain assumption The NPC LLM's JSON output fields (context_type, context_subtype, last_trade_context) accurately represent the true dialogue state and behavior.
- domain assumption A virtual player LLM at temperature 0.7 with fixed seeds produces dialogue behavior representative of human players.
- domain assumption Prompt-following of structured JSON and placeholder rules is reliable enough for the measured metrics to reflect the method rather than lucky generation.
- ad hoc to paper The five trade subcontexts plus NONE and END_CONVERSATION are a sufficient decomposition of trustworthy trading.
Cite this review
Pith. "Pith review of State-Inference-Based Prompting for Natural Language Trading with Game NPCs." pith.science (2026). https://pith.science/paper/KW4IJ2CL
@misc{pith2026250707203,
author = {Pith},
title = {Pith review of: State-Inference-Based Prompting for Natural Language Trading with Game NPCs},
year = {2026},
howpublished = {\url{https://pith.science/paper/KW4IJ2CL}},
note = {Machine review of arXiv:2507.07203}
}
read the original abstract
Large Language Models enable dynamic game interactions but struggle with rule-governed trading systems. Current implementations suffer from rule violations, such as item hallucinations and calculation errors, that erode player trust. Here, State-Inference-Based Prompting (SIBP) enables reliable trading through autonomous dialogue state inference and context-specific rule adherence. The approach decomposes trading into six states within a unified prompt framework, implementing context-aware item referencing and placeholder-based price calculations. Evaluation across 100 trading dialogues demonstrates >97% state compliance, >95% referencing accuracy, and 99.7% calculation precision. SIBP maintains computational efficiency while outperforming baseline approaches, establishing a practical foundation for trustworthy NPC interactions in commercial games.
Figures
Reference graph
Works this paper leans on
-
[1]
A survey on large language model based autonomous agents
Junyu Luo et al. A survey on large language model based autonomous agents. arXiv preprint arXiv:2503.21460, 2024
arXiv 2024
-
[2]
Procedural content generation in games: A survey with insights on emerging llm integration
Mahdi Farrokhi Maleki and Richard Zhao. Procedural content generation in games: A survey with insights on emerging llm integration. arXiv preprint arXiv:2410.15644, 2024
arXiv 2024
-
[3]
Joon Sung Park, Joseph C. O’Brien, Carrie J. Cai, Meredith Ringel Morris, Percy Liang, and Michael S. Bernstein. Generative agents: Interactive simulacra of human behavior. arXiv preprint arXiv:2304.03442v2, Aug 2023
arXiv 2023
-
[4]
LLM-Driven NPCs: Cross-Platform Dialogue System for Games and Social Platforms
Li Song. LLM-Driven NPCs: Cross-Platform Dialogue System for Games and Social Platforms. arXiv preprint arXiv:2504.13928, 2025
work page Pith review arXiv 2025
-
[5]
Character generalization: A study on out-of-domain role-playing dialogue agents
Xuefeng Li et al. Character generalization: A study on out-of-domain role-playing dialogue agents. arXiv preprint arXiv:2501.15427, 2025
arXiv 2025
-
[6]
A quest for information: Enhancing game-based learning with LLM-Driven NPCs
Tereza Todova and Peter Bruza. A quest for information: Enhancing game-based learning with LLM-Driven NPCs. In Proceedings of CESCG 2025: The 29th Central European Seminar on Computer Graphics , 2025
work page 2025
-
[7]
K. I. Gero et al. Procedural Artificial Narrative using Generative AI (PANGeA). In AAAI Conference on Artificial Intelligence and Interactive Digital Entertainment (AIIDE), 2024
work page 2024
-
[8]
CivAgent: A Human-like Agent for Complex Strategy Games
Chen Wang et al. CivAgent: A Human-like Agent for Complex Strategy Games. arXiv preprint arXiv:2502.20807, 2025
arXiv 2025
Show all 35 references
-
[9]
Magical Mike Senpai, Here My Spell Is
Relu Games. Magical Mike Senpai, Here My Spell Is. https://store. steampowered.com/app/2696400, 2025
2025
-
[10]
Uncover The Smoking Gun
Relu Games. Uncover The Smoking Gun. https://store.steampowered.com/ app/2492290, 2025
2025
-
[11]
SmartJoy in inZOI: LLM-based Live Simulation Engine
Jaewoong Cho. SmartJoy in inZOI: LLM-based Live Simulation Engine. Presen- tation at Game Developers Conference (GDC) 2025. Information based on report from Eneba Hub News [3], 2025
2025
-
[12]
Leveraging large language models for active merchant non-player characters
Byungjun Kim, Minju Kim, Dayeon Seo, and Bugeun Kim. Leveraging large language models for active merchant non-player characters. arXiv preprint arXiv:2412.11189v2, Dec 2024
2024 arXiv
-
[13]
RPGBench: A Comprehensive Benchmark for Evaluating Large Language Models as Role-Playing Game Engines
Zhi Yuan et al. RPGBench: A Comprehensive Benchmark for Evaluating Large Language Models as Role-Playing Game Engines. arXiv preprint arXiv:2502.00595, 2024
2024 arXiv
-
[14]
The effect of llm-based npc emotional states on player emotions: An analysis of interactive game play
Alessandro Marincioni, Myriana Miltiadous, Katerina Zacharia, Rick Heemskerk, Georgios Doukeris, Giulio Barbero, and Mike Preuss. The effect of llm-based npc emotional states on player emotions: An analysis of interactive game play. In 2024 IEEE Conference on Games (CoG) , pag...
2024
-
[15]
Exploring presence in interactions with llm-driven npcs: A comparative study of speech recognition and dialogue options
Frederik Roland Christiansen, Linus Nørgaard Hollensberg, Niko Bach Jensen, Kristian Julsgaard, Kristian Nyborg Jespersen, and Ivan Nikolov. Exploring presence in interactions with llm-driven npcs: A comparative study of speech recognition and dialogue options. In 30th ACM Sym...
2024
-
[16]
LLM-Stakeholders Interactive Negotiation: A Benchmark for Cooperation, Competition, and Maliciousness
Shahin Abdelnabi et al. LLM-Stakeholders Interactive Negotiation: A Benchmark for Cooperation, Competition, and Maliciousness. arXiv preprint [6, 7] , 2024
2024
-
[17]
AgreeMate: Teaching LLMs to Haggle
Ainesh Chatterjee, Samuel Miller, and Nithin Parepally. AgreeMate: Teaching LLMs to Haggle. arXiv preprint arXiv:2412.18690, 2024
2024 arXiv
-
[18]
item_id",
Yujie Feng, Zexin Lu, Bo Liu, Li-Ming Zhan, and Xiao-Ming Wu. Instruction- Tuning for Dialogue State Tracking. OpenReview [4], 2023. Minkyung Kim, Junsik Kim, Hwidong Bae, Woongcheol Yang, Sangdon Park, and Sohee Bae A Appendix: NPC Prompt NPC Prompt Template <SYSTEM_INSTRUCTI...
2023
-
[19]
- Trading proposals are absolutely prohibited in this context
General conversation: - context_type: NONE - Converse with the player about the game world, character background, or items from <GAME_ITEMS_LIST>. - Trading proposals are absolutely prohibited in this context
-
[20]
End conversation: - context_type: END_CONVERSATION - When the player is rude or the conversation naturally concludes
-
[21]
</CONTEXT_GUIDELINES> <TRADE_GUIDELINES> - In trading context, strictly follow the following trade flow
Trading: - context_type: TRADE - context_subtype: Refer to <TRADE_GUIDELINES>. </CONTEXT_GUIDELINES> <TRADE_GUIDELINES> - In trading context, strictly follow the following trade flow. - Identify the most recent trading sub-context from <DIALOGUE_HISTORY>. - In this prompt, ’sh...
-
[22]
Here are sturdy ropes, health potions, etc
When NPC shows items: - Only select items where item_name can be completely found in <CHARACTER_INVENTORY>and quantity >0. If not found, mention unavailability for sale. - Describe characteristics, uses, and quality of selected valid items. Do not mention prices unless asked. ...
-
[23]
__PRICE__
When player shows purchase intent and the ’shopping cart’ is newly formed or changed: - Regardless of the last trading sub-context, generate OFFER_SELL response. - Only select items where item_name can be completely found in <CHARACTER_INVENTORY>and quantity >0. Describe quali...
-
[24]
- Respond with {character_name}’s final selling price as sale_price
When attempting price negotiation with last trading sub-context being OFFER_SELL or NEGOTIATE_PRICE: - Negotiate or refuse based on character personality. - Respond with {character_name}’s final selling price as sale_price. - context_type: TRADE, context_subtype: NEGOTIATE_PRICE
-
[25]
So, will you buy it?
When last trading sub-context is OFFER_SELL or NEGOTIATE_PRICE and player gives positive response: - Must generate CHECK_CONFIRMATION response. Do not omit. Must end conversation with a question (re)confirming the purchase (e.g., "So, will you buy it?"). - If player gives tip ...
-
[26]
Yes, let’s proceed with the trade
When last trading sub-context is CHECK_CONFIRMATION and player gives positive response: - Be sure to check if the last trading sub-context is CHECK_CONFIRMATION. - Even if player responds "Yes, let’s proceed with the trade", "I’ll pay", "I’ll buy", etc., if the last trading su...
-
[27]
last_trade_context (string): Last trading context, respond with empty string if not confirmed
-
[28]
context_reason (string): Context summary
-
[29]
NONE", "TRADE
context_type (string): "NONE", "TRADE", "END_CONVERSATION"
-
[30]
SHOW_INVENTORY
context_details (object): - When context_type is NONE or END_CONVERSATION: Prohibited to create fields - When context_type is TRADE: - context_subtype (string): "SHOW_INVENTORY", "OFFER_SELL", "NEGOTIATE_PRICE", "CHECK_CONFIRMATION", "CON- FIRM_SELL", "REJECT_TRADE". - items (...
-
[31]
item_id",
npc_dialogue (string): Natural conversation, reflecting items. </RESPONSE_FORMAT> <RESPONSE_GUIDELINES> - Respond as ’{character_name}’, reflecting character’s personality, emotions, and background. - Complete colloquial style, no parentheses. - All responses must be in comple...
-
[32]
Try price negotiation
Optional Action Types: a. Try price negotiation. You may attempt persistent or rude negotiation. b. Try adding items to purchase before completing the transaction. (Randomly select from <GAME_ITEMS_LIST>.) c. If the NPC rejects the trade, you may respond rudely. d. You may sim...
-
[33]
Termination Conditions: If <DIALOGUE_HISTORY>contains CONFIRM_SELL or END_CONVERSATION, your next output must be “End"
Mandatory Action Rules: You must strictly follow these rules. Termination Conditions: If <DIALOGUE_HISTORY>contains CONFIRM_SELL or END_CONVERSATION, your next output must be “End". </DIALOGUE_GUIDELINES> <DIALOGUE_HISTORY> {formatted_history} </DIALOGUE_HISTORY> Player: C App...
-
[34]
Ask detailed questions about the price, performance, usage methods, etc
Optional Action Types: a. Ask detailed questions about the price, performance, usage methods, etc. of recommended items. b. If there are items among the recommended ones that you like, express your purchase intent. c. Try price negotiation. d. If the NPC rejects the trade, you...
-
[35]
Termination Conditions: If <DIALOGUE_HISTORY>contains CONFIRM_SELL or END_CONVERSATION, your next output must be “End"
Mandatory Action Rules: You must strictly follow these rules. Termination Conditions: If <DIALOGUE_HISTORY>contains CONFIRM_SELL or END_CONVERSATION, your next output must be “End". </DIALOGUE_GUIDELINES> <DIALOGUE_HISTORY> {formatted_history} </DIALOGUE_HISTORY> Player:
Reviewed August 6, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.