Pith. sign in

REVIEW 3 major objections 4 minor 51 references

Integrating Neural and Symbolic Components in a Model of Pragmatic Question-Answering

T0 review · 3 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that LLM proposers and evaluators inside a probabilistic cognitive model can match or exceed the original model's fit to human question-answering data, with truth-conditional semantics as the weak link.

desk verdict A useful systematic comparison of LLM/PCM hybrids, but the 'open-ended' headline is held up by manual response supplementation. read the letter →

arxiv 2506.01474 v1 pith:PUWVAYOY submitted 2025-06-02 cs.CL

Polina Tsvilodub , Robert D. Hawkins , Michael Franke This is my paper
classification cs.CL
keywords neuro-symbolicmodelspragmaticquestionansweringprobabilisticcognitivelargelanguagerationalspeechactLLMproposersevaluatorsoverinformativeanswers
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to show that hand-built probabilistic models of pragmatic question answering can be opened up to natural language by handing specific subtasks to large language models (LLMs) while keeping the symbolic reasoning scaffold intact. It tests a family of such hybrid models on a classic scenario: a person asks for an unavailable item and the respondent mentions a relevant alternative. Across systematic variations, several hybrids match or beat the original hand-specified model in predicting which response category humans produce. The paper also claims that the integration style matters: LLMs work best as proposers of alternative responses, questions, and goals, and as evaluators of utility, but are the weak link for truth-conditional semantic evaluation. If true, this is a path toward cognitive models that handle open-ended language instead of fixed vignettes.

What carries the argument

The central object is the QA model, a probabilistic cognitive model that formalizes pragmatic question answering as recursive reasoning: a base-level respondent answers literally, a pragmatic questioner chooses a question by expected utility over a decision problem, and a pragmatic respondent infers the questioner's decision problem and chooses an answer that maximizes expected utility plus informativeness. The neuro-symbolic variants replace discrete components of this model, namely the utility function, literal semantics, and the sets of possible questions, answers, and goals, with LLM calls, while leaving the Bayesian inference steps symbolic. The key contrast is between models with LLM evaluators (utility and semantics) and LLM proposers (alternatives and goals), and between decomposed multiple-LLM architectures and a single monolithic prompt.

What would settle it

Run a monolithic one-shot LLM and the decomposed neuro-symbolic model with matched total token or compute budgets and compare Jensen-Shannon divergence to human response categories; if the monolithic model matches the decomposed model's fit once budgets are equal, the paper's scaffolding explanation is undermined.

Watch

Extended reading notes

Core claim

The central claim is that a neuro-symbolic model, in which LLM modules propose alternatives and evaluate utility while symbolic Bayesian inference over those outputs is preserved, can match or exceed the traditional probabilistic model's fit to human answer patterns. In the main comparison, the fully decomposed neuro-symbolic model and a prompt-based questioner conditioned on inferred goals reach or exceed the probabilistic baseline in Jensen-Shannon improvement over chance. The paper further finds that the way LLMs are integrated is decisive: they are reliable for generating alternative utterances and questions, for proposing plausible questioner goals, and for rating the utility of options (correlating with human utility ratings at R = 0.92), but LLM-based semantic evaluation of the pragmatic respondent's answers overpredicts unrelated responses. This leads the authors to identify truth-conditional semantic evaluation as the bottleneck of the neuro-symbolic approach.

Load-bearing premise

The load-bearing premise is that the improved fit of the scaffolded models comes from the cognitive task decomposition, not from the larger inference-time compute budget that multiple LLM calls consume.

Editorial extensions

If this is right

  • Hand-specified sets of utterances, questions, and goals can be replaced by LLM proposals without sacrificing predictive fit.
  • LLM utility ratings correlate strongly with human utility ratings, so one human rating elicitation step can be automated.
  • Models with at least some task decomposition fit human data better than a single monolithic chain-of-thought prompt.
  • Truth-conditional semantic evaluation is the weak point; models using LLM semantics overpredict unrelated responses.
  • Open-source LLMs can reproduce the main pattern when formatting failures are controlled, suggesting the framework is not tied to one proprietary model.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the compute-budget confound is set aside, the results suggest a general design rule for neuro-symbolic cognitive models: use LLMs for generation and rating, and keep a symbolic or rule-based component for literal truth checking.
  • The goal-proposer analysis points to a testable bias: LLM-generated goals over-anchor on the target and underrepresent social goals, which could distort respondent inferences in domains where social reasons dominate.
  • A natural extension is to replace fixed n=10 response sampling with sampling until all conceptual response types are covered, which would test whether proposer coverage, rather than the symbolic scaffold, drives the fit.
  • The improvement observed as more LLM modules are added within the PCM family suggests an open question: whether a fully open-ended model with no manually specified response categories can reach human-level fit once semantic evaluation is fixed.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper proposes a neuro-symbolic framework for pragmatic question-answering in which LLM modules replace or augment components of a Rational Speech Act-style probabilistic cognitive model (PCM). The authors systematically vary which components are neural (utility evaluation, literal semantics, proposers of responses/questions/goals, prompt-based questioner, one-shot chain-of-thought) and compare model predictions to human response-category proportions using Jensen-Shannon divergence. The central claim is that hybrid models can match or exceed the traditional PCM while 'eliminating the need for manual specification.' The paper also includes qualitative module analyses and an appendix with an open-source LLM (Qwen) replication. The main empirical finding is that the 'full NeSy' model and the 'prompt-based questioner with goals' achieve the best JSD fit, with several other scaffolded models close to the PCM baseline.

Significance. If the central claims hold, this is a valuable contribution to computational pragmatics: it provides a systematic architecture for combining LLMs with probabilistic cognitive models, tests multiple integration points, and validates an LLM utility evaluator against human ratings (R = 0.92). The paper is transparent about many limitations, includes prompts and materials, and reports an open-source LLM replication, which strengthens reproducibility. However, the flagship 'full NeSy' model's best fit relies on manual supplementation of the response set, directly undercutting the abstract's 'eliminating the need for manual specification' claim. The compute-budget confound is acknowledged but not addressed, and the headline comparisons lack inferential statistics. These issues affect the strength of the central contribution as currently stated, although they are addressable within the manuscript's scope.

major comments (3)
  1. [§3.3, Abstract] The abstract claims that the framework eliminates the need for manual specification, but the best-performing 'full NeSy' model depends on exactly the kind of manual specification the paper aims to avoid. In Section 3.3, the authors state that because the LLM response proposer with n=10 was 'unlikely to contain' the 'no-options' and 'all-options' response types, they 'appended these two response types to set of alternatives manually nonetheless.' These two rule-based additions are two of the five human response categories, so the model's ability to match human data in Figure 2 is not a clean test of open-ended generation of utterance sets. The 'prompt-based questioner with goals' model also uses the pre-specified symbolic response set from the PCM. The open-endedness claim should be substantially tempered, or the authors should provide a version of the full NeSy model whose response types are generated entirely by the LLM proposer (e.g., with larger n or adjusted prompting) and show that the fit is preserved.
  2. [§6 (last paragraph)] The paper acknowledges that 'one other potential source of improved performance with scaffolding of the LLM could be due to higher inference time compute budget that comes with decomposing the task into several LLM calls.' Since the monolithic one-shot CoT model uses a single LLM call while scaffolded models use several, the observed improvement in fit may be driven by additional compute rather than by the cognitive task decomposition. This confound is not controlled anywhere in the manuscript: there is no matching of total tokens, API calls, or sampling effort across conditions. The claim that task decomposition, rather than compute, drives the improvement is therefore unsupported. I recommend adding a control condition (e.g., sampling multiple responses from the monolithic model, or otherwise equating compute) or rephrasing the conclusion to present the result as a comparison of architectures under unequal compute budgets.
  3. [§4, Figure 2] The central quantitative comparisons are based on means and 95% bootstrapped confidence intervals over only five simulation iterations, with no inferential statistics. The statement in the abstract that hybrid models 'can match or exceed the performance of traditional probabilistic models' is stronger than what the data show: in Figure 2, the CIs of the best models overlap with or lie near the PCM reference line, and differences among models are not tested. I ask for either a permutation or bootstrap significance test on the JSD differences, or a rephrasing of the claim to 'comparable to' rather than 'match or exceed.' Reporting per-item JSD distributions would also help assess whether the aggregate fit is driven by a few vignettes.
minor comments (4)
  1. [§3.1] The response category labels ('target,' 'similar,' 'unrelated,' 'all options,' 'no options') are introduced in the text but Figure 3's category names should be explicitly matched to these labels to avoid confusion, especially for readers not familiar with Tsvilodub et al. (2023).
  2. [§4, Formula for Δ] The definition Δ_i = JSD(B, humans) − JSD(M_i, humans) is clear, but the text should state explicitly that higher Δ means closer to human data (it is implied by the caption) and report the uncertainty in Δ (e.g., bootstrap CIs on the difference rather than on each JSD separately).
  3. [§3.3 and Appendix D] The main results rely on a single closed-source LLM (GPT-4o-mini), and the open-source replication in Appendix D does not report results for the full neuro-symbolic model due to parsing errors. This limitation is acknowledged in the Discussion, but it would strengthen the paper to state in Section 4 that the headline 'full NeSy' result is specific to GPT-4o-mini and has not been replicated with an open-source model.
  4. [§6] The phrase 'open-ended' is used both for the framework's goal (in principle unbounded sets) and for the actual implementations (n=10 or n=3 proposals with manual supplementation). Clarifying this distinction early would avoid overreading the results as demonstrating fully open-ended generation.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the LLM-based modules are zero-shot, no parameter is fitted to the target human answer distribution, and the hand-supplemented response categories alter availability, not the predicted proportions.

full rationale

The paper's derivation chain is self-contained against an external benchmark. The target human answer proportions come from Tsvilodub et al. (2023), and the neuro-symbolic models are evaluated by Jensen-Shannon divergence to those proportions; no model parameter is fitted to the human answer data. The utility evaluator is validated against independent human utility ratings (R=0.92), not used to define the target response categories. The probabilistic machinery (Bayesian inference, soft-max utilities, KL informativity) is specified in Appendix A and executed symbolically, with LLM modules supplying proposals and evaluations zero-shot. The one point that resembles an input/output overlap is the manual append of 'no-options' and 'all-options' responses to the proposer's sampled alternatives (Section 3.3). This does affect the open-endedness claim, but it is not a circular reduction: the appended items merely make those response categories available; their predicted frequencies are still computed by the QA model's decision-theoretic reasoning and are not set equal to human frequencies. The paper itself acknowledges a related leakage concern, that LLM proposals may contain a learned prior over human preferences from training data; this is an external generalizability caveat, not an in-paper construction in which a fitted input is renamed a prediction. Self-citations to the QA model scaffolding (Hawkins et al., to appear) and to the human-data paper (Tsvilodub et al., 2023) are load-bearing but are independent published results, and the scaffolding is additionally supported by a new explanation experiment reported in Section 2. The acknowledged compute-budget confound is likewise an external validity threat, not a circular derivation. Hence no step in the derivation reduces by definition to its own inputs.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

The framework introduces no new physical or theoretical entities. It depends on the cognitive validity of the QA model scaffold, on treating LLM outputs as cognitive estimates, and on the completeness of the human response categories. Several hand-set modeling choices (n proposals, temperature, manual response supplementation) are free parameters that could influence the reported fit.

free parameters (5)
  • response proposal count n = 10
    Hand-set number of alternatives sampled from the LLM response proposer; the authors report n<10 risks missing response categories and n=50 is computationally expensive.
  • question proposal count n = 3
    Hand-set number of alternative questions sampled from the LLM question proposer.
  • LLM sampling temperature = 0.1
    Fixed temperature for GPT-4o-mini modules; affects stochastic outputs.
  • simulation iterations = 5
    Number of stochastic simulation runs averaged; no seed is reported.
  • manual response-type supplementation = rule-based append after qualitative inspection
    'no-options' and 'all-options' responses are appended to the LLM proposal set to ensure all human response categories are representable, potentially inflating the full NeSy model's fit.
assumptions (5)
  • domain assumption The QA model's recursive decomposition (base-level respondent, pragmatic questioner, pragmatic respondent) accurately describes human pragmatic answering.
    Used as scaffolding for all neuro-symbolic variants; only mildly supported by the exploratory explanation study in Section 2, where goal-based explanations made up 0.43 of responses.
  • domain assumption LLM modules yield valid estimates of the cognitive quantities they replace (utilities, truth values, proposal distributions, question likelihoods).
    Utility ratings correlate with human ratings (R = 0.92), semantic evaluators are checked qualitatively, but question likelihoods from the prompt-based questioner are never validated.
  • domain assumption The human response categories (target, similar, unrelated, all, no options) from Tsvilodub et al. 2023 are exhaustive and correctly coded.
    All model fits are computed against these proportions; the preprint does not report N, coding reliability, or per-category uncertainty.
  • ad hoc to paper Flat priors and zero utterance costs are appropriate for the respondent's utility function.
    Stated in Section 3.3: 'We assume flat priors and no utterance costs throughout the model.' This simplifies but may not reflect human production preferences.
  • ad hoc to paper The improved fit of scaffolded models is attributable to task decomposition rather than to higher inference-time compute from multiple LLM calls.
    The authors mention this alternative in Section 6 citing Yu et al. 2024, but no control or ablation for compute budget is reported.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Integrating Neural and Symbolic Components in a Model of Pragmatic Question-Answering." pith.science (2026). https://pith.science/paper/PUWVAYOY

@misc{pith2026250601474,
  author       = {Pith},
  title        = {Pith review of: Integrating Neural and Symbolic Components in a Model of Pragmatic Question-Answering},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/PUWVAYOY}},
  note         = {Machine review of arXiv:2506.01474}
}
read the original abstract

Computational models of pragmatic language use have traditionally relied on hand-specified sets of utterances and meanings, limiting their applicability to real-world language use. We propose a neuro-symbolic framework that enhances probabilistic cognitive models by integrating LLM-based modules to propose and evaluate key components in natural language, eliminating the need for manual specification. Through a classic case study of pragmatic question-answering, we systematically examine various approaches to incorporating neural modules into the cognitive model -- from evaluating utilities and literal semantics to generating alternative utterances and goals. We find that hybrid models can match or exceed the performance of traditional probabilistic models in predicting human answer patterns. However, the success of the neuro-symbolic model depends critically on how LLMs are integrated: while they are particularly effective for proposing alternatives and transforming abstract goals into utilities, they face challenges with truth-conditional semantic evaluation. This work charts a path toward more flexible and scalable models of pragmatic language use while illuminating crucial design considerations for balancing neural and symbolic components.

Figures

Figures reproduced from arXiv: 2506.01474 by the authors.

Figure 1
Figure 1. Probabilistic cognitive model (PCM) of prag [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Upper panel: Improvement of the model fit to human data in terms of Jensen-Shannon divergence over a uniform response distribution baseline (higher is better, y-axis) of all analyzed models (x-axis). The horizontal line indicates performance of the probabilistic model. Dots indicate the means across simulations, error bars indicate 95% bootstrapped CIs. Lower panel: Overview of tested models. Each box shows a schema… view at source ↗
Figure 3
Figure 3. Proportions of different response categories produced by humans (left column) and predicted by different [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (9 more)
Figure 4
Figure 4. Figure 4: Formal definitions of the pragmatic questioner [PITH_FULL_IMAGE:figures/full_fig_p012_4.png]
Figure 5
Figure 5. Figure 5: Left: GPT-4o-mini utilities plotted against human utilities, R = 0.92. Right: Qwen-2.5-32B-Instruct utilities plotted against human utilities, R = 0.93. 1-shot CoT 1-shot example 1-shot explanation 0-shot Qwen-2.5-32B-Instruct 0.21 0.15 0.25 0.28 Qwen-2.5-14B-Instruct …
Figure 7
Figure 7. Figure 7: Base-level Evaluator Prompt [PITH_FULL_IMAGE:figures/full_fig_p013_7.png]
Figure 11
Figure 11. Figure 11: Goal Proposer Prompt Prompt-based questioner with goals 1 We are interested in how likely a person would be to ask the following question in a simple context , given their goal . 2 Please return only the likelihood , provided on a scale between 0 and 1. 3 Goal : { goa…
Figure 12
Figure 12. Figure 12: Prompt-based questioner with goals Prompt-based questioner without goals 1 We are interested in how likely a person would be to ask the following question in a simple context . 2 Please return only the likelihood , provided on a scale between 0 and 1. 3 Context : { st…
Figure 13
Figure 13. Figure 13: Prompt-based questioner without goals [PITH_FULL_IMAGE:figures/full_fig_p014_13.png]
Figure 14
Figure 14. Figure 14: One-shot chain-of-thought prompt D Simulation Results with an Open-Source LLM Additionally to the main experiments performed with GPT-4o-mini, we ran all experiments with an open-source LLM — Qwen-2.5-32B-Instruct (Team, 2024), providing insights about advantages and …
Figure 15
Figure 15. Figure 15: Improvement of the fit to human data of a model with an open-source Qwen-2.5-32B-Instruct backbone [PITH_FULL_IMAGE:figures/full_fig_p016_15.png]
Figure 16
Figure 16. Figure 16: Proportions of different response categories predicted by Qwen-2.5-32B-Instruct used in different models [PITH_FULL_IMAGE:figures/full_fig_p016_16.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

51 extracted references · 32 canonical work pages

  1. [1]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Chinmaya Andukuri, Jan-Philipp Fr \"a nken, Tobias Gerstenberg, and Noah Goodman. 2024. https://openreview.net/forum?id=CrzAj0kZjR ST ar- GATE : Teaching language models to ask clarifying questions . In First Conference on Language Modeling

    work page 2024

  4. [4]

    Sebastian Bader, Pascal Hitzler, and Steffen Hoelldobler. 2004. https://arxiv.org/abs/cs/0408069 The integration of connectionism and first-order knowledge representation and reasoning as a challenge for artificial intelligence . Preprint, arXiv:cs/0408069

    work page Pith review arXiv 2004

  5. [5]

    Anna Bavaresco, Raffaella Bernardi, Leonardo Bertolazzi, Desmond Elliott, Raquel Fern \'a ndez, Albert Gatt, Esam Ghaleb, Mario Giulianelli, Michael Hanna, Alexander Koller, et al. 2024. LLMs instead of human judges? A large scale empirical study across 20 NLP evaluation tasks . URL https://arxiv. org/abs/2406.18403

    arXiv 2024

  6. [6]

    Bender, Timnit Gebru, Angelina McMillan-Major, and Margaret Mitchell

    Emily M. Bender, Timnit Gebru, Angelina McMillan-Major, and Margaret Mitchell. 2021. https://doi.org/10.1145/3442188.3445922 On the dangers of stochastic parrots: Can language models be too big? In Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency, FAccT '21, page 610–623, New York, NY, USA. Association for Computing Machinery

    arXiv 2021

  7. [7]

    Anton Benz. 2006. Utility and relevance of answers. Springer

    work page 2006

  8. [8]

    Bikram Pratim Bhuyan, Amar Ramdane-Cherif, Ravi Tomar, and TP Singh. 2024. Neuro-symbolic artificial intelligence: a survey. Neural Computing and Applications, pages 1--36

    work page 2024

Show all 51 references
  1. [9]

    Herbert H Clark. 1979. Responding to indirect speech acts. Cognitive psychology, 11(4):430--477

    1979

  2. [10]

    Collins, Catherine Wong, Jiahai Feng, Megan Wei, and Joshua B

    Katherine M. Collins, Catherine Wong, Jiahai Feng, Megan Wei, and Joshua B. Tenenbaum. 2022. https://arxiv.org/abs/2205.05718 Structured, flexible, and robust: benchmarking and improving large language models towards more human-like behavior in out-of-distribution reasoning ta...

    2022 arXiv

  3. [11]

    Antonia Creswell, Murray Shanahan, and Irina Higgins. 2022. Selection-inference: Exploiting large language models for interpretable logical reasoning. arXiv preprint arXiv:2205.09712

    2022 arXiv

  4. [12]

    Judith Degen. 2023. The rational speech act framework. Annual Review of Linguistics, 9(1):519--540

    2023

  5. [13]

    Simon Farrell and Stephan Lewandowsky. 2018. Computational modeling of cognition and behavior. Cambridge University Press

    2018

  6. [14]

    Michael C Frank and Noah D Goodman. 2012. Predicting pragmatic reasoning in language games. Science, 336(6084):998--998

    2012

  7. [15]

    Michael Franke, Polina Tsvilodub, and Fausto Carcassi. 2024. Bayesian statistical modeling with predictors from LLM s. arXiv preprint arXiv:2406.09012

    2024 arXiv

  8. [16]

    Noah D Goodman and Andreas Stuhlm\" u ller. 2014. The Design and Implementation of Probabilistic Programming Languages . http://dippl.org. Accessed: 2025-1-30

    2014

  9. [17]

    Jeroen Antonius Gerardus Groenendijk and Martin Johan Bastiaan Stokhof. 1984. Studies on the Semantics of Questions and the Pragmatics of Answers. Ph.D. thesis, Univ. Amsterdam

    1984

  10. [18]

    Auli Hakulinen. 2001. Minimal and non-minimal answers to yes-no questions. Pragmatics, 11(1):1--15

    2001

  11. [19]

    CL Hamblin. 1973. Questions in M ontague E nglish. Foundations of Language, 10(1):41--53

    1973

  12. [20]

    Hawkins and Noah D

    Robert D. Hawkins and Noah D. Goodman. 2017. Why do you ask? T he informational dynamics of questions and answers. PsyArXiv

    2017

  13. [21]

    Hawkins, Andreas Stuhlm \"u ller, Judith Degen, and Noah D

    Robert D. Hawkins, Andreas Stuhlm \"u ller, Judith Degen, and Noah D. Goodman. 2015. https://api.semanticscholar.org/CorpusID:4687637 Why do you ask? G ood questions provoke informative answers . Cognitive Science

    2015

  14. [22]

    Hawkins, Polina Tsvilodub, Claire Augusta Bergey, Noah D

    Robert D. Hawkins, Polina Tsvilodub, Claire Augusta Bergey, Noah D. Goodman, and Michael Franke. to appear. Relevant answers to polar questions. Philosophical Transactions B

  15. [23]

    Joy He-Yueya, Gabriel Poesia, Rose E Wang, and Noah D. Goodman. 2023. Solving math word problems by combining language models with symbolic solvers. arXiv preprint arXiv:2304.09102

    2023 arXiv

  16. [24]

    Jennifer Hu, Sammy Floyd, Olessia Jouravlev, Evelina Fedorenko, and Edward Gibson. 2023. https://doi.org/10.18653/v1/2023.acl-long.230 A fine-grained comparison of pragmatic language understanding in humans and language models . In Proceedings of the 61st Annual Meeting of the...

    2023 doi

  17. [25]

    Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang, Andrea Madotto, and Pascale Fung. 2023. https://doi.org/10.1145/3571730 Survey of hallucination in natural language generation . ACM Comput. Surv., 55(12)

    2023 doi

  18. [26]

    Subbarao Kambhampati, Karthik Valmeekam, Lin Guan, Mudit Verma, Kaya Stechly, Siddhant Bhambri, Lucas Saldyt, and Anil Murthy. 2024. https://arxiv.org/abs/2402.01817 LLM s can't plan, but can help planning in LLM - M odulo frameworks . Preprint, arXiv:2402.01817

    2024 arXiv

  19. [27]

    Alexander K Lew, Michael Henry Tessler, Vikash K Mansinghka, and Joshua B Tenenbaum. 2020. Leveraging unstructured statistical knowledge in a probabilistic language of thought. In Proceedings of the annual conference of the cognitive science society

    2020

  20. [28]

    Lew, Tan Zhi-Xuan, Gabriel Grand, and Vikash K

    Alexander K. Lew, Tan Zhi-Xuan, Gabriel Grand, and Vikash K. Mansinghka. 2023. https://arxiv.org/abs/2306.03081 Sequential M onte C arlo steering of large language models using probabilistic programs . Preprint, arXiv:2306.03081

    2023 arXiv

  21. [29]

    u ttler, Mike Lewis, Wen-tau Yih, Tim Rockt \

    Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich K \"u ttler, Mike Lewis, Wen-tau Yih, Tim Rockt \"a schel, et al. 2020. Retrieval-augmented generation for knowledge-intensive NLP tasks. Advances in Neural Information Proc...

    2020

  22. [30]

    Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang

    Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. 2023. https://arxiv.org/abs/2307.03172 Lost in the middle: How language models use long contexts . Preprint, arXiv:2307.03172

    2023 arXiv

  23. [31]

    Lea L \"o hn, Niklas Kiehne, Alexander Ljapunov, and Wolf-Tilo Balke. 2024. https://aclanthology.org/2024.inlg-main.19/ Is machine psychology here? O n requirements for using human psychological tests on large language models . In Proceedings of the 17th International Natural ...

    2024

  24. [32]

    Maxwell Nye, Michael Tessler, Josh Tenenbaum, and Brenden M Lake. 2021. Improving coherence and consistency in neural sequence models with dual-system, neuro-symbolic reasoning. Advances in Neural Information Processing Systems, 34:25192--25204

    2021

  25. [33]

    Kathryn Pruitt and Floris Roelofsen. 2011. Disjuntive questions: P rosody, syntax, and semantics. Handout, G\"ottingen

    2011

  26. [34]

    Ben Prystawski, Paul Thibodeau, Christopher Potts, and Noah Goodman. 2023. Psychologically-informed chain-of-thought prompts for metaphor understanding in large language models. In Proceedings of the Annual Meeting of the Cognitive Science Society, volume 45

    2023

  27. [35]

    Anselm Rothe, Brenden M Lake, and Todd Gureckis. 2017. Question asking as program generation. Advances in neural information processing systems, 30

    2017

  28. [36]

    Laura Ruis, Akbir Khan, Stella Biderman, Sara Hooker, Tim Rockt \"a schel, and Edward Grefenstette. 2023. The goldilocks of pragmatic understanding: Fine-tuning strategy matters for implicature resolution by LLM s

    2023

  29. [37]

    Timo Schick, Jane Dwivedi-Yu, Roberto Dessì, Roberta Raileanu, Maria Lomeli, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. 2023. https://arxiv.org/abs/2302.04761 Toolformer: Language models can teach themselves to use tools . Preprint, arXiv:2302.04761

    2023 arXiv

  30. [38]

    Richard Shiffrin and Melanie Mitchell. 2023. Probing the psychology of AI models. Proceedings of the National Academy of Sciences, 120(10):e2300963120

    2023

  31. [39]

    Jon Scott Stevens, Anton Benz, Sebastian Reu e, and Ralf Klabunde. 2016. Pragmatic question answering: A game-theoretic approach. Data & Knowledge Engineering, 106:52--69

    2016

  32. [40]

    Theodore R Sumers, Shunyu Yao, Karthik Narasimhan, and Thomas L Griffiths. 2023. Cognitive architectures for language agents. arXiv preprint arXiv:2309.02427

    2023 arXiv

  33. [41]

    Qwen Team. 2024. https://qwenlm.github.io/blog/qwen2.5/ Qwen2.5: A party of foundation models

    2024

  34. [42]

    Polina Tsvilodub, Michael Franke, and Fausto Carcassi. 2024 a . https://openreview.net/forum?id=gnGhkVfhje Cognitive modeling with scaffolded LLM s: A case study of referential expression generation . In ICML 2024 Workshop on LLMs and Cognition

    2024

  35. [43]

    Polina Tsvilodub, Michael Franke, Robert Hawkins, and Noah D. Goodman. 2023. Overinformative question answering by humans and machines. In Proceedings of the 45th Annual Conference of the Cognitive Science Society. Cognitive Science Society

    2023

  36. [44]

    Polina Tsvilodub, Paul Marty, Sonia Ramotowska, Jacopo Romoli, and Michael Franke. 2024 b . Experimental pragmatics with machines: Testing LLM predictions for the inferences of plain and embedded disjunctions. In Proceedings of CogSci , pages 3960--3967

    2024

  37. [45]

    Robert van Rooy. 2003. Questioning to resolve decision problems. Linguistics and Philosophy, 26(6):727--763

    2003

  38. [46]

    Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. 2023. https://arxiv.org/abs/2203.11171 Self-consistency improves chain of thought reasoning in language models . Preprint, arXiv:2203.11171

    2023 arXiv

  39. [47]

    Julia White, Jesse Mu, and Noah D. Goodman. 2020. https://arxiv.org/abs/2006.00418 Learning to refer informatively by amortizing pragmatic reasoning . Preprint, arXiv:2006.00418

    2020 arXiv

  40. [48]

    Lew, Noah D

    Li Siang Wong, Gabriel Grand, Alexander K. Lew, Noah D. Goodman, Vikash K. Mansinghka, Jacob Andreas, and Joshua B. Tenenbaum. 2023. https://api.semanticscholar.org/CorpusID:259224900 From word models to world models: Translating from natural language to the probabilistic lang...

    2023 arXiv

  41. [49]

    Griffiths, Yuan Cao, and Karthik Narasimhan

    Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L. Griffiths, Yuan Cao, and Karthik Narasimhan. 2023. https://arxiv.org/abs/2305.10601 Tree of thoughts: Deliberate problem solving with large language models . Preprint, arXiv:2305.10601

    2023 arXiv

  42. [50]

    Ping Yu, Jing Xu, Jason Weston, and Ilia Kulikov. 2024. https://arxiv.org/abs/2407.06023 Distilling system 2 into system 1 . Preprint, arXiv:2407.06023

    2024 arXiv

  43. [51]

    Haiyan Zhao, Hanjie Chen, Fan Yang, Ninghao Liu, Huiqi Deng, Hengyi Cai, Shuaiqiang Wang, Dawei Yin, and Mengnan Du. 2023. Explainability for large language models: A survey. ACM Transactions on Intelligent Systems and Technology

    2023

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.