Pith. sign in

REVIEW 5 major objections 5 minor 3 cited by

Referential ambiguity and clarification requests: comparing human and LLM behaviour

T0 review · 5 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read In task-oriented Minecraft dialogues, humans rarely ask clarification questions about referential ambiguity, while LLMs ask them disproportionately; reasoning (trained or prompted) increases both frequency and relevance of model questions.

desk verdict Useful merged corpus and some real behavioral observations, but the central human/LLM asymmetry in the abstract is never actually measured. read the letter →

arxiv 2507.10445 v1 pith:5ZF22WC3 submitted 2025-07-14 cs.CL cs.AI

Chris Madge , Matthew Purver , Massimo Poesio This is my paper
classification cs.CLcs.AI
keywords referentialambiguityclarificationquestionstask-orienteddialogueMinecraftCorpuslargelanguagemodelschain-of-thoughtreasoningdiscoursedeixisspatial
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper asks a deceptively simple question: when a speaker's words can point to more than one thing, do dialogue partners ask for clarification? It builds a merged annotation of 100 Minecraft instruction-following dialogues that aligns reference/ambiguity labels (discourse and spatial deixis with multiple antecedents) with discourse-structure labels for clarification and confirmation questions. The answer it finds is surprising: humans almost never clarify referential ambiguity, and instead ask when the task itself is uncertain—orientation, color, whether to repeat a pattern. Large language models show the opposite pattern, asking more about referential ambiguity and less about task uncertainty; reasoning models and chain-of-thought prompting ask more questions overall and more relevant ones.

What carries the argument

The load-bearing piece is the merged MMAX corpus: MDC-R's reference annotations (which flag a referent as ambiguous when it carries both segment_phrase_antecedent_2 for discourse deixis or object2 for spatial deixis, i.e., two or more antecedents) are aligned with MSDC's SDRT annotations (which mark clarification and confirmation questions as discourse relations). This alignment lets the authors test whether the presence of an annotated ambiguity predicts a clarification question at or near that point, and lets them count how often each LLM approach responds with a question at those points. The evaluation also uses a single human coder to judge relevance for the spatial-deixis subset.

What would settle it

Count in the merged corpus how many of the human clarification/confirmation questions have as their direct subject a phrase that also carries a two-antecedent ambiguity attribute. The paper reports essentially a single such instance; a replication that finds many such overlaps, or a re-annotation that marks potential misunderstanding more broadly, would overturn the weak-link finding.

Watch

Extended reading notes

Core claim

Using the merged corpus, the paper finds that of 182 clarification questions and 218 confirmation questions in the 100-dialogue subset, only a single discourse-deixis ambiguity lies in the vicinity of a later clarification question; humans tend to act presumptively when language is ambiguous and reserve questions for task-level uncertainty such as orientation ('how should they be oriented') or property values ('what color blocks?'). LLMs, in contrast, produce direct disambiguating questions for expressions like 'the previous one' or 'the same thing on the other side,' and their overall tendency to ask any question rises from about 276–383 responses across the tested approaches. Reasoning-trained models (deepseek-r1:8b, llama3.3:70b) and chain-of-thought-prompted non-reasoning models (llama2:13b) ask more questions and yield higher counts of relevant spatial-deixis questions (up to 5 for llama3.3:70b) than plain non-reasoning models (0 for llama2:13b and humans).

Load-bearing premise

The paper's central comparison depends on treating a referent with two or more annotated antecedents as the operational definition of 'referential ambiguity' that should trigger clarification; if that annotation mostly captures deliberately vague, low-stakes phrases that humans reasonably leave unclarified, the weak human–ambiguity link is an artifact of the annotation rather than a fact about human behavior.

Editorial extensions

If this is right

  • Dialogue agents that treat referential ambiguity as the primary trigger for clarification will produce unnecessary questions in instruction-follower settings.
  • Human clarification behavior is driven by task-level uncertainty, so human-compatible agents should ask when task progress or properties are uncertain rather than when words are ambiguous.
  • Reasoning-oriented models and chain-of-thought prompting raise both question frequency and relevance, suggesting that clarification ability is tied to simulated reasoning.
  • The merged corpus gives a reusable testbed for studying clarification in situated, grounded dialogue.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If human question-asking is a metacognitive act aimed at task efficiency rather than linguistic repair, then the right test for LLM clarification is not "did it ask at the ambiguous mention?" but "did the question reduce wasted actions?"—a metric the paper explicitly leaves for future work.
  • The paper's one discovered propagation case (an ambiguous "the previous one" that led to a wrong action and a later confirmation) suggests a follow-up: measuring how often acted-on ambiguities actually cause task failure, which would quantify the utility of the LLM's extra questions.
  • The relevance evaluation covers only spatial deixis; extending it to discourse deixis (where the annotation is less literal) and to end-to-end task success could change the ranking of approaches.
  • The "humans act presumptively" strategy may be specific to this Minecraft build task and its low cost of correction; in higher-stakes tasks the human-to-ambiguity link could be stronger.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper introduces a merged version of the Minecraft Dialogue Corpus annotations (MDC-R reference annotations and MSDC SDRT annotations) in MMAX format, and uses it to compare how humans and several LLMs (with and without chain-of-thought prompting) produce clarification questions in response to referential ambiguity and task-based uncertainty. The central claims are (i) there is only a weak link between referential ambiguity and human clarification requests; (ii) there is a low correlation between human and LLM clarification behaviour; (iii) humans rarely ask clarification questions for referential ambiguity but often do so for task-based uncertainty, while LLMs show the reverse pattern; and (iv) reasoning-oriented models and CoT prompting increase question frequency and relevance. The paper reports qualitative examples and two quantitative tables: total question counts per model (Table 3) and counts of relevant spatial-deixis clarification questions per model (Table 4).

Significance. If the central asymmetry claim were supported, the paper would be a useful contribution to the study of clarification in task-oriented dialogue, showing that referential ambiguity detection alone does not predict human clarification behaviour and that reasoning-enabled LLMs tend to over-clarify referential ambiguity. The merged MDC-R/MSDC corpus is a concrete resource that can benefit future work on reference and clarification. The qualitative examples are instructive and the authors are transparent about some limitations (e.g., not measuring the utility of clarification). However, the paper's most distinctive quantitative claims about the human/LLM asymmetry and about correlation are not actually backed by the reported experiments, which limits the significance of the findings as they currently stand.

major comments (5)
  1. [Abstract and Section 5.3] The central claim that 'LLMs produce more clarification questions for referential ambiguity, but less so for task uncertainty' is not quantified anywhere. Table 4 only counts relevant spatial-deixis (referential) questions, and Table 3 aggregates all questions without distinguishing referential from task-based triggers. There is no count of task-uncertainty clarification questions for either humans or LLMs, so the 'conversely' half of the asymmetry is unsupported by data. The qualitative examples in Figures 5–7 actually show LLMs asking task-oriented questions (orientation, position, colour), which suggests that the claimed reverse pattern may not hold once question target type is classified. The authors should classify all model-generated and human clarification questions by target type and report the rates for both categories.
  2. [Abstract and Section 4] The abstract asserts 'low correlation between humans and LLMs', but no correlation statistic is ever reported. Section 4 only mentions checking 'correlation between clarification questions, and the instances of linguistic ambiguity', which is a different comparison. The paper does not state the correlation coefficient, the unit of analysis, or the confidence interval for any human/LLM correlation. To support this part of the abstract, the authors need to compute and report a correlation (e.g., per-dialogue or per-instance agreement between human and LLM question behaviour) or remove the claim.
  3. [Section 5.3, Table 4 and Methodology] The relevance counts in Table 4 are based on a single human coder with no inter-annotator agreement score or detailed annotation guidelines, as stated in Section 4 ('use a single human coder to examine specific instances of ambiguity and their responses'). With counts as small as 0–5, even one coding error changes the qualitative ranking. The authors should have at least two annotators judge a subset of responses, report agreement (e.g., Cohen's kappa or Krippendorff's alpha), and discuss any disagreements.
  4. [Section 5.3, Table 3] The conclusion that reasoning approaches increase question frequency is not cleanly supported by Table 3 because the models differ in size, family, and training. For example, llama3.2:3b asks 278 questions, barely more than llama2:13b (276) and much fewer than deepseek-r1:8b (354), while llama3.3:70b (383) is also the largest model. Without paired comparisons using the same base model with and without CoT or reasoning training, the effect of reasoning is confounded with model scale and architecture. A statistical test (e.g., permutation test over dialogues) and a controlled comparison are needed before concluding that 'reasoning does appear to increase question frequency'.
  5. [Section 5.1 and Section 4] The finding that humans 'hardly ever produce clarification questions for referential ambiguity' depends entirely on the operational definition in Section 4, which counts a referent as ambiguous when it has 'segment_phrase_antecedent_2' or 'object2' attributes from MDC-R. The paper itself notes in Section 5.1 that many such instances are deliberate underspecifications ('that's great', 'like this?') that are not task-relevant ambiguities. If the definition captures mostly low-stakes, conventionalised expressions, the weak link between ambiguity and clarification may be an artifact of the annotation scheme. The authors should either validate this definition against human judgments of genuinely ambiguous referents or report how many of the 149 ambiguous instances fall into the 'approval/polite' category versus other types, to show that the conclusion is robust.
minor comments (5)
  1. [Section 5.3, final paragraph] 'Another limitation to our experiment. It's challenging to communicate a sense of perspective to the LLMs. As a consequence, some instructions do not make sense...' is a sentence fragment and is disconnected from the rest of the quantitative analysis; it should be integrated into a limitations subsection or the discussion.
  2. [Table 2] The table row 'space 16 0.3%' is unclear; it likely refers to spatial deixis instances, but the label is ambiguous. Also, the percentages for confirmation and clarification questions are given relative to utterances; please clarify the denominator and report the overlap between dialogues containing clarification and confirmation questions, since the text says 77% and 75% respectively.
  3. [Section 4 / Throughout] The term 'task-based uncertainty' is used in the abstract and Section 7 but is never formally defined. Consider defining it operationally, for example as clarification/confirmation questions whose subject is not a referentially ambiguous markable, so that the classification is reproducible.
  4. [Section 6] The discussion connecting clarification to metacognition and theory of mind is speculative and is not tested by the experiments. It would be helpful to add a sentence explicitly stating that this framework is a post-hoc interpretation rather than a mechanism established by the reported results.
  5. [Appendix A.1] The system prompt for the Architect uses the placeholder '[]' for the target and current world state, but it is not explained how these are instantiated. Please clarify the prompt construction, including whether the full dialogue history is concatenated and whether the world-state representation is given as coordinates or a natural-language description.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the study is empirical and self-contained; the unsupported asymmetry claim is a missing comparison, not a circular reduction.

full rationale

The paper is an empirical corpus study, not a derivational argument. Its central comparisons (human versus LLM clarification behavior) are measured from annotated data and generated outputs; no equation is fitted to the target result and then reported as a prediction. The ambiguity ground truth comes from the authors' own MDC-R corpus, but that corpus is a public, inspectable annotation artifact, and the clarification annotations come from the separate MSDC effort; neither is an unverified theorem invoked to force the conclusion. The only self-citation (MDC-R) supplies data rather than a load-bearing premise that determines the outcome, so it does not raise the circularity score under the stated rules. The 'reasoning increases questions' result comes from a controlled comparison of prompt and model variants, not from a parameter fitted to that result. The abstract's 'conversely' asymmetry is indeed not quantified in Section 5.3, and the relevance judgments rely on a single coder, but these are correctness and evaluation-validity concerns, not cases where a claim reduces by construction to its input. No step in the paper's derivation chain is equivalent to its own premises, so the appropriate circularity score is 0.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The paper's empirical claims rest on the quality and completeness of two existing annotation layers (MDC-R reference and MSDC SDRT), the representativeness of the 100-dialogue subset, and the assumption that LLM prompts supply context equivalent to the human builder's situation. No free parameters or invented entities are introduced.

assumptions (4)
  • domain assumption MDC-R reference annotations correctly identify referential ambiguity (two or more antecedents) in the 100-dialogue subset.
    Section 3 and Section 4 use the 'segment_phrase_antecedent_2' and 'object2' attributes as the operational definition of ambiguity; if these are inaccurate, the central comparison is affected.
  • domain assumption MSDC SDRT annotations correctly identify all clarification and confirmation questions.
    Section 3 relies on MSDC to count clarification and confirmation question subjects.
  • domain assumption The 100-dialogue subset is representative of the full Minecraft Dialogue Corpus for question behavior.
    Section 3 compares question frequencies between subset and full corpus to argue representativeness.
  • domain assumption The LLM prompts give the models the same information as the human builder, so differences in question behavior reflect underlying capabilities rather than information access.
    Section 4 and Appendix A.1 describe feeding conversation context and world state; the authors acknowledge perspective is hard to convey, so this assumption is only partially met.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Referential ambiguity and clarification requests: comparing human and LLM behaviour." pith.science (2026). https://pith.science/paper/5ZF22WC3

@misc{pith2026250710445,
  author       = {Pith},
  title        = {Pith review of: Referential ambiguity and clarification requests: comparing human and LLM behaviour},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/5ZF22WC3}},
  note         = {Machine review of arXiv:2507.10445}
}
read the original abstract

In this work we examine LLMs' ability to ask clarification questions in task-oriented dialogues that follow the asynchronous instruction-giver/instruction-follower format. We present a new corpus that combines two existing annotations of the Minecraft Dialogue Corpus -- one for reference and ambiguity in reference, and one for SDRT including clarifications -- into a single common format providing the necessary information to experiment with clarifications and their relation to ambiguity. With this corpus we compare LLM actions with original human-generated clarification questions, examining how both humans and LLMs act in the case of ambiguity. We find that there is only a weak link between ambiguity and humans producing clarification questions in these dialogues, and low correlation between humans and LLMs. Humans hardly ever produce clarification questions for referential ambiguity, but often do so for task-based uncertainty. Conversely, LLMs produce more clarification questions for referential ambiguity, but less so for task uncertainty. We question if LLMs' ability to ask clarification questions is predicated on their recent ability to simulate reasoning, and test this with different reasoning approaches, finding that reasoning does appear to increase question frequency and relevancy.

Figures

Figures reproduced from arXiv: 2507.10445 by the authors.

Figure 1
Figure 1. Referential Ambiguity Annotation Example [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Clarification Question Annotation Example [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Excerpt from B4-A19-C4-1522882189483 demonstrating propagation of referential ambiguity Similarly, in [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (3 more)
Figure 5
Figure 5. Figure 5: Excerpt from B19-A36-C118- 1524093668453 showing the subject of a clarification request in relation to orientation uncertainty Architect great now one orange below that one Builder (model vs. original human dialogue): Model Is it correct to assume ’now one orange below…
Figure 6
Figure 6. Figure 6: Excerpt from B16-A29-C144- 1524067263550 showing the subject of a clarification request in relation to individual position uncertainty lective, individual, absolute and relative property. LLMs seem to also approach these well. Another common example is colour (shown in…
Figure 7
Figure 7. Figure 7: Excerpt from B37-A23-C102-1523629957597 showing the subject of a clarification request in relation to the colour property 5.3 Quantitative Analysis In this section we aim to quantify the differences between approaches and how they respond to situa￾tions that may benefi…

Discussion (0). Sign in to comment.

Forward citations

Cited by 3 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Talking to a Know-It-All GPT or a Second-Guesser Claude? How Repair reveals unreliable Multi-Turn Behavior in LLMs

    cs.CL 2026-04 unverdicted novelty 6.0 of 10

    Each tested LLM shows its own characteristic unreliability when engaging in repair during extended math-question dialogues.

  2. Discriminatory Compliance: How LLMs Answer Queries from Protected Groups

    cs.CY 2026-06 unverdicted novelty 4.0 of 10

    State-of-the-art LLMs respond inconsistently to queries from protected-group personas, with some responses omitting key information that should be provided.

  3. Frame of Reference: Addressing the Challenges of Common Ground Representation in Situational Dialogs

    cs.CL 2026-01 unverdicted novelty 4.0 of 10

    Reinforcement learning on synthetic data improves language models' ability to represent and use common ground with relational references in situated dialogs.

Reference graph

Works this paper leans on

53 extracted references · 33 canonical work pages · cited by 3 Pith papers

  1. [1]

    Mohammad Aliannejadi, Hamed Zamani, Fabio Crestani, and W Bruce Croft. 2019. Asking clarifying questions in open-domain information-seeking conversations. In Proceedings of the 42nd international acm sigir conference on research and development in information retrieval, pages 475--484

    work page 2019

  2. [2]

    Chinmaya Andukuri, Jan-Philipp Fr \"a nken, Tobias Gerstenberg, and Noah D Goodman. 2024. Star-gate: Teaching language models to ask clarifying questions. arXiv preprint arXiv:2403.19154

    arXiv 2024

  3. [3]

    Nicholas Asher and Alex Lascarides. 2003. Logics of conversation. Cambridge University Press

    work page 2003

  4. [4]

    Cristian-Paul Bara, Sky CH-Wang, and Joyce Chai. 2021. Mindcraft: Theory of mind modeling for situated dialogue in collaborative tasks. arXiv preprint arXiv:2109.06275

    arXiv 2021

  5. [5]

    Claire Bonial, Mitchell Abrams, David Traum, and Clare Voss. 2021. Builder, we have done it: evaluating & extending dialogue-amr nlu pipeline for two collaborative domains. In Proceedings of the 14th International Conference on Computational Semantics (IWCS), pages 173--183

    work page 2021

  6. [6]

    Julia Bonn, Martha Palmer, Jon Cai, and Kristin Wright-Bettner. 2020. Spatial AMR : Expanded spatial annotation in the context of a grounded minecraft corpus. In Proceedings of the 12th Conference on Language Resources and Evaluation (LREC 2020),

    work page 2020

  7. [7]

    Ta-Chung Chi, Minmin Shen, Mihail Eric, Seokhwan Kim, and Dilek Hakkani-Tur. 2020. Just ask: An interactive learning framework for vision and language navigation. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pages 2459--2466

    work page 2020

  8. [8]

    Yang Deng, Lizi Liao, Liang Chen, Hongru Wang, Wenqiang Lei, and Tat-Seng Chua. 2023. Prompting and evaluating large language models for proactive dialogues: Clarification, target-guided, and non-collaboration. arXiv preprint arXiv:2305.13626

    arXiv 2023

Show all 53 references
  1. [9]

    Shehzaad Dhuliawala, Mojtaba Komeili, Jing Xu, Roberta Raileanu, Xian Li, Asli Celikyilmaz, and Jason Weston. 2023. https://arxiv.org/abs/2309.11495 Chain-of-verification reduces hallucination in large language models . Preprint, arXiv:2309.11495

    2023 arXiv

  2. [10]

    Aniket Didolkar, Anirudh Goyal, Nan Rosemary Ke, Siyuan Guo, Michal Valko, Timothy Lillicrap, Danilo Jimenez Rezende, Yoshua Bengio, Michael C Mozer, and Sanjeev Arora. 2024. Metacognitive capabilities of llms: An exploration in mathematical problem solving. Advances in Neural...

    2024

  3. [11]

    John H Flavell. 1979. Metacognition and cognitive monitoring: A new area of cognitive--developmental inquiry. American psychologist, 34(10):906

    1979

  4. [12]

    Malte Gabsdil. 2003. Clarification in spoken dialogue systems. In Proceedings of the 2003 AAAI Spring Symposium. Workshop on Natural Language Generation in Spoken and Written Dialogue, pages 28--35

    2003

  5. [13]

    Spandana Gella, Aishwarya Padmakumar, Patrick Lange, and Dilek Hakkani-Tur. 2022. Dialog acts for task-driven embodied agents. arXiv preprint arXiv:2209.12953

    2022 arXiv

  6. [14]

    Felix Gervits, Antonio Roque, Gordon Briggs, Matthias Scheutz, and Matthew Marge. 2021. How should agents ask questions for situated learning? an annotated dialogue corpus. arXiv preprint arXiv:2106.06504

    2021 arXiv

  7. [15]

    Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, and 1 others. 2025. Deepseek-r1: Incentivizing reasoning capability in llms via reinforcement learning. arXiv preprint arXiv:2501.12948

    2025 arXiv

  8. [16]

    Rebecca MM Hicke and David Mimno. 2024. [lions: 1] and [tigers: 2] and [bears: 3], oh my! literary coreference annotation with llms. arXiv preprint arXiv:2401.17922

    2024 arXiv

  9. [17]

    Matthew Johnson, Katja Hofmann, Tim Hutton, and David Bignell. 2016. The malmo platform for artificial intelligence experimentation. In IJCAI , volume 16, pages 4246--4247

    2016

  10. [18]

    Jin-Hwa Kim, Nikita Kitaev, Xinlei Chen, Marcus Rohrbach, Byoung-Tak Zhang, Yuandong Tian, Dhruv Batra, and Devi Parikh. 2017. Codraw: Collaborative drawing as a testbed for grounded goal-driven communication. arXiv preprint arXiv:1712.05558

    2017 arXiv

  11. [19]

    Julia Kiseleva, Ziming Li, Mohammad Aliannejadi, Shrestha Mohanty, Maartje ter Hoeve, Mikhail Burtsev, Alexey Skrynnik, Artem Zholus, Aleksandr Panov, Kavya Srinet, and 1 others. 2022. Interactive grounded language understanding in a collaborative environment: IGLU 2021. In Ne...

    2022

  12. [20]

    Michal Kosinski. 2023. Theory of mind may have spontaneously emerged in large language models. arXiv preprint arXiv:2302.02083, 4:169

    2023 arXiv

  13. [21]

    Lorenz Kuhn, Yarin Gal, and Sebastian Farquhar. 2022. Clam: Selective clarification for ambiguous questions with generative language models. arXiv preprint arXiv:2212.07769

    2022 arXiv

  14. [22]

    Alex Lascarides and Nicholas Asher. 2007. Segmented discourse representation theory: Dynamic semantics with discourse structure. In Computing meaning, pages 87--124. Springer

    2007

  15. [23]

    Le and Alan Ritter

    Nghia T. Le and Alan Ritter. 2023. https://arxiv.org/abs/2305.14489 Are large language models robust coreference resolvers? Preprint, arXiv:2305.14489

    2023 arXiv

  16. [24]

    Belinda Z Li, Alex Tamkin, Noah Goodman, and Jacob Andreas. 2023. Eliciting human preferences with language models. arXiv preprint arXiv:2310.11589

    2023 arXiv

  17. [25]

    Haau-Sing Li, Mohsen Mesgar, Andr \'e FT Martins, and Iryna Gurevych. 2022. Asking clarification questions for code generation in general-purpose programming language. arXiv preprint arXiv:2212.09885

    2022 arXiv

  18. [26]

    Chris Madge, Maris Camilleri, Paloma Carretero Garcia, Mladen Karan, Juexi Shao, Prashant Jayannavar, Julian Hough, Benjamin Roth, and Massimo Poesio. 2025. Mdc-r: The minecraft dialogue corpus with reference. arXiv preprint arXiv:2506.22062

    2025 arXiv

  19. [27]

    Chris Madge and Massimo Poesio. 2024. A llm benchmark based on the minecraft builder dialog agent task. In Proceedings of the 28th Workshop on the Semantics and Pragmatics of Dialogue

    2024

  20. [28]

    Brielen Madureira and David Schlangen. 2023 a . " are you telling me to put glasses on the dog?''content-grounded annotation of instruction clarification requests in the codraw dataset. arXiv preprint arXiv:2306.02377

    2023 arXiv

  21. [29]

    Brielen Madureira and David Schlangen. 2023 b . Instruction clarification requests in multimodal collaborative dialogue games: Tasks, and an analysis of the codraw dataset. arXiv preprint arXiv:2302.14406

    2023 arXiv

  22. [30]

    Bodhisattwa Prasad Majumder, Sudha Rao, Michel Galley, and Julian McAuley. 2021. Ask what's missing and what's useful: Improving clarification question generation using global knowledge. arXiv preprint arXiv:2104.06828

    2021 arXiv

  23. [31]

    Anjali Narayan-Chen, Prashant Jayannavar, and Julia Hockenmaier. 2019. Collaborative dialogue in minecraft. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5405--5415

    2019

  24. [32]

    Miguel Ortega-Mart \' n, \'O scar Garc \' a-Sierra, Alfonso Ardoiz, Jorge \'A lvarez, Juan Carlos Armenteros, and Adri \'a n Alonso. 2023. Linguistic ambiguity analysis in chatgpt. arXiv preprint arXiv:2302.06426

    2023 arXiv

  25. [33]

    Aishwarya Padmakumar, Jesse Thomason, Ayush Shrivastava, Patrick Lange, Anjali Narayan-Chen, Spandana Gella, Robinson Piramuthu, Gokhan Tur, and Dilek Hakkani-Tur. 2022. Teach: Task-driven embodied agents that chat. In Proceedings of the AAAI Conference on Artificial Intellige...

    2022

  26. [34]

    Jing-Cheng Pang, Heng-Bo Fan, Pengyuan Wang, Jia-Hao Xiao, Nan Tang, Si-Hang Yang, Chengxing Jia, Sheng-Jun Huang, and Yang Yu. 2024. Empowering language models with active inquiry for deeper understanding. arXiv preprint arXiv:2402.03719

    2024 arXiv

  27. [35]

    Massimo Poesio, Maris Camilleri, Paloma Carretero Garcia, and Ron Artstein. 2024. https://sites.google.com/view/arrau/corpus The ARRAU 3 Annotation Manual , v. 1.1 edition. Queen Mary University of London

    2024

  28. [36]

    Sameer Pradhan, Alessandro Moschitti, Nianwen Xue, Olga Uryupina, and Yuchen Zhang. 2012. Conll-2012 shared task: Modeling multilingual unrestricted coreference in ontonotes. In Joint conference on EMNLP and CoNLL-shared task, pages 1--40

    2012

  29. [37]

    David Premack and Guy Woodruff. 1978. Does the chimpanzee have a theory of mind? Behavioral and brain sciences, 1(4):515--526

    1978

  30. [38]

    Matthew Purver, Jonathan Ginzburg, and Patrick Healey. 2003. On the means for clarification in dialogue. Current and new directions in discourse and dialogue, pages 235--255

    2003

  31. [39]

    Verena Rieser and Johanna Moore. 2005. Implications for generating clarification requests in task-oriented dialogues. In Proceedings of the 43rd Annual Meeting of the ACL , pages 239--246, Ann Arbor

    2005

  32. [40]

    Kepa Rodr \' guez and David Schlangen. 2004. Form, intonation and function of clarification requests in G erman task-oriented spoken dialogues. In Proceedings of the 8th Workshop on the Semantics and Pragmatics of Dialogue ( Catalog ) , pages 101--108, Barcelona

    2004

  33. [41]

    David Schlangen. 2004. Causes and strategies for requesting clarification in dialogue. In Proceedings of the 5th SIGdial Workshop on Discourse and Dialogue at HLT-NAACL 2004, pages 136--143

    2004

  34. [42]

    Ying Shen and Ismini Lourentzou. 2023. Learning by asking for embodied visual navigation and task completion. arXiv preprint arXiv:2302.04865

    2023 arXiv

  35. [43]

    Francesc Sidera, Georgina Perpi \ n \`a , J \`e ssica Serrano, and Carles Rostan. 2018. Why is theory of mind important for referential communication? Current Psychology, 37:82--97

    2018

  36. [44]

    Alberto Testoni and Raquel Fern \'a ndez. 2024. Asking the right question at the right time: Human and model uncertainty guidance to ask clarification questions. arXiv preprint arXiv:2402.06509

    2024 arXiv

  37. [45]

    Kate Thompson, Julie Hunter, and Nicholas Asher. 2024. Discourse structure for the minecraft corpus. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pages 4957--4967

    2024

  38. [46]

    Yuqing Wang and Yun Zhao. 2023. Metacognitive prompting improves understanding in large language models. arXiv preprint arXiv:2308.05342

    2023 arXiv

  39. [47]

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, and 1 others. 2022. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824--24837

    2022

  40. [48]

    An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, and 22 others. 2024. Qwen2.5 technical repo...

    2024 arXiv

  41. [49]

    Michael JQ Zhang and Eunsol Choi. 2023. Clarify when necessary: Resolving ambiguity through interaction with lms. arXiv preprint arXiv:2311.09469

    2023 arXiv

  42. [50]

    Tong Zhang, Peixin Qin, Yang Deng, Chen Huang, Wenqiang Lei, Junhong Liu, Dingnan Jin, Hongru Liang, and Tat-Seng Chua. 2024. Clamber: A benchmark of identifying and clarifying ambiguous information needs in large language models. arXiv preprint arXiv:2405.12063

    2024 arXiv

  43. [51]

    Yue Zhang, Yafu Li, Leyang Cui, Deng Cai, Lemao Liu, Tingchen Fu, Xinting Huang, Enbo Zhao, Yu Zhang, Yulong Chen, and 1 others. 2023. Siren's song in the ai ocean: a survey on hallucination in large language models. arXiv preprint arXiv:2309.01219

    2023 arXiv

  44. [52]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  45. [53]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.