Pith. sign in

REVIEW 6 major objections 6 minor 45 references

CLAIM: Leading Open-domain Active Clarification of Large Language Models with Uncertainty Measurement

T0 review · 6 major / 6 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read When several language models disagree on an answer, that disagreement can be turned into training data that teaches a model when to ask clarifying questions.

desk verdict Low-annotation clarification training shows promise, but the paper must document where its 10k training queries come from before the SOTA claims can be trusted. read the letter →

arxiv 2608.11631 v1 pith:L5NV2W22 submitted 2026-08-12 cs.AI

Kuangzhao Yang , Ziliang Zhao , Zhicheng Dou This is my paper · ORCID
classification cs.AI
keywords clarifyingquestionsopen-domainclarificationuncertaintyestimationsemanticentropysyntheticdatainformationgaingrouprelativepolicyoptimizationlargelanguagemodels
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that the disagreement among several language models answering the same user query is a usable signal for teaching a model when to ask clarifying questions and what to ask. It builds CLAIM, an offline pipeline that turns that signal into roughly ten thousand synthetic training examples without human preference labels, then trains a single 8B model with supervised fine-tuning and group-relative policy optimization. On three open-domain and task-oriented benchmarks, the trained model matches or approaches the best prior clarification system, which was trained on about 120k manually supervised and preference-annotated examples. A sympathetic reader would care because the result points to a low-cost, scalable route to proactive clarification: the supervision comes from model disagreement itself rather than expensive annotation.

What carries the argument

The load-bearing object is the entropy-driven synthetic data pipeline: a multi-model answer sampler using five models, semantic clustering over answers, and entropy as an uncertainty score, followed by a threshold, an LLM-based completeness judgment, and conflict arbitration. Clarifying-question candidates are generated with history-based diversity constraints, and the chosen question is the one that maximizes information gain, $IG(q, c_q) = E_1(q) - E_2(q, c_q, A)$, where $A$ is a simulated user answer and $E_2$ is the post-clarification entropy. That synthetic data defines a structured decision-generation target: either answer directly, or name a clarification dimension and ask a targeted question. Training then runs in two stages: supervised fine-tuning to internalize the behaviors and group-relative policy optimization to stabilize decisions for high-uncertainty queries. The multi-model cost is paid only offline; at inference CLAIM is a single model.

What would settle it

Build a held-out set of queries judged ambiguous by human raters but for which the five models produce nearly identical answers; if CLAIM answers directly on most of them, the entropy signal misses an entire class of missing-information queries and the synthetic labels inherit that blind spot.

Watch

Extended reading notes

Core claim

The paper's central claim is that query ambiguity can be operationalized as entropy over semantically clustered answers from a set of five heterogeneous models. For a user query $q$, the pipeline collects candidate direct answers, clusters them by semantic similarity, and computes $E_1(q) = -\sum_i p_i \log p_i$. If entropy exceeds a fixed threshold, or an LLM completeness judgment also flags missing information, with a third model arbitrating disagreements, the query is labeled as needing clarification. The pipeline then generates several candidate clarifying questions and keeps the one with the largest information gain, defined as the drop in entropy after a simulated user answer is added. Those labels form about 10k training instances; SFT teaches the decision and generation format, and GRPO sharpens decisions near the boundary. The paper reports that the resulting single model achieves top or near-top results on the clarification-necessity and question-quality metrics of ClariLM-test, IN3, and CLAMBER, using far less supervised data than the ClariLM baseline.

Load-bearing premise

Everything rests on the assumption that the spread of answers among a fixed set of five models, combined with the fixed threshold and LLM completeness judgments, faithfully identifies which queries need clarification and which clarifying question is most useful to real users.

Editorial extensions

If this is right

  • Clarification policies can be trained without human preference labels, since the supervision is synthesized from model outputs.
  • Roughly ten thousand uncertainty-constructed examples are enough to approach a system trained on roughly one hundred twenty thousand supervised and preference-annotated examples.
  • A multi-model agent pipeline can be distilled into a single 8B model that needs one inference call per query at deployment.
  • The information-gain selection step is what improves clarification-question quality: removing it drops dimension accuracy and semantic similarity while necessity accuracy stays similar.
  • Adding GRPO after SFT further improves decision accuracy on the harder boundary cases, as shown by the gains from SFT-Full to CLAIM.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The same disagreement-as-uncertainty label generator could plausibly be reused for other underspecification decisions, such as when a model should ask for confirmation, request a missing tool argument, or abstain from answering.
  • Because the threshold is fixed at 0.45 based on the five-model sampling scheme, the method's calibration is tied to that specific model pool; an inexpensive extension would be to re-estimate the threshold for each pool rather than using a global constant.
  • The information-gain ranking rests on a simulated user answer, so questions that the simulating model finds easy to disambiguate may be favored even when real users would not answer that way; testing with real user answers would show whether the ranking holds.
  • Extending the same uncertainty measurement across turns, which the paper leaves to future work, would turn CLAIM into an active information-acquisition loop that re-estimates entropy after each user reply.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

6 major / 6 minor

Summary. The paper proposes CLAIM, an uncertainty-driven framework for open-domain clarification in LLMs. The method estimates query uncertainty via entropy over semantic clusters of answers sampled from k1=5 heterogeneous LLMs, decides whether clarification is needed using a hand-set entropy threshold tau=0.45 combined with LLM-based judgment and conflict arbitration, and selects among k2=3 candidate clarifying questions by information gain computed with a simulated user answer. Training uses SFT followed by GRPO on roughly 10k synthetic instances generated without human annotations. Evaluation on ClariLM-test, IN3, and CLAMBER compares against LLMs, reasoning models, SFT variants, and the previous ClariLM system. The paper claims SOTA or near-SOTA on the majority of metrics and superior data efficiency versus ClariLM's ~120k training examples.

Significance. If the results hold, the paper provides a low-cost, scalable alternative to human-annotated clarification training: a single 8B model trained on ~10k automatically generated examples can match or beat a system trained on ~120k examples on two of three benchmarks. The multi-model semantic-entropy signal is principled and well-motivated by prior uncertainty-estimation work. The paper includes useful ablations (SFT-Entropy only, SFT-LLM only, SFT-without IG, SFT-IN3 vs. SFT-Full) that isolate the contributions of the judgment signals, the information-gain selection, and the domain-generalization behavior. The release of code and prompts (claimed in Section 3) would aid reproducibility. However, the strength of the central claims is weakened by the lack of statistical support, the unvalidated synthetic labels, and the absence of provenance for the training queries.

major comments (6)
  1. [Sections 3.2-3.6 and 4.1] The paper never states the source of the user queries used to construct the approximately 10k synthetic training instances. Section 4.1 only states that the three benchmarks are used 'exclusively for evaluation,' but this phrasing does not rule out that queries from the ClariLM-test, IN3, or CLAMBER test sets were fed into the CLAIM-Agent pipeline before evaluation. If the training queries overlap with the test queries, the data-efficiency claim in Section 4.4 and the generalization analysis in Section 4.5 become circular. I request an explicit statement of where the training queries came from and a documented decontamination procedure (e.g., n-gram overlap removal) between the training queries and all test sets.
  2. [Sections 3.2-3.3, Eq. (5), and 3.6.2] The training labels are generated entirely by the authors' own pipeline: the hand-set entropy threshold tau=0.45, the LLM-based judgment, and the conflict-arbitration step. There is no independent validation that these synthetic labels agree with human judgments of clarification necessity or question quality, and the GRPO reward is defined as alignment with these same synthetic labels. Without such validation, the trained model may be learning the pipeline's biases rather than user needs, and the only support for the pipeline's correctness is the final benchmark accuracy, which risks a self-referential loop. Please provide a label-quality study, for example human agreement on a sample of the training labels or a comparison of the pipeline's decisions against the human-annotated labels in IN3 and CLAMBER.
  3. [Section 5 vs. Table 1] The conclusion states that CLAIM 'consistently outperforms' strong baseline models, but Table 1 shows that ClariLM outperforms CLAIM on all four metrics on IN3: Accuracy 89.72 vs. 87.04, F1 94.36 vs. 92.55, CDA 66.32 vs. 63.16, and CQSS 72.68 vs. 72.23. The claim should be qualified to reflect that CLAIM's advantage is benchmark-specific (ClariLM-test and CLAMBER) rather than consistent across all benchmarks.
  4. [Section 4.4, Table 1] All results are reported as single runs without variance or significance tests. Several key comparisons are within 1-2 points, for example ClariLM-test F1 (CLAIM 84.97 vs. ClariLM 85.48) and IN3 CQSS (CLAIM 72.23 vs. ClariLM 72.68). Without standard deviations across seeds or pairwise significance tests, the 'SOTA or near-SOTA' claim is not statistically supported for those metrics. Please report mean and standard deviation over at least three seeds and, where appropriate, significance tests for the main comparisons.
  5. [Section 3.5 and Appendix A] The information gain in Eq. (7) depends on a simulated user answer A generated by DeepSeek-V3. The selection of the optimal clarifying question is therefore conditioned on the assumption that real users answer the way DeepSeek-V3 does. No validation is provided that these simulated answers are representative of real user responses. This premise is load-bearing for the IG-based selection claim (the SFT-without IG vs. SFT-Full comparison in Section 4.5) and should be tested, for example by comparing simulated and real user answers on a sample of queries.
  6. [Section 3.3.1 and Appendix A] The entropy threshold tau=0.45 is motivated by an argument specific to the choice of k1=5 sampling models, but the paper reports no sensitivity analysis for tau, k1, the semantic clustering similarity threshold, or the sampling temperature. These are free parameters of the synthetic data generation pipeline, and the robustness of the method to their variation is unestablished. I request an ablation or sensitivity study for at least the threshold and the number of answer-sampling models.
minor comments (6)
  1. [Figure 3] Figure 3 appears garbled in the manuscript, with sequences such as '/uni00000013' replacing the plotted content. A readable figure is needed.
  2. [Table 1] Table 1 has formatting errors where numbers appear merged, for example '54.4470.2084.26' in the Qwen3-32B row. The table should be cleaned up so that each metric is clearly separated.
  3. [Section 4.4] The phrase 'approximately 10k' is used for the training instance count, but the exact number is never stated. Please provide the exact count and clarify whether it includes both clarification and non-clarification instances.
  4. [Section 4.4] The term 'near-SOTA' is not defined. Please specify the margin that qualifies as near state-of-the-art.
  5. [Section 4.6] The human evaluation uses three expert annotators and general users, but no inter-annotator agreement (e.g., Cohen's kappa or Fleiss' kappa) is reported. Please include this information.
  6. [Section 3.2] The notation for the cluster probability p_i is defined assuming k1 answers, but the clustering threshold used for semantic clustering is never specified in the main text or appendix. Please document it.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the training signal is a model-intrinsic uncertainty proxy, and the headline results are evaluated on independent, externally labeled benchmarks.

full rationale

The derivation chain has two stages. First, CLAIM-Agent constructs training labels: it samples k1=5 models' answers, clusters them, computes semantic entropy E1(q) (Eq. 4), thresholds it with tau=0.45 (Eq. 5), supplements with LLM-based completeness judgements and arbitration, generates CQ candidates, and selects the CQ maximizing IG=E1-E2 (Eq. 7). Second, CLAIM is trained by SFT (Eq. 9) and GRPO (Eq. 14) to imitate those decisions, then evaluated on ClariLM-test, IN3 test, and CLAMBER for clarification necessity (ACC/F1) and question quality (CDA/CQSS). The evaluation labels are human or semantic ground truths from external datasets, not the pipeline's own E1 or IG values; therefore the SOTA/near-SOTA claim is not forced by construction. The GRPO reward measures 'the extent to which the generated result aligns with the target decision induced by the synthetic data process,' which is ordinary self-supervised training on synthetic labels; it would be circular only if those same synthetic labels were also the evaluation metric, which they are not. Section 4.1 explicitly states that the three benchmarks are 'used exclusively for evaluation' and that IN3's training data is not used, with evaluation restricted to its test set. Appendix B and C do run evaluation queries through the pipeline for diagnostic statistics (e.g., Table 3), but diagnostics on held-out data are not training. One transparency gap is real: the provenance of the roughly 10k training queries is never documented, so if those queries overlapped the test benchmarks, the data-efficiency and generalization claims would collapse. However, the paper provides no equation or construction exhibiting such overlap, and its explicit evaluation-only statement, if accurate, rules it out; this is a correctness/verification risk, not a demonstrated circularity. The paper also cites the authors' prior ClariLM [38] as a benchmark source and baseline, but that citation points to a published, reproducible system and dataset and is not load-bearing for the core derivation. No self-definitional step, fitted-parameter-renamed-as-prediction, or self-citation uniqueness argument appears. Verdict: no significant circularity, score 0.

Assumptions & free parameters 4 free parameters · 3 assumptions · 0 invented entities

The central claim depends on several hand-chosen thresholds and domain assumptions about the validity of model disagreement and simulated user answers as proxies for real ambiguity and utility. None of these are derived from first principles; they are the cost of removing human annotation.

free parameters (4)
  • Entropy threshold tau = 0.45
    Global threshold for clarification judgement (Eq. 5), set by the heuristic that a one-model disagreement yields entropy 0.50. It directly controls which queries are labeled as needing clarification, and its sensitivity is not reported.
  • Number of answer-sampling models k1 = 5
    Number of heterogeneous LLMs used for entropy estimation; hand-chosen with no sensitivity analysis.
  • Number of candidate clarifying questions k2 = 3
    Number of generated CQ candidates per query; hand-chosen with no sensitivity analysis.
  • Semantic clustering threshold = not specified
    Section 3.2 says 'perform clustering based on semantic similarity' but never defines the distance metric threshold or clustering algorithm, so the entropy values depend on an undisclosed arbitrary choice.
assumptions (3)
  • domain assumption Semantic answer disagreement across models reflects query ambiguity
    Central premise of CLAIM, stated in Section 1 and formalized in Section 3.2; supported by prior work [10,30] but not proven here.
  • domain assumption Information gain computed with simulated user answers measures real clarification value
    Section 3.5 uses a simulated user answer A to compute post-clarification entropy; this assumes the simulation reflects real user responses.
  • domain assumption LLM-based completeness judgement and conflict arbitration are reliable annotators
    The LLM judge and arbitration model produce labels that are used as ground truth for SFT and GRPO, without human verification on the training set.

how reviews work

0 comments
Cite this review

Pith. "Pith review of CLAIM: Leading Open-domain Active Clarification of Large Language Models with Uncertainty Measurement." pith.science (2026). https://pith.science/paper/L5NV2W22

@misc{pith2026260811631,
  author       = {Pith},
  title        = {Pith review of: CLAIM: Leading Open-domain Active Clarification of Large Language Models with Uncertainty Measurement},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/L5NV2W22}},
  note         = {Machine review of arXiv:2608.11631}
}
read the original abstract

In open-domain human-computer interaction scenarios, large language models (LLMs) frequently encounter user queries that are ambiguous or incomplete. In such cases, directly producing an answer often leads to overgeneralized, erroneous, or low-information responses. In contrast, asking clarifying questions can substantially improve interaction quality. However, existing approaches still rely heavily on manually annotated data or preference alignment to address two fundamental challenges: when clarification is necessary, and which aspect of the query should be clarified. This reliance incurs high annotation costs and limits generalization. To address these challenges, we propose CLAIM, an uncertainty-driven framework for active clarification learning in open-domain settings. CLAIM eliminates the need for explicit human preference annotations by quantifying query uncertainty through the entropy induced by answer disagreements across multiple models. This uncertainty signal is then used to construct high-quality synthetic data, enabling the training of a unified clarification decision model through a combination of supervised learning and reinforcement learning. Specifically, we propose an entropy-driven synthetic data generation pipeline that integrates entropy-based uncertainty estimation with semantic clustering and reasoning-based judgments, enabling reliable automatic annotation of clarification requirements. To train CLAIM, we formulate the clarification process as a structured decision generation problem and adopt a training paradigm that combines supervised fine-tuning (SFT) with group-relative policy optimization (GRPO). Experimental results demonstrate that CLAIM can learn stable and generalizable clarification strategies without relying on manually labeled data, offering a low-cost and robust solution for proactive understanding in real-world open-domain interactions with LLMs.

Figures

Figures reproduced from arXiv: 2608.11631 by the authors.

Figure 1
Figure 1. Examples of clarification in human–LLM interac [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. The overall framework of CLAIM, consisting of entropy-driven uncertainty estimation, clarification judgement, [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Pairwise comparative evaluation of CLAIM against [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

45 extracted references · 15 canonical work pages

  1. [1]

    Mohammad Aliannejadi, Leif Azzopardi, Krisztian Balog, and Mark Sanderson

    work page

  2. [2]

    Mohammad Aliannejadi, Julia Kiseleva, Aleksandr Chuklin, Jeff Dalton, and Mikhail Burtsev. 2020. ConvAI3: Generating Clarifying Questions for Open- Domain Dialogue Systems (ClariQ). arXiv:2009.11352 [cs.CL] https://arxiv.org/ abs/2009.11352 arXiv preprint / shared task overview

    arXiv 2020

  3. [3]

    Bruce Croft

    Mohammad Aliannejadi, Hamed Zamani, Fabio Crestani, and W. Bruce Croft

  4. [4]

    Yizhou Chi, Jessy Lin, Kevin Lin, and Dan Klein. 2024. CLARINET: Aug- menting Language Models to Ask Clarification Questions for Retrieval. arXiv:2405.15784 [cs.CL] https://arxiv.org/abs/2405.15784 arXiv preprint

    arXiv 2024

  5. [5]

    InProceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval

    Asking Clarifying Questions in Open-Domain Information-Seeking Con- versations. InProceedings of the 42nd International ACM SIGIR Conference on Research and Development in Information Retrieval. Association for Computing Machinery, New York, NY, USA, 475–484. doi:10.1145/3331184.3331265

    arXiv

  6. [6]

    Sebastian Farquhar, Jannik Kossen, Lorenz Kuhn, and Yarin Gal. 2024. Detecting hallucinations in large language models using semantic entropy.Nature630, 8017 (2024), 625–630. doi:10.1038/s41586-024-07421-0

    doi:10.1038/s41586-024-07421-0 2024

  7. [7]

    Shihan Dou, Shichun Liu, Yuming Yang, Yicheng Zou, Yunhua Zhou, Shuhao Xing, Chenhao Huang, Qiming Ge, Demin Song, Haijun Lv, Songyang Gao, Chengqi Lv, Enyu Zhou, Honglin Guo, Zhiheng Xi, Wenwei Zhang, Qipeng Guo, Qi Zhang, Xipeng Qiu, Xuanjing Huang, Tao Gui, and Kai Chen. 2025. Pre-Trained Policy Discriminators are General Reward Models. arXiv:2507.0519...

    arXiv 2025

  8. [8]

    Jiawei Gu, Xuhui Jiang, Zhichao Shi, Hexiang Tan, Xuehao Zhai, Chengjin Xu, Wei Li, Yinghan Shen, Shengjie Ma, Honghao Liu, Saizhuo Wang, Kun Zhang, Yuanzhuo Wang, Lionel Ni, Jian Guo, and Wen Gao. 2024. A Survey on LLM- as-a-Judge. arXiv:2411.15594 [cs.CL] https://arxiv.org/abs/2411.15594 arXiv preprint

    arXiv 2024

Show all 45 references
  1. [9]

    Yujian Gan, Changling Li, Jinxia Xie, Luou Wen, Matthew Purver, and Massimo Poesio. 2024. ClarQ-LLM: A Benchmark for Models Clarifying and Requesting Information in Task-Oriented Dialog. arXiv:2409.06097 [cs.CL] https://arxiv.org/ abs/2409.06097 arXiv preprint

    2024 arXiv

  2. [10]

    Saurabh Kadavath, Aman Arora, John Schulman, Tom Henighan, Jacob Steinhardt, Jared Kaplan, Prafulla Dhariwal, and Dario Amodei. 2022. Language Models (Mostly) Know What They Know. arXiv:2207.05221 [cs.CL] https://arxiv.org/ abs/2207.05221 arXiv preprint

    2022 arXiv

  3. [11]

    Minsuh Joo and Hyunsoo Cho. 2025. Cleanse: Uncertainty Estimation Approach Using Clustering-based Semantic Consistency in LLMs. InProceedings of the Fourth Workshop on Generation, Evaluation and Metrics (GEM 2). Association for Computational Linguistics, Online, 291–301. https...

    2025

  4. [12]

    Lorenz Kuhn, Yarin Gal, and Sebastian Farquhar. 2023. Semantic Uncertainty: Lin- guistic Invariances for Uncertainty Estimation in Natural Language Generation. arXiv:2302.09664 [cs.CL] https://arxiv.org/abs/2302.09664 arXiv preprint

    2023 arXiv

  5. [13]

    Jannik Kossen, Jiatong Han, Muhammed Razzak, Lisa Schut, Shreshth Malik, and Yarin Gal. 2024. Semantic Entropy Probes: Robust and Cheap Hallucination Detection in LLMs. arXiv:2406.15927 [cs.CL] https://arxiv.org/abs/2406.15927 arXiv preprint

    2024 arXiv

  6. [14]

    Li, Been Kim, and Zi Wang

    Belinda Z. Li, Been Kim, and Zi Wang. 2025. QuestBench: Can LLMs ask the right question to acquire information in reasoning tasks? arXiv:2503.22674 [cs.AI] https://arxiv.org/abs/2503.22674 arXiv preprint

    2025

  7. [15]

    Dongryeol Lee, Segwang Kim, Minwoo Lee, Hwanhee Lee, Joonsuk Park, Sang- Woo Lee, and Kyomin Jung. 2023. Asking Clarification Questions to Handle Ambiguity in Open-Domain QA. InFindings of the Association for Computational Linguistics: EMNLP 2023. Association for Computational...

    2023 doi

  8. [16]

    Jiaju Ma, Lei Shi, Kenneth Robertsen, and Peggy Chi. 2025. AmbigChat: Interac- tive Hierarchical Clarification for Ambiguous Open-Domain Question Answering. InProceedings of the 38th Annual ACM Symposium on User Interface Software and Technology. Association for Computing Mach...

    2025

  9. [17]

    Wenhan Liu, Ziliang Zhao, Yutao Zhu, and Zhicheng Dou. 2024. Mining Ex- ploratory Queries for Conversational Search. InProceedings of The Web Conference

    2024

  10. [18]

    Sewon Min, Julian Michael, Hannaneh Hajishirzi, and Luke Zettlemoyer. 2020. AmbigQA: Answering Ambiguous Open-domain Questions. InProceedings of the 2020 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, Online, 5239–525...

    2020 doi

  11. [19]

    Ali Montazeralghaem, Guy Tennenholtz, Craig Boutilier, and Ofer Meshi. 2025. Asking Clarifying Questions for Preference Elicitation With Large Language Models. arXiv:2510.12015 [cs.AI] https://arxiv.org/abs/2510.12015 arXiv preprint

    2025

  12. [20]

    Bodhisattwa Prasad Majumder, Sudha Rao, Michel Galley, and Julian McAuley

  13. [21]

    Cheng Qian, Yuhan Liu, Zhenzhong Lan, Yixuan Liu, Jing Zhang, and Minlie Huang. 2024. Tell Me More! Towards Implicit User Intention Understanding in Agent Interaction. InProceedings of the 62nd Annual Meeting of the Association for CLAIM: Leading Open-domain Active Clarificati...

    2024

  14. [22]

    Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Chelsea Finn, and Sergey Levine. 2023. Direct Preference Optimization: Your Language Model Is Secretly a Reward Model. InAdvances in Neural Information Processing Systems, Vol. 36. Curran Associates, Inc., Red Hook,...

    2023

  15. [23]

    Sudha Rao and Hal Daumé III. 2018. Learning to Ask Good Questions: Ranking Clarification Questions Using Neural Expected Value of Perfect Information. InProceedings of the 56th Annual Meeting of the Association for Computational Linguistics. Association for Computational Lingu...

    2018 doi

  16. [24]

    Lake, and Todd M

    Anselm Rothe, Brenden M. Lake, and Todd M. Gureckis. 2017. Question Asking as Program Generation. arXiv:1711.06351 [cs.CL] https://arxiv.org/abs/1711.06351 arXiv preprint

    2017 arXiv

  17. [25]

    Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F. Christiano, Jan Leik...

    2022

  18. [26]

    Ivan Sekulić, Mohammad Aliannejadi, and Fabio Crestani. 2021. Towards Facet- Driven Generation of Clarifying Questions for Conversational Search. InPro- ceedings of the 2021 ACM SIGIR International Conference on Theory of Information Retrieval. Association for Computing Machin...

    2021

  19. [27]

    Claude E. Shannon. 1948. A Mathematical Theory of Communication.Bell System Technical Journal27, 3 (1948), 379–423, 623–656

    1948

  20. [28]

    Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, Y. K. Li, Y. Wu, and Daya Guo. 2024. DeepSeek- Math: Pushing the Limits of Mathematical Reasoning in Open Language Models. arXiv:2402.03300 [cs.CL] https://arxiv.org/abs/2402....

    2024 arXiv

  21. [29]

    Wenxuan Wang, Juluan Shi, Zixuan Ling, Yuk-Kit Chan, Chaozheng Wang, Cheryl Lee, Youliang Yuan, Jen-tse Huang, Wenxiang Jiao, and Michael R. Lyu. 2025. Learning to Ask: When LLM Agents Meet Unclear Instruction. InProceedings of the 2025 Conference on Empirical Methods in Natur...

    2025

  22. [30]

    Marzieh Saeidi, Max Bartolo, Patrick Lewis, Sameer Singh, Tim Rocktäschel, Mike Sheldon, Guillaume Bouchard, and Sebastian Riedel. 2018. Interpretation of Natural Language Rules in Conversational Machine Reading. InProceedings of the 2018 Conference on Empirical Methods in Nat...

    2018 doi

  23. [31]

    Julia White, Gabriel Poesia, Robert Hawkins, Dorsa Sadigh, and Noah Goodman

  24. [32]

    Jie JW Wu. 2023. Large Language Models Should Ask Clarifying Questions to Increase Confidence in Generated Code. arXiv:2308.13507 [cs.SE] https: //arxiv.org/abs/2308.13507 arXiv preprint

    2023 arXiv

  25. [33]

    Yifei Yuan, Clemencia Siro, Mohammad Aliannejadi, Maarten de Rijke, and Wai Lam. 2024. Asking Multimodal Clarifying Questions in Mixed-Initiative Con- versational Search. InProceedings of the ACM Web Conference 2024. Association for Computing Machinery, New York, NY, USA, 1474...

    2024 doi

  26. [34]

    Trippas, Jeff Dalton, and Filip Radlinski

    Hamed Zamani, Johanne R. Trippas, Jeff Dalton, and Filip Radlinski. 2020. MIM- ICS: A Large-Scale Data Collection for Search Clarification. InProceedings of the 29th ACM International Conference on Information and Knowledge Management. ACM, Galway, Ireland, 3189–3198. doi:10.1...

    2020

  27. [35]

    Le, Ed H

    Xuezhi Wang, Jason Wei, Dale Schuurmans, Maarten Bosma, Quoc V. Le, Ed H. Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. 2023. Self- Consistency Improves Chain-of-Thought Reasoning in Language Models. In Proceedings of the International Conference on Learning Represe...

    2023

  28. [36]

    Tong Zhang, Jiali Mao, Shunyu Yao, Rui Wang, and Yixin Cao. 2024. CLAMBER: A Benchmark of Identifying and Clarifying Ambiguous Information Needs in Large Language Models. InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics. Association for...

    2024

  29. [37]

    InProceedings of the 2021 Conference on Empirical Methods in Natural Language Processing

    Open-domain clarification question generation without question examples. InProceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics, Online and Punta Cana, Dominican Republic, 563–570. doi:10.18653/v1/202...

    2021 doi

  30. [38]

    Ziliang Zhao, Haonan Chen, Shiren Song, Jian Xie, and Zhicheng Dou. 2025. ClariLM: Enhancing Open-domain Clarification Ability for Large Language Mod- els. InProceedings of the 34th ACM International Conference on Information and Knowledge Management. Association for Computing...

    2025

  31. [39]

    Ziliang Zhao, Zhicheng Dou, and Yujia Zhou. 2024. Generating Intent-aware Clar- ifying Questions in Conversational Information Retrieval Systems. InProceedings of the 33rd ACM International Conference on Information and Knowledge Manage- ment. Association for Computing Machine...

    2024

  32. [40]

    Zhanke Zhou, Xiao Feng, Zhaocheng Zhu, Jiangchao Yao, Sanmi Koyejo, and Bo Han. 2025. From Passive to Active Reasoning: Can Large Language Models Ask the Right Questions under Incomplete Information? arXiv:2506.08295 [cs.CL] https://arxiv.org/abs/2506.08295 arXiv preprint

    2025 arXiv

  33. [41]

    Michael J. Q. Zhang, W. Bradley Knox, and Eunsol Choi. 2024. Modeling Future Conversation Turns to Teach Large Language Models to Ask Clarifying Ques- tions. arXiv:2410.13788 [cs.CL] https://arxiv.org/abs/2410.13788 arXiv preprint; submitted to ICLR 2025

    2024 arXiv

  34. [43]

    Yiming Zhang, Lingfei Wu, Qi Shen, Yitong Pang, Zhihua Wei, Fangli Xu, Bo Long, and Jian Pei. 2022. Multiple Choice Questions Based Multi-Interest Policy Learning for Conversational Recommendation. InProceedings of the ACM Web Conference 2022. ACM, Lyon, France, 2153–2162. doi...

    2022

  35. [2021]

    InProceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies

    Ask What’s Missing and What’s Useful: Improving Clarification Question Generation Using Global Knowledge. InProceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Association for Computati...

    2021

  36. [2024]

    doi:10.1145/3589334.3645424

    Association for Computing Machinery, New York, NY, USA, 1386–1394. doi:10.1145/3589334.3645424

  37. [4312]

    https://aclanthology.org/2021.naacl-main.340/

    2021

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.