Pith. sign in

REVIEW 3 major objections 5 minor 57 references

Curiosity by Design: An LLM-based Coding Assistant Asking Clarification Questions

T0 review · 3 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that an LLM coding assistant that detects vague prompts and asks a clarification question before generating code produces answers users prefer over a one-shot baseline.

desk verdict An honest and clearly written systems paper with a defensible RQ1 result but a confounded RQ2 comparison; worth reviewing with major revision. read the letter →

arxiv 2507.21285 v1 pith:ZXJWJNFO submitted 2025-07-28 cs.AI

Harsh Darji , Thibaud Lutellier This is my paper
classification cs.AI
keywords clarificationquestionscodingassistantsunder-specifiedpromptsintentclassificationcodegenerationfine-tuninguserstudyLLM
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that an LLM coding assistant can do better than guessing when a developer's prompt is vague: it should detect the under-specification and ask a targeted clarification question before generating code. To test this, the authors built a two-stage pipeline out of a DistilBERT clarity classifier and a fine-tuned Gemma-3-1B-IT question generator, and compared it against a one-shot baseline in two user studies. Users preferred the clarification questions in 68% of cases on precision and focus, and rated the final answers from the full pipeline higher in 82% of cases for precision and focus, 78% for contextual fit, 80% for answer faithfulness, and 66% for correctness, all statistically significant. If the result holds, interactive clarification becomes a cheap, trainable improvement for coding assistants rather than a prompt-engineering trick.

What carries the argument

The engine of the system is a two-stage clarification loop: a DistilBERT classifier fine-tuned on 4,161 synthetic examples rates each prompt on a four-point clarity scale and, if under-specified, routes it to a Gemma-3-1B-IT model fine-tuned with LoRA on 9,969 prompt–clarification pairs; the generated question is answered by the user, the augmented prompt is re-checked, and only then does the answering module produce code. The loop is what converts an ambiguous request into a concrete specification before generation begins.

What would settle it

Run an A/B user study on a held-out set of naturally occurring under-specified prompts collected from public issue trackers or Q&A sites, using the same rating scales; if the clarification pipeline no longer beats the one-shot baseline at a statistically significant margin on precision/focus and correctness, the central claim fails outside the synthetic distribution.

Watch

Extended reading notes

Core claim

On its own terms, the paper claims that under-specified coding prompts are better handled by an explicit clarification turn than by one-shot generation. It reports that a fine-tuned Gemma-3-1B-IT clarification module, trained on 9,969 synthetic prompt–question pairs generated by GPT-4o-mini, produces questions users significantly prefer to a one-shot baseline across precision and focus (mean 3.9 out of 5, 68% favoring), immediate editability (mean 3.55, 53%), and contextual fit (mean 3.56, 45%, with 87% equivalent-or-better), at p < 0.001. When those questions are inserted into a full pipeline with a DistilBERT intent classifier and the user's answer is fed back before final code generation, users prefer the pipeline's final answer in 82% of cases on precision and focus, 78% on contextual fit, 80% on answer faithfulness, and 66% on correctness (with 100% at least as correct as the baseline), again at p < 0.001. The paper presents these findings as evidence that clarification-driven interaction is a promising direction for LLM-based coding assistants.

Load-bearing premise

The load-bearing premise is that synthetic under-specified prompts generated by GPT-4o-mini with the same template as the training data represent how real developers phrase vague coding requests, since both user studies are conducted on that distribution.

Editorial extensions

If this is right

  • Users prefer clarification-generated answers over one-shot answers on four quality axes, so interactive clarification can be a default behavior rather than a special mode.
  • A 1B-parameter model fine-tuned on synthetic pairs can generate questions users rate useful, suggesting the clarification skill is learnable without frontier-scale models.
  • A lightweight classifier can gate whether to ask, keeping detection overhead below 0.1 seconds; the main cost is question generation (about 133 seconds per prompt on the tested setup).
  • Because answer faithfulness and correctness also improve, clarification plausibly reduces the need for users to engineer prompts or supply tests upfront.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because both user studies use prompts synthesized with the same template as the training data, a natural next test is to rerun the comparison on naturally occurring under-specified prompts from version-control issues or Q&A sites; the paper's own note that real GitHub data was too noisy for training suggests this distribution gap is the main open risk.
  • The clarification model's tendency to ask failure-handling questions (for example, what should happen if the API call fails) hints that clarification may push generated code toward defensive or security-aware patterns; a controlled vulnerability comparison with and without clarification would test this.
  • The same two-stage ask-then-answer loop could transfer to other under-specified generation tasks, such as data-analysis or documentation requests, where the intent gap is similarly wide.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes an end-to-end LLM-based coding assistant that first classifies whether a coding prompt is under-specified and, if so, generates clarification questions before producing a final code answer. The system consists of a DistilBERT intent-clarity classifier and a LoRA-fine-tuned Gemma-3-1B-IT clarification module, both trained on synthetic prompt–clarification data generated by GPT-4o-mini. Two user studies are reported: RQ1 compares the clarification questions from the fine-tuned module against a one-shot baseline, and RQ2 compares the full pipeline's final answer (including a simulated user response) against a baseline that receives only the original prompt. The authors report strong user preferences for both the clarification questions and the final answers, and also report classifier accuracy, perplexity, and pipeline overhead results.

Significance. If the findings hold, the paper makes a practical contribution: it shows that a small, fine-tuned model can generate clarification questions that users prefer over a one-shot baseline, and that inserting a clarification turn can improve the perceived quality of final code answers. The authors explicitly release their code, datasets, user study templates, and models, which strengthens reproducibility. The use of a small DeciLM/Gemma-3-1B-IT model with LoRA is a reasonable efficiency-oriented design. However, the central external-validity claims rest on user studies conducted entirely on synthetic prompts generated with the same template used for training, and the RQ2 comparison is confounded by the inclusion of a simulated user response that provides extra information only to the treatment condition. These issues limit what can currently be concluded about real-world developer queries.

major comments (3)
  1. [§4.2 and §5.1] The external validity of both user studies is the load-bearing weakness. Section 4.2 states that the RQ1 evaluation prompts were synthesized with GPT-4o-mini using the same prompt template as the training dataset, and Section 5.1 (Internal Validity) reports that real GitHub issue data was too noisy and led to lower model performance. Consequently, the user preferences reported in Findings 1 and 2 may reflect in-distribution performance on the training distribution rather than an ability to handle realistic, out-of-distribution developer queries. To support the generalization claim, the authors should evaluate on genuinely held-out real-world queries (e.g., StackOverflow or GitHub issues) or on systematically out-of-distribution prompts, reporting classifier accuracy and clarification quality separately for that data.
  2. [§4.4 and §5.1] The RQ2 comparison is confounded. The treatment condition receives the original under-specified prompt, the generated clarification question, and a simulated user response that supplies the missing specifications, while the baseline receives only the original prompt before generating its final answer. The reported 66–82% preference for the pipeline's final answer could therefore be driven by the additional information in the simulated response rather than by the quality of the generated clarification question. A proper control would give the baseline the same simulated user response without the clarification question (or an oracle-prompt baseline with gold specifications), and a clarification-generating baseline such as ClarifyGPT should be included to isolate the contribution of the trained module. The paper's own Section 5.1 conclusion-validity caveat acknowledges that simulation may misrepresent real interactions, but the confound remains in the RQ2 design.
  3. [§4.3 and Finding 1] The effect-size reporting is internally inconsistent. The RQ1 findings state a medium effect for precision and focus (d = 0.61) and small effects for the other metrics (0.3 < d < 0.5), but the same paragraph and Finding 1 claim 'large effect sizes (Cohen's d > 0.8)'. Additionally, the statistical test used to test the null hypothesis that the rating equals 3 is not named, and no correction for multiple comparisons is reported. Please reconcile the effect-size statements and specify the exact test procedure.
minor comments (5)
  1. [§4.2] The text says participants reviewed 10 interactions for a total of 100 unique interactions, which implies 10 participants, but the number of participants should be stated explicitly.
  2. [§4.5] The sentence '100% of cases were deemed at least as correct as the baseline' should be clarified by stating the breakdown, e.g., 66% better and 34% equal, so that the distribution is unambiguous.
  3. [§4.6] The Intent Classifier's routing threshold on the 4-point clarity scale is not reported; please describe how the 4-point scores were mapped to the binary route decision and report a small sensitivity analysis around that cutoff.
  4. [§4.6] The reported 4% perplexity reduction is presented without variance or a significance test; confidence intervals or a paired test would make the quantitative result more informative.
  5. [§2.3 and §4.2] ClarifyGPT is discussed in related work but is not used as a baseline in either user study; adding it as a clarification-generating baseline would strengthen the RQ1 comparison.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity found; the central preference claims rest on human ratings, not on the training data or self-citations.

full rationale

The paper's central claims (Finding 1 and Finding 2) are supported by pairwise A/B user studies in which human raters judged clarification questions and final code answers. These human judgments are external to the model's training objective and are not derived from the same equations or fitted parameters used to build the system. The fine-tuned clarification module and intent classifier are trained on synthetic GPT-4o-mini data, and the RQ1 evaluation prompts are synthesized with the same prompt template (Section 4.2); this is a genuine external-validity threat, especially given the authors' report that real GitHub data was too noisy and led to lower model performance (Section 5.1). However, this is a distribution-matching concern rather than a circular reduction: the measured outcome is a human preference rating, not a quantity that is equal to the training loss, the data-generation template, or a self-cited result. RQ2's simulated user responses also introduce a confound, because the pipeline receives extra specification information that the baseline does not receive, but again this is an experimental design issue rather than a case where a predicted quantity is equivalent to an input by construction. The paper does not rely on load-bearing self-citations, uniqueness theorems, or ansatz smuggling, and no quoted passage exhibits a step where a claimed prediction reduces to its own input. Accordingly, no specific circular step can be exhibited, and the appropriate circularity score is 0.

Assumptions & free parameters 2 free parameters · 3 assumptions · 0 invented entities

All substantive claims depend on synthetic data fidelity and on small user-study ratings; the two free parameters are unspecified routing thresholds. No invented entities are proposed.

free parameters (2)
  • Clarity routing cutoff on the 4-point scale = not reported
    Section 3.2 uses a 4-point clarity scale, but the paper never states which score separates clear prompts from under-specified ones, so the operating point of the pipeline is undefined.
  • Clarification loop iteration limit = not reported
    Section 3.1 says the loop continues until the prompt is clear or a threshold is reached, but no threshold value or stopping rule is given.
assumptions (3)
  • domain assumption GPT-4o-mini-generated synthetic prompts and clarification pairs are suitable training and evaluation data for detecting and resolving under-specified coding requests.
    Invoked in Sections 3.2, 3.3, and 4.2; the paper justifies this with references to prior work but provides no external validation against real developer queries.
  • domain assumption A DistilBERT classifier with 73% cross-validation accuracy is accurate enough that routing errors do not degrade the overall system.
    Stated in Section 5.1 (Construct Validity): false positives are claimed to be low-cost, but this claim is not measured in the user study.
  • domain assumption Simulated user responses in RQ2 (generated by a larger LLM) approximate real user replies closely enough to support conclusions about final answer quality.
    Acknowledged as a threat in Section 5.1; the paper mitigates by separate rating, but the assumption remains load-bearing for RQ2.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Curiosity by Design: An LLM-based Coding Assistant Asking Clarification Questions." pith.science (2026). https://pith.science/paper/ZXJWJNFO

@misc{pith2026250721285,
  author       = {Pith},
  title        = {Pith review of: Curiosity by Design: An LLM-based Coding Assistant Asking Clarification Questions},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ZXJWJNFO}},
  note         = {Machine review of arXiv:2507.21285}
}
read the original abstract

Large Language Models (LLMs) are increasingly used as coding assistants. However, the ambiguity of the developer's prompt often leads to incorrect code generation, as current models struggle to infer user intent without extensive prompt engineering or external context. This work aims to build an LLM-based coding assistant that mimics the human code review process by asking clarification questions when faced with ambiguous or under-specified queries. Our end-to-end system includes (1) a query classifier trained to detect unclear programming-related queries and (2) a fine-tuned LLM that generates clarification questions. Our evaluation shows that the fine-tuned LLM outperforms standard zero-shot prompting in generating useful clarification questions. Furthermore, our user study indicates that users find the clarification questions generated by our model to outperform the baseline, demonstrating that our coding assistant produces more accurate and helpful code responses compared to baseline coding assistants.

Figures

Figures reproduced from arXiv: 2507.21285 by the authors.

Figure 1
Figure 1. Example of under-specified prompt to an LLM. [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Overview of our approach 3 Approach This section presents our proposed approach. Section 3.1 presents an overview of our system, Section 3.2 describes our Intent Clarity Classifier, Section 3.3 presents the Clarification Module, and Section 3.4 describes the Answering Module we use. 3.1 Overview [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Clarification Questions and final answer for the prompt in Figure 1. [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: a shows box-plot results of the RQ1 user study for the three metrics. A rating of 1 indicates that the user strongly preferred the baseline while a rating of 5 indicates that the user rated the questions generated by our approach as strongly preferable to the baseline.…

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

57 extracted references · 38 canonical work pages

  1. [1]

    Mohammad Aliannejadi, Julia Kiseleva, Aleksandr Chuklin, Jeff Dalton, and Mikhail Burtsev

    work page

  2. [2]

    Chinmaya Andukuri, Jan-Philipp Fränken, Tobias Gerstenberg, and Noah D Goodman

    work page

  3. [3]

    Yasharth Bajpai, Bhavya Chopra, Param Biyani, Cagri Aslan, Dustin Coleman, Sumit Gulwani, Chris Parnin, Arjun Radhakrishna, and Gustavo Soares. 2024. Let’s Fix this Together: Conver- sational Debugging with GitHub Copilot. In 2024 IEEE Symposium on Visual Languages and Human-Centric Computing (VL/HCC). IEEE, 1–12

    work page 2024

  4. [4]

    Shraddha Barke, Michael B James, and Nadia Polikarpova. 2023. Grounded copilot: How programmers interact with code-generating models. Proceedings of the ACM on Programming Languages 7, OOPSLA1 (2023), 85–111

    2023

  5. [5]

    Lenz Belzner, Thomas Gabor, and Martin Wirsing. 2023. Large language model assisted software engineering: prospects, challenges, and a case study. In International Conference on Bridging the Gap between AI and Reality. Springer, 355–374

    work page 2023

  6. [6]

    Pavel Braslavski, Denis Savenkov, Eugene Agichtein, and Alina Dubatovka. 2017. What do you mean exactly? Analyzing clarification questions in CQA. In Proceedings of the 2017 conference on conference human information interaction and retrieval. 345–348

    work page 2017

  7. [7]

    Valerie Chen, Alan Zhu, Sebastian Zhao, Hussein Mozannar, David Sontag, and Ameet Tal- walkar. 2024. Need Help? Designing Proactive AI Assistants for Programming. arXiv preprint arXiv:2410.04596 (2024)

    arXiv 2024

  8. [8]

    James CL Chow, Valerie Wong, and Kay Li. 2024. Generative pre-trained transformer- empowered healthcare conversations: Current trends, challenges, and future directions in large language model-enabled medical chatbots. BioMedInformatics 4, 1 (2024), 837–852

    work page 2024

Show all 57 references
  1. [9]

    Xin Luna Dong, Seungwhan Moon, Yifan Ethan Xu, Kshitiz Malik, and Zhou Yu. 2023. Towards next-generation intelligent assistants leveraging llm techniques. InProceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining. 5792–5793

    2023

  2. [10]

    Madeline Endres, Sarah Fakhoury, Saikat Chakraborty, and Shuvendu K Lahiri. 2024. Can large language models transform natural language intent into formal method postconditions? Proceedings of the ACM on Software Engineering 1, FSE (2024), 1889–1912

    2024

  3. [11]

    Deborah Etsenake and Meiyappan Nagappan. 2024. Understanding the Human-LLM Dynamic: A Literature Survey of LLM Use in Programming Tasks. arXiv preprint arXiv:2410.01026 (2024)

    2024 arXiv

  4. [12]

    Sarah Fakhoury, Aaditya Naik, Georgios Sakkas, Saikat Chakraborty, and Shuvendu K. Lahiri. 2024. LLM-Based Test-Driven Interactive Code Generation: User Study and Em- pirical Evaluation. IEEE Transactions on Software Engineering 50, 9, 2254–2268. https: //doi.org/10.1109/TSE.2...

    2024

  5. [13]

    Sarah Fakhoury, Aaditya Naik, Georgios Sakkas, Saikat Chakraborty, and Shuvendu K Lahiri

  6. [14]

    Yujian Gan, Changling Li, Jinxia Xie, Luou Wen, Matthew Purver, and Massimo Poesio. 2024. Clarq-llm: A benchmark for models clarifying and requesting information in task-oriented dialog. arXiv preprint arXiv:2409.06097 (2024)

    2024 arXiv

  7. [15]

    Heng Gu, Chadha Degachi, U˘gur Genç, Senthil Chandrasegaran, and Himanshu Verma. 2023. On the effectiveness of creating conversational agent personalities through prompting. arXiv preprint arXiv:2310.11182 (2023). 10

    2023 arXiv

  8. [16]

    IEEE Transactions on Software Engineering (2024)

    Llm-based test-driven interactive code generation: User study and empirical evaluation. IEEE Transactions on Software Engineering (2024)

    2024

  9. [17]

    Hao Guan, Guangdong Bai, and Yepang Liu. 2024. Large language models can connect the dots: Exploring model optimization bugs with domain knowledge-aware prompts. In Proceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis. 1579– 1591

    2024

  10. [18]

    Kailun Jin, Chung-Yu Wang, Hung Viet Pham, and Hadi Hemmati. 2024. Can chatgpt sup- port developers? an empirical evaluation of large language models for code generation. In Proceedings of the 21st International Conference on Mining Software Repositories. 167–171

    2024

  11. [19]

    Ken Gu, Madeleine Grunde-McLaughlin, Andrew McNutt, Jeffrey Heer, and Tim Althoff. 2024. How do data analysts respond to ai assistance? a wizard-of-oz study. InProceedings of the 2024 CHI Conference on Human Factors in Computing Systems. 1–22

    2024

  12. [20]

    Majeed Kazemitabaar, Xinying Hou, Austin Henley, Barbara Jane Ericson, David Weintrop, and Tovi Grossman. 2023. How novices use LLM-based code generators to solve CS1 coding tasks in a self-paced learning environment. In Proceedings of the 23rd Koli calling international confe...

    2023

  13. [21]

    Kimiya Keyvan and Jimmy Xiangji Huang. 2022. How to approach ambiguous queries in conversational search: A survey of techniques, approaches, tools, and challenges. Comput. Surveys 55, 6 (2022), 1–40

    2022

  14. [22]

    Samia Kabir, David N Udo-Imeh, Bonan Kou, and Tianyi Zhang. 2024. Is stack overflow obsolete? an empirical study of the characteristics of chatgpt answers to stack overflow questions. In Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems. 1–17

    2024

  15. [23]

    Sylvain Kouemo Ngassom, Arghavan Moradi Dakhel, Florian Tambon, and Foutse Khomh

  16. [24]

    Vaibhav Kumar and Alan W Black. 2020. ClarQ: A large-scale and diverse dataset for Clar- ification Question Generation. In Proceedings of the 58th Annual Meeting of the Associa- tion for Computational Linguistics , Dan Jurafsky, Joyce Chai, Natalie Schluter, and Joel Tetreault...

    2020 doi

  17. [25]

    Ranim Khojah, Mazen Mohamad, Philipp Leitner, and Francisco Gomes de Oliveira Neto. 2024. Beyond code generation: An observational study of chatgpt usage in software engineering practice. Proceedings of the ACM on Software Engineering 1, FSE (2024), 1819–1840

    2024

  18. [26]

    Chaitanya Malaviya, Joseph Chee Chang, Dan Roth, Mohit Iyyer, Mark Yatskar, and Kyle Lo

  19. [27]

    In Proceedings of the 1st ACM International Conference on AI-Powered Software

    Chain of Targeted Verification Questions to Improve the Reliability of Code Generated by LLMs. In Proceedings of the 1st ACM International Conference on AI-Powered Software. 122–130

  20. [28]

    Ahmad Mohsin, Helge Janicke, Adrian Wood, Iqbal H Sarker, Leandros Maglaras, and Naeem Janjua. 2024. Can we trust large language models generated code? a framework for in- context learning, security patterns, and code evaluations across diverse llms. arXiv preprint arXiv:2406....

    2024 arXiv

  21. [29]

    Jenny T Liang, Chenyang Yang, and Brad A Myers. 2024. A large-scale survey on the usability of ai programming assistants: Successes and challenges. In Proceedings of the 46th IEEE/ACM international conference on software engineering. 1–13

    2024

  22. [30]

    Fangwen Mu, Lin Shi, Song Wang, Zhuohao Yu, Binquan Zhang, ChenXue Wang, Shichao Liu, and Qing Wang. 2024. ClarifyGPT: A Framework for Enhancing LLM-Based Code Generation via Requirements Clarification. Proc. ACM Softw. Eng.1, FSE, Article 103 (July 2024), 23 pages. https://do...

    2024 doi

  23. [31]

    arXiv preprint arXiv:2411.07237 (2024)

    Contextualized evaluations: Taking the guesswork out of language model evaluations. arXiv preprint arXiv:2411.07237 (2024)

    2024 arXiv

  24. [32]

    Noble Saji Mathews and Meiyappan Nagappan. 2024. Test-Driven Development and LLM- based Code Generation. In Proceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering. 1583–1594

    2024

  25. [33]

    Soya Park and Chinmay Kulkarni. 2023. Thinking assistants: Llm-based conversational assistants that help users think by asking rather than answering.arXiv preprint arXiv:2312.06024 (2023)

    2023 arXiv

  26. [34]

    Nickolas Anselmo Carneiro Mororo, Jorge Luiz Araújo, Rafael Bomfim, and Vasco Furtado

  27. [35]

    Steven I Ross, Fernando Martinez, Stephanie Houde, Michael Muller, and Justin D Weisz

  28. [36]

    Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2019. DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter.arXiv preprint arXiv:1910.01108 (2019)

    2019 arXiv

  29. [37]

    Sydney Nguyen, Hannah McLean Babe, Yangtian Zi, Arjun Guha, Carolyn Jane Anderson, and Molly Q Feldman. 2024. How beginning programmers and code llms (mis) read each other. In Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems. 1–26

    2024

  30. [38]

    Elise Paradis, Kate Grey, Quinn Madison, Daye Nam, Andrew Macvean, Vahid Meimand, Nan Zhang, Ben Ferrari-Church, and Satish Chandra. 2024. How much does AI impact development speed? An enterprise-based randomized controlled trial. arXiv preprint arXiv:2410.12944 (2024)

    2024 arXiv

  31. [39]

    David H Smith IV , Paul Denny, and Max Fowler. 2024. Prompting for comprehension: Exploring the intersection of explain in plain english questions and prompt writing. In Proceedings of the Eleventh ACM Conference on Learning@ Scale. 39–50

    2024

  32. [40]

    Hossein A Rahmani, Xi Wang, Yue Feng, Qiang Zhang, Emine Yilmaz, and Aldo Lipani. 2023. A survey on asking clarification questions datasets in conversational systems. arXiv preprint arXiv:2305.15933 (2023)

    2023 arXiv

  33. [41]

    Gemma Team, Aishwarya Kamath, Johan Ferret, Shreya Pathak, Nino Vieillard, Ramona Merhej, Sarah Perrin, Tatiana Matejovicova, Alexandre Ramé, Morgane Rivière, et al. 2025. Gemma 3 technical report. arXiv preprint arXiv:2503.19786 (2025)

    2025 arXiv

  34. [42]

    Rebeka Tóth, Tamas Bisztray, and László Erd˝odi. 2024. LLMs in Web Development: Evalu- ating LLM-Generated PHP Code Unveiling Vulnerabilities and Limitations. InInternational Conference on Computer Safety, Reliability, and Security. Springer, 425–437

    2024

  35. [43]

    Man-Fai Wong, Shangxin Guo, Ching-Nam Hang, Siu-Wai Ho, and Chee-Wei Tan. 2023. Natural language generation and understanding of big code for AI-assisted programming: A review. Entropy 25, 6 (2023), 888

    2023

  36. [44]

    Agnia Sergeyuk, Yaroslav Golubev, Timofey Bryksin, and Iftekhar Ahmed. 2025. Using AI- based coding assistants in practice: State of affairs, perceptions, and ways forward. Information and Software Technology178 (2025), 107610

    2025

  37. [45]

    Agnia Sergeyuk, Sergey Titov, and Maliheh Izadi. 2024. In-ide human-ai experience in the era of large language models; a literature review. InProceedings of the 1st ACM/IEEE Workshop on Integrated Development Environments. 95–100

    2024

  38. [46]

    Huan Zhang, Wei Cheng, Yuhan Wu, and Wei Hu. 2024. A Pair Programming Framework for Code Generation via Multi-Plan Exploration and Feedback-Driven Refinement. In Proceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering. 1319– 1331

    2024

  39. [47]

    Leitian Tao, Xiang Chen, Tong Yu, Tung Mai, Ryan Rossi, Yixuan Li, and Saayan Mitra. 2024. Codelutra: Boosting llm code generation via preference-guided refinement. arXiv preprint arXiv:2411.05199 (2024)

    2024 arXiv

  40. [48]

    Ziyao Zhang, Yanlin Wang, Chong Wang, Jiachi Chen, and Zibin Zheng. 2024. Llm halluci- nations in practical code generation: Phenomena, mechanism, and mitigation. arXiv preprint arXiv:2409.20550 (2024)

    2024 arXiv

  41. [49]

    Wenting Zhao, Alexander M Rush, and Tanya Goyal. 2024. Challenges in trustworthy human evaluation of chatbots. arXiv preprint arXiv:2412.04363 (2024)

    2024 arXiv

  42. [50]

    Zibin Zheng, Kaiwen Ning, Qingyuan Zhong, Jiachi Chen, Wenqing Chen, Lianghong Guo, Weicheng Wang, and Yanlin Wang. 2025. Towards an understanding of large language models in software engineering tasks. Empirical Software Engineering 30, 2 (2025), 50. 13

    2025

  43. [51]

    Fan Yang and Jiansong Zhang. 2024. Prompt-based automation of building code information transformation for compliance checking. Automation in Construction 168 (2024), 105817. 12

    2024

  44. [52]

    Hamed Zamani, Johanne R Trippas, Jeff Dalton, Filip Radlinski, et al. 2023. Conversational information seeking. Foundations and Trends® in Information Retrieval 17, 3-4 (2023), 244– 456

    2023

  45. [54]

    Quanjun Zhang, Chunrong Fang, Yang Xie, Yaxin Zhang, Yun Yang, Weisong Sun, Shengcheng Yu, and Zhenyu Chen. 2023. A survey on large language models for software engineering. arXiv preprint arXiv:2312.15223 (2023)

    2023 arXiv

  46. [2020]

    arXiv preprint arXiv:2009.11352 (2020)

    ConvAI3: Generating clarifying questions for open-domain dialogue systems (ClariQ). arXiv preprint arXiv:2009.11352 (2020)

    2020 arXiv

  47. [2023]

    In Proceedings of the 28th International Conference on Intelligent User Interfaces

    The programmer’s assistant: Conversational interaction with a large language model for software development. In Proceedings of the 28th International Conference on Intelligent User Interfaces. 491–514

  48. [2024]

    arXiv preprint arXiv:2403.19154 (2024)

    Star-gate: Teaching language models to ask clarifying questions. arXiv preprint arXiv:2403.19154 (2024)

    2024 arXiv

  49. [2025]

    In Simpósio Brasileiro de Sistemas de Informação (SBSI)

    Providing Task Execution Capabilities in LLM-Based Conversational Assistants. In Simpósio Brasileiro de Sistemas de Informação (SBSI). SBC, 743–750. 11

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.