Pith. sign in

REVIEW 4 major objections 6 minor 2 cited by

From Critique to Clarity: A Pathway to Faithful and Personalized Code Explanations with Large Language Models

T0 review · 4 major / 6 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Two separate iterative critique loops make LLM code explanations both more faithful to the code and more tailored to the reader.

desk verdict Nice two-loop pipeline with a real execution anchor, but the faithfulness metric measures solvability rather than fidelity to the given solution, and the personalization evidence relies on proxy metrics with no human validation. read the letter →

arxiv 2501.14731 v1 pith:K3XZLCMJ submitted 2024-12-08 cs.SE cs.AIcs.CL

Zexing Xu , Zhuang Luo , Yichuan Li , Kyumin Lee , S. Rasoul Etesami This is my paper · ORCID
classification cs.SEcs.AIcs.CL
keywords codeexplanationgenerationlargelanguagemodelspersonalizationfaithfulnessiterativerefinementself-correctionrole-playingjudgeContests
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that code explanations from large language models can be made both faithful and personalized by iterating through two separate refinement loops: one that checks an explanation by using it to regenerate working code and running that code against tests, and one that rewrites the explanation to match a user profile inferred from their Stack Overflow history. The authors argue that a single pass, or even self-selection among several samples, is not enough; repeated critique-and-revise cycles with external verification are what push quality up. They validate the claim on competitive-programming problems from CodeContests, where the iterative method improves the rate at which an explanation lets another LLM solve the problem and raises personalization metrics such as ROUGE-L and word overlap with users' past questions. If true, this gives developers and stakeholders a practical route to explanations that are both technically accurate and tailored to the reader's background.

What carries the argument

The central mechanism is the Self-Iteration refinement pipeline: two independent three-stage loops, each with reflection (summarizing the problem or the user profile), iterative explanation (chain-of-thought initialization plus revision conditioned on prior output and critique), and verification-and-analysis (external execution or a role-playing judge). The faithfulness loop's verification step is the load-bearing device: the explanation is judged by whether it lets another LLM generate code that passes the test cases, turning faithfulness into an observable, executable criterion. The personalization loop's load-bearing device is the role-playing judge that rates whether an explanation matches the inferred user profile and supplies revision suggestions until the rating is satisfactory.

What would settle it

A user study in which the same ten (or more) Stack Overflow users, or a comparable panel, rate the personalized explanations blind: if users do not prefer Self-Iteration's personalized explanations over the baseline, or if their preferences do not correlate with higher ROUGE-L, word overlap, and win-rate scores, then the personalization claim fails.

Watch

Extended reading notes

Core claim

The central claim, stated on the paper's own terms, is that faithful and personalized code explanation is achievable as a generation task by separating the two objectives into independent iterative loops. The faithfulness loop generates an explanation, uses it to prompt a fresh code solution, executes that solution against public tests, and feeds any failure back as a revision suggestion; the personalization loop infers a six-aspect programming profile from a user's recent Python, data-structure, and algorithm questions on Stack Overflow, drafts a tailored explanation, and has a role-playing judge LLM score it against the profile, repeating until satisfied. The final output concatenates the faithful explanation and the personalized explanation. On CodeContests, with GPT-3.5-turbo as backbone, the method reports higher Pass@k than greedy self-consistency and self-selection baselines, and higher ROUGE-L, word overlap, and win-rate scores for personalization, leading the paper to conclude that iterative refinement with critique-response loops improves both qualities.

Load-bearing premise

The load-bearing premise is that the personalization metrics—word and phrase overlap with a user's five most recent Stack Overflow questions and ratings from a role-playing LLM judge—actually measure whether an explanation is useful to that user, because the paper reports no human evaluation of personalization.

Editorial extensions

If this is right

  • Explanations produced by Self-Iteration should transfer to a fresh LLM: a downstream model given only the explanation solves CodeContests problems more often than with baseline explanations, so the explanations carry real problem-solving content.
  • The method can serve as a data-annotation pipeline: iteratively refined explanations of the CodeContests training set could produce large numbers of 'silver' explanations for fine-tuning reasoning models, a direction the paper explicitly points to.
  • Because the two loops are independent, practitioners can apply faithfulness refinement without personalization or vice versa, and can tune each loop's stopping criteria separately.
  • The reported gains come from iteration depth (four rounds) rather than simply sampling more candidates, since Self-Iteration outperforms Self-Selection with the same number of samples.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A testable extension is to swap the role-playing judge for a human-in-the-loop rating in the personalization loop; if human ratings disagree with the LLM judge, the loop would need a different reward signal.
  • The faithfulness loop implicitly defines 'faithful' as 'sufficient for re-implementation.' That is a particular, executable notion of faithfulness; other notions, such as line-by-line correspondence to the oracle solution, might rank the same explanations differently.
  • Since user profiles are inferred from only five recent Stack Overflow inquiries, the approach could be sensitive to the recency and topic mix of those questions; profiles built from longer or multi-source history would test how robust the personalization loop is.
  • The two-loop separation suggests a modular product design: a general faithfulness service plus a pluggable personalization layer keyed to a user model, which could be reused beyond competitive programming.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes Self-Iteration, a two-loop iterative refinement method for generating code explanations for competitive programming problems. A faithfulness loop generates an explanation of a problem-solution pair and verifies it indirectly by asking an LLM to produce a code solution from the explanation and executing that solution on public test cases; a personalization loop infers a user profile from Stack Overflow inquiry history and uses a role-playing LLM judge to rate and refine the explanation. Experiments on the CodeContests validation and test sets compare Self-Iteration against a greedy baseline and a Self-Selection baseline, reporting Pass@k, win rate, ROUGE-L, and word overlap ratio. The paper claims that the method produces more faithful and more personalized code explanations and states in the abstract and conclusion that both automatic and human evaluations were performed, although no human evaluation is reported in the text.

Significance. If the results held, this would be a useful contribution to code explanation generation, particularly for the new task of personalized code explanations. The architecture is clearly described, and the faithfulness loop has a genuinely external anchor through the Python executor and public test cases. The paper also takes reasonable steps to mitigate data leakage by using only the CodeContests validation and test splits, and it reports error bars on most automatic metrics. However, the two central claims rest on metrics whose validity is not established: Pass@k is not shown to measure faithfulness to the oracle solution, and the personalization metrics are lexical overlap with past Stack Overflow inquiries plus a self-referential role-playing judge. The claimed human evaluation is absent. The contribution is therefore conditional on additional validation.

major comments (4)
  1. [§5, Table 1; §3.2] Pass@k measures whether a fresh LLM, given the problem and the candidate explanation, can generate a code solution that passes public tests; it does not measure whether the explanation is faithful to the particular oracle solution s. Because Eq. (1) defines the explanation as conditioned on the pair (p, s), a generic problem-solving hint could raise Pass@k without describing the logic of s at all. The verification loop in §3.2 never checks whether any statement in e_i is entailed by s, so the reported Pass@k gains do not substantiate the claim that Self-Iteration produces more faithful explanations. A direct evaluation against s (e.g., human ratings of explanation-solution consistency or an automatic entailment-style check) is needed.
  2. [Abstract and §6] The abstract states that the authors evaluate with 'both automatic and human assessments,' and the conclusion repeats that the method is validated 'no matter evaluated by the automatic evaluation or the human evaluation.' No human evaluation protocol, results, or analysis appears anywhere in Sections 4 or 5. This is a missing support for a stated contribution. The authors should either add the human study or remove the claim from the abstract and conclusion.
  3. [§3.3, §4.1, Tables 2-3, Figure 2] The personalization evidence rests on three proxy metrics: ROUGE-L, word overlap ratio, and win rate. The first two measure lexical overlap between the generated explanation and the user's five most recent Stack Overflow inquiries, and the win rate is assigned by a role-playing LLM judge whose persona is based on a profile inferred by the same GPT-3.5-turbo model that generates the explanations. The paper provides no evidence that lexical similarity to past questions or self-ratings by the generating model track what a user actually needs. Without a human evaluation or an independent external validation of the judge, the personalization claim in the paper's conclusion is not supported.
  4. [§3, Eq. (3), §5] The final output is defined in Eq. (3) as o = e ⊕ p_e, combining the faithfulness-loop explanation and the personalization-loop explanation. However, the reported evaluations treat the two components separately: Table 1 evaluates e through Pass@k, while Tables 2-3 and Figure 2 evaluate p_e through lexical and judge-based metrics. The actual combined output o is never evaluated. Thus the paper does not demonstrate that the method simultaneously produces a final explanation that is both faithful and personalized, despite the stated goal of balancing the two objectives.
minor comments (6)
  1. [§4, Backbone Model] All experiments use only GPT-3.5-turbo as the backbone and judge; the paper does not test whether the observed improvements hold for other LLMs, so the generality of the method remains unclear.
  2. [Table 1] The two 'Commercial Product' baselines are unnamed and undescribed, which prevents a reader from understanding the comparison or reproducing it.
  3. [§4.1] The Pass@k description says the solutions are checked 'against ground-truth outputs, derived from private or generated test cases,' while §3.2 refers to 'public test cases.' The paper should clarify which test cases are actually used and whether private tests exist.
  4. [§4.1] The reported standard deviations are said to come from '40 calculations (10 chosen users * 4 samples per user),' but it is not stated whether the averaging is over users, over problems, or over both; the aggregation procedure should be specified.
  5. [References] Several reference entries have garbled or malformed author lists (e.g., [15], [19], [30]), and [37] and [38] are duplicate entries for the same Self-Refine paper; the reference list needs proofreading.
  6. [§3.1, §4] The iteration count is fixed to 4 based on a citation to prior work, but no ablation is reported to justify this choice; varying the iteration count would strengthen the claim that 4 iterations are appropriate.

Circularity Check

2 steps flagged · score 6.0 of 10

Personalization results are partially circular: the win-rate metric is awarded by the same role-playing GPT-3.5-turbo judge used as the loop's revision reward, and ROUGE-L/word overlap compare against the same Stack Overflow history h that conditions generation.

  1. fitted input called prediction [Section 3.3 (Personalization Loop, revision step) and Section 4.1 (Win Rate metric)]
    "The revision step generates an improved personalized explanation 𝑝𝑒𝑖+1 based on the problem 𝑝, the code solution 𝑠, the explanation𝑒 from the faithfulness loop, the user’s profile 𝑢𝑝, the previous personalized explanation 𝑝𝑒𝑖, and the rating𝑟𝑖 on the previous personalized explanation that is generated by a role-playing LLM. ... Win Rate. This metric evaluates personalized explanations generated by different methods using the win rate metric [54]. It compares how often one explanation outperforms another when simulating an individual with a specific user profile."

    The personalization loop uses the role-playing LLM's rating r_i as the feedback signal to revise pe_i, and the loop stops only when that same kind of role-playing judge is satisfied. The Win Rate metric is then computed by the same kind of role-playing simulation ('simulating an individual with a specific user profile') using the same backbone model (GPT-3.5-turbo) that generated and revised the explanations. Thus the evaluation measures how well the method satisfies the very judge it was iteratively optimized against; the win-rate advantage over baselines that were not optimized against this judge is a fitted outcome, not an independent test of real user alignment.

  2. self definitional [Eq. (2) (Problem Definition) and Section 4.1 (Rouge-L / Word Overlap Ratio)]
    "𝑝𝑒∼ 𝑃𝑀(·| ℘⊕𝑝⊕𝑠⊕𝑒⊕ℎ) ... Rouge-L. This metric evaluates personalized explanations by measuring overlap between model predictions and user queries on Stack Overflow [28, 30]. A higher value indicates better alignment with the user’s skill level and background."

    Eq. (2) conditions the personalized explanation pe on the user's historical Stack Overflow inquiries h. The Rouge-L and Word Overlap metrics then score pe by lexical overlap with h, and the paper states that a higher value 'indicates better alignment with the user's skill level and background.' The reference for personalization is therefore the same h that is fed into the generator as input; a model that echoes vocabulary from h will trivially score higher, and no independent profile (or human judgment of the inferred profile) is used. The higher Rouge-L/word overlap for Self-Iteration is thus partly an artifact of conditioning on h, making the 'better personalization' claim self-referential rather than an externally validated outcome.

full rationale

The faithfulness half of the paper is not materially circular: the verification loop uses an external Python executor with public test cases, and Pass@k is computed on private or generated test cases, so there is an external, non-self-referential anchor. However, Pass@k measures downstream solvability rather than fidelity to the particular oracle solution s, which is a construct-validity gap, and the human evaluation claimed in the Abstract and Conclusion is not reported anywhere in the manuscript; these are evidence gaps but not circular reductions. The circularity is concentrated in the personalization half. The win-rate evaluation is awarded by the same role-playing GPT-3.5-turbo judge that serves as the revision reward inside the personalization loop, so the headline win-rate result is a fitted outcome. The Rouge-L and Word Overlap metrics compare outputs to the same user history h that is an input to generation (Eq. 2), making those metrics self-referential. Because the central claim of the paper is two-pillared and the personalization pillar reduces to the loop's own reward/judge, a score of 6 (partial circularity) is appropriate. No self-citation chain or imported-uniqueness issue is present.

Assumptions & free parameters 6 free parameters · 6 assumptions · 2 invented entities

The method introduces no new physical entities, but the personalization claim rests on the invented user-profile representation and on the self-referential role-playing judge. The evaluation additionally relies on hand-set decisions (iteration count, sampling temperatures, number of users and inquiries) and on unvalidated assumptions that lexical overlap and LLM self-ratings measure personalization. The faithfulness half has independent grounding through code execution against public test cases.

free parameters (6)
  • iteration_count = 4
    The number of refinement rounds for both loops was fixed at 4, based on prior work by Gou et al. rather than tuned per problem; the text also says stopping conditions exist, but no adaptive stopping is described.
  • code_generation_sampling = temperature=0.2, top_p=0.1
    Sampling parameters for verification code generation and Pass@k evaluation; no sensitivity analysis is reported.
  • text_generation_sampling = temperature=0.7, top_p=0.8
    Sampling parameters for explanation and personalization generation; no sensitivity analysis is reported.
  • number_of_users = 10
    Personalization experiments use 10 Stack Overflow users, a small and possibly non-representative sample.
  • inquiries_per_user = 5
    Each user's profile is built from their five most recent Python, data structures, and algorithms inquiries; the window is small and arbitrary.
  • samples_per_problem_user_pair = 4
    Each problem and user is sampled 4 times, and metrics are averaged over 40 calculations (10 users times 4 samples); this limits precision.
assumptions (6)
  • domain assumption CodeContests validation and test sets are a valid benchmark for code explanation faithfulness and are free from data leakage.
    Section 4 says only validation/test sets were used to mitigate leakage, but no explicit leakage checks, duplicate detection, or contamination analysis are reported.
  • domain assumption Pass@k with a separate LLM using the explanation measures explanation faithfulness.
    Section 4.1 and Section 5 equate faithfulness with the fraction of generated programs that pass test cases; this measures how useful the explanation is as a problem-solving hint, not necessarily how accurately it reflects the given code's logic.
  • domain assumption Lexical overlap with a user's Stack Overflow history measures personalization quality.
    Section 4.1 defines ROUGE-L and word overlap ratio as alignment with user skill and background; no evidence links lexical similarity to better comprehension or user satisfaction.
  • domain assumption A role-playing LLM judge's rating reflects real user satisfaction with personalization.
    Section 3.3 uses a role-playing LLM to rate the personalized explanation and terminate the loop; the judge is in the same model family as the generator and its ratings are not calibrated against human judgments.
  • domain assumption A user's five recent Stack Overflow inquiries are a sufficient representation of their programming profile.
    Section 3.3 infers six profile aspects from five sampled inquiries per user; this assumes the small sample is representative of the user's background and needs.
  • domain assumption The external Python executor and public test cases provide correct ground truth for verification.
    Section 3.2 relies on executing verification code and comparing outputs to public test cases; this assumes the executor and test set are correct and complete enough to detect errors.
invented entities (2)
  • User programming profile with six aspects (programming languages, skill level, topics of interest, problem-solving approach, experience, other)
    purpose: Drives the personalization loop and provides the basis for the role-playing judge's ratings.
    The profile is inferred by the LLM from only five Stack Overflow inquiries per user and is never validated against the user's true abilities or against human judgments.
  • Role-playing judging LLM persona
    purpose: Rates whether a personalized explanation aligns with the inferred user profile and decides when the personalization loop stops.
    The judge is in the same model family as the generator and evaluates against a profile created by the same model, so its ratings are self-referential.

how reviews work

0 comments
Cite this review

Pith. "Pith review of From Critique to Clarity: A Pathway to Faithful and Personalized Code Explanations with Large Language Models." pith.science (2026). https://pith.science/paper/K3XZLCMJ

@misc{pith2026250114731,
  author       = {Pith},
  title        = {Pith review of: From Critique to Clarity: A Pathway to Faithful and Personalized Code Explanations with Large Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/K3XZLCMJ}},
  note         = {Machine review of arXiv:2501.14731}
}
read the original abstract

In the realm of software development, providing accurate and personalized code explanations is crucial for both technical professionals and business stakeholders. Technical professionals benefit from enhanced understanding and improved problem-solving skills, while business stakeholders gain insights into project alignments and transparency. Despite the potential, generating such explanations is often time-consuming and challenging. This paper presents an innovative approach that leverages the advanced capabilities of large language models (LLMs) to generate faithful and personalized code explanations. Our methodology integrates prompt enhancement, self-correction mechanisms, personalized content customization, and interaction with external tools, facilitated by collaboration among multiple LLM agents. We evaluate our approach using both automatic and human assessments, demonstrating that our method not only produces accurate explanations but also tailors them to individual user preferences. Our findings suggest that this approach significantly improves the quality and relevance of code explanations, offering a valuable tool for developers and stakeholders alike.

Figures

Figures reproduced from arXiv: 2501.14731 by the authors.

Figure 1
Figure 1. The Illustration of Iterative Code Explanation Refinement. The system generates code explanations through two [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Win Rate Model Set Method Rouge-L GPT-3.5 Validation Baseline 0.0230 ± 0.0085 Self-Selection 0.0232 ± 0.0086 Self-Iteration 0.0363 ± 0.0133 Test Baseline 0.0227 ± 0.0083 Self-Selection 0.0230 ± 0.0085 Self-Iteration 0.0361 ± 0.0131 [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. REVA: Supporting LLM-Generated Programming Feedback Validation at Scale Through User Attention-based Adaptation

    cs.HC 2025-07 conditional novelty 6.0 of 10

    REVA uses instructors' highlighting and edits to reorder AI-generated feedback reviews and propagate revisions, and a 12-instructor lab study reports higher feedback precision and recall than a baseline without these ...

  2. An Intelligent Fault Self-Healing Mechanism for Cloud AI Systems via Integration of Large Language Models and Deep Reinforcement Learning

    cs.AI 2025-06 reject novelty 3.0 of 10

    An LLM-plus-deep-RL hybrid is proposed for cloud fault self-healing, claiming 37% faster recovery on unknown faults with weak experimental documentation.

Reference graph

Works this paper leans on

66 extracted references · 28 canonical work pages · cited by 2 Pith papers

  1. [1]

    Wasi Uddin Ahmad, Saikat Chakraborty, Pang Wei He, and Jidong Guo. 2022. Contextualized code completion with neural language models. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing . 675–685

    work page 2022

  2. [2]

    Hassan Alhuzali, Antonios Anastasopoulos, Parisa Kordjamshidi, and Dan Roth

    work page

  3. [3]

    Miltiadis Allamanis, Earl T Barr, Christian Bird, and Charles Sutton. 2018. A Survey of Machine Learning for Big Code and Naturalness. ACM Computing Surveys (CSUR) 51, 4 (2018), 1–37

    work page 2018

  4. [4]

    Pepa Atanasova, Grégoire Cardon, Thomas Demeester, and Isabelle Augenstein

    work page

  5. [5]

    Paheli Bhattacharya, Manojit Chakraborty, Kartheek N S N Palepu, Vikas Pandey, Ishan Dindorkar, Rakesh Rajpurohit, and Rishabh Gupta. 2023. Exploring Large Language Models for Code Explanation. ArXiv abs/2310.16673 (2023). https: //api.semanticscholar.org/CorpusID:264451660

    arXiv 2023

  6. [6]

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems 33 (2020), 1877–1901

    2020

  7. [7]

    Peter Brusilovsky, Arun-Balajiee Lekshmi-Narayanan, Priti Oli, Jeevan Chapa- gain, Mohammad Hassany, Rabin Banjade, and Vasile Rus. 2023. Explaining code examples in introductory programming courses: Llm vs humans. arXiv preprint arXiv:2403.05538 (2023)

    arXiv 2023

  8. [8]

    Eason Chen, Ray Huang, Han-Shin Chen, Yuen-Hsien Tseng, and Liang-Yi Li

Show all 66 references
  1. [9]

    Jiangjie Chen, Xintao Wang, Rui Xu, Siyu Yuan, Yikai Zhang, Wei Shi, Jian Xie, Shuang Li, Ruihan Yang, Tinghui Zhu, et al. 2024. From Persona to Personaliza- tion: A Survey on Role-Playing Language Agents.arXiv preprint arXiv:2404.18231 (2024)

    2024 arXiv

  2. [10]

    Pei Chen, Boran Han, and Shuai Zhang. 2024. CoMM: Collaborative Multi-Agent, Multi-Reasoning-Path Prompting for Complex Problem Solving. In Proceedings of the 2024 Conference of the North American Chapter of the Association for Com- putational Linguistics. https://brickee.gith...

    2024

  3. [11]

    Zhihao Chen and Hongyu Ji. 2021. Evaluating the Faithfulness of Importance Measures in NLP by Recursively Masking Allegedly Important Tokens and Retraining. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. 2669–2675

    2021

  4. [12]

    Liu Dan, Yang Shi, Yu Zhang, and Wei Gao. 2021. Improving Faithfulness of Attention-based Explanations with Task-Specific Information for Text Classifica- tion. In Proceedings of the 59th Annual Meeting of the Association for Computa- tional Linguistics and the 11th Internatio...

    2021

  5. [13]

    Zhangyin Feng, Daya Guo, Duyu Tang, Nan Duan, Xiaocheng Feng, Ming Gong, Linjun Shou, Bing Qin, Ting Liu, Daxin Jiang, and Ming Zhou. 2020. CodeBERT: A Pre-Trained Model for Programming and Natural Languages. arXiv preprint arXiv:2002.08155 (2020)

    2020 arXiv

  6. [14]

    Zhibin Gou, Zhihong Shao, Yeyun Gong, Yelong Shen, Yujiu Yang, Nan Duan, and Weizhu Chen. 2023. Critic: Large language models can self-correct with tool-interactive critiquing. arXiv preprint arXiv:2305.11738 (2023)

    2023 arXiv

  7. [15]

    Daya Guo, Shuo Ren, Shuai Lu, Zhangyin Feng, Duyu Tang, Nan Duan, and Ming Zhou. 2022. UnixCoder: Unified Cross-Modal Pre-Training for Code Rep- resentation. arXiv preprint arXiv:2203.01679 (2022)

    2022 arXiv

  8. [16]

    Peter Hase and Mohit Bansal. 2021. Evaluating explainable AI: Which algorithmic explanations help users predict model behavior? Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics (ACL) (2021), 5544–5553

    2021

  9. [17]

    Bairu Hou, Joe O’connor, Jacob Andreas, Shiyu Chang, and Yang Zhang. 2023. Promptboosting: Black-box text classification with ten forward passes. In Inter- national Conference on Machine Learning . PMLR, 13309–13324

    2023

  10. [18]

    Xing Hu, Ge Li, Xin Xia, David Lo, and Zhi Jin. 2018. Deep code comment genera- tion. In 2018 IEEE/ACM 26th International Conference on Program Comprehension (ICPC). IEEE, 200–20010

    2018

  11. [19]

    Hamel Husain, Ho-Hsiang Siddiqui, Huy Feng, Usama Chowdhury, Eric Ham- mond, Boris Tran, Vinod Mangal, Dima Kang, and Ankur Taly. 2019. CodeSearch- Net Challenge: Evaluating the State of Semantic Code Search. In arXiv preprint arXiv:1909.09436

    2019 arXiv

  12. [20]

    Hanlei Jin, Yang Zhang, Dan Meng, Jun Wang, and Jinghua Tan. 2024. A com- prehensive survey on process-oriented automatic text summarization with ex- ploration of llm-based methods. arXiv preprint arXiv:2403.02901 (2024)

    2024

  13. [21]

    Alexander LeClair, Collin McMillan, Mustafa Kocakulak, Shan Jiang, Jingzhou Lou, and Lingling Liu. 2019. Neural Models for Code Summarization: A Review and Evaluation. In Proceedings of the 34th IEEE/ACM International Conference on Automated Software Engineering (ASE). 151–162

    2019

  14. [22]

    Juho Leinonen, Paul Denny, Stephen MacNeil, Sami Sarsa, Seth Bernstein, Joanne Kim, Andrew Tran, and Arto Hellas. 2023. Comparing code explanations created by students and large language models. In Proceedings of the 2023 Conference on Innovation and Technology in Computer Sci...

    2023

  15. [23]

    Aitor Lewkowycz, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Ramasesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo, et al. 2022. Solving quantitative reasoning problems with language models. Advances in Neural Information Processing Syste...

    2022

  16. [24]

    Cheng Li, Ziang Leng, Chenxi Yan, Junyi Shen, Hao Wang, Weishi MI, Yaying Fei, Xiaoyang Feng, Song Yan, HaoSheng Wang, Linkang Zhan, Yaokai Jia, Pingyu Wu, and Haozhen Sun. 2023. ChatHaruhi: Reviving Anime Character in Reality via Large Language Model. arXiv:2308.09597 [cs.CL]

    2023 arXiv

  17. [25]

    Jierui Li, Szymon Tworkowski, Yingying Wu, and Raymond Mooney. 2023. Ex- plaining competitive-level programming solutions using llms. arXiv preprint arXiv:2307.05337 (2023)

    2023 arXiv

  18. [26]

    Yujia Li, David Choi, Junyoung Chung, Nate Kushman, Julian Schrittwieser, Rémi Leblond, Tom Eccles, James Keeling, Felix Gimeno, Agustin Dal Lago, et al. 2022. Competition-level code generation with alphacode. Science 378, 6624 (2022), 1092–1097

    2022

  19. [27]

    Yuan Li, Yanlin Wang, and Hongyu Liu. 2018. Automatic code summarization via deep learning-based attention mechanism. InProceedings of the 2018 International Joint Conference on Neural Networks (IJCNN) . 1–8

    2018

  20. [28]

    Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out. 74–81

    2004

  21. [29]

    Xi Victoria Lin, Diane Belgrave, and Shubhomoy Dasgupta. 2022. Program synthesis with large language models. arXiv preprint arXiv:2203.13474 (2022)

    2022 arXiv

  22. [30]

    Zhen Lin, Shubhendu Trivedi, and Jimeng Sun. 2023. Generating with confidence: Uncertainty quantification for black-box large language models. arXiv preprint arXiv:2305.19187 (2023)

    2023 arXiv

  23. [31]

    Natasha Linnell, Nabeel Gillani, Ece Kamar, and Eric Horvitz. 2019. Generating Automated Explanations for Numerical Data Insights. Proceedings of the AAAI Conference on Artificial Intelligence 33 (2019), 9656–9661

    2019

  24. [32]

    Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. 2023. Pre-train, prompt, and predict: A systematic survey of From Critique to Clarity: A Pathway to Faithful and Personalized Code Explanations with Large Language Models prompting methods...

    2023 doi

  25. [33]

    Stephen MacNeil, Andrew Tran, Arto Hellas, Joanne Kim, Sami Sarsa, Paul Denny, Seth Bernstein, and Juho Leinonen. 2023. Experiences from using code explanations generated by large language models in a web software development e-book. In Proceedings of the 54th ACM Technical Sy...

    2023

  26. [34]

    Stephen MacNeil, Andrew Tran, Dan Mogil, Seth Bernstein, Erin Ross, and Ziheng Huang. [n. d.]. Generating Diverse Code Explanations using the GPT-3 Large Language Model. ICER ’22: Proceedings of the 2022 ACM Conference on International Computing Education ([n. d.]). https://do...

    2022 doi

  27. [35]

    Stephen MacNeil, Andrew Tran, Dan Mogil, Seth Bernstein, Erin Ross, and Ziheng Huang. 2022. Generating diverse code explanations using the gpt-3 large language model. In Proceedings of the 2022 ACM Conference on International Computing Education Research-Volume 2. 37–39

    2022

  28. [36]

    Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, et al

  29. [37]

    Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, Shashank Gupta, Bodhisattwa Prasad Majumder, Katherine Hermann, Sean Welleck, Amir Yazdanbakhsh, and Peter Clark. 2023. Self-Refine: It...

    2023

  30. [38]

    Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, Shashank Gupta, Bodhisattwa Prasad Majumder, Katherine Hermann, Sean Welleck, Amir Yazdanbakhsh, and Peter Clark. 2023. Self-Refine: It...

    2023 arXiv

  31. [39]

    Andrea Madotto, Zhaojiang Lin, Chien-Sheng Wu, and Pascale Fung. 2019. Personalizing Dialogue Agents via Meta-Learning. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics . 5454–5459. https://doi.org/10.18653/v1/P19-1542

    2019 doi

  32. [40]

    Vera Liao, Justin Cheng, Justin D

    Stefania Miceli, Q. Vera Liao, Justin Cheng, Justin D. Weisz, and Michael Muller

  33. [41]

    Jiří Milička, Anna Marklová, Klára VanSlambrouck, Eva Pospíšilová, Jana Šimsová, Samuel Harvan, and Ondřej Drobil. 2024. Large language models are able to downplay their cognitive abilities to fit the persona they simulate. Plos one 19, 3 (2024), e0298522

    2024

  34. [42]

    Ansong Ni, Srini Iyer, Dragomir Radev, Veselin Stoyanov, Wen-tau Yih, Sida Wang, and Xi Victoria Lin. 2023. Lever: Learning to verify language-to-code generation with execution. In International Conference on Machine Learning . PMLR, 26106–26128

    2023

  35. [43]

    Priti Oli, Rabin Banjade, Jeevan Chapagain, and Vasile Rus. 2023. The Behavior of Large Language Models When Prompted to Generate Code Explanations. arXiv preprint arXiv:2311.01490 (2023)

    2023 arXiv

  36. [44]

    2023 OpenAI. 2023. Introducing ChatGPT. https://openai.com/index/chatgpt/

    2023

  37. [45]

    In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems

    Studying the Impact of Explanation Faithfulness on the Performance of Interactive Machine Learning Systems. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems . 1–12

    2021

  38. [46]

    Debjit Paul, Mete Ismayilzada, Maxime Peyrard, Beatriz Borges, Antoine Bosselut, Robert West, and Boi Faltings. 2023. Refiner: Reasoning feedback on intermediate representations. arXiv preprint arXiv:2304.01904 (2023)

    2023 arXiv

  39. [47]

    Silviu Pitis, Michael R Zhang, Andrew Wang, and Jimmy Ba. 2023. Boosted prompt ensembles for large language models. arXiv preprint arXiv:2304.05970 (2023)

    2023 arXiv

  40. [48]

    Jack W Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, et al. 2021. Scaling language models: Methods, analysis & insights from training gopher. arXiv preprint arXiv:2112.11446 (2021)

    2021 arXiv

  41. [49]

    Laria Reynolds and Kyle McDonell. 2021. Prompt Programming for Large Lan- guage Models: Beyond the Few-Shot Paradigm. In CHI Conference on Human Factors in Computing Systems . https://doi.org/10.1145/3411764.3445647

    2021

  42. [50]

    Bhargavi Paranjape, Scott Lundberg, Sameer Singh, Hannaneh Hajishirzi, Luke Zettlemoyer, and Marco Tulio Ribeiro. 2023. Art: Automatic multi-step reasoning and tool-use for large language models. arXiv preprint arXiv:2303.09014 (2023)

    2023 arXiv

  43. [51]

    Alejandro Sarsa. 2022. Automatic code explanation with large language models in CS education. In Proceedings of the 27th ACM Conference on on Innovation and Technology in Computer Science Education Vol. 2 . 92–98

    2022

  44. [52]

    Yunfan Shao, Linyang Li, Junqi Dai, and Xipeng Qiu. 2023. Character-LLM: A Trainable Agent for Role-Playing. arXiv:2310.10158 [cs.CL]

    2023 arXiv

  45. [53]

    Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. 2023. Self-Consistency Improves Chain of Thought Reasoning in Language Models. arXiv:2203.11171 [cs.CL]

    2023 arXiv

  46. [54]

    Zekun Moore Wang, Zhongyuan Peng, Haoran Que, Jiaheng Liu, Wangchunshu Zhou, Yuhan Wu, Hongcheng Guo, Ruitong Gan, Zehao Ni, Man Zhang, et al

  47. [55]

    Tal Ridnik, Dedy Kredo, and Itamar Friedman. 2024. Code Generation with AlphaCodium: From Prompt Engineering to Flow Engineering. arXiv preprint arXiv:2401.08500 (2024)

    2024 arXiv

  48. [56]

    Yixuan Weng, Minjun Zhu, Fei Xia, Bin Li, Shizhu He, Shengping Liu, Bin Sun, Kang Liu, and Jun Zhao. 2022. Large language models are better reasoners with self-verification. arXiv preprint arXiv:2212.09561 (2022)

    2022 arXiv

  49. [57]

    Qingyun Wu, Gagan Bansal, Jieyu Zhang, Yiran Wu, Shaokun Zhang, Erkang Zhu, Beibin Li, Li Jiang, Xiaoyun Zhang, and Chi Wang. 2023. Autogen: Enabling next-gen llm applications via multi-agent conversation framework.arXiv preprint arXiv:2308.08155 (2023)

    2023 arXiv

  50. [58]

    Siyu Yuan, Kaitao Song, Jiangjie Chen, Xu Tan, Yongliang Shen, Ren Kan, Dong- sheng Li, and Deqing Yang. 2024. Easytool: Enhancing llm-based agents with concise tool instruction. arXiv preprint arXiv:2401.06201 (2024)

    2024 arXiv

  51. [59]

    Raluca Zamfirescu and Bjoern Hartmann. 2023. Iterative Disambiguation: To- wards LLM-Supported Programming and System Design. In ICML Workshop on Interpretable Machine Learning . https://people.eecs.berkeley.edu/~bjoern/ papers/zamfirescu-iterdis-icmlws2023.pdf

    2023

  52. [60]

    arXiv preprint arXiv:2310.00746 (2023)

    Rolellm: Benchmarking, eliciting, and enhancing role-playing abilities of large language models. arXiv preprint arXiv:2310.00746 (2023)

    2023 arXiv

  53. [61]

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022. Chain-of-thought prompting elicits reason- ing in large language models. Advances in neural information processing systems 35 (2022), 24824–24837

    2022

  54. [66]

    Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, et al. 2023. A survey of large language models. arXiv preprint arXiv:2303.18223 (2023)

    2023 arXiv

  55. [2020]

    Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) (2020), 4015–4028

    Diagnostic dataset construction to evaluate NLP models for critical infor- mation extraction in the biomedical domain. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP) (2020), 4015–4028

    2020

  56. [2021]

    In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing

    A Model-Agnostic Data-Free Approach for Extracting Fair Representations. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. 2894–2906

    2021

  57. [2023]

    In International Conference on Artificial Intelligence in Education

    GPTutor: a ChatGPT-powered programming tool for code explanation. In International Conference on Artificial Intelligence in Education. Springer, 321–327

  58. [2024]

    Advances in Neural Information Processing Systems 36 (2024)

    Self-refine: Iterative refinement with self-feedback. Advances in Neural Information Processing Systems 36 (2024)

    2024

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.