{"id":"ce2533ab-3465-4b23-b37e-3930be5ff0b1","arxiv_id":"2501.14731","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"An iterative two-loop LLM pipeline (a faithfulness loop with execution-based checks and a personalization loop with a role-playing judge) produces code explanations that score higher on automatic metrics than simpler baselines.","lead":"Researchers built an AI system that first writes a technically accurate explanation of a competitive programming solution, checks it by generating and running code from the explanation, and then rewrites it to match a user's coding background. They report that this two-stage refinement improves both correctness and personalization compared to simpler prompting methods on a Codeforces-based benchmark.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Pass@k measures downstream solvability, not whether an explanation is faithful to the oracle solution; the faithfulness claim is therefore unsupported without a direct consistency check.","rationale":"The reader's weakest assumption concerned the personalization metrics (ROUGE-L and role-playing judge). I examined the faithfulness pillar and found a more fundamental construct-validity issue: Pass@k measures whether an explanation helps solve the problem, not whether it is faithful to the oracle solution s. In §3.2 the verification loop checks whether a code solution generated from p and e_i passes public tests; it never checks consistency between e_i and the given s. Since §2 defines e as sampled for the pair (p,s), an explanation that is a generic problem-solving hint would score well on Pass@k without explaining the specific code. No direct faithfulness evaluation appears in §5. This is not an internal contradiction, but it means the paper's 'more faithful' claim rests on a proxy that may measure the wrong construct. The personalization issue remains valid as well, but the faithfulness concern is prior: if Pass@k is not a faithfulness measure, the first half of the central claim fails even before personalization is considered. The proposed concrete test (swapping s2 for s1 in the Pass@k protocol) settles whether explanations are solution-specific; if not, the faithfulness claim is unsupported. Thus I would keep the reader's CONDITIONAL verdict, adding this test as a condition.","tokens_in":11092,"tokens_out":6366,"duration_ms":65812,"concrete_test":"For 20 CodeContests test problems that have at least two oracle solutions, take each Self-Iteration explanation e (generated for the shortest solution s1) and rerun the Pass@k protocol with the problem p and e, but substitute a different oracle solution s2 as the reference code in the prompt. If Pass@k does not drop by a meaningful margin compared to using s1, then e is not explaining s1 specifically; it is a generic problem-solving text, and Pass@k is not a faithfulness measure.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central faithfulness claim (Section 5) is supported only by Pass@k (Table 1): an explanation is deemed more faithful if a fresh LLM, given the problem and the explanation, produces a solution that passes public tests. This does not measure whether the explanation describes the logic of the particular oracle solution s. In the faithfulness loop (§3.2), the verification step checks whether a code solution generated from p and e_i passes tests; the loop never checks whether any statement in e_i is entailed by s. Consequently, an explanation could be a generic problem-solving hint (e.g., 'use memoization') that raises Pass@k without explaining the given code. The paper's problem definition (Eq. 1) explicitly ties e to the pair (p,s), so Pass@k is at best a necessary condition, not a sufficient one. No human or other direct faithfulness evaluation is reported, despite the abstract and conclusion claiming human assessment. This leaves the 'more faithful' portion of the central claim unvalidated.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Self-Iteration, a two-loop iterative refinement method for generating code explanations for competitive programming problems. A faithfulness loop generates an explanation of a problem-solution pair and verifies it indirectly by asking an LLM to produce a code solution from the explanation and executing that solution on public test cases; a personalization loop infers a user profile from Stack Overflow inquiry history and uses a role-playing LLM judge to rate and refine the explanation. Experiments on the CodeContests validation and test sets compare Self-Iteration against a greedy baseline and a Self-Selection baseline, reporting Pass@k, win rate, ROUGE-L, and word overlap ratio. The paper claims that the method produces more faithful and more personalized code explanations and states in the abstract and conclusion that both automatic and human evaluations were performed, although no human evaluation is reported in the text.","tokens_in":11287,"tokens_out":4630,"duration_ms":45776,"significance":"If the results held, this would be a useful contribution to code explanation generation, particularly for the new task of personalized code explanations. The architecture is clearly described, and the faithfulness loop has a genuinely external anchor through the Python executor and public test cases. The paper also takes reasonable steps to mitigate data leakage by using only the CodeContests validation and test splits, and it reports error bars on most automatic metrics. However, the two central claims rest on metrics whose validity is not established: Pass@k is not shown to measure faithfulness to the oracle solution, and the personalization metrics are lexical overlap with past Stack Overflow inquiries plus a self-referential role-playing judge. The claimed human evaluation is absent. The contribution is therefore conditional on additional validation.","major_comments":[{"comment":"Pass@k measures whether a fresh LLM, given the problem and the candidate explanation, can generate a code solution that passes public tests; it does not measure whether the explanation is faithful to the particular oracle solution s. Because Eq. (1) defines the explanation as conditioned on the pair (p, s), a generic problem-solving hint could raise Pass@k without describing the logic of s at all. The verification loop in §3.2 never checks whether any statement in e_i is entailed by s, so the reported Pass@k gains do not substantiate the claim that Self-Iteration produces more faithful explanations. A direct evaluation against s (e.g., human ratings of explanation-solution consistency or an automatic entailment-style check) is needed.","section":"§5, Table 1; §3.2"},{"comment":"The abstract states that the authors evaluate with 'both automatic and human assessments,' and the conclusion repeats that the method is validated 'no matter evaluated by the automatic evaluation or the human evaluation.' No human evaluation protocol, results, or analysis appears anywhere in Sections 4 or 5. This is a missing support for a stated contribution. The authors should either add the human study or remove the claim from the abstract and conclusion.","section":"Abstract and §6"},{"comment":"The personalization evidence rests on three proxy metrics: ROUGE-L, word overlap ratio, and win rate. The first two measure lexical overlap between the generated explanation and the user's five most recent Stack Overflow inquiries, and the win rate is assigned by a role-playing LLM judge whose persona is based on a profile inferred by the same GPT-3.5-turbo model that generates the explanations. The paper provides no evidence that lexical similarity to past questions or self-ratings by the generating model track what a user actually needs. Without a human evaluation or an independent external validation of the judge, the personalization claim in the paper's conclusion is not supported.","section":"§3.3, §4.1, Tables 2-3, Figure 2"},{"comment":"The final output is defined in Eq. (3) as o = e ⊕ p_e, combining the faithfulness-loop explanation and the personalization-loop explanation. However, the reported evaluations treat the two components separately: Table 1 evaluates e through Pass@k, while Tables 2-3 and Figure 2 evaluate p_e through lexical and judge-based metrics. The actual combined output o is never evaluated. Thus the paper does not demonstrate that the method simultaneously produces a final explanation that is both faithful and personalized, despite the stated goal of balancing the two objectives.","section":"§3, Eq. (3), §5"}],"minor_comments":[{"comment":"All experiments use only GPT-3.5-turbo as the backbone and judge; the paper does not test whether the observed improvements hold for other LLMs, so the generality of the method remains unclear.","section":"§4, Backbone Model"},{"comment":"The two 'Commercial Product' baselines are unnamed and undescribed, which prevents a reader from understanding the comparison or reproducing it.","section":"Table 1"},{"comment":"The Pass@k description says the solutions are checked 'against ground-truth outputs, derived from private or generated test cases,' while §3.2 refers to 'public test cases.' The paper should clarify which test cases are actually used and whether private tests exist.","section":"§4.1"},{"comment":"The reported standard deviations are said to come from '40 calculations (10 chosen users * 4 samples per user),' but it is not stated whether the averaging is over users, over problems, or over both; the aggregation procedure should be specified.","section":"§4.1"},{"comment":"Several reference entries have garbled or malformed author lists (e.g., [15], [19], [30]), and [37] and [38] are duplicate entries for the same Self-Refine paper; the reference list needs proofreading.","section":"References"},{"comment":"The iteration count is fixed to 4 based on a citation to prior work, but no ablation is reported to justify this choice; varying the iteration count would strengthen the claim that 4 iterations are appropriate.","section":"§3.1, §4"}],"recommendation":"major_revision","confidential_remarks":"The gap between the claimed human evaluation and the reported experiments is substantial and may warrant a direct query to the authors about whether such an evaluation was conducted. The unnamed commercial baselines and the absence of code/data availability statements also complicate independent verification. The paper's central claims are plausible but currently rest on proxy metrics; a revision that adds a direct faithfulness evaluation and a human personalization study would make the contribution much stronger."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"This paper builds a two-loop LLM pipeline for code explanations — one loop checks faithfulness by executing code written from the explanation, the other personalizes using Stack Overflow history. The execution anchor is a real strength, but the faithfulness metric measures downstream solvability, not fidelity to the given solution, and the personalization evidence is proxy-based with no human data despite the abstract claiming it.\n\nWhat's new and good: The two independent loops are a reasonable architecture. The faithfulness loop's use of an external Python executor and public test cases is a solid way to ground iterative refinement. The paper reports error bars on Pass@k over validation and test splits, which is honest. The idea of profiling users from Stack Overflow histories rather than demographic tags is a nice touch.\n\nSoft spots: The \"faithfulness\" claim is overreach. Pass@k tells you whether an explanation helps another LLM pass tests; it does not check whether the explanation describes the logic of the specific oracle solution s. A generic \"use memoization\" hint could improve Pass@k without being faithful to the code at hand. The problem definition (Eq. 1) conditions on (p,s), so the metric should at least be supplemented with a direct consistency check. No human evaluation is reported despite the abstract and conclusion explicitly claiming it. The personalization metrics — ROUGE-L, word overlap with five past Stack Overflow questions, and a role-playing LLM judge from the same model family as the generator — are weak proxies. The two commercial baselines are unnamed, which makes the comparison in Table 1 hard to interpret. No code or prompts are released, so the results are not reproducible as-is.\n\nThe core ideas are sensible and the execution-anchored loop is worth building on, but the paper currently overstates what the evaluation supports. It deserves peer review because the problem is real and the architecture is a reasonable contribution, but it needs heavy revision: temper the claims, add a human study or rename the metrics, name the baselines, and release artifacts.\n\nWho's it for: People working on LLM-based code explanation, evaluation methodology for generated explanations, and tool-integrated self-correction.\n\nRecommendation: send to peer review, with a clear request for major revision.","headline":"Nice two-loop pipeline with a real execution anchor, but the faithfulness metric measures solvability rather than fidelity to the given solution, and the personalization evidence relies on proxy metrics with no human validation.","tokens_in":11834,"tokens_out":2485,"would_cite":false,"duration_ms":23545,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Two separate iterative critique loops make LLM code explanations both more faithful to the code and more tailored to the reader.","keywords":["code explanation generation","large language models","personalization","faithfulness","iterative refinement","self-correction","role-playing judge","CodeContests"],"falsifier":"A user study in which the same ten (or more) Stack Overflow users, or a comparable panel, rate the personalized explanations blind: if users do not prefer Self-Iteration's personalized explanations over the baseline, or if their preferences do not correlate with higher ROUGE-L, word overlap, and win-rate scores, then the personalization claim fails.","tokens_in":10878,"feed_emoji":"🔁","tokens_out":6864,"duration_ms":60545,"temperature":0.7,"pith_summary":"This paper tries to establish that code explanations from large language models can be made both faithful and personalized by iterating through two separate refinement loops: one that checks an explanation by using it to regenerate working code and running that code against tests, and one that rewrites the explanation to match a user profile inferred from their Stack Overflow history. The authors argue that a single pass, or even self-selection among several samples, is not enough; repeated critique-and-revise cycles with external verification are what push quality up. They validate the claim on competitive-programming problems from CodeContests, where the iterative method improves the rate at which an explanation lets another LLM solve the problem and raises personalization metrics such as ROUGE-L and word overlap with users' past questions. If true, this gives developers and stakeholders a practical route to explanations that are both technically accurate and tailored to the reader's background.","feed_headline":"Iterative critique loops make code explanations more faithful and personalized","feed_subtitle":"Self-Iteration beats greedy and self-selection baselines on CodeContests, raising Pass@1 and personalization overlap scores.","key_machinery":"The central mechanism is the Self-Iteration refinement pipeline: two independent three-stage loops, each with reflection (summarizing the problem or the user profile), iterative explanation (chain-of-thought initialization plus revision conditioned on prior output and critique), and verification-and-analysis (external execution or a role-playing judge). The faithfulness loop's verification step is the load-bearing device: the explanation is judged by whether it lets another LLM generate code that passes the test cases, turning faithfulness into an observable, executable criterion. The personalization loop's load-bearing device is the role-playing judge that rates whether an explanation matches the inferred user profile and supplies revision suggestions until the rating is satisfactory.","core_discovery":"The central claim, stated on the paper's own terms, is that faithful and personalized code explanation is achievable as a generation task by separating the two objectives into independent iterative loops. The faithfulness loop generates an explanation, uses it to prompt a fresh code solution, executes that solution against public tests, and feeds any failure back as a revision suggestion; the personalization loop infers a six-aspect programming profile from a user's recent Python, data-structure, and algorithm questions on Stack Overflow, drafts a tailored explanation, and has a role-playing judge LLM score it against the profile, repeating until satisfied. The final output concatenates the faithful explanation and the personalized explanation. On CodeContests, with GPT-3.5-turbo as backbone, the method reports higher Pass@k than greedy self-consistency and self-selection baselines, and higher ROUGE-L, word overlap, and win-rate scores for personalization, leading the paper to conclude that iterative refinement with critique-response loops improves both qualities.","pith_inferences":["A testable extension is to swap the role-playing judge for a human-in-the-loop rating in the personalization loop; if human ratings disagree with the LLM judge, the loop would need a different reward signal.","The faithfulness loop implicitly defines 'faithful' as 'sufficient for re-implementation.' That is a particular, executable notion of faithfulness; other notions, such as line-by-line correspondence to the oracle solution, might rank the same explanations differently.","Since user profiles are inferred from only five recent Stack Overflow inquiries, the approach could be sensitive to the recency and topic mix of those questions; profiles built from longer or multi-source history would test how robust the personalization loop is.","The two-loop separation suggests a modular product design: a general faithfulness service plus a pluggable personalization layer keyed to a user model, which could be reused beyond competitive programming."],"forward_implications":["Explanations produced by Self-Iteration should transfer to a fresh LLM: a downstream model given only the explanation solves CodeContests problems more often than with baseline explanations, so the explanations carry real problem-solving content.","The method can serve as a data-annotation pipeline: iteratively refined explanations of the CodeContests training set could produce large numbers of 'silver' explanations for fine-tuning reasoning models, a direction the paper explicitly points to.","Because the two loops are independent, practitioners can apply faithfulness refinement without personalization or vice versa, and can tune each loop's stopping criteria separately.","The reported gains come from iteration depth (four rounds) rather than simply sampling more candidates, since Self-Iteration outperforms Self-Selection with the same number of samples."],"supporting_citations":[{"why":"Supplies the CodeContests dataset of competitive programming problems, oracle solutions, and test cases used for all faithfulness and personalization experiments.","marker":"[26]"},{"why":"Defines the task of explaining competitive-level programming solutions with LLMs and serves as the primary baseline the method extends.","marker":"[25]"},{"why":"Provides the chain-of-thought prompting strategy used to initialize both refinement loops.","marker":"[55]"},{"why":"Supplies the self-refine paradigm of iterative generation, feedback, and revision that the two loops operationalize.","marker":"[37]"},{"why":"Motivates tool-interactive critiquing and the choice of four refinement iterations.","marker":"[14]"},{"why":"Self-consistency is used to strengthen the greedy baseline that Self-Iteration is compared against.","marker":"[53]"},{"why":"Provides the win-rate metric and role-playing methodology used by the personalization judge.","marker":"[54]"},{"why":"Defines ROUGE-L, one of the automatic personalization metrics measuring overlap with users' Stack Overflow inquiries.","marker":"[28]"},{"why":"Supplies the anonymized Stack Overflow dump from which the ten user inquiry profiles are built.","marker":"[48]"},{"why":"Reports that LLMs downplay their displayed abilities to fit a simulated persona, which the paper cites as the reason for keeping the personalization loop separate from the faithfulness loop.","marker":"[41]"}],"fun_headline_variants":["Separate critique loops make code explanations faithful and personalized","Two iterative LLM loops tailor and verify code explanations","Dual-loop LLM approach yields faithful, personalized code explanations","Iterative self-correction personalizes code explanations for users","Critique loops in LLMs improve code explanation fidelity and fit"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the personalization metrics—word and phrase overlap with a user's five most recent Stack Overflow questions and ratings from a role-playing LLM judge—actually measure whether an explanation is useful to that user, because the paper reports no human evaluation of personalization.","fun_headline_variants_meta":{"raw":{"variants":["Separate critique loops make code explanations faithful and personalized","Two iterative LLM loops tailor and verify code explanations","Dual-loop LLM approach yields faithful, personalized code explanations","Iterative self-correction personalizes code explanations for users","Critique loops in LLMs improve code explanation fidelity and fit"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000842,"raw_usage":{"total_tokens":3647,"prompt_tokens":901,"completion_tokens":2746,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":517,"completion_tokens_details":{"reasoning_tokens":2665}},"tokens_in":517,"tokens_out":2746,"duration_ms":18937,"temperature":1.0,"reasoning_tokens":2665,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T20:16:22.339641+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A user study in which the same ten (or more) Stack Overflow users, or a comparable panel, rate the personalized explanations blind: if users do not prefer Self-Iteration's personalized explanations over the baseline, or if their preferences do not correlate with higher ROUGE-L, word overlap, and win-rate scores, then the personalization claim fails.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the self-refine paradigm of iterative generation, feedback, and revision that the two loops operationalize."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the win-rate metric and role-playing methodology used by the personalization judge."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Reports that LLMs downplay their displayed abilities to fit a simulated persona, which the paper cites as the reason for keeping the personalization loop separate from the faithfulness loop."}],"review_version":1}