{"id":"471e6b93-1814-4b0a-9991-5c04f2a86fd4","arxiv_id":"2509.03269","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Expert rubric ratings overstate student-perceived helpfulness of AI-generated hints in 26.6% of cases, with mismatch reasons grouped into five categories and preliminary fixes proposed.","lead":"This study compares students' and experts' ratings of AI-generated programming hints in a Python course and finds that experts' rubric-based ratings often call hints high-quality when students found them unhelpful. It categorizes the reasons for these mismatches and proposes an extended rubric and prompt changes, though the validation of these fixes is preliminary.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The reported chi-square test in RQ1 is under-specified and may not test the paired asymmetry; with 139 ratings nested in 34 students, the 37-vs-11 imbalance needs a cluster-aware McNemar test to support the claim.","rationale":"I read the paper as an exploratory, in-progress study whose central descriptive finding is the mismatch between student and expert ratings, particularly the 37 expert-high/student-unhelpful cases. The reader's verdict of CONDITIONAL is appropriate. The most load-bearing concern I find is not the inter-rater reliability of the two experts, which the reader emphasized, but the statistical inference attached to Table 1. The paper uses a chi-square test to claim that students' and experts' ratings differ significantly, yet it does not specify whether this is McNemar's test for paired marginal asymmetry or a standard chi-square test of independence. These answer different questions, and the nested structure of the data (multiple hints per student) is not addressed. A correct re-analysis could still confirm the asymmetry, and the descriptive pattern is plausible, so I do not recommend changing the overall verdict. However, the authors should be required to report the exact test, the test statistic, and a cluster-robust variant before the claim is accepted as statistically grounded. This concern is concrete and checkable with the existing data, unlike a request for new annotation studies, which is why I focus on it here.","tokens_in":7930,"tokens_out":8581,"duration_ms":85900,"concrete_test":"Use the public repository's data to reconstruct the paired ratings for all 139 hints with student IDs. Re-run (1) McNemar's exact test on the discordant pairs (37 vs 11) and (2) a cluster-robust version, e.g., bootstrap resampling students 10,000 times and recomputing the McNemar statistic, or fit a mixed-effects logistic regression with a random intercept for student predicting student 'unhelpful' among expert-high hints. Report the test statistic, confidence interval, and p-value. If the cluster-robust p-value exceeds 0.05, the claimed asymmetry is not statistically robust and RQ1's conclusion should be downgraded.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central quantitative evidence for 'expert evaluations may be overly optimistic' is the significant chi-square test reported in RQ1 (Section 4, Table 1). As reported, this test is under-specified and may not be testing the claim. The data are paired: student and expert rate the same hint, and the meaningful null for 'experts are more optimistic' is that the two off-diagonal counts (37 and 11) are equal. That requires McNemar's test or an exact binomial test on discordant pairs, not a standard chi-square test of independence, which would instead test whether the two ratings are associated. In addition, the 139 rated hints are nested in 34 students (about 4 hints per student), so ratings from the same student are not independent; this can inflate significance in either test. The paper cites Cochran (1952) without reporting the statistic, the degrees of freedom, or any treatment of clustering, so the reader cannot verify that p <= 0.001 actually refers to the asymmetry. If the correct cluster-aware McNemar or mixed-effects test does not preserve significance, the headline mismatch count (37 unhelpful-student/expert-high hints) becomes a descriptive observation without statistical support, and the conclusion that rubric-based expert evaluation is 'overly optimistic' is not established by RQ1. The inter-rater reliability concern raised by the reader is real but secondary: it affects generalizability, whereas this statistical issue directly affects the main numerical finding.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper reports on a classroom deployment of GPT-4-generated programming hints in a Python data science course. Of 74 enrolled students, 34 requested and rated hints; 139 rated hints are compared with ratings given by two experts who jointly applied a four-attribute rubric. The authors report 65.5% agreement between students and experts, with 37 cases in which experts rated a hint high-quality while the student rated it unhelpful, and 11 reverse mismatches. A chi-square test is reported as significant at p≤0.001, and the authors interpret the imbalance as evidence that rubric-based expert evaluations may be overly optimistic. They then classify 26 of the 37 mismatch cases into five qualitative categories and propose four additional rubric attributes and five adjustments to the hint-generation process, with preliminary validation on the same 16 mismatch cases that motivated the new rubric.","tokens_in":8233,"tokens_out":4241,"duration_ms":41171,"significance":"If the descriptive mismatch result is robust, the paper offers a useful, actionable contribution to the learning-at-scale community: it provides concrete evidence that expert-only rubric evaluation can diverge from students' perceived helpfulness, and it names five plausible reasons for that divergence. The strengths are the real classroom deployment, the public repository, the clear worked example in Figure 1, and the transparent reporting of all counts in Table 1. However, the main inferential claim currently rests on an underspecified statistical test, the expert and qualitative coding lack reliability evidence, and the preliminary validation of the extended rubric is circular. These issues do not destroy the descriptive value of the mismatch counts, but they do limit the strength of the conclusion that rubric-based expert evaluation is systematically overly optimistic.","major_comments":[{"comment":"The reported chi-square test is not the appropriate test for the claim being made and is under-specified. The data are paired, because the same hint receives both a student rating and an expert rating, so the null hypothesis relevant to 'experts are more optimistic' is that the two discordant counts (37 and 11) are equal. That hypothesis should be tested with McNemar's test or an exact binomial test on the discordant pairs, not with a chi-square test of independence on the 2×2 table, which tests association rather than asymmetry. In addition, the 139 rated hints are nested within 34 students, roughly four hints per student, so unadjusted tests may overstate significance; a cluster-aware paired test or a mixed-effects model is needed. The manuscript cites Cochran (1952) without reporting the test statistic, degrees of freedom, or any treatment of clustering, so the reader cannot verify that p≤0.001 refers to the asymmetry. Please re-run the analysis with a paired, cluster-adjusted test and report the statistic; if significance does not survive, the mismatch finding should be presented as descriptive rather than inferential.","section":"§4, Table 1 (RQ1)"},{"comment":"The preliminary validation of the extended rubric is circular. The four new attributes in Table 3a were derived from the same 16 mismatch cases that are then re-rated with the extended rubric, so the finding that all 16 hints are rated low-quality is largely by construction. This does not establish that the extended rubric aligns with student perceptions on unseen cases. Please validate the extended rubric on held-out mismatch cases, on cases where the original rubric agreed with students, and ideally with independent coders who were not involved in designing the attributes.","section":"§5, Table 3a (preliminary validation)"},{"comment":"The expert side of the comparison is a joint consensus rating produced by two experts discussing each hint, and no inter-rater reliability is reported. Because the central comparison treats expert ratings as ground truth, the absence of an independent-rating agreement statistic leaves open the possibility that the expert ratings are idiosyncratic. Similarly, the RQ2 categories in Table 2 were assigned by the authors without a second coder or an agreement measure such as Cohen's kappa. Please report inter-rater reliability for the rubric and coding reliability for the taxonomy, or explicitly characterize these as consensus/exploratory judgments whose generalizability is not yet established.","section":"§3 'Rubric and expert ratings'; §4 RQ2"}],"minor_comments":[{"comment":"The caption says the adjusted prompt incorporates student thoughts with 'changes highlighted in yellow,' but the yellow highlighting is not visible in the printed text; please state exactly which parts were added.","section":"Figure 1 caption"},{"comment":"The sentence 'Since resolving pedagogical conflicts is not the main consideration of this work' excludes the largest mismatch category from the 16-case validation; please clarify the rationale, since the pedagogical-objectives category is 27.0% of the known-reason cases.","section":"§5"},{"comment":"The paper moves between '74 students' in the course and '34 students requested hints'; consider stating both numbers in the abstract to avoid ambiguity about the effective sample size.","section":"Abstract and §1"},{"comment":"The five mismatch categories are presented as mutually exclusive, but a single hint could plausibly fit more than one category; please state whether the categories were treated as exclusive and, if so, how ties were resolved.","section":"Table 2"}],"recommendation":"major_revision","confidential_remarks":"This is a short, in-progress L@S paper whose descriptive finding is plausible and potentially useful, but the main inferential claim is statistically underspecified and the proposed rubric validation is circular. I believe both are fixable within the scope of a revision: the authors should use a cluster-aware paired test and validate the extended rubric on held-out cases. The paper would also benefit from reporting inter-rater reliability for the expert ratings and the qualitative categories. I therefore recommend major revision rather than rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nFirst thing to know: this paper does the direct comparison we've been missing. Prior work used either expert rubrics or student ratings; here the same 139 AI-generated hints get both, from 34 students in a real Python course, and experts rate them on a standard four-attribute rubric. The mismatch is visible: 37 hints experts called high-quality were rated unhelpful by the student, while only 11 went the other way. That asymmetry is the paper's main event.\n\nWhat's good: the five-category taxonomy for why experts over-rate is plausible and grounded in examples. The extended rubric attributes (accounting for student's concern, informative given history, etc.) follow directly from the failure modes and are practical for anyone building hint generators. The paper is honest about being in-progress, small-sample, and single-course; the repo is public.\n\nNow the soft spots. The chi-square test in RQ1 is the one real load-bearing issue. The comparison is paired—student and expert on the same hint—and the claim of 'overly optimistic' rests on the 37-vs-11 asymmetry. That needs McNemar or an exact binomial test on discordant pairs, not a standard chi-square of association. They cite Cochran 1952 but give no statistic or df, and with 139 hints nested in 34 students, the independence assumption is violated anyway. So p≤0.001 likely isn't the right test, and the conclusion that expert evaluations are 'overly optimistic' is, statistically speaking, still up in the air. The descriptive mismatch count stands, but the confidence level should be lower.\n\nOther issues are softer. Two experts rate together with no IRR reported, so we don't know how much the expert side is a single opinion. The RQ2 categories are single-coder. And the preliminary validation of the extended rubric scores the same 16 cases it was built from—no surprise that it catches them. The authors label this as preliminary, so this is a minor over-claim, not a fatal one.\n\nBottom line: worth a serious referee. The comparative design and taxonomy are useful for the AI-tutoring community, and the questions are the right ones. The authors should be pushed to report a proper paired test with clustering, get a second coder on the taxonomy, and ideally validate the extended rubric on held-out data. If they do that, this becomes a genuinely useful reference point.","headline":"A useful small-study comparison of student vs. expert ratings of AI hints, but the headline mismatch is statistically underspecified and the fixes are validated on the cases that motivated them.","tokens_in":8740,"tokens_out":2866,"would_cite":true,"duration_ms":25848,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that expert rubric ratings of AI-generated programming hints are systematically more optimistic than student helpfulness ratings, identifies five reasons for the mismatch, and proposes rubric extensions and prompt…","keywords":["generative AI in education","programming feedback","hint generation","expert rubrics","student helpfulness ratings","mismatch taxonomy","personalized learning at scale"],"falsifier":"Have two or more independent expert pairs rate the same 139 hints with the four-attribute rubric; if expert-pair agreement is low, the 37 'high-quality but unhelpful' count is an artifact of one pair's judgments. A complementary test: in a fresh course, ask students who rate a hint unhelpful to choose from the five mismatch categories, and check whether the categories actually account for their reasons.","tokens_in":7762,"feed_emoji":"🎓","tokens_out":9360,"duration_ms":75014,"temperature":0.7,"pith_summary":"This paper argues that expert rubric ratings of AI-generated programming hints are systematically more optimistic than the ratings of the students who receive them. In a Python course, students and experts disagreed on 48 of 139 rated hints, and in 37 cases the experts called a hint high-quality while the student called it unhelpful—a pattern the paper judges unlikely to be chance ($p \\le 0.001$ by a $\\chi^2$ test). It names five reasons for these over-optimistic ratings, such as hints that ignore the student's stated concern or previous hints, and it proposes an extended rubric plus prompt changes that use the student's thoughts and request history. If the paper is right, offline expert-only evaluation pipelines can overstate the quality of AI hints, and the new rubric attributes would make expert ratings a better stand-in for student helpfulness.","feed_headline":"Expert rubrics call 37 unhelpful AI hints 'high-quality'","feed_subtitle":"Rating AI hints by rubric alone is too optimistic; adding student context and request history closes the gap.","key_machinery":"The argument runs on a $2\\times2$ contingency table of expert versus student ratings, tested with a $\\chi^2$ test, plus a qualitative taxonomy built by hand-coding the 37 high-quality-but-unhelpful cases. The deployed hint generator is a two-stage pipeline: it runs the student's buggy program to obtain the buggy output and a fixed program, then asks a large language model for a chain-of-thought explanation followed by a Socratic-style hint. The taxonomy of five mismatch categories is the central explanatory device, and the proposed remedy is an augmented rubric whose four new attributes—Accounting student's concern, Informative given history, Tackling overarching bug, and Guiding—are designed to capture what the original four attributes miss.","core_discovery":"Comparing a four-attribute expert rubric (Correct, Informative, Conceal, Comprehensible) with student Helpful/Unhelpful ratings on 139 hints from a deployed AI hint generator yielded agreement in 91 cases (65.5%) and disagreement in 48 cases (34.5%), with the disagreement pattern significant by a $\\chi^2$ test. Of the 48 disagreements, 37 were hints the experts rated high-quality but the student rated unhelpful; the paper concludes that rubric-based expert evaluations may be overly optimistic. Coding those 37 cases produces five mismatch categories: mismatched pedagogical objectives, the student's concern not addressed, the student's trajectory ignored, the student's struggle in approach ignored, and the student's progress not acknowledged, with 11 cases of unknown reason. The paper then proposes four new rubric attributes and five prompt adjustments aimed at the four explainable categories, and reports preliminary results that the extended rubric re-rates all 16 targeted mismatch cases as low-quality.","pith_inferences":["If the over-optimism bias generalizes, earlier benchmark comparisons of hint generators that relied only on expert rubrics may need to be re-run with student-helpfulness data, since their relative rankings could shift.","The five-category taxonomy could be turned into a lightweight classifier that flags likely-unhelpful hints at generation time, trained on the student code, hint text, and interaction history.","The same student-versus-expert gap may appear in other formative AI-feedback settings, such as essay comments or math step hints, because the root cause is a rubric describing content quality rather than situational relevance.","An online protocol that asks students who rate a hint unhelpful to select a reason from the five categories would collect ground-truth labels at scale, replacing the hand-coding that currently limits the taxonomy."],"forward_implications":["Rubric-only offline evaluation of hint generators is likely to overstate quality; techniques tuned to the original four attributes may be overfitting to criteria students do not weigh heavily.","The five mismatch categories give a concrete diagnostic checklist: a hint can fail by ignoring the student's concern, earlier hints, the overarching bug, the student's current approach, or the student's progress.","The extended rubric, with the four new attributes, should make expert ratings a closer proxy for student helpfulness, reducing the need to deploy unfinished techniques in classrooms.","Feeding pre-hint thoughts, post-hint thoughts, and request history into the hint-generation prompt should produce hints that target the student's actual difficulty rather than a generic bug in the code.","The extended rubric's effect on hints where students and experts already agree is not yet measured, so the authors' own caution is that the rubric needs validation on those cases before being adopted."],"supporting_citations":[{"why":"It supplies the two-stage hint-generation technique (buggy output plus fixed program) deployed in the course, and one source of the four-attribute rubric.","marker":"[25]"},{"why":"It provides the four-attribute rubric (Correct, Informative, Conceal, Comprehensible) used for expert ratings.","marker":"[15]"},{"why":"It gives the chi-square test used to assess whether the student-expert disagreement departs from chance.","marker":"[8]"},{"why":"It is the earlier conversational programming-support deployment that the hint-generation pipeline adapts.","marker":"[35]"},{"why":"It is the feedback-theory literature that guided the coding of the mismatch categories.","marker":"[5]"},{"why":"It contributes student perceptions of quality feedback that shaped the mismatch taxonomy.","marker":"[9]"},{"why":"It supplies guidance on critical-thinking and problem-solving instruction used to inform the categories.","marker":"[30]"},{"why":"It provides evidence on what students consider good written feedback, used to define the mismatch reasons.","marker":"[31]"}],"fun_headline_variants":["Rubric marks 37 unhelpful AI hints as high-quality","Expert rubric calls 37 useless AI hints 'good'","Student ratings clash with rubrics on 37 AI hints","When rubrics overrate: 37 unhelpful AI hints"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The expert side of the comparison is trusted as ground truth: two experts jointly scored every hint with one rubric and no inter-rater agreement statistic is reported, so if their ratings are idiosyncratic, the mismatch counts and the five categories lose their foundation.","fun_headline_variants_meta":{"raw":{"variants":["Rubric marks 37 unhelpful AI hints as high-quality","Expert rubric calls 37 useless AI hints 'good'","Student ratings clash with rubrics on 37 AI hints","When rubrics overrate: 37 unhelpful AI hints"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000431,"raw_usage":{"total_tokens":2205,"prompt_tokens":956,"completion_tokens":1249,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":572,"completion_tokens_details":{"reasoning_tokens":1178}},"tokens_in":572,"tokens_out":1249,"duration_ms":9656,"temperature":1.0,"reasoning_tokens":1178,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T16:32:17.637377+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Have two or more independent expert pairs rate the same 139 hints with the four-attribute rubric; if expert-pair agreement is low, the 37 'high-quality but unhelpful' count is an artifact of one pair's judgments. A complementary test: in a fresh course, ask students who rate a hint unhelpful to choose from the five mismatch categories, and check whether the categories actually account for their reasons.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"It supplies the two-stage hint-generation technique (buggy output plus fixed program) deployed in the course, and one source of the four-attribute rubric."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"It provides the four-attribute rubric (Correct, Informative, Conceal, Comprehensible) used for expert ratings."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"It gives the chi-square test used to assess whether the student-expert disagreement departs from chance."},{"cited_title":"Zamfirescu-Pereira, Laryn Qi, Bjorn Hartmann, John DeNero, and Narges Norouzi","cited_arxiv_id":null,"evidence_quote":"It is the earlier conversational programming-support deployment that the hint-generation pipeline adapts."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"It is the feedback-theory literature that guided the coding of the mismatch categories."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"It contributes student perceptions of quality feedback that shaped the mismatch taxonomy."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"It supplies guidance on critical-thinking and problem-solving instruction used to inform the categories."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"It provides evidence on what students consider good written feedback, used to define the mismatch reasons."}],"review_version":2}