Pith. sign in

REVIEW 4 major objections 4 minor 100 references

Reflection prompts that deepen student thinking also lower their satisfaction with AI-generated hints, two field experiments show.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-04 06:37 UTC pith:W6L5UIVR

load-bearing objection A timely, honestly reported pair of field experiments whose headline tradeoff is plausible but currently outruns the evidence: the strongest quality result is partly built into the prompt wording, and the abstract overstates non-significant findings. the 4 major comments →

arxiv 2512.04630 v2 pith:W6L5UIVR submitted 2025-12-04 cs.CY

Reflection-Satisfaction Tradeoff: Investigating Impact of Reflection on Student Engagement with AI-Generated Programming Hints

Heeryung Choi , Tung Phung , Mengyan Wu , Adish Singla , Christopher Brooks This is my paper
classification cs.CY
keywords AI-generated hintsreflection promptsself-regulated learningstudent satisfactionprogramming educationfield experimentmetacognitive lazinessgenerative AI
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper claims that in a programming course, prompting students to reflect before receiving an AI-generated hint, focusing on planning, or using directed prompts produces higher-quality reflective writing, but lowers how helpful students rate the AI hints. The inverse link between reflection quality and hint satisfaction appeared consistently across two randomized field trials, while immediate problem-solving performance did not differ across conditions. A sympathetic reader should care because the finding implies that satisfaction-based evaluation of educational AI may push design choices that discourage exactly the effortful thinking the tools are meant to support.

Core claim

Across two field experiments in an online introductory data-science programming course, the study establishes a consistent inverse relationship: conditions that raised the quality of students' reflective responses—reflecting before seeing a hint rather than after, planning-oriented prompts rather than monitoring or evaluation, and directed rather than open prompts—all produced lower student satisfaction with the AI-generated hints. There was no corresponding change in immediate success on the next submission. The authors interpret the pattern as evidence that effortful reflection and user satisfaction are in tension in current AI tutoring systems, and that aligning AI to user preferences alo

What carries the argument

The central mechanism is the reflection prompt itself, varied along three design axes: placement (before vs. after the hint), targeted self-regulated-learning phase (planning, monitoring, evaluation), and amount of guidance (open vs. directed). The study's outcome measures pair reflective-writing quality—coded into components such as what/why/how and SRL phases—with students' binary helpful/unhelpful ratings of each AI hint. The inverse relationship between these two measures carries the argument.

Load-bearing premise

The claim rests on the coding of reflection quality as a valid proxy for deeper learning, but the directed prompt's wording in Trial 2 explicitly asks 'what do you think is a way to fix the bug?', which maps directly onto the 'how' code, so part of the higher reflection quality in that condition may be an artifact of the prompt wording rather than evidence of deeper engagement.

What would settle it

Re-analyze or replicate Trial 2 with a directed prompt that asks students to identify the bug and its impact but omits the explicit 'what is a way to fix the bug?' question. If the 'how' code advantage (χ²=9.73, p=0.002) disappears or reverses, the link between directed guidance and deeper reflection is called into question. Similarly, a replication measuring students' ratings on a multi-item satisfaction scale rather than a binary helpful/unhelpful button could check whether the inverse relationship is an artifact of the rating instrument.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • If the inverse relationship holds, educational AI systems evaluated mainly by user satisfaction will systematically favor designs that produce shallower reflection.
  • Prompt placement before the hint, planning prompts, and directed prompts are concrete design levers that deepen reflection, even at a cost to immediate satisfaction.
  • Immediate success rates did not differ, so the tradeoff is not visible if only performance is measured.
  • The finding motivates alignment methods for educational AI that incorporate metacognitive and self-regulated-learning objectives rather than preference ratings alone.
  • The authors suggest that pre-training on pedagogically aligned data and integrating desirable difficulties into AI optimization could be a more scalable resolution.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Since Trial 2 included reflections in hint generation, the directed prompt's higher 'how' frequency may be inflated by the prompt wording itself; testing a directed prompt that asks for diagnosis without prescribing the 'how' question would isolate the effect.
  • If the tradeoff generalizes, a natural testable extension is whether delayed post-tests (not measured here) would show learning gains for the lower-satisfaction conditions; the paper explicitly does not measure long-term learning.
  • The findings suggest a possible reframing of AI 'helpfulness' in education: satisfaction could be redefined to include perceived progress in self-regulated learning, not just immediate utility.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper reports two randomized field experiments in an online introductory programming course pairing AI-generated hints with reflection prompts. Trial 1 varies prompt placement (before vs. after the hint) and SRL phase (planning/monitoring/evaluation); Trial 2 varies prompt guidance (open vs. directed). The central claim is an inverse relationship between reflection quality and student satisfaction with AI-generated hints: before-hint, planning, and directed prompts are said to yield higher-quality reflections but lower satisfaction. Immediate performance did not differ across conditions. The findings are presented as evidence for a 'reflection-satisfaction tradeoff' with implications for training and evaluating educational AI.

Significance. If the tradeoff were established, the paper would make a useful contribution to the growing literature on metacognitive laziness and AI in education, and its call to reconsider satisfaction-based AI alignment would be timely. The study has real strengths: it is a randomized intervention in an authentic course, includes two trials, reports inter-rater reliability for the qualitative coding, and makes its coding schemes explicit. However, the central claim depends on the construct validity of 'reflection quality' as measured by the author-defined qualitative codes, and that validity is not established. Several prominent results are confounded with prompt wording, and some headline comparisons are not backed by significance tests. The paper is therefore best treated as an exploratory field study with suggestive patterns; it needs substantial revision before the causal and generalizable language in the abstract and conclusions can be supported.

major comments (4)
  1. [§5.3.2 / Table 2 / Table 6] The directed prompt asks 'What do you think is a way to fix the bug?', which is nearly a verbatim definition of the 'How' code ('Suggesting how to resolve the error or its cause'). The significant 'How' difference (14.4% vs 4.8%, p=0.002) is therefore at least partly a demand characteristic: students in the directed condition were instructed to produce the response that the coding scheme counts as high quality. This is load-bearing for RQ3 and for the claimed tradeoff. Please provide evidence that the coding measures reflection depth independent of prompt wording — for example, restrict the analysis to the 'Why' code, compare only unprompted solution suggestions, or run a validation study where the same response is coded under both prompt framings.
  2. [§4.1 / §5.1.4 / Table 1] The before/after comparison varies prompt placement together with prompt content. Before-hint prompts ask about 'your submission and the feedback you have gotten from the system thus far', while after-hint prompts ask about 'the hint you just received'. Reflections after hints that say 'hint was not relevant' are coded as Hint Assessment and interpreted as low-quality reflection, but this may simply reflect the different question being asked. Moreover, no statistical test is reported for RQ1d; the 'higher-quality reflections' claim in the abstract is not backed by a significance test. The placement conclusion requires either a design that holds prompt content constant across placements or an analysis on a common subset of codes.
  3. [§5.2 / Figure 4] RQ2 explicitly reports no statistical tests (n=16–19 per prompt), yet the abstract and §6.1.1 state that planning prompts 'produced higher-quality reflections' and lower satisfaction. The evidence for higher quality is dominated by the 'what' code (100% planning vs 72.7% and 63.6% for monitoring/evaluation, respectively), which is the lowest descriptive level of the 5Rs framework cited in §4.1. Thus the conclusion reverses the paper's own depth hierarchy. At minimum, these claims must be hedged as descriptive trends; ideally the authors should add appropriate significance tests or state clearly why such tests are not possible.
  4. [Abstract / §6.1.1] The 'consistent inverse relationship' across RQ1–RQ3 is not statistically established. RQ1 satisfaction difference is non-significant (χ²=0.46, p=.796), RQ2 had no tests, and only RQ3 satisfaction (p=.017) and the 'How' code (p=.002) reach nominal significance. Given the large number of comparisons (satisfaction, participation, success rate, and multiple codes across RQs), some nominal significance is expected by chance. The paper should either present a pre-specified analysis plan and correction for multiple comparisons, or explicitly characterize the synthesis as exploratory rather than confirmatory.
minor comments (4)
  1. [References] The reference to Bardach et al. contains typos ('Sstudent', 'Oonline'); also, Girden (1992) is a book on repeated-measures ANOVA, which is not the standard citation for a between-subjects ANOVA. Please verify the citation.
  2. [§4.3 / Figure 5d] Krippendorff's alpha after resolving disagreements (0.977, 0.916, 0.950) is not an inter-rater reliability estimate; report the pre-resolution values (0.791, 0.649, 0.670) as the reliability evidence. Also clarify how unresolved cases (N_what=6, N_why=16, N_how=5) were excluded from the denominator and whether the exclusion could bias results.
  3. [Tables 7/8] The abbreviation 'D/E/A' in Table 7 corresponds to 'what/why/how' in Table 6 but is not defined in the table; unify the notation across tables and figure panels.
  4. [§6.1.4] The limitations paragraph states that 'definitive conclusions about when and how reflection practices affect AI support usage and learning gains remain elusive', but the abstract and conclusions draw definitive conclusions about the reflection-satisfaction tradeoff. Align the language between the abstract, limitations, and conclusions.

Circularity Check

0 steps flagged

No significant circularity: satisfaction and reflection quality are independently measured, and the reported inverse relationship is an empirical correlation rather than a construction.

full rationale

The paper's central claim is an empirical inverse relationship between students' satisfaction with AI-generated hints (binary helpful/unhelpful ratings) and the quality of their written reflections (manually coded themes). These two constructs are measured independently: satisfaction comes from in-system hint ratings, and reflection quality comes from expert thematic coding of free-text responses. No equation-level reduction, fitted parameter renamed as a prediction, or self-citation chain forces the result. The reflection-quality coding scheme is author-defined, but it is not derived from satisfaction data and the tradeoff is not true by construction. The strongest validity concern is that prompt wording may be confounded with what the coding counts as 'deep' reflection—e.g., the directed prompt explicitly asks 'What do you think is a way to fix the bug?', which maps closely onto the 'How' code ('Suggesting how to resolve the error or its cause'), and the before/after prompts differ in content as well as placement. However, this is a measurement-validity or construct-confounding threat, not circularity: the coding still requires human judgment of open-ended responses, and the inverse satisfaction relationship is an observed correlation rather than an analytic identity. Self-citations to prior work by overlapping authors (e.g., Choi et al. 2023 for reflection benefits, Phung et al. 2024 for the hint-generation technique) are used for background and system implementation, not as load-bearing justifications of the main empirical finding; there is no invoked uniqueness theorem or imported ansatz that determines the results. The paper also candidly notes limitations such as the lack of long-term outcome measures and small sample size, but these limitations do not indicate circularity. Accordingly, no specific circular step can be exhibited, and the appropriate score is 0.

Axiom & Free-Parameter Ledger

0 free parameters · 4 axioms · 0 invented entities

No model fitting or invented entities; the ledger entries are the qualitative-coding validity assumption, small-sample randomization assumption, satisfaction measurement assumption, and cross-trial comparability assumption. The central empirical claim rests on these domain assumptions.

axioms (4)
  • domain assumption Reflection quality is validly captured by the SRL-phase and Critical Engagement coding scheme (what/why/how).
    The paper's core claim compares 'higher-quality reflections' across conditions, but the coding scheme (Section 4.1, Table 6) overlaps with the directed prompt's wording; inter-rater reliability is reported only for Trial 2.
  • domain assumption Random assignment produced comparable groups despite small N.
    Trial 1 has only 34 students who activated the system; no baseline equivalence checks are reported (Section 3.4, Table 3).
  • domain assumption Self-reported binary hint ratings measure satisfaction.
    Satisfaction is operationalized as a helpful/unhelpful binary rating (Section 3.4); this is a coarse but standard measure.
  • domain assumption GPT-4-generated hints are comparable across trials.
    Trial 2 changed hint generation to include reflections and changed system messages, hint limit, and onboarding (Section 3.2); cross-trial interpretation assumes the tradeoff is not an artifact of these changes.

pith-pipeline@v1.3.0-alltime-deepseek · 24633 in / 8459 out tokens · 83296 ms · 2026-08-04T06:37:30.930312+00:00 · methodology

0 comments
read the original abstract

Generative AI tools, such as AI-generated hints, are increasingly integrated into programming education to offer timely, personalized support. However, little is known about how to effectively leverage these hints while ensuring autonomous and meaningful learning. One promising approach involves pairing AI-generated hints with reflection prompts, asking students to review and analyze their learning, when they request hints. This study investigates the interplay between AI-generated hints and different designs of reflection prompts in an online introductory programming course. We conducted a two-trial field experiment. In Trial 1, students were randomly assigned to receive prompts either before or after receiving hints, or no prompt at all. Each prompt also targeted one of three SRL phases: planning, monitoring, and evaluation. In Trial 2, we examined two types of prompt guidance: directed (offering more explicit and structured guidance) and open (offering more general and less constrained guidance). Findings show that students in the before-hint (RQ1), planning (RQ2), and directed (RQ3) prompt groups produced higher-quality reflections but reported lower satisfaction with AI-generated hints than those in other conditions. Immediate performance did not differ across conditions. This negative relationship between reflection quality and hint satisfaction aligns with previous work on student mental effort and satisfaction. Our results highlight the need to reconsider how AI models are trained and evaluated for education, as prioritizing user satisfaction can undermine deeper learning.

Figures

Figures reproduced from arXiv: 2512.04630 by Adish Singla, Christopher Brooks, Heeryung Choi, Mengyan Wu, Tung Phung.

Figure 1
Figure 1. Figure 1: Illustration of the two randomized controlled trials involved in our study. Sub [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Example illustrating a course question and an interaction between a student [PITH_FULL_IMAGE:figures/full_fig_p011_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Results for RQ1: Impacts of reflection practices and placement. [PITH_FULL_IMAGE:figures/full_fig_p021_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Results for RQ2: Impacts of SRL phase associated with reflection prompts. [PITH_FULL_IMAGE:figures/full_fig_p025_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Results for RQ3: Impacts of prompt types on students’ hint-seeking experience [PITH_FULL_IMAGE:figures/full_fig_p027_5.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

100 extracted references · 1 canonical work pages

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in ":" * " " * FUNCTION f...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in ":" * " " * FUNCTION f...

  3. [3]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in ":" * " " * FUNCTION f...

  4. [4]

    , author Mclaren, B

    author Aleven, V. , author Mclaren, B. , author Roll, I. , author Koedinger, K. , year 2006 . title Toward meta-cognitive tutoring: A model of help seeking with a cognitive tutor . journal International Journal of Artificial Intelligence in Education volume 16 , pages 101--128

    2006

  5. [5]

    , author Wijnia, L

    author Baars, M. , author Wijnia, L. , author de Bruin, A. , author Paas, F. , year 2020 . title Training self-regulation: The effects of self-regulation training on learning and performance . journal Learning and Instruction volume 68 , pages 101325

    2020

  6. [6]

    , author Kadavath, S

    author Bai, Y. , author Kadavath, S. , author Kundu, S. , author Askell, A. , author Kernion, J. , author Jones, A. , author Chen, A. , author Goldie, A. , author Mirhoseini, A. , author McKinnon, C. , et al., year 2022 . title Constitutional ai: Harmlessness from ai feedback . journal arXiv preprint arXiv:2212.08073

    Pith/arXiv arXiv 2022

  7. [7]

    , author Ballantyne, R

    author Bain, J.D. , author Ballantyne, R. , author Packer, J. , author Mills, C. , year 1999 . title Using journal writing to enhance student teachers’ reflectivity during field experience placements . journal Teach. Teach. volume 5 , pages 51--73

    1999

  8. [8]

    , author Mills, C

    author Bain, J.D. , author Mills, C. , author Ballantyne, R. , author Packer, J. , year 2002 . title Developing reflection on practice through journal writing: Impacts of variations in the focus and level of feedback . journal Teach. Teach. volume 8 , pages 171--196

    2002

  9. [9]

    , author Klassen, R.M

    author Bardach, L. , author Klassen, R.M. , author Durksen, T.L. , author Rushby, J.V. , author Bostwick, K.C. , author Sheridan, L. , year 2021 . title T he P ower of F eedback and R eflection: T esting an O online S cenario-based L earning I ntervention for S student T eachers . journal Computers & Education volume 169

    2021

  10. [10]

    , author Bastani, O

    author Bastani, H. , author Bastani, O. , author Sungu, A. , author Ge, H. , author Kabakc , O. , author Mariman, R. , year 2024 . title G enerative AI C an H arm L earning . journal Available at SSRN volume 4895486

    2024

  11. [11]

    , author Denny, P

    author Becker, B.A. , author Denny, P. , author Finnie - Ansley, J. , author Luxton - Reilly, A. , author Prather, J. , author Santos, E.A. , year 2023 . title P rogramming I s H ard - O r at L east I t U sed to B e: E ducational O pportunities and C hallenges of AI C ode G eneration , in: booktitle Proceedings of the Technical Symposium on Computer Scien...

    2023

  12. [12]

    , author Bjork, R.A

    author Bjork, E.L. , author Bjork, R.A. , et al., year 2011 . title Making things hard on yourself, but in a good way: Creating desirable difficulties to enhance learning . journal Psychology and the real world: Essays illustrating fundamental contributions to society volume 2 , pages 56--64

    2011

  13. [13]

    , year 1994

    author Bjork, R.A. , year 1994 . title Memory and metamemory considerations in the training of human beings . journal Metacognition: Knowing about Knowing , pages 185--205

    1994

  14. [14]

    , year 2011

    author Boekaerts, M. , year 2011 . title Self-regulation in the classroom: A perspective on assessment and intervention . journal Applied Psychology volume 54 , pages 199--231

    2011

  15. [15]

    , year 2013

    author Boud, D. , year 2013 . title E nhancing L earning through S elf- A ssessment . publisher Routledge

    2013

  16. [16]

    , author Keogh, R

    author Boud, D. , author Keogh, R. , author Walker, D. , year 1985 . title What is reflection in learning . journal Reflection: Turning experience into learning , pages 7--17

    1985

  17. [17]

    , year 2011

    author Bradford, G.R. , year 2011 . title A relationship study of student satisfaction with learning online and cognitive load: Initial results . journal The Internet and Higher Education volume 14 , pages 217--226

    2011

  18. [18]

    , author Clarke, V

    author Braun, V. , author Clarke, V. , year 2012 . title T hematic A nalysis. publisher American Psychological Association

    2012

  19. [19]

    , author Poon, W.Y

    author Broadbent, J. , author Poon, W.Y. , year 2015 . title Self-regulated learning strategies & academic achievement in online higher education learning environments: A systematic review . journal Internet and Higher Education volume 27 , pages 1--13

    2015

  20. [20]

    , author Baars, M

    author de Bruin, A.B. , author Baars, M. , author van Merriënboer, J.J. , year 2020 . title The effort monitoring and regulation (emr) model: A systematic review of the literature . journal Educational Psychology Review volume 32 , pages 903--926

    2020

  21. [21]

    W e S hould N ot B e L ike a D inosaur

    author Burner, T. , author Lindvig, Y. , author W rness, J.I. , year 2025 . title “ W e S hould N ot B e L ike a D inosaur”— U sing AI T echnologies to P rovide F ormative F eedback to S tudents . journal Education Sciences volume 15

    2025

  22. [22]

    , author Wang, Q

    author Chen, Y. , author Wang, Q. , author Chen, J. , year 2023 . title Investigating students' satisfaction with online collaborative learning: The role of cognitive load . journal Computers & Education volume 201 , pages 104843

    2023

  23. [23]

    , author Jovanovic, J

    author Choi, H. , author Jovanovic, J. , author Poquet, O. , author Brooks, C. , author Joksimovic, S. , author Williams, J.J. , year 2023 . title T he B enefit of R eflection P rompts for E ncouraging L earning with H ints in an O nline P rogramming C ourse . journal The Internet and Higher Education volume 58

    2023

  24. [24]

    , author Yeh, I.J

    author Chow, N.C.H. , author Yeh, I.J. , year 2022 . title Correlation between learning motivation and satisfaction in synchronous on-the-job online training in the public sector . journal Frontiers in Psychology volume 13 , pages 789252

    2022

  25. [25]

    , author Leike, J

    author Christiano, P.F. , author Leike, J. , author Brown, T. , author Martic, M. , author Legg, S. , author Amodei, D. , year 2017 . title Deep reinforcement learning from human preferences . journal Advances in Neural Information Processing Systems volume 30 , pages 4299--4307

    2017

  26. [26]

    , author Braun, V

    author Clarke, V. , author Braun, V. , year 2017 . title Thematic analysis . journal The Journal of Positive Psychology volume 12 , pages 297--298

    2017

  27. [27]

    , author Harvey, M

    author Coulson, D. , author Harvey, M. , year 2012 . title Scaffolding student reflection for experience-based learning: A framework . journal Teaching in Higher Education volume 18 , pages 401--413

    2012

  28. [28]

    , year 2003

    author Davis, E.A. , year 2003 . title Prompting middle school science students for productive reflection: Generic and directed prompts . journal The Journal of the Learning Sciences volume 12 , pages 91--142

    2003

  29. [29]

    , author Linn, M.C

    author Davis, E.A. , author Linn, M.C. , year 2000 . title Scaffolding students' knowledge integration: Prompts for reflection in kie . journal International Journal of Science Education volume 22 , pages 819--837

    2000

  30. [30]

    , author Gulwani, S

    author Denny, P. , author Gulwani, S. , author Heffernan, N.T. , author K \"a ser, T. , author Moore, S. , author Rafferty, A.N. , author Singla, A. , year 2024 a. title G enerative AI for E ducation ( GAIED ): A dvances, O pportunities, and C hallenges . journal CoRR volume abs/2402.01580

    Pith/arXiv arXiv 2024

  31. [31]

    , author MacNeil, S

    author Denny, P. , author MacNeil, S. , author Savelka, J. , author Porter, L. , author Luxton - Reilly, A. , year 2024 b. title D esirable C haracteristics for AI T eaching A ssistants in P rogramming E ducation , in: booktitle Proceedings of the Innovation and Technology in Computer Science Education (ITiCSE) , pp. pages 408--414

    2024

  32. [32]

    , year 1997

    author Dewey, J. , year 1997 . title H ow W e T hink

    1997

  33. [33]

    , author Kim, J

    author Dhollande, P. , author Kim, J. , author Lee, S. , year 2025 . title Fostering autonomous learning practices through reflective frameworks . journal Journal of Learning Analytics volume 12 , pages 50--68

    2025

  34. [34]

    , year 2017

    author Edwards, S. , year 2017 . title Reflecting differently. new dimensions: reflection-before-action and reflection-beyond-action . journal International Practice Development Journal volume 7

    2017

  35. [35]

    , year 2004 a

    author Edwards, S.H. , year 2004 a. title Using software testing to move students from trial-and-error to reflection-in-action , in: booktitle Proceedings of the 35th SIGCSE technical symposium on Computer science education , pp. pages 26--30

    2004

  36. [36]

    , year 2004 b

    author Edwards, S.H. , year 2004 b. title Using software testing to move students from trial-and-error to reflection-in-action , in: booktitle Proceedings of the 35th SIGCSE technical symposium on Computer science education , publisher ACM , address New York, NY, USA

    2004

  37. [37]

    , year 2011

    author Efklides, A. , year 2011 . title Interactions of metacognition with motivation and affect in self-regulated learning: The masrl model . journal Educational Psychologist volume 46 , pages 6--25

    2011

  38. [38]

    , author Tang, L

    author Fan, Y. , author Tang, L. , author Le, H. , author Shen, K. , author Tan, S. , author Zhao, Y. , author Shen, Y. , author Li, X. , author Ga s evi \'c , D. , year 2025 . title B eware of M etacognitive L aziness: E ffects of G enerative A rtificial I ntelligence on L earning M otivation, P rocesses, and P erformance . journal British Journal of Edu...

    2025

  39. [39]

    , author Denny, P

    author Finnie - Ansley, J. , author Denny, P. , author Becker, B.A. , author Luxton - Reilly, A. , author Prather, J. , year 2022 . title T he R obots A re C oming: E xploring the I mplications of O pen AI C odex on I ntroductory P rogramming , in: booktitle Australasian Computing Education Conference (ACE) , pp. pages 10--19

    2022

  40. [40]

    , year 2025

    author Gerlich, M. , year 2025 . title Ai tools in society: Impacts on cognitive offloading and the future of critical thinking . journal Societies volume 15 , pages 6

    2025

  41. [41]

    , year 1992

    author Girden, E.R. , year 1992 . title ANOVA : R epeated M easures

    1992

  42. [42]

    , author Abdelnabi, S

    author Greshake, K. , author Abdelnabi, S. , author Mishra, S. , author Endres, C. , author Holz, T. , author Fritz, M. , year 2023 . title Not what you've signed up for: Compromising real-world llm-integrated applications with indirect prompt injection . journal arXiv preprint arXiv:2302.12173

    Pith/arXiv arXiv 2023

  43. [43]

    , author Järvelä, S

    author Hadwin, A.F. , author Järvelä, S. , author Miller, M. , year 2011 . title Self-regulated learning in the digital age: A theoretical review . journal Educational Psychologist volume 46 , pages 267--284

    2011

  44. [44]

    , author Bent, D

    author Handa, K. , author Bent, D. , author Tamkin, A. , author McCain, M. , author Durmus, E. , author Stern, M. , author Schiraldi, M. , author Huang, S. , author Ritchie, S. , author Syverud, S. , author Jagadish, K. , author Vo, M. , author Bell, M. , author Ganguli, D. , year 2025 . title Anthropic education report: How university students use claude

    2025

  45. [45]

    , author Lossius, M.H

    author Janssen, E.M. , author Lossius, M.H. , author Muis, K.R. , year 2023 . title Overcoming misconceptions about desirable difficulties in learning . journal Educational Psychology Review volume 35 , pages 1--28

    2023

  46. [46]

    , author Mayer, R.E

    author Johnson, C.I. , author Mayer, R.E. , year 2010 . title Applying the self-explanation principle to multimedia learning in a computer-based game-like environment . journal Computers in Human Behavior volume 26 , pages 1246--1252 . :10.1016/j.chb.2010.03.025

    work page doi:10.1016/j.chb.2010.03.025 2010

  47. [47]

    , author Hauptmann, E

    author Kosmyna, N. , author Hauptmann, E. , author Yuan, Y.T. , author Situ, J. , author Liao, X.H. , author Beresnitzky, A.V. , author Braunstein, I. , author Maes, P. , year 2025 . title Y our B rain on C hatgpt: A ccumulation of C ognitive D ebt W hen U sing an A i A ssistant for E ssay W riting T ask . journal ArXiv Preprint ArXiv:2506.08872

    Pith/arXiv arXiv 2025

  48. [48]

    , year 2011

    author Krippendorff, K. , year 2011 . title Computing krippendorff's alpha-reliability

    2011

  49. [49]

    , author Xiao, R

    author Kumar, H. , author Xiao, R. , author Lawson, B. , author Musabirov, I. , author Shi, J. , author Wang, X. , author Luo, H. , author Williams, J.J. , author Rafferty, A.N. , author Stamper, J.C. , author Liut, M. , year 2024 . title S upporting S elf- R eflection at S cale with L arge L anguage M odels: I nsights from R andomized F ield E xperiments...

    2024

  50. [50]

    , author Sarkar, A

    author Lee, H.P. , author Sarkar, A. , author Tankelevitch, L. , author Drosos, I. , author Rintel, S. , author Banks, R. , author Wilson, N. , year 2025 . title The impact of generative ai on critical thinking: Self-reported reductions in cognitive effort and confidence effects from a survey of knowledge workers , in: booktitle Proceedings of the 2025 CH...

    2025

  51. [51]

    , year 2022

    author Lin, G. , year 2022 . title Using metacognitive prompts to enhance self-regulated learning and learning outcomes: A meta-analysis of experimental studies in computer-based learning environments . journal Journal of Computer Assisted Learning volume 38 , pages 811--832

    2022

  52. [52]

    , author Yadav, A

    author Lishinski, A. , author Yadav, A. , year 2021 a. title Self-evaluation interventions: Impact on self-efficacy and performance in introductory programming . journal ACM Transactions on Computing Education (TOCE) volume 21 , pages 1--28

    2021

  53. [53]

    , author Yadav, A

    author Lishinski, A. , author Yadav, A. , year 2021 b. title Self-evaluation interventions: Impact on self-efficacy and performance in introductory programming: Impact on self-efficacy and performance in introductory programming . journal ACM Trans. Comput. Educ. volume 21 , pages 1--28

    2021

  54. [54]

    , author Xie, B

    author Loksa, D. , author Xie, B. , author Kwik, H. , author Ko, A.J. , year 2020 . title Investigating novices' in situ reflections on their programming process , in: booktitle Proceedings of the 51st ACM technical symposium on computer science education , pp. pages 149--155

    2020

  55. [55]

    , author Chen, L

    author Ma, B. , author Chen, L. , author Konomi, S. , year 2024 . title E nhancing P rogramming E ducation with C hatgpt: A C ase S tudy on S tudent P erceptions and I nteractions in a P ython C ourse , in: booktitle Proceedings of the Artificial Intelligence in Education ( AIED ) , pp. pages 113--126

    2024

  56. [56]

    , author Delahunt, B

    author Maguire, M. , author Delahunt, B. , year 2017 . title D oing a T hematic A nalysis: A P ractical, S tep-by- S tep G uide for L earning and T eaching S cholars. journal All Ireland Journal of Higher Education volume 9

    2017

  57. [57]

    , author Forrest, K

    author Mandernach, B.J. , author Forrest, K. , author Babutzke, J. , author Manker, L. , year 2011 . title The effects of student engagement, student satisfaction, and perceived learning in online learning environments . journal Online Learning volume 15 , pages 74--88

    2011

  58. [58]

    , author Gordon, J

    author Mann, K. , author Gordon, J. , author MacLeod, A. , year 2009 . title R eflection and R eflective P ractice in H ealth P rofessions E ducation: a S ystematic R eview . journal Advances in Health Sciences Education volume 14

    2009

  59. [59]

    , author Prather, J

    author Margulieux, L.E. , author Prather, J. , author Reeves, B.N. , author Becker, B.A. , author Uzun, G.C. , author Loksa, D. , author Leinonen, J. , author Denny, P. , year 2024 . title S elf- R egulation, S elf- E fficacy, and F ear of F ailure I nteractions with H ow N ovices U se LLM s to S olve P rogramming P roblems , in: booktitle Proceedings of ...

    2024

  60. [60]

    , author Williams, J.J

    author Marwan, S. , author Williams, J.J. , author Price, T.W. , year 2019 . title A n E valuation of the I mpact of A utomated P rogramming H ints on P erformance and L earning , in: booktitle Conference on International Computing Education Research ( ICER ) , pp. pages 61--70

    2019

  61. [61]

    , year 2004

    author Moon, J.A. , year 2004 . title A Handbook of Reflective and Experiential Learning: Theory and Practice . publisher Routledge

    2004

  62. [62]

    , author Valiente, J.A.R

    author Mu \ n oz-Merino, P.J. , author Valiente, J.A.R. , author Kloos, C.D. , year 2013 . title Inferring higher level learning information from low level data for the khan academy platform , in: booktitle Proceedings of the third international conference on learning analytics and knowledge , pp. pages 112--116

    2013

  63. [63]

    , author colleagues , year 2023

    author Naeem, M. , author colleagues , year 2023 . title A pragmatic approach to thematic analysis . journal International Journal of Qualitative Methods volume 22 , pages 1--12

    2023

  64. [64]

    title Chat GPT

    author OpenAI , year 2023 . title Chat GPT . howpublished https://openai.com/blog/chatgpt

    2023

  65. [65]

    , year 1998

    author O'Rourke, R. , year 1998 . title The learning journal: from chaos to coherence . journal Assessment & Evaluation in Higher Education volume 23 , pages 403--413

    1998

  66. [66]

    , author Wu, J

    author Ouyang, L. , author Wu, J. , author Jiang, X. , author Almeida, D. , author Wainwright, C. , author Mishkin, P. , author Zhang, C. , author Agarwal, S. , author Slama, K. , author Ray, A. , et al., year 2022 . title Training language models to follow instructions with human feedback . journal Advances in Neural Information Processing Systems volume...

    2022

  67. [67]

    , author Baker, R.S

    author Pankiewicz, M. , author Baker, R.S. , year 2023 . title L arge L anguage M odels (GPT) for A utomating F eedback on P rogramming A ssignments . journal CoRR volume abs/2307.00150

    Pith/arXiv arXiv 2023

  68. [68]

    , author Baker, R.S

    author Pankiewicz, M. , author Baker, R.S. , year 2024 . title N avigating C ompiler E rrors with AI A ssistance - A S tudy of GPT H ints in an I ntroductory P rogramming C ourse , in: booktitle Proceedings of Innovation and Technology in Computer Science Education (ITiCSE) , pp. pages 94--100

    2024

  69. [69]

    , author Ribeiro, I

    author Perez, F. , author Ribeiro, I. , year 2022 . title Ignore previous prompt: Attack techniques for language models . journal arXiv preprint arXiv:2211.09527

    Pith/arXiv arXiv 2022

  70. [70]

    , author P a durean, V

    author Phung, T. , author P a durean, V. , author Cambronero, J. , author Gulwani, S. , author Kohn, T. , author Majumdar, R. , author Singla, A. , author Soares, G. , year 2023 . title G enerative AI for P rogramming E ducation: B enchmarking C hatgpt, GPT -4, and H uman T utors , in: booktitle Proceedings of the Conference on International Computing Edu...

    2023

  71. [71]

    , author Padurean, V

    author Phung, T. , author Padurean, V. , author Singh, A. , author Brooks, C. , author Cambronero, J. , author Gulwani, S. , author Singla, A. , author Soares, G. , year 2024 . title A utomating H uman T utor- S tyle P rogramming F eedback: L everaging GPT -4 T utor M odel for H int G eneration and GPT -3.5 S tudent M odel for H int V alidation , in: book...

    2024

  72. [72]

    , author Sane, A

    author Prasad, P. , author Sane, A. , year 2024 . title A S elf- R egulated L earning F ramework using G enerative AI and its A pplication in CS E ducational I ntervention D esign , in: booktitle Proceedings of the Technical Symposium on Computer Science Education ( SIGCSE ) , pp. pages 1070--1076

    2024

  73. [73]

    , author Denny, P

    author Prather, J. , author Denny, P. , author Leinonen, J. , author Becker, B.A. , author Albluwi, I. , author Craig, M. , author Keuning, H. , author Kiesler, N. , author Kohn, T. , author Luxton - Reilly, A. , author MacNeil, S. , author Petersen, A. , author Pettit, R. , author Reeves, B.N. , author Savelka, J. , year 2023 . title T he R obots A re H ...

    2023

  74. [74]

    , author Sharma, A

    author Rafailov, R. , author Sharma, A. , author Mitchell, E. , author Ermon, S. , author Manning, C.D. , author Finn, C. , year 2023 . title Direct preference optimization: Your language model is secretly a reward model . journal arXiv preprint arXiv:2305.18290

    Pith/arXiv arXiv 2023

  75. [75]

    , author Ismail, M.A

    author Rum, S.N.M. , author Ismail, M.A. , year 2017 . title Metocognitive support accelerates computer assisted learning for novice programmers . journal Journal of Educational Technology & Society volume 20 , pages 170--181

    2017

  76. [76]

    , author Ryan, M

    author Ryan, M. , author Ryan, M. , year 2014 . title The pedagogical balancing act: Teaching reflection in higher education . journal Teaching in Higher Education volume 19 , pages 144--155

    2014

  77. [77]

    , author Kandimalla, S.R

    author Sankaranarayanan, S. , author Kandimalla, S.R. , author Bogart, C.A. , author Murray, R.C. , author Hilton, M. , author Sakr, M.F. , author Ros \'e , C.P. , year 2022 . title Collaborative programming for work-relevant learning: Comparing programming practice with example-based reflection for student learning and transfer task performance . journal...

    2022

  78. [78]

    , year 2017

    author Sch \"o n, D.A. , year 2017 . title T he R eflective P ractitioner: H ow P rofessionals T hink in A ction . volume volume 73

    2017

  79. [79]

    , author Schüler, A

    author Seufert, T. , author Schüler, A. , author Lehmann, J. , year 2024 . title The role of cognitive load in self-regulated learning . journal Educational Psychology Review volume 36 , pages 441--465

    2024

  80. [80]

    , author Ai, X

    author Shen, Y. , author Ai, X. , author Raj, A.G.S. , author John, R.J.L. , author Syamkumar, M. , year 2024 . title I mplications of C hat GPT for D ata S cience E ducation , in: booktitle Proceedings of the Technical Symposium on Computer Science Education ( SIGCSE ) , pp. pages 1230--1236

    2024

Showing first 80 references.