Pith. sign in

REVIEW 4 major objections 4 minor 69 references

Investigating Student Interaction Patterns with Large Language Model-Powered Course Assistants in Computer Science Courses

T0 review · 4 major / 4 minor · reviewed 2026-08-04 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read LLM course assistant usage peaks at night and in introductory courses, filling gaps in traditional academic support.

desk verdict Real deployment data on an LLM course assistant, but the headline cognitive-level conclusion rests on unreliability-checked self-annotation. read the letter →

arxiv 2509.08862 v1 pith:5NF2UVIM submitted 2025-09-10 cs.CY cs.AIcs.HC

Chang Liu , Loc Hoang , Andrew Stolman , Rene F. Kizilcec , Bo Wu This is my paper
classification cs.CYcs.AIcs.HC
keywords LLMcourseassistantsstudent-LLMinteractionretrieval-augmentedgenerationBloom'staxonomyinquiry-basedlearninghomeworkmodetemporalsupportgapseducationaltechnology
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper reports on a deployed LLM-powered course assistant—a retrieval-augmented chatbot with educator-curated course content and separate general, homework, and practice modes—used by about 2,000 students across six courses at three universities by Spring 2024. Analyzing 12,060 conversations from three computer science courses, the authors aim to show the assistant fills temporal and individual support gaps: usage stays high from 6 PM to 1 AM, peaks before exams, and is proportionally stronger in introductory courses. They also aim to characterize its pedagogical limits: 53.79% of conversations used homework mode, and manual annotation of 200 conversations per course found 92.33% of responses correct and helpful, 2.17% erroneous, 5.5% unhelpful, and only 4% containing worked examples. When the system posed Socratic follow-up questions, they were often ignored, and the questions rarely reached higher cognitive levels. If these patterns hold, LLM assistants can extend academic support beyond campus hours, but realizing deeper inquiry will require more educator involvement in prompts, content, and policies.

What carries the argument

The load-bearing mechanism is the deployed course assistant: a retrieval-augmented generation (RAG) pipeline on a GPT-4 backbone that retrieves chunks from an educator-curated knowledge database, applies course-specific instructions, and routes each question into general, homework, or practice mode. A homework auto-detection feature directs homework-like queries to a hints-instead-of-answers response. The prompt template also instructs the assistant to end some responses with Socratic follow-up questions; this feature is the probe for testing inquiry-based learning, and the logged conversations are the dataset behind every reported pattern.

What would settle it

An independent, blinded re-annotation of the same 600 sampled conversations by two or more coders who were not involved in building the system, with reported inter-rater agreement, would settle whether the correctness and Bloom's-level findings are stable; low agreement or a substantially higher error rate would overturn the central quality claims.

Watch

Extended reading notes

Core claim

The central discovery is a data-driven characterization of how students use an LLM-powered course assistant in real courses. Usage stayed consistently high from 6 PM to 1 AM, ramped up before midterms and finals, and was proportionally stronger in introductory computer science than in advanced courses—evidence, the authors argue, that the assistant reaches students when human support is unavailable and is especially important to novices. In 600 manually annotated conversations, 92.33% of responses were judged correct and helpful, 2.17% contained errors (about half computational, half conceptual), and 5.5% were unhelpful or off-target; only 4% included worked examples. The system's Socratic f

Load-bearing premise

The paper's quality and cognitive-level results rest on the authors' own manual annotation of sampled conversations, with no reported inter-rater reliability or independent coder check.

Editorial extensions

If this is right

  • If the usage patterns hold, LLM course assistants can demonstrably extend academic support into the evening and night hours when human help is unavailable.
  • If introductory-course students are the heaviest users, institutions adopting such assistants should expect the largest impact—and the largest responsibility—in early CS courses.
  • The prevalence of homework-mode use and the 21% of sessions with no question suggest that homework support with hints rather than answers is a key function, and that many students may need more scaffolding to ask productive questions.
  • If LLM-generated follow-up questions remain at lower Bloom's levels and are mostly ignored, prompt-level changes alone are unlikely to create deep inquiry; educator-designed question sets and policies will be needed.
  • The observed computational and conceptual errors, though a small share of responses, imply that monitoring and educator oversight of correctness remain necessary in deployed systems.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial inference: the nighttime usage result could generalize to other institutions with similar course pacing, but the absence of learning-outcome measures means the paper does not establish that off-hours use improves grades or retention.
  • Editorial inference: the low follow-up response rate may partly reflect interface design—follow-up questions appearing at the end of long text blocks can be easy to miss—rather than a fixed student disinterest in Socratic interaction.
  • Editorial inference: the Bloom's-level ceiling is likely configuration-dependent; explicitly prompting for 'Evaluate' or 'Create' level questions or using a secondary classifier could push the distribution higher than observed here.
  • Editorial inference: a testable extension would be to compare courses with and without educator-authored follow-up questions to see whether content-based questions receive higher response rates than LLM-generated ones.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. This paper reports on a deployed LLM-powered course assistant built on GPT-4 with retrieval-augmented generation, used by roughly 2,000 students across six courses at three institutions, and analyzes 12,060 conversations from three computer science courses at one institution in Spring 2024. The authors characterize usage patterns (temporal distribution, conversation mode, duration, and dialogue rounds), manually annotate sampled conversations for response correctness/helpfulness and Bloom's taxonomy levels, and analyze LLM-generated follow-up questions. They conclude that the assistant addresses temporal and novice support gaps, that homework mode is heavily used, that response quality is generally high with occasional computational/contextual errors, and that current LLM capabilities are limited in generating higher-order cognitive questions.

Significance. If fully supported, the paper would be a valuable empirical contribution to human-LLM interaction in education: it draws on a large real-world corpus, spans courses of different levels, and documents concrete failure cases (Figures 12-13). The descriptive usage results—temporal patterns, mode distribution, conversation lengths—are likely sound and constitute a useful baseline for future deployments. However, the headline cognitive-level and response-quality findings rest on manual annotation with no reported reliability evidence, and the generalization from one system/prompt configuration to 'current LLM capabilities' exceeds the evidence. The authors disclose the commercial affiliation and IRB review, which is commendable.

major comments (4)
  1. [§4.4, Figures 15-18] The Bloom's taxonomy classification, correctness/helpfulness coding, and follow-up response annotation are described as manual annotation of sampled conversations, but no rubric, coder count, training procedure, or inter-rater reliability (e.g., Cohen's kappa) is reported. Two authors are HiTA AI employees, so the coders are not independent of the system's developers. The 92.33% correct-and-helpful figure and the claim that LLM-generated questions are 'typically at the Apply level' therefore cannot be distinguished from coder expectation. This is load-bearing for RQ3 and for the Abstract's conclusion that current LLM capabilities are limited in generating higher-order cognitive questions.
  2. [Abstract; §4.4.2; §6(C)] The paper generalizes from this specific system to 'current LLM capabilities.' The system uses GPT-4 with a prompt that explicitly instructs a Socratic dialogue style, and a particular RAG configuration (top-2 chunks, 4,000-character chunk size, §3.3). The observed low cognitive level of LLM-generated questions may be an artifact of this prompt and retrieval setup rather than an intrinsic property of current LLMs. The claim should be reframed as applying to this system/configuration, or supported by comparative evidence across prompts/models.
  3. [§4.1, §4.2, footnote 1] Developer/test conversations are not excluded from the usage analysis. Footnote 1 states that 4.13% of the 12,060 conversations come from developer users, and that 44.89% of developer conversations contain no questions. Since all reported usage statistics (temporal distribution, mode distribution, dialogue-round distribution) appear to be computed over the full corpus, the reported percentages include non-student traffic. The paper should report the analyses with developer conversations removed, or show that the results are robust to their inclusion.
  4. [§4.2.2; Abstract] The claim that 'usage remains strong in the evenings and nights ... indicating that our system helps address temporal support gaps and novice learner needs' goes beyond the data. The study shows that students queried the system at night and in introductory courses, but it does not measure whether these interactions resolved students' problems, improved learning, or effectively substituted for unavailable human support. Since RQ1 asks whether the assistant 'addresses gaps,' this needs direct evidence (e.g., resolution rates, follow-up success, student-reported outcomes) or a more cautious wording such as 'is used during hours when conventional support is unavailable.'
minor comments (4)
  1. [§4.3.1] The linguistic feature annotations (grammar, politeness, relevance, copied content) also lack a detailed coding rubric and reliability statistics. If these annotations were double-coded, please report agreement; otherwise, present them as exploratory.
  2. [Figures 15-18] The figures report counts and relative frequencies without confidence intervals. Given the sampled nature of the annotation (200 conversations per course), confidence intervals or error bars would aid interpretation and prevent over-reading small differences.
  3. [§4.1] The sentence describing student recruitment says students were 'invited and given access,' but does not clarify whether participation was opt-in or whether informed consent was obtained as part of the IRB protocol. Please clarify the consent procedure.
  4. [Various] Minor typographical and stylistic issues: the arXiv title has inconsistent capitalization ('LARGELANGUAGEMODEL' run-on), the subsection label '(A) Asking while Learning' is awkward, and reference [5] is a bare URL lacking an access date.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the usage findings are log-derived and the annotated quality/Bloom claims, while validity-limited, do not reduce to their inputs by construction.

full rationale

The paper's central usage claims—evening/night use, higher introductory-course use, homework-mode concentration, and follow-up question response rates—are computed directly from anonymized interaction logs (§4.2), with no fitted parameter later renamed as a prediction. The quality and cognitive-level findings come from manual annotation of 200 sampled conversations per course (§4.1, §4.4.1); the annotation lacks reported inter-rater reliability and is performed by the system's own developer-authors, which is a genuine internal-validity limitation, and the generalization to 'current LLM capabilities' (§4.4.2) exceeds the single GPT-4, single-prompt configuration studied. But this is not circularity in the defined sense: the Bloom categories and correctness labels are not defined in terms of the conclusions, no equation or fitted value is reused as an output, and the conclusions could in principle have come out differently under the same annotation scheme. Self-citations [36,37] describe the same platform in prior venue papers; they are not load-bearing evidence for the empirical claims, and no uniqueness theorem or ansatz is imported from the authors' prior work. The competing-interest declaration also warrants caution in interpreting the annotations, but it does not demonstrate a circular derivation. Hence no step in the derivation chain is equivalent to its inputs by construction.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The central claims rest on log data and subjective annotation. The RAG retrieval parameters (top-k=2, chunk size 4000 characters) are hand-chosen. The representativeness of the samples and the validity of Bloom's taxonomy in this context are assumed.

free parameters (2)
  • RAG top-k retrieval count = 2
    Number of most similar chunks retrieved per query; chosen by empirical practice, not validated.
  • Knowledge base chunk size = 4000 characters
    Default chunk size for splitting course materials; chosen by empirical practice.
assumptions (4)
  • domain assumption Conversation logs accurately capture user interaction with the assistant
    The entire analysis relies on log data as ground truth for engagement.
  • domain assumption Bloom's taxonomy is a valid and reliable instrument for classifying cognitive levels of user and LLM questions
    Used to draw conclusions about cognitive depth; no validation in this context.
  • domain assumption Randomly sampled 200 conversations per course are representative of all conversations
    Sampling is stated but no stratification or representativeness checks are reported.
  • ad hoc to paper GPT-4 responses through this specific system are representative of current LLM capabilities
    Generalizations about LLM cognitive limits are drawn from one model and one prompt configuration.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Investigating Student Interaction Patterns with Large Language Model-Powered Course Assistants in Computer Science Courses." pith.science (2026). https://pith.science/paper/5NF2UVIM

@misc{pith2026250908862,
  author       = {Pith},
  title        = {Pith review of: Investigating Student Interaction Patterns with Large Language Model-Powered Course Assistants in Computer Science Courses},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/5NF2UVIM}},
  note         = {Machine review of arXiv:2509.08862}
}
read the original abstract

Providing students with flexible and timely academic support is a challenge at most colleges and universities, leaving many students without help outside scheduled hours. Large language models (LLMs) are promising for bridging this gap, but interactions between students and LLMs are rarely overseen by educators. We developed and studied an LLM-powered course assistant deployed across multiple computer science courses to characterize real-world use and understand pedagogical implications. By Spring 2024, our system had been deployed to approximately 2,000 students across six courses at three institutions. Analysis of the interaction data shows that usage remains strong in the evenings and nights and is higher in introductory courses, indicating that our system helps address temporal support gaps and novice learner needs. We sampled 200 conversations per course for manual annotation: most sampled responses were judged correct and helpful, with a small share unhelpful or erroneous; few responses included dedicated examples. We also examined an inquiry-based learning strategy: only around 11% of sampled conversations contained LLM-generated follow-up questions, which were often ignored by students in advanced courses. A Bloom's taxonomy analysis reveals that current LLM capabilities are limited in generating higher-order cognitive questions. These patterns suggest opportunities for pedagogically oriented LLM-based educational systems and greater educator involvement in configuring prompts, content, and policies.

Figures

Figures reproduced from arXiv: 2509.08862 by the authors.

Figure 1
Figure 1. System Workflow: Users interact with the web interface where their questions are processed by RAG and [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. User Interface: A user interacts with the interface by posing a question. (Course and user information are [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Prompt Structure As of Spring 2024, our system has been deployed across six courses reaching approximately 2,000 students from three universities: Colorado School of Mines, Cornell University, and the University of Colorado Denver. The courses range from entry-level subjects such as Computer Science for STEM to advanced topics like Operating Systems. 6 [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figures from the paper (15 more)
Figure 4
Figure 4. Figure 4: Analysis of User and Conversation Counts [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: Analysis of Conversation Durations 0 1 2 3 4 5 6 7 Number of Dialogue Rounds per Conversation 0 500 1000 1500 2000 2500 3000 3500 Frequency CSS CO OS (a) Frequency of Dialogue Rounds per Conversation 0 1 2 3 4 5 6 7 Number of Dialogue Rounds per Conversation 0% 5% 10% …
Figure 6
Figure 6. Figure 6: Analysis of Dialogue Rounds per Conversation [PITH_FULL_IMAGE:figures/full_fig_p008_6.png]
Figure 7
Figure 7. Figure 7: Analysis of Conversation Initiation by Week of Semester and Hour of Day - CSS [PITH_FULL_IMAGE:figures/full_fig_p009_7.png]
Figure 8
Figure 8. Figure 8: Analysis of Conversation Initiation by Week of Semester and Hour of Day - CO [PITH_FULL_IMAGE:figures/full_fig_p009_8.png]
Figure 9
Figure 9. Figure 9: Analysis of Conversation Initiation by Week of Semester and Hour of Day - OS [PITH_FULL_IMAGE:figures/full_fig_p010_9.png]
Figure 10
Figure 10. Figure 10: Number of Conversations by Mode Practice mode represents another option available to users. Our system does not include an auto-detection feature for this mode. Therefore, despite the pre-defined course document buttons displayed on the webpage, we anticipate lower en…
Figure 11
Figure 11. Figure 11: Frequency of Dialogue Rounds per Conversation by Mode [PITH_FULL_IMAGE:figures/full_fig_p011_11.png]
Figure 12
Figure 12. Figure 12: Example Dialogue Showing Contextual Error [PITH_FULL_IMAGE:figures/full_fig_p011_12.png]
Figure 13
Figure 13. Figure 13: Example Dialogue Showing Computational Error [PITH_FULL_IMAGE:figures/full_fig_p012_13.png]
Figure 14
Figure 14. Figure 14: Distribution of Linguistic Features in User Questions by Course [PITH_FULL_IMAGE:figures/full_fig_p013_14.png]
Figure 15
Figure 15. Figure 15: Analysis of Cognitive Level Classification in User Questions by Course [PITH_FULL_IMAGE:figures/full_fig_p014_15.png]
Figure 16
Figure 16. Figure 16: Analysis of Linguistic and Educational Features in System Responses [PITH_FULL_IMAGE:figures/full_fig_p015_16.png]
Figure 17
Figure 17. Figure 17: Cognitive Level Classification in User-LLM Dialogues: User question refers to the question posed before [PITH_FULL_IMAGE:figures/full_fig_p016_17.png]
Figure 18
Figure 18. Figure 18: Cognitive Level Classification of User-LLM Dialogues by Course [PITH_FULL_IMAGE:figures/full_fig_p016_18.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

69 extracted references · 43 canonical work pages

  1. [1]

    Becoming a teacher: Student teachers’ experiences and perceptions about teaching practice.European journal of teacher education, 35(2):163–178, 2012

    Susana Caires, Leandro Almeida, and Diana Vieira. Becoming a teacher: Student teachers’ experiences and perceptions about teaching practice.European journal of teacher education, 35(2):163–178, 2012

    work page 2012

  2. [2]

    A survey on evaluation of large language models.ACM Transactions on Intelligent Systems and Technology, 15(3):1–45, 2024

    Yupeng Chang, Xu Wang, Jindong Wang, Yuan Wu, Linyi Yang, Kaijie Zhu, Hao Chen, Xiaoyuan Yi, Cunxiang Wang, Yidong Wang, et al. A survey on evaluation of large language models.ACM Transactions on Intelligent Systems and Technology, 15(3):1–45, 2024

    2024

  3. [3]

    When large language models meet personalization: Perspectives of challenges and opportunities.World Wide Web, 27(4):42, 2024

    Jin Chen, Zheng Liu, Xu Huang, Chenwang Wu, Qi Liu, Gangwei Jiang, Yuanhao Pu, Yuxuan Lei, Xiaolong Chen, Xingmei Wang, et al. When large language models meet personalization: Perspectives of challenges and opportunities.World Wide Web, 27(4):42, 2024

    2024

  4. [4]

    Evaluating chatgpt’s information extraction capabilities: An assessment of performance, explainability, calibration, and faithfulness

    Bo Li, Gexiang Fang, Yang Yang, Quansen Wang, Wei Ye, Wen Zhao, and Shikun Zhang. Evaluating chatgpt’s information extraction capabilities: An assessment of performance, explainability, calibration, and faithfulness. arXiv preprint arXiv:2304.11633, 2023

    arXiv 2023

  5. [5]

    https://www.intelligent.com/one-third-of-college-students-used-chatgpt-for-schoolwork-during-the-2022-23-academic-year/ ,

    Intelligent.com. https://www.intelligent.com/one-third-of-college-students-used-chatgpt-for-schoolwork-during-the-2022-23-academic-year/ ,

    work page 2022

  6. [6]

    Students’ voices on generative ai: Perceptions, benefits, and challenges in higher education.International Journal of Educational Technology in Higher Education, 20(1):43, 2023

    Cecilia Ka Yuk Chan and Wenjie Hu. Students’ voices on generative ai: Perceptions, benefits, and challenges in higher education.International Journal of Educational Technology in Higher Education, 20(1):43, 2023

    work page 2023

  7. [7]

    Student perspectives on using a large language model (llm) for an assignment on professional ethics

    Virginia Grande, Natalie Kiesler, and María Andreína Francisco R. Student perspectives on using a large language model (llm) for an assignment on professional ethics. InProceedings of the 2024 on Innovation and Technology in Computer Science Education V . 1, pages 478–484. 2024

    work page 2024

  8. [8]

    Chatgpt has entered the classroom: how llms could transform education.Nature, 623(7987):474– 477, 2023

    Andy Extance. Chatgpt has entered the classroom: how llms could transform education.Nature, 623(7987):474– 477, 2023

    work page 2023

Show all 69 references
  1. [9]

    Large language models for education: A survey and outlook.arXiv preprint arXiv:2403.18105, 2024

    Shen Wang, Tianlong Xu, Hang Li, Chaoli Zhang, Joleen Liang, Jiliang Tang, Philip S Yu, and Qingsong Wen. Large language models for education: A survey and outlook.arXiv preprint arXiv:2403.18105, 2024

    2024 arXiv

  2. [10]

    A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions.arXiv preprint arXiv:2311.05232, 2023

    Lei Huang, Weijiang Yu, Weitao Ma, Weihong Zhong, Zhangyin Feng, Haotian Wang, Qianglong Chen, Weihua Peng, Xiaocheng Feng, Bing Qin, et al. A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions.arXiv preprint arXiv:2311.05232, 2023

    2023 arXiv

  3. [11]

    On assessing the faithfulness of llm-generated feedback on student assignments

    Qinjin Jia, Jialin Cui, Ruijie Xi, Chengyuan Liu, Parvez Rashid, Ruochi Li, and Edward Gehringer. On assessing the faithfulness of llm-generated feedback on student assignments. InProceedings of the 17th International Conference on Educational Data Mining, pages 491–499, 2024

    2024

  4. [12]

    Chatgpt.https://chatgpt.com/, 2024

    OpenAI. Chatgpt.https://chatgpt.com/, 2024. Accessed: 2024-10-24

    2024

  5. [13]

    Sparks of artificial general intelligence: Early experiments with gpt-4.arXiv preprint arXiv:2303.12712, 2023

    Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, et al. Sparks of artificial general intelligence: Early experiments with gpt-4.arXiv preprint arXiv:2303.12712, 2023

    2023 arXiv

  6. [14]

    Student-ai interaction: A case study of cs1 students.arXiv preprint arXiv:2407.00305, 2024

    Matin Amoozadeh, Daye Nam, Daniel Prol, Ali Alfageeh, James Prather, Michael Hilton, Sruti Srinivasa Ragavan, and Mohammad Amin Alipour. Student-ai interaction: A case study of cs1 students.arXiv preprint arXiv:2407.00305, 2024

    2024 arXiv

  7. [15]

    Student interaction with newtbot: An llm-as-tutor chatbot for secondary physics education

    Anna Lieb and Toshali Goel. Student interaction with newtbot: An llm-as-tutor chatbot for secondary physics education. InExtended Abstracts of the CHI Conference on Human Factors in Computing Systems, pages 1–8, 2024

    2024

  8. [16]

    The life cycle of large language models in education: A framework for understanding sources of bias.British Journal of Educational Technology, 55(5):1982–2002, 2024

    Jinsook Lee, Yann Hicke, Renzhe Yu, Christopher Brooks, and René F Kizilcec. The life cycle of large language models in education: A framework for understanding sources of bias.British Journal of Educational Technology, 55(5):1982–2002, 2024

    1982

  9. [17]

    Phases of inquiry-based learning: Definitions and the inquiry cycle.Educational research review, 14:47–61, 2015

    Margus Pedaste, Mario Mäeots, Leo A Siiman, Ton De Jong, Siswa AN Van Riesen, Ellen T Kamp, Constantinos C Manoli, Zacharias C Zacharia, and Eleftheria Tsourlidaki. Phases of inquiry-based learning: Definitions and the inquiry cycle.Educational research review, 14:47–61, 2015

    2015

  10. [18]

    The effect of inquiry-based learning method on students’ academic achievement in science course

    Ali Abdi. The effect of inquiry-based learning method on students’ academic achievement in science course. Universal journal of educational Research, 2(1):37–41, 2014

    2014

  11. [19]

    A complete survey on llm-based ai chatbots.arXiv preprint arXiv:2406.16937, 2024

    Sumit Kumar Dam, Choong Seon Hong, Yu Qiao, and Chaoning Zhang. A complete survey on llm-based ai chatbots.arXiv preprint arXiv:2406.16937, 2024

    2024 arXiv

  12. [20]

    Chat gpt in supporting education instruction sector: an empirical literature review

    Azlini Anuar Tan, Miftachul Huda, Moh Abdul Rohim, Tuan Rusmawati Raja Hassan, Azman Ismail, and Maragustam Siregar. Chat gpt in supporting education instruction sector: an empirical literature review. In International Congress on Information and Communication Technology, page...

    2024

  13. [21]

    Claude.https://www.anthropic.com/, 2024

    Anthropic. Claude.https://www.anthropic.com/, 2024. Accessed: 2024-10-24

    2024

  14. [22]

    A survey on retrieval-augmented text generation

    Huayang Li, Yixuan Su, Deng Cai, Yan Wang, and Lemao Liu. A survey on retrieval-augmented text generation. arXiv preprint arXiv:2202.01110, 2022

    2022 arXiv

  15. [23]

    Retrieval-augmented generation for knowledge- intensive nlp tasks.Advances in Neural Information Processing Systems, 33:9459–9474, 2020

    Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. Retrieval-augmented generation for knowledge- intensive nlp tasks.Advances in Neural Information Processing Systems...

    2020

  16. [24]

    Shamane Siriwardhana, Rivindu Weerasekera, Elliott Wen, Tharindu Kaluarachchi, Rajib Rana, and Suranga Nanayakkara. Improving the domain adaptation of retrieval augmented generation (rag) models for open domain question answering.Transactions of the Association for Computation...

    2023

  17. [25]

    Search engine guided neural machine translation

    Jiatao Gu, Yong Wang, Kyunghyun Cho, and Victor OK Li. Search engine guided neural machine translation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018

    2018

  18. [26]

    Teaching cs50 with ai: leveraging generative artificial intelligence in computer science education

    Rongxin Liu, Carter Zenke, Charlie Liu, Andrew Holmes, Patrick Thornton, and David J Malan. Teaching cs50 with ai: leveraging generative artificial intelligence in computer science education. InProceedings of the 55th ACM Technical Symposium on Computer Science Education V . 1...

    2024

  19. [27]

    Large language models in education: A focus on the complementary relationship between human teachers and chatgpt.Education and Information Technologies, 28(12):15873–15892, 2023

    Jaeho Jeon and Seongyong Lee. Large language models in education: A focus on the complementary relationship between human teachers and chatgpt.Education and Information Technologies, 28(12):15873–15892, 2023

    2023

  20. [28]

    Chatgpt for good? on opportunities and challenges of large language models for education.Learning and individual differences, 103:102274, 2023

    Enkelejda Kasneci, Kathrin Seßler, Stefan Küchemann, Maria Bannert, Daryna Dementieva, Frank Fischer, Urs Gasser, Georg Groh, Stephan Günnemann, Eyke Hüllermeier, et al. Chatgpt for good? on opportunities and challenges of large language models for education.Learning and indiv...

    2023

  21. [29]

    Chatgpt user experience: Implications for education.Available at SSRN 4312418, 2022

    Xiaoming Zhai. Chatgpt user experience: Implications for education.Available at SSRN 4312418, 2022

    2022

  22. [30]

    Artificial intelligence and the future of teaching and learning

    MA Cardona, RJ Rodríguez, and K Ishmael. Artificial intelligence and the future of teaching and learning. Department of Education, 2023

    2023

  23. [31]

    Oatutor: An open-source adaptive tutoring system and curated content library for learning sciences research

    Zachary A Pardos, Matthew Tang, Ioannis Anastasopoulos, Shreya K Sheel, and Ethan Zhang. Oatutor: An open-source adaptive tutoring system and curated content library for learning sciences research. InProceedings of the 2023 chi conference on human factors in computing systems,...

    2023

  24. [32]

    Codeaid: Evaluating a classroom deployment of an llm-based programming assistant that balances student and educator needs.arXiv preprint arXiv:2401.11314, 2024

    Majeed Kazemitabaar, Runlong Ye, Xiaoning Wang, Austin Z Henley, Paul Denny, Michelle Craig, and Tovi Grossman. Codeaid: Evaluating a classroom deployment of an llm-based programming assistant that balances student and educator needs.arXiv preprint arXiv:2401.11314, 2024

    2024 arXiv

  25. [33]

    Jill watson doesn’t care if you’re pregnant: Grounding ai ethics in empirical studies

    Bobbie Eicher, Lalith Polepeddi, and Ashok Goel. Jill watson doesn’t care if you’re pregnant: Grounding ai ethics in empirical studies. InProceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society, pages 88–94, 2018

    2018

  26. [34]

    How do students interact with an llm-powered virtual teaching assistant in different educational settings?arXiv preprint arXiv:2407.17429, 2024

    Pratyusha Maiti and Ashok K Goel. How do students interact with an llm-powered virtual teaching assistant in different educational settings?arXiv preprint arXiv:2407.17429, 2024

    2024 arXiv

  27. [35]

    Educhat: A large-scale language model-based chatbot system for intelligent education.arXiv preprint arXiv:2308.02773, 2023

    Yuhao Dan, Zhikai Lei, Yiyang Gu, Yong Li, Jianghao Yin, Jiaju Lin, Linhao Ye, Zhiyan Tie, Yougen Zhou, Yilei Wang, et al. Educhat: A large-scale language model-based chatbot system for intelligent education.arXiv preprint arXiv:2308.02773, 2023

    2023 arXiv

  28. [36]

    Hita: A rag-based educational platform that centers educators in the instructional loop

    Chang Liu, Loc Hoang, Andrew Stolman, and Bo Wu. Hita: A rag-based educational platform that centers educators in the instructional loop. InInternational Conference on Artificial Intelligence in Education, pages 405–412. Springer, 2024

    2024

  29. [37]

    Understanding student engagement with large language model-powered course assistants

    Chang Liu, Loc Hoang, Andrew Stolman, Rene F Kizilcec, and Bo Wu. Understanding student engagement with large language model-powered course assistants. InInternational Conference on Artificial Intelligence in Education, pages 3–10. Springer, 2025

    2025

  30. [38]

    Automatic generation of programming exercises and code explanations using large language models

    Sami Sarsa, Paul Denny, Arto Hellas, and Juho Leinonen. Automatic generation of programming exercises and code explanations using large language models. InProceedings of the 2022 ACM Conference on International Computing Education Research-Volume 1, pages 27–43, 2022

    2022

  31. [39]

    Assessing the efficacy of large language models in generating accurate teacher responses.arXiv preprint arXiv:2307.04274, 2023

    Yann Hicke, Abhishek Masand, Wentao Guo, and Tushaar Gangavarapu. Assessing the efficacy of large language models in generating accurate teacher responses.arXiv preprint arXiv:2307.04274, 2023

    2023 arXiv

  32. [40]

    How novices use llm-based code generators to solve cs1 coding tasks in a self-paced learning environment.arXiv preprint arXiv:2309.14049, 2023

    Majeed Kazemitabaar, Xinying Hou, Austin Henley, Barbara J Ericson, David Weintrop, and Tovi Grossman. How novices use llm-based code generators to solve cs1 coding tasks in a self-paced learning environment.arXiv preprint arXiv:2309.14049, 2023

    2023 arXiv

  33. [41]

    Llms are biased teachers: Evaluating llm bias in personalized education.arXiv preprint arXiv:2410.14012, 2024

    Iain Weissburg, Sathvika Anand, Sharon Levy, and Haewon Jeong. Llms are biased teachers: Evaluating llm bias in personalized education.arXiv preprint arXiv:2410.14012, 2024. 21

    2024 arXiv

  34. [42]

    Leveraging large language models to power chatbots for collecting user self-reported data.Proceedings of the ACM on Human-Computer Interaction, 8(CSCW1):1–35, 2024

    Jing Wei, Sungdong Kim, Hyunhoon Jung, and Young-Ho Kim. Leveraging large language models to power chatbots for collecting user self-reported data.Proceedings of the ACM on Human-Computer Interaction, 8(CSCW1):1–35, 2024

    2024

  35. [43]

    Significant productivity gains through programming with large language models.Proceedings of the ACM on Human-Computer Interaction, 8(EICS):1–29, 2024

    Thomas Weber, Maximilian Brandmaier, Albrecht Schmidt, and Sven Mayer. Significant productivity gains through programming with large language models.Proceedings of the ACM on Human-Computer Interaction, 8(EICS):1–29, 2024

    2024

  36. [44]

    Is a seat at the table enough? engaging teachers and students in dataset specification for ml in education.Proceedings of the ACM on Human-Computer Interaction, 8(CSCW1):1–32, 2024

    Mei Tan, Hansol Lee, Dakuo Wang, and Hari Subramonyam. Is a seat at the table enough? engaging teachers and students in dataset specification for ml in education.Proceedings of the ACM on Human-Computer Interaction, 8(CSCW1):1–32, 2024

    2024

  37. [45]

    it’s all about conversation

    Tiffany Wenting Li, Karrie Karahalios, and Hari Sundaram. " it’s all about conversation" challenges and concerns of faculty and students in the arts, humanities, and the social sciences about education at scale.Proceedings of the ACM on Human-Computer Interaction, 4(CSCW3):1–37, 2021

    2021

  38. [46]

    Characterizing student engagement moods for dropout prediction in question pool websites.Proc

    Reza Hadi Mogavi, Xiaojuan Ma, and Pan Hui. Characterizing student engagement moods for dropout prediction in question pool websites.Proc. ACM Hum. Comput. Interact., 5(CSCW1):12:1–12:22, 2021

    2021

  39. [47]

    Using cipherbot: An exploratory analysis of student interaction with an llm-based educational chatbot

    Joni Salminen, Soon-gyo Jung, Johanne Medina, Kholoud Aldous, Jinan Azem, Waleed Akhtar, and Bernard J Jansen. Using cipherbot: An exploratory analysis of student interaction with an llm-based educational chatbot. In Proceedings of the Eleventh ACM Conference on Learning@ Scal...

    2024

  40. [48]

    Investigating interaction modes and user agency in human-llm collaboration for domain-specific data analysis

    Jiajing Guo, Vikram Mohanty, Jorge H Piazentin Ono, Hongtao Hao, Liang Gou, and Liu Ren. Investigating interaction modes and user agency in human-llm collaboration for domain-specific data analysis. InExtended Abstracts of the CHI Conference on Human Factors in Computing Syste...

    2024

  41. [49]

    Elahi Hossain, Anna L Cox, Anna Dowthwaite, and Yvonne Rogers. Adaptive, sociable and ready for anything: Undergraduate students are resilient when faced with technological change.Proceedings of the ACM on Human- Computer Interaction, 8(CSCW1):1–32, 2024

    2024

  42. [50]

    Gpt-4 technical report.arXiv preprint arXiv:2303.08774, 2023

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report.arXiv preprint arXiv:2303.08774, 2023

    2023 arXiv

  43. [51]

    Stephan Diederich, Alfred Benedikt Brendel, Stefan Morana, and Lutz Kolbe. On the design of and interaction with conversational agents: An organizing and assessing review of human-computer interaction research.Journal of the Association for Information Systems, 23(1):96–138, 2022

    2022

  44. [52]

    Pearson Education India, 2010

    Ben Shneiderman and Catherine Plaisant.Designing the user interface: strategies for effective human-computer interaction. Pearson Education India, 2010

    2010

  45. [53]

    Text-embedding-ada-002

    OpenAI. Text-embedding-ada-002. https://platform.openai.com/docs/models/ text-embedding-ada-002, 2025. Accessed: 2025-8-18

    2025

  46. [54]

    Addison Wesley Longman, Inc., 2001

    Lorin W Anderson and David R Krathwohl.A taxonomy for learning, teaching, and assessing: A revision of Bloom’s taxonomy of educational objectives: complete edition. Addison Wesley Longman, Inc., 2001

    2001

  47. [55]

    Taxonomy of educational objectives, handbook i: the cognitive domain

    Benjamin S Bloom, Max D Englehart, Edward J Furst, Walker H Hill, David R Krathwohl, et al. Taxonomy of educational objectives, handbook i: the cognitive domain. new york: David mckay co, 1956

    1956

  48. [56]

    Table meets llm: Can large language models understand structured table data? a benchmark and empirical study

    Yuan Sui, Mengyu Zhou, Mingjie Zhou, Shi Han, and Dongmei Zhang. Table meets llm: Can large language models understand structured table data? a benchmark and empirical study. InProceedings of the 17th ACM International Conference on Web Search and Data Mining, pages 645–654, 2024

    2024

  49. [57]

    Evaluating large language models on graphs: Performance insights and comparative analysis.arXiv preprint arXiv:2308.11224, 2023

    Chang Liu and Bo Wu. Evaluating large language models on graphs: Performance insights and comparative analysis.arXiv preprint arXiv:2308.11224, 2023

    2023 arXiv

  50. [58]

    Inquiry-based learning: A review of the research literature.Alberta Ministry of Education, 32:1–32, 2013

    Sharon Friesen and David Scott. Inquiry-based learning: A review of the research literature.Alberta Ministry of Education, 32:1–32, 2013

    2013

  51. [59]

    Gsm-symbolic: Understanding the limitations of mathematical reasoning in large language models.arXiv preprint arXiv:2410.05229, 2024

    Iman Mirzadeh, Keivan Alizadeh, Hooman Shahrokhi, Oncel Tuzel, Samy Bengio, and Mehrdad Farajtabar. Gsm-symbolic: Understanding the limitations of mathematical reasoning in large language models.arXiv preprint arXiv:2410.05229, 2024

    2024 arXiv

  52. [60]

    Logicbench: Towards systematic evaluation of logical reasoning ability of large language models

    Mihir Parmar, Nisarg Patel, Neeraj Varshney, Mutsumi Nakamura, Man Luo, Santosh Mashetty, Arindam Mitra, and Chitta Baral. Logicbench: Towards systematic evaluation of logical reasoning ability of large language models. InProceedings of the 62nd Annual Meeting of the Associati...

    2024

  53. [61]

    Mindful-rag: A study of points of failure in retrieval augmented generation.arXiv preprint arXiv:2407.12216, 2024

    Garima Agrawal, Tharindu Kumarage, Zeyad Alghamdi, and Huan Liu. Mindful-rag: A study of points of failure in retrieval augmented generation.arXiv preprint arXiv:2407.12216, 2024. 22

    2024 arXiv

  54. [62]

    Ragas: Automated evaluation of retrieval augmented generation.arXiv preprint arXiv:2309.15217, 2023

    Shahul Es, Jithin James, Luis Espinosa-Anke, and Steven Schockaert. Ragas: Automated evaluation of retrieval augmented generation.arXiv preprint arXiv:2309.15217, 2023

    2023 arXiv

  55. [63]

    Hallucination is inevitable: An innate limitation of large language models.arXiv preprint arXiv:2401.11817, 2024

    Ziwei Xu, Sanjay Jain, and Mohan Kankanhalli. Hallucination is inevitable: An innate limitation of large language models.arXiv preprint arXiv:2401.11817, 2024

    2024 arXiv

  56. [64]

    Sean B Eom, H Joseph Wen, and Nicholas Ashill. The determinants of students’ perceived learning outcomes and satisfaction in university online education: An empirical investigation.Decision Sciences Journal of Innovative Education, 4(2):215–235, 2006

    2006

  57. [65]

    Kizilcec, and Allison Koenecke

    Emma Harvey, Rene F. Kizilcec, and Allison Koenecke. A framework for auditing chatbots for dialect-based quality-of-service harms. InThe 2025 ACM Conference on Fairness, Accountability, and Transparency, Athens, Greece, June 2025. ACM

    2025

  58. [66]

    Towards an educator-centered method for measuring bias in large language model-based chatbot tutors

    Emma Harvey, Allison Koenecke, and Rene F Kizilcec. Towards an educator-centered method for measuring bias in large language model-based chatbot tutors. InAI for Education: Bridging Innovation and Responsibility at the 38th AAAI Annual Conference on AI, 2024

    2024

  59. [67]

    don’t forget the teachers

    Emma Harvey, Allison Koenecke, and Rene F Kizilcec. "don’t forget the teachers": Towards an educator-centered understanding of harms from large language models in education. InProceedings of the 2025 CHI Conference on Human Factors in Computing Systems, pages 1–19, 2025

    2025

  60. [68]

    To advance ai use in education, focus on understanding educators.International Journal of Artificial Intelligence in Education, 34(1):12–19, 2024

    René F Kizilcec. To advance ai use in education, focus on understanding educators.International Journal of Artificial Intelligence in Education, 34(1):12–19, 2024. 23

    2024

  61. [2023]

    Accessed: 2024-10-22

    2024

Pith tools

Reviewed August 4, 2026 · model on record in the stance chip above.