REVIEW 3 major objections 9 minor 2 cited by
From Interests to Insights: An LLM Approach to Course Recommendations Using Natural Language Queries
T0 review · 3 major / 9 minor · reviewed 2026-08-11 · deepseek-v4-flash
Pith's one-line read The paper claims that an LLM can turn a student's casual interests into an 'ideal' course description and use that description to retrieve genuinely relevant courses from a catalog alone.
desk verdict A clear, honest exploratory system paper for LLM-based course recommendations that never tests its core design choice, leaving its main novelty unproven. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
Two-stage retrieval with LLM-generated query expansion. Stage one prompts a language model to write the 'ideal' course description for the user's query, embeds that text, and retrieves the fifty course descriptions with highest cosine similarity to it (a min-priority queue over 1,536-dimensional embeddings). Stage two feeds those fifty descriptions plus the original query to a second LLM, which returns ten courses with short rationales and confidence ratings. The load-bearing identity is that similarity between the ideal-description embedding and course-description embeddings captures true curricular relevance better than direct similarity between the user's own words and the catalog.
What would settle it
Run the retrieval stage twice on the same query set: once with the LLM-generated ideal description as the query vector, and once with the user's raw query embedded directly (and optionally with BM25). Have human advisers blind-rank the top fifty results for relevance. If raw-query or BM25 results match or beat the ideal-description results, the paper's central design choice is not doing the work.
Extended reading notes
Core claim
The paper's central claim is that an LLM plus RAG pipeline built only from course descriptions consistently generates contextually appropriate recommendations across diverse query types, from broad exploratory prompts to focused technical ones. The system does not use enrollment histories, grades, or demographic features. Instead it operationalizes the idea that an LLM can translate a student's casual phrasing into an 'ideal' course description in catalog language; that description, embedded, becomes a better retrieval query than the student's raw words; the top fifty courses by cosine similarity form the context from which a second LLM draws ten recommendations, each with a rationale and a confidence level. The authors support the claim with hand-inspected examples, subject-level embedding networks, a similarity-rank analysis showing the top-ranked course appears among final recommendations about 85% of the time, and paired demographic queries whose top recommendations do not differ.
Load-bearing premise
The whole pipeline rests on the claim that having a language model rewrite the user's query as an 'ideal' course description makes the embedding search more relevant than searching with the user's own words, and the paper never compares those two options directly.
Editorial extensions
If this is right
- A pilot deployed within a campus course-exploration service could let students and advisers query in natural language and receive explainable suggestions in seconds.
- Because the method needs only course descriptions, institutions without rich enrollment data can deploy it quickly.
- Recommendations can be constrained by course level (100-200 vs 300-400 etc.) to match student stage.
- The context window of fifty courses is a pragmatic balance; similarity rank strongly predicts recommendation likelihood, with the top twelve ranks contributing around 60% of recommended courses.
- Top recommendations are stable across demographic variants in paired tests, though lower-ranked courses vary.
Reading between the lines
- A head-to-head retrieval test would settle whether the ideal-description step is doing the work: compare it against directly embedding the user's raw query and against a sparse baseline like BM25 on the same relevance judgments.
- The demographic variation observed at lower ranks could be probed without invoking demographics at all, by running paraphrased queries that differ only in wording and checking whether similar recommendation variance appears.
- The sharp drop in recommendation likelihood beyond the top twelve ranks suggests that a shorter context window might preserve most recommendation quality at lower computational cost, though the paper does not test this.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper describes an LLM-based course recommendation system for the University of Michigan. A user's natural-language query is first transformed by GPT-3.5-turbo into an 'ideal' course description, which is embedded with text-embedding-ada-002 and matched by cosine similarity against a corpus of course descriptions. The top-50 most similar courses are then passed to GPT-4o, which returns ten recommendations with rationales and confidence ratings. The paper reports exploratory analyses of the embedding space, a rank-versus-recommendation-likelihood study, two illustrative recommendation examples, a preliminary bias study across gender, race, and sexuality, and speed measurements. The authors explicitly frame the work as exploratory and acknowledge the lack of ground truth in course recommendation evaluation.
Significance. If the claimed behavior holds, the paper offers a practical, low-data recipe for course discovery that does not require enrollment histories or credentialing rules, which could be useful for students and advisers on large campuses. The two-stage retrieval idea—generating an idealized description before embedding—is intuitive and potentially valuable, and the RAG grounding in current course descriptions is a sensible way to mitigate hallucination. I credit the authors for candidly labeling the work exploratory, for including speed measurements, and for structuring the bias study with paired demographic queries. The paper does not provide machine-checked proofs, reproducible code (the repository is only promised), or parameter-free derivations; its value rests on the soundness of the empirical claims, which are currently supported mainly by qualitative inspection.
major comments (3)
- [Section 4.2, Algorithm 1] The central methodological novelty—using an LLM-generated 'ideal' course description as the embedding query—is never tested against the alternatives it is meant to improve upon. The paper motivates the design by a claimed 'lexical and semantic gap' between student queries and course descriptions, but no experiment compares this two-stage retrieval against direct embedding of the raw query, BM25, DPR-style retrieval, or a hybrid baseline. Because the final GPT-4o recommender is restricted to the top-50 courses returned by this retrieval step, and Section 5.2 shows that recommendation likelihood is concentrated in the top similarity ranks, retrieval errors propagate directly into the final output. The rank-likelihood analysis in Section 5.2 does not fill this gap, since it measures internal consistency rather than retrieval quality against a reference set. I recommend adding a small hand-labeled query-course relevance set and reporting retrieval metrics (e.g., recall at 20/50, nDCG) for the proposed method and at least one baseline.
- [Section 5.3.3, Figure 4] The paper's central claim that 'the system consistently generates contextually appropriate recommendations across diverse query types' rests on manual inspection of two example outputs and informal 'extended testing.' The authors themselves state in Section 5.3.3 that they 'manually inspect the course descriptions associated with each recommendation and verify that the provided rationales are accurate.' This is not sufficient evidence for a consistency claim. I would ask for a structured human evaluation: multiple annotators, a scoring rubric (relevance, rationale accuracy, coverage), inter-annotator agreement, and ideally a preference comparison against an adviser-based or rule-based baseline. Without such an evaluation, the central assertion remains anecdotal.
- [Section 5.4, Figure 5] The bias-testing conclusion that recommendation variations 'appear unrelated to learned societal biases' and arise from 'attractors in the recommendation space' is not supported by the reported evidence. The experiments use 100 trials per demographic pair but report no confidence intervals, significance tests, or multiple-comparison corrections, and the effect sizes vary widely across courses (e.g., EECS 551 at ~0.9 vs ~0.0 across query pairs). The dynamical-systems 'attractor' explanation is introduced post hoc, with no predictive test, and the paper itself concedes it 'cannot conclusively demonstrate this mechanism.' At minimum, the authors should report effect sizes with uncertainty, run permutation tests with matched demographic descriptors, and present the attractor account as one of several hypotheses rather than as the explanation.
minor comments (9)
- [Section 5.4, Figure 5 caption] The caption contains a typo: 'man/women' should be 'man/woman.'
- [Section 5.1, Figure 2d text] The text says 'We see a strong cultual studies network'; 'cultual' should be 'cultural.'
- [Appendix B, EECS 551 description] The course description contains the typo 'engenvalue'; it should be 'eigenvalue.'
- [Section 5.5, Table 1] The table reports retrieval and total times without standard deviations or the number of trials per level, and the retrieval-time values have inconsistent decimal places (2.10, 2.57, 2.8, 5.22); please clarify the trial counts and report variability.
- [Section 5.2, Figure 3] The interpretation that high recommendation rates for top-ranked courses indicate 'alignment between embedding-based filtering and the language model's reasoning' should be tempered, because LLMs are known to be sensitive to the order of items in the context; an alternative explanation is position bias, and a control experiment with shuffled context order would strengthen the claim.
- [Section 4.3] The text states that temperature is set to 0 but that 'significant output variability persists'; please clarify whether this variability is due to the retrieval step (e.g., non-deterministic generation of the ideal description) or to the final model, since temperature=0 should make GPT-4o deterministic.
- [Section 5.4] The queries use 'man/woman' and the section title says 'birth sex'; sex and gender are distinct concepts, and the terminology should be made consistent and precise.
- [Author affiliations] The first author is listed with 'New York, NY' while the other authors are at the University of Michigan in Ann Arbor; this appears to be an error and should be corrected.
- [Availability section] The paper says source code 'will be made available' at a GitHub URL; for reproducibility, please provide the actual repository link or at least specify a release date, and consider adding a data availability statement for the course-description corpus.
Circularity Check
No significant circularity: the paper is an exploratory, non-derivational pipeline description whose evaluations are heuristic self-assessments, not predictions forced by fitted inputs or self-citation.
full rationale
This paper contains no mathematical derivation chain, no fitted parameters, and no quantity that is predicted from a value to which it is definitionally tied. The two-stage retrieval pipeline (ideal-description generation followed by embedding similarity search and GPT-4o ranking) is described operationally rather than derived, and the central claims are supported by manual inspection of example outputs, embedding-space visualizations, and rank-frequency statistics. The only author-overlapping citation, to the Atlas platform [Evrard et al., 2023], provides deployment context and is not load-bearing for any stated result. The absence of a baseline comparison for the 'ideal course description' retrieval step is a genuine empirical weakness, but it is a correctness or evaluation gap, not a circular reduction: the paper never claims to have tested that step against alternatives, so nothing is being presented as a validated prediction. Likewise, the statement in Section 5.3.3 that 'we manually inspect the course descriptions ... and verify that the provided rationales are accurate' is a subjective quality assessment, not a derivation of the system's quality from its own outputs. Under the hard rules, no circular step can be quoted and exhibited, so the appropriate score is 0.
Assumptions & free parameters
free parameters (3)
- Context window size k =
50
- Number of final recommendations =
10
- Sampling temperature for GPT-4o =
0
assumptions (4)
- domain assumption text-embedding-ada-002 maps course descriptions into a space where cosine similarity reflects meaningful semantic relatedness.
- ad hoc to paper An LLM-generated ideal description is a more effective retrieval query than the user's original text.
- domain assumption GPT-4o, given a context of 50 retrieved courses and the original query at temperature 0, returns reliable and constraint-respecting recommendations.
- domain assumption Course descriptions alone contain enough information to recommend courses well.
invented entities (1)
-
Weak and strong attractors in the LLM recommendation space
Cite this review
Pith. "Pith review of From Interests to Insights: An LLM Approach to Course Recommendations Using Natural Language Queries." pith.science (2026). https://pith.science/paper/IYWPOUTU
@misc{pith2026241219312,
author = {Pith},
title = {Pith review of: From Interests to Insights: An LLM Approach to Course Recommendations Using Natural Language Queries},
year = {2026},
howpublished = {\url{https://pith.science/paper/IYWPOUTU}},
note = {Machine review of arXiv:2412.19312}
}
read the original abstract
Most universities in the United States encourage their students to explore academic areas before declaring a major and to acquire academic breadth by satisfying a variety of requirements. Each term, students must choose among many thousands of offerings, spanning dozens of subject areas, a handful of courses to take. The curricular environment is also dynamic, and poor communication and search functions on campus can limit a student's ability to discover new courses of interest. To support both students and their advisers in such a setting, we explore a novel Large Language Model (LLM) course recommendation system that applies a Retrieval Augmented Generation (RAG) method to the corpus of course descriptions. The system first generates an 'ideal' course description based on the user's query. This description is converted into a search vector using embeddings, which is then used to find actual courses with similar content by comparing embedding similarities. We describe the method and assess the quality and fairness of some example prompts. Steps to deploy a pilot system on campus are discussed.
Figures
Figures from the paper (6 more)
Forward citations
Cited by 2 Pith papers
-
KNOWPLAN: Knowledge-Driven AI Agents for Smart Degree Pathway Planning
An extraction-first pipeline, CatalogBrowse plus DegreeMap, reconstructs university curricula from heterogeneous public sources and certifiably optimizes feasible personalized degree plans, though no code or data is released.
-
SmartCourse: A Contextual AI-Powered Course Advising System for Undergraduates
Giving an LLM a transcript and degree plan makes its course suggestions align more with that plan, but the evaluation partly assumes that plan is the correct answer.
Reference graph
Works this paper leans on
-
[4]
doi:10.1109/IISA.2017.8316368. August E. Evrard, Rebecca Matz, Erin Murray, Ben Hayward, Mark Mills, and Caitlin Hayward. Innovating at campus scale: The case of michigan’s atlas. In Proceedings of the Tenth ACM Conference on Learning @ Scale , L@S ’23, page 306–310, New York, NY , USA,
-
[9]
Shuyuan Xu, Wenyue Hua, and Yongfeng Zhang
URL https://arxiv.org/abs/ 2203.13366. Shuyuan Xu, Wenyue Hua, and Yongfeng Zhang. Openp5: An open-source platform for developing, training, and evaluating llm-based recommender systems. In Proceedings of the 47th International ACM SIGIR Conference on Research and Development in Information Retrieval , SIGIR ’24, page 386–394, New York, NY , USA,
-
[11]
URL https://arxiv.org/abs/2205.08084. Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. Retrieval-augmented generation for knowledge-intensive nlp tasks,
-
[12]
Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Ming-Wei Chang
URL https://arxiv.org/abs/2005.11401. Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Ming-Wei Chang. Realm: Retrieval-augmented language model pre-training,
arXiv 2005
-
[13]
Stephen Robertson and Hugo Zaragoza
URL https://arxiv.org/abs/2002.08909. Stephen Robertson and Hugo Zaragoza. The probabilistic relevance framework: Bm25 and beyond. Found. Trends Inf. Retr., 3(4):333–389, April
arXiv 2002
-
[15]
Alex Mallen, Akari Asai, Victor Zhong, Rajarshi Das, Daniel Khashabi, and Hannaneh Hajishirzi
URL https://arxiv.org/abs/2004.04906. Alex Mallen, Akari Asai, Victor Zhong, Rajarshi Das, Daniel Khashabi, and Hannaneh Hajishirzi. When not to trust language models: Investigating effectiveness of parametric and non-parametric memories. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki, editors, Proceedings of the 61st Annual Meeting of the Associa...
arXiv 2004
-
[16]
doi:10.18653/v1/2023.acl-long.546
Association for Computational Linguistics. doi:10.18653/v1/2023.acl-long.546. URL https://aclanthology.org/2023. acl-long.546. Nicholas Carlini, Florian Tramèr, Eric Wallace, Matthew Jagielski, Ariel Herbert-V oss, Katherine Lee, Adam Roberts, Tom Brown, Dawn Song, Úlfar Erlingsson, Alina Oprea, and Colin Raffel. Extracting training data from large lan- g...
-
[17]
doi:10.18653/v1/2021.eacl-main.74
Association for Computational Linguistics. doi:10.18653/v1/2021.eacl-main.74. URL https:// aclanthology.org/2021.eacl-main.74. Sebastian Borgeaud, Arthur Mensch, Jordan Hoffmann, Trevor Cai, Eliza Rutherford, Katie Millican, George van den Driessche, Jean-Baptiste Lespiau, Bogdan Damoc, Aidan Clark, Diego de Las Casas, Aurelia Guy, Jacob Menick, Roman Rin...
Show all 23 references
-
[18]
URL https://arxiv.org/abs/ 2112.04426. OpenAI. Text-embedding-ada-002,
-
[19]
Accessed: 2024-12-04
URL https://platform.openai.com/docs/guides/embeddings. Accessed: 2024-12-04. Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-V oss, Gretc...
2024
-
[20]
URL https://arxiv.org/abs/2005.14165. OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Baltescu,...
2005 arXiv
-
[21]
Jiasheng Gu, Hongyu Zhao, Hanzi Xu, Liangyu Nie, Hongyuan Mei, and Wenpeng Yin
URL https://arxiv.org/abs/2303.08774. Jiasheng Gu, Hongyu Zhao, Hanzi Xu, Liangyu Nie, Hongyuan Mei, and Wenpeng Yin. Robustness of learning from task instructions. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki, editors, Findings of the Association for Computational L...
2023 arXiv
-
[22]
doi:10.18653/v1/2023.findings-acl.875
Association for Computational Linguistics. doi:10.18653/v1/2023.findings-acl.875. URL https://aclanthology.org/2023. findings-acl.875. Bo Liu, Yuqian Jiang, Xiaohan Zhang, Qiang Liu, Shiqi Zhang, Joydeep Biswas, and Peter Stone. Llm+p: Empowering large language models with opt...
2023 doi
-
[23]
Need to improve my data analysis skills
URL https://arxiv.org/abs/2304.11477. A Similarity Rank Experiments We analyze recommendation likelihoods and similarity rank distributions across different query types. For each query, we plot: (1) the distribution of similarity ranks in recommendations over 10 trials, and (2...
-
[1998]
URL https://doi.org/10
doi:10.1080/0142569980190401. URL https://doi.org/10. 1080/0142569980190401. N D Lynn and A W R Emanuel. A review on recommender systems for course selection in higher education. IOP Con- ference Series: Materials Science and Engineering, 1098(3):032039,
-
[2009]
doi:10.1561/1500000019
ISSN 1554-0669. doi:10.1561/1500000019. URL https://doi.org/10.1561/ 1500000019. Vladimir Karpukhin, Barlas O˘guz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen tau Yih. Dense passage retrieval for open-domain question answering,
-
[2017]
Kathleen Lynch and Claire O’riordan
doi:10.1146/annurev-soc-060116-053622. Kathleen Lynch and Claire O’riordan. Inequality in higher education: a study of class barriers. British Journal of Sociology of Education , 19(4):445–478,
-
[2019]
URL http: //dx.doi.org/10.14569/IJACSA.2019.0101122
doi:10.14569/IJACSA.2019.0101122. URL http: //dx.doi.org/10.14569/IJACSA.2019.0101122. Zachary A. Pardos, Zihao Fan, and Weijie Jiang. Connectionist recommendation in the wild: On the utility and scrutability of neural networks for personalized course guidance,
2019
-
[2020]
ISBN 9781450377126
Association for Computing Machinery. ISBN 9781450377126. doi:10.1145/3375462.3375524. URL https://doi.org/10.1145/3375462.3375524. Shijie Geng, Shuchang Liu, Zuohui Fu, Yingqiang Ge, and Yongfeng Zhang. Recommendation as language processing (rlp): A unified pretrain, personali...
-
[2021]
URL https://dx.doi.org/10.1088/1757-899X/1098/3/032039
doi:10.1088/1757-899X/1098/3/032039. URL https://dx.doi.org/10.1088/1757-899X/1098/3/032039. Yiu-Kai Ng and Jane Linn. Crsrecs: A personalized course recommendation system for college students. In 2017 8th International Conference on Information, Intelligence, Systems & Applic...
2017 doi
-
[2022]
ISBN 9781450391580
Association for Computing Machinery. ISBN 9781450391580. doi:10.1145/3491140.3528270. URL https://doi.org/10.1145/3491140.3528270. Norazuwa Binti Salehudin, Hasan Kahtan, Mansoor Abdullateef Abdulgabber, and Hael Al-bashiri. A proposed course recommender model based on collabo...
-
[2023]
ISBN 9798400700255
Association for Computing Machinery. ISBN 9798400700255. doi:10.1145/3573051.3596177. URL https://doi.org/10.1145/3573051.3596177. Youjie Chen, Annie Fu, Jennifer Jia-Ling Lee, Ian Wilkie Tomasik, and René F. Kizilcec. Pathways: Exploring academic interests with historical cou...
-
[2024]
ISBN 9798400704314
Association for Computing Machinery. ISBN 9798400704314. doi:10.1145/3626772.3657883. URL https: //doi.org/10.1145/3626772.3657883. 12 From Interests to Insights A PREPRINT Zeyu Cui, Jianxin Ma, Chang Zhou, Jingren Zhou, and Hongxia Yang. M6-rec: Generative pretrained language...
Reviewed August 11, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.