REVIEW 3 major objections 4 minor 52 references
Designing and Evaluating an Educational Recommender System with Different Levels of User Control
T0 review · 3 major / 4 minor · reviewed 2026-08-10 · deepseek-v4-flash
Pith's one-line read User control in an educational recommender is associated with higher transparency, trust, and satisfaction, a 30-user study finds.
desk verdict A solid design contribution with an evaluation that cannot support the causal claims; the paper should be revised toward an exploratory design case. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The carrying mechanism is a set of interactive control widgets organized along the three standard levels of a recommender. At the input level, learners select "did not understand" concepts, adjust concept weights with sliders, and include or exclude concepts with checkboxes. At the process level, they choose among four recommendation algorithms via radio buttons and adjust ranking-factor weights with sliders whose effects appear in real-time progress bars. At the output level, they mark recommendations as helpful or not helpful, with a follow-up selection of which concepts were clarified, sort by similarity, recency, or views, and save items for later. This three-level control design is evaluated with a post-task questionnaire based on a standard user-centric evaluation framework, and the paper's statistical claims rest on Pearson correlations with bootstrap confidence intervals.
What would settle it
A controlled experiment that runs the same educational recommender with the control widgets active for one group and hidden for another, measuring the same questionnaire items, would falsify the causal claim if the control group shows no significantly higher transparency, trust, or satisfaction scores.
Extended reading notes
Core claim
The central discovery is that user control at all three levels of a recommender, input, process, and output, is associated with positive user-perceived benefits in an educational setting, and specifically that user control strongly correlates with transparency, moderately with trust, and moderately with satisfaction. In the same data, transparency moderately correlates with satisfaction, trust strongly correlates with satisfaction, but transparency and trust are less correlated with each other. The authors interpret this as evidence for "transparency through controllability": users understand why items are recommended because they can shape the profile, algorithm, and ranking themselves. They also argue that because transparency and trust move somewhat independently, evaluations of interactive educational recommenders should treat them as separate constructs.
Load-bearing premise
The study assumes that a single post-task questionnaire from 30 users, with no baseline system and no check that participants actually understood or used the control features, is enough to attribute the positive ratings and correlations to the user control design.
Editorial extensions
If this is right
- Designers of educational recommender systems can treat controllability as a practical route to transparency, since control and transparency showed the strongest correlation in the study.
- Because trust strongly correlates with satisfaction, improving either one is likely to carry the other upward.
- Transparency and trust should be measured as separate evaluation dimensions in interactive educational recommenders, since they correlated only weakly with each other.
- Providing control at all three levels, input, process, and output, is feasible inside a MOOC platform and was rated positively by learners with varied backgrounds.
- The authors' account implies a trade-off: too much control can raise cognitive load, which may explain why transparency and trust do not always move together.
Reading between the lines
- A natural next test, not run in this paper, would be to compare the three control levels separately to see whether input, process, or output control drives most of the transparency gain.
- If transparency through controllability is real, adding explanation text alongside the control widgets should push trust up further; the authors themselves hypothesize this as transparency through explanation.
- A baseline condition without any control widgets would be needed to separate the effect of control from a general novelty or interface-quality effect; the current design has no such comparison.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper presents the design of an interactive educational recommender system (ERS) built into the MOOC platform CourseMapper, with user control at three levels: input (selecting and weighting misunderstood concepts), process (choosing a recommendation algorithm and ranking weights), and output (sorting, saving, and giving feedback on recommendations). The authors report a single-group online user study (N=30) in which participants performed guided tasks and then completed a post-task questionnaire based on the ResQue framework. The paper claims that user control has a positive impact on users' perceptions, that user control correlates strongly with transparency and moderately with trust and satisfaction, and that transparency and trust each correlate with satisfaction but less with each other.
Significance. If the causal claims were supported, the paper would make a useful contribution by systematically combining input-, process-, and output-level control in an educational recommender and jointly evaluating transparency, trust, and satisfaction. The UI design is informed by a literature review and iterative prototyping, and the evaluation uses a recognized framework (ResQue) with bootstrap confidence intervals and adjusted p-values. However, the evaluation design is a one-arm post-test study with no baseline or manipulation check, so the central claim that user control causes transparency, trust, or satisfaction is not supported by the data. The paper is nevertheless informative as a descriptive account of user perceptions of a control-rich ERS and as a design exploration, which may be of interest to practitioners.
major comments (3)
- [Section 4.1 and RQ1 (Section 1)] The study uses a single condition: all 30 participants interacted with the full ERS containing all user-control features. There is no control condition without user control, no within-subjects manipulation, and no manipulation check. Consequently, RQ1 ('How does complementing an ERS with user control impact users' perceptions of the ERS?') cannot be answered from these data. The high mean scores in Table 2 and Figure 5 could be due to the recommender's underlying quality, the guided task structure, the demo video, or demand characteristics, rather than to the presence of user-control features. The causal language in the abstract and in Section 5.1 ('user control over the ERS can lead to relevant and novel recommendations') is therefore unsupported.
- [Section 5.2 and Figure 6] RQ2 asks about the 'effects of user control' on transparency, trust, and satisfaction, and Section 5.2 states that user control 'leads to increased transparency' and that 'user control over the RS positively influenced their satisfaction.' These claims rest on Pearson correlations between a self-reported two-item control measure and other self-reported measures collected in the same post-task questionnaire. This is a common-method correlation analysis, which cannot establish causal effects. The absence of any behavioral measure of control use or a manipulation check further weakens the inference. The paper itself acknowledges in Section 6 that 'we plan to conduct a more comprehensive user study,' which is consistent with the present design being insufficient for the causal claims made.
- [Section 5.3] The paper explains the relatively low correlation between transparency and trust with speculative mechanisms such as cognitive load from too much control and the absence of explanations. However, no cognitive-load measures, open-ended responses, or explanation manipulations were collected, so these explanations are untestable in the current design. The text should be clearly framed as hypotheses for future work rather than as findings from this study.
minor comments (4)
- [Section 4.1] The sentence 'Most of the participants were familiar with the use of RSs (n=24, 63%)' contains an inconsistency: 24 out of 30 is 80%, not 63%; the 63% figure applies to the following phrase about interacting with RSs (n=19, 63%).
- [Table 2] Several constructs (e.g., Interface Adequacy, Perceived Usefulness, Use Intentions) are measured with multiple items, but only a single mean and SD are reported for the whole construct. Reporting item-level statistics or reliability coefficients (e.g., Cronbach's alpha) would improve interpretability.
- [Figure 6] The exact Pearson correlation coefficients and adjusted p-values are not stated in the text; the figure shows them visually, but numeric values in the text or a table would make the results more transparent and reproducible.
- [Section 4.1] There is a typo in the list of participant countries: '2 Chineese' should be '2 Chinese.'
Circularity Check
No significant circularity: the paper reports an empirical single-arm user study, and its central claims are correlational findings from its own data rather than constructed derivations.
full rationale
The paper does not attempt a formal derivation chain, so there is no equation or construction step in which an output is equivalent to an input by definition. The central claims ('user control strongly correlates with transparency and moderately correlates with trust and satisfaction') are Pearson correlations computed from the post-task questionnaire items listed in Table 2. Perceived control is measured with its own two items, and transparency, trust, and satisfaction are measured with separate items from the ResQue framework; no fitted parameter is renamed as a prediction, and no construct is defined in terms of another construct. The causal language in Section 5.2 ('user control with the ERS leads to an increased transparency') is not supported by the single-condition, no-baseline design, and the correlations may reflect common-method variance because all measures come from the same questionnaire after the same interaction. However, that is a methodological validity threat, not circular reasoning. The self-citations in the paper are also not load-bearing: [32] describes the CourseMapper platform, [33] describes the underlying recommendation algorithms, and [49,50,52] are related empirical and conceptual works; none of these is invoked as a uniqueness theorem or as the source of the reported correlations. The paper even acknowledges the need for a more comprehensive study in Section 6. Accordingly, the derivation is self-contained as an empirical study, and any circularity score should be low. A score of 1 reflects the presence of minor self-citations and the weak causal framing, without any construction-equivalent circular step.
Assumptions & free parameters
assumptions (4)
- ad hoc to paper A single-condition post-test design is sufficient to infer the impact of user control.
- domain assumption Self-reported Likert items from ResQue measure the intended constructs validly and reliably.
- domain assumption Participants used the control features as intended and answered truthfully.
- standard math Pearson correlation and bootstrap confidence intervals are appropriate for five-point Likert responses with N=30.
Cite this review
Pith. "Pith review of Designing and Evaluating an Educational Recommender System with Different Levels of User Control." pith.science (2026). https://pith.science/paper/XVBOBJOK
@misc{pith2026250112894,
author = {Pith},
title = {Pith review of: Designing and Evaluating an Educational Recommender System with Different Levels of User Control},
year = {2026},
howpublished = {\url{https://pith.science/paper/XVBOBJOK}},
note = {Machine review of arXiv:2501.12894}
}
read the original abstract
Educational recommender systems (ERSs) play a crucial role in personalizing learning experiences and enhancing educational outcomes by providing recommendations of personalized resources and activities to learners, tailored to their individual learning needs. However, their effectiveness is often diminished by insufficient user control and limited transparency. To address these challenges, in this paper, we present the systematic design and evaluation of an interactive ERS, in which we introduce different levels of user control. Concretely, we introduce user control around the input (i.e., user profile), process (i.e., recommendation algorithm), and output (i.e., recommendations) of the ERS. To evaluate our system, we conducted an online user study (N=30) to explore the impact of user control on users' perceptions of the ERS in terms of several important user-centric aspects. Moreover, we investigated the effects of user control on multiple recommendation goals, namely transparency, trust, and satisfaction, as well as the interactions between these goals. Our results demonstrate the positive impact of user control on user perceived benefits of the ERS. Moreover, our study shows that user control strongly correlates with transparency and moderately correlates with trust and satisfaction. In terms of interaction between these goals, our results reveal that transparency moderately correlates and trust strongly correlates with satisfaction. Whereas, transparency and trust stand out as less correlated with each other.
Figures
Figures from the paper (3 more)
Reference graph
Works this paper leans on
-
[1]
N. Manouselis, H. Drachsler, R. Vuorikari, H. Hummel, R. Koper, Recommender systems in technology enhanced learning, Recommender systems handbook (2011) 387–415
work page 2011
-
[2]
S. S. Khanal, P. Prasad, A. Alsadoon, A. Maag, A systematic review: machine learning based recommendation systems for e-learning, Education and Information Technologies (2020)
work page 2020
-
[3]
S. Valtolina, R. A. Matamoros, F. Epifania, Design of a conversational recommender system in education, User modeling and user-adapted interaction (2024) 1–29
work page 2024
-
[4]
H. Chau, J. Barria-Pineda, P. Brusilovsky, Learning content recommender system for instructors of programming courses, in: Artificial Intelligence in Education: 19th International Conference, AIED 2018, London, UK, June 27–30, 2018, Proceedings, Part II 19, Springer, 2018, pp. 47–51
work page 2018
-
[5]
F. Bousbahi, H. Chorfi, Mooc-rec: A case based recommender system for moocs, Procedia - Social and Behavioral Sciences 195 (2015) 1813–1822. World Conference on Technology, Innovation and Entrepreneurship
work page 2015
-
[6]
O. C. Santos, M. Saneiro, J. G. Boticario, M. C. Rodriguez-Sanchez, Toward interactive context- aware affective educational recommendations in computer-assisted language learning, New Review of Hypermedia and Multimedia 22 (2016) 27–57
work page 2016
-
[7]
Y. Jin, N. Tintarev, K. Verbert, Effects of personal characteristics on music recommender sys- tems with different levels of controllability, in: Proceedings of the 12th ACM Conference on Recommender Systems, 2018, pp. 13–21
work page 2018
-
[8]
C. He, D. Parra, K. Verbert, Interactive recommender systems: A survey of the state of the art and future research challenges and opportunities, Expert Systems with Applications 56 (2016) 9–27
work page 2016
Show all 52 references
-
[9]
Jugovac, D
M. Jugovac, D. Jannach, Interacting with recommenders—overview and research directions, ACM Transactions on Interactive Intelligent Systems (TiiS) 7 (2017) 1–46
2017
-
[10]
Jannach, S
D. Jannach, S. Naveed, M. Jugovac, User control in recommender systems: Overview and interaction challenges, in: E-Commerce and Web Technologies: 17th International Conference, EC-Web 2016, Porto, Portugal, September 5-8, 2016, Revised Selected Papers 17, Springer, 2017, pp. 21–33
2016
-
[11]
Harambam, D
J. Harambam, D. Bountouridis, M. Makhortykh, J. van Hoboken, Designing for the better by taking users into account: a qualitative evaluation of user control mechanisms in (news) recommender systems, in: Proceedings of 13th ACM Conference on Recommender Systems, RecSys ’19, 2019
2019
-
[12]
B. P. Knijnenburg, S. Bostandjiev, J. O’Donovan, A. Kobsa, Inspectability and control in social recommenders, in: Proceedings of the sixth ACM conference on Recommender systems, 2012
2012
-
[13]
Barria-Pineda, P
J. Barria-Pineda, P. Brusilovsky, Explaining educational recommendations through a concept-level knowledge visualization, in: Companion Proceedings of the 24th International Conference on Intelligent User Interfaces, 2019, pp. 103–104
2019
-
[14]
Schaffer, T
J. Schaffer, T. Hollerer, J. O’Donovan, Hypothetical recommendation: A study of interactive profile manipulation behavior for recommender systems, in: 28th international flairs conference, 2015
2015
-
[15]
O’Donovan, B
J. O’Donovan, B. Smyth, B. Gretarsson, S. Bostandjiev, T. Höllerer, Peerchooser: visual interactive recommendation, in: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI ’08, Association for Computing Machinery, NY, USA, 2008, p. 1085–1088
2008
-
[16]
Bostandjiev, J
S. Bostandjiev, J. O’Donovan, T. Höllerer, Tasteweights: a visual interactive hybrid recommender system, in: Proceedings of the 6th ACM Conference on Recommender Systems, Association for Computing Machinery, New York, USA, 2012
2012
-
[17]
F. M. Harper, F. Xu, H. Kaur, K. Condiff, S. Chang, L. Terveen, Putting users in control of their recommendations, in: Proceedings of the 9th ACM Conference on Recommender Systems, RecSys ’15, Association for Computing Machinery, New York, NY, USA, 2015, p. 3–10
2015
-
[18]
Gretarsson, J
B. Gretarsson, J. O’Donovan, S. Bostandjiev, C. Hall, T. Höllerer, Smallworlds: Visualizing social recommendations, Computer Graphics Forum 29 (2010) 833–842
2010
-
[19]
C.-H. Tsai, P. Brusilovsky, Providing control and transparency in a social recommender system for academic conferences, in: Proceedings of the 25th conference on user modeling, adaptation and personalization, 2017, pp. 313–317
2017
-
[20]
Kangasrääsiö, D
A. Kangasrääsiö, D. Glowacka, S. Kaski, Improving controllability and predictability of interactive recommendation interfaces for exploratory search, in: Proceedings of the 20th International Conference on Intelligent User Interfaces, IUI ’15, 2015
2015
-
[21]
Bruns, A
S. Bruns, A. C. Valdez, C. Greven, M. Ziefle, U. Schroeder, What should i read next? a personalized visual publication recommender system, in: International Conference on Human Interface and the Management of Information, Springer, 2015, pp. 89–100
2015
-
[22]
S. Zhao, M. X. Zhou, Q. Yuan, X. Zhang, R. Zheng, Who is talking about what: social map-based recommendation for content-centric social websites, in: Proceedings of the 4th ACM Conference on Recommender Systems, New York, USA, 2010
2010
-
[23]
Tintarev, B
N. Tintarev, B. Kang, T. Höllerer, J. O’Donovan, Inspection mechanisms for community-based content discovery in microblogs., in: IntRS@ RecSys, 2015, pp. 21–28
2015
-
[24]
Y. Chen, P. Pu, Cofeel: Using emotions for social interaction in group recommender systems (2012)
2012
-
[25]
C.-H. Tsai, P. Brusilovsky, The effects of controllability and explainability in a social recommender system, User Modeling and User-Adapted Interaction 31 (2021) 591–627
2021
-
[26]
Loepp, T
B. Loepp, T. Hussein, J. Ziegler, Choice-based preference elicitation for collaborative filtering recommender systems, in: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, 2014, pp. 3085–3094
2014
-
[27]
Parra, P
D. Parra, P. Brusilovsky, C. Trattner, See what you want to see: visual user-driven approach for hybrid recommendation, in: Proceedings of the 19th International Conference on Intelligent User Interfaces, IUI ’14, Association for Computing Machinery, New York, USA, 2014, p. 235–240
2014
-
[28]
Y. Jin, K. Seipp, E. Duval, K. Verbert, Go with the flow: effects of transparency and user control on targeted advertising using flow charts, in: Proceedings of the international working conference on advanced visual interfaces, 2016, pp. 68–75
2016
-
[29]
Bostandjiev, J
S. Bostandjiev, J. O’Donovan, T. Höllerer, Linkedvis: Exploring social and semantic career recom- mendations, 2013, pp. 107–116. doi: 10.1145/2449396.2449412
2013
-
[30]
Verbert, D
K. Verbert, D. Parra, P. Brusilovsky, E. Duval, Visualizing recommendations to support explo- ration, transparency and controllability, in: Proceedings of the 2013 International Conference on Intelligent User Interfaces, IUI ’13, ACM, New York, NY, USA, 2013
2013
-
[31]
Tintarev, J
N. Tintarev, J. Masthoff, Explaining recommendations: Design and evaluation, in: Recommender systems handbook, Springer, 2015, pp. 353–382
2015
-
[32]
Q. U. Ain, M. A. Chatti, S. Joarder, I. Nassif, B. S. Wobiwo Teda, M. Guesmi, R. Alatrash, Learning channels to support interaction and collaboration in coursemapper, in: Proceedings of the 14th International Conference on Education Technology and Computers, ICETC ’22, 2023
2023
-
[33]
Q. U. Ain, M. A. Chatti, P. A. Meteng Kamdem, R. Alatrash, S. Joarder, C. Siepmann, Learner modeling and recommendation of learning resources using personal knowledge graphs, in: Proceedings of the 14th Learning Analytics and Knowledge Conference, 2024, pp. 273–283
2024
-
[34]
Fotopoulou, A
E. Fotopoulou, A. Zafeiropoulos, M. Feidakis, D. Metafas, S. Papavassiliou, An interactive rec- ommender system based on reinforcement learning for improving emotional competences in educational groups, in: International Conference on Intelligent Tutoring Systems, Springer, 2020
2020
-
[35]
Bustos López, G
M. Bustos López, G. Alor-Hernández, J. L. Sánchez-Cervantes, M. A. Paredes-Valverde, M. d. P. Salas-Zárate, Edurecomsys: an educational resource recommender system based on collaborative filtering and emotion detection, Interacting with Computers 32 (2020) 407–432
2020
-
[36]
F. L. da Silva, B. K. Slodkowski, K. K. A. da Silva, S. C. Cazella, A systematic literature review on educational recommender systems for teaching and learning: research trends, limitations and opportunities, Education and Information Technologies 28 (2023) 3289–3328
2023
-
[37]
Zapata, V
A. Zapata, V. H. Menéndez, M. E. Prieto, C. Romero, Evaluation and selection of group recom- mendation strategies for collaborative searching of learning objects, International Journal of Human-Computer Studies 76 (2015) 22–39
2015
-
[38]
S. Abdi, H. Khosravi, S. Sadiq, D. Gasevic, Complementing educational recommender systems with open learner models, in: Proceedings of the tenth international conference on learning analytics & knowledge, 2020, pp. 360–365
2020
-
[39]
Vlachos, D
M. Vlachos, D. Svonava, Graph embeddings for movie visualization and recommendation, in: First International Workshop on Recommendation Technologies for Lifestyle Change, 2012
2012
-
[40]
J. B. Schafer, J. A. Konstan, J. Riedl, Meta-recommendation systems: user-controlled integration of diverse recommendations, in: Proceedings of the Eleventh International Conference on Information and Knowledge Management, Association for Computing Machinery, NY, USA, 2002
2002
-
[41]
Saito, T
Y. Saito, T. Itoh, Musicube: a visual music recommendation system featuring interactive evolution- ary computing, in: Proceedings of the 2011 Visual Information Communication - International Symposium, VINCI ’11, Association for Computing Machinery, New York, USA, 2011
2011
-
[42]
D. Wong, S. Faridani, E. Bitton, B. Hartmann, K. Goldberg, The diversity donut: enabling participant control over the diversity of recommended responses, in: CHI’11 Extended Abstracts on Human Factors in Computing Systems, 2011, pp. 1471–1476
2011
-
[43]
M. D. Ekstrand, D. Kluver, F. M. Harper, J. A. Konstan, Letting users choose recommender algorithms: An experimental study, Proceedings of the 9th ACM Conference on Recommender Systems (2015)
2015
-
[44]
J. Ooge, L. Dereu, K. Verbert, Steering recommendations and visualising its impact: Effects on adolescents’ trust in e-learning platforms, in: Proceedings of the 28th International Conference on Intelligent User Interfaces, IUI ’23, 2023, p. 156–170
2023
-
[45]
Jannach, M
D. Jannach, M. Jugovac, I. Nunes, Explanations and user control in recommender systems, in: Proceedings of the 23rd International Workshop on Personalization and Recommendation on the Web and Beyond, 2019, pp. 31–31
2019
-
[46]
P. Pu, L. Chen, R. Hu, A user-centric evaluation framework for recommender systems, in: ACM Conference on Recommender Systems, 2011
2011
-
[47]
Tintarev, J
N. Tintarev, J. Masthoff, A survey of explanations in recommender systems, in: 2007 IEEE 23rd International Conference on Data Engineering Workshop, 2007, pp. 801–810
2007
-
[48]
Balog, F
K. Balog, F. Radlinski, Measuring recommendation explanation quality: The conflicting goals of explanations, in: Proceedings of the 43rd international ACM SIGIR conference on research and development in information retrieval, 2020, pp. 329–338
2020
-
[49]
Guesmi, M
M. Guesmi, M. A. Chatti, S. Joarder, Q. U. Ain, R. Alatrash, C. Siepmann, T. Vahidi, Interactive explanation with varying level of details in an explainable scientific literature recommender system, International Journal of Human–Computer Interaction (2023) 1–22
2023
-
[50]
Guesmi, M
M. Guesmi, M. A. Chatti, S. Joarder, Q. U. Ain, C. Siepmann, H. Ghanbarzadeh, R. Alatrash, Justifi- cation vs. transparency: Why and how visual explanations in a scientific literature recommender system, Information 14 (2023) 401
2023
-
[51]
Gedikli, D
F. Gedikli, D. Jannach, M. Ge, How should i explain? a comparison of different explanation types for recommender systems, International Journal of Human-Computer Studies 72 (2014) 367–382
2014
-
[52]
Siepmann, M
C. Siepmann, M. A. Chatti, Trust and transparency in recommender systems, arXiv preprint arXiv:2304.08094 (2023)
2023 arXiv
Reviewed August 10, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.