{"id":"04974579-40db-401f-86fa-9ce8dceeee3c","arxiv_id":"2605.01400","paper_version":1,"verdict":"UNVERDICTED","confidence":"LOW","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Input-level control over user profiles in an educational recommender system suffices to boost perceived control and positively shapes transparency, trust, satisfaction, and perceived quality, while further controls mainly reinforce impressions.","lead":"Researchers tested an interactive educational recommender system in a MOOC platform where users could control their profile, the algorithm, and the recommendations at different levels. A study with 184 participants found that profile control alone is enough to improve how users feel about the system, with extra controls adding little.","discovery_kind":"extension","skeptic_critique":{"model":"grok-4.3","headline":"Between-subjects lab study with only self-reported perceptions leaves open whether input-control sufficiency holds in real, sustained MOOC usage.","rationale":"The reader’s weakest assumption correctly isolates the methodological gap that most directly threatens external validity of the central empirical claim. Full-text inspection confirms the study is a standard short-term between-subjects perception experiment with no objective usage data or retention measures, so the concern lands without needing further invention.","tokens_in":1811,"tokens_out":324,"duration_ms":28848,"concrete_test":"Re-run the study as a two-week longitudinal within-subjects or mixed design on the live CourseMapper platform; log actual profile edits, recommendation views/accepts, and re-administer the same scales at day 14. If the input-only condition no longer differs from higher-control conditions on behavioral metrics or if perceived-control differences shrink below significance, the sufficiency claim weakens.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The headline result—that building/refining the user profile is sufficient and that only perceived control shows significant differences—rests on immediate post-task Likert ratings collected after a single, scripted interaction in a controlled setting (N=184). No behavioral logs (click-through rates, profile edits retained over time, recommendation acceptance) or longitudinal follow-up are reported, so the claim that additional process/output controls “mainly reinforce” impressions cannot be distinguished from demand characteristics or novelty effects. The between-subjects assignment also leaves individual differences in prior MOOC experience unaccounted for as potential confounds on the transparency/trust/satisfaction scales.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.3","summary":"The paper reports a between-subjects user study (N=184) on an interactive educational recommender system (ERS) embedded in the CourseMapper MOOC platform. Participants received varying levels of control over the input (user profile construction and refinement), process (recommendation algorithm), and output (recommendations). The central claims are that input control alone is sufficient to produce positive perceptions of the ERS, that additional process and output controls mainly reinforce those impressions, that perceived control is the only recommendation goal showing statistically significant differences across conditions (with input control exerting the strongest influence), and that the other goals (transparency, trust, satisfaction, perceived quality) are affected in distinct but interconnected ways.","tokens_in":1911,"tokens_out":616,"duration_ms":29184,"significance":"If the results hold under stronger validation, the work supplies actionable design guidance for ERSs by indicating that profile-building controls can deliver most of the perceptual benefits without the overhead of full algorithmic or output control. This could simplify interfaces in educational settings while still supporting transparency, trust, and satisfaction. The study is strengthened by its use of a deployed platform and a reasonably powered sample; it adds empirical data to the HCI/recommender-systems literature on control granularity in learning contexts.","major_comments":[{"comment":"The headline claim that input control is 'sufficient to promote positive perceptions' (abstract and results) rests exclusively on immediate post-task self-reported Likert ratings collected after a single scripted interaction. No behavioral logs (profile-edit counts retained, recommendation acceptance rates, click-through data) or longitudinal follow-up are reported, leaving open whether the sufficiency finding generalizes beyond lab demand characteristics or novelty effects.","section":"User Study and Results"},{"comment":"The assertion that 'perceived control is the only goal significantly affected' and that 'input control exert[s] the strongest influence' requires full statistical reporting (exact p-values, effect sizes, power analysis, and multiple-comparison corrections) to be load-bearing; the abstract supplies none, and the between-subjects design leaves prior MOOC experience unaccounted for as a potential confound on the transparency/trust/satisfaction scales.","section":"Results and Discussion"}],"minor_comments":[{"comment":"The abstract should include at least the key statistical outcomes (p-values, effect sizes) that support the significance claims.","section":null},{"comment":"Clarify the exact operationalization of the three control levels (e.g., which UI elements were enabled/disabled in each condition) and how the between-subjects assignment was randomized.","section":"Methodology"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is a reasonable fit for an HCI/educational-technology venue, but the absence of any behavioral or retention metrics weakens its claim to inform real MOOC practice; a revision that adds even modest log data or a short follow-up would substantially increase impact."},"author_rebuttal":{"model":"grok-4.3","summary":"We thank the referee for their constructive and detailed feedback on our manuscript. We address each major comment below, indicating where revisions will be made to strengthen the paper while maintaining the integrity of our study design and findings.","responses":[{"response":"We acknowledge that the study relies on immediate post-interaction self-reported Likert ratings without accompanying behavioral logs or longitudinal data. This design was chosen to isolate perceptual effects in a controlled between-subjects setup, consistent with common HCI evaluation practices for recommender systems. We agree that behavioral metrics and follow-up measures would provide additional robustness. In the revised manuscript, we will expand the limitations and future work sections to explicitly discuss the absence of behavioral data, potential novelty or demand effects, and the need for longitudinal validation. We will also moderate phrasing in the abstract and results to avoid overgeneralization while preserving the core contribution of the perceptual findings.","revision_made":"partial","referee_comment":"[User Study and Results] The headline claim that input control is 'sufficient to promote positive perceptions' (abstract and results) rests exclusively on immediate post-task self-reported Likert ratings collected after a single scripted interaction. No behavioral logs (profile-edit counts retained, recommendation acceptance rates, click-through data) or longitudinal follow-up are reported, leaving open whether the sufficiency finding generalizes beyond lab demand characteristics or novelty effects."},{"response":"We appreciate the call for complete statistical transparency. The revised manuscript will include exact p-values, effect sizes, a post-hoc power analysis, and details on multiple-comparison corrections. The abstract will be updated to summarize these statistics. On the potential confound of prior MOOC experience, we will report the distribution across conditions and, where relevant, include it as a covariate or discuss it explicitly as a limitation. Our randomization procedure aimed to balance such factors, but we will add this analysis to address the concern directly.","revision_made":"yes","referee_comment":"[Results and Discussion] The assertion that 'perceived control is the only goal significantly affected' and that 'input control exert[s] the strongest influence' requires full statistical reporting (exact p-values, effect sizes, power analysis, and multiple-comparison corrections) to be load-bearing; the abstract supplies none, and the between-subjects design leaves prior MOOC experience unaccounted for as a potential confound on the transparency/trust/satisfaction scales."}],"tokens_in":1508,"tokens_out":509,"duration_ms":30229,"standing_objections":[]},"desk_editor":{"model":"grok-4.3","letter":"The main result is that letting users build and refine their profile in this CourseMapper ERS is enough to lift ratings on transparency, trust, satisfaction, and quality, while extra controls on the algorithm or recommendations add little beyond reinforcement. Only perceived control showed clear differences across conditions, and input control had the strongest effect. The other goals moved in connected but smaller ways.","headline":"Profile control alone drives most of the perception gains here, but one-shot self-reports after a scripted task limit how much we can trust the real-world claims.","tokens_in":2422,"tokens_out":150,"would_cite":false,"duration_ms":28686,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.3","headline":"Enabling users to build and refine their profiles in an educational recommender system is sufficient to promote positive perceptions of control, transparency, trust, satisfaction, and quality.","keywords":["educational recommender systems","user control","perceived control","transparency","trust","satisfaction","perceived quality","user study"],"falsifier":"A follow-up study that finds no difference in actual engagement or satisfaction between users limited to profile control and users given full control over multiple weeks of real course activity would undermine the claim that input control alone is sufficient.","tokens_in":2696,"feed_emoji":"📚","tokens_out":708,"duration_ms":34219,"temperature":0.7,"pith_summary":"This paper tests the effects of giving learners different amounts of control over an educational recommender system. Users could adjust their own profiles as input, tweak the recommendation process, or edit the final output list. The results indicate that profile control by itself drives most of the gains in how users view the system, while extra controls over the algorithm and recommendations add only modest reinforcement. A reader would care because these systems aim to personalize learning paths in online courses, and knowing the minimal effective level of control could simplify interfaces without sacrificing benefits. If the pattern holds, builders can concentrate on clear profile tools rather than full algorithmic interfaces.","feed_headline":"Profile control alone improves perceptions in learning recommenders","feed_subtitle":"Input control suffices for trust and satisfaction while further options mainly reinforce impressions.","key_machinery":"The staged control options over input (user profile construction), process (recommendation algorithm), and output (displayed recommendations).","core_discovery":"The work shows that in an interactive educational recommender system, allowing users to build and refine their profiles is sufficient to generate positive perceptions of the recommendation goals, while providing additional control over the algorithm and recommendations mainly reinforces those impressions. Perceived control is the only goal significantly affected by the different levels of control, and input control produces the strongest effect on it. The levels of control influence transparency, trust, satisfaction, and perceived quality in distinct yet interconnected patterns, and user control in general positively shapes these perceptions to varying degrees.","pith_inferences":["Interface designers could prioritize simple profile-editing features over complex algorithm controls to achieve most perception benefits.","The observed perception gains might support higher completion rates in online courses if profile control reduces the time users spend searching for suitable materials.","Similar control patterns could be tested in non-educational recommender systems to check whether input control remains the dominant stage.","Longer deployments might reveal whether the reinforced impressions translate into sustained use or measurable learning improvements."],"forward_implications":["Input control alone promotes positive perceptions of the educational recommender system.","Additional process and output controls mainly reinforce existing positive impressions rather than creating substantial new effects.","Perceived control is the only recommendation goal significantly affected by control levels, with input control exerting the strongest influence.","Different control levels affect transparency, trust, satisfaction, and perceived quality in distinct but interconnected ways.","User control overall positively shapes transparency, trust, satisfaction, and perceived quality, though to varying extents."],"fun_headline_variants":["Profile refinement suffices for positive ERS perceptions","Extra controls reinforce input impressions in recommenders","Input control strongest on perceived control in ERS","Control levels distinctly affect trust and satisfaction in ERS"],"cache_read_input_tokens":2112,"weakest_assumption_plain":"Self-reported perceptions collected in a short-term between-subjects experiment reflect the real effects of control levels during extended everyday use across different educational platforms.","fun_headline_variants_meta":{"raw":{"variants":["Profile refinement suffices for positive ERS perceptions","Extra controls reinforce input impressions in recommenders","Input control strongest on perceived control in ERS","Control levels distinctly affect trust and satisfaction in ERS"]},"model":"grok-4.3","cost_usd":0.006599,"raw_usage":{"total_tokens":3029,"prompt_tokens":725,"num_sources_used":0,"completion_tokens":56,"cost_in_usd_ticks":65990500,"prompt_tokens_details":{"text_tokens":725,"audio_tokens":0,"image_tokens":0,"cached_tokens":64},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":2248,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":725,"tokens_out":56,"duration_ms":19218,"temperature":1.0,"reasoning_tokens":2248,"cache_read_input_tokens":64,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-05-09T18:18:36.389092+00:00","model_set":{"reader":"grok-4.3"},"falsifier":"A follow-up study that finds no difference in actual engagement or satisfaction between users limited to profile control and users given full control over multiple weeks of real course activity would undermine the claim that input control alone is sufficient.","supporting_citations":[],"review_version":1}