{"id":"ce8262f5-72cb-466d-bd9c-845da9ab30d6","arxiv_id":"2509.06126","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":3.0,"correctness_risk":"high","formal_verification":"none","parameter_count":1,"one_line_summary":"A case study reports higher pass rates and satisfaction in a gamified biomedical engineering course, but the evidence is weakened by a non-equivalent historical control and a grading formula that ties the main outcome to the gamified components.","lead":"A teaching team added game-like activities (online quizzes, leaderboards, and a startup pitch contest) to a biomedical engineering course and compared grades and survey answers with an earlier class that had no games. The paper argues the games raised pass rates and satisfaction, but the comparison group is from years before with no controls, so the causal claim is not well supported.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The gamification effect is unidentified: the 2017-18 control is separated by a two-year gap and the pandemic, with no adjustment for cohort, grading, or course changes, so the 100% vs 70.6% pass-rate comparison does not support a causal claim.","rationale":"The reader identifies the comparability of the 2017-18 control as the weakest assumption. I agree; this is the single most load-bearing issue because the entire causal claim depends on it. If the control is not exchangeable, the pass-rate difference (100% vs 70.6%) and the mediation results are both uninterpretable as effects of gamification. The two-year gap in which the course was not offered, the pandemic-era grading context, and the tripled enrollment all make exchangeability implausible. I considered whether the internal inconsistency in the mediation coefficients (Section 4.4 vs Table 6) is even more fundamental; it is serious but secondary because it affects the theoretical mechanism, not the primary outcome comparison. A concrete check—comparing grade trends in contemporaneous non-gamified courses—would settle the causal question. Since the paper currently provides no such control, the reader's REJECT verdict is unchanged.","tokens_in":20319,"tokens_out":5073,"duration_ms":53322,"concrete_test":"Request the university's anonymized grade records for all Biomedical Engineering courses from 2016-17 through 2022-23. Identify a non-gamified elective (or two) with similar enrollment and content continuity across these years and run a difference-in-differences regression: pass rate (or grade) on gamification indicator, year fixed effects, course fixed effects, and an interaction term. If the comparison courses show a comparable pass-rate increase (e.g., from ~70% to ~100%) over the same period, the gamification effect is confounded by year-level trends; if they remain flat, the historical-control comparison gains credibility.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim—that adding Nanogames and NanoTechStart improved learning outcomes—rests entirely on the pass-rate comparison in Table 1 (100% vs 70.6%) and the mediation analysis in Tables 5-6. The control group is from 2017-18, but the course was not taught in 2018-19 and 2019-20 (Section 3.1.3). For the causal claim to hold, the 2017-18 cohort must be exchangeable with the 2020-23 cohorts on every factor that affects grades: instructor behavior, exam difficulty, grading standards, syllabus content, student ability/motivation, and external shocks. The paper provides no evidence on any of these. In particular, the 2020-23 period overlaps the COVID-19 pandemic, during which many institutions relaxed grading and pass thresholds; the simultaneous rise in enrollment (17 to 45) also signals a different selection pool. The statistical tests (Kruskal-Wallis, two-proportion z-test) are unadjusted and cannot distinguish gamification from a secular time trend or grading shift. Consequently, the headline causal inference is unsupported; the data are at best consistent with a descriptive improvement in a course redesign.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper reports on a gamified redesign of a Biomedical Engineering course at Universidad Carlos III de Madrid, adding two gamified activities (Nanogames and NanoTechStart) to continuous evaluation. It compares a 2017-2018 control cohort (N=17) with 2020-2023 gamified cohorts (total N=45), reporting higher pass rates (100% vs 70.6%), higher final scores, positive student surveys, and a moderated-mediation analysis grounded in Landers' Theory of Gamified Learning. The central claim is that the gamification intervention improved learning outcomes. The manuscript also includes faculty survey results and a detailed description of the course design.","tokens_in":20615,"tokens_out":4948,"duration_ms":53437,"significance":"If the causal claim were credible, the paper would provide a useful, theory-guided example of gamification in a STEM course, with detailed descriptions of two transferable activities and associated survey instruments. The use of Landers' framework is a strength, as is the transparent reporting of activity structures and scoring. However, the design is a single historical-control before-after comparison with no adjustment for major confounders, the mediation analysis is partly tautological because the course total is defined as 60% continuous evaluation plus 40% final exam, and the manuscript contains several internal inconsistencies (population description, sample sizes, and regression coefficients). These issues undermine the central quantitative claims, reducing the paper's contribution to a descriptive case study.","major_comments":[{"comment":"The causal claim that gamification improved pass rates and final scores is unsupported by the design. The control group is a single cohort from 2017-2018, separated by a two-year gap during which the course was not taught (2018-2019 and 2019-2020) and overlapping the COVID-19 pandemic. Enrollment grew from 17 to 45, and the paper provides no evidence that instructor behavior, exam difficulty, grading standards, syllabus content, or student selection were stable across this period. The unadjusted p-values in Table 1 (e.g., pass-rate p=0.0033) cannot distinguish the intervention from secular trends or grading shifts. The 100% vs 70.6% pass-rate comparison does not support the paper's headline conclusion.","section":"§3.1.3, §4.1, Table 1"},{"comment":"The mediation analysis is partly tautological. Course Total (CT) is defined as 60% Continuous Evaluation (CE) plus 40% final exam. The regression of CT on CE therefore should yield a coefficient near 0.6; Table 5 reports β=0.58, which is precisely the grading weight. Consequently, Hypothesis 2 is a restatement of the grading formula, and the indirect path Gamification→CE→CT in Table 6 partially reflects the definition of CT rather than a psychological or pedagogical mechanism. The claim that the data support Hypothesis 1 is therefore overstated.","section":"§3.2.4, §4.4, Table 5"},{"comment":"The population is described inconsistently. The abstract says the intervention involved \"master's-level students in Biomedical Image Processing,\" while the main text (Section 3.1.3 and Section 3.2) describes a fourth-year Bachelor of Engineering course, \"Biomedical Applications of Nanotechnology.\" This mismatch is not a minor wording issue: it obscures the educational level and the specific course, and it is load-bearing for interpreting the results and their generalizability.","section":"Abstract vs. §3.1.3 and §3.2"},{"comment":"The mediation analysis reports N=41 in Table 4, but Section 3.1.3 states there were 45 participants in the gamified cohorts and 17 in the control group, for a total of 62. No explanation is given for the missing 21 observations. If the mediation model excludes some cohorts or some students, this must be stated and justified; as written, the discrepancy makes it impossible to assess the analysis's validity.","section":"§4.4, Table 4 vs. §3.1.3"},{"comment":"The reported regression coefficients are inconsistent. Table 5 reports a direct effect of Gamification→CT of β=0.28 (p=0.031) in the complete model, while Table 6 reports a conditional direct effect of 0.09 (p<0.05). The total effect in Table 6 is 0.19, but the sum of the direct and indirect effects is 0.09+0.10=0.19, which does not reconcile with the 0.28 coefficient in Table 5. Additionally, the standard error for Gamification→CE is reported as 0.02 with a coefficient of 0.54, which implies an implausibly large effect given the reported CE distribution (mean 0.89, SD 0.08). These internal inconsistencies undermine confidence in the mediation results.","section":"§4.4, Tables 5-6"}],"minor_comments":[{"comment":"The text states \"a p-value of 0.00003 indicating a significant difference in the pass rates,\" but Table 1 reports p=0.0033 for pass rates and <0.0001 for final scores. Please correct the reported value and clarify which test corresponds to which comparison.","section":"§4.1"},{"comment":"Typo: \"2017-2028\" should be \"2017-2018\" in the first paragraph.","section":"§4.2"},{"comment":"Typo: \"Nanogrames\" should be \"Nanogames\" in the sentence about unanimous approval.","section":"§4.3"},{"comment":"Typo: \"Continous evaluation\" appears multiple times; should be \"Continuous evaluation.\"","section":"§4.4"},{"comment":"The text says \"Similarly to what Landers (2014) and Landers (2014)\" — the citation is duplicated. Also, Section 5.2 refers to \"Econplus Champions League model discussed by Murillo et al. (2021)\", but the reference list has \"Murillo-Zamorano et al.\" and the citation format is inconsistent.","section":"§5.2.1"},{"comment":"The manuscript states \"Ethical Approval: Not applicable\" and \"Informed Consent: Not applicable\" despite collecting student grades, surveys, and anonymized data. At most institutions this requires at least an exemption or approval; please clarify or document the institutional review process.","section":"§3.5.1, §5.3"}],"recommendation":"reject","confidential_remarks":"The paper is better framed as a descriptive implementation report than as an impact evaluation. The causal and mediation claims are not supported by the design and contain internal inconsistencies that cannot be resolved by minor revision. The abstract/body mismatch about the course level and the unexplained N in the mediation analysis are additional concerns that would need to be addressed before any resubmission. If a future version is a descriptive case study, it should be submitted to an education-oriented venue rather than a computing/cyber venue, and it should clearly state that causal claims are not being made."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dear colleague,\n\nQuick take: this is a single-course case study with real descriptive value, but the causal claim at the center doesn't survive contact with the design. The 100% vs. 70.6% pass-rate comparison is a before/after comparison with a two-year gap, no controls for cohort or grading changes, and the mediation analysis partly restates the grading formula. I'd want a major rewrite before treating it as evidence about gamification.\n\nWhat's genuinely new: the specific activities (Nanogames, NanoTechStart) in a biomedical engineering course, and the detailed tables of grades, pass rates, and student surveys across three intervention years plus one historical control. The authors also take Landers' theory seriously and lay out hypotheses explicitly. That's more than most gamification case studies do.\n\nThe soft spots, in order of size. First, the control group. The course wasn't offered in 2018-19 and 2019-20; the intervention years are 2020-23. That's a pandemic-era comparison, and enrollment jumped from 17 to 45. The authors use unadjusted tests and don't discuss grade inflation, exam difficulty, or instructor changes. A descriptive claim that grades improved after a redesign is fine; a causal claim that gamification did it is not. Second, the mediation results are partly definitional. Section 3.2.4 says course total is 60% continuous evaluation plus 40% final exam, so regressing CT on CE should give a coefficient around 0.6. They get 0.58 and call that mediation; it's just the grading formula. The moderation interaction is also non-significant, so the \"moderated-mediation\" model is really just a direct effect. Third, internal consistency problems: the abstract describes master's-level students; the methods say bachelor's. The regression N is 41 but the intervention N is 45; Table 1 shows 17 control and 45 intervention, so I can't tell what 41 refers to. And Table 5 and Table 6 give conflicting direct effects (0.28 vs. 0.09). These aren't fatal for a descriptive report, but they make the quantitative sections hard to trust.\n\nWho gets value: instructors in engineering education who want a template for two gamified activities, or people studying how not to do mediation analysis. As a claim about gamification's impact, it shouldn't be taken at face value. I'd ask the authors to reframe it as a course design case study, remove the causal language and the mediation claims, and show the descriptive before/after data with appropriate caveats. That would be a publishable contribution to the teaching literature. As it stands, I'd read it skeptically but not ignore it.","headline":"A useful descriptive case study undermined by an uncontrolled before/after design and a mediation analysis that partly restates the grading formula; the causal claim doesn't hold.","tokens_in":21109,"tokens_out":2315,"would_cite":false,"duration_ms":24699,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that adding two gamified activities to a biomedical engineering course raised the pass rate from 70.6% to 100% and significantly raised final scores.","keywords":["gamification","learning outcomes","biomedical engineering education","project-based learning","peer assessment","STEM education","continuous evaluation","mediation analysis"],"falsifier":"Re-run the same course in a later year with identical exams and grading rubrics but without Nanogames and NanoTechStart; if the pass rate stays near 100%, the attribution to gamification collapses. A simpler observational check is to compare the 2017-2018 cohort's grades in other courses with the 2020-2023 cohorts' grades in those same courses; if the cohorts differ globally, the control gap is not specific to this course.","tokens_in":1595,"feed_emoji":"🎮","tokens_out":2620,"duration_ms":71952,"temperature":0.7,"pith_summary":"This paper argues that adding game elements to a project-based biomedical engineering course improves learning outcomes. Across three academic years (2020-2023), 45 students took a version with two gamified activities, while 17 students from 2017-2018 served as a non-gamified control. The intervention cohort had a 100% pass rate and higher final scores than the control's 70.6%, and regression analysis reported a significant direct effect of the gamified design on the course total score. The authors frame the result with the Theory of Gamified Learning: the games work partly by lifting continuous-assessment performance, although the moderation path was not significant.","feed_headline":"Gamified course hits 100% pass rate vs 70.6% control","feed_subtitle":"Team quiz games and a startup-pitch project lifted final scores and engagement in a biomedical engineering course.","key_machinery":"The engine is the Theory of Gamified Learning, applied as a mediation-moderation model: gamification (binary 0/1) is the independent variable, continuous evaluation score is the proposed mediator and moderator, and normalized course total score is the outcome. The two concrete gamified instruments—Nanogames, a team quiz with leaderboards and a final Alphabet Game, and NanoTechStart, a student-run startup pitch with a jury and vote—operationalize the independent variable. The regression paths Gamification→CE, CE→CT, Gamification→CT, and Gamification×CE→CT carry the causal claim.","core_discovery":"The central empirical claim is that a redesigned fourth-year course on biomedical applications of nanotechnology, which appended two collaborative game-based activities, is associated with a large jump in academic performance. Grading combined continuous evaluation (60%) with a final exam (40%); the intervention added Nanogames to the midterm and NanoTechStart, a simulated investor-pitch event, to the project work. Compared with the 2017-2018 control, the 2020-2023 cohorts moved from a 70.6% to a 100% pass rate, with no failing grades, and the distribution of final scores differed at p<0.0001. In a mediation-moderation regression, gamification had a significant direct effect on course total","pith_inferences":["Editorial extension: the 100% versus 70.6% comparison is vulnerable to cohort drift, since the course was not taught in 2018-2019 or 2019-2020; a concurrent control or a replication with the same exams and grading rubric would isolate the gamification effect more cleanly.","Editorial extension: the null moderation result is consistent with gamification working through motivational channels such as engagement or time-on-task rather than through better performance on homework and labs; that mechanism could be tested by measuring attendance, participation, and study time directly.","Editorial extension: because Nanogames and NanoTechStart bundle competition, teamwork, peer feedback, mentoring, and public presentation, the design does not identify which element carries the effect; a component-wise study could separate leaderboard competition from the entrepreneurial pitch format.","Editorial extension: self-reported satisfaction is near ceiling, so future work should pair subjective ratings with objective skill assessments or delayed retention tests to distinguish enjoyment from durable learning."],"forward_implications":["If the central claim is correct, adding collaborative game elements to a project-based STEM course can move pass rates from roughly 70% to near 100% in that specific course.","The significant direct path (β=0.28) implies gamification's effect on final grades is not fully explained by improved continuous-assessment scores; the design adds value beyond homework, labs, and project grades.","Because the moderation interaction was not significant, the data do not support the idea that continuous-evaluation performance changes how strongly gamification affects course totals.","Student survey responses indicate that both activities were perceived as improving subject knowledge and soft skills, suggesting the benefit may extend beyond grades.","The authors present the activity structure as a replicable model for other science and technology courses that use project-based learning."],"supporting_citations":[{"why":"Supplies the mediation-moderation theory of gamified learning that defines the study's hypotheses and regression model.","marker":"Landers (2014)"},{"why":"Provides the meta-analytic baseline that active learning improves STEM performance, which motivates the intervention.","marker":"Freeman et al. (2014)"},{"why":"Supplies the game-attribute taxonomy used to identify which game elements to apply.","marker":"Bedwell, Pavlas, Heyne, Lazzara, and Salas (2012)"},{"why":"Names the online platform on which the Nanogames activities and leaderboards were implemented.","marker":"Educaplay (2023)"},{"why":"Supports the claim that challenge and competition in game-based learning enhance engagement and outcomes.","marker":"Hamari et al. (2016)"},{"why":"Provides evidence that gamified e-quizzes improve student learning and engagement, a direct comparison for the Nanogames results.","marker":"Zainuddin et al. (2020)"},{"why":"Documents the effect of gamified quizzes on student learning, supporting the quiz-based component.","marker":"Sanchez, Langer, and Kaur (2020)"},{"why":"Supplies an example of e-gamification increasing engagement and the Champions-League-style team competition model.","marker":"Murillo-Zamorano et al. (2021)"},{"why":"Earlier gamification experiment showing improved scores and completion rates, used to position the present findings.","marker":"Cuevas-Martínez et al. (2019)"}],"fun_headline_variants":["Gamified BME course: 100% pass rate vs 70.6% baseline","Gamification in BME course: pass rate jumps from 70.6% to 100%","Gamified BME course achieves 100% pass rate vs 70.6% control","From 70.6% to 100%: gamified BME course pass rate","Gamified BME course: 70.6% to 100% pass rate"],"cache_read_input_tokens":22912,"weakest_assumption_plain":"The load-bearing premise is that the 2017-2018 control group is a valid counterfactual for the 2020-2023 intervention groups, even though the course was not taught in 2018-2019 or 2019-2020, so student ability, grading standards, and teaching may have shifted.","fun_headline_variants_meta":{"raw":{"variants":["Gamified BME course: 100% pass rate vs 70.6% baseline","Gamification in BME course: pass rate jumps from 70.6% to 100%","Gamified BME course achieves 100% pass rate vs 70.6% control","From 70.6% to 100%: gamified BME course pass rate","Gamified BME course: 70.6% to 100% pass rate"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000684,"raw_usage":{"total_tokens":2925,"prompt_tokens":710,"completion_tokens":2215,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":454,"completion_tokens_details":{"reasoning_tokens":2094}},"tokens_in":454,"tokens_out":2215,"duration_ms":16615,"temperature":1.0,"reasoning_tokens":2094,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T04:20:38.011174+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the same course in a later year with identical exams and grading rubrics but without Nanogames and NanoTechStart; if the pass rate stays near 100%, the attribution to gamification collapses. A simpler observational check is to compare the 2017-2018 cohort's grades in other courses with the 2020-2023 cohorts' grades in those same courses; if the cohorts differ globally, the control gap is not specific to this course.","supporting_citations":[{"cited_title":"(2014, 12)","cited_arxiv_id":null,"evidence_quote":"Supplies the mediation-moderation theory of gamified learning that defines the study's hypotheses and regression model."},{"cited_title":"(2012, 12)","cited_arxiv_id":null,"evidence_quote":"Supplies the game-attribute taxonomy used to identify which game elements to apply."},{"cited_title":"(2016, 1)","cited_arxiv_id":null,"evidence_quote":"Supports the claim that challenge and competition in game-based learning enhance engagement and outcomes."},{"cited_title":"(2020, 2)","cited_arxiv_id":null,"evidence_quote":"Provides evidence that gamified e-quizzes improve student learning and engagement, a direct comparison for the Nanogames results."},{"cited_title":"(2020, 1)","cited_arxiv_id":null,"evidence_quote":"Documents the effect of gamified quizzes on student learning, supporting the quiz-based component."},{"cited_title":"(2021, 12)","cited_arxiv_id":null,"evidence_quote":"Supplies an example of e-gamification increasing engagement and the Champions-League-style team competition model."}],"review_version":1}