A Causal Framework for Estimating Heterogeneous Effects of On-Demand Tutoring

Bibi Groot; Danielle R Thomas; Digory Smith; Justin Reich; Kirk Vanacore; Rene Kizilcec

REVIEW 4 major objections 5 minor 51 references

A Causal Framework for Estimating Heterogeneous Effects of On-Demand Tutoring

T0 review · 4 major / 5 minor · reviewed 2026-08-02 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A student-initiated tutoring session inside an adaptive learning platform raises next-problem accuracy by about 4 percentage points, recoverable from observational logs by conditioning on a learned knowledge state.

desk verdict A serious applied-causal-inference paper whose session-level CATE pipeline is genuinely useful, but whose own placebo test undercuts the 'robust to unmeasured confounders' claim—worth peer review, not worth taking the 4pp estimate at face value. read the letter →

arxiv 2602.19296 v2 pith:P7CNQ7TW submitted 2026-02-22 cs.HC stat.AP

Kirk Vanacore , Danielle R Thomas , Digory Smith , Bibi Groot , Justin Reich , Rene Kizilcec This is my paper

classification cs.HCstat.AP

keywords causalinferenceon-demandtutoringdeepknowledgetracingheterogeneoustreatmenteffectsforestsadaptivelearningsystemsobservationalstudyeffectiveness

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that a student-initiated tutoring session inside an adaptive learning platform has a measurable, short-term causal effect on subsequent performance, and that this effect can be estimated from observational logs rather than requiring randomized assignment. The core methodological claim is that the severe self-selection problem — students call for help exactly at moments of confusion — can be handled by conditioning on a deep-model estimate of each student's evolving knowledge state at the moment right before the request. Using that strategy on more than five thousand real tutoring sessions, the authors estimate that asking for a human tutor raises the probability of answering the next problem correctly by 4.01 percentage points and the first problem on the next skill by 2.73 percentage points. They also claim the average hides large session-level variation, with estimated effects ranging from about -20 to +20 percentage points, and that the framework's session-level estimates can serve as training labels and benchmarks for AI tutors. A sympathetic reader would care because, if these estimates are right, embedded on-demand tutoring produces a small but real and transferable gain, and the field gains a practical template for causally evaluating help at scale.

What carries the argument

The load-bearing mechanism is the latent knowledge state: a 50-dimensional hidden vector produced by a deep recurrent network (deep knowledge tracing, DKT) trained on held-out student interaction logs, extracted at the moment just before a student could request help. Alongside the model's predicted probabilities for the current and next problems, these hidden states enter a causal forest that residualizes both the outcome and the treatment indicator against the covariates and then computes doubly robust average and unit-level effects by augmented inverse probability weighting. The knowledge state is what does the causal work: it is the framework's answer to the core confound that students se

What would settle it

A randomized trial that varies who gets to request a tutor in the same platform — or randomly delays access by a few minutes for a treatment arm — and measures the same next-problem and next-skill outcomes would settle the claim; if the experimental effect is materially smaller than 4.01 percentage points or absent, the observational estimate is inflated by the selection process.

Watch

Extended reading notes

Core claim

The paper's central discovery is a causal estimate with a methodological warrant. On its own terms: for middle-school students working math problems in an adaptive system, the act of requesting a human tutoring chat increases the probability of a correct answer on the very next problem by an average of 4.01 percentage points (95% CI 2.51 to 5.51) and on the first problem of the next skill by 2.73 percentage points (95% CI 1.12 to 4.35). The authors report that this estimate is essentially unchanged when external demographic and pretest covariates are added, drops to about 2.2 points under a washout-sample control definition, and remains positive after a pre-treatment placebo check once corre

Load-bearing premise

The load-bearing assumption is that, once the model conditions on the student's latent knowledge state and problem-level probabilities, whether the student asks for tutoring carries no additional information about how they would have done without it; the paper's own pre-treatment placebo check shows a small residual imbalance, so this is the vulnerable point.

Editorial extensions

If this is right

If the 4-point and 3-point estimates are correct, on-demand tutoring embedded in adaptive platforms produces an immediate, transferable learning gain comparable in size to other platform supports, strengthening the case for offering human help just-in-time.
The wide spread of session-level estimates means average effects understate reality; using CATEs as outcome labels would let researchers identify which tutor behaviors produce high-impact sessions and which are neutral or harmful.
The stability of the estimate when external test scores and demographics are added implies that platform-derived learning traces may already capture most of the relevant confounding, so large-scale observational evaluation of tutoring features becomes cheaper.
The monotonic relationship between prior tutoring experience and session impact suggests students improve at using on-demand help; platforms could train students in help-seeking before they are in a struggle moment.
The attenuated benefit for lower-income students implies simply granting access is unlikely to narrow achievement gaps, and that access designs must address additional barriers.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Editorial extension: the same pipeline could be pointed at AI-generated hints and AI tutor dialogues, putting human and machine tutoring on the same causal scale and making session-level ATE/CATE estimates a common currency for model comparison.
Editorial extension: because conditional ignorability is unverifiable, a natural stronger test is a randomized encouragement or wait-list design; if the resulting experimental effect is materially smaller than 4 points, the residual imbalance hinted by the -1.67pp placebo estimate is doing real work.
Editorial extension: the DKT representation is one of many possible mastery encodings; swapping in Bayesian or transformer-based knowledge tracing and tracking how much the 4-point estimate moves would quantify how much of the conclusion depends on the choice of representation.
Editorial extension: predicted CATEs could drive an adaptive help-seeking policy — prompting a student to request a tutor only in sessions with high predicted benefit — though such a policy would need its own causal evaluation.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Colleague,

Short version: this is a serious applied-causal-inference paper, and the method contribution is real, but the headline “robust to unmeasured confounders” does not survive contact with their own placebo test. If you read it as evidence that on-demand tutoring probably has a small immediate effect, it is useful. If you read it as a template for benchmarking AI tutors, it is worth taking seriously. If you read it as securing the 4pp estimate, you are reading the abstract too generously.

The genuinely new piece is the pipeline: DKT trained on a holdout gives a 50-dim latent knowledge state at the moment before help-seeking; those states plus predicted probabilities go into an honest causal forest with student-level clustering; ATE/ATT/CATEs come out via AIPW. That combination for session-level tutoring effects is not in the citations I know. The empirical CATE distribution is also a contribution: wide heterogeneity (-20pp to +20pp) is exactly what you would need to train or evaluate AI tutors. The sample construction is careful—never-treated controls, overlap restriction, washout robustness. The data/code link is on OSF. This deserves credit.

The problem: their placebo outcome is a problem three items before treatment, and the estimated effect is -1.67pp with CI [-3.10, -0.24]. That CI excludes zero before Bonferroni. They call it non-significant, but that is only true after the correction. The negative sign is convenient—it suggests treated students were lower-performing, which would bias against finding a positive tutoring effect—but a negative point estimate on the placebo does not establish that the post-treatment estimate has no residual bias of the other sign. The washout specification dropping to 2.17pp is additional evidence that the never-treated control group is not perfectly comparable. The sensitivity analysis (RV = 3x most influential covariate) is not crazy, but it is not proof. The right fix is either a design that explicitly balances the pre-treatment outcome (e.g., use placebo performance in the outcome model, or reweight on pre-treatment trends) or much softer language in the abstract.

Minor notes: the paper says “non-significant” only after Bonferroni; the Appendix is referenced but I did not execute the OSF deposit. The citation pattern looks fine; no obvious loaded self-cite.

Who this is for: EDM/learning-engineering people and applied causal methodologists. I would send it to peer review; the framework and heterogeneity results are worth referee time. I would not let the current abstract through without revision.

Referee Report

4 major / 5 minor

Summary. The paper introduces a three-stage causal inference framework for estimating session-level effects of on-demand human tutoring embedded in an adaptive learning platform. The stages are: (1) analytic sample construction that pairs tutored problem attempts with subsequent untutored attempts and uses never-treated students as controls; (2) Deep Knowledge Tracing (DKT) trained on a held-out sample to produce 50-dimensional latent knowledge-state covariates; and (3) doubly robust estimation of average and conditional treatment effects using causal forests with AIPW (Eq. 3). Applying this to roughly 5,000 tutoring sessions, the authors report an ATE of 4.01pp (CI 2.51–5.51) on next-problem correctness and 2.73pp (CI 1.12–4.35) on next-skill accuracy, with a placebo test and several robustness checks. They also report substantial CATE heterogeneity and student-level moderators, including larger effects for lower-mastery students and slightly smaller effects for low-SES students.

Significance. The contribution is potentially valuable: it demonstrates a practical pipeline for estimating causal effects from fine-grained learning logs, combines DKT hidden states with a modern heterogeneous-effect estimator, and includes useful diagnostics (placebo outcomes, propensity overlap, washout controls). The DKT model is trained on a separate hold-out, the causal forest uses honesty, and the analytic sample construction addresses SUTVA by excluding previously treated students from the primary control group. The ATE magnitude and direction are plausible and consistent with prior tutoring literature. If the identification assumption were supported, this would be a strong template for evaluating both human and AI tutoring in adaptive systems. However, the paper's own placebo test shows a pre-treatment baseline imbalance that directly undercuts the claim that conditioning on DKT states removes self-selection confounding. The washout specification also attenuates the effect by nearly half. These issues must be resolved or substantially reframed before the central quantitative claim can be accepted.

major comments (4)

[Table 3 and Section 6.1.1] The placebo test is the most load-bearing diagnostic, and it does not support the identification strategy. In Table 3, the pre-intervention placebo ATE is -1.67pp with 95% CI [-3.10, -0.24], which excludes zero before Bonferroni adjustment. Because this outcome temporally precedes tutoring, a null effect is required under conditional ignorability (Section 5.4.3). The CI excluding zero indicates that, conditional on the DKT features and other covariates, treated and control students are not exchangeable on baseline performance. The authors' interpretation that 'if anything, students may be biased toward lower baseline performance' does not establish that the post-treatment estimate is unbiased; residual confounding of unknown sign and magnitude can affect the post-treatment outcome. Please report a calibration analysis (e.g., including the placebo outcome as a covariate or using it as a n
[Sections 5.4.2 and 6.1.1] The washout robustness specification cuts the ATE from 4.01pp to 2.17pp, a 46% reduction. The authors note that the confidence intervals overlap, but overlap is a weak criterion; the point estimate moves substantially, and the two specifications target different control-group assumptions. If the washout controls reduce selection bias from using only never-treated students, then the true effect may be closer to 2pp than to 4pp. The discussion should explicitly acknowledge that the never-treated control group may not be comparable despite DKT adjustment, and should report a formal comparison (e.g., a Hausman-type test or a discussion of which specification is more credible) rather than treating the overlapping CIs as evidence of robustness.
[Sections 6.2 and 6.3, Eq. (4)] The heterogeneity analysis uses estimated CATE points as outcomes in a linear mixed model without accounting for CATE estimation uncertainty. The reported ranges (-20.25pp to +19.91pp for immediate performance) are likely inflated by noise: causal-forest local estimates are regularized but still carry substantial sampling variability, especially in a sample of 5,163 treated observations. Using noisy CATEs as dependent variables can bias moderator coefficients and overstate the precision of the heterogeneity findings. Please report confidence intervals or calibration diagnostics for the CATEs, and use an estimator that propagates CATE uncertainty into the moderator regressions, or clearly frame the heterogeneity results as exploratory.
[Section 6.1.1 / Appendix A.1] The omitted-variable sensitivity claim is not quantified. The text states that an unmeasured confounder would need to be 'three times as predictive of both treatment assignment and the outcome as our most influential observed covariates,' but it does not report the numeric Robustness Value (RV q=1) or the partial R² of the strongest observed covariate against which the comparison is made. Without these numbers, the reader cannot assess whether the RV is large relative to plausible unmeasured confounders such as motivation or help-seeking propensity. Please report the exact RV values and benchmark covariates.

minor comments (5)

[Section 6.1.1 vs Table 3] Several confidence intervals in the text do not match the table: the external-variables ATE is given as CI (2.73, 5.51) in the text but (2.49, 5.37) in Table 3; the washout ATE is given as CI (1.17, 6.16) in the text but (1.17, 3.16) in the table. Please reconcile.
[Abstract / Section 6.2] The near-transfer CATE range is reported as '-23.60pp to +47.70pp' in the abstract and as '-47.70pp to 23.60pp' in Section 6.2. The ordering and signs are inconsistent; please correct.
[Section 4.2] The citation for the Eedi RCT is missing ('[?]'). Please add the reference or remove the placeholder.
[Various] Typos: 'conrell.edu' should be 'cornell.edu'; 'SUTV A' should be 'SUTVA'; 'T echnology' in the affiliation should be 'Technology'.
[Table 3 footnote] The footnote says 'Bonferroni-corrected significance' but it is unclear whether the correction applies only to the placebo row or to all significance stars in the table. Please clarify the multiple-testing correction procedure.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the ATE is produced by out-of-sample DKT covariates and honest causal forests, not by a fitted input or self-citation chain.

full rationale

The reported treatment effects are not derived from any fitted constant equal to the target, nor from a self-citation. DKT is trained on a separate holdout sample ('The hold-out sample was used exclusively for DKT estimation'), and its validation is on data not used for training ('The model achieved an acceptable AUC of 0.72 on the analytic control sample (which was not used for training)'), so the DKT hidden states and predicted probabilities are out-of-sample covariates. The causal forest uses honesty ('separate subsamples are used for tree construction and treatment effect estimation'), so CATE/ATE estimates are not in-sample predictions. The AIPW estimator in Eq. 3 combines these out-of-sample nuisance functions in a standard doubly-robust way; no equation reduces tau_ATE to an input definition. The few self-citations (e.g., [14], [19], [20], [28], [39]) appear in literature review and heterogeneity discussion and are not load-bearing in the identification or estimation argument. The pre-treatment placebo imbalance (-1.67pp, CI -3.10 to -0.24 before Bonferroni) is a residual-confounding concern, not a circularity. The missing '[?]' citation in Section 4.2 for the underlying RCT is an evidence-completeness issue, not a circular step in the derivation chain.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The framework rests on standard causal assumptions (ignorability, positivity, SUTVA) and a strong modeling assumption that DKT hidden states summarize the confounding knowledge trajectory. It introduces no new physical entities or novel statistical objects beyond the CATE estimates themselves. The DKT and GRF hyperparameters are tuned, not pulled out of thin air, and the washout window is a hand-chosen robustness parameter.

free parameters (3)

DKT LSTM hyperparameters (hidden_dim=50) = hidden_dim=50; other training details not fully specified
Model capacity choice affecting latent state quality; tuned on holdout AUC.
Causal forest hyperparameters (trees=500, honesty, mtry, etc.) = selected via cross-validation
Affect CATE estimates but not a single fitted constant; standard ML tuning.
Washout window ('two skills') = 2 skills
Hand-chosen cutoff for robustness control definition; affects robustness estimate (2.17pp vs 4.01pp).

assumptions (4)

domain assumption Conditional ignorability: Y(0),Y(1) independent of Z given X
Core identification assumption; stated in Limitations Section 8; challenged by the placebo test result.
domain assumption Positivity/overlap: 0 < P(Z=1|X) < 1
Checked via propensity score range 0.03-0.89 in Section 5.3.2.
domain assumption SUTVA: no interference between students/sessions
Paper restricts control to never-treated students and excludes treated students' non-tutored attempts to avoid cross-unit contamination (Section 5.1).
domain assumption The 50-dimensional DKT hidden state is a sufficient summary of the time-varying knowledge state
The framework conditions on DKT hidden states as if they capture all confounding; no formal guarantee is provided (Section 5.2).

how reviews work

0 comments

Cite this review

Pith. "Pith review of A Causal Framework for Estimating Heterogeneous Effects of On-Demand Tutoring." pith.science (2026). https://pith.science/paper/P7CNQ7TW

@misc{pith2026260219296,
  author       = {Pith},
  title        = {Pith review of: A Causal Framework for Estimating Heterogeneous Effects of On-Demand Tutoring},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/P7CNQ7TW}},
  note         = {Machine review of arXiv:2602.19296}
}

abstract

This paper introduces a scalable causal inference framework for estimating the immediate, session-level effects of on-demand human tutoring embedded within adaptive learning systems. Because students seek assistance at moments of difficulty, conventional evaluation is confounded by self-selection and time-varying knowledge states. We address these challenges by integrating principled analytic sample construction with Deep Knowledge Tracing (DKT) to estimate latent mastery, followed by doubly robust estimation using Causal Forests. Applying this framework to over 5,000 middle-school mathematics tutoring sessions, we find that requesting human tutoring increases next-problem correctness by approximately 4 percentage points and accuracy on the subsequent skill encountered by approximately 3 percentage points, suggesting that the effects of tutoring have proximal transfer across knowledge components. This effect is robust to various forms of model specification and potential unmeasured confounders. Notably, these effects exhibit significant heterogeneity across sessions and students, with session-level effect estimates ranging from $-20.25pp$ to $+19.91pp$. Our follow-up analyses suggest that typical behavioral indicators, such as student talk time, do not consistently correlate with high-impact sessions. Furthermore, treatment effects are larger for students with lower prior mastery and slightly smaller for low-SES students. This framework offers a rigorous, practical template for the evaluation and continuous improvement of on-demand human tutoring, with direct applications for emerging AI tutoring systems.

Figures

Figures reproduced from arXiv: 2602.19296 by the authors.

**Figure 2.** Distribution of Conditional Average Treatment Effects (CATE) on both Immediate Performance and Near Transfer. [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗

**Figure 3.** Effect Sizes on Immediate Performance by Tutoring Session Characteristics. Each effect is decomposed by quartiles [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗

Figures from the paper (1 more)

**Figure 4.** Figure 4: Interactions between prior knowledge in terms of predicted mastery (A) or standardized test scores (B) and socio [PITH_FULL_IMAGE:figures/full_fig_p010_4.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

51 extracted references · 2 linked inside Pith

[1]

high-dosage

INTRODUCTION Tutoring is widely recognized as one of the most effective educational interventions, particularly for students strug- gling academically [11, 16, 24, 22]. However, recent large- scale evidence suggests that the magnitude of K-12 tutor- ing effects varies widely across contexts, populations, and implementations, complicating efforts to genera...

arXiv 2026
[2]

Hybrid” or

BACKGROUND 2.1 Efficacy and Heterogeneity in Tutoring Interventions Tutoring is widely recognized as one of the most effective academic interventions for improving student learning out- comes [11, 16, 24]. Comprehensive meta-analyses have found that tutoring consistently yields significant positive effects on student learning [22, 24]. These effects are i...
[3]

CURRENT STUDY The current study proposes an observational approach for evaluating tutoring sessions embedded within digital learn- ing platforms by estimating their causal impact on stu- dents’ subsequent performance. The approach integrates DKT–based estimates of students’ latent knowledge states into a causal inference framework that combines causal for...
[4]

CONTEXT 4.1 Data The data used for this analysis came from the treatment arm of a multi-year randomized controlled trial (RCT) de- signed to estimate the causal impact of Eedi on middle school mathematics achievement. Schools were randomly assigned to treatment or control conditions at the school level, with the treatment group receiving access to Eedi fo...
[5]

negative control

METHODS: CAUSAL FRAMEWORK FOR LOCAL EFFECT ESTIMATION Our causal inference framework consists of three intercon- nected stages, illustrated in Figure 1. (1) First, we construct an analytic dataset that links each tutoring intervention to its immediate outcome while defining an appropriate con- trol group. (2) Second, we train a DKT model on a held- out sa...
[6]

We find a statistically significant positive effect of tutoring on immediate perfor- mance and near transfer (Table 3)

RESULTS 6.1 Treatment Effect Estimates Table 3 presents the doubly robust average treatment ef- fect estimates from the Causal Forest analysis, with p-values adjusted for multiple corrections. We find a statistically significant positive effect of tutoring on immediate perfor- mance and near transfer (Table 3). The estimated average treatment effect (ATE)...
[7]

DISCUSSION & CONCLUSION We introduced a scalable causal framework for estimating problem-level effects of on-demand human tutoring embed- ded in an adaptive learning system. Across specifications, requesting tutoring produces modest but reliable gains in immediate performance (next-problem correctness; ATE of 4 percentage points) and near transfer (first ...
[8]

LIMITATIONS & FUTURE WORK Several limitations of this study provide productive av- enues for future research. First, while our causal identi- fication strategy relies on the conditional ignorability as- sumption, our sensitivity analysis suggests that unmea- sured factors—such as student motivation or teacher prac- tices—would need to be substantially mor...

Show all 51 references

[9]

Abdelrahman, Q

G. Abdelrahman, Q. Wang, and B. Nunes. Knowledge tracing: A survey.ACM Computing Surveys, 55(11):224:1–224:37, 2023

2023
[10]

Aleven, I

V. Aleven, I. Roll, B. M. McLaren, and K. R. Koedinger. Help helps, but only so much: Research on help seeking with intelligent tutoring systems. International Journal of Artificial Intelligence in Education, 26(1):205–223, 2016

2016
[11]

Athey, J

S. Athey, J. Tibshirani, and S. Wager. Generalized random forests.The Annals of Statistics, 47(2):1148–1178, 2019

2019
[12]

Athey and S

S. Athey and S. Wager. Estimating treatment effects with causal forests: An application.Observational studies, 5(2):37–51, 2019

2019
[13]

M. T. Chi, S. A. Siler, H. Jeong, T. Yamauchi, and R. G. Hausmann. Learning from human tutoring. Cognitive science, 25(4):471–533, 2001

2001
[14]

D. R. Chine, C. Brentley, C. Thomas-Browne, J. E. Richey, A. Gul, P. F. Carvalho, and K. R. Koedinger. Educational equity through combined human-ai personalization: A propensity matching evaluation. In International Conference on Artificial Intelligence in Education, pages 366...

2022
[15]

Cinelli and C

C. Cinelli and C. Hazlett. Making sense of sensitivity: Extending omitted variable bias.Journal of the Royal Statistical Society Series B: Statistical Methodology, 82(1):39–67, 2020

2020
[16]

Cinelli and C

C. Cinelli and C. Hazlett. sensemakr: Sensitivity analysis tools for omitted variable bias in r.Journal of Statistical Software, 94(11):1–34, 2020

2020
[17]

R. K. Crump, J. V. Hotz, G. W. Imbens, and O. A. Mitnik. Dealing with limited overlap in estimation of average treatment effects.Biometrika, 96(1):187–199, 2009

2009
[18]

Deacon and G

G. Deacon and G. Chojnacki. Impacts of upchieve on-demand tutoring on students’ math knowledge and perceptions. Technical report, Mathematica, 2023

2023
[19]

Dietrichson, M

J. Dietrichson, M. Bøg, T. Filges, and A. M. Klint Jørgensen. Academic interventions for elementary and middle school students with low socioeconomic status: A systematic review and meta-analysis.Review of Educational Research, 87(2):243–282, 2017

2017
[20]

A. C. Eggers, G. Tu˜ n´ on, and A. Dafoe. Placebo tests for causal inference.American Journal of Political Science, 68(3):1106–1121, 2024

2024
[21]

A. N. Glynn and K. M. Quinn. An introduction to the augmented inverse propensity weighted estimator. Political Analysis, 18:36–56, 2010

2010
[22]

Gurung, S

A. Gurung, S. Baral, M. P. Lee, A. C. Sales, A. Haim, K. P. Vanacore, A. A. McReynolds, H. Kreisberg, C. Heffernan, and N. T. Heffernan. How common are common wrong answers? crowdsourcing remediation at scale. InProceedings of the Tenth ACM Conference on Learning @ Scale, page...

2023
[23]

Gurung, J

A. Gurung, J. Lin, J. Gutterman, D. R. Thomas, A. Houk, S. Gupta, and K. Koedinger. Human tutoring improves the impact of ai tutor use on learning outcomes. InInternational Conference on Artificial Intelligence in Education, pages 393–407. Springer, 2025

2025
[24]

Guryan, J

J. Guryan, J. Ludwig, M. P. Bhatt, P. J. Cook, J. M. Davis, K. Dodge, and G. Stoddard. Not too late: Improving academic outcomes among adolescents. American Economic Review, 113(3):738–765, 2023

2023
[25]

P. R. Hahn, J. S. Murray, and C. M. Carvalho. Bayesian regression tree models for causal inference: Regularization, confounding, and heterogeneous effects (with discussion).Bayesian Analysis, 15(3):965–1056, 2020

2020
[26]

D. W. Harrison, D. J. Brown, and S. Higgins. A pilot impact study to evaluate the effectiveness of eedi on raising attainment in mathematics at ks3. Technical report, What Worked Education, 2023

2023
[27]

R. F. Kizilcec, G. M. Davis, and G. L. Cohen. Towards equal opportunities in moocs: affirmation reduces gender & social-class achievement gaps in china. InProceedings of the fourth (2017) ACM conference on learning@ scale, pages 121–130, 2017

2017
[28]

R. F. Kizilcec, A. Saltarelli, P. Bonfert-Taylor, M. Goudzwaard, E. Hamonic, and R. Sharrock. Welcome to the course: Early social cues influence women’s persistence in computer science. In Proceedings of the 2020 CHI conference on human factors in computing systems, pages 1–13, 2020

2020
[29]

M. A. Kraft, D. S. Edwards, and M. Cannata. The scaling dynamics and causal effects of a district-operated tutoring program. edworkingpaper no. 24-1030.Annenberg Institute for School Reform at Brown University, 2024

2024
[30]

M. A. Kraft, B. E. Schueler, and G. Falken. What impacts should we expect from tutoring at scale? exploring meta-analytic generalizability, 2024

2024
[31]

Mojarad, A

S. Mojarad, A. Essa, S. Mojarad, and R. S. Baker. Studying adaptive learning efficacy using propensity score matching. InCompanion proceedings of the 8th international conference on learning analytics and knowledge (LAK’18), pages 5–9, 2018

2018
[32]

Nickow, P

A. Nickow, P. Oreopoulos, and V. Quan. The promise of tutoring for prek–12 learning: A systematic review and meta-analysis of the experimental evidence. American Educational Research Journal, 61(1):74–107, 2024

2024
[33]

Z. A. Pardos and N. T. Heffernan. Modeling individualization in a bayesian networks implementation of knowledge tracing. InInternational conference on user modeling, adaptation, and personalization, pages 255–266. Springer, 2010

2010
[34]

Y. Pei, A. Sales, and J. Gagnon-Bartsch. Boosting precision in educational a/b tests using auxiliary information and design-based estimators. In Proceedings of the 17th International Conference on Educational Data Mining, pages 990–993, 2024

2024
[35]

Pel´ anek

R. Pel´ anek. Bayesian knowledge tracing, logistic models, and beyond: an overview of learner modeling techniques.User modeling and user-adapted interaction, 27(3):313–350, 2017

2017
[36]

D. M. Pham, K. P. Vanacore, A. C. Sales, and J. A. Gagnon-Bartsch. Lool: Towards personalization with flexible & robust estimation of heterogeneous treatment effects. InProceedings of the 17th International Conference on Educational Data Mining, pages 376–384. International Ed...

2024
[37]

Piech et al

C. Piech et al. Deep knowledge tracing. InAdvances in Neural Information Processing Systems, pages 505–513, 2015

2015
[38]

Prihar, A

E. Prihar, A. Moore, and N. Heffernan. Identifying explanations within student-tutor chat logs. In Proceedings of the 15th International Conference on Educational Data Mining, page 773–777, 2022

2022
[39]

Prihar, T

E. Prihar, T. Patikorn, A. Botelho, A. Sales, and N. Heffernan. Toward personalizing students’ education with crowdsourced tutoring. page 37–45. Association for Computing Machinery, Inc, 2021

2021
[40]

Prihar, M

E. Prihar, M. Syed, and K. Ostrow. Exploring common trends in online educational experiments. page 12, 2022

2022
[41]

C. D. Robinson and S. Loeb. High-impact tutoring: State of the research and priorities for future learning. National Student Support Accelerator, 21(284):1–53, 2021

2021
[42]

C. D. Robinson, C. Pollard, S. Novicoff, S. White, and S. Loeb. The effects of virtual tutoring on young readers: Results from a randomized controlled trial. Educational Evaluation and Policy Analysis, 47(4):1245–1265, 2025

2025
[43]

D. B. Rubin. Estimating causal effects of treatments in randomized and nonrandomized studies.Journal of Educational Psychology, 66:688–701, 1974

1974
[44]

A. C. Sales, B. B. Hansen, and B. Rowan. Rebar: Reinforcing a matching estimator with predictions from high-dimensional covariates.Journal of Educational and Behavioral Statistics, 43(1):3–31, 2018

2018
[45]

A. C. Sales and J. F. Pane. The effect of teachers reassigning students to new cognitive tutor sections. International Educational Data Mining Society, 2020

2020
[46]

D. R. Thomas, J. Lin, E. Gatz, A. Gurung, S. Gupta, K. Norberg, and K. R. Koedinger. Improving student learning with hybrid human-ai tutoring: A three-study quasi-experimental investigation. InProceedings of the 14th Learning Analytics and Knowledge Conference, pages 404–415, 2024

2024
[47]

Vanacore, A

K. Vanacore, A. Sales, A. Liu, and E. Ottmar. Benefit of gamification for persistent learners: Propensity to replay problems moderates algebra-game effectiveness. InTenth ACM Conference on Learning @ Scale (L@S ’23), Copenhagen, Denmark, 2023. ACM

2023
[48]

Wager and S

S. Wager and S. Athey. Estimation and inference of heterogeneous treatment effects using random forests. Journal of the American Statistical Association, 113(523):1228–1242, 2018

2018
[49]

A. Wang, A. Rysbek, A. Huber, A. Nambiar, A. Kenolty, B. Caulfield, B. Lilley-Draper, B. Groot, B. Veprek, C. Burdett, C. Willis, C. Barton, D. Smith, G. Mu, H. Walters, I. Jurenka, I. Hulls, and V. Braz˜ ao. Ai tutoring can safely and effectively support students: An explorat...

2025
[50]

R. E. Wang, A. T. Ribeiro, C. D. Robinson, S. Loeb, and D. Demszky. Tutor copilot: A human-ai approach for scaling real-time expertise.arXiv preprint arXiv:2410.03017, 2024

2024 arXiv
[51]

Wu and J

E. Wu and J. A. Gagnon-Bartsch. The loop estimator: Adjusting for covariates in randomized experiments. Evaluation review, 42(4):458–488, 2018

2018

Pith tools

Reviewed August 2, 2026 · model on record in the stance chip above.

[1] [1]

high-dosage

INTRODUCTION Tutoring is widely recognized as one of the most effective educational interventions, particularly for students strug- gling academically [11, 16, 24, 22]. However, recent large- scale evidence suggests that the magnitude of K-12 tutor- ing effects varies widely across contexts, populations, and implementations, complicating efforts to genera...

arXiv 2026

[2] [2]

Hybrid” or

BACKGROUND 2.1 Efficacy and Heterogeneity in Tutoring Interventions Tutoring is widely recognized as one of the most effective academic interventions for improving student learning out- comes [11, 16, 24]. Comprehensive meta-analyses have found that tutoring consistently yields significant positive effects on student learning [22, 24]. These effects are i...

[3] [3]

CURRENT STUDY The current study proposes an observational approach for evaluating tutoring sessions embedded within digital learn- ing platforms by estimating their causal impact on stu- dents’ subsequent performance. The approach integrates DKT–based estimates of students’ latent knowledge states into a causal inference framework that combines causal for...

[4] [4]

CONTEXT 4.1 Data The data used for this analysis came from the treatment arm of a multi-year randomized controlled trial (RCT) de- signed to estimate the causal impact of Eedi on middle school mathematics achievement. Schools were randomly assigned to treatment or control conditions at the school level, with the treatment group receiving access to Eedi fo...

[5] [5]

negative control

METHODS: CAUSAL FRAMEWORK FOR LOCAL EFFECT ESTIMATION Our causal inference framework consists of three intercon- nected stages, illustrated in Figure 1. (1) First, we construct an analytic dataset that links each tutoring intervention to its immediate outcome while defining an appropriate con- trol group. (2) Second, we train a DKT model on a held- out sa...

[6] [6]

We find a statistically significant positive effect of tutoring on immediate perfor- mance and near transfer (Table 3)

RESULTS 6.1 Treatment Effect Estimates Table 3 presents the doubly robust average treatment ef- fect estimates from the Causal Forest analysis, with p-values adjusted for multiple corrections. We find a statistically significant positive effect of tutoring on immediate perfor- mance and near transfer (Table 3). The estimated average treatment effect (ATE)...

[7] [7]

DISCUSSION & CONCLUSION We introduced a scalable causal framework for estimating problem-level effects of on-demand human tutoring embed- ded in an adaptive learning system. Across specifications, requesting tutoring produces modest but reliable gains in immediate performance (next-problem correctness; ATE of 4 percentage points) and near transfer (first ...

[8] [8]

LIMITATIONS & FUTURE WORK Several limitations of this study provide productive av- enues for future research. First, while our causal identi- fication strategy relies on the conditional ignorability as- sumption, our sensitivity analysis suggests that unmea- sured factors—such as student motivation or teacher prac- tices—would need to be substantially mor...