{"id":"e2ec0fa3-a068-47b2-9cf1-2439a69c945a","arxiv_id":"2412.13395","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"Classroom talk-move classifiers, fine-tuned with longer context and speaker labels, transfer to math tutoring and outperform tutoring-only training on the new SAGA22 dataset.","lead":"The authors built a small annotated dataset of real math tutoring sessions and tested whether models trained on classroom teaching talk can be adapted to classify tutor and student talk moves. The adapted models beat training from scratch, but missing data-split details and code make the exact gains hard to verify.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"SAGA22 is included in supplementary pretraining (P contains 's'), yet the paper never defines the train/validation/test split of the 121 sessions; if test transcripts appeared in pretraining, the best student model's 76.5 F1 is partly leaked.","rationale":"The paper's central transfer claim is that supplementary pretraining on classroom data improves tutoring talk-move models, especially with longer context and speaker information. This is plausible and partly supported by internal comparisons: the best tutor model uses only teaching data (t+n) before fine-tuning and reaches 82.4 F1, and the zero-shot row {t+n,0} reaches 81.8 for tutors, so the classroom-to-tutoring transfer is not solely dependent on SAGA22 pretraining. However, the strongest abstract claim cites both the 82.4 tutor number and the 76.5 student number. The student model was pretrained on SAGA22 itself (t+n+s) and then fine-tuned on SAGA22, creating a direct route for test-set leakage unless the split was enforced. The paper does not define the split at all; Section 5 only says \"held-out test sets\" and Section 3.2 describes annotating 121 sessions without a split. This is exactly the load-bearing weak point: a small dataset of 121 sessions, 80 searched configurations, and no documented split means the student headline cannot be verified. I do not accuse the authors of deliberate leakage; the issue is that the experiment as described is not reproducible and the reported student gain may be inflated. The concrete test of releasing session IDs and rerunning with a train-only pretraining corpus would settle it. If leakage is confirmed, the 76.5 number and Table 4's 's' rows should be corrected or clearly caveated; if leakage is absent, the central claim stands largely intact. Hence the reader's CONDITIONAL verdict is appropriate, with the condition being explicit split disclosure and a leakage-safe rerun.","tokens_in":19102,"tokens_out":3788,"duration_ms":31084,"concrete_test":"Independently re-derive the SAGA22 session-level split: request the session IDs used for train, validation, and test, or have the authors release a stratified split, and verify that no test session appears in the P='s' pretraining corpora or in further fine-tuning. Then rerun S^{t+n+s,1}_{±7,spk} and the Table 4 rows involving 's' with SAGA22 pretraining restricted to training sessions only, using identical hyperparameters and validation-based checkpoint selection. If the 76.5 F1 drops materially (for example, by more than 2-3 macro F1) relative to the published value, the headline student-model gain is at least partly leakage-driven.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing risk is data leakage via pretraining. Section 4.3 defines supplementary pretraining datasets P ∈ {∅, \"t\", \"t+n\", \"t+s\", \"t+n+s\"}, with lower-case 's' denoting SAGA22. Table 4 evaluates rows {s,0}, {t+s,0}, {t+n+s,0}, and {t+n+s,1} on the SAGA22 test set, and the best student model S^{t+n+s,1}_{±7,spk} (Table 3) is first pretrained on all three datasets, including SAGA22, and then further fine-tuned on SAGA22. Section 5 says models are selected by validation macro F1 and reported on \"held-out test sets\", but the paper nowhere specifies how the 121 sessions were split into train, validation, and test, nor states that test transcripts were excluded from the supplementary pretraining phase. If the same sessions were used for pretraining and test, the reported 76.5 student F1 is inflated by memorization rather than transfer. The teacher-side best model T^{t+n,1} does not include SAGA22 in pretraining, so the classroom-to-tutoring transfer claim has some independent support from the tutor results and from zero-shot teaching-only rows, but the headline student number and the Table 4 ablations involving 's' cannot be trusted without split documentation. This is a reproducibility and transparency gap, not evidence of bad intent.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper introduces SAGA22, a dataset of 121 annotated high-school mathematics tutoring sessions labeled with 7 teacher/tutor and 5 student talk moves, and studies whether models trained for classroom discourse transfer to tutoring under a pretrain-fine-tune framework. Using RoBERTa-base, the authors vary dialogue context (previous-one vs ±7 utterances), speaker prefixes, supplementary pretraining datasets (∅, TALK MOVES, +NCTE-119, +SAGA22), and whether to further fine-tune on SAGA22. They report a best tutor model T{t+n,1}_{±7,spk} reaching 82.4 macro F1 and a best student model S{t+n+s,1}_{±7,spk} reaching 76.5, compared with 70.6 and 63.6 for from-scratch baselines, plus extensive ablations on context, speaker information, pretraining, and fine-tuning.","tokens_in":19653,"tokens_out":10254,"duration_ms":83054,"significance":"If the reported results hold, the paper is a useful empirical contribution: it adds a real-world tutoring corpus aligned with an existing talk-move annotation scheme, and it gives evidence that supplementary pretraining on classroom teaching data (TALK MOVES, NCTE-119) can improve tutoring talk-move classification, especially with ±7 context and speaker prefixes. The cleanest transfer comparisons—T{t,1} vs T{∅,1}, T{t+n,1} vs T{∅,1}, and the no-target-fine-tuning rows {t,0}/{t+n,0}—support this conclusion and are a strength, as are the explicit TALK MOVES re-split description and the confusion-matrix analyses. The main caveat is that the SAGA22 split is never defined, and the headline student model and several Table 4 rows include SAGA22 in supplementary pretraining; without proof that test transcripts were excluded, the student-side number is at risk of measuring memorization rather than transfer. The lack of variance reporting also weakens the small-difference ablation claims.","major_comments":[{"comment":"The SAGA22 train/validation/test split is never defined. Section 3.3 describes the TALK MOVES split (441/63/63), but for SAGA22 the paper only states that models are selected by validation macro F1 and reported on 'held-out test sets' (§5). This matters because §4.3 allows SAGA22 ('s') in supplementary pretraining, and Table 4 includes the rows {s,0}, {t+s,0}, {t+n+s,0}, and {t+n+s,1}; the headline student model S{t+n+s,1}_{±7,spk} is pretrained on TALK MOVES + NCTE-119 + SAGA22 and then fine-tuned on SAGA22. If the 121 sessions were not split before the pretraining stage, or if the split was performed at utterance rather than session level, the reported 76.5 student macro F1 and the 's' rows of Table 4 partly reflect memorization of test transcripts rather than transfer. Please specify the split protocol (session-level vs utterance-level, sizes, random seed), state explicitly that test transcripts were excluded from all stages including supplementary pretraining, and if they were not, rerun the affected experiments with a clean split. The tutor-side result T{t+n,1}_{±7,spk} is unaffected because it does not include SAGA22 in pretraining, but the student-side headline and the table rows involving 's' cannot be interpreted without this documentation.","section":"§4.3, §5; Tables 3–4"},{"comment":"All reported numbers appear to come from a single run; no variance, confidence intervals, or significance tests are presented. Statements such as 'significantly outperforms all the existing talk move models' (Section 5, Best Student Model) and the ablation conclusions in §6.3 and §6.4 are therefore not statistically grounded. The model differences in Table 4 are often small enough that seed variation could change the conclusions; for example, the tutor F1 difference between {t,0} and {t+n,0} is 78.9 vs 81.8, while the student difference between {t+s,0} and {t+n+s,0} is 77.1 vs 76.2. Please report the number of seeds, mean and standard deviation for the key comparisons, or a paired significance test across sessions, so the reader can distinguish robust effects from noise.","section":"§5; Tables 2–6"}],"minor_comments":[{"comment":"Please provide per-label Cohen's kappa for the 10-video IAA subset, the number of annotators per video, and the adjudication procedure; the current summary ('more than 80 Cohen's kappa on most labels, with 75 on one') is too coarse for a new annotation dataset.","section":"§3.2"},{"comment":"No code or data release link is given; footnote 5 says to contact the first author. Please provide a public repository with the SAGA22 split indices, preprocessing scripts, and training configuration, or explain in the paper why this is not possible.","section":"Reproducibility"},{"comment":"There are several typos and informal expressions, including 'Summerization' and 'Tutotring' in Table 1, 'whiling' in Section 5, 'pertaining' in Section 4.4, 'coorelated' in Section 5, and 'premilinary' in the Limitations section.","section":"Throughout"},{"comment":"The use of 'zero-shot' for rows with F=0 is misleading; those models were supervised on TALK MOVES and optionally NCTE-119 during supplementary pretraining. Please call this 'no target fine-tuning' instead.","section":"§6.3"},{"comment":"The asterisk and the footnote about 'highlighted numbers' are unexplained; please clarify which numbers are highlighted and why the selected best model is not the one with the highest validation F1.","section":"Table 4"},{"comment":"Please clarify how utterances are truncated or padded when the dialogue has fewer than 7 previous or subsequent utterances; the statement about empty utterances being prepended and padded is not precise about boundary handling.","section":"§4.1"}],"recommendation":"major_revision","confidential_remarks":"The reader's stress-test concern is well placed: the missing SAGA22 split is the main blocking issue, and the student-side result should not be cited as classroom transfer until the split is documented or the affected rows are rerun. That said, the tutor-side transfer result and the zero-target-fine-tuning rows give independent support for the paper's central claim, so I would not reject on the leakage concern alone. The paper would also benefit from more careful claims about 'significance' without error bars, and from a public release of code and splits."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper delivers two real things: SAGA22, a new math-tutoring corpus annotated with Accountable Talk moves, and a systematic transfer study showing that classroom discourse models can be adapted to tutoring with solid gains. The tutor-side result is the cleanest evidence — T^{t+n,1}_{±7,spk} reaches 82.4 macro F1 without ever seeing SAGA22 in pretraining, so the classroom-to-tutoring transfer claim is not circular. The ablations on context length and speaker prefixes are thorough and the findings are plausible: longer context helps when there is enough pretraining data, speaker prefixes almost always help, and adding more pretraining data yields diminishing returns. That is a useful empirical map for anyone building tutor feedback tools.\n\nThe soft spot is real and it is in the student-model headline. SAGA22 is included in supplementary pretraining for the best student model (S^{t+n+s,1}_{±7,spk}, 76.5 F1) and for the rows in Table 4 that contain 's', yet the paper never states how the 121 sessions were split into train, validation, and test, nor that the test transcripts were excluded from pretraining. If test sessions appeared in pretraining, that number is inflated. This is a transparency gap, not evidence of bad intent, and it is fixable: release the session IDs or a split definition. The zero-shot rows without SAGA22 (e.g., {t,0} at 74.4 student F1) suggest the transfer story survives even if the 76.5 number is discounted, so the central conclusion is not hanging by this thread.\n\nMinor issues: IAA is reported on only 10 of 121 videos, and there are no significance tests or error bars, so some of the finer ablations (differences of 1–2 F1) are probably noise. The paper's own limitations section is honest about modeling constraints but does not mention the split omission, which is worth flagging to the authors. The dataset is not publicly released—readers must contact the first author—which limits immediate reuse.\n\nWho is this for? Researchers and practitioners working on automated discourse analysis for tutoring or on transfer learning for dialogue act classification. It deserves a serious referee: the dataset is valuable, the experiments are broad, and the main transfer claim is independently supported. I would want the split clarified and the student-model results re-checked before trusting the headline, but desk rejection would be wrong. Send it to review with a request for splits and uncertainty estimates.","headline":"A genuinely useful new tutoring dataset and a transfer-learning result that mostly survives scrutiny, but the best student model's headline number rests on an undisclosed train/test split.","tokens_in":19978,"tokens_out":1365,"would_cite":true,"duration_ms":14949,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that supplementary pretraining on classroom mathematics discourse, combined with longer dialogue context and speaker prefixes, substantially improves automatic talk-move classification in mathematics tutoring.","keywords":["talk moves","mathematics tutoring","classroom discourse","transfer learning","dialogue act classification","SAGA22 dataset","speaker information","dialogue context"],"falsifier":"Rerun the best student configuration with a session-level split in which the test sessions are provably excluded from all pretraining and fine-tuning data; if the 76.5 macro F1 drops substantially, the headline transfer gain is partly leakage rather than classroom-to-tutoring transfer.","tokens_in":18976,"feed_emoji":"📐","tokens_out":6937,"duration_ms":58666,"temperature":0.7,"pith_summary":"Automatic analysis of tutoring dialogue usually requires expensive, tutoring-specific annotated data. This paper asks whether models trained to recognize 'talk moves' in mathematics classrooms can be transferred to one-on-one and small-group tutoring, and introduces SAGA22, a new corpus of 121 high-school math tutoring sessions, to test that. The central result is that supplementary pretraining on two classroom datasets, combined with a longer dialogue context (seven utterances before and after the target) and speaker prefixes, lifts macro F1 to 82.4 for tutor moves and 76.5 for student moves, versus 70.6 and 63.6 for models trained only on the tutoring data. If the transfer result holds, tutoring-discourse analysis can ride on existing classroom resources instead of requiring large new annotation campaigns.","feed_headline":"Classroom data lifts tutoring AI by 12 F1 points","feed_subtitle":"Pretraining on teaching transcripts with speaker labels beats training only on a new 121-session tutoring dataset.","key_machinery":"The load-bearing mechanism is a supervised pretrain-finetune pipeline whose input representation controls transfer. Each target utterance is classified into Accountable Talk move categories: 7 teacher/tutor moves and 5 student moves. Three variables matter most: a ±7 utterance context window (previous and subsequent utterances concatenated with sentence-boundary tokens), speaker prefixes ('T:' and 'S:') prepended to every utterance, and the choice of supplementary pretraining corpus. The paper argues that longer context only helps when enough pretraining data is available, and that speaker prefixes make that context usable by disambiguating who said what; ablations show each ingredient contributes positive, additive gains.","core_discovery":"The paper claims that the gap between classroom and tutoring discourse is bridgeable by transfer learning. Using RoBERTa-base as the backbone, a pretrained transformer language model, it defines a model search over context window (-1 vs ±7 utterances), speaker prefixes ('T:'/'S:'), supplementary pretraining on TALK MOVES and NCTE-119, and optional fine-tuning on SAGA22. The best tutor model, pretrained on both teaching datasets with ±7 context and speaker prefixes then fine-tuned on SAGA22, reaches 82.4 macro F1 on the SAGA22 test set, and the best student model, which additionally includes SAGA22 in pretraining, reaches 76.5. These numbers compare with 70.6 and 63.6 for the strongest from-scratch models, and zero-shot evaluation with teaching pretraining alone reaches 81.8 and 74.4. The paper concludes that classroom talk-move resources are reusable for tutoring, provided the input representation includes speaker identity and a sufficiently long dialogue window.","pith_inferences":["The paper's emphasis on bi-party speaker prefixes suggests a testable extension: distinguishing individual students ('Student-1' vs 'Student-2') in SAGA22 could specifically recover the rare RELTO and ASKMI moves that the unified 'S:' prefix loses.","Because the best label-level F1 scores come from different pretraining mixtures, a weighted mixture or multi-task objective tuned per talk-move label could outperform any single corpus combination.","If classroom-to-tutoring transfer works this well, the same recipe may transfer talk-move models across subjects or grade bands, but that remains untested since all datasets here are U.S. English-only mathematics discourse."],"forward_implications":["Tutoring talk-move classifiers can be built from existing classroom corpora plus a small tutoring adaptation set, reducing the annotation burden for new tutoring programs.","Adding speaker prefixes and a seven-utterance context window matters more than scaling pretraining data once a classroom corpus is available.","Zero-shot transfer from teaching data alone reaches near-best performance on tutor moves, so some tutoring applications may not need tutoring-specific fine-tuning at all.","Tutor and student move classifiers have different optimal pretraining mixtures, so practical systems should tune them separately."],"supporting_citations":[{"why":"Supplies the TalkMoves classroom dataset and the original talk-move classification setup that this paper transfers from.","marker":"[Suresh et al., 2022a]"},{"why":"Supplies the longer-context (previous and subsequent seven utterances) modeling strategy and prior classroom-domain baselines.","marker":"[Suresh et al., 2022b]"},{"why":"Supplies the NCTE-119 elementary classroom transcript corpus used as a second supplementary pretraining source.","marker":"[Demszky and Hill, 2022]"},{"why":"Supplies the in-context-learning baseline that the paper re-implements with a different 7B model.","marker":"[Wang et al., 2023]"},{"why":"Provides an earlier one-to-one tutoring dialogue corpus used to position SAGA22 as a real-world math tutoring dataset.","marker":"[Stasaski et al., 2020]"},{"why":"Provides MathDial, another math tutoring dialogue corpus, used to contrast with the new SAGA22 dataset.","marker":"[Macina et al., 2023]"}],"fun_headline_variants":["Tutoring AI gains 12 F1 points via classroom pretraining","Classroom talk moves transfer to improve tutor AI","Speaker-aware context boosts math tutoring dialogue models","Teaching data sharpens AI talk moves in tutoring","From classroom to tutoring: AI talk moves improve"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper never states how the 121 SAGA22 sessions were split into training, validation, and test sets, and the best student model was pretrained on SAGA22 itself; if any test transcripts appeared in that pretraining, the reported student-model gain is inflated.","fun_headline_variants_meta":{"raw":{"variants":["Tutoring AI gains 12 F1 points via classroom pretraining","Classroom talk moves transfer to improve tutor AI","Speaker-aware context boosts math tutoring dialogue models","Teaching data sharpens AI talk moves in tutoring","From classroom to tutoring: AI talk moves improve"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000187,"raw_usage":{"total_tokens":1301,"prompt_tokens":888,"completion_tokens":413,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":504,"completion_tokens_details":{"reasoning_tokens":338}},"tokens_in":504,"tokens_out":413,"duration_ms":4360,"temperature":1.0,"reasoning_tokens":338,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T13:09:58.411669+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Rerun the best student configuration with a session-level split in which the test sessions are provably excluded from all pretraining and fine-tuning data; if the 76.5 macro F1 drops substantially, the headline transfer gain is partly leakage rather than classroom-to-tutoring transfer.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the in-context-learning baseline that the paper re-implements with a different 7B model."}],"review_version":1}