MSA at BEA 2025 Shared Task: Disagreement-Aware Instruction Tuning for Multi-Dimensional Evaluation of LLMs as Math Tutors

Ali Hamdi; Baraa Hikal; Islam Oshallah; Mohamed Basem

REVIEW 4 major objections 3 minor 24 references

MSA at BEA 2025 Shared Task: Disagreement-Aware Instruction Tuning for Multi-Dimensional Evaluation of LLMs as Math Tutors

T0 review · 4 major / 3 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A single LoRA-fine-tuned Mathstral model, combined with a disagreement-aware ensemble vote that protects the minority 'To some extent' label, ranks first in Providing Guidance and top-five on every track of the BEA 2025 math-tutor…

desk verdict Solid leaderboard system that makes an unsupported ensemble-reliability claim; the rank results stand, but the contribution needs an ablation and a clarified run-selection description. read the letter →

arxiv 2505.18549 v1 pith:7RJ4CSGD submitted 2025-05-24 cs.CL

Baraa Hikal , Mohamed Basem , Islam Oshallah , Ali Hamdi This is my paper

classification cs.CL

keywords BEA2025sharedtaskAItutorevaluationinstructiontuningLoRAMathstral-7Bdisagreement-awareensemblemacro-F1minority-classrecall

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that one unified recipe—instruction-tune the open-weights math model Mathstral-7B-v0.1 with LoRA adapters per task dimension, then combine five runs through a disagreement-aware vote that favors the minority label 'To some extent'—is enough to rank among the best systems at the BEA 2025 shared task on evaluating AI math tutors. The system placed 1st in Providing Guidance, 3rd in Actionability, and 4th in both Mistake Identification and Mistake Location out of dozens of teams. If true, this matters because the field gets a cheap, reproducible baseline for grading tutor feedback: no task-specific architectures, no proprietary models, and the minority-class improvements come from inference-time arbitration rather than extra training data. The paper also implies the same pipeline could generalize across pedagogical dimensions, so automatic evaluation of AI tutors becomes a standard instruction-following problem rather than four bespoke tasks.

What carries the argument

The load-bearing mechanism is the combination of LoRA (Low-Rank Adaptation) fine-tuning of Mathstral-7B-v0.1, an open-weight 7B model specialized for mathematical reasoning, with the adapter update $\Delta W = \alpha \cdot A B$ added to frozen attention weights, and a disagreement-aware ensemble: five fine-tuned runs per track, greedy decoding, and a deterministic arbitration step that retains a class only when all runs agree, otherwise biases toward 'To some extent' to match the development set's label distribution. The arbitration exists because the official metric is macro-F1, which weights the minority class equally with 'Yes'; without it, single-model predictions collapse ambiguous cases into the majority label. LoRA adapters keep the system cheap to deploy since only adapter weights are loaded at inference.

What would settle it

Retrain the five runs with five different seeds (or with different LoRA ranks or data orders), compute the average pairwise agreement rate on the development set, and compare the disagreement-aware ensemble's macro-F1 against a single model plus a fixed prior shift toward 'To some extent.' If the runs are near-identical (pairwise agreement near 100%) or if the prior-shift baseline matches the ensemble's macro-F1, the claimed reliability gain from ensemble disagreement is not supported.

Watch

Extended reading notes

Core claim

The central claim is that a single fine-tuned model—Mathstral-7B-v0.1 adapted with LoRA—can handle all four tracks of the BEA 2025 MRBench-based shared task and that a disagreement-aware ensemble inference strategy is what makes the difference on the minority 'To some extent' class. The paper reports official leaderboard ranks of 1st in Providing Guidance (strict macro-F1), 3rd in Actionability, and 4th in both Mistake Identification and Mistake Location, with top-3 ranks on several secondary metrics. The innovation is in the inference policy: five fine-tuned runs per track are each decoded greedily; when they disagree, the system keeps predictions that preserve the development set's relative frequency of 'To some extent' instead of applying majority voting. This is presented as the mechanism that lifts minority-class recall under macro-F1 evaluation, where 'Yes' labels dominate the data.

Load-bearing premise

The whole arbitration policy assumes the five fine-tuned runs genuinely disagree in meaningful ways, but the paper fixes the random seed at 42 and does not say what varies between runs, so the disagreement signal could be near-identical outputs and the 'To some extent' preference would then be applied to noise.

Editorial extensions

If this is right

The same LoRA-tuned Mathstral backbone, with no track-specific architecture changes, reaches 1st place in Providing Guidance, 3rd in Actionability, and 4th in Mistake Identification and Mistake Location, so the unified pipeline transfers across all four pedagogical dimensions.
The disagreement-aware ensemble improves coverage of the minority 'To some extent' label, which is key to macro-F1 ranking because 'Yes' dominates the development set.
Only LoRA adapter weights are needed for the five ensemble members, so the added inference cost is a sequential pass of lightweight adapters rather than five full-model deployments.
Because model selection and early stopping are done on development macro-F1, the same recipe can be run by any team with a single GPU node and open weights.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Retraining the five runs with varied seeds or data orders and computing pairwise agreement would test whether the disagreement signal is substantive, since the paper fixes seed 42 but calls the runs independent.
A prior-shift-only baseline—a single model whose probability outputs are reweighted to the dev 'To some extent' frequency—would isolate whether the ensemble adds macro-F1 value beyond the label-shift policy.
The lenient protocol's merging of 'Yes' and 'To some extent' implies the strict-metric gains could come from boundary shifts; a per-class error analysis would reveal whether the ensemble improves true minority recall or just re-prioritizes borderline cases.
If the mechanism transfers, any three-way annotation task with a dominant majority class could reuse the unified instruction-template plus disagreement arbitration recipe.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Quick take: this is a decent shared-task system description with external validation—first in Providing Guidance, top-five across the board—but the methodological hook, disagreement-aware ensemble inference, is not actually quantified anywhere. The leaderboard numbers are what they are; the reliability claim about the ensemble doesn't have a single comparative statistic behind it.

What's new and good: the authors fine-tune one open 7B model (Mathstral) with LoRA on all four BEA 2025 tracks without any task-specific architecture changes. That's a clean, reproducible engineering contribution. The writing is clear, and the limitations section is honest—they explicitly note that ensembles are only useful if the members are diverse, which is exactly the thing they don't demonstrate. The shared task's blind test set gives external validity to the ranking claims.

Where it gets soft: the abstract and Section 3.4 claim a disagreement-aware policy that improves minority 'To some extent' coverage and prediction reliability. But Table 1 lists per-run macro-F1 only; there's no ensemble row, no ensemble-versus-single-model ablation, and no test-set tie between the submitted predictions and any aggregation method. So the leaderboard ranks could as easily come from picking the best single run. Figure 3 shows label distributions, but that's not a performance metric.

Compounding this, Section 3.3 says training used 'fixed seed 42 for reproducibility,' while Section 3.4 says 'five independent models per track.' Those two statements conflict unless something else varied, and the paper doesn't say what. If the five runs are near-identical, the disagreement signal is noise and the filtering rule is just undoing noise. There's also a minor count mismatch: the Providing Guidance rows in Table 1 show four runs, not five.

None of this invalidates the ranking results, which were obtained through the official evaluation server. But the stated contribution, the disagreement-aware ensemble, is unsupported as written. This is a moderate issue, not a fatal one: the authors can address it with a simple ablation, and they should be asked for it.

Bottom line: this paper deserves peer review. It's a useful system description for the educational NLP community, with genuine leaderboard results and an open model. The referee should require the ensemble ablation and a clarification of what differs across the five runs. If that is fixed, the paper is a fine archival write-up. If the ablation shows no ensemble benefit, the authors can still keep the paper, just with a less ambitious claim.

Who it's for: anyone working on automatic evaluation of AI tutors, shared task participants, and people using LoRA fine-tuning for educational classification.

Referee Report

4 major / 3 minor

Summary. The paper describes MSA-MathEval, the authors' submission to the BEA 2025 Shared Task on evaluating AI tutor responses along four dimensions: Mistake Identification, Mistake Location, Providing Guidance, and Actionability. The method is a unified LoRA fine-tuning pipeline applied to Mathstral-7B-v0.1 for all four tracks, followed by a disagreement-aware ensemble inference strategy that is intended to preserve minority-class predictions, especially for the 'To some extent' label. The authors report strong leaderboard results, including 1st place in Providing Guidance, 3rd in Actionability, and 4th in both Mistake Identification and Mistake Location. The central methodological claim is that disagreement-aware ensemble inference improves prediction reliability, but the paper does not provide a quantitative comparison between ensemble output and single-model output.

Significance. If the results hold, the paper demonstrates that a single instruction-tuned open-weight model with a lightweight ensemble policy can rank at or near the top of a multi-dimensional educational evaluation benchmark. The leaderboard ranks are externally validated on a blind test set, which is a genuine strength, and the training configuration is described with enough detail to be reproduced. The paper also honestly lists limitations concerning domain specialization, ensemble diversity, and the coarseness of macro-F1. However, the distinctive contribution advertised in the abstract and introduction—disagreement-aware ensemble inference—is not actually substantiated by any ensemble-versus-single-model evaluation. The main empirical tables report per-run scores only, and no row or statistic identifies the ensemble's macro-F1 or shows that the submitted predictions came from the ensemble rather than from a selected single run. This gap is load-bearing because the leaderboard ranks alone cannot distinguish the effect of ensembling from the effect of the underlying fine-tuned model.

major comments (4)

[§3.3–§3.4, Table 1, Figure 3] The abstract and Section 3.4 claim that disagreement-aware ensemble inference improves prediction reliability, but no single-model-versus-ensemble macro-F1 ablation is reported anywhere. Table 1 lists only per-run scores; Figure 3 shows label distributions but no F1 values; Section 4.3 discusses improved minority coverage without quantifying any classification gain. In addition, Section 3.3 states that training used 'fixed seed 42 for reproducibility,' while Section 3.4 says 'we fine-tuned five independent models per track.' If the seed is fixed and all other settings are identical, the runs are not independent, and the disagreement signal is either unreported variation or noise. Please specify what varies across the five runs, add an explicit ensemble row to Table 1 comparing ensemble macro-F1 with each constituent run, and state which aggregation method produced the official test-set predictions.
[§3.4, §4.3] The ensemble filtering policy is explicitly designed to preserve the development-set frequency of 'To some extent': when the ensemble disagrees, the policy 'prefer[s] predictions that preserve the relative frequency of "To some extent" observed in the development set.' Therefore, the observation in Section 4.3 and Figure 3 that the ensemble's label distribution better matches the development distribution is partly definitional, not independent evidence of improved classification. The metric that matters for the shared task is macro-F1, and no ensemble macro-F1 is provided. Please report strict and lenient macro-F1 for the ensemble output versus every single run, and distinguish the effect of the filtering rule from the effect of simply rebalancing labels to match the development set.
[§5.1, Table 1, §5.2, Table 2] The relationship between the reported runs and the leaderboard ranks is not established. Section 5.1 says 'five independent fine-tuning runs per track,' but Table 1 lists only four runs for Providing Guidance. More importantly, Section 5.2 presents leaderboard ranks without stating which prediction set—aggregated ensemble or a particular single run—was submitted. Since Section 5.1 highlights individual run scores (e.g., Track 1 Run 1, Track 2 Run 5), the rank results could equally be explained by selecting the best single run. The paper must state explicitly which run or which aggregation produced the official test predictions, and it must show that this submission outperformed the constituent single runs on the development set.
[§6] The limitations section concedes that the ensemble benefit 'may diminish if the base models exhibit correlated predictions,' and it cites prior work on diversity in ensembles, but the paper does not measure diversity across the five runs. Given that Section 3.3 fixes the seed and Section 3.4 calls the models independent, the diversity assumption is exactly what needs to be tested. Please report a diversity measure (e.g., pairwise agreement or predicted-label overlap) and relate it to the claimed ensemble gain.

minor comments (3)

[Abstract, §1] The system name is rendered inconsistently as 'MSA-MathEval' in the abstract and 'MSA-M ATHEVAL' in Section 1; use a single consistent spelling throughout.
[References] The reference for Mathstral appears as 'Math σtral 7b v0.1' with a Greek-letter artifact, and the LLaMA 2 entry lists 'Hugo Touvron and 1 others.' Please correct these citation entries before publication.
[§4.2, Eq. (2)] Equation (2) defines macro-F1 as an unweighted average, but it would be clearer to state explicitly that the sum runs over the three class-specific F1 scores, matching the three-way label scheme used elsewhere.

Circularity Check

1 steps flagged · score 6.0 of 10

Ensemble's dev-set minority-coverage benefit is fixed by the policy; leaderboard remains externally scored.

self definitional [Section 3.4 (Inference and Ensemble Strategy); Section 4.3 (Effect of Ensemble Disagreement on Label Distribution)]
"If the ensemble disagrees, we analyze the class distribution and prefer predictions that preserve the relative frequency of "To some extent" observed in the development set. ... The ensemble strategy improves minority-class coverage—especially for "To some extent"—by better matching the development distribution and mitigating dominant-class bias."

The disagreement filter is defined to target the dev-set relative frequency of "To some extent" whenever the five runs disagree. Section 4.3 then reports, as an empirical benefit, that the ensemble 'better matches the development distribution' and improves minority-class coverage on the dev set. That outcome is guaranteed by construction: the policy explicitly optimizes the same quantity that is later measured as the result. The paper presents this as evidence that the ensemble strategy works, but the dev-set improvement is a definitional consequence of the filtering rule, not an independent validation. The external leaderboard rankings, obtained from the shared-task server on blind test labels, are not part of this circular step.

full rationale

The leaderboard results in Tables 1 and 2 come from an external blind test set scored through the shared-task server, so the reported ranks (1st in Providing Guidance, top-5 elsewhere) are not circular. No test-set prediction is reconstructed from a fitted parameter. The one genuine circular step is the supporting analysis of the ensemble's benefit: Section 3.4 defines the disagreement policy to prefer 'To some extent' so as to preserve its dev-set relative frequency, and Section 4.3 then claims the ensemble improves minority-class coverage by matching the dev distribution. The measured quantity and the optimized target are the same, making that sub-claim self-definitional. Separately, the manuscript under-specifies the ensemble: Section 3.3 states 'fixed seed 42 for reproducibility' while Sections 3.4 and 5.1 say 'five independent models per track,' and Table 1 lists only four Providing Guidance runs; Section 6 also concedes the benefit may diminish if base-model predictions are correlated. These are evidentiary gaps about whether the runs truly differ, not additional circularity. Overall, the central leaderboard claim is independent and externally validated, but the paper's central methodological claim about disagreement-aware minority-coverage improvement rests on a dev-set comparison that reduces to the policy's own definition, hence partial circularity.

Assumptions & free parameters 5 free parameters · 4 assumptions · 0 invented entities

The system depends on standard hyperparameters, a domain-specific base model, and a hand-tuned ensemble policy. No new entities or theoretical constructs are introduced, and the central results rest on external benchmark evaluation rather than fitted derivations.

free parameters (5)

LoRA rank r = 64
Hand-chosen rank for low-rank adapters; not swept or justified beyond standard practice.
LoRA scaling alpha = 2.0
Hand-chosen scaling factor for the LoRA update.
Learning rate = 4e-5
AdamW learning rate set by the authors; likely from standard ranges.
Batch size and max steps = batch size 2, 500 steps
Hand-chosen training budget.
Disagreement filtering rule = prefer 'To some extent' when ensemble disagrees
Policy chosen to match development-set label frequency; it directly shapes predictions and is not derived from first principles.

assumptions (4)

domain assumption Dev set labels are reliable expert gold annotations and the three-way rubric is meaningful.
All training and validation use the shared task development set as ground truth (Sections 3.1, 4.1).
domain assumption Mathstral-7B-v0.1 is a suitable base model for this classification task.
Model choice is justified by math benchmark scores, not by a task-specific experiment (Section 3.2).
domain assumption Macro-F1 is the right objective and matching dev label distribution is a valid inference heuristic.
The ensemble policy is built around this assumption (Sections 3.4, 4.3); the paper itself notes this metric lacks granularity for pedagogical severity.
standard math The LoRA low-rank update formula is a standard result.
Equation (1) is taken from prior work on parameter-efficient fine-tuning.

how reviews work

0 comments

Cite this review

Pith. "Pith review of MSA at BEA 2025 Shared Task: Disagreement-Aware Instruction Tuning for Multi-Dimensional Evaluation of LLMs as Math Tutors." pith.science (2026). https://pith.science/paper/7RJ4CSGD

@misc{pith2026250518549,
  author       = {Pith},
  title        = {Pith review of: MSA at BEA 2025 Shared Task: Disagreement-Aware Instruction Tuning for Multi-Dimensional Evaluation of LLMs as Math Tutors},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/7RJ4CSGD}},
  note         = {Machine review of arXiv:2505.18549}
}

read the original abstract

We present MSA-MathEval, our submission to the BEA 2025 Shared Task on evaluating AI tutor responses across four instructional dimensions: Mistake Identification, Mistake Location, Providing Guidance, and Actionability. Our approach uses a unified training pipeline to fine-tune a single instruction-tuned language model across all tracks, without any task-specific architectural changes. To improve prediction reliability, we introduce a disagreement-aware ensemble inference strategy that enhances coverage of minority labels. Our system achieves strong performance across all tracks, ranking 1st in Providing Guidance, 3rd in Actionability, and 4th in both Mistake Identification and Mistake Location. These results demonstrate the effectiveness of scalable instruction tuning and disagreement-driven modeling for robust, multi-dimensional evaluation of LLMs as educational tutors.

Figures

Figures reproduced from arXiv: 2505.18549 by the authors.

**Figure 1.** Overview of our unified MSA-MATHEVAL framework for the BEA 2025 Shared Task. The pipeline includes preprocessing, LoRA-based fine-tuning of Mathstral-7B-v0.1, and disagreement-aware ensemble inference. 2 Related Work Evaluating the pedagogical capabilities of AI tutors builds upon long-standing research in intelligent tutoring systems (ITS) and more recent advances in large language models (LLMs). Early ITS such as … view at source ↗

**Figure 2.** LoRA adaptation adds trainable low-rank matrices A and B to frozen attention weights W0, producing an effective weight W = W0 + αAB during training. Only A and B are updated, enabling memoryefficient fine-tuning (Hu et al., 2021). Training was capped at 500 steps with gradient norm clipping (∥g∥2 < 1.0) and a maximum sequence length of 2048 tokens. We used a batch size of 2, single micro-batching, and fixed seed … view at source ↗

**Figure 3.** Label distribution comparison across tracks and systems. Each group shows the percentage of predictions [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

24 extracted references · 4 canonical work pages

[1]

online" 'onlinestring :=

ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...
[2]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...
[3]

Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. 2021. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168

arXiv 2021
[4]

Nico Daheim, Jakub Macina, Manu Kapur, Iryna Gurevych, and Mrinmaya Sachan. 2024. https://doi.org/10.18653/v1/2024.emnlp-main.478 Stepwise verification and remediation of student reasoning errors with large language model tutors . In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 8386--8411, Miami, Florida, U...

doi:10.18653/v1/2024.emnlp-main.478 2024
[5]

Dorottya Demszky, Jing Liu, Zid Mancenido, Julie Cohen, Heather Hill, Dan Jurafsky, and Tatsunori Hashimoto. 2021. https://doi.org/10.18653/v1/2021.acl-long.130 Measuring conversational uptake: A case study on student-teacher interactions . In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th Internationa...

doi:10.18653/v1/2021.acl-long.130 2021
[6]

Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. 2023. Qlora: Efficient finetuning of quantized llms. Advances in neural information processing systems, 36:10088--10115

2023
[7]

Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021 a . https://arxiv.org/abs/2009.03300 Measuring massive multitask language understanding . In International Conference on Learning Representations ( ICLR )

arXiv 2021
[8]

Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. 2021 b . https://arxiv.org/abs/2103.03874 Measuring mathematical problem solving with the MATH dataset . In Advances in Neural Information Processing Systems (NeurIPS) Datasets and Benchmarks Track

arXiv 2021

Show all 24 references

[9]

Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Weizhu Wang, and Zichao Chen

Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Weizhu Wang, and Zichao Chen. 2021. https://arxiv.org/abs/2106.09685 Lora: Low-rank adaptation of large language models . arXiv preprint arXiv:2106.09685

2021 arXiv
[10]

Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, L\' e lio Renard-Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Tho...

2023 arXiv
[11]

Ekaterina Kochmar, Kaushal Kumar Maurya, Kseniia Petukhova, KV Aditya Srivatsa, Ana \"i s Tack, and Justin Vasselli. 2025. Findings of the BEA 2025 shared task on pedagogical ability assessment of AI -powered tutors. In Proceedings of the 20th Workshop on Innovative Use of NLP...

2025
[12]

Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell. 2017. Simple and scalable predictive uncertainty estimation using deep ensembles. In Advances in Neural Information Processing Systems 30 (NeurIPS 2017), pages 6402--6413

2017
[13]

Jakub Macina, Nico Daheim, Sankalan Chowdhury, Tanmay Sinha, Manu Kapur, Iryna Gurevych, and Mrinmaya Sachan. 2023. https://doi.org/10.18653/v1/2023.findings-emnlp.372 M ath D ial: A dialogue tutoring dataset with rich pedagogical properties grounded in math reasoning problems...

2023 doi
[14]

Kaushal Kumar Maurya, Kv Aditya Srivatsa, Kseniia Petukhova, and Ekaterina Kochmar. 2025. https://aclanthology.org/2025.naacl-long.57/ Unifying AI tutor evaluation: An evaluation taxonomy for pedagogical ability assessment of LLM -powered AI tutors . In Proceedings of the 2025...

2025
[15]

Mistral AI Team . 2024. Math tral 7b v0.1: A math reasoning and scientific discovery model. https://mistral.ai/news/mathstral

2024
[16]

Nye, Arthur C

Benjamin D. Nye, Arthur C. Graesser, and Xiangen Hu. 2014. https://doi.org/10.1007/s40593-014-0029-5 Autotutor and family: A review of 17 years of natural language tutoring . International Journal of Artificial Intelligence in Education, 24(4):427--469

2014 doi
[17]

OpenAI . 2022. Chatgpt: Optimizing language models for dialogue. https://openai.com/blog/chatgpt

2022
[18]

OpenAI . 2023. GPT-4 technical report. arXiv preprint arXiv:2303.08774

2023 arXiv
[19]

Ana \"i s Tack and Chris Piech. 2022. https://doi.org/10.5281/zenodo.6853187 The AI teacher test: Measuring the pedagogical ability of blender and GPT-3 in educational dialogues . In Proceedings of the 15th International Conference on Educational Data Mining, pages 522--529, D...

2022 doi
[20]

Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timoth\' e e Lacroix, Baptiste Rozi\`ere, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023 a . LLaMA : Open and efficient fou...

2023 arXiv
[21]

Hugo Touvron and 1 others. 2023 b . Llama 2 : Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288

2023 arXiv
[22]

Rose Wang, Qingyang Zhang, Carly Robinson, Susanna Loeb, and Dorottya Demszky. 2024. https://doi.org/10.18653/v1/2024.naacl-long.120 Bridging the novice-expert gap via models of decision-making: A case study on remediating math mistakes . In Proceedings of the 2024 Conference ...

2024 doi
[23]

Lingling Xu, Haoran Xie, Si-Zhao Joe Qin, Xiaohui Tao, and Fu Lee Wang. 2023. Parameter-efficient fine-tuning methods for pretrained language models: A critical review and assessment. arXiv preprint arXiv:2312.12148

2023 arXiv
[24]

Da-Wei Zhou, Zi-Wen Cai, Han-Jia Ye, De-Chuan Zhan, and Ziwei Liu. 2023. Revisiting class-incremental learning with pre-trained models: Generalizability and adaptivity are all you need. arXiv preprint arXiv:2303.07338

2023 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.

[1] [1]

online" 'onlinestring :=

ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

[2] [2]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

[3] [3]

Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman. 2021. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168

arXiv 2021

[4] [4]

Nico Daheim, Jakub Macina, Manu Kapur, Iryna Gurevych, and Mrinmaya Sachan. 2024. https://doi.org/10.18653/v1/2024.emnlp-main.478 Stepwise verification and remediation of student reasoning errors with large language model tutors . In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 8386--8411, Miami, Florida, U...

doi:10.18653/v1/2024.emnlp-main.478 2024

[5] [5]

Dorottya Demszky, Jing Liu, Zid Mancenido, Julie Cohen, Heather Hill, Dan Jurafsky, and Tatsunori Hashimoto. 2021. https://doi.org/10.18653/v1/2021.acl-long.130 Measuring conversational uptake: A case study on student-teacher interactions . In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th Internationa...

doi:10.18653/v1/2021.acl-long.130 2021

[6] [6]

Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. 2023. Qlora: Efficient finetuning of quantized llms. Advances in neural information processing systems, 36:10088--10115

2023

[7] [7]

Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021 a . https://arxiv.org/abs/2009.03300 Measuring massive multitask language understanding . In International Conference on Learning Representations ( ICLR )

arXiv 2021

[8] [8]

Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt. 2021 b . https://arxiv.org/abs/2103.03874 Measuring mathematical problem solving with the MATH dataset . In Advances in Neural Information Processing Systems (NeurIPS) Datasets and Benchmarks Track

arXiv 2021