FEAT: A Preference Feedback Dataset through a Cost-Effective Auto-Generation and Labeling Framework for English AI Tutoring

Hyein Seo; Sangkeun Jung; Taewook Hwang; Yohan Lee

REVIEW 4 major objections 6 minor 54 references

FEAT: A Preference Feedback Dataset through a Cost-Effective Auto-Generation and Labeling Framework for English AI Tutoring

T0 review · 4 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A mostly LLM-generated preference dataset for English tutoring feedback, augmented with just 5-10% human-annotated pairs, trains ranking models that match or beat training on the full human-annotated set.

desk verdict Useful datasets and framework, but the headline 5–10% DM superiority claim is an overgeneralization from one model with within-noise margins. read the letter →

arxiv 2506.19325 v2 pith:46ERXH3S submitted 2025-06-24 cs.AI

Hyein Seo , Taewook Hwang , Yohan Lee , sangkeun Jung This is my paper · ORCID

classification cs.AI

keywords teacherfeedbackgenerationpreferencedatasetEnglishAItutoringLLMannotationlearningtorankrank-biasedoverlapdataaugmentationcriteria

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that high-quality preference data for training feedback-ranking models in English AI tutoring does not need to be fully human-annotated. It proposes FEAT, a pipeline in which an LLM generates candidate teacher feedback with and without a set of five pedagogical criteria, and the criteria-present version is automatically labeled as the preferred feedback. The resulting LLM-only dataset (DG) produces rankings that are competitive with those from a fully human-annotated dataset (DM) when used to train pairwise rankers. The paper's main empirical claim is that mixing just 5-10% human-annotated DM pairs into the DG pool (DA) yields ranking models whose agreement with human rankings exceeds that of models trained on 100% DM. If true, this substantially lowers the cost of building feedback-ranking systems for tutoring while matching or improving quality.

What carries the argument

The central object is the FEAT dataset-construction pipeline, specifically its automatic labeling rule: for each tutoring prompt, the LLM produces two feedback responses, one generated with five pedagogical criteria (Correct, Revealing, Guidance, Diagnostic, Encouragement) and one generated without them; the criteria-conditioned response is labeled chosen and the other rejected, forming preference pairs without human annotation. A second component is the augmentation schedule: sampling a small proportion (5-100%) of human-ranked DM pairs into the DG pool to form DA, and training pairwise ranking models on the mixed data. The chain of the argument runs from the criteria-aware generation, through the automatic chosen/rejected labeling, to the measured rank-biased overlap (RBO) agreement on the human-ranked DM test set.

What would settle it

Have human annotators rank a sample of DG pairs: if a meaningful share of the 'rejected' (no-criteria) responses is judged equal or better than the paired 'chosen' (criteria) response, the automatic labels are unreliable. Then retrain the DA-to-DM models with the DG chosen/rejected labels replaced by those human judgments; if the 5-10% advantage over DM-to-DM disappears, the reported gain comes from the labeling convention rather than feedback quality.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is that a small amount of human preference signal can be leveraged much further than usual: augmenting LLM-generated preference pairs with 5-10% human-ranked examples produces a training set whose resulting rankers generalize better to the human test set than rankers trained on the entire human-annotated dataset. Across the ranking approaches tested (binary classifier, reward model, DPO, RankNet, and an ensemble), the DA-trained models reached or surpassed the DM-to-DM baseline, with the effect clearest on Llama-3B-IT, where the binary classifier, DPO, and ensemble exceeded the full-DM baseline with only 5% DM. The paper also reports that DG alone, without any human data, stays close to the human-trained baseline, and that using all five feedback criteria rather than two improves ranking generalization for most models.

Load-bearing premise

The load-bearing premise is that feedback generated with the five criteria is better than feedback generated without them, so labeling the former chosen and the latter rejected gives correct training labels; the paper reports no human verification of these automatic labels.

Editorial extensions

If this is right

Training feedback rankers for tutoring can move from fully human-annotated preferences to a mostly LLM-generated pool with a 5-10% human core, cutting annotation cost by roughly 90-95% for the same or better measured ranking agreement.
LLM-only preference data is a viable fallback: DG-trained rankers stay within roughly 0.04-0.12 RBO of DM-trained ones across the tested 1B and 3B models and approaches.
Using richer feedback criteria during generation transfers to ranking quality: switching from two to five criteria raised DPO's RBO by up to 11.41% on Llama-1B and generally improved most approaches.
The same construction pattern (criteria-aware generation, automatic pairwise labeling, small human augmentation) can be applied to other educational tasks, such as hint generation or distractor ranking, without re-collecting large human-preference sets.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

One testable extension: the 5-10% advantage may partly come from the DG pair structure, where chosen and rejected responses differ by whether the criteria prompt appeared, so a ranker could learn to detect prompt-format cues rather than feedback quality; regenerating DG with both responses criteria-based and human-labeled would separate these effects.
If the labeling assumption survives a human audit, the result suggests a general 'small human anchor' recipe: massive automatically contrasted pairs plus a tiny human-ranked core can dominate much larger human-only training sets, because the LLM supplies coverage and the human core corrects systematic bias; this may transfer to other domains with scarce pairwise preference data.
The paper evaluates agreement with human rankings of five fixed feedback candidates; a stronger downstream test is whether DA-trained rankers improve actual tutoring outcomes, such as student learning gains or preference in live teacher-student interaction, since RBO with annotator rankings is a proxy for feedback quality.
The model comparison hints that the optimal augmentation ratio is model-dependent: Qwen-3B-IT needed 50-75% DM to beat its DM-to-DM baseline while Llama-3B-IT did so at 5-10%, so the headline ratio should be re-tuned per base model rather than treated as universal.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Hey — quick read of FEAT. The dataset release is the thing: three preference datasets for English tutoring feedback, built through a cost-effective LLM generation and labeling pipeline, with external evaluation on human-annotated rankings. That's a real contribution. But the headline claim — that adding 5–10% human-annotated DM to LLM-generated DG beats 100% DM — is an overgeneralization. The stress-test note is right: only Llama-3B-IT shows the effect across multiple methods, and even there the margins (0.006–0.022 RBO) sit inside the DM→DM baseline's standard deviation (up to 0.056). Qwen-3B-IT needs 50–75% DM. The abstract states it as a general result, and it isn't one.

What makes the paper worth reading despite that: the evaluation design is not circular. Models trained on DG or DA are tested against human-annotated DM rankings, so the transfer result is meaningful. The framework is a sensible combination of known pieces — LLM generation with criteria, human ranking, hybrid augmentation — and the authors release the data. That's reproducible work. The paper also owns its limitations: only 1B–3B models, only MCTest-derived scenarios, pairwise ranking only.

The soft spots, in order. First, the DG labels are constructed, not validated: w/ criteria feedback is called chosen and w/o criteria rejected, on the assumption that criteria-based feedback is better. No human evaluation of those labels is reported. If that assumption is wrong, the DG signal is systematically mislabeled. Second, the RBO metric on five candidate lists has a random baseline of ~0.4, so deltas of 0.01–0.02 are a small fraction of usable scale. Table 7 gives standard deviations, but Figure 6 omits error bars for the DA condition, so you can't tell if those differences are noise. The paper needs significance testing or confidence intervals before the 5–10% conclusion is stated as established.

Bottom line: the datasets and framework deserve referee time and a place in the educational NLP literature. The empirical claim needs to be tempered to 'can' rather than 'leads to' until the evidence is stronger. I'd send it to review, with a request for tighter statistics and a reworded abstract.

Referee Report

4 major / 6 minor

Summary. The paper introduces FEAT, a framework for cost-effectively building teacher-feedback preference datasets for English AI tutoring. Three datasets are constructed: DIRECT-Manual (DM), where feedback candidates from five sources are ranked by humans; DIRECT-Generated (DG), where LLMs generate feedback with or without five pedagogical criteria and the criteria-based output is automatically labeled as chosen; and DIRECT-Augmented (DA), which mixes DG with small amounts of human-annotated DM. The authors train five ranking approaches (binary classifier, reward model, DPO, RankNet, and an ensemble) on five open-weight 1B/3B models and evaluate them against human-annotated DM rankings using RBO. The central claim is that adding only 5-10% DM to DG yields better ranking performance than using 100% DM alone.

Significance. If the central claim were robust, FEAT would offer a practical way to reduce human annotation costs for building preference models in tutoring, and the public release of the three datasets would be a useful community resource. The paper's evaluation is externally grounded: models trained on DG or DA are tested against human-annotated DM rankings, so the main result is not a tautology. Additional strengths include the breadth of ranking methods and base models tested, the explicit reporting of multiple-seed standard deviations for the DM->DM baseline (Table 7), and honest limitations covering model scale, pairwise-only training, and the single MCTest domain. The main weakness is that the headline 5-10% claim rests on a small number of model-method combinations, with margins that are frequently inside the reported baseline noise; the paper also relies on an unvalidated assumption for the DG chosen/rejected labels.

major comments (4)

[Abstract and Section 5] The claim that 'incorporating a small portion of DM (5-10%) into DG leads to superior performance compared to using 100% DM alone' is not supported across the reported models. In Figure 6, only Llama-3B-IT shows 5% DA above the DM->DM baseline for a plurality of methods (binary classifier 0.783 vs 0.773, DPO 0.792 vs 0.770, ensemble 0.804 vs 0.798), and even these margins are tiny. Qwen-3B-IT is below the baseline at 5% for every method (e.g., binary classifier 0.787 vs 0.806, ensemble 0.797 vs 0.811), and Figure 13 shows Llama-1B, Llama-1B-IT, and Llama-3B generally reaching or exceeding the baseline only at higher DM ratios. The abstract and conclusion should either qualify this statement to the specific model(s) and method(s) for which it holds or provide a significance test aggregated across models and methods.
[Figure 6 and Table 7] The DA->DM results in Figure 6 are plotted without error bars, while Table 7 reports one-standard-deviation widths for the DM->DM baseline that reach 0.056 for Llama-3B binary classifier and 0.023 for Llama-3B-IT DPO. The reported 5% DA improvements are 0.006-0.022, which is within one baseline standard deviation in every instance. To support a superiority claim, the authors should report per-seed results with error bars for the DA conditions and run paired significance tests (e.g., across the five shared seeds) against the DM->DM baseline. The challenge is compounded by the metric: RBO on five-candidate lists has a chance baseline of about 0.4, as the paper itself notes in Section 4.2, so a 0.01-0.02 RBO difference occupies only a small fraction of the usable scale between chance and perfect agreement.
[Section 2.2] The DG preference labels are constructed by labeling feedback generated with five criteria as chosen and feedback generated without criteria as rejected, 'assuming criteria-based feedback is of higher quality.' This assumption is load-bearing: DG->DM and DA->DM transfer are central results, and if the assumption is wrong, the training signal is systematically mislabeled and the apparent transfer could reflect the prompt-format distinction between the two generation conditions rather than feedback quality. The paper reports no human evaluation of DG chosen/rejected pairs. I request either a human validation sample of DG pairs or a control experiment (e.g., training on reversed labels) to demonstrate that the learned preference is driven by pedagogic quality rather than by the presence or absence of the criteria instruction.
[Section 4.4] The statement that 'incorporating richer feedback information enhances model generalization' is an overgeneralization of the criteria-ablation results. Figure 7 shows mixed directions across models: Llama-1B improves on all approaches, but Llama-1B-IT shows a decrease for binary classifier (-2.00%) and RankNet (-3.88%), and Llama-3B-IT shows decreases for binary classifier (-1.96%) and RankNet (-2.26%). Given the small number of model-method conditions and the lack of significance testing, the conclusion should be limited to the specific conditions that improve, or supported by a paired test across the five seeds.

minor comments (6)

[Abstract] The abstract contains a punctuation error: 'lower quality;, and' should read 'lower quality; and'.
[Figure 3] The ranking shown in Figure 3 lists 'Tutor' as the lowest-ranked candidate, but the candidate list labels the same feedback as 'DIRECT'; these labels should be made consistent.
[Table 5] The model name 'Qwen-3B' in Table 5 with version 'Qwen2.5-3B-Instruct' is inconsistent with the 'Qwen-3B-IT' label used throughout the text; use one unified name.
[Table 7] Table 7 states 'Best results are highlighted in bold,' but the boldface formatting is not visible in the manuscript text; either render the boldface or remove the note.
[Section 4.3] There is a typographical spacing issue in the line 'DM →DM'; the arrow should be formatted consistently as 'DM→DM'.
[References] The references list Scarlatos et al. 2024b and 2024c with the same title 'Improving the validity of automatically generated feedback via reinforcement learning'; these should be merged or clearly disambiguated by venue.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: the central claim is tested against held-out human-annotated DM rankings, not defined by the training labels.

full rationale

The paper's core experimental claim is that training on DG (LLM-generated, LLM-labeled) plus a small portion of DM (human-LLM generated, human-ranked) outperforms training on the full DM set, when both are evaluated on the held-out DM test set. This evaluation is external to the training labels: DG/DA models are scored against human preference rankings on DM, which are not used to construct the DG labels. The DG labeling rule (Section 2.2) — "We labeled w/ criteria feedback as chosen and w/o criteria as rejected, assuming criteria-based feedback is of higher quality" — is an assumption about label quality, but it is not circular because the empirical test checks whether that assumption transfers to independent human judgments. The self-citations (Seo et al. 2025 for feedback criteria; Huang et al. 2023 and Liermann et al. 2024 for the DIRECT dataset) provide design choices and dataset provenance, but they do not force the experimental result: the ranking models must generalize to new feedback texts in the DM test set, and the reported RBO scores give non-trivial evidence. The claim that 5–10% DM in DA outperforms 100% DM alone is an empirical outcome; while it may be overgeneralized from one model (Llama-3B-IT) and lack significance testing, that is a robustness/correctness concern, not a circularity concern. The paper does not define its predictions in terms of its inputs, nor does it fit any parameter to the test set. The only mild self-reference is the use of the authors' own criteria as a generation guideline, but the evaluation does not reduce to that citation; therefore no circular step can be exhibited.

Assumptions & free parameters 2 free parameters · 5 assumptions · 0 invented entities

No new theoretical entities are introduced; DM, DG, and DA are datasets, not postulated mechanisms, and DA is released publicly. The central claims rest on imported assumptions about feedback criteria, label construction, scenario validity, and evaluation metric choice, plus the swept DM mixing ratio.

free parameters (2)

DM proportion in DA = 5-10% (best window in sweep)
The headline result is a range read off Figures 6 and 13 by sweeping DM ratios from 0% to 100%; it is not derived from a cost-quality model and varies across models and methods.
Number of feedback criteria = 5 (vs 2 in Section 4.4)
The criteria count is a design choice imported from Seo et al. (2025); Section 4.4 compares 2 versus 5 criteria and finds mixed effects depending on the model, so the 5-criterion choice is not independently justified by the data.

assumptions (5)

domain assumption The five FEAT criteria (Correct, Revealing, Guidance, Diagnostic, Encouragement) define educationally effective teacher feedback.
Invoked in Section 2 and in DG prompts; no evidence in this paper links these criteria to learning outcomes, and they are inherited from the authors' earlier work (Seo et al., 2025).
ad hoc to paper Feedback generated with criteria is better than feedback generated without criteria.
Section 2.2 assigns chosen and rejected labels by this rule; Table 4 shows qualitative examples, but no human evaluation quantifies the assumed superiority.
domain assumption MCTest items converted to story, question, correct answer, and one randomly selected wrong answer are a valid stand-in for teacher-student tutoring dialogues.
Section 2.2 and Appendix C.1; DG has no multi-turn dialogue history, so the domain gap to DM may affect transfer results.
domain assumption Human-ranked feedback in DM is a valid ground truth for feedback quality.
DM rankings are made by human annotators using Correct and Revealing, but no inter-annotator agreement or adjudication process is reported.
domain assumption RBO similarity on five-item ranking lists is an appropriate evaluation metric.
The authors use RBO and note the random baseline is at least 0.4, but do not adjust for chance or report significance.

how reviews work

0 comments

Cite this review

Pith. "Pith review of FEAT: A Preference Feedback Dataset through a Cost-Effective Auto-Generation and Labeling Framework for English AI Tutoring." pith.science (2026). https://pith.science/paper/46ERXH3S

@misc{pith2026250619325,
  author       = {Pith},
  title        = {Pith review of: FEAT: A Preference Feedback Dataset through a Cost-Effective Auto-Generation and Labeling Framework for English AI Tutoring},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/46ERXH3S}},
  note         = {Machine review of arXiv:2506.19325}
}

read the original abstract

In English education tutoring, teacher feedback is essential for guiding students. Recently, AI-based tutoring systems have emerged to assist teachers; however, these systems require high-quality and large-scale teacher feedback data, which is both time-consuming and costly to generate manually. In this study, we propose FEAT, a cost-effective framework for generating teacher feedback, and have constructed three complementary datasets: (1) DIRECT-Manual (DM), where both humans and large language models (LLMs) collaboratively generate high-quality teacher feedback, albeit at a higher cost; (2) DIRECT-Generated (DG), an LLM-only generated, cost-effective dataset with lower quality;, and (3) DIRECT-Augmented (DA), primarily based on DG with a small portion of DM added to enhance quality while maintaining cost-efficiency. Experimental results showed that incorporating a small portion of DM (5-10%) into DG leads to superior performance compared to using 100% DM alone.

Figures

Figures reproduced from arXiv: 2506.19325 by the authors.

**Figure 2.** The architecture of the FEAT framework, illustrating the construction process of the DIRECT-Manual, [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗

**Figure 4.** Sample from the MCTest. ties to prefer chosen feedback, using log probability differences between chosen and rejected pairs. RankNet (Burges et al., 2005) learns score differences between feedback pairs using Binary CrossEntropy loss. Reward Model, DPO, and RankNet share the same prompt, shown in [PITH_FULL_IMAGE:figures/full_fig_p003_4.png] view at source ↗

Figures from the paper (11 more)

**Figure 5.** Figure 5: Ranking model performance across different approaches (with 5-seed standard deviation). Lines indicate [PITH_FULL_IMAGE:figures/full_fig_p004_5.png]

**Figure 6.** Figure 6: Llama-3B-IT and Qwen-3B-IT performance in the DA [PITH_FULL_IMAGE:figures/full_fig_p004_6.png]

**Figure 7.** Figure 7: Overall performance across varying numbers of feedback criteria. [PITH_FULL_IMAGE:figures/full_fig_p005_7.png]

**Figure 8.** Figure 8: Prompt for generating DIRECT-Manual. • GPT-4: Feedback generated using GPT-4- 0613. B.2 Feedback Ranking Process The feedback candidates were ranked by human annotators based on two criteria: • Correct: The feedback provides specific factual information based on the st…

**Figure 9.** Figure 9: Prompt for generating the teacher’s feedback with criteria provided. [PITH_FULL_IMAGE:figures/full_fig_p012_9.png]

**Figure 10.** Figure 10: Prompt for generating the teacher’s feedback without criteria provided. [PITH_FULL_IMAGE:figures/full_fig_p012_10.png]

**Figure 11.** Figure 11: Input data format for binary classifier. [PITH_FULL_IMAGE:figures/full_fig_p013_11.png]

**Figure 12.** Figure 12: Prompt for reward model, DPO, and RankNet [PITH_FULL_IMAGE:figures/full_fig_p013_12.png]

**Figure 13.** Figure 13: Overall performance across varying DM dataset proportions in the DA [PITH_FULL_IMAGE:figures/full_fig_p014_13.png]

**Figure 14.** Figure 14: An example with a high RBO score between the ground-truth ranking and the predicted ranking. [PITH_FULL_IMAGE:figures/full_fig_p015_14.png]

**Figure 15.** Figure 15: An example with a low RBO score between the ground-truth ranking and the predicted ranking. [PITH_FULL_IMAGE:figures/full_fig_p015_15.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

54 extracted references · 27 canonical work pages

[1]

Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774

arXiv 2023
[2]

Nischal Ashok Kumar and Andrew Lan. 2024. https://aclanthology.org/2024.bea-1.10 Improving socratic question generation using data augmentation and preference optimization . In Proceedings of the 19th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2024), pages 108--118, Mexico City, Mexico. Association for Computational Linguistics

work page 2024
[3]

Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. 2023. Qwen technical report. arXiv preprint arXiv:2309.16609

arXiv 2023
[4]

Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. 2022. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862

arXiv 2022
[5]

Chris Burges, Tal Shaked, Erin Renshaw, Ari Lazier, Matt Deeds, Nicole Hamilton, and Greg Hullender. 2005. https://doi.org/10.1145/1102351.1102363 Learning to rank using gradient descent . In Proceedings of the 22nd International Conference on Machine Learning, ICML '05, page 89–96, New York, NY, USA. Association for Computing Machinery

arXiv 2005
[6]

Christopher Burges, Robert Ragno, and Quoc Le. 2006. https://proceedings.neurips.cc/paper_files/paper/2006/file/af44c4c56f385c43f2529f9b1b018f6a-Paper.pdf Learning to rank with nonsmooth cost functions . In Advances in Neural Information Processing Systems, volume 19. MIT Press

work page 2006
[7]

Zhe Cao, Tao Qin, Tie-Yan Liu, Ming-Feng Tsai, and Hang Li. 2007. https://doi.org/10.1145/1273496.1273513 Learning to rank: from pairwise approach to listwise approach . In Proceedings of the 24th International Conference on Machine Learning, ICML '07, page 129–136, New York, NY, USA. Association for Computing Machinery

arXiv 2007
[8]

Nadine Chang, Francesco Ferroni, Michael J Tarr, Martial Hebert, and Deva Ramanan. 2023. Thinking like an annotator: Generation of dataset labeling instructions. arXiv preprint arXiv:2306.14035

work page Pith review arXiv 2023

Show all 54 references

[9]

Koby Crammer and Yoram Singer. 2001. https://proceedings.neurips.cc/paper_files/paper/2001/file/5531a5834816222280f20d1ef9e95f69-Paper.pdf Pranking with ranking . In Advances in Neural Information Processing Systems, volume 14. MIT Press

2001
[10]

Ganqu Cui, Lifan Yuan, Ning Ding, Guanming Yao, Bingxiang He, Wei Zhu, Yuan Ni, Guotong Xie, Ruobing Xie, Yankai Lin, et al. 2023. Ultrafeedback: Boosting language models with scaled ai feedback. arXiv preprint arXiv:2310.01377

2023 arXiv
[11]

Wei Dai, Jionghao Lin, Hua Jin, Tongguang Li, Yi-Shan Tsai, Dragan Gašević, and Guanliang Chen. 2023. https://doi.org/10.1109/ICALT58122.2023.00100 Can large language models provide feedback to students? a case study on chatgpt . In 2023 IEEE International Conference on Advanc...

2023
[12]

Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783

2024 arXiv
[13]

Wanyong Feng, Jaewook Lee, Hunter McNichols, Alexander Scarlatos, Digory Smith, Simon Woodhead, Nancy Ornelas, and Andrew Lan. 2024. https://doi.org/10.18653/v1/2024.findings-naacl.193 Exploring automated distractor generation for math multiple-choice questions via large langu...

2024 doi
[14]

Davide Fossati. 2008. The role of positive feedback in intelligent tutoring systems. In Proceedings of the ACL-08: HLT Student Research Workshop, pages 31--36

2008
[15]

Yoav Freund, Raj Iyer, Robert E Schapire, and Yoram Singer. 2003. An efficient boosting algorithm for combining preferences. Journal of machine learning research, 4(Nov):933--969

2003
[16]

Wensheng Gan, Zhenlian Qi, Jiayang Wu, and Jerry Chun-Wei Lin. 2023. https://doi.org/10.1109/BigData59044.2023.10386291 Large language models in education: Vision and opportunities . In 2023 IEEE International Conference on Big Data (BigData), pages 4776--4785

2023
[17]

John Hattie and Helen Timperley. 2007. The power of feedback. Review of educational research, 77(1):81--112

2007
[18]

Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685

2021 arXiv
[19]

Jin-Xia Huang, Yohan Lee, and Oh-Woog Kwon. 2023. https://doi.org/10.1109/ACCESS.2022.3233224 Direct: Toward dialogue-based reading comprehension tutoring . IEEE Access, 11:8978--8987

2023
[20]

Jaeho Jeon and Seongyong Lee. 2023. Large language models in education: A focus on the complementary relationship between human teachers and chatgpt. Education and Information Technologies, 28(12):15873--15892

2023
[21]

Thorsten Joachims. 2002. https://doi.org/10.1145/775047.775067 Optimizing search engines using clickthrough data . In Proceedings of the Eighth ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD '02, page 133–142, New York, NY, USA. Association for...

2002
[22]

Gerd Kortemeyer. 2024. Performance of the pre-trained large language model gpt-4 on automated short answer grading. Discover Artificial Intelligence, 4(1):47

2024
[23]

Charles Koutcheme, Nicola Dainese, Arto Hellas, Sami Sarsa, Juho Leinonen, Syed Ashraf, and Paul Denny. 2024. http://arxiv.org/abs/2407.04873 Evaluating language models for generating and judging programming feedback

2024 arXiv
[24]

Yanyan Lan, Yadong Zhu, Jiafeng Guo, Shuzi Niu, and Xueqi Cheng. 2014. Position-aware listmle: a sequential learning process for ranking. In Proceedings of the Thirtieth Conference on Uncertainty in Artificial Intelligence, UAI'14, page 449–458, Arlington, Virginia, USA. AUAI Press

2014
[25]

Yu-Ju Lan and Nian-Shing Chen. 2024. Teachers’ agency in the era of llm and generative ai. Educational Technology & Society, 27(1):I--XVIII

2024
[26]

Jaewook Lee, Digory Smith, Simon Woodhead, and Andrew Lan. 2024. https://doi.org/10.5281/zenodo.12730005 Math multiple choice question generation via human-large language model collaboration . In Proceedings of the 17th International Conference on Educational Data Mining, page...

2024 doi
[27]

Ping Li, Qiang Wu, and Christopher Burges. 2007. https://proceedings.neurips.cc/paper_files/paper/2007/file/b86e8d03fe992d1b0e19656875ee557c-Paper.pdf Mcrank: Learning to rank using multiple classification and gradient boosting . In Advances in Neural Information Processing Sy...

2007
[28]

Wencke Liermann, Jin-Xia Huang, Yohan Lee, and Kong Joo Lee. 2024. https://doi.org/10.18653/v1/2024.emnlp-main.605 More insightful feedback for tutoring: Enhancing generation mechanisms and automatic evaluation . In Proceedings of the 2024 Conference on Empirical Methods in Na...

2024 doi
[29]

Haochen Liu, Zitao Liu, Zhongqin Wu, and Jiliang Tang. 2020. https://doi.org/10.18653/v1/2020.coling-main.166 Personalized multimodal feedback generation in education . In Proceedings of the 28th International Conference on Computational Linguistics, pages 1826--1840, Barcelon...

2020 doi
[30]

Haohao Luo, Yang Deng, Ying Shen, See-Kiong Ng, and Tat-Seng Chua. 2024 a . https://doi.org/10.18653/v1/2024.acl-long.432 Chain-of-exemplar: Enhancing distractor generation for multimodal educational question generation . In Proceedings of the 62nd Annual Meeting of the Associ...

2024 doi
[31]

Jian Luo, Xuanang Chen, Ben He, and Le Sun. 2024 b . https://doi.org/10.18653/v1/2024.acl-long.313 PRP -graph: Pairwise ranking prompting to LLM s with graph aggregation for effective text re-ranking . In Proceedings of the 62nd Annual Meeting of the Association for Computatio...

2024 doi
[32]

Wenting Ma, Olusola O Adesope, John C Nesbit, and Qing Liu. 2014. Intelligent tutoring systems and learning outcomes: A meta-analysis. Journal of educational psychology, 106(4):901

2014
[33]

Liebenow, Marlene Steinbach, Andrea Horbach, and Johanna Fleckenstein

Jennifer Meyer, Thorben Jansen, Ronja Schiller, Lucas W. Liebenow, Marlene Steinbach, Andrea Horbach, and Johanna Fleckenstein. 2024. https://doi.org/https://doi.org/10.1016/j.caeai.2023.100199 Using llms to bring evidence-based feedback into the classroom: Ai-generated feedba...

2024
[34]

Nikahat Mulla and Prachi Gharpure. 2023. Automatic question generation: a review of methodologies, datasets, evaluation metrics, and applications. Progress in Artificial Intelligence, 12(1):1--32

2023
[35]

David J Nicol and Debra Macfarlane-Dick. 2006. Formative assessment and self-regulated learning: A model and seven principles of good feedback practice. Studies in higher education, 31(2):199--218

2006
[36]

Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F Christiano, Jan Leike, a...

2022
[37]

Zhen Qin, Rolf Jagerman, Kai Hui, Honglei Zhuang, Junru Wu, Le Yan, Jiaming Shen, Tianqi Liu, Jialu Liu, Donald Metzler, Xuanhui Wang, and Michael Bendersky. 2024. https://doi.org/10.18653/v1/2024.findings-naacl.97 Large language models are effective text rankers with pairwise...

2024 doi
[38]

Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. 2023. https://proceedings.neurips.cc/paper_files/paper/2023/file/a85b405ed65c6477a4fe8302b5e06ce7-Paper-Conference.pdf Direct preference optimization: Your language model is ...

2023
[39]

Burges, and Erin Renshaw

Matthew Richardson, Christopher J.C. Burges, and Erin Renshaw. 2013. https://aclanthology.org/D13-1020 MCT est: A challenge dataset for the open-domain machine comprehension of text . In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pa...

2013
[40]

Alexander Scarlatos, Wanyong Feng, Andrew Lan, Simon Woodhead, and Digory Smith. 2024 a . https://aclanthology.org/2024.bea-1.19 Improving automated distractor generation for math multiple-choice questions with overgenerate-and-rank . In Proceedings of the 19th Workshop on Inn...

2024
[41]

Alexander Scarlatos, Digory Smith, Simon Woodhead, and Andrew Lan. 2024 b . http://arxiv.org/abs/2403.01304 Improving the validity of automatically generated feedback via reinforcement learning

2024 arXiv
[42]

Alexander Scarlatos, Digory Smith, Simon Woodhead, and Andrew Lan. 2024 c . Improving the validity of automatically generated feedback via reinforcement learning. In International Conference on Artificial Intelligence in Education, pages 280--294. Springer

2024
[43]

Hyein Seo, Taewook Hwang, Jeesu Jung, Hyeonseok Kang, Hyuk Namgoong, Yohan Lee, and Sangkeun Jung. 2025. Large language models as evaluators in education: Verification of feedback consistency and accuracy. Applied Sciences, 15(2):671

2025
[44]

Valerie J Shute. 2008. Focus on formative feedback. Review of educational research, 78(1):153--189

2008
[45]

Maja Stahl, Leon Biermann, Andreas Nehring, and Henning Wachsmuth. 2024. https://aclanthology.org/2024.bea-1.23 Exploring LLM prompting strategies for joint essay scoring and feedback generation . In Proceedings of the 19th Workshop on Innovative Use of NLP for Building Educat...

2024
[46]

Jacob Steiss, Tamara Tate, Steve Graham, Jazmin Cruz, Michael Hebert, Jiali Wang, Youngsun Moon, Waverly Tseng, Mark Warschauer, and Carol Booth Olson. 2024. https://doi.org/https://doi.org/10.1016/j.learninstruc.2024.101894 Comparing the quality of human and chatgpt feedback ...

2024
[47]

Michael Taylor, John Guiver, Stephen Robertson, and Tom Minka. 2008. https://doi.org/10.1145/1341531.1341544 Softrank: optimizing non-smooth rank metrics . In Proceedings of the 2008 International Conference on Web Search and Data Mining, WSDM '08, page 77–86, New York, NY, US...

2008
[48]

Ming-Feng Tsai, Tie-Yan Liu, Tao Qin, Hsin-Hsi Chen, and Wei-Ying Ma. 2007. https://doi.org/10.1145/1277741.1277808 Frank: a ranking method with fidelity loss . In Proceedings of the 30th Annual International ACM SIGIR Conference on Research and Development in Information Retr...

2007
[49]

Shen Wang, Tianlong Xu, Hang Li, Chaoli Zhang, Joleen Liang, Jiliang Tang, Philip S Yu, and Qingsong Wen. 2024. Large language models for education: A survey and outlook. arXiv preprint arXiv:2403.18105

2024 arXiv
[50]

Baraniuk

Zichao Wang, Jakob Valdez, Debshila Basu Mallick, and Richard G. Baraniuk. 2022. Towards human-like educational question generation with large language models. In Artificial Intelligence in Education, pages 153--166, Cham. Springer International Publishing

2022
[51]

Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, R \'e mi Louf, Morgan Funtowicz, et al. 2019. Huggingface's transformers: State-of-the-art natural language processing. arXiv preprint arXiv:1910.03771

2019 arXiv
[52]

Zhaohui Zheng, Keke Chen, Gordon Sun, and Hongyuan Zha. 2007. https://doi.org/10.1145/1277741.1277792 A regression framework for learning ranking functions using relative relevance judgments . In Proceedings of the 30th Annual International ACM SIGIR Conference on Research and...

2007
[53]

URL: " 'urlintro :=

ENTRY address author booktitle chapter edition editor howpublished institution journal key month note number organization pages publisher school series title type volume year eprint doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before...
[54]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.

[1] [1]

Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774

arXiv 2023

[2] [2]

Nischal Ashok Kumar and Andrew Lan. 2024. https://aclanthology.org/2024.bea-1.10 Improving socratic question generation using data augmentation and preference optimization . In Proceedings of the 19th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2024), pages 108--118, Mexico City, Mexico. Association for Computational Linguistics

work page 2024

[3] [3]

Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. 2023. Qwen technical report. arXiv preprint arXiv:2309.16609

arXiv 2023

[4] [4]

Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, et al. 2022. Training a helpful and harmless assistant with reinforcement learning from human feedback. arXiv preprint arXiv:2204.05862

arXiv 2022

[5] [5]

Chris Burges, Tal Shaked, Erin Renshaw, Ari Lazier, Matt Deeds, Nicole Hamilton, and Greg Hullender. 2005. https://doi.org/10.1145/1102351.1102363 Learning to rank using gradient descent . In Proceedings of the 22nd International Conference on Machine Learning, ICML '05, page 89–96, New York, NY, USA. Association for Computing Machinery

arXiv 2005

[6] [6]

Christopher Burges, Robert Ragno, and Quoc Le. 2006. https://proceedings.neurips.cc/paper_files/paper/2006/file/af44c4c56f385c43f2529f9b1b018f6a-Paper.pdf Learning to rank with nonsmooth cost functions . In Advances in Neural Information Processing Systems, volume 19. MIT Press

work page 2006

[7] [7]

Zhe Cao, Tao Qin, Tie-Yan Liu, Ming-Feng Tsai, and Hang Li. 2007. https://doi.org/10.1145/1273496.1273513 Learning to rank: from pairwise approach to listwise approach . In Proceedings of the 24th International Conference on Machine Learning, ICML '07, page 129–136, New York, NY, USA. Association for Computing Machinery

arXiv 2007

[8] [8]

Nadine Chang, Francesco Ferroni, Michael J Tarr, Martial Hebert, and Deva Ramanan. 2023. Thinking like an annotator: Generation of dataset labeling instructions. arXiv preprint arXiv:2306.14035

work page Pith review arXiv 2023