Simulation as Reality? The Effectiveness of LLM-Generated Data in Open-ended Question Assessment

Long Zhang; Meng Zhang; Wei Lin Wang; Yu Luo

REVIEW 4 major objections 5 minor 1 cited by

Simulation as Reality? The Effectiveness of LLM-Generated Data in Open-ended Question Assessment

T0 review · 4 major / 5 minor · reviewed 2026-08-08 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Fine-tuning on LLM-made student answers yields an assessment agent that generally outperforms GPT-4o on real open-ended responses, with a persistent simulation-to-reality gap.

desk verdict The real-world superiority claim is not established, but the paper is a useful, honest case study in synthetic-data evaluation pitfalls. read the letter →

arxiv 2502.06371 v1 pith:JBJODFYO submitted 2025-02-10 cs.CY

Long Zhang , Meng Zhang , Wei Lin Wang , Yu Luo This is my paper · ORCID

classification cs.CY

keywords automatedassessmentopen-endedquestionssyntheticdataLLM-generatedDeBERTaretrieval-augmentedgenerationeducationalnoise

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tests whether synthetic responses generated by a large language model can substitute for human-annotated real student data when training an automated open-ended question assessment system. The authors fine-tune a DeBERTa-based agent on about 322,538 GPT-3.5-generated question-response pairs and find that it scores 0.936 MAP@3 on a public science-exam test set, far above the 0.532 of DeBERTa without their retrieval-augmented pipeline. In a real classroom study with 60 students and three teacher raters, the synthetic-trained agent generally beats GPT-4o on identifying the best response, recognizing top-three responses, and overall ranking, while losing slightly on one question category. The paper's cautionary conclusion is that simulation works well but not completely: models trained only on clean synthetic data lose precision when moved from the simulator to real-world noise, so future systems should blend synthetic and real data or deliberately inject realistic noise and bias.

What carries the argument

The central object is a fine-tuned DeBERTa v3-large model embedded in a retrieval-augmented generation loop. Text from a large encyclopedia corpus and a STEM subset is embedded into vectors; for each student response the top ten related passages are retrieved and appended to the query; DeBERTa is fine-tuned on 322,538 synthetic question-response pairs with a known best answer, then ranks each candidate response as it would a multiple-choice option. The identity doing the work is the synthetic pairing of open-ended prompts with a single best response, which converts assessment into a ranking task the model can learn without human annotation.

What would settle it

A fresh out-of-sample experiment would settle it: collect real student answers from a different population and question set, have independent teachers rank them, and pit a synthetic-only DeBERTa agent against GPT-4o. If the agent no longer beats GPT-4o on Top1 accuracy and NDCG, or if a version trained on aggressively cleaned real data outperforms one trained on raw noisy real data, the paper's central claim—that synthetic data suffices and that over-cleaning removes useful signal—would be undercut.

Watch

Extended reading notes

Core claim

The paper's central claim is that a DeBERTa-based assessment agent fine-tuned on 322,538 GPT-3.5-generated question-response pairs can assess open-ended answers well enough to beat a frontier general-purpose model on real student responses. On the public science-exam test set the agent scores MAP@3 = 0.936, up from 0.532 for DeBERTa without the retrieval-augmented pipeline. On real responses rated by three teachers, it reaches overall Top1 accuracy 0.442 versus GPT-4o's 0.408, Top3 F1 0.731 versus 0.706, NDCG 0.920 versus 0.914, and MAP 0.660 versus 0.627; GPT-4o wins only on the Theories and Relationships category in ranking metrics. The authors read this as evidence that synthetic data can bootstrap useful assessment tools, while the gap between Phase I and Phase II shows that noise and variation missing from synthetic data are information the model still needs.

Load-bearing premise

The load-bearing premise is that the public competition test set used in Phase I is a valid out-of-sample test of assessment skill, even though it was created with the same generator and source corpus as the training data; if that premise fails, the high Phase I score reflects in-distribution consistency with the generator rather than transferable assessment ability.

Editorial extensions

If this is right

Educational assessment systems for open-ended questions can be bootstrapped without large human-annotation campaigns, because synthetic question-response pairs with known best answers are sufficient to fine-tune a ranking model.
A compact fine-tuned discriminative model can match or exceed a frontier general-purpose model on this task, while avoiding the cost and hallucination risk of using a generative model directly as a grader.
Deployment of synthetic-only agents should be paired with human validation, especially for question types such as Theories and Relationships where the synthetic-trained agent does not clearly beat GPT-4o.
Training pipelines should preserve or intentionally introduce realistic noise and bias rather than aggressively cleaning data, since the paper argues over-processed real data suffers from the same simulation gap.
Reporting performance by question type matters, because overall averages hide category-level differences relevant to classroom use.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The paper implies a testable prescription: if missing noise is the cause of the sim-to-real gap, then injecting realistic distractors, typos, or partially correct answers into synthetic training data should narrow the gap; this can be measured by comparing Top1 accuracy on fresh real responses with and without such augmentation.
The same hybrid-training recipe may transfer to other domains where annotated responses are scarce, such as short-answer scoring in medicine or law, but the size of the synthetic-to-real gap would need to be re-measured in each domain.
The Phase I MAP@3 number probably overstates real-world skill because the test set shares its generator and source corpus with the training data; the honest evidence of transfer is Phase II, where the margin over GPT-4o is much smaller.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The headline finding is not established. The Phase II claim that your synthetic-trained DeBERTa agent outperforms GPT-4o on real student responses rests on a 44.2% vs 40.8% Top-1 accuracy with no significance test, and the Phase I MAP@3 of 0.936 is measured on a test set generated by the same GPT-3.5-from-Wikipedia recipe used to create the training data. That score mostly shows the model fits the generator, not that it generalizes to real student writing.

What the paper does well: it asks a practical question, builds a full RAG plus fine-tuning pipeline, runs an actual classroom pilot, and is candid about the sim-to-real gap. The recommendation to mix synthetic and real data is sensible and not overclaimed. The clustering into four question types is reasonable, and the honest acknowledgment of limitations—single benchmark, science-only domain, small participant pool—counts in the authors' favor.

The load-bearing soft spot is the evaluation. In Phase I, the Kaggle LLM Science Exam test set was created by GPT-3.5 from Wikipedia, exactly the same generator and corpus used to produce the 322k training entries. So MAP@3=0.936 is in-distribution consistency, not an out-of-sample check. The improvement over DeBERTa alone (0.532) is interesting, but it is still within that same distribution. Phase II has roughly 120 question-answer sets total. The overall Top-1 margin of 0.442 vs 0.408 corresponds to about four correct decisions. Even in the most favorable discordant split, the two-sided McNemar p-value is 0.125; with any realistic discordance it is much larger. So the phrase "significantly outperforming" is not supported by the data. The ground-truth ranking also uses an arbitrary 7/3/1 weighting of three teachers' votes with no reported inter-rater agreement, which further weakens the labels.

Minor but worth noting: no code or data is released, and the paper is written in places as if the real-world result is definitive, which contradicts its own careful discussion of limitations.

Who this is for: researchers in AI-based educational assessment who want a cautionary example of why synthetic-data evaluation needs a genuinely out-of-distribution test set and proper statistical testing. It would be an excellent reading-group case study, but not a source to cite for quantitative conclusions.

My recommendation: send this to peer review rather than desk-reject—the question matters and the flaws are fixable with significance testing, a true out-of-distribution benchmark, and released artifacts. But expect major revision and do not accept the current claims as stated.

Referee Report

4 major / 5 minor

Summary. The paper investigates whether LLM-generated synthetic data can train an automated open-ended question assessment agent. In Phase I, the authors fine-tune a DeBERTa-v3-large model on 322,538 synthetic question-answer entries generated with GPT-3.5 from Wikipedia and WikiSTEM, and they report a MAP@3 of 0.936 on the 200-item Kaggle LLM Science Exam test set, versus 0.532 for a DeBERTa baseline. In Phase II, they recruit 60 university students, have three teachers rank the student responses, and compare their agent's Top1, Top3, NDCG, and MAP scores against GPT-4o, reporting an overall Top1 accuracy of 0.442 versus 0.408. The paper concludes that synthetic data can effectively bootstrap an assessment agent, that real-world performance shows a gap, and that future systems should mix synthetic and real data.

Significance. If the empirical claims were sound, the paper would make a useful contribution to reducing annotation costs in educational assessment. The authors are transparent about the synthetic data generation recipe and make a concrete attempt to validate the agent on real student responses with human teacher judgments, which is a strength. However, the study's load-bearing evidence does not support the headline claims: the Phase I test set shares its generator and source corpus with the training data, and the Phase II comparison lacks any inferential statistics and is likely within sampling noise. The paper is therefore more valuable as a cautionary demonstration of the difficulty of synthetic-to-real transfer than as evidence that synthetic-data-trained agents outperform frontier LLMs on real student responses.

major comments (4)

[§2.1.1] The Kaggle LLM Science Exam test set was generated by GPT-3.5 from Wikipedia text fragments, and the 322,538 synthetic training entries were generated with the same GPT-3.5-from-Wikipedia recipe (using Wikipedia and WikiSTEM). The reported Phase I MAP@3 of 0.936 therefore largely measures the model's fit to the distribution of its own synthetic generator rather than its ability to assess real student responses. Since this score is the primary quantitative evidence for the effectiveness of synthetic data, an out-of-sample evaluation on independently collected responses is required before the central claim can be accepted.
[§3.2, Table 2] The statement that the agent 'significantly outperforms' GPT-4o is unsupported. The overall Top1 difference is 0.442 versus 0.408; with 120 question-answer sets (12 groups × 10 questions), this is about 4 correct decisions. No confidence intervals, significance tests, or effect sizes are reported. Even in the most favorable discordant-pair split, McNemar's exact test gives p≈0.125 (two-sided), so the observed margin is within sampling noise. The same caution applies to the Top3 and ranking differences in Tables 2 and 3.
[§2.2.1] The teacher composite score Score(A_i) = (7×H_i + 3×M_i + 1×L_i)/3 is adopted without justification or sensitivity analysis. The weights determine the ground-truth ordering of responses, and all Phase II metrics are computed against this ordering. No inter-rater reliability (e.g., Cohen's kappa or ICC) is reported, so it is unclear whether the three teachers agree sufficiently to support the composite ranking. A sensitivity analysis over the weights and an agreement statistic are needed.
[§3.2, Tables 2–3] The abstract and discussion claim the model outperforms GPT-4o 'in most question types', but the per-type results are mixed: GPT-4o has higher Top1 accuracy in 'Theories and Relationships' (0.533 vs 0.400) and higher NDCG there (0.946 vs 0.932); GPT-4o also has slightly higher Top3 accuracy in 'Concepts and Definitions' (0.529 vs 0.519). The claim of superiority should be qualified accordingly.

minor comments (5)

[§2.1.3] The displayed MAP@3 formula is corrupted (missing summation bounds and division by U); please replace it with a clean equation.
[§4.1] The Phase I MAP@3 is reported as 0.935 in §4.1 but as 0.936 in §3.1; please reconcile the inconsistency.
[Figures] There are two 'Figure 1' labels and a reference to Figure 3 that does not appear; please renumber the figures and ensure all are present.
[Throughout] The spellings 'GPT4o' and 'GPT-4o' are used inconsistently; please standardize.
[Throughout] The manuscript contains numerous typographical and grammatical errors (e.g., 'real-ward' in the Introduction); a careful language edit is needed.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation found: the claims rest on empirical experiments, not on definitions, fitted predictions, or load-bearing self-citation.

full rationale

I walked the derivation chain and found no step in which a prediction or claimed result is equivalent to its inputs by construction. Phase I fine-tunes DeBERTa on 322,538 GPT-3.5-generated Wikipedia-derived entries and evaluates on the 200-item Kaggle LLM Science Exam set. The paper itself notes this test set was also generated by GPT-3.5 from Wikipedia text, so the high MAP@3 of 0.936 is an in-distribution evaluation rather than a proof of real-world generalization. That is a validity limitation, not circularity: the test labels are not constructed from the model's fitted outputs, no parameter is fitted to the test set, and the model's ranking is not forced by the training procedure. Phase II is the paper's real-world check, using 60 students, four question sets, and three independent teacher judges; the 7/3/1 weighting rule for combining teacher votes is arbitrary and simplifies an external ground truth, but it is not an output of the assessed agent and does not make the comparison self-referential. The absence of significance testing, confidence intervals, or inter-rater reliability in Phase II is a statistical and evidentiary weakness, not a circularity. Cited prior work (Park et al., DeBERTa, RAG, embedding methods) supplies methodological components, but the central effectiveness claim is supported by the paper's own experiments rather than by a self-citation chain. Because no specific reduction to inputs can be exhibited, the appropriate verdict is no significant circularity, score 0.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

The central claims rest on the representativeness of GPT-3.5-generated data, the validity of the teacher scoring formula, and the comparability of the Phase II baseline; none of these are independently verified.

free parameters (3)

Teacher scoring weights = H weight 7, M weight 3, L weight 1
Ad hoc weights in Section 2.2.1 used to combine three teachers' Top1/Top3 judgments into a single ranking; no validation or sensitivity analysis.
Number of question clusters = 4
Chosen by elbow and silhouette methods, but the cluster labels are used to report per-category performance differences; the boundaries are data-dependent, not hypothesis-driven.
Retrieval top-k = 10
Number of context passages retrieved per query in the RAG pipeline; chosen without ablation, influences the augmented query used for fine-tuning and inference.

assumptions (5)

domain assumption GPT-3.5-generated open-ended questions and responses are representative of real student discourse in educational assessment.
The whole synthetic data strategy rests on this, stated in Section 2.1.1 and Discussion; if false, the sim-to-real transfer claim collapses.
domain assumption The Kaggle LLM Science Exam dataset, generated by GPT-3.5 from Wikipedia, is a valid test set for measuring assessment quality.
Used as Phase I test set; the paper itself notes the dataset is GPT-3.5-generated, so the test distribution overlaps the training distribution.
ad hoc to paper The teacher composite score formula (7,3,1) produces the correct ordering of student responses.
Introduced in Section 2.2.1 without prior validation; all Phase II labels derive from it.
standard math Standard supervised fine-tuning and RAG assumptions about gradient-based learning from task-specific labels.
Implicit in the fine-tuning step; not proven, but standard in ML practice.
domain assumption GPT-4o is a valid state-of-the-art baseline for automated assessment and is evaluated under comparable conditions.
GPT-4o is used as the reference benchmark in Phase II, yet the prompting, context, and decoding settings for GPT-4o are not specified.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Simulation as Reality? The Effectiveness of LLM-Generated Data in Open-ended Question Assessment." pith.science (2026). https://pith.science/paper/JBJODFYO

@misc{pith2026250206371,
  author       = {Pith},
  title        = {Pith review of: Simulation as Reality? The Effectiveness of LLM-Generated Data in Open-ended Question Assessment},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/JBJODFYO}},
  note         = {Machine review of arXiv:2502.06371}
}

read the original abstract

The advancement of Artificial Intelligence (AI) has created opportunities for e-learning, particularly in automated assessment systems that reduce educators' workload and provide timely feedback to students. However, developing effective AI-based assessment tools remains challenging due to the substantial resources required for collecting and annotating real student data. This study investigates the potential and gap of simulative data to address this limitation. Through a two-phase experimental study, we examined the effectiveness and gap of Large Language Model generated synthetic data in training educational assessment systems. Our findings reveal that while simulative data demonstrates promising results in training automated assessment models, outperforming state-of-the-art GPT-4o in most question types, its effectiveness has notable limitations. Specifically, models trained on synthetic data show excellent performance in simulated environment but need progress when applied to real-world scenarios. This performance gap highlights the limitations of only using synthetic data in controlled experimental settings for AI training. The absence of real-world noise and biases, which are also present in over-processed real-world data, contributes to this limitation. We recommend that future development of automated assessment agents and other AI tools should incorporate a mixture of synthetic and real-world data, or introduce more realistic noise and biases patterns, rather than relying solely on synthetic or over-processed data.

Figures

Figures reproduced from arXiv: 2502.06371 by the authors.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

GOBench: Benchmarking Geometric Optics Generation and Understanding of MLLMs
cs.CV 2025-06 conditional novelty 6.0 of 10

GOBench measures how well multimodal AI models generate and understand geometric optics, finding that even top models make frequent physical errors.

Reference graph

Works this paper leans on

6 extracted references · 5 canonical work pages · cited by 1 Pith paper

[1]

Why is Paris the most memorable place you have ever visited?

Simulation as Reality? The Effectiveness of LLM-Generated Data in Open-ended Question Assessment Long (Jim) Zhang[ 0000-0001-5768-2549] (u3008540@connect.hku.hk) Faculty of Education, The University of Hong Kong Meng (Joe) Zhang[0009-0007-6002-9963] (zhangmeng2@kingfa.com) Kingfa Science and Technology Company, Limited Wei Lin (William) Wang[0000-0002-170...

work page 2024
[2]

Applications and Rules

Concepts and Definitions: Help students grasp fundamental definitions and concepts. Theories and Relationships: Require students to understand and explain more complex theories and relationships. Phenomena and Effects: Focus on specific phenomena and effects and their scientific explanations. Applications and Rules: Emphasize practical applications and op...

work page 2024
[3]

In Phase I, our primary objective is to implement an open-ended automated evaluation agent using generated synthetic data

Consider integrating controlled noise into synthetic data generation 2 Experimental Design This study consists of two phases. In Phase I, our primary objective is to implement an open-ended automated evaluation agent using generated synthetic data. We will discuss this phase in terms of three components: data sources, modeling process, and performance. In...

work page 2023
[4]

In the Data Preprocessing stage, datasets from open-ended questions, Wikipedia, and WikiSTEM are cleaned and formatted to ensure consistency and quality

Utilizing Simulation for Fine-tuning process of DeBERTa The model depicted in Figure 1 comprises four main stages: Data Preprocessing, Retrieval, Augmentation, and Generation (RAG, suggested by Siriwardhana et al., 2023), with a crucial Fine-tuning step within the Generation stage to enhance accuracy and performance. In the Data Preprocessing stage, datas...

work page 2023
[514]

The End of History

https://doi.org/10.3390/app12010514 Koçak, D. (2020). Investigation of rater tendencies and reliability in different assessment methods with many facet rasch model. Lnternational Electronic Journal of Elementary Education, 12(4), 349-358. https://doi.org/10.26822/iejee.2020459464 Myagmar, B. and Li, J. (2019). Cross-domain sentiment classification with bi...

work page arXiv 2020
[2018]

over-sanitization

- may inadvertently eliminate crucial contextual information. Just like "over-sanitization" of training data, whether synthetic or real, presents a paradox in AI development: the very processes intended to enhance model performance may actually limit its ability to handle real-world variability. The standardized patterns learned from cleaned and synthetic...

arXiv 2023

Pith tools

Reviewed August 8, 2026 · model on record in the stance chip above.

[1] [1]

Why is Paris the most memorable place you have ever visited?

Simulation as Reality? The Effectiveness of LLM-Generated Data in Open-ended Question Assessment Long (Jim) Zhang[ 0000-0001-5768-2549] (u3008540@connect.hku.hk) Faculty of Education, The University of Hong Kong Meng (Joe) Zhang[0009-0007-6002-9963] (zhangmeng2@kingfa.com) Kingfa Science and Technology Company, Limited Wei Lin (William) Wang[0000-0002-170...

work page 2024

[2] [2]

Applications and Rules

Concepts and Definitions: Help students grasp fundamental definitions and concepts. Theories and Relationships: Require students to understand and explain more complex theories and relationships. Phenomena and Effects: Focus on specific phenomena and effects and their scientific explanations. Applications and Rules: Emphasize practical applications and op...

work page 2024

[3] [3]

In Phase I, our primary objective is to implement an open-ended automated evaluation agent using generated synthetic data

Consider integrating controlled noise into synthetic data generation 2 Experimental Design This study consists of two phases. In Phase I, our primary objective is to implement an open-ended automated evaluation agent using generated synthetic data. We will discuss this phase in terms of three components: data sources, modeling process, and performance. In...

work page 2023

[4] [4]

In the Data Preprocessing stage, datasets from open-ended questions, Wikipedia, and WikiSTEM are cleaned and formatted to ensure consistency and quality

Utilizing Simulation for Fine-tuning process of DeBERTa The model depicted in Figure 1 comprises four main stages: Data Preprocessing, Retrieval, Augmentation, and Generation (RAG, suggested by Siriwardhana et al., 2023), with a crucial Fine-tuning step within the Generation stage to enhance accuracy and performance. In the Data Preprocessing stage, datas...

work page 2023

[5] [514]

The End of History

https://doi.org/10.3390/app12010514 Koçak, D. (2020). Investigation of rater tendencies and reliability in different assessment methods with many facet rasch model. Lnternational Electronic Journal of Elementary Education, 12(4), 349-358. https://doi.org/10.26822/iejee.2020459464 Myagmar, B. and Li, J. (2019). Cross-domain sentiment classification with bi...

work page arXiv 2020

[6] [2018]

over-sanitization

- may inadvertently eliminate crucial contextual information. Just like "over-sanitization" of training data, whether synthetic or real, presents a paradox in AI development: the very processes intended to enhance model performance may actually limit its ability to handle real-world variability. The standardized patterns learned from cleaned and synthetic...

arXiv 2023