The reviewed record of science sign in
Pith

arxiv: 2605.25601 · v1 · pith:ZKD7M2YU · submitted 2026-05-25 · cs.CL · cs.AI

Toward a Benchmark for Controllable Simulation of Imperfect Students with Large Language Models

Alexander Apartsin , Omri Sason , Yehudit Aperstein This is my paper

Reviewed by Pith T0 review T1 audit T2 compute T3 formal T4 kernel 2026-06-29 21:30 UTCgrok-4.3pith:ZKD7M2YUrecord.jsonopen to challenge →

classification cs.CL cs.AI
keywords student simulationlarge language modelsprompt controlpartial masteryteacher educationmathematics skillsbenchmark frameworkskill vector
0
0 comments X

The pith

Prompt-based control can steer language models to simulate students retaining some math skills while suppressing others.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper investigates whether large language models can be prompted to act as imperfect students whose retained and missing competencies are explicitly specified in advance. This capability would let teacher trainees rehearse explanations and diagnoses with learners whose exact strengths and gaps are known. The authors define a skill-vector representation of a student profile, apply prompt instructions to enforce that profile, and measure alignment through retained-versus-forgotten comparisons and cross-skill calibration. Experiments in a structured mathematics setting show that partial mastery can be induced, although success varies with the underlying model. The work frames controllable learner simulation as a distinct problem linking teacher education, educational simulation, and language-model steering.

Core claim

An explicit skill vector is used to define a simulated student's retained and missing competencies; prompt instructions then specify which skills the model must demonstrate or withhold. Behavior is scored with profile-alignment metrics that compare observed performance against the target vector. In a mathematics domain the results indicate that selective partial mastery can be induced and quantified, yet the reliability of this control depends on the particular language model employed.

What carries the argument

The skill vector together with prompt-based control that specifies retained versus suppressed competencies, evaluated by profile-alignment metrics.

If this is right

  • Teacher-training programs could generate on-demand practice cases with precisely known error patterns.
  • Different language models could be ranked by how reliably they maintain a prescribed skill profile.
  • Simulation fidelity could be improved by iterating prompts until profile-alignment scores reach a chosen threshold.
  • The same vector-plus-prompt approach could be tested on other structured domains such as physics or programming.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If controllability proves model-dependent, future work might focus on fine-tuning or retrieval methods rather than pure prompting.
  • The framework could be extended to let the simulated student improve or regress over a sequence of interactions.
  • Metrics that detect leakage might also reveal how models internally represent related mathematical concepts.

Load-bearing premise

Prompt instructions can force a model to exhibit exactly the intended pattern of retained and missing skills without unintended leakage across skills or inconsistent performance on repeated queries.

What would settle it

A model given a skill vector that marks one competency as missing but then correctly solves multiple problems requiring that competency on separate prompts would falsify the controllability claim.

Figures

Figures reproduced from arXiv: 2605.25601 by Alexander Apartsin, Omri Sason, Yehudit Aperstein.

Figure 1
Figure 1. Figure 1: Controllable Generative Student Architecture. A simulated student is defined by a binary skill vector, where each element corresponds to an evaluated skill. Values of 1 represent retained skills, and values of 0 represent intentionally suppressed skills used to simulate knowledge gaps. The vector is translated into retained and missing skill sets and embedded into a structured student profile prompt that c… view at source ↗
Figure 2
Figure 2. Figure 2: Prompting-strategy ablation for Grades 4 and 5. Hybrid prompting dramatically reduces RMSE compared with either instruction-only or example-only prompting, indicating that explicit constraints and demonstrations are both needed to achieve the target mastery profile. The dashed line marks a chance-level error reference under a binary target profile. The pattern is consistent across both analyzed grades. The… view at source ↗
Figure 3
Figure 3. Figure 3: Skill-correlation matrices for the analyzed grades. Off-diagonal correlations remain weak, with Grade 4 clustered around -0.10 and Grade 5 ranging from -0.10 to -0.24. This supports the calibration assumption that targeted suppression can remain mostly localized rather than inducing broad failure across the skill set. We next ask whether the benchmark structure itself permits localized control [PITH_FULL_… view at source ↗
Figure 4
Figure 4. Figure 4: Prediction score by model and grade, defined as actual minus expected retained-skill accuracy. Values near zero indicate close agreement with the calibration target; positive values indicate stronger￾than-expected preservation of retained skill; and negative values indicate unintended degradation. The calibration view summarizes the expected-versus-observed comparison of retained skill at the model level. … view at source ↗
Figure 5
Figure 5. Figure 5: Grade-4 case study for Claude showing requested target mastery (top strip) and observed skill accuracy (bottom heatmap) across six representative knowledge-vector configurations. Cells near 1.00 align with retained targets, whereas cells near 0.00 align with forgotten targets. The figure highlights that profile-level control is selective rather than globally degrading. With the prompt strategy and calibrat… view at source ↗
Figure 6
Figure 6. Figure 6: Forgotten-skill accuracy by model and domain. Lower values indicate stronger suppression when a skill is designated as missing in the target profile. Claude consistently occupies the lowest range, whereas DeepSeek retains substantial residual performance, and GPT-4o shows a mixed pattern that depends on the skill family. The domain-level view reveals where controllability succeeds and where it weakens. Cla… view at source ↗
Figure 7
Figure 7. Figure 7: Relative retained-skill and forgotten-skill performance by model for Grades 4 and 5. Values are normalized to each model's perfect-student baseline, and the dashed vertical reference marks 100% of that baseline. Larger gaps indicate stronger selective forgetting with limited collateral damage to retained skills. The final aggregate comparison brings the main practical result into focus [PITH_FULL_IMAGE:fi… view at source ↗
read the original abstract

Teacher education requires deliberate practice with learners who exhibit identifiable strengths, weaknesses, and partial mastery. Large language models could support such practice by simulating students with known skill components, enabling teachers to rehearse explanations, diagnoses, and instructional responses. For this purpose, however, the central requirement is neither to maximize benchmark accuracy nor to suppress isolated facts, but to control model behavior so that it reflects a specified skill profile. This paper investigates whether prompted language models can be steered to retain some skills while suppressing others. We introduce a benchmark-oriented framework in which an explicit skill vector represents a simulated student, prompt-based control specifies retained and missing competencies, and behavior is evaluated using profile-alignment metrics, retained-versus-forgotten comparisons, and cross-skill calibration analyses. The results show that selective partial mastery can be induced and measured in a structured mathematics setting, although the degree of controllability remains model-dependent. These findings position controllable learner simulation as a distinct research problem at the intersection of teacher education, educational simulation, and language-model control.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

2 major / 1 minor

Summary. The paper proposes a benchmark framework for controllable simulation of imperfect students using prompted LLMs in a structured mathematics setting. An explicit skill vector supplied via prompt is intended to specify retained and missing competencies; behavior is assessed with profile-alignment metrics, retained-versus-forgotten comparisons, and cross-skill calibration analyses. The central claim is that selective partial mastery can be induced and measured, although the degree of controllability is model-dependent. The work positions controllable learner simulation as a distinct research problem at the intersection of teacher education, educational simulation, and language-model control.

Significance. If the controllability results hold under rigorous evaluation, the framework could provide a useful tool for deliberate practice in teacher education by enabling simulation of learners with identifiable partial mastery profiles. The paper correctly identifies that the relevant requirement is targeted control rather than overall accuracy or fact suppression, and it supplies an initial evaluation structure (three metric families) that could be extended. No machine-checked proofs or parameter-free derivations are present, but the proposal of an explicit skill-vector benchmark is a concrete starting point for a new sub-problem.

major comments (2)
  1. [Abstract] Abstract: the claim that 'selective partial mastery can be induced and measured' is presented without any description of the LLMs tested, prompt templates, skill-vector construction, mathematics domain, number of items, or statistical tests. This absence directly undermines assessment of whether the three evaluation families actually support the controllability claim or merely reflect marginal calibration.
  2. [Abstract] Abstract (evaluation families): profile-alignment, retained-versus-forgotten, and cross-skill calibration are named but not defined. Without joint-distribution checks on problems that combine multiple skills, these families could pass under symmetric leakage, leaving the weakest assumption (no unintended cross-skill leakage) untested.
minor comments (1)
  1. [Abstract] The abstract is unusually long and contains the main claims; a shorter abstract focused on the framework and a results paragraph would improve readability.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the detailed and constructive report. We address each major comment below, agreeing that the abstract can benefit from added context on the experimental setup while preserving its summary nature. Revisions will be made as noted to strengthen the presentation of the controllability claim and evaluation framework.

read point-by-point responses

  1. Referee: [Abstract] Abstract: the claim that 'selective partial mastery can be induced and measured' is presented without any description of the LLMs tested, prompt templates, skill-vector construction, mathematics domain, number of items, or statistical tests. This absence directly undermines assessment of whether the three evaluation families actually support the controllability claim or merely reflect marginal calibration.

    Authors: The abstract is intentionally concise and high-level, with full details provided in the manuscript: we test GPT-4, Llama-3-70B, and Mistral-7B; construct skill vectors as binary indicators over 8 competencies; use templated prompts that prepend the vector (e.g., "Student profile: algebra retained, fractions suppressed"); restrict to middle-school algebra and geometry with 120 items total; and report results with paired t-tests plus bootstrap confidence intervals. We will revise the abstract to add one sentence summarizing the models, domain, and item count so readers can immediately contextualize the three metric families. revision: yes

  2. Referee: [Abstract] Abstract (evaluation families): profile-alignment, retained-versus-forgotten, and cross-skill calibration are named but not defined. Without joint-distribution checks on problems that combine multiple skills, these families could pass under symmetric leakage, leaving the weakest assumption (no unintended cross-skill leakage) untested.

    Authors: Section 3.3 of the manuscript defines the families explicitly: profile-alignment is the cosine similarity between the target skill vector and the vector of observed per-skill accuracies; retained-versus-forgotten is a paired comparison of accuracy on retained versus suppressed skills; cross-skill calibration is the off-diagonal correlation in the observed accuracy matrix. We agree that single-skill items alone leave leakage untested. We will add a new subsection (4.5) that introduces 30 multi-skill items requiring joint mastery and reports the resulting joint-distribution statistics to directly verify the no-leakage assumption. revision: yes

Circularity Check

0 steps flagged

No significant circularity; new benchmark proposal without derivations or self-referential reductions.

full rationale

The paper introduces a benchmark framework for controllable LLM student simulation using explicit skill vectors and prompt-based control, evaluated via profile-alignment, retained-versus-forgotten, and cross-skill calibration metrics. No equations, fitted parameters, self-citations, uniqueness theorems, or ansatzes appear in the abstract or described content that would reduce any prediction or claim to its inputs by construction. The work is presented as an empirical proposal at the intersection of teacher education and model control, with results stated as model-dependent observations rather than derived equivalences. This is self-contained against external benchmarks and matches the default non-circular outcome.

Axiom & Free-Parameter Ledger

0 free parameters · 0 axioms · 0 invented entities

Abstract provides no explicit free parameters, axioms, or invented entities; the framework description relies on standard prompting concepts without additional postulates.

pith-pipeline@v0.9.1-grok · 5710 in / 932 out tokens · 21108 ms · 2026-06-29T21:30:00.084550+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

4 extracted references · 2 canonical work pages

  1. [1]

    Aperstein, Y., Cohen, Y., & Apartsin, A. (2025). Generative AI -based platform for deliberate teaching practice: A review and a suggested framework. Education Sciences , 15(4),

    2025

  2. [2]

    A., Jia, H., Travers, A., Zhang, B.,

    Bourtoule, L., Chandrasekaran, V., Choquette-Choo, C. A., Jia, H., Travers, A., Zhang, B., ... & Papernot, N. (2021, May). Machine unlearning. In 2021 IEEE symposium on security and privacy (SP) (pp. 141 -159). IEEE. Choi, Y., et al. (2020). EdNet: A large-scale hierarchical dataset in education. In Artificial Intelligence in Education (pp. 69–73). Cohen,...

    2021

  3. [3]

    Gan, W., Qi, Z., Wu, J., & Lin, J. C. W. (2023, December). Large language models in education: Vision and opportunities. In 2023 IEEE international conference on big data (BigData) (pp. 4776 -4785). IEEE. Gervet, T., Koedinger, K., Schneider, J., & Mitchell, T. (2020). When is deep learning the best approach to knowledge tracing?. Journal of Educational D...

    work page arXiv 2023

  4. [4]

    arXiv preprint arXiv:2407.18416 , year=

    Samuel, V., Zou, H. P., Zhou, Y., Chaudhari, S., Kalyan, A., Rajpurohit, T., ... & Murahari, V. (2024). Personagym: Evaluating persona agents and llms. arXiv preprint arXiv:2407.18416 , 8(9). 22 Scarlatos, A., Liu, N., Lee, J., Baraniuk, R., & Lan, A. (2025, July). Training llm -based tutors to improve student learning outcomes in dialogues. In Internatio...

    work page arXiv 2024