{"id":"db2b89ab-04b4-47ba-8256-7a152ff9e40b","arxiv_id":"2608.10154","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"Fine-tuned multimodal LLMs that imitate students at discrete ability levels recover item difficulty parameters on held-out math items, reaching Pearson 0.85, better than text-only regression baselines.","lead":"The paper trains a large language model to imitate how students of different ability levels answer multiple-choice math questions that include images. The authors report that reading the model's predicted answer probabilities gives item difficulty estimates that beat text-only regression baselines.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"No vision-capable regression baseline: the 0.85-vs-0.75 'clear margin' is confounded by input modality, not isolating the simulated-respondent mechanism.","rationale":"The reader's conditional verdict already identifies the modality confound as a key weakness. I agree that this is the load-bearing issue because it determines whether the central mechanism is supported: without a vision-capable regression baseline, the reported superiority of the simulated-respondent approach over direct regression is not established. I considered the token-normalization issue from Section 2.4.2 and Eq. 4 as a secondary concern; the next-token probabilities v are not renormalized over the option set, and a raw softmax over the full vocabulary need not sum to one over A-D. This is worth a separate check, but it would not by itself overturn the absolute reconstruction result; a normalization correction could be additive. The absence of a multimodal regression baseline, by contrast, means the claimed advantage over direct regression has not been demonstrated at all. The proposed multimodal-regression test directly targets that confound and would settle whether the comparison holds. Since the reader already made the verdict conditional on exactly this kind of evidence, no verdict adjustment is needed.","tokens_in":9272,"tokens_out":8050,"duration_ms":90834,"concrete_test":"Run a vision-capable regression baseline: fine-tune Qwen3.5-9B with the same LoRA budget, the same combined image+text input from Figure 2, and a linear head trained with MSE to predict the calibrated b directly (no ability-conditioned simulation, no Eq. 4 target), on the same train/dev/test item-model split. If its held-out Pearson/QWK are within bootstrap error of the proposed model's 0.85/0.835, the reported advantage is explained by multimodal input and the simulated-respondent framing contributes little; if it lands near the MetaMath 0.75/0.692 level, the framing is supported. Report 95% bootstrap CIs over the 146 test items.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central comparative claim (Section 3, Table 5) is that simulating students of varying ability recovers difficulty-relevant signal that direct stimulus-to-parameter regression does not capture. This claim is not actually tested. The proposed Qwen3.5 models consume the combined stimulus-and-option image constructed in Figure 2, while both regression baselines, MathBERT and MetaMath, are text-only. The corpus is of mathematics items that the authors themselves call 'inherently visual'; a text-only model cannot see the figure that the 9B multimodal model is given. The reported margin (Pearson 0.85 vs. 0.75; QWK 0.835 vs. 0.692) is therefore fully consistent with the alternative explanation that the advantage comes from the image input alone, not from the simulated-respondent training objective in Eq. 4. The Discussion's inference that a model 'that merely reads the question cannot easily infer how often low-ability students will be drawn to a particular distractor' is also based on this confounded comparison: the proposed model does not merely read the question, it sees the image. The absolute reconstruction result may survive, but the mechanism claim and the 'clear margin' headline do not.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes and evaluates a method for estimating item parameters in educational assessments, specifically the difficulty (b), discrimination (a), and guessing (c) parameters of the three-parameter logistic (3PL) model and the option probabilities of the multiple-choice model (MCM). The method fine-tunes a multimodal LLM (Qwen3.5-4B/9B) to act as a simulated respondent: given an item (text plus a combined image), a labeled ability level, and the correct answer, the model is trained with an MSE loss to match empirically derived option choice probabilities for that ability bin. At inference, the model's four option-token probabilities are treated as response probabilities, and the 3PL/MCM parameters are recovered by weighted least-squares fitting to the resulting curves. The method is tested on a large corpus of mathematics items (4,848 items from 970 item models) and is reported to achieve Pearson correlation 0.85 and QWK 0.835 for difficulty, exceeding text-only regression baselines (MathBERT, MetaMath). The authors interpret this as evidence that the simulated-respondent framing recovers response behavior beyond surface features of the stimulus.","tokens_in":9457,"tokens_out":9986,"duration_ms":85066,"significance":"The paper addresses a practically important problem—estimating item parameters without costly field testing—and extends the emerging line of work on LLM-simulated respondents to a multimodal setting. Its strengths include the large real-response dataset, the clean train/dev/test split by item model that prevents item-level leakage, an explicit loss function and reconstruction objective, and the use of multiple metrics (Pearson, RMSE, QWK). If the method's advantage over baselines could be attributed to the simulated-respondent mechanism rather than to the mere addition of image input, it would be a genuine contribution. As it stands, the headline comparative claim is not supported by the experimental design; the absolute reconstruction results, however, remain plausible and worth further study.","major_comments":[{"comment":"The comparison between the proposed Qwen3.5 models and the regression baselines is confounded by input modality: the baselines (MathBERT, MetaMath) are text-only, while the proposed models consume the combined stimulus-option image constructed in Figure 2. Since the corpus is described as \"inherently visual,\" the reported margin (Pearson 0.85 vs. 0.75) is fully consistent with the alternative explanation that the advantage comes from image input alone, not from the simulated-respondent training objective in Eq. (4). This undermines the abstract's and Section 3's central claim that the simulated-respondent framing \"recovers difficulty-relevant signal that direct stimulus-to-parameter regression does not capture.\" To establish the mechanism, the authors must add a vision-capable regression baseline (e.g., the same multimodal encoder with a linear head regressing directly to b) or an ablation of the proposed model with text-only input. Without such controls, the comparative headline is not supported.","section":"Section 3, Table 5"},{"comment":"The training loss uses the raw softmax probabilities v_{\\iota_k} for the four option tokens as response probabilities. Because the softmax is over the entire vocabulary, these token probabilities are not the probabilities of selecting each option; they are marginal token probabilities that can be influenced by the rest of the vocabulary. The same issue applies at inference, where the \"predicted option probabilities\" are read directly from the token logits. This is a load-bearing assumption: if the model assigns non-negligible probability to other tokens, the reconstructed IRT parameters will be systematically biased. The authors should either renormalize over the four option tokens or provide empirical evidence (e.g., the sum of the four option-token probabilities over a sample of prompts) that the remaining vocabulary mass is negligible.","section":"Section 2.4.2, Eq. (4)"},{"comment":"The title and abstract promise reconstruction of \"MCM curves,\" and Eq. (iv) defines an objective for recovering the MCM parameters {a_ik, b_ik, d_ik}, but Table 5 evaluates only the 3PL parameters (a, b, c). No results are reported for distractor-level parameters or for the MCM fit itself. As written, the paper does not support the stated claim of reconstructing MCM curves; at minimum, report the recovery accuracy for MCM parameters or explicitly narrow the contribution to 3PL difficulty and related parameters. This is essential for assessing whether the approach captures \"the strength of distractor items\" as discussed in Sections 2.2 and 4.","section":"Sections 2.3 and 3"},{"comment":"The reported correlations treat test items as independent, but items are generated from only 146 test item models, so the effective sample size for evaluating item-level generalization is at most 146, not 730. Items within an item model share structure and the model's predictions are likely correlated, yet no account is taken of this clustering (e.g., bootstrapping by item model). Without a measure of uncertainty, the \"clear margin\" between 0.85 and 0.75 cannot be assessed; this issue is especially relevant to the claim of a marked improvement over the MetaMath baseline.","section":"Section 3"},{"comment":"The computation of the target probabilities \\rho_{ijk} in Eq. (iii) requires integrating a normal density times the MCM probability over each ability interval, but the paper does not state how these integrals are evaluated (e.g., Gauss-Hermite quadrature, adaptive quadrature, or Monte Carlo). Without this detail, the method is not reproducible, and the choice of approximation could affect the targets and hence the trained model. Please specify the numerical integration scheme and any relevant convergence settings.","section":"Section 2.3"}],"minor_comments":[{"comment":"Table 1 is difficult to read: the columns for item models, items, and responses are not clearly aligned, and response counts such as \"363310.2\\times 10^6\" are confusingly formatted; please use plain numbers or clearly labeled units.","section":"Table 1"},{"comment":"The system prompt includes the correct answer in the instruction to the simulated student; please clarify whether this is intentional and how it interacts with the goal of reproducing students' systematic errors, since providing the correct answer could make the task of selecting wrong options artificially driven.","section":"Section 2.4.2"},{"comment":"No error bars, standard deviations, or multiple-seed results are reported for any model; for the \"clear margin\" claim in Section 3, at least a variance estimate across fine-tuning seeds or item-model resamples is needed.","section":"Section 2.5"},{"comment":"The RMSE of 0.55 for the difficulty parameter is reported only after the \"linear development-set correction.\" Because Pearson correlation is invariant to affine transformations, the reported correlation is unaffected, but the raw RMSE before the correction should be reported to make the description of the procedure unambiguous.","section":"Section 3"},{"comment":"The descriptive labels for ability bands (e.g., \"Emerging foundations\") and difficulty bands (e.g., \"Very Hard\") are nonstandard; please provide conventional labels or a reference for the band definitions, which are used in the QWK computation.","section":"Tables 2 and 4"},{"comment":"The paper cites \"Ormerod, 2026\" as prior work without a full bibliographic entry, making it impossible for readers to verify the relationship to the present manuscript; please add the arXiv identifier or venue.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The stress-test note correctly identifies the modality confound, and I agree that the paper currently cannot support the mechanism claim. The item-model clustering issue is also important: the effective sample size for the test set is 146 item models, not 730 items. In revision, the authors should be strongly encouraged to add a vision-capable regression baseline or a text-only ablation, and to report uncertainty estimates clustered by item model."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Thanks for passing this along. The headline result is plausible and the paper is a genuine extension of the author's text-only ICC reconstruction framework to multimodal math items and to 3PL and MCM curves, on a large real College Board corpus. The absolute difficulty reconstruction (Pearson 0.85 on held-out items, clean item-model split) is worth taking seriously, and the c-parameter recovery at 0.48, with text baselines near zero, is the most honest evidence that the simulated-respondent objective captures something about distractor behavior rather than surface text features.\n\nThe soft spot is exactly what the stress-test note says. Section 3 compares the multimodal 9B Qwen against text-only baselines, on items the paper itself calls inherently visual. The margin (0.85 vs 0.75; QWK 0.835 vs 0.692) is fully consistent with the image input alone driving the advantage, so the mechanism claim -- that simulating students recovers signal direct regression misses -- is not established by this comparison. The Discussion's inference that a model 'merely reading the question' cannot infer low-ability distractor pull is undercut by the model's own input: it sees the image. You need a vision-capable regression baseline, e.g. the same Qwen with a linear head trained directly to b, before you can attribute the margin to the simulated-respondent framing. The absolute reconstruction claim may survive; the comparative one does not.\n\nMinor but real: the paper reads next-token probabilities for the four option tokens without renormalizing over the option set. The softmax is over the whole vocabulary, so those four values don't sum to one, and the MSE in Eq. 4 treats them as if they did. That needs a fix or a justification. Also, there are no error bars or multiple seeds; with one run, the 4B-to-9B difference (0.80 vs 0.85) could be noise. The development-set linear correction is a calibrated step and should be applied identically to baselines or the RMSE comparison is unbalanced.\n\nCredit where due: real response data with rich per-item counts, item-model split to avoid leakage, interval-integrated ability bins as an improvement over point evaluation, and an honest account of design constraints. No code or data, which is a limit but not a killer in this proprietary setting.\n\nBottom line: worth a serious referee, but the referee should demand a vision-capable baseline, renormalization, and uncertainty before publication. I would not cite it in its current form, and I would bring it to reading group mainly to discuss what the modality confound means for the simulated-respondent literature.","headline":"A real multimodal extension of the simulated-respondent method with a plausible 0.85 difficulty correlation, but the margin over baselines is confounded by image input and the mechanism claim needs a vision-capable regression baseline.","tokens_in":10049,"tokens_out":2884,"would_cite":false,"duration_ms":27602,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Fine-tuning a multimodal language model to imitate students at fixed ability levels lets it recover the response curves behind test items and predict held-out difficulty with a Pearson correlation of 0.85.","keywords":["item response theory","three-parameter logistic model","multiple-choice model","multimodal LLM fine-tuning","simulated student responses","item difficulty prediction","quadratic weighted kappa","Qwen3.5"],"falsifier":"Run the same simulated-respondent fine-tuning on a text-only version of the items that strips out every image; if the difficulty correlation stays near 0.85, the image channel is not the source of the gain, whereas a drop toward the 0.68–0.75 baseline range would show the comparison drove the result. As a second check, renormalize the four option probabilities over only the four option tokens before refitting the curves; large shifts in fitted b values would reveal that the full-vocabulary softmax assumption is load-bearing.","tokens_in":9004,"feed_emoji":"🎓","tokens_out":11045,"duration_ms":89699,"temperature":0.7,"pith_summary":"This paper tries to show that item parameters—especially difficulty—can be estimated from a fine-tuned multimodal language model without collecting new student response data. The strategy is to fine-tune the model to act like a student at each of twenty discrete ability levels, so its next-token probabilities for the four answer options match empirically calibrated response curves. On a held-out set of items, the 9-billion-parameter model reaches a Pearson correlation of 0.85 and a quadratic weighted kappa of 0.835 for difficulty, exceeding the MathBERT (0.68) and MetaMath (0.75) regression baselines. If true, this would reduce the expense and security risk of field-testing by letting item parameters be read off the stimulus itself.","feed_headline":"Student-simulating language model predicts item difficulty at 0.85","feed_subtitle":"A 9B multimodal model exceeds text-only baselines by reading option probabilities from simulated students.","key_machinery":"The object that carries the argument is the discrete ability-level response-probability target: for item $i$, option $k$, and ability interval $I_j = (\\gamma_{j-1}, \\gamma_j]$, the model is trained to match $\\rho_{ijk} = \\frac{1}{w_j}\\int_{\\gamma_{j-1}}^{\\gamma_j} f(\\theta) P_{i\\theta k}\\,d\\theta$, where $w_j$ is the normal-prior probability of that ability interval and $P_{i\\theta k}$ is the option probability from the multiple-choice model. The 3PL is $P(X=1)=c+(1-c)\\sigma(a(\\theta-b))$, and the MCM sets option probabilities from $\\exp(a_{ik}\\theta+b_{ik})+d_{ik}$ normalized across options. The fine-tuning loss is the mean squared error between the LLM's four next-token option probabilities and the $\\rho$ values; after training, the same predicted probabilities are used to refit 3PL and MCM parameters by weighted least squares. This bridges raw response data, item response curves, and LLM outputs.","core_discovery":"The central claim is that training a multimodal LLM to replicate the systematic option-choice patterns of students across a labeled range of ability levels makes it implicitly capture the response probabilities encoded in the three-parameter logistic and multiple-choice models. The fine-tuned Qwen3.5-9B model scores 0.85 Pearson correlation and 0.835 QWK on held-out difficulty, with an RMSE of 0.55 after a linear correction, while recovering the guessing parameter at 0.48 and the discrimination parameter at only 0.31. The authors interpret the strong difficulty and guessing results as evidence that the simulated-respondent framing captures response behavior rather than surface stimulus features, and the weak discrimination recovery as a cost of discretizing ability.","pith_inferences":["A text-only variant of the simulated-respondent training would isolate how much of the improvement comes from image input; the paper's current design cannot separate image access from response simulation.","Because the option probabilities are read from a full-vocabulary softmax, renormalizing over the four option tokens is a natural robustness check that could change the fitted parameters.","The same prompting scheme could be extended to produce joint response patterns across a set of items, which would let test assemblers simulate form-level statistics before any field test."],"forward_implications":["Item difficulty can be obtained directly from a fine-tuned model's option probabilities, so routine field-testing for difficulty calibration could be reduced or supplemented by simulation.","The same setup handles items whose stimuli and options contain images, unifying text-only and image-plus-text items under one input representation.","The guessing parameter is recovered far better than by regression baselines (0.48 versus 0.25–0.28), suggesting distractor attraction is genuinely modeled.","Discrimination recovery is weak (0.31), and the paper attributes the loss to ability discretization, implying finer ability bins should improve slope estimates.","Under the five-band difficulty labels, predicted and calibrated labels agree at QWK 0.835, so the model supports the coarse labels educators actually receive."],"supporting_citations":[{"why":"Defines the three-parameter logistic model whose parameters the paper reconstructs.","marker":"(Thissen and Steinberg, 1986)"},{"why":"Defines the multiple-choice model for per-option response probabilities, the second reconstruction target.","marker":"(Thissen and Steinberg, 1984)"},{"why":"Introduces the discrete ability-level reconstruction of item characteristic curves that this paper extends to multimodal items.","marker":"(Ormerod, 2026)"},{"why":"Supplies the parameter-efficient fine-tuning approach for simulating students of varying ability.","marker":"(Scarlatos et al., 2025)"},{"why":"Establishes the simulated-respondent use of LLMs that the paper builds on.","marker":"(Maeda, 2025)"},{"why":"Provides the LoRA adaptation used to fine-tune the Qwen models.","marker":"(Hu et al., 2021)"},{"why":"Provides the Qwen3.5 multimodal base model that is fine-tuned.","marker":"(Qwen Team, 2026)"},{"why":"Supplies MathBERT, one of the regression baselines.","marker":"(Peng et al., 2021)"},{"why":"Supplies MetaMath, the stronger regression baseline.","marker":"(Yu et al., 2024)"},{"why":"Supplies the standard evaluation metrics used for difficulty prediction.","marker":"(AlKhuzaey et al., 2024)"}],"fun_headline_variants":["LLM that plays student nails item difficulty at 0.85","Simulated students help LLM predict test question difficulty","Multimodal LLM mimics test-takers to estimate item difficulty","Fine-tuned Qwen3.5 reads option probabilities for item difficulty","Student-simulating model cracks item difficulty estimation"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing comparison assumes that text-only regression models are the proper baselines for a multimodal model, so the reported gain could come from seeing images rather than from the simulated-respondent training.","fun_headline_variants_meta":{"raw":{"variants":["LLM that plays student nails item difficulty at 0.85","Simulated students help LLM predict test question difficulty","Multimodal LLM mimics test-takers to estimate item difficulty","Fine-tuned Qwen3.5 reads option probabilities for item difficulty","Student-simulating model cracks item difficulty estimation"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000149,"raw_usage":{"total_tokens":1124,"prompt_tokens":805,"completion_tokens":319,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":421,"completion_tokens_details":{"reasoning_tokens":234}},"tokens_in":421,"tokens_out":319,"duration_ms":3219,"temperature":1.0,"reasoning_tokens":234,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T04:11:30.795835+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same simulated-respondent fine-tuning on a text-only version of the items that strips out every image; if the difficulty correlation stays near 0.85, the image channel is not the source of the gain, whereas a drop toward the 0.68–0.75 baseline range would show the comparison drove the result. As a second check, renormalize the four option probabilities over only the four option tokens before refitting the curves; large shifts in fitted b values would reveal that the full-vocabulary softmax assumption is load-bearing.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the three-parameter logistic model whose parameters the paper reconstructs."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Establishes the simulated-respondent use of LLMs that the paper builds on."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the Qwen3.5 multimodal base model that is fine-tuned."}],"review_version":1}