{"id":"7bb75445-7638-41ff-8df4-28d5743bbb0c","arxiv_id":"2601.09886","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Language-model surprisal predicts reading times better than cloze surprisal because it is higher-resolution, semantically discriminating, and frequency-sensitive.","lead":"This paper tests why language-model (LM) probabilities predict reading times better than human cloze responses. It finds the advantage comes from the LM's fine-grained probability resolution, its ability to distinguish semantically similar words, and its treatment of low-frequency words.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"H1's resolution manipulation confounds sampling noise and heavy add-one smoothing (S=200) with reduced resolution; a noiseless compression control is needed before attributing the fit drop to low resolution.","rationale":"The reader's weakest_assumption identifies the same core issue: the manipulations are assumed to selectively remove the hypothesized property, but they introduce other changes. I focus on H1 because it is the most consequential: the paper interprets the H1 result as showing that matching the number of responses eliminates GPT2's advantage, which motivates practical recommendations about cloze study resolution. The proposed control directly tests whether the smoothing compression—a confound the authors do not address—can reproduce the loss of fit. If the control produces a similar drop, the conclusion is not about resolution per se. This is a concrete, executable check that would settle the concern. The reader's verdict of CONDITIONAL already accounts for this type of issue, so my critique does not change the verdict; it sharpens the specific test needed.","tokens_in":15564,"tokens_out":7988,"duration_ms":95623,"concrete_test":"Construct a noiseless compression control for H1: for each context, take the original full-resolution GPT2 probabilities p(w) and pass them through the same add-one smoothing formula as Eq. 2 without sampling, i.e., compute p_ctrl(w) = (N·p(w)+1)/(N+S), using the same N and S=200, then renormalize if needed. This preserves the full vocabulary and relative ordering but applies exactly the same probability compression as the H1 manipulation, while removing sampling noise and count discretization. Evaluate this control in the same LME cross-validation pipeline. If p_ctrl loses as much held-out log-likelihood relative to unmanipulated GPT2 as the actual H1 predictor, then the H1 result is attributable to the heavy smoothing rather than to resolution, and the resolution hypothesis is not specifically supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim—that GPT2 surprisal outperforms cloze surprisal because of fine-grained resolution, semantic distinctions, and low-frequency accuracy—rests on Experiment 2's manipulations being selective. The H1 manipulation (Eq. 2) is the most load-bearing because Section 5 uses it to argue that matching the number of responses makes GPT2 surprisal no stronger than cloze surprisal. However, the manipulation does not only reduce resolution: it injects sampling noise and applies add-one smoothing with S=200, where N≈40–90. The denominator N+S is dominated by S, compressing all probabilities into a narrow range (roughly 1/(N+S) to (N+1)/(N+S)). This compression alone would degrade any predictor, independent of the discrete sampling that defines resolution. The paper lacks a control condition that applies a comparable amount of probability compression without the sampling step (e.g., smoothing the original GPT2 probabilities directly). Without such a control, the observed decrease in fit for GPT2-H1 could be due to the smoothing transformation or to generic noise, not specifically to the loss of fine-grained resolution. The same selectivity issue applies to H2 (k-means coarse-graining removes arbitrary distinctions, not just semantic ones) and H3 (thresholding removes most vocabulary and renormalizes, a drastic transformation), but H1 is the linchpin because it is the most direct and novel evidence for the resolution hypothesis.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper investigates why GPT-2 surprisal predicts human reading times better than cloze surprisal. Experiment 1 compares cloze and GPT-2 surprisal in linear mixed-effects models across four reading-time datasets (BK21 SPR; Provo FP/GP; UCL SPR/FP/GP) and finds, consistent with prior work, that GPT-2 surprisal generally subsumes cloze surprisal. Experiment 2 then manipulates GPT-2 probabilities in three ways intended to selectively remove properties that distinguish LMs from cloze estimates: H1 samples a limited number of responses to match cloze resolution (Eq. 2), H2 aggregates probabilities over semantic clusters (Eq. 3), and H3 removes/re-normalizes low-frequency vocabulary (Eq. 4). The authors report that all three manipulations significantly reduce fit to reading times and conclude that the LM advantage is attributable to fine-grained resolution, semantic distinctions, and accurate low-frequency probabilities. Experiment 3 examines similarity-adjusted surprisal and finds that both cloze-based and GPT-2-based SA surprisal fit worse than count-and-divide cloze surprisal. The paper closes by proposing methodological triangulation and future experiments on human sensitivity to fine-grained distinctions.","tokens_in":15962,"tokens_out":2726,"duration_ms":32811,"significance":"If the manipulations in Experiment 2 are truly selective, the paper would provide a valuable mechanistic account of why LM surprisal outperforms cloze surprisal, with direct implications for how to improve cloze-based predictability measures and for the interpretation of prior LM/cloze comparisons. The study is carefully designed in several respects: it uses multiple datasets, cross-validated model comparison, permutation tests with Bonferroni correction, and it makes code available. The core comparison in Experiment 1 is not circular because LM probabilities and manipulated predictors are evaluated against independently collected reading times. However, the central inferences in Experiment 2 depend on the assumption that each manipulation changes only the hypothesized property. The H1 manipulation, in particular, confounds resolution with sampling noise and heavy add-one smoothing, and the H2/H3 results are not uniform across the parameter settings reported in Appendix C. These issues are load-bearing for the paper's main claim, so the manuscript requires substantial revision before the conclusions can be accepted.","major_comments":[{"comment":"The H1 manipulation is not a selective test of resolution. Eq. 2 samples N words and then applies add-one smoothing with S=200, where N is approximately 40–90. The denominator N+S is therefore dominated by S, compressing all probabilities into a narrow range roughly from 1/(N+S) to (N+1)/(N+S). This transformation alone would degrade any predictor, independent of the discrete sampling that defines resolution. The paper lacks a control condition that applies comparable probability compression without sampling, e.g., smoothing the original GPT-2 probabilities directly with the same S and then comparing fit. Without such a control, the observed decrease in fit for GPT2-H1 cannot be attributed specifically to low resolution rather than generic degradation from smoothing or sampling noise. This is especially important because Section 5 highlights the H1 result as the paper's most noteworthy f","section":"Section 3.1, Eq. 2 (H1)"},{"comment":"The headline H2/H3 results are presented for k=80 and a frequency threshold of 10^4 per billion, described in footnote 7 as 'representative examples of the similar trend observed across settings.' Appendix C shows that the pattern is not uniform. For H2 with k=20, GPT2-H2 surprisal predicts RTs over and above cloze surprisal on BK21 SPR; with k=40, cloze surprisal predicts over GPT2-H2 on BK21 SPR and UCL FP, and there is no significant difference on UCL GP. For H3, raising the threshold to 10^5 removes the significant advantage on Provo GP. The claim of a 'similar trend' is therefore too strong. The authors should either report results across all settings in the main text, or provide a principled, pre-registered criterion for selecting k and the threshold that is not outcome-dependent.","section":"Section 3.2 and Appendix C (H2/H3)"},{"comment":"The cloze preprocessing choices—smoothing factor S=200 and the squared-surprisal transform S(wt)^2—are selected by maximizing in-sample fit on approximately 50% of the same RT data used for all subsequent comparisons. These choices are then applied not only to cloze probabilities but also to every manipulated GPT-2 variant (e.g., Eq. 2 and the H2/H3 definitions), even though the manipulated distributions have different sample sizes, vocabularies, and scales. This raises a risk that the observed differences reflect a mismatch between the fixed transform/smoothing and the manipulated distributions rather than the hypothesized cognitive property. The authors should provide a sensitivity analysis showing that the conclusions hold under alternative smoothing factors and transforms, or justify why the cloze-selected settings are appropriate for the manipulated LM distributions.","section":"Section 2.4 and Section 3.1"},{"comment":"The paper concludes from Experiment 3 that 'count-and-divide is not an unreasonable method to convert cloze responses into probabilities.' However, the comparison in Figure 5 shows that both similarity-adjusted predictors perform poorly, and the results are described as 'inconclusive' about the quality of alternatives. This is a secondary claim, but as written the discussion somewhat overinterprets null or weak effects. If this conclusion is retained, the authors should clarify the inferential limitations of using a null difference to support the adequacy of count-and-divide cloze surprisal.","section":"Section 4 (Experiment 3) and discussion"}],"minor_comments":[{"comment":"Typo: 'processces' should be 'processes' in the second paragraph.","section":"Introduction"},{"comment":"The aggregate panel pools 60 folds from six different measures and reports error bars over folds. Since folds within a dataset are not independent (they share subjects and items), the displayed SEM and the permutation test over these folds may overstate precision. Consider reporting per-dataset results alongside the aggregate, or using a hierarchical/blocked bootstrap.","section":"Figure 3"},{"comment":"Table 2 reports increases in log likelihood but not standard errors or significance tests across smoothing/transform variants. Given that the authors conclude 'the smoothing factor and the power transforms of surprisal did not have a very large effect,' it would be helpful to show the variability across folds or datasets.","section":"Appendix B, Table 2"},{"comment":"The formula for renormalization in Eq. 4 is slightly opaque; in particular, the role of |VF|/(|VF|+1) as a smoothing-like term is not explained. A brief derivation or intuitive explanation would improve readability.","section":"Section 3.1, Eq. 4 (H3)"}],"recommendation":"major_revision","confidential_remarks":"The paper addresses an important question and has a solid overall framework, but the load-bearing manipulations in Experiment 2 need stronger assumptions of selectivity. In particular, H1 requires a control for smoothing/noise, and the H2/H3 parameter dependence shown in Appendix C needs to be addressed rather than relegated to a footnote. I would be willing to review a revised version."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Worth knowing: this is the first paper I've seen that tries to explain why LM surprisal beats cloze surprisal by directly manipulating LM probabilities. Experiment 1 is a clean re-establishment of the LM advantage. Experiment 2 is the novel part: three concrete hypotheses, four datasets, cross-validated LMEs, and the authors release code. The H1 result — that matching the number of responses makes GPT-2 no better than cloze — is genuinely striking.\n\nThat said, the manipulations aren't as selective as the headline suggests. H1 samples N tokens and then applies add-one smoothing with S=200. With N around 40-90, the denominator is dominated by S, so the resulting probabilities are compressed into a narrow range. A large part of the fit drop could be generic degradation from heavy smoothing plus sampling noise, not specifically from low resolution. There's no noiseless compression control. This matters because the discussion leans on H1 to argue that resolution alone explains the advantage. That claim is under-supported. The stress-test note is right.\n\nH2 and H3 have a different but related problem: the main text reports k=80 and threshold 10^4 as representative. The appendix shows k=20 reverses some comparisons and threshold 10^5 loses Provo GP. They at least report this, but 'support for all three hypotheses' overstates the stability of the pattern.\n\nA smaller point: the cloze predictor uses S=200 and a squared-surprisal transform selected by fitting half the RT data. That's a defensible modeling choice, but it means the comparison is tuned cloze against untuned GPT-2, which biases the test against cloze, not the other way round.\n\nWho this is for: people who work on predictability measures in sentence processing. It deserves a serious referee — the core question is important and the experimental template is reusable. The flaws are addressable, not fatal. I'd want the H1 control and a more honest treatment of the parameter sensitivity before I'd trust the three-factor explanation, but I'd still engage with the paper.","headline":"A genuinely new decomposition attempt with a clean Experiment 1, but the H1 manipulation conflates resolution with heavy smoothing, and H2/H3 parameter choices are post hoc.","tokens_in":16388,"tokens_out":4318,"would_cite":true,"duration_ms":47664,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that language-model surprisal beats cloze surprisal at predicting reading times because it assigns fine-grained probabilities, distinguishes semantically similar words, and handles low-frequency words—and it supports this b","keywords":["language model surprisal","cloze task","reading times","psycholinguistics","predictability","GPT2","resolution","word frequency"],"falsifier":"One concrete test: apply a control manipulation to GPT2 probabilities that adds the same amount of random perturbation as the H1 sampling (e.g., sampling from a flat or shuffled distribution) without reducing resolution, and check whether the drop in reading-time fit disappears; if it still drops, the H1 result is not specific to resolution. Alternatively, double or triple the number of cloze responses per context and see whether cloze surprisal's fit rises to match GPT2 surprisal's; if it does not, resolution alone is insufficient.","tokens_in":15469,"feed_emoji":"🧠","tokens_out":4812,"duration_ms":51276,"temperature":0.7,"pith_summary":"The paper asks why language-model surprisal predicts human reading times more strongly than surprisal derived from human cloze responses, and tries to identify the exact source of that advantage rather than simply endorsing LMs. It tests three hypotheses: LMs assign more finely-grained probabilities than the roughly 40–90 human responses per context, they distinguish semantically similar words such as 'couch' vs. 'sofa,' and they assign accurate probabilities to low-frequency words. The authors manipulate GPT2's output to degrade each of these properties and find that every manipulation significantly reduces the fit to reading times, supporting all three hypotheses. If the conclusion holds, it frames the LM advantage as coming from specific statistical properties, which in turn suggests both how cloze studies could be improved and what experiments could test whether human prediction is as fine-grained as LMs.","feed_headline":"Three degradations expose the source of LM surprisal's advantage","feed_subtitle":"Targeted manipulations of GPT2 probabilities show that high resolution, semantic sensitivity, and low-frequency accuracy each carry the edge","key_machinery":"The paper's key mechanism is a set of three interventions on a language model's conditional next-word distribution. To mimic limited cloze resolution, words are sampled from the model's distribution (same count as human responses) and converted to probabilities with add-one smoothing. To erase semantic sensitivity, the model's vocabulary is k-means clustered (k=80 reported as representative) and each word's probability is replaced by the total probability of its cluster. To remove low-frequency information, all tokens below a frequency threshold (10^4 per billion) have their probability set to zero and the remaining probabilities are renormalized. These manipulated surprisals are then compar","core_discovery":"The central discovery is that the modeling advantage of GPT2 surprisal over cloze surprisal can be recreated or annulled by targeted interventions. When GPT2 probabilities are degraded to have the resolution of a typical cloze study (tens of responses with add-one smoothing), to lose distinctions among words in the same semantic cluster, or to ignore low-frequency tokens, the resulting surprisal becomes no better—and on several measures worse—than cloze surprisal at predicting reading times. This indicates that high resolution, sensitivity to semantically similar alternatives, and accurate low-frequency probabilities are each load-bearing components of the LM's predictive advantage.","pith_inferences":["A practical design implication: if resolution is the dominant factor, then cloze norming studies that aggregate many more respondents (or use crowd-sourced response distributions) might make cloze surprisal approach LM surprisal's fit, even without changing the underlying human prediction mechanism.","The H1 manipulation's drop in fit could be partly an artifact of smoothing and sampling noise rather than resolution per se; a control condition that adds the same amount of random noise to the LM distribution without changing its resolution would separate these.","The frequency hypothesis suggests a testable behavioural prediction: low-frequency but plausible continuations should produce measurable reading-time delays if readers' predictions are as frequency-sensitive as LMs; if not, the LM's low-frequency advantage may be a corpus artifact.","The similarity-adjusted surprisal results (Experiment 3) suggest that replacing count-based cloze probabilities with embedding-weighted sums does not help, implying that the relevant semantic information may already be captured by the LM distribution itself."],"forward_implications":["Cloze surprisal's predictive power is limited by response count; increasing the number of cloze responses per context should narrow the gap with LM surprisal.","Collecting more cloze responses alone will not address limitations caused by human insensitivity to semantic near-synonyms and low-frequency words, so the cloze task may need complementary measures.","The findings motivate testing whether human readers genuinely differentiate semantically similar and low-frequency alternatives, using LM-informed stimuli.","Methodological triangulation with timed cloze or maze tasks could help separate deliberate reflection from fast predictive processing."],"fun_headline_variants":["Why LM surprisal beats cloze: three load-bearing strengths","Degrading GPT2 to cloze level kills its reading-time advantage","Cloze's low resolution, semantics, and rare words cost it the edge","LM surprisal wins on resolution, semantic nuance, and rare words","Three cloze flaws explain why GPT2 surprisal predicts better"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"Each manipulation must selectively remove only the hypothesised property from the LM distribution, but the H1 manipulation, in particular, also adds sampling noise and drastically shrinks probabilities through smoothing, so the reduced fit could reflect generic degradation rather than specifically low resolution.","fun_headline_variants_meta":{"raw":{"variants":["Why LM surprisal beats cloze: three load-bearing strengths","Degrading GPT2 to cloze level kills its reading-time advantage","Cloze's low resolution, semantics, and rare words cost it the edge","LM surprisal wins on resolution, semantic nuance, and rare words","Three cloze flaws explain why GPT2 surprisal predicts better"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00023,"raw_usage":{"total_tokens":1273,"prompt_tokens":652,"completion_tokens":621,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":396,"completion_tokens_details":{"reasoning_tokens":527}},"tokens_in":396,"tokens_out":621,"duration_ms":6403,"temperature":1.0,"reasoning_tokens":527,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-03T10:26:49.700005+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"One concrete test: apply a control manipulation to GPT2 probabilities that adds the same amount of random perturbation as the H1 sampling (e.g., sampling from a flat or shuffled distribution) without reducing resolution, and check whether the drop in reading-time fit disappears; if it still drops, the H1 result is not specific to resolution. Alternatively, double or triple the number of cloze responses per context and see whether cloze surprisal's fit rises to match GPT2 surprisal's; if it does not, resolution alone is insufficient.","supporting_citations":[],"review_version":1}