Clozing the Gap: Exploring Why Language Model Surprisal Outperforms Cloze Surprisal

Byung-Doh Oh; Sathvik Nair

REVIEW 4 major objections 4 minor 3 references

Clozing the Gap: Exploring Why Language Model Surprisal Outperforms Cloze Surprisal

T0 review · 4 major / 4 minor · reviewed 2026-08-03 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims that language-model surprisal beats cloze surprisal at predicting reading times because it assigns fine-grained probabilities, distinguishes semantically similar words, and handles low-frequency words—and it supports this b

desk verdict A genuinely new decomposition attempt with a clean Experiment 1, but the H1 manipulation conflates resolution with heavy smoothing, and H2/H3 parameter choices are post hoc. read the letter →

arxiv 2601.09886 v2 pith:F5A5Z43W submitted 2026-01-14 cs.CL

Sathvik Nair , Byung-Doh Oh This is my paper · ORCID

classification cs.CL

keywords languagemodelsurprisalclozetaskreadingtimespsycholinguisticspredictabilityGPT2resolutionwordfrequency

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper asks why language-model surprisal predicts human reading times more strongly than surprisal derived from human cloze responses, and tries to identify the exact source of that advantage rather than simply endorsing LMs. It tests three hypotheses: LMs assign more finely-grained probabilities than the roughly 40–90 human responses per context, they distinguish semantically similar words such as 'couch' vs. 'sofa,' and they assign accurate probabilities to low-frequency words. The authors manipulate GPT2's output to degrade each of these properties and find that every manipulation significantly reduces the fit to reading times, supporting all three hypotheses. If the conclusion holds, it frames the LM advantage as coming from specific statistical properties, which in turn suggests both how cloze studies could be improved and what experiments could test whether human prediction is as fine-grained as LMs.

What carries the argument

The paper's key mechanism is a set of three interventions on a language model's conditional next-word distribution. To mimic limited cloze resolution, words are sampled from the model's distribution (same count as human responses) and converted to probabilities with add-one smoothing. To erase semantic sensitivity, the model's vocabulary is k-means clustered (k=80 reported as representative) and each word's probability is replaced by the total probability of its cluster. To remove low-frequency information, all tokens below a frequency threshold (10^4 per billion) have their probability set to zero and the remaining probabilities are renormalized. These manipulated surprisals are then compar

What would settle it

One concrete test: apply a control manipulation to GPT2 probabilities that adds the same amount of random perturbation as the H1 sampling (e.g., sampling from a flat or shuffled distribution) without reducing resolution, and check whether the drop in reading-time fit disappears; if it still drops, the H1 result is not specific to resolution. Alternatively, double or triple the number of cloze responses per context and see whether cloze surprisal's fit rises to match GPT2 surprisal's; if it does not, resolution alone is insufficient.

Watch

Extended reading notes

Core claim

The central discovery is that the modeling advantage of GPT2 surprisal over cloze surprisal can be recreated or annulled by targeted interventions. When GPT2 probabilities are degraded to have the resolution of a typical cloze study (tens of responses with add-one smoothing), to lose distinctions among words in the same semantic cluster, or to ignore low-frequency tokens, the resulting surprisal becomes no better—and on several measures worse—than cloze surprisal at predicting reading times. This indicates that high resolution, sensitivity to semantically similar alternatives, and accurate low-frequency probabilities are each load-bearing components of the LM's predictive advantage.

Load-bearing premise

Each manipulation must selectively remove only the hypothesised property from the LM distribution, but the H1 manipulation, in particular, also adds sampling noise and drastically shrinks probabilities through smoothing, so the reduced fit could reflect generic degradation rather than specifically low resolution.

Editorial extensions

If this is right

Cloze surprisal's predictive power is limited by response count; increasing the number of cloze responses per context should narrow the gap with LM surprisal.
Collecting more cloze responses alone will not address limitations caused by human insensitivity to semantic near-synonyms and low-frequency words, so the cloze task may need complementary measures.
The findings motivate testing whether human readers genuinely differentiate semantically similar and low-frequency alternatives, using LM-informed stimuli.
Methodological triangulation with timed cloze or maze tasks could help separate deliberate reflection from fast predictive processing.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

A practical design implication: if resolution is the dominant factor, then cloze norming studies that aggregate many more respondents (or use crowd-sourced response distributions) might make cloze surprisal approach LM surprisal's fit, even without changing the underlying human prediction mechanism.
The H1 manipulation's drop in fit could be partly an artifact of smoothing and sampling noise rather than resolution per se; a control condition that adds the same amount of random noise to the LM distribution without changing its resolution would separate these.
The frequency hypothesis suggests a testable behavioural prediction: low-frequency but plausible continuations should produce measurable reading-time delays if readers' predictions are as frequency-sensitive as LMs; if not, the LM's low-frequency advantage may be a corpus artifact.
The similarity-adjusted surprisal results (Experiment 3) suggest that replacing count-based cloze probabilities with embedding-weighted sums does not help, implying that the relevant semantic information may already be captured by the LM distribution itself.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Worth knowing: this is the first paper I've seen that tries to explain why LM surprisal beats cloze surprisal by directly manipulating LM probabilities. Experiment 1 is a clean re-establishment of the LM advantage. Experiment 2 is the novel part: three concrete hypotheses, four datasets, cross-validated LMEs, and the authors release code. The H1 result — that matching the number of responses makes GPT-2 no better than cloze — is genuinely striking.

That said, the manipulations aren't as selective as the headline suggests. H1 samples N tokens and then applies add-one smoothing with S=200. With N around 40-90, the denominator is dominated by S, so the resulting probabilities are compressed into a narrow range. A large part of the fit drop could be generic degradation from heavy smoothing plus sampling noise, not specifically from low resolution. There's no noiseless compression control. This matters because the discussion leans on H1 to argue that resolution alone explains the advantage. That claim is under-supported. The stress-test note is right.

H2 and H3 have a different but related problem: the main text reports k=80 and threshold 10^4 as representative. The appendix shows k=20 reverses some comparisons and threshold 10^5 loses Provo GP. They at least report this, but 'support for all three hypotheses' overstates the stability of the pattern.

A smaller point: the cloze predictor uses S=200 and a squared-surprisal transform selected by fitting half the RT data. That's a defensible modeling choice, but it means the comparison is tuned cloze against untuned GPT-2, which biases the test against cloze, not the other way round.

Who this is for: people who work on predictability measures in sentence processing. It deserves a serious referee — the core question is important and the experimental template is reusable. The flaws are addressable, not fatal. I'd want the H1 control and a more honest treatment of the parameter sensitivity before I'd trust the three-factor explanation, but I'd still engage with the paper.

Referee Report

4 major / 4 minor

Summary. The paper investigates why GPT-2 surprisal predicts human reading times better than cloze surprisal. Experiment 1 compares cloze and GPT-2 surprisal in linear mixed-effects models across four reading-time datasets (BK21 SPR; Provo FP/GP; UCL SPR/FP/GP) and finds, consistent with prior work, that GPT-2 surprisal generally subsumes cloze surprisal. Experiment 2 then manipulates GPT-2 probabilities in three ways intended to selectively remove properties that distinguish LMs from cloze estimates: H1 samples a limited number of responses to match cloze resolution (Eq. 2), H2 aggregates probabilities over semantic clusters (Eq. 3), and H3 removes/re-normalizes low-frequency vocabulary (Eq. 4). The authors report that all three manipulations significantly reduce fit to reading times and conclude that the LM advantage is attributable to fine-grained resolution, semantic distinctions, and accurate low-frequency probabilities. Experiment 3 examines similarity-adjusted surprisal and finds that both cloze-based and GPT-2-based SA surprisal fit worse than count-and-divide cloze surprisal. The paper closes by proposing methodological triangulation and future experiments on human sensitivity to fine-grained distinctions.

Significance. If the manipulations in Experiment 2 are truly selective, the paper would provide a valuable mechanistic account of why LM surprisal outperforms cloze surprisal, with direct implications for how to improve cloze-based predictability measures and for the interpretation of prior LM/cloze comparisons. The study is carefully designed in several respects: it uses multiple datasets, cross-validated model comparison, permutation tests with Bonferroni correction, and it makes code available. The core comparison in Experiment 1 is not circular because LM probabilities and manipulated predictors are evaluated against independently collected reading times. However, the central inferences in Experiment 2 depend on the assumption that each manipulation changes only the hypothesized property. The H1 manipulation, in particular, confounds resolution with sampling noise and heavy add-one smoothing, and the H2/H3 results are not uniform across the parameter settings reported in Appendix C. These issues are load-bearing for the paper's main claim, so the manuscript requires substantial revision before the conclusions can be accepted.

major comments (4)

[Section 3.1, Eq. 2 (H1)] The H1 manipulation is not a selective test of resolution. Eq. 2 samples N words and then applies add-one smoothing with S=200, where N is approximately 40–90. The denominator N+S is therefore dominated by S, compressing all probabilities into a narrow range roughly from 1/(N+S) to (N+1)/(N+S). This transformation alone would degrade any predictor, independent of the discrete sampling that defines resolution. The paper lacks a control condition that applies comparable probability compression without sampling, e.g., smoothing the original GPT-2 probabilities directly with the same S and then comparing fit. Without such a control, the observed decrease in fit for GPT2-H1 cannot be attributed specifically to low resolution rather than generic degradation from smoothing or sampling noise. This is especially important because Section 5 highlights the H1 result as the paper's most noteworthy f
[Section 3.2 and Appendix C (H2/H3)] The headline H2/H3 results are presented for k=80 and a frequency threshold of 10^4 per billion, described in footnote 7 as 'representative examples of the similar trend observed across settings.' Appendix C shows that the pattern is not uniform. For H2 with k=20, GPT2-H2 surprisal predicts RTs over and above cloze surprisal on BK21 SPR; with k=40, cloze surprisal predicts over GPT2-H2 on BK21 SPR and UCL FP, and there is no significant difference on UCL GP. For H3, raising the threshold to 10^5 removes the significant advantage on Provo GP. The claim of a 'similar trend' is therefore too strong. The authors should either report results across all settings in the main text, or provide a principled, pre-registered criterion for selecting k and the threshold that is not outcome-dependent.
[Section 2.4 and Section 3.1] The cloze preprocessing choices—smoothing factor S=200 and the squared-surprisal transform S(wt)^2—are selected by maximizing in-sample fit on approximately 50% of the same RT data used for all subsequent comparisons. These choices are then applied not only to cloze probabilities but also to every manipulated GPT-2 variant (e.g., Eq. 2 and the H2/H3 definitions), even though the manipulated distributions have different sample sizes, vocabularies, and scales. This raises a risk that the observed differences reflect a mismatch between the fixed transform/smoothing and the manipulated distributions rather than the hypothesized cognitive property. The authors should provide a sensitivity analysis showing that the conclusions hold under alternative smoothing factors and transforms, or justify why the cloze-selected settings are appropriate for the manipulated LM distributions.
[Section 4 (Experiment 3) and discussion] The paper concludes from Experiment 3 that 'count-and-divide is not an unreasonable method to convert cloze responses into probabilities.' However, the comparison in Figure 5 shows that both similarity-adjusted predictors perform poorly, and the results are described as 'inconclusive' about the quality of alternatives. This is a secondary claim, but as written the discussion somewhat overinterprets null or weak effects. If this conclusion is retained, the authors should clarify the inferential limitations of using a null difference to support the adequacy of count-and-divide cloze surprisal.

minor comments (4)

[Introduction] Typo: 'processces' should be 'processes' in the second paragraph.
[Figure 3] The aggregate panel pools 60 folds from six different measures and reports error bars over folds. Since folds within a dataset are not independent (they share subjects and items), the displayed SEM and the permutation test over these folds may overstate precision. Consider reporting per-dataset results alongside the aggregate, or using a hierarchical/blocked bootstrap.
[Appendix B, Table 2] Table 2 reports increases in log likelihood but not standard errors or significance tests across smoothing/transform variants. Given that the authors conclude 'the smoothing factor and the power transforms of surprisal did not have a very large effect,' it would be helpful to show the variability across folds or datasets.
[Section 3.1, Eq. 4 (H3)] The formula for renormalization in Eq. 4 is slightly opaque; in particular, the role of |VF|/(|VF|+1) as a smoothing-like term is not explained. A brief derivation or intuitive explanation would improve readability.

Circularity Check

0 steps flagged · score 2.0 of 10

No significant circularity: core comparisons are against external RT data; tuning and post-hoc settings are validity concerns, not reductions to inputs.

full rationale

The paper's central chain—LM surprisal predicts reading times over cloze surprisal, and ablating fine-grained probability distinctions reduces that advantage—is evaluated against external by-word reading times (BK21, Provo, UCL). The predictors are computed from cloze responses or from GPT-2, and the RT data are not used to construct the predictors; therefore the core LM-vs-cloze comparison is not definitionally circular. The three manipulations are ablations: Eq. 2 uses the same smoothed count-and-divide form as Eq. 1 with counts sampled from GPT-2, but the outcome (decrease in held-out RT fit) is measured, not derived from the equation. H2 and H3 are k-means and frequency-threshold transformations whose effects on RT fit are likewise empirical. The paper does cite prior work by the same authors (Oh and Schuler 2023/2024/2025; Oh and Linzen 2025; Nair and Resnik 2023), but these support peripheral methodological choices (whitespace tokenization, frequency effects) and are not invoked as a uniqueness theorem or as the sole justification for the main claim. The most concrete concerns are methodological rather than circular: the cloze smoothing factor and squared-surprisal transform were selected on the same datasets ('we apply S(wt)^2 with a smoothing factor of S=200 to all cloze probabilities, as this setup achieved the best fit to the six measures'), and the representative H2/H3 settings were chosen post hoc ('we report results using 80 clusters and a threshold of 10^4 per billion as representative examples'), with Appendix C showing some non-uniformity. These affect interpretation and generalizability, but they are not cases where a claimed prediction reduces by construction to a fitted parameter or to a self-citation chain.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The paper introduces no new physical or theoretical entities. Its assumptions are domain-level: cloze-as-sampling, embeddings-as-semantics, wordfreq-as-frequency. The main extra 'parameters' are the smoothing/transform choices and the H2/H3 hyperparameters, which are tuned post hoc and may not be neutral across conditions.

free parameters (4)

Cloze smoothing factor S = 200
Selected to maximize cloze probability fit to RT data on ~50% of observations (Section 2.4, Table 2). Applied to cloze and to all manipulated GPT2 variants.
Surprisal power transform exponent = 2
S(w_t)^2 chosen as best functional form among six transforms on RT fit (Section 2.4).
Number of k-means clusters k (H2) = 80
Chosen as representative from {20, 40, 80, 100, 500, 1000}; results vary by k (Appendix C, Figure 6).
Frequency threshold (H3) = 10^4 per billion
Chosen as representative from {10^3, 10^4, 10^5}; at 10^5 some comparisons reverse (Appendix C, Figure 7).

assumptions (4)

domain assumption Cloze responses can be modeled as samples from a subject's subjective probability distribution (Smith & Levy, 2011)
Justifies H1's simulation of cloze by sampling from GPT2; Section 3.1.
domain assumption k-means clusters over GPT2 token embeddings capture semantic similarity among words
Underpins H2's cluster-probability manipulation; Section 3.1.
domain assumption wordfreq frequency counts are appropriate for partitioning GPT2's subword vocabulary into frequent/infrequent sets
Used in H3 thresholding; Section 3.1.
ad hoc to paper The chosen smoothing factor and transform (S=200, squared surprisal) are appropriate for all manipulated probability variants despite different sample sizes and distributions
A single transform is applied across all predictors; this choice was tuned to cloze fit on the same RT data, potentially penalizing manipulated variants.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Clozing the Gap: Exploring Why Language Model Surprisal Outperforms Cloze Surprisal." pith.science (2026). https://pith.science/paper/F5A5Z43W

@misc{pith2026260109886,
  author       = {Pith},
  title        = {Pith review of: Clozing the Gap: Exploring Why Language Model Surprisal Outperforms Cloze Surprisal},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/F5A5Z43W}},
  note         = {Machine review of arXiv:2601.09886}
}

read the original abstract

How predictable a word is can be quantified in two ways: using human responses to the cloze task or using probabilities from language models (LMs).When used as predictors of processing effort, LM probabilities outperform probabilities derived from cloze data. However, it is important to establish that LM probabilities do so for the right reasons, since different predictors can lead to different scientific conclusions about the role of prediction in language comprehension. We present evidence for three hypotheses about the advantage of LM probabilities: not suffering from low resolution, distinguishing semantically similar words, and accurately assigning probabilities to low-frequency words. These results call for efforts to improve the resolution of cloze studies, coupled with experiments on whether human-like prediction is also as sensitive to the fine-grained distinctions made by LM probabilities.

Figures

Figures reproduced from arXiv: 2601.09886 by the authors.

**Figure 2.** Increase in per-observation log likelihood over [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗

**Figure 3.** (Top) Increase in per-observation log likelihood over the baseline regression models due to including cloze surprisal, GPT2 surprisal, and manipulated variants of GPT2 surprisal, averaged over all 60 folds (10 folds of six measures) used in cross-validation. Error bars denote one SEM across all 60 folds. Among the three comparisons between GPT2 and its manipulated variants, differences that achieve significance … view at source ↗

Figures from the paper (4 more)

**Figure 4.** Figure 4: Increase in per-observation log likelihood in [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]

**Figure 5.** Figure 5: Increase in per-observation log likelihood, [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]

**Figure 6.** Figure 6: Increase in per-observation log likelihood over the baseline regression models due to including cloze [PITH_FULL_IMAGE:figures/full_fig_p013_6.png]

**Figure 7.** Figure 7: Increase in per-observation log likelihood over the baseline regression models due to including cloze [PITH_FULL_IMAGE:figures/full_fig_p013_7.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

3 extracted references · 2 linked inside Pith

[2]

Cloze procedure

Towards a similarity-adjusted surprisal theory. InProceedings of the 2024 Conference on Empiri- cal Methods in Natural Language Processing, pages 16485–16498. James A. Michaelov, Seana Coulson, and Benjamin K. Bergen. 2023. So cloze yet so far: N400 ampli- tude is better predicted by distributional information than human predictability judgements.IEEE Tra...

arXiv 2024
[2023]

InFindings of the Associ- ation for Computational Linguistics: EMNLP 2023, pages 15711–15721

The linearity of the effect of surprisal on read- ing times across languages. InFindings of the Associ- ation for Computational Linguistics: EMNLP 2023, pages 15711–15721. A Number of Observations for Each RT Measure Table 1 outlines the number of observations that were analyzed for each RT measure. B Effect of Smoothing and Functional Form on the Fit of ...

2023
[2024]

Cassandra L

Large-scale cloze evaluation reveals that token prediction tasks are neither lexically nor semantically aligned.arXiv preprint, arXiv:2410.12057v2. Cassandra L. Jacobs, Ryan J. Hubbard, Loïc Grobol, and Kara D. Federmeier. 2025. Uncovering patterns of se- mantic predictability in sentence processing.Journal of Memory and Language, 144:104653. Tatsuki Kuri...

arXiv 2025

Pith tools

Reviewed August 3, 2026 · model on record in the stance chip above.

[1] [2]

Cloze procedure

Towards a similarity-adjusted surprisal theory. InProceedings of the 2024 Conference on Empiri- cal Methods in Natural Language Processing, pages 16485–16498. James A. Michaelov, Seana Coulson, and Benjamin K. Bergen. 2023. So cloze yet so far: N400 ampli- tude is better predicted by distributional information than human predictability judgements.IEEE Tra...

arXiv 2024

[2] [2023]

InFindings of the Associ- ation for Computational Linguistics: EMNLP 2023, pages 15711–15721

The linearity of the effect of surprisal on read- ing times across languages. InFindings of the Associ- ation for Computational Linguistics: EMNLP 2023, pages 15711–15721. A Number of Observations for Each RT Measure Table 1 outlines the number of observations that were analyzed for each RT measure. B Effect of Smoothing and Functional Form on the Fit of ...

2023

[3] [2024]

Cassandra L

Large-scale cloze evaluation reveals that token prediction tasks are neither lexically nor semantically aligned.arXiv preprint, arXiv:2410.12057v2. Cassandra L. Jacobs, Ryan J. Hubbard, Loïc Grobol, and Kara D. Federmeier. 2025. Uncovering patterns of se- mantic predictability in sentence processing.Journal of Memory and Language, 144:104653. Tatsuki Kuri...

arXiv 2025