Pith. sign in

REVIEW 3 major objections 4 minor 1 cited by

How Well Does First-Token Entropy Approximate Word Entropy as a Psycholinguistic Predictor?

T0 review · 3 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read First-token entropy systematically underpredicts word entropy and yields reading-time predictions that diverge from Monte Carlo word-level estimates.

desk verdict A useful empirical caution about first-token entropy approximations, but the headline divergence rests on shaky aggregate statistics. read the letter →

arxiv 2507.22209 v1 pith:ZHMUJYBK submitted 2025-07-29 cs.CL

Christian Clark , Byung-Doh Oh , William Schuler This is my paper · ORCID
classification cs.CL
keywords contextualentropyfirst-tokenapproximationMonteCarloestimationreadingtimepsycholinguisticssurprisalRényisubwordtokenization
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Contextual entropy, the reader's uncertainty about the next word before seeing it, is usually estimated from a language model's probabilities over a word's first subword token because summing over all possible words is intractable. This paper argues that the first-token approximation is not just a slight underestimate: it systematically distorts which words look predictable, since multi-token words are common for open-class parts of speech. To test this, the authors build Monte Carlo estimates of word-level entropy that sample full words of variable token length, and compare both measures as predictors of reading times in self-paced reading and eye-tracking corpora. The two measures yield divergent regression results, with Monte Carlo word entropy fitting self-paced reading better and improving predictions in most eye-tracking evaluations. The paper concludes that first-token entropy may not be a reliable approximation for psycholinguistic modeling.

What carries the argument

The load-bearing mechanism is a Monte Carlo estimator of next-word entropy built from the language model's subword distribution. The vocabulary is split into whitespace-initial tokens and tokens without a leading whitespace; a word is sampled by drawing an initial token from the whitespace-initial set renormalized to sum to one, then drawing continuation tokens from the non-whitespace set together with an end-of-word symbol whose probability is the summed probability of all whitespace-initial tokens. The sampled word's surprisal is the sum of its token surprisals, and entropy is estimated by averaging over 512 samples; the same sampling supports a Rényi-entropy estimator. This procedure converts an intractable sum over an infinite word vocabulary into a tractable, unbiased expectation, and it is what lets the paper compare word-level entropy against first-token entropy.

What would settle it

Compute exact word-level entropies for a set of contexts by exhaustively summing over all token sequences up to the length cap, or by using a word-level language model, and compare them to the MC estimates; if the MC estimates deviate systematically from the exact values, the sampler is biased and the paper's divergence claim collapses.

Watch

Extended reading notes

Core claim

The paper's central claim is that first-token entropy is a lower bound on true word entropy and that the gap is large enough to change experimental conclusions. Using Monte Carlo sampling that draws whole words from GPT2-small by treating whitespace-initial tokens as word boundaries, the authors obtain unbiased word-level estimates of Shannon entropy and Rényi entropy with alpha = 1/2. In linear mixed-effects regressions over five reading-time corpora, replacing first-token entropy with Monte Carlo entropy changes the size and sometimes the sign of entropy's contribution, and permutation tests show the two predictors differ significantly. The authors read this as evidence that conclusions drawn from first-token entropy should be re-examined, and that word-level Monte Carlo estimates are a safer basis for studying anticipatory processing.

Load-bearing premise

The MC comparison is only as good as the sampler: if sampling words by renormalizing whitespace-initial tokens does not reproduce the language model's true next-word distribution, the Monte Carlo estimates are biased and the divergence from first-token entropy would not reflect word-level truth.

Editorial extensions

If this is right

  • Self-paced reading results in this paper improve when entropy is measured at the word level rather than the first-token level, so word-level estimates should be preferred in studies of anticipatory processing.
  • First-token entropy not only underpredicts entropy but distorts it across parts of speech, so previous entropy effects estimated this way may mix tokenization artifacts with genuine processing difficulty.
  • Monte Carlo Rényi entropy with alpha = 1/2 outperforms first-token Rényi entropy on both self-paced and most eye-tracking evaluations, suggesting the word-level measure more consistently captures anticipatory processing.
  • Permutation tests over squared errors find the two predictors differ significantly, so the divergence is not just noise in one corpus.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper, because distortion scales with multi-token word probability, larger subword vocabularies should shrink the first-token-versus-Monte-Carlo gap; this can be tested by rerunning the comparison with GPT2-large or different tokenizers.
  • Beyond the paper, the word-level Rényi advantage suggests the anticipatory strategy that best fits readers lies between Shannon averaging and max-probability tracking, which could be probed by fitting alpha as a free parameter.
  • Beyond the paper, the sampling estimator is not specific to entropy: the same whitespace-partition sampler could compute other word-level information measures, such as mutual information between a word and its context, without changing the data pipeline.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper compares first-token (subword-level) approximations of contextual Shannon and Rényi entropy with Monte Carlo estimates that allow a word to span multiple subword tokens, using GPT2-small as the language model. The two entropy estimators are evaluated as predictors of reading times in two self-paced reading corpora and three eye-tracking corpora, with surprisal and other baseline predictors controlled in linear mixed-effects regressions. The paper reports that first-token entropy is a lower bound on word entropy, demonstrates in Figure 1 that MC entropy is on average higher and differs across parts of speech, and reports in Table 2 that the two estimators yield different log-likelihood improvements. It concludes that the concrete differences between first-token and MC entropy warrant caution against using first-token entropy in psycholinguistic modeling.

Significance. If the reported divergence is real, this is a useful methodological caution for a research community that frequently uses first-token entropy as a proxy for word-level contextual entropy. The paper has several concrete strengths: the code is released, the connection to the whitespace-initial-token confound is explicitly acknowledged with references to Oh and Schuler (2024) and Pimentel and Meister (2024), the MC sampling procedure is described in enough detail to be reproduced, and Appendix A.2 provides a bootstrap-based variance analysis of sample count. The theoretical lower-bound point is not new, so the paper's contribution is primarily empirical. The main weakness is that the central empirical claim is supported by point estimates and an aggregated permutation test rather than by per-corpus uncertainty quantification, which is the load-bearing gap for the Discussion's conclusion.

major comments (3)
  1. [§4.3 / Table 2] The claim that first-token and MC entropy yield a 'concrete difference' rests on ΔLL point estimates in Table 2 that have no standard errors, confidence intervals, or per-corpus significance tests. Several entries are close to zero or negative (e.g., Provo FP Shannon first-token −0.2; GECO FP MC Shannon −0.4), so the sign of an individual comparison is not informative without uncertainty. The permutation test described in §4.3 is run over squared errors aggregated across all reading-time corpora, which mixes self-paced reading times, first-pass durations, and go-past durations with different units and noise levels; the test statistic, null hypothesis, and clustering are not specified. One or two large effects, such as Natural Stories SPR Shannon 29 vs. 72, could drive the reported p < 0.01 / p < 0.001 even if most per-corpus comparisons were null or reversed. The Discussion in §5 requires per-corpus or otherwise appropriately conditioned uncertainty quantification to support the caution it draws.
  2. [§3.3 / Eq. (5)] The Introduction states that the MC method produces unbiased estimates, but this is not correct for the Rényi estimator in Eq. (5). That estimator applies a logarithm to a sample average of P(s)^{β−1}, so Jensen's inequality introduces finite-sample bias; for α = 1/2 the term P(s)^{−1/2} is particularly sensitive to rare sampled words and can produce high variance. The bootstrap coefficient-of-variation analysis in Appendix A.2 measures dispersion across resamples but does not measure bias or the influence of rare-word tail events. The Rényi results are an important part of Table 2, so the finite-sample properties of Eq. (5) should be acknowledged and, if possible, quantified.
  3. [Appendix A.1] The validity of the MC estimator as an approximation to true word entropy depends on the whitespace-partition sampling procedure producing samples from the LM's word-level conditional distribution. The paper follows Oh and Schuler (2024) by renormalizing over whitespace-initial tokens and defining the end-of-word probability as the sum over those tokens, but it does not validate on GPT2-small that this procedure yields the correct marginal word probabilities on the actual corpus tokenization. A small-scale exact enumeration or a comparison against the method of Pimentel and Meister (2024) would strengthen the claim that the MC estimates approximate true word entropy rather than another token-boundary quantity. The 20-token cap is acknowledged to underestimate true surprisal, but the magnitude of its effect on the entropy estimates is not quantified.
minor comments (4)
  1. [§3.3] The sentence 'These approximation are a lower bound' should read 'These approximations are a lower bound'.
  2. [Appendix A.2] The variance analysis uses only the first story of the Natural Stories corpus; the authors should note that this corpus may not be representative of the lexical distributions in the eye-tracking corpora.
  3. [§4.2 / Table 1] The regression setup would be easier to evaluate if the exact model formula and the method for obtaining held-out log-likelihoods (e.g., whether random effects were refit) were stated explicitly.
  4. [Figure 2] The y-axis ranges of the two panels differ substantially; using a common scale or noting the difference in the caption would help readers compare the Shannon and Rényi coefficients of variation.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the entropy predictors are computed from GPT-2 probabilities independently of reading-time outcomes, and the regression comparison is an external empirical test.

full rationale

The paper's chain of reasoning derives first-token and Monte Carlo entropy estimates from the same conditional token probabilities (Eqs. 3-5) and then evaluates them as predictors of held-out reading times in LME regressions (Sec. 4). No parameter of either entropy estimate is fitted to the behavioral data; the entropy values are fixed functions of the LM and the corpora. The MC estimator's unbiasedness and the first-token lower-bound property are cited from external work (Giulianelli et al., 2024; Pimentel et al., 2023), and the fact that MC estimates exceed first-token estimates on average is mathematically expected rather than presented as an empirical prediction. The only self-citation (Oh and Schuler, 2024) supplies a whitespace-boundary sampling technique, and the same methodological point is independently cited to Pimentel and Meister (2024); this does not by construction determine which predictor fits reading times better. The observed divergence in Table 2 is not forced by the definitions; indeed, several eye-tracking comparisons favor first-token entropy. Concerns about missing per-corpus uncertainty intervals are statistical-evidence concerns, not circularity, and are outside the scope of this pass.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The paper introduces no new theoretical entities. Its free parameters are computational choices (sample count, token cap) and a hyperparameter (Rényi alpha) adopted from prior work, none of which are fitted to the behavioral data. The central statistical framework rests on standard definitions and on the assumption that the LM and the sampling procedure faithfully represent word-level prediction.

free parameters (3)
  • MC sample count |S| = 512
    Number of Monte Carlo samples chosen due to compute budget; variance analysis in Appendix A.2 suggests reasonable stability, but it is a manually chosen constant affecting estimate precision.
  • Maximum subword tokens per word (cap) = 20
    Cap on sampled token length for tractability; paper notes it causes slight underestimation of surprisal but affects only about 1/50,000 samples.
  • Rényi entropy order alpha = 0.5
    Adopted from Pimentel et al. (2023) as a strong predictor; not fitted in this paper, but a hyperparameter selection that affects the measure.
assumptions (4)
  • standard math Standard definitions of Shannon and Rényi entropy, and the unbiasedness of the sample-mean estimator for Shannon entropy.
    The paper relies on these definitions and properties to justify the MC estimator in Eq. 4.
  • domain assumption GPT2-small's conditional probability distribution provides a valid proxy for readers' predictive distributions.
    This assumption underlies the use of LM probabilities to compute psycholinguistic predictors; Section 4.2 uses GPT2-small for both surprisal and entropy.
  • domain assumption The sampling procedure of Oh and Schuler (2024) correctly samples from the word-level conditional distribution of GPT2.
    Appendix A.1 relies entirely on this procedure for the MC estimates; if it is flawed, the MC estimates are biased and the comparison to first-token entropy is compromised.
  • domain assumption Reading times (SPR, FP, GP) reflect processing difficulty and are appropriately modeled by linear mixed-effects regression with the specified baseline predictors.
    This assumption is inherent to the regression experiments in Section 4, though it is standard in the field.

how reviews work

0 comments
Cite this review

Pith. "Pith review of How Well Does First-Token Entropy Approximate Word Entropy as a Psycholinguistic Predictor?." pith.science (2026). https://pith.science/paper/ZHMUJYBK

@misc{pith2026250722209,
  author       = {Pith},
  title        = {Pith review of: How Well Does First-Token Entropy Approximate Word Entropy as a Psycholinguistic Predictor?},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/ZHMUJYBK}},
  note         = {Machine review of arXiv:2507.22209}
}
read the original abstract

Contextual entropy is a psycholinguistic measure capturing the anticipated difficulty of processing a word just before it is encountered. Recent studies have tested for entropy-related effects as a potential complement to well-known effects from surprisal. For convenience, entropy is typically estimated based on a language model's probability distribution over a word's first subword token. However, this approximation results in underestimation and potential distortion of true word entropy. To address this, we generate Monte Carlo (MC) estimates of word entropy that allow words to span a variable number of tokens. Regression experiments on reading times show divergent results between first-token and MC word entropy, suggesting a need for caution in using first-token approximations of contextual entropy.

Figures

Figures reproduced from arXiv: 2507.22209 by the authors.

Figure 1
Figure 1. Average Shannon entropy of the 10 most fre [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Coefficient of variation by sample count for [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. LODESTAR: Trustworthy Entropy Is Navigated, Not Merely Measured -- Reinforced Polarizer Keeps a Frozen LLM from Being Confidently Misled by the Wrong Evidence

    cs.CL 2026-08 conditional novelty 6.0 of 10

    A fixed learned sentence inserted into a frozen LLM's prompt steers its uncertainty so that lowest-entropy answer selection avoids misleading passages, raising mean F1 from 0.5148 to 0.5339 across five QA benchmarks.

Reference graph

Works this paper leans on

31 extracted references · 22 canonical work pages · cited by 1 Pith paper

  1. [1]

    Douglas Bates, Martin M \" a chler, Ben Bolker, and Steve Walker. 2015. https://doi.org/10.18637/jss.v067.i01 Fitting linear mixed-effects models using lme4 . Journal of Statistical Software, 67(1):1--48

    doi:10.18637/jss.v067.i01 2015

  2. [2]

    Benedetta Cevoli, Chris Watkins, and Kathleen Rastle. 2022. Prediction as a basis for skilled reading: Insights from modern language models. Royal Society open science, 9(6):211837

    work page 2022

  3. [3]

    Agatha Christie. 1920. https://www.gutenberg.org The mysterious affair at Styles . John Lane. Retrieved from Project Gutenberg

    work page 1920

  4. [4]

    Uschi Cop, Nicolas Dirix, Denis Drieghe, and Wouter Duyck. 2017. https://doi.org/10.3758/s13428-016-0734-0 Presenting GECO: An eyetracking corpus of monolingual and bilingual sentence reading . Behavior Research Methods, 49(2):602--615

    work page doi:10.3758/s13428-016-0734-0 2017

  5. [5]

    Bradley Efron. 1992. Bootstrap methods: another look at the jackknife. In Breakthroughs in statistics: Methodology and distribution, pages 569--593. Springer

    work page 1992

  6. [6]

    Stefan L Frank. 2013. Uncertainty reduction as a measure of cognitive load in sentence comprehension. Topics in cognitive science, 5(3):475--494

    work page 2013

  7. [7]

    Tily, Idan Blank, Anastasia Vishnevetsky, Steven Piantadosi, and Evelina Fedorenko

    Richard Futrell, Edward Gibson, Harry J. Tily, Idan Blank, Anastasia Vishnevetsky, Steven Piantadosi, and Evelina Fedorenko. 2021. The Natural Stories corpus: A reading-time corpus of English texts containing rare syntactic constructions . Language Resources and Evaluation, 55:63--77

    work page 2021

  8. [8]

    Mario Giulianelli, Andreas Opedal, and Ryan Cotterell. 2024. https://doi.org/10.18653/v1/2024.findings-emnlp.682 Generalized measures of anticipation and responsivity in online language processing . In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 11648--11669, Miami, Florida, USA. Association for Computational Linguistics

    doi:10.18653/v1/2024.findings-emnlp.682 2024

Show all 31 references
  1. [9]

    Aaron Gokaslan and Vanya Cohen. 2019. OpenWebText Corpus . http://Skylion007.github.io/OpenWebTextCorpus

    2019

  2. [10]

    John Hale . 2001. A probabilistic earley parser as a psycholinguistic model. In Proceedings of the second meeting of the North American chapter of the Association for Computational Linguistics, pages 159--166, Pittsburgh, PA

    2001

  3. [11]

    John Hale. 2003. The information conveyed by words in sentences. Journal of psycholinguistic research, 32(2):101--123

    2003

  4. [12]

    John Hale. 2006. Uncertainty about the rest of the sentence. Cognitive science, 30(4):643--672

    2006

  5. [13]

    Clark, and Philipp Koehn

    Kenneth Heafield, Ivan Pouzyrevsky, Jonathan H. Clark, and Philipp Koehn. 2013. https://www.aclweb.org/anthology/P13-2121/ Scalable modified Kneser-Ney language model estimation . In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics, pages...

    2013

  6. [14]

    Alan Kennedy , James Pynte , and Robin Hill . 2003. The D undee corpus. In Proceedings of the 12th E uropean conference on eye movement

    2003

  7. [15]

    Roger Levy . 2008. Expectation-based syntactic comprehension. Cognition, 106(3):1126--1177

    2008

  8. [16]

    Tal Linzen and T Florian Jaeger. 2016. Uncertainty and expectation in sentence processing: Evidence from subcategorization distributions. Cognitive science, 40(6):1382--1411

    2016

  9. [17]

    Luke and Kiel Christianson

    Steven G. Luke and Kiel Christianson. 2018. https://doi.org/10.3758/s13428-017-0908-4 The Provo Corpus: A large eye-tracking corpus with predictability norms . Behavior Research Methods, 50(2):826--833

    2018 doi

  10. [18]

    Mitch Marcus, Beatrice Santorini, and Mary Ann Marcinkiewicz. 1993. Building a large annotated corpus of English: The Penn Treebank . Computational linguistics, 19(2):313--330

    1993

  11. [19]

    Nicholas Metropolis and Stanislaw Ulam. 1949. The monte carlo method. Journal of the American statistical association, 44(247):335--341

    1949

  12. [20]

    Byung-Doh Oh and William Schuler. 2024. https://doi.org/10.18653/v1/2024.emnlp-main.202 Leading whitespaces of language models' subword vocabulary pose a confound for calculating word probabilities . In Proceedings of the 2024 Conference on Empirical Methods in Natural Languag...

    2024 doi

  13. [21]

    Tiago Pimentel and Clara Meister. 2024. https://doi.org/10.18653/v1/2024.emnlp-main.1020 How to compute the probability of a word . In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 18358--18375, Miami, Florida, USA. Association f...

    2024 doi

  14. [22]

    Tiago Pimentel, Clara Meister, Ethan G Wilcox, Roger P Levy, and Ryan Cotterell. 2023. On the effect of anticipation on reading times. Transactions of the Association for Computational Linguistics, 11:1624--1642

    2023

  15. [23]

    Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. ArXiv

    2019

  16. [24]

    Alfr \'e d R \'e nyi. 1961. On measures of entropy and information. In Proceedings of the fourth Berkeley symposium on mathematical statistics and probability, volume 1: contributions to the theory of statistics, volume 4, pages 547--562. University of California Press

    1961

  17. [25]

    Brian Roark, Asaf Bachrach, Carlos Cardenas, and Christophe Pallier. 2009. Deriving lexical and syntactic expectation-based measures for psycholinguistic modeling via incremental top-down parsing. In Proceedings of the 2009 conference on empirical methods in natural language p...

    2009

  18. [26]

    Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. https://doi.org/10.18653/v1/P16-1162 Neural machine translation of rare words with subword units . In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages ...

    2016 doi

  19. [27]

    Cory Shain, Clara Meister, Tiago Pimentel, Ryan Cotterell, and Roger Levy. 2024. https://doi.org/10.1073/pnas.2307876121 Large-scale evidence for logarithmic effects of word predictability on reading time . Proceedings of the National Academy of Sciences, 121(10):e2307876121

    2024 doi

  20. [28]

    Claude E Shannon. 1948. A mathematical theory of communication. The Bell system technical journal, 27(3):379--423

    1948

  21. [29]

    Smith and Roger Levy

    Nathaniel J. Smith and Roger Levy. 2013. https://doi.org/10.1016/j.cognition.2013.02.013 The effect of word predictability on reading time is logarithmic . Cognition, 128:302--319

    2013 doi

  22. [30]

    Marten van Schijndel and Tal Linzen. 2019. https://doi.org/10.7275/qtbb-9d05 Can entropy explain successor surprisal effects in reading? In Proceedings of the Society for Computation in Linguistics ( SC i L ) 2019 , pages 1--7

    2019 doi

  23. [31]

    Ethan Gotlieb Wilcox, Tiago Pimentel, Clara Meister, Ryan Cotterell, and Roger P. Levy. 2023. https://doi.org/10.1162/tacl_a_00612 Testing the predictions of surprisal theory in 11 languages . Transactions of the Association for Computational Linguistics, 11:1451--1470

    2023 doi

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.