REVIEW 3 major objections 4 minor 49 references
Estimating Musical Surprisal in Audio
T0 review · 3 major / 4 minor · reviewed 2026-08-10 · deepseek-v4-flash
Pith's one-line read The information content of a Transformer's next-step predictions on compressed audio latents predicts EEG brain responses to songs, supporting an audio-domain proxy for musical surprisal.
desk verdict A genuinely new audio-level surprisal estimator with a real EEG validation, but the perceptual claim currently rests on a single acoustic nuisance regressor, so the result is promising rather than conclusive. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is the information content of one-step predictions from a GIVT, a causal Transformer whose output head parameterizes a 32-component Gaussian mixture model over the next 64-dimensional frame of a Music2Latent audio latent sequence at roughly 11 frames per second. The information content is the negative log-density of the observed next frame under the predicted mixture, replacing the softmax over a finite token vocabulary used in symbolic music models. This lets the surprisal estimate flow from the raw audio content itself, including timbre and dynamics, and the same quantity is then compared with repetition structure, section type, musical features, and EEG responses.
What would settle it
Run the EEG prediction with spectral flux, loudness, onset density, and timbre descriptors added as competing regressors before the information content; if the information content no longer improves prediction, its apparent perceptual role is an acoustic confound rather than surprisal.
Extended reading notes
Core claim
The paper claims that musical surprisal in audio can be modeled without preselected audio features by computing the information content of next-step predictions from a GIVT, a causal Transformer whose output head parameterizes a 32-component Gaussian mixture model over the next frame of a Music2Latent latent sequence. The resulting audio-domain information content behaves like symbolic-domain surprisal: it decreases with repetition on the model's training distribution, it is higher for segment types that appear later in a piece while outros show lower values, and it correlates more strongly with loudness and spectral flux than with dissonance, rhythmic complexity, or onset density. The decisive validation is that this information content, competing with the energy envelope as a predictor in a linear encoding model, significantly improves prediction of EEG responses to songs, supporting the methodology's potential as a perceptual model of human musical surprisal.
Load-bearing premise
The result assumes that the EEG variance explained by the information content after removing only the energy envelope is due to musical surprisal, not to other acoustic properties, like timbre or loudness changes, that happen to move together with the information content.
Editorial extensions
If this is right
- Repetition lowers the mean information content on a large popular-music training set, showing that the model learns musical structure from audio alone.
- Segment types appearing later in a piece have higher information content than earlier ones, while outros have lower information content, supporting the idea that later sections are composed to contrast earlier material.
- Information content correlates most strongly with loudness and spectral flux, and less with dissonance, rhythmic complexity, and onset density; these correlations are highest at the start of a piece and decay over time.
- Information content explains a small but significant portion of EEG variance beyond the energy envelope, so the audio-based surprisal signal tracks human neural responses.
- Because the method needs no preselected features, it can be applied to arbitrary full-length audio recordings, including timbral and dynamic content unavailable in symbolic scores.
Reading between the lines
- A natural next test is whether the information content's EEG contribution survives when spectral flux and loudness are added as competing regressors; if it does not, the perceptual claim would need to be attributed to acoustic surprise rather than musical expectation.
- The same GIVT-plus-Music2Latent recipe could be used to condition generative music models on surprisal curves, turning the estimator into a controllable knob for compositional contrast.
- Applying the method to genres outside popular music, or to recordings with strong production effects, would clarify whether the repetition and segment-contrast effects generalize beyond the training distribution.
- Because the model is trained on audio itself rather than on score-like events, its information content likely mixes acoustic novelty and music-theoretic expectation; separating these would require a model whose predictions are evaluated over semantically distinct event types.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes a method for estimating musical surprisal directly from audio by training an autoregressive GIVT model to predict Music2Latent latent representations and defining information content (IC) as the negative log density of a Gaussian mixture model's next-frame prediction. The authors evaluate the resulting IC in four ways: it decreases with repetition on the largest dataset, later-occurring segment types have higher IC than earlier ones (while outros have lower IC), it correlates with several audio and musical features (dissonance, IOI entropy, onset density, loudness, spectral flux), and it significantly improves prediction of EEG responses to sung music beyond an energy-envelope baseline. Code is released, and evaluation includes public datasets.
Significance. If the results hold, the paper extends information-theoretic surprisal modeling from the symbolic domain to full-length audio without relying on hand-selected audio features, which is a useful step for music cognition and MIR. The experimental design has notable strengths: the model is evaluated on multiple datasets, the EEG test uses an external benchmark with a shuffled-IC null model, the repetition and segment-contrast analyses are simple and falsifiable, and code is provided. However, the central perceptual claim rests on the EEG analysis, and that analysis currently leaves open a plausible acoustic-confound explanation. The free parameters of the pipeline (latent representation, frame rate, context length, GMM components, architecture) are not ablated, so the contribution is best read as a proof-of-concept rather than a systematic study.
major comments (3)
- [Section V-D] The EEG validation controls only for the energy envelope, but the paper's own Section V-C and Table I show that IC is strongly correlated with loudness (rl up to 0.70) and spectral flux (rf up to 0.57). The Hilbert envelope captures amplitude modulations but not spectral or timbral changes. The shuffled-IC baseline only breaks the temporal alignment between IC and the EEG signal; it does not control for acoustic features that are correlated with IC. As a result, the incremental EEG variance attributed to IC could be carried by uncontrolled loudness- or timbre-related acoustics, which would undermine the central conclusion that IC reflects musical surprisal. Please include spectral flux and loudness (and ideally other acoustic descriptors) as additional nuisance regressors, or otherwise show that the IC effect survives when these features compete for the same variance.
- [Table I and Section V-C] All complexity correlations are reported only as point estimates, with no confidence intervals, standard errors, or cross-validation, and no correction for the multiple comparisons across five metrics and three datasets. The statement that 'all correlations are significant' is therefore under-supported, especially for the smaller public datasets. Since these correlations are used both to support the complexity associations and to quantify the potential acoustic confounds in the EEG analysis, please provide interval estimates and describe the exact significance-testing procedure.
- [Section V-A] The repetition analysis is described as a 'one-tailed t-test for positivity,' but the reported mean difference for PR is negative (-1.30) and is described as a significant decrease. This is confusing: the sign convention for the computed difference (first segment minus second, or vice versa) and the alternative hypothesis of the t-test need to be stated explicitly. As written, a reader cannot tell whether the test is for a positive mean difference or for a negative mean difference.
minor comments (4)
- [Equation (2)] The text says the IC 'is unbounded,' but it may be worth clarifying that the negative log density of a continuous GMM can also take negative values when the density exceeds 1. This is a minor notational point but affects how readers interpret IC values.
- [Figure 1] The temporal correlation curves are plotted without any confidence bands or significance shading, so it is not clear whether the visible decrease over time is statistically reliable. Please add error bars or a significance annotation.
- [Section V-D] The EEG stimuli are monophonic sung music, while the model is trained on multi-stem popular music. The fine-tuning on vocal stems is mentioned, but the paper does not report how IC is computed and aligned for the EEG stimuli (e.g., whether the same 11 Hz frame rate and context length are used, and how the IC time series is resampled to the EEG sampling rate). A brief description of the alignment procedure would improve reproducibility.
- [Throughout] There are several typographical errors ('rhymic complexity' in Section II, 'suprisal' in the introduction) and the phrase 't-test for positivity' in Section V-A. A careful proofreading pass is needed.
Circularity Check
No circularity: the IC is defined from an independently trained next-step density model and validated against external EEG data with a shuffled-IC baseline.
full rationale
The paper's derivation chain is self-contained with respect to its central claim. The IC is defined in Eq. (2) as the negative log density of a Gaussian mixture model parameterized by a causally trained Transformer, with no EEG or perceptual labels in the training objective. The EEG validation in Section V-D uses an external dataset (Cantisani et al., [47]), a linear temporal response model, and a shuffled-IC baseline to assess incremental predictive power; the model parameters are not fitted to neural responses. The repetition, segment-contrast, and feature-correlation analyses are empirical checks, not built into the definition of IC. The paper does cite prior work by its own authors (e.g., Music2Latent [19] and the EEG dataset [47]), but these are external artifacts and not load-bearing in a circular way: the method would stand or fall on the independent reproducibility of those components. The main scientific risk, that IC's EEG predictive power may be carried by acoustic features such as spectral flux or loudness beyond the Hilbert envelope, is a validity concern about confounds, not a circularity in which the prediction is equivalent to a fitted input by construction.
Assumptions & free parameters
free parameters (5)
- GMM components =
32
- Latent dimension =
64
- Frame rate =
~11 Hz
- Context length =
4600 frames
- Model architecture =
12-layer Transformer
assumptions (5)
- domain assumption Music2Latent's latent representation preserves enough audio information to capture surprisal-relevant acoustic properties.
- domain assumption Deep12's automatic segment labels and boundaries are accurate enough for the repetition and contrast analyses.
- domain assumption The mTRF linear encoding model adequately captures the relationship between IC and EEG responses.
- domain assumption The EEG dataset and preprocessing (including interpolation of unvoiced segments) are appropriate and unbiased.
- domain assumption TIV dissonance, IOI entropy, onset density, and spectral flux are valid proxies for musical complexity.
Cite this review
Pith. "Pith review of Estimating Musical Surprisal in Audio." pith.science (2026). https://pith.science/paper/WUDE36R2
@misc{pith2026250107474,
author = {Pith},
title = {Pith review of: Estimating Musical Surprisal in Audio},
year = {2026},
howpublished = {\url{https://pith.science/paper/WUDE36R2}},
note = {Machine review of arXiv:2501.07474}
}
read the original abstract
In modeling musical surprisal expectancy with computational methods, it has been proposed to use the information content (IC) of one-step predictions from an autoregressive model as a proxy for surprisal in symbolic music. With an appropriately chosen model, the IC of musical events has been shown to correlate with human perception of surprise and complexity aspects, including tonal and rhythmic complexity. This work investigates whether an analogous methodology can be applied to music audio. We train an autoregressive Transformer model to predict compressed latent audio representations of a pretrained autoencoder network. We verify learning effects by estimating the decrease in IC with repetitions. We investigate the mean IC of musical segment types (e.g., A or B) and find that segment types appearing later in a piece have a higher IC than earlier ones on average. We investigate the IC's relation to audio and musical features and find it correlated with timbral variations and loudness and, to a lesser extent, dissonance, rhythmic complexity, and onset density related to audio and musical features. Finally, we investigate if the IC can predict EEG responses to songs and thus model humans' surprisal in music. We provide code for our method on github.com/sonycslparis/audioic.
Figures
Reference graph
Works this paper leans on
-
[1]
Meaning in music and information theory,
L. B. Meyer, “Meaning in music and information theory,” The Journal of Aesthetics and Art Criticism , vol. 15, no. 4, pp. 412–424, 1957
work page 1957
-
[2]
Multiple viewpoint systems for music prediction,
D. Conklin and I. H. Witten, “Multiple viewpoint systems for music prediction,” Journal of New Music Research , vol. 24, no. 1, pp. 51–73, 1995
work page 1995
-
[3]
M. Pearce, “The construction and evaluation of statistical models of melodic structure in music perception and composition,” Ph.D. disser- tation, Department of Computing, City University, London, UK, 2005
work page 2005
-
[4]
Auditory expectation: The information dynamics of music perception and cognition,
M. T. Pearce and G. A. Wiggins, “Auditory expectation: The information dynamics of music perception and cognition,” Top. Cogn. Sci. , vol. 4, no. 4, pp. 625–652, 2012
work page 2012
-
[5]
Information-theoretic modeling of perceived musical complexity,
S. A. Sauv ´e and M. T. Pearce, “Information-theoretic modeling of perceived musical complexity,” Music Perception: An Interdisciplinary Journal, vol. 37, no. 2, pp. 165–178, 2019
work page 2019
-
[6]
Exploring sampling techniques for generating melodies with a transformer language model,
M. R. Bjare, S. Lattner, and G. Widmer, “Exploring sampling techniques for generating melodies with a transformer language model,” in ISMIR, 2023, pp. 810–816
work page 2023
-
[7]
Predictability and uncertainty in the pleasure of music: a reward for learning?
B. P. Gold, M. T. Pearce, E. Mas-Herrero, A. Dagher, and R. J. Zatorre, “Predictability and uncertainty in the pleasure of music: a reward for learning?” Journal of Neuroscience , vol. 39, no. 47, pp. 9397–9409, 2019
work page 2019
-
[8]
Guided music synthesis with variable markov oracle,
C.-i. Wang and S. Dubnov, “Guided music synthesis with variable markov oracle,” in AAAI, vol. 10, no. 5, 2014, pp. 55–62
work page 2014
Show all 49 references
-
[9]
Developing and evaluating computational models of musical style,
T. Collins, R. Laney, A. Willis, and P. H. Garthwaite, “Developing and evaluating computational models of musical style,” AI EDAM , vol. 30, no. 1, pp. 16–43, 2016
2016
-
[10]
Controlling surprisal in music generation via information content curve matching,
M. R. Bjare, S. Lattner, and G. Widmer, “Controlling surprisal in music generation via information content curve matching,” in ISMIR, 2024
2024
-
[11]
Audio oracle: A new algorithm for fast learning of audio structures,
S. Dubnov, G. Assayag, and A. Cont, “Audio oracle: A new algorithm for fast learning of audio structures,” in ICMC, 2007, pp. 224–227
2007
-
[12]
Deep generative models of music expectation,
N. L. Masclef and T. A. Keller, “Deep generative models of music expectation,” NeurIPS ML for Audio Workshop 2023 , 2023
2023
-
[13]
Retrieving musical information from neural data: how cognitive features enrich acoustic ones
E. Abrams, E. M. Vidal, C. Pelofi, and P. Ripoll ´es, “Retrieving musical information from neural data: how cognitive features enrich acoustic ones.” in ISMIR, 2022, pp. 160–168
2022
-
[14]
Differentiable short-term models for efficient online learning and prediction in monophonic music,
M. R. Bjare, S. Lattner, and G. Widmer, “Differentiable short-term models for efficient online learning and prediction in monophonic music,” Trans. Int. Soc. Music. Inf. Retr . , vol. 5, no. 1, p. 190, 2022
2022
-
[15]
On the typicality of musical sequences,
M. R. Bjare and S. Lattner, “On the typicality of musical sequences,” in ISMIR Late Breaking and Demo Papers , 2022
2022
-
[16]
Attention is all you need,
A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin, “Attention is all you need,” in NeurIPS, 2017, pp. 5998–6008
2017
-
[17]
Long-form music generation with latent diffusion,
Z. Evans, J. D. Parker, C. Carr, Z. Zukowski, J. Taylor, and J. Pons, “Long-form music generation with latent diffusion,” in ISMIR, 2024
2024
-
[18]
GIVT: generative infinite- vocabulary transformers,
M. Tschannen, C. Eastwood, and F. Mentzer, “GIVT: generative infinite- vocabulary transformers,” CoRR, vol. abs/2312.02116, 2023
2023 arXiv
-
[19]
Music2latent: Consistency autoencoders for latent audio compression,
M. Pasini, S. Lattner, and G. Fazekas, “Music2latent: Consistency autoencoders for latent audio compression,” in ISMIR, 2024
2024
-
[20]
Unsupervised statistical learning underpins computational, behavioural, and neural manifestations of musical expectation,
M. T. Pearce, M. H. Ruiz, S. Kapasi, G. A. Wiggins, and J. Bhattacharya, “Unsupervised statistical learning underpins computational, behavioural, and neural manifestations of musical expectation,” NeuroImage, vol. 50, no. 1, pp. 302–313, 2010
2010
-
[21]
Cortical encoding of melodic expectations in human temporal cortex,
G. M. Di Liberto, C. Pelofi, R. Bianco, P. Patel, A. D. Mehta, J. L. Herrero, A. De Cheveign ´e, S. Shamma, and N. Mesgarani, “Cortical encoding of melodic expectations in human temporal cortex,” Elife, vol. 9, p. e51784, 2020
2020
-
[22]
Predictive uncertainty in auditory sequence processing,
N. C. Hansen and M. T. Pearce, “Predictive uncertainty in auditory sequence processing,” Frontiers in psychology , vol. 5, p. 1052, 2014
2014
-
[23]
Pupil responses to pitch deviants reflect predictability of melodic sequences,
R. Bianco, L. E. Ptasczynski, and D. Omigie, “Pupil responses to pitch deviants reflect predictability of melodic sequences,” Brain and Cognition, vol. 138, p. 103621, 2020
2020
-
[24]
Statistical learning of melodic patterns influences the brain’s response to wrong notes,
T. Moldwin, O. Schwartz, and E. S. Sussman, “Statistical learning of melodic patterns influences the brain’s response to wrong notes,”Journal of cognitive neuroscience , vol. 29, no. 12, pp. 2114–2122, 2017
2017
-
[25]
Detecting change in stochastic sound sequences,
B. Skerritt-Davis and M. Elhilali, “Detecting change in stochastic sound sequences,” PLoS Comput. Biol. , vol. 14, no. 5, 2018
2018
-
[26]
A model for statistical regularity extraction from dynamic sounds,
B. Skerritt-Davis and M. Elhilali, “A model for statistical regularity extraction from dynamic sounds,” Acta Acustica united with Acustica , vol. 105, no. 1, pp. 1–4, 2019
2019
-
[27]
Consistency models,
Y . Song, P. Dhariwal, M. Chen, and I. Sutskever, “Consistency models,” in ICML, vol. 202. PMLR, 2023, pp. 32 211–32 252
2023
-
[28]
wav2vec 2.0: A framework for self-supervised learning of speech representations,
A. Baevski, Y . Zhou, A. Mohamed, and M. Auli, “wav2vec 2.0: A framework for self-supervised learning of speech representations,” in NeurIPS, 2020, pp. 12 449–12 460
2020
-
[29]
Contrastive learning of musical representations,
J. Spijkervet and J. A. Burgoyne, “Contrastive learning of musical representations,” in ISMIR, 2021, pp. 673–681
2021
-
[30]
Jukebox: A generative model for music,
P. Dhariwal, H. Jun, C. Payne, J. W. Kim, A. Radford, and I. Sutskever, “Jukebox: A generative model for music,” CoRR, vol. abs/2005.00341, 2020
2005 arXiv
-
[31]
SoundStream: An end-to-end neural audio codec,
N. Zeghidour, A. Luebs, A. Omran, J. Skoglund, and M. Tagliasacchi, “SoundStream: An end-to-end neural audio codec,” IEEE ACM Trans. Audio Speech Lang. Process. , vol. 30, pp. 495–507, 2022
2022
-
[32]
High fidelity neural audio compression,
A. D ´efossez, J. Copet, G. Synnaeve, and Y . Adi, “High fidelity neural audio compression,” Trans. Mach. Learn. Res. , vol. 2023, 2023
2023
-
[33]
High- fidelity audio compression with improved RVQGAN,
R. Kumar, P. Seetharaman, A. Luebs, I. Kumar, and K. Kumar, “High- fidelity audio compression with improved RVQGAN,” inNeurIPS, 2023
2023
-
[34]
Language models are unsupervised multitask learners,
A. Radford, J. Wu, R. Child, D. Luan, D. Amodei, and I. Sutskever, “Language models are unsupervised multitask learners,” 2019
2019
-
[35]
Medleydb 2.0: New data and a system for sustainable data collection,
R. M. Bittner, J. Wilkins, H. Yip, and J. P. Bello, “Medleydb 2.0: New data and a system for sustainable data collection,” ISMIR Late Breaking and Demo Papers , vol. 36, 2016
2016
-
[36]
The MUSDB18 corpus for music separation,
Z. Rafii, A. Liutkus, F.-R. St ¨oter, S. I. Mimilakis, and R. Bittner, “The MUSDB18 corpus for music separation,” Dec. 2017. [Online]. Available: https://doi.org/10.5281/zenodo.1117372
2017 doi
-
[37]
Kishi, N
H. Kishi, N. Polouliakh, and T. Akama, “Deep12,” https://www.sonycsl.co.jp/tokyo/14621
-
[38]
Roformer: Enhanced transformer with rotary position embedding,
J. Su, M. H. M. Ahmed, Y . Lu, S. Pan, W. Bo, and Y . Liu, “Roformer: Enhanced transformer with rotary position embedding,” Neurocomput- ing, vol. 568, p. 127063, 2024
2024
-
[39]
FlashAttention: Fast and memory-efficient exact attention with io-awareness,
T. Dao, D. Y . Fu, S. Ermon, A. Rudra, and C. R ´e, “FlashAttention: Fast and memory-efficient exact attention with io-awareness,” CoRR, vol. abs/2205.14135, 2022
2022 arXiv
-
[40]
Adam: A method for stochastic optimization,
D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” in ICLR (Poster), 2015
2015
-
[41]
Conchord: an application for generating musical harmony by navigating in a perceptually motivated tonal interval space,
G. Bernardes, D. Cocharro, C. Guedes, and M. Davies, “Conchord: an application for generating musical harmony by navigating in a perceptually motivated tonal interval space,” inCMMR, 2015, pp. 71–86
2015
-
[42]
A multi-level tonal interval space for modelling pitch relatedness and musical consonance,
G. Bernardes, D. Cocharro, M. Caetano, C. Guedes, and M. E. Davies, “A multi-level tonal interval space for modelling pitch relatedness and musical consonance,” Journal of New Music Research , vol. 45, no. 4, pp. 281–294, 2016
2016
-
[43]
A. A. Moles, Information theory and esthetic perception . The Univer- sity of Illinois Press, Urbana and London, 1966
1966
-
[44]
Perceptual scaling of synthesized musical timbres: Common dimensions, specificities, and latent subject classes,
S. McAdams, S. Winsberg, S. Donnadieu, G. De Soete, and J. Krim- phoff, “Perceptual scaling of synthesized musical timbres: Common dimensions, specificities, and latent subject classes,” Psychological re- search, vol. 58, pp. 177–192, 1995
1995
-
[45]
Maximum filter vibrato suppression for onset detection,
S. B ¨ock and G. Widmer, “Maximum filter vibrato suppression for onset detection,” in DAFx, vol. 7. Citeseer, 2013, p. 4
2013
-
[46]
ITU- R BS.1770-4,
ITU Radiocommunication Sector - Broadcasting service (sound), “ITU- R BS.1770-4,” https://www.itu.int/rec/R-REC-BS.1770/en
-
[47]
Inves- tigating the cortical tracking of speech and music with sung speech,
G. Cantisani, A. Chalehchaleh, G. Di Liberto, and S. Shamma, “Inves- tigating the cortical tracking of speech and music with sung speech,” in INTERSPEECH. ISCA, 2023, pp. 5157–5161
2023
-
[48]
The multivariate temporal response function (mtrf) toolbox: a matlab toolbox for relating neural signals to continuous stimuli,
M. J. Crosse, G. M. Di Liberto, A. Bednar, and E. C. Lalor, “The multivariate temporal response function (mtrf) toolbox: a matlab toolbox for relating neural signals to continuous stimuli,” Frontiers in human neuroscience, vol. 10, p. 604, 2016
2016
-
[49]
Neural signatures of musical and linguistic interactions during natural song listening,
G. Cantisani, S. Shamma, and G. M. Di Liberto, “Neural signatures of musical and linguistic interactions during natural song listening,” Hal preprint, 2024
2024
Reviewed August 10, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.