{"id":"9c970c5e-64bd-4f8b-935d-b39a9a305c02","arxiv_id":"2501.11570","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Common uncertainty-estimation methods fail to predict inter-rater disagreement in music emotion recognition, even when the true per-song disagreement is supplied during training.","lead":"This paper tests five common ways to estimate uncertainty in music emotion recognition, and finds none reliably capture how much listeners disagree on a song's emotional rating. It is a useful warning for systems that claim to know their own confidence on subjective ratings.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The NLL and KLD loss equations in Section 2.2 are not the stated losses, so two of the five failure rows in Table 3 do not test the named methods.","rationale":"The reader correctly flagged the loss-formula errors as issues, but identified the noisy per-song SD as the weakest assumption. I disagree with that ranking: with roughly ten raters per song, the sample SD is noisy but the attenuation ceiling for a perfect model is not near zero. Concretely, for n=10 and the observed SD distribution, the maximum achievable correlation between a perfect prediction of the true SD and the empirical SD is approximately 0.4-0.6, while the observed correlations in Table 3 are near zero or negative. Thus target noise alone cannot account for the complete absence of correlation. The more decisive defect is that the NLL and KLD losses in Section 2.2 are mathematically not the named losses, so two of the five rows in the central evidence table are invalid tests of those methods. The claim that 'none of the investigated methods' works is therefore only securely supported by the MSE, random-seeds, and MC-dropout conditions. Since those three conditions do fail, the paper's broad conclusion is plausible but needs the loss corrections to be fully substantiated. The reader's conditional verdict already captures this need, so no verdict change is required; the stress-test pass sharpens the specific check that should settle the issue.","tokens_in":11795,"tokens_out":9293,"duration_ms":105786,"concrete_test":"Reimplement both methods with the correct loss functions on the same DEAM splits and frozen MusicFM-MSD features: NLL as 0.5*((mu_hat-mu)/sigma_hat)^2 + log(sigma_hat) (constants omitted), and KLD as log(sigma/sigma_hat) + (sigma_hat^2 + (mu_hat-mu)^2)/(2*sigma^2) - 0.5. Retrain the same FCN with the same seeds and report Table 3 metrics. If corrected NLL and KLD still give Pearson/Spearman correlations near zero and R^2 below zero, the central claim is robust; if either shows positive correlation with the empirical SD, the paper's 'none of the investigated methods' conclusion is falsified for those methods.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central negative claim is that none of the investigated methods can model inter-rater uncertainty, with Table 3 as the quantitative evidence. For two of the five methods, however, the published loss functions are not the losses they are claimed to be. Equation (3) gives L_NLL = 0.5*((mu_hat-mu)/sigma_hat)^2 + 0.5*log(sigma_hat), but the Gaussian negative log-likelihood for a target mu is 0.5*((mu_hat-mu)/sigma_hat)^2 + log(sigma_hat) up to constants; the regularization term is off by a factor of two. Equation (2) is not the KL divergence between N(mu_hat, sigma_hat^2) and N(mu, sigma^2): the first term should have the target variance sigma^2 in the denominator, not sigma_hat^2, and the expression is missing the log(sigma) and constant terms. These are not harmless transcription choices because the balance between the data-fit term and the variance penalty changes, and for KLD the wrong denominator actively encourages the model to inflate sigma_hat. Consequently, the NLL and KLD rows of Table 3 do not establish that the correctly implemented NLL and KLD methods fail. The MSE, random-seeds, and MC-dropout rows still fail, so the broad conclusion may survive, but the published study does not substantiate its 'none of the investigated methods' claim for these two methods. The authors' own Section 5 limitation about the frozen foundation model is also relevant, but the loss-function discrepancy is more fundamental because it invalidates specific evidence rather than merely narrowing generalizability.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper investigates whether deep-learning models can predict not only the mean but also the standard deviation of valence and arousal ratings in music emotion recognition, using the DEAM dataset and treating per-song SD as ground-truth interrater uncertainty. Five methods are compared: an MSE loss with an SD target, a KLD loss between predicted and empirical Gaussians, a Gaussian NLL loss, multiple random training seeds, and Monte Carlo dropout. The models use a frozen MusicFM foundation model with a small FCN head. The main empirical result is that mean prediction works reasonably (R2 around 0.6), whereas all methods fail to predict the per-song SD, with R2 well below zero and correlations near zero, leading the authors to conclude that none of the investigated methods can effectively model interrater uncertainty.","tokens_in":12131,"tokens_out":7414,"duration_ms":77706,"significance":"Assuming the negative result survives correction of the loss functions, the paper provides a useful, reproducible benchmark of standard UQ methods on a real subjective-regression task with external data. Its strengths are the use of a public benchmark dataset (DEAM), a held-out split with published song IDs, multiple evaluation metrics, and an honest discussion of limitations, including the frozen foundation model and the small number of raters. The finding that mean predictions are reasonable while SD predictions are essentially uncorrelated with empirical interrater SDs is important for MER and for psychometric machine learning more generally. However, as written, the evidence for the NLL and KLD rows of Table 3 does not test the methods named, so the strongest version of the conclusion is not yet established.","major_comments":[{"comment":"Equation (2) is not the KL divergence between N(mu_hat, sigma_hat^2) and N(mu, sigma^2). The correct KL, up to constants with respect to the predicted parameters, is 0.5*((mu_hat-mu)^2)/sigma^2 + 0.5*(sigma_hat^2/sigma^2) - log(sigma_hat). The first term in Eq. (2) divides by sigma_hat^2 instead of sigma^2, which changes the gradient so that increasing sigma_hat reduces the data-fit penalty; the log(sigma) and constant terms are also omitted. Consequently, the KLD row of Table 3 does not test the named KLD loss, and this is load-bearing for the paper's claim that 'none of the investigated methods' can model interrater uncertainty.","section":"Section 2.2, Eq. (2)"},{"comment":"Equation (3) is not the Gaussian negative log-likelihood for a target mu under N(mu_hat, sigma_hat^2). Up to an additive constant, the correct NLL is 0.5*((mu_hat-mu)/sigma_hat)^2 + log(sigma_hat); Eq. (3) has 0.5*log(sigma_hat), halving the variance penalty. Because this changes the balance between data fit and regularization, the NLL row of Table 3 does not establish that a correctly implemented NLL method fails. Since Table 3 is the core evidence for the central negative claim, this must be corrected and the experiments re-run or re-reported.","section":"Section 2.3, Eq. (3)"},{"comment":"The per-song standard deviation targets are computed from at least 10 annotators per song (Section 3.1). With about 10 Likert ratings, the sample SD has large sampling error, so even a perfect predictor of the true interrater SD would show substantially attenuated correlation with these targets. Section 5 mentions insufficient ratings as a possible cause, but the paper does not quantify this attenuation. The strong conclusion that the methods 'cannot effectively model' interrater uncertainty should be qualified or supported by a reliability analysis or simulation before the negative result is generalized.","section":"Section 3.1 and Section 5"}],"minor_comments":[{"comment":"The Random Seeds row reports no standard deviations or confidence intervals, unlike the other rows; please explain or add them.","section":"Table 3"},{"comment":"The notation sigma_hat is used for a standard deviation in Eq. (2), but Eq. (3) is written in a way that is ambiguous about whether sigma_hat denotes a standard deviation or a variance; please define the parameterization explicitly and keep it consistent throughout.","section":"Sections 2.2 and 2.3"},{"comment":"The colorbar in Figure 5 is labeled 'Emp. Mean' although the figure displays predicted and empirical SDs; if the points are colored by the empirical mean, please state this in the caption and explain the relevance, and otherwise correct the label.","section":"Figure 5"},{"comment":"The phrase 'it is possible to arrive the negative log-likelihood loss' should read 'arrive at'.","section":"Section 2.3"}],"recommendation":"major_revision","confidential_remarks":"For the editor: the self-citations for the NLL loss are standard and do not raise circularity concerns. The central issue is that two of the five evaluated methods are not implemented as named; this is fixable within the scope of the manuscript by correcting the losses and re-running. The target-noise issue should also be addressed quantitatively before the broad negative claim is accepted."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's my take. The paper does something genuinely useful: it tests several standard UQ methods on a subjective regression task (MER) and shows that none of them predict inter-rater SD well. That's an important negative result for anyone building trustworthy music recommendation or therapy tools. The evaluation is careful—held-out splits, 15 seeds, code released—and the discussion, including the frozen-foundation caveat, is honest.\n\nThe problem is in Section 2.2. Equation (3) is not the Gaussian negative log-likelihood; it has 0.5 log(σ̂) instead of log(σ̂), so the variance penalty is halved. Equation (2) is not a KL divergence at all: the first term should have the target variance in the denominator, and the log(σ) term is missing. These aren't cosmetic transcription issues. They change the loss landscape, and for the KLD the wrong denominator actively encourages the model to enlarge σ̂. So the NLL and KLD rows of Table 3 do not represent the correctly implemented methods. That undercuts the “none of the investigated methods” claim, at least for two of the five.\n\nThe other three methods—MSE, random seeds, and MC dropout—still fail, so the broad negative result might survive a corrected run. But the current paper overreaches; the conclusion should be limited to the methods as implemented, or the authors need to rerun with correct losses.\n\nThe other soft spots are minor by comparison: the per-song SD targets come from roughly ten annotators, so the target itself is noisy; the paper mentions this but doesn't quantify it. And they report R² as “≪0” without exact numbers, which makes meta-analysis harder. Neither undermines the study's core.\n\nBottom line: this is a legitimate empirical study with a real flaw in two of the five method implementations. It deserves a serious referee—the question it asks is important and the setup is solid—but the NLL and KLD results need to be corrected before the paper can be believed on those rows. If I were editing, I'd send it out, ask for the corrected experiments, and then judge whether the negative result holds.","headline":"A useful negative benchmark for UQ in music emotion recognition, but the NLL and KLD loss equations are not the stated losses, so the 'none of the methods' claim overreaches until they are fixed.","tokens_in":12668,"tokens_out":4754,"would_cite":false,"duration_ms":48401,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper tests whether standard deep-learning uncertainty techniques can capture how much human raters disagree about the emotion in a piece of music, and reports that none of them can.","keywords":["uncertainty quantification","music emotion recognition","emotion recognition","psychometric machine learning","aleatoric uncertainty","interrater disagreement","valence-arousal","Monte Carlo dropout"],"falsifier":"Take a set of songs with substantially more ratings per song (for example, fifty or more), recompute the per-song standard deviations, and rerun the same five methods; if any method's predicted spread correlates strongly with these more reliable targets despite failing on the ten-rater targets, the paper's conclusion would collapse into an artifact of noisy evaluation labels. A complementary check is a simulation study in which songs have known true variances and the same methods are trained on exact rather than estimated targets.","tokens_in":11609,"feed_emoji":"🎵","tokens_out":9257,"duration_ms":90569,"temperature":0.7,"pith_summary":"This paper asks whether modern deep-learning systems can do more than predict the average emotion a piece of music evokes: can they also predict how much listeners will disagree? The authors work with a dataset of 1,744 songs, each rated on valence and arousal by at least ten people, and compare five uncertainty-estimation methods: two losses that receive the empirical rating variance as an extra training target (mean squared error and KL divergence), a negative log-likelihood loss that learns variance without such a target, and two inference-time sampling schemes (multiple random seeds and Monte Carlo dropout). The central finding is negative. All five methods predict the mean valence and arousal reasonably well, but none produces standard-deviation estimates that correlate with the observed per-song spread: every $R^2$ value for standard-deviation prediction is effectively well below zero. The paper concludes that the uncertainty arising from interrater disagreement cannot currently be modeled by these standard approaches, even when the empirical uncertainty is supplied during training.","feed_headline":"Music emotion AI predicts averages, not rater disagreement","feed_subtitle":"All five methods score below zero on predicting rating spread, even when the true spread is the training target.","key_machinery":"The central object is the conditional Gaussian model $Y \\mid X = x \\sim \\mathcal{N}(\\mu_x, \\Sigma_x)$ for a song's valence-arousal ratings, with $\\Sigma_x$ taken diagonal so each song has a mean and a per-dimension standard deviation. The paper attaches five uncertainty estimators to a frozen pretrained music-feature extractor followed by a small two-layer network: direct regression of the distribution parameters via an MSE loss, minimization of the KL divergence between predicted and empirical Gaussians, negative log-likelihood training that learns $\\hat\\sigma$ without an empirical variance target, and two inference-time sampling schemes (separate training seeds and dropout masks) whose output spread is used as the uncertainty estimate. The comparison that carries the argument is between these predicted standard deviations and the empirical per-song standard deviations computed from the annotator ratings.","core_discovery":"On the paper's own terms, the discovery is that interrater disagreement in music emotion recognition is not captured by any of the investigated uncertainty-quantification methods. Treating each song's ratings as a bivariate Gaussian with diagonal covariance, the paper trains models to output both a mean and a standard deviation for valence and arousal. Whether the empirical standard deviation is supplied as a training target (MSE and KLD losses) or learned indirectly (NLL loss) or replaced by inference-time variation (random seeds, MC dropout), the predicted standard deviations are essentially uncorrelated with the empirical ones, with $R^2$ values effectively well below zero and Pearson and Spearman correlations near zero or negative. The failure is not a matter of scale: the MSE- and KLD-trained models produce predicted spreads in the right range, but the ordering across songs is wrong. The authors' conclusion is that these methods, and the deterministic deep-learning machinery underneath them, are insufficient for modeling the variability of subjective responses.","pith_inferences":["The paper does not go this far, but its negative result could be partly an artifact of the evaluation target: with roughly ten ratings per song, the empirical standard deviation is a noisy estimate of the true interrater spread, and regressing toward such noisy targets would suppress even a genuine correlation. A dataset with many more raters per stimulus would separate this possibility from a tru","The paper does not test ordinal or count-based likelihoods. Since the raw ratings are discrete values normalized to a bounded interval, the Gaussian assumption may itself be a bottleneck; an ordinal likelihood that treats each rater's response directly could recover interrater spread where the Gaussian losses do not.","The paper notes that its feature extractor is frozen and that uncertainty information may have been discarded during embedding extraction; a natural test of this conjecture would be to fine-tune the backbone or draw features from several layers before applying the same five uncertainty heads."],"forward_implications":["If the paper is right, a music emotion recognition system that reports only a mean valence and arousal is not merely omitting a detail: the standard uncertainty-estimation toolkits attached to such systems will not supply the missing interrater spread.","The MSE- and KLD-trained models fail even though the empirical standard deviation is explicitly provided as a training target, so the deficit cannot be blamed on the absence of uncertainty supervision.","The random-seed and Monte Carlo dropout estimates systematically underestimate human disagreement, because the variation they measure is variation across model parameters, not across raters.","The negative result aligns with the paper's observation that recent large-scale uncertainty benchmarks outside music also fail to obtain accurate uncertainty estimates, suggesting the difficulty is not specific to this dataset or task.","The paper's conclusion points future work toward less common modeling strategies for subjective response distributions rather than incremental variants of the tested losses."],"supporting_citations":[{"why":"Supplies the dataset of 1,744 songs with multiple annotators per song; the empirical mean and standard-deviation targets are computed from these ratings.","marker":"[2]"},{"why":"Provides the Bayesian dropout approximation that justifies Monte Carlo dropout as an inference-time sampling method.","marker":"[7]"},{"why":"Frames heteroscedastic aleatoric uncertainty in deep learning and supports the negative log-likelihood loss and MC dropout choices.","marker":"[11]"},{"why":"Recent benchmark showing that standard methods fail to disentangle data and model uncertainty, which the paper cites to contextualize its negative findings.","marker":"[16]"},{"why":"One of the cited sources of the negative log-likelihood loss used when empirical uncertainty targets are unavailable.","marker":"[18]"},{"why":"Introduces the probabilistic valence-arousal modeling approach for music emotion that this paper's Gaussian formulation extends.","marker":"[21]"},{"why":"Another cited source of the negative log-likelihood loss and its application to predicting subjective responses.","marker":"[31]"},{"why":"Provides the frozen pretrained music foundation model used as the feature extractor for all experiments.","marker":"[32]"}],"fun_headline_variants":["Music emotion AI can't gauge rater disagreement, even when trained on it","Uncertainty in music emotion: methods fail to capture rater spread","Even with true spread as target, AI models can't predict emotion uncertainty","For music emotion, predicting average is easy, spread is not","Interrater disagreement in music emotion: no method captures it"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the per-song standard deviation computed from about ten raters accurately represents true interrater disagreement, even though a standard deviation based on so few ratings carries enough sampling error to hide a genuine signal.","fun_headline_variants_meta":{"raw":{"variants":["Music emotion AI can't gauge rater disagreement, even when trained on it","Uncertainty in music emotion: methods fail to capture rater spread","Even with true spread as target, AI models can't predict emotion uncertainty","For music emotion, predicting average is easy, spread is not","Interrater disagreement in music emotion: no method captures it"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000623,"raw_usage":{"total_tokens":2847,"prompt_tokens":868,"completion_tokens":1979,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":484,"completion_tokens_details":{"reasoning_tokens":1900}},"tokens_in":484,"tokens_out":1979,"duration_ms":13325,"temperature":1.0,"reasoning_tokens":1900,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T18:05:55.205788+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a set of songs with substantially more ratings per song (for example, fifty or more), recompute the per-song standard deviations, and rerun the same five methods; if any method's predicted spread correlates strongly with these more reliable targets despite failing on the ten-rater targets, the paper's conclusion would collapse into an artifact of noisy evaluation labels. A complementary check is a simulation study in which songs have known true variances and the same methods are trained on exact rather than estimated targets.","supporting_citations":[{"cited_title":"PLOS ONE12(3), e0173392 (Mar 2017).https: //doi.org/10.1371/journal.pone.0173392","cited_arxiv_id":null,"evidence_quote":"Supplies the dataset of 1,744 songs with multiple annotators per song; the empirical mean and standard-deviation targets are computed from these ratings."},{"cited_title":"In: Workshop Track Proceed- ings of the 4th International Conference on Learning Representations (2016)","cited_arxiv_id":null,"evidence_quote":"Provides the Bayesian dropout approximation that justifies Monte Carlo dropout as an inference-time sampling method."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Frames heteroscedastic aleatoric uncertainty in deep learning and supports the negative log-likelihood loss and MC dropout choices."},{"cited_title":"In: 38th Annual Conference on Neural Information Processing Systems","cited_arxiv_id":null,"evidence_quote":"Recent benchmark showing that standard methods fail to disentangle data and model uncertainty, which the paper cites to contextualize its negative findings."},{"cited_title":"In: Proceedings of the 11th International Society for Music Information Retrieval Conference’","cited_arxiv_id":null,"evidence_quote":"Introduces the probabilistic valence-arousal modeling approach for music emotion that this paper's Gaussian formulation extends."},{"cited_title":"IEEE Signal Processing Letters29, 1749–1753 (2022)","cited_arxiv_id":null,"evidence_quote":"Another cited source of the negative log-likelihood loss and its application to predicting subjective responses."}],"review_version":1}