Exploring the Feasibility of LLMs for Automated Music Emotion Annotation

Jon McCormack; Maria Teresa Llano; Meng Yang; Wanchao Su

REVIEW 3 major objections 3 minor 26 references

Exploring the Feasibility of LLMs for Automated Music Emotion Annotation

T0 review · 3 major / 3 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A large language model, GPT-4o, can annotate music emotion labels that are less accurate than expert consensus yet fall within the natural spread of disagreement among human experts.

desk verdict Useful empirical check on GPT-4o for music-emotion annotation, but the scalability claim depends on a contamination control that the abstract does not show and the full text does not reveal. read the letter →

arxiv 2508.12626 v1 pith:SYIK5QFF submitted 2025-08-18 cs.SD

Meng Yang , Jon McCormack , Maria Teresa Llano , Wanchao Su This is my paper · ORCID

classification cs.SD

keywords musicemotionannotationGPT-4ovalence-arousalframeworkinter-annotatoragreementGiantMIDI-Pianolargelanguagemodelsaffectivecomputingautomatic

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks whether a large language model, GPT-4o, can take over the costly manual labelling of music emotion. The authors annotated the classical piano pieces of GiantMIDI-Piano in a four-quadrant valence-arousal space (pleasantness and energy) and compared the model's labels with those of three human experts. Their central finding is that GPT-4o is less accurate overall and less nuanced on specific emotional states, but its variability as an annotator falls inside the range of normal disagreement among human experts. The paper argues that this makes GPT-based annotation a scalable, cost-effective alternative for building large emotion-labelled music collections, even though it does not yet match expert judgement.

What carries the argument

The load-bearing object is the four-quadrant valence-arousal framework, a two-axis map of emotion in which one axis is valence (pleasant to unpleasant) and the other is arousal (energetic to calm), with each quadrant carrying a broad emotional family. The paper's mechanism is to treat GPT-4o as one more annotator on this map and to compare its label distribution and agreement pattern against three human experts. The decisive metric is weighted accuracy that accounts for inter-expert agreement: it rewards the model when its label is close to the expert majority, and the inter-annotator agreement metrics supply the yardstick that makes GPT-4o's variability look like natural expert disagreement rather than random noise.

What would settle it

Annotate the same GiantMIDI-Piano excerpts with a much larger expert panel, for example twenty or more, and also run GPT-4o repeatedly with the same prompt. If GPT-4o's labels fall systematically outside the panel's disagreement envelope, for instance clustering in one quadrant while the panel spreads across several, or varying more across its own runs than any single expert varies on re-annotation, the paper's claim of expert-level reliability would be refuted.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is a measured feasibility result: GPT-4o can be prompted to place classical piano works into a four-quadrant valence-arousal framework, and the resulting labels, while less accurate than the human experts' consensus and coarser in distinguishing specific emotional states, sit within the observed spread of expert disagreement. The evaluation uses standard accuracy, a weighted accuracy that credits answers close to what most experts chose, inter-annotator agreement metrics, and distributional similarity of label sets. Because the model's disagreement with expert labels is comparable to the disagreement experts show among themselves, the paper concludes that the remaining error is not a sign of model unreliability but of the inherent subjectivity of music emotion, and that GPT-4o is therefore a viable scalable annotator despite its lower overall accuracy.

Load-bearing premise

The evaluation treats the annotations of just three human experts as the ground truth for the emotion of each piece, so if those experts are not representative of how listeners perceive these works, the conclusion that GPT-4o's variability falls within expert disagreement may not generalize.

Editorial extensions

If this is right

Large classical-music collections such as GiantMIDI-Piano can be annotated for emotion at a fraction of the time and cost of manual labelling, enabling datasets far larger than expert annotation alone could produce.
For downstream tasks such as music recommendation or emotion-based retrieval, GPT-4o labels could serve as training data wherever coarse valence-arousal categories suffice.
The weighted-accuracy-with-expert-agreement evaluation gives future automated annotation studies a way to judge whether machine disagreement is acceptable relative to human disagreement.
Because GPT-4o is less accurate overall and less nuanced on specific emotional states, it is not a drop-in replacement where fine-grained emotion distinctions matter; the paper's claim is specifically about scalable coarse annotation.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

If disagreement among experts is used as the acceptance threshold, the same standard should apply to other subjective annotation tasks, such as aesthetic quality or sentiment intensity, where a model's disagreement with a single gold label may overstate its failure.
One test the paper leaves implicit is repeated prompting: GPT-4o's within-model consistency across multiple runs of the same excerpt would directly measure the stability that the inter-rater metrics are proxying for.
The corpus is symbolic MIDI piano music, so the result does not automatically extend to full audio with timbre, vocals, or non-classical genres; a natural next test is the same prompting protocol on audio excerpts.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

You should know two things before spending time on this preprint. First, the evaluation design described in the abstract is sensible: GPT-4o annotations are compared against three human experts on GiantMIDI-Piano using accuracy, inter-annotator agreement, and distributional similarity. Second, the full text I received is corrupted—garbled beyond the abstract—so any judgment has to be provisional. That limits how much I can credit the details.

What the paper does well: it addresses a real bottleneck. Music emotion annotation is labor-intensive, and a systematic test of whether GPT-4o can produce usable labels is a legitimate contribution to MIR. The abstract is honest about where GPT falls short—lower overall accuracy, less nuance—while showing its variability falls within expert disagreement. That is the kind of balanced result that is actually useful. The metric set is more thorough than a simple accuracy number.

Soft spots, in proportion to how soft they are. The three-expert gold standard is small. Without confidence intervals or significance tests, I can't tell if 'within natural disagreement' is a strong claim or just noise. The larger concern is contamination. GiantMIDI-Piano consists of well-known classical piano pieces. GPT-4o was trained on web text that discusses these pieces and their emotional character. If the annotation prompt included titles or composers, or if the model identifies the piece from the MIDI, the reported accuracy might reflect memorized associations rather than music-emotion perception. That would undermine the 'scalable alternative' claim, since the pipeline would not generalize to arbitrary new music. The abstract shows no control for this—no anonymized prompts, no unfamiliar pieces. My stress-test note is conditional: it could be that the full paper includes such a control. But as presented, the claim is unsupported.

Also worth noting, though minor: 'acceptable variability' is defined by the same three experts used as ground truth. That is a defensible design choice, but it makes the conclusion partially circular.

Who is this for? Researchers in music information retrieval and affective computing, especially those building annotation pipelines. It is a useful empirical data point, not a conceptual breakthrough. I would send it to peer review: the question is important, the design can be checked, and the contamination concern deserves a formal answer. My own verdict remains unverdictable until I can read the actual methods and tables.

Referee Report

3 major / 3 minor

Summary. The paper examines whether GPT-4o can serve as a scalable annotator of music emotion. It annotates GiantMIDI-Piano, a classical piano MIDI dataset, in a four-quadrant valence-arousal framework, and compares GPT-4o outputs against labels from three human experts. The evaluation reportedly covers standard accuracy, weighted accuracy that accounts for inter-expert agreement, inter-annotator agreement metrics, and distributional similarity. The authors find that GPT-4o is less accurate overall and less nuanced than the human experts, but that its inter-annotator variability falls within the range of natural disagreement among the experts, and they conclude that GPT-based annotation is a promising, cost-effective scalable alternative. The full text as provided to the referee is severely corrupted, so the experimental protocol and exact quantitative results cannot be verified from the manuscript itself.

Significance. If the results hold, the paper provides a useful benchmark for LLM-based music emotion annotation and some evidence about the reliability of GPT-4o labels in a four-quadrant valence-arousal space. The study has two strengths: it compares the model against human experts rather than only self-consistency, and it examines multiple evaluation perspectives (accuracy, agreement, distributional similarity) instead of a single metric. No circularity is apparent: GPT outputs are assessed against independently elicited human labels. However, the significance is conditional on two load-bearing points that the abstract alone does not settle: whether the model is actually inferring emotion from the music content, and whether the three-expert ground truth supports the statistical claims about "natural disagreement." Because the supplied full text is unreadable, these points cannot currently be checked.

major comments (3)

[Abstract / experimental protocol] The central scalability claim presupposes that GPT-4o infers emotion from the musical content itself. GiantMIDI-Piano consists largely of well-known classical piano works whose emotional character is discussed extensively in web-scale training text, so if the annotation prompt includes title or composer information, or if the model recognizes a piece, the reported accuracy and "within-expert-disagreement" variability may reflect retrieval of memorized associations rather than perceptual judgment. The abstract and the readable fragments do not describe any control, such as anonymized prompts, newly composed pieces, or obscure held-out stimuli. Please report the exact prompt used and add such a control; without it, the conclusion that GPT-4o is a scalable alternative for arbitrary new music is unsupported.
[Evaluation design / metrics (abstract)] The evaluation rests on labels from only three human experts. The claim that GPT-4o's variability is "within the range of natural disagreement among experts" requires a statistical comparison of the GPT-expert agreement distribution with the expert-expert agreement distribution; with three experts, the latter has only three pairwise values and is too coarse to establish equivalence unless the data are substantially richer. Please report per-item and per-expert agreement matrices, confidence intervals for the relevant agreement metrics, and a significance test comparing GPT-to-expert agreement with expert-to-expert agreement.
[Full text (all sections)] The complete text supplied to the referee is corrupted and unreadable, and no experimental detail can be verified: the prompt, the number of pieces annotated, the number of GPT runs, the exact definition of the weighted accuracy, the handling of the four quadrants, and the computation of inter-annotator metrics are all unrecoverable from the given rendering. This renders the reported evaluation unverifiable in its current form. Please provide a properly rendered manuscript so that the claims can be checked against the actual protocol.

minor comments (3)

[Abstract] The number of pieces annotated and the number of independent GPT annotation runs should be stated in the abstract, because the interpretation of inter-annotator reliability differs depending on whether the reported spread is across runs or across prompts within a single run.
[Abstract / Methods] The phrase "weighted accuracy that accounts for inter-expert agreement" is ambiguous: it should state whether the weights are derived from confusion patterns across experts, from per-item confidence, or from another source.
[Figures and tables] The figure and table captions are illegible in the supplied rendering; please ensure the final PDF uses properly embedded fonts and standard character encoding so that all quantitative results can be read.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity found: GPT-4o labels are evaluated against independent human expert annotations, with no fitted parameter or self-citation chain forcing the result.

full rationale

The paper's central comparison is external: GPT-4o annotations of GiantMIDI-Piano are measured against annotations provided by three human experts (abstract). No parameter is fitted to the human labels and then renamed as a prediction; the metrics (standard accuracy, weighted accuracy accounting for inter-expert agreement, inter-annotator agreement, distributional similarity) are evaluative statistics, not quantities derived from the model's own outputs by construction. The 'within the range of natural disagreement among experts' claim is an empirical finding about dispersion relative to the human judges, not an equation that forces GPT's labels to equal the human labels. The paper does not rely on a self-citation chain or an imported uniqueness theorem; its premises are the dataset, the annotation prompt, and the human gold standard. The skeptic's memorization concern is a legitimate external-validity threat but is not circularity: even if GPT-4o relies on memorized associations, the comparison is still against independent human labels. The provided full text is heavily corrupted, so no specific equation could be quoted exhibiting a definitional reduction; absent such evidence, the appropriate finding is no significant circularity.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

No free parameters or invented entities are identifiable from the abstract. The central evaluation rests on the valence-arousal framework, the three-expert human baseline, and the classical-piano dataset, each of which is a domain assumption rather than an ad hoc construct.

assumptions (3)

domain assumption Valence-arousal four-quadrant framework is a valid and sufficient representation of music emotion.
The study maps all annotations into four quadrants; if this coarse framework misses relevant emotion dimensions, the evaluation is incomplete. This is a modeling assumption stated in the abstract.
domain assumption The three human experts' annotations are a reliable ground truth despite inter-expert disagreement.
GPT performance is measured against human labels; the study defines 'natural disagreement' from the same experts, so the human baseline is load-bearing.
domain assumption GiantMIDI-Piano is representative of the music domain for which annotation is needed.
The feasibility conclusion is drawn from classical piano MIDI; generalization to other genres and audio formats is not established in the abstract.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Exploring the Feasibility of LLMs for Automated Music Emotion Annotation." pith.science (2026). https://pith.science/paper/SYIK5QFF

@misc{pith2026250812626,
  author       = {Pith},
  title        = {Pith review of: Exploring the Feasibility of LLMs for Automated Music Emotion Annotation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/SYIK5QFF}},
  note         = {Machine review of arXiv:2508.12626}
}

read the original abstract

Current approaches to music emotion annotation remain heavily reliant on manual labelling, a process that imposes significant resource and labour burdens, severely limiting the scale of available annotated data. This study examines the feasibility and reliability of employing a large language model (GPT-4o) for music emotion annotation. In this study, we annotated GiantMIDI-Piano, a classical MIDI piano music dataset, in a four-quadrant valence-arousal framework using GPT-4o, and compared against annotations provided by three human experts. We conducted extensive evaluations to assess the performance and reliability of GPT-generated music emotion annotations, including standard accuracy, weighted accuracy that accounts for inter-expert agreement, inter-annotator agreement metrics, and distributional similarity of the generated labels. While GPT's annotation performance fell short of human experts in overall accuracy and exhibited less nuance in categorizing specific emotional states, inter-rater reliability metrics indicate that GPT's variability remains within the range of natural disagreement among experts. These findings underscore both the limitations and potential of GPT-based annotation: despite its current shortcomings relative to human performance, its cost-effectiveness and efficiency render it a promising scalable alternative for music emotion annotation.

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

26 extracted references · 23 canonical work pages

[1]

write newline

" write newline "" initialize.prev.this.status FUNCTION begin.bib " write newline preamble empty 'skip preamble write newline if " thebibliography " longest.label * " " * write newline " [1] #1 " write newline " url@samestyle " write newline " " write newline " [2] #2 " write newline " =0pt " write newline " " ALTinterwordstretchfactor * " " * write newli...
[2]

Yang and H

Y.-H. Yang and H. H. Chen, ``Machine recognition of music emotion: A review,'' ACM Transactions on Intelligent Systems and Technology, vol. 3, no. 3, May 2012

work page 2012
[3]

Aljanaki, y.-h

A. Aljanaki, y.-h. Yang, and M. Soleymani, ``Developing a benchmark for emotional analysis of music,'' PLOS ONE, vol. 12, p. e0173392, 03 2017

work page 2017
[4]

Aljanaki, F

A. Aljanaki, F. Wiering, and R. C. Veltkamp, ``Studying emotion induced by music through a crowdsourcing game,'' Information Processing & Management, vol. 52, no. 1, p. 115–128, Jan. 2016

work page 2016
[5]

L. N. Ferreira and J. Whitehead, ``Learning to generate music with sentiment,'' in Proceedings of the Conference of the International Society for Music Information Retrieval, Delft, Netherlands, 2019, pp. 384--390

work page 2019
[6]

H.-T. Hung, J. Ching, S. Doh, N. Kim, J. Nam, and Y.-H. Yang, ``Emopia: A multi-modal pop piano dataset for emotion recognition and emotion-based music generation,'' in International Society for Music Information Retrieval Conference, 2021

work page 2021
[7]

Barthet, G

M. Barthet, G. Fazekas, and M. Sandler, ``Music emotion recognition: From content- to context-based models,'' in From Sounds to Music and Emotions, M. Aramaki, M. Barthet, R. Kronland-Martinet, and S. Ystad, Eds. 1em plus 0.5em minus 0.4em Berlin, Heidelberg: Springer Berlin Heidelberg, 2013, pp. 228--252

work page 2013
[8]

Delbouys, R

R. Delbouys, R. Hennequin, F. Piccoli, J. Royo-Letelier, and M. Moussallam, ``Music mood detection based on audio and lyrics with deep neural net,'' ArXiv, vol. abs/1809.07276, 2018

arXiv 2018

Show all 26 references

[9]

F. H. Rachman, R. Sarno, and C. Fatichah, ``Music emotion detection using weighted of audio and lyric features,'' in 2020 6th Information Technology International Seminar (ITIS), 2020, pp. 229--233

2020
[10]

X. Hu, K. Choi, and J. S. Downie, ``A framework for evaluating multimodal music mood classification,'' Journal of the Association for Information Science and Technology, vol. 68, 2017. [Online]. Available: https://api.semanticscholar.org/CorpusID:45480061

2017
[11]

Agrawal, R

Y. Agrawal, R. G. R. Shanker, and V. Alluri, ``Transformer-based approach towards music emotion recognition from lyrics,'' in Advances in Information Retrieval: 43rd European Conference on IR Research, ECIR 2021, Virtual Event, March 28 – April 1, 2021, Proceedings, Part II, B...

2021
[12]

Hu and J

X. Hu and J. S. Downie, ``Exploring mood metadata: Relationships with genre, artist and usage metadata,'' in International Society for Music Information Retrieval Conference, 2007. [Online]. Available: https://api.semanticscholar.org/CorpusID:16794525

2007
[13]

Q. Kong, B. Li, J. Chen, and Y. Wang, ``Giantmidi-piano: A large-scale midi dataset for classical piano music,'' Transactions of the International Society for Music Information Retrieval, May 2022

2022
[14]

Turnbull, L

D. Turnbull, L. Barrington, D. Torres, and G. Lanckriet, ``Towards musical query-by-semantic-description using the cal500 data set,'' in Proceedings of the 30th Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, ser. SIGIR '07. 1em ...

2007
[15]

P. L. Louro, H. Redinho, R. Santos, R. Malheiro, R. Panda, and R. P. Paiva, ``Merge -- a bimodal dataset for static music emotion recognition,'' 2025. [Online]. Available: https://arxiv.org/abs/2407.06060

2025 arXiv
[16]

Y. E. Kim, E. M. Schmidt, R. Migneco, B. G. Morton, P. Richardson, J. J. Scott, J. A. Speck, and D. Turnbull, ``Music emotion recognition: A state of the art review,'' in International Society for Music Information Retrieval Conference, 2010

2010
[17]

Donnelly and A

P. Donnelly and A. Beery, ``Evaluating large-language models for dimensional music emotion prediction from social media discourse,'' in Proceedings of the 5th International Conference on Natural Language and Speech Processing (ICNLSP 2022). 1em plus 0.5em minus 0.4em Trento, I...

2022
[18]

Y. E. Kim, E. M. Schmidt, and L. Emelle, ``Moodswings: A collaborative game for music mood label collection,'' in International Society for Music Information Retrieval Conference, 2008. [Online]. Available: https://api.semanticscholar.org/CorpusID:14382686

2008
[19]

Z. Tan, D. Li, S. Wang, A. Beigi, B. Jiang, A. Bhattacharjee, M. Karami, J. Li, L. Cheng, and H. Liu, ``Large language models for data annotation and synthesis: A survey,'' in Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing. 1em plus 0.5e...

2024
[20]

Gilardi, M

F. Gilardi, M. Alizadeh, and M. Kubli, ``Chatgpt outperforms crowd workers for text-annotation tasks,'' Proceedings of the National Academy of Sciences of the United States of America, vol. 120, 2023

2023
[21]

He, C.-Y

Z. He, C.-Y. Huang, C.-K. C. Ding, S. Rohatgi, and T.-H. K. Huang, ``If in a crowdsourced data annotation pipeline, a gpt-4,'' in Proceedings of the 2024 CHI Conference on Human Factors in Computing Systems, ser. CHI '24. 1em plus 0.5em minus 0.4em New York, NY, USA: Associati...

2024
[22]

S. Wang, Y. Liu, Y. Xu, C. Zhu, and M. Zeng, ``Want to reduce labeling cost? GPT -3 can help,'' in Findings of the Association for Computational Linguistics: EMNLP 2021. 1em plus 0.5em minus 0.4em Punta Cana, Dominican Republic: Association for Computational Linguistics, Nov. ...

2021
[23]

Edmonds and J

D. Edmonds and J. Sedoc, ``Multi-emotion classification for song lyrics,'' in Proceedings of the Eleventh Workshop on Computational Approaches to Subjectivity, Sentiment and Social Media Analysis. 1em plus 0.5em minus 0.4em Online: Association for Computational Linguistics, Ap...

2021
[24]

Donnelly and A

P. Donnelly and A. Beery, ``Evaluating large-language models for dimensional music emotion prediction from social media discourse,'' in Proceedings of the 5th International Conference on Natural Language and Speech Processing (ICNLSP 2022), M. Abbas and A. A. Freihat, Eds. 1em...

2022
[25]

Russell, `` A circumplex model of affect ,'' Journal of personality and social psychology, vol

J. Russell, `` A circumplex model of affect ,'' Journal of personality and social psychology, vol. 39, no. 6, pp. 1161--1178, 1980

1980
[26]

Gabrielsson, ``Emotion perceived and emotion felt: Same or different?'' Musicae Scientiae, vol

A. Gabrielsson, ``Emotion perceived and emotion felt: Same or different?'' Musicae Scientiae, vol. 5, no. 1\_suppl, pp. 123--147, 2001

2001

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.

[1] [1]

write newline

" write newline "" initialize.prev.this.status FUNCTION begin.bib " write newline preamble empty 'skip preamble write newline if " thebibliography " longest.label * " " * write newline " [1] #1 " write newline " url@samestyle " write newline " " write newline " [2] #2 " write newline " =0pt " write newline " " ALTinterwordstretchfactor * " " * write newli...

[2] [2]

Yang and H

Y.-H. Yang and H. H. Chen, ``Machine recognition of music emotion: A review,'' ACM Transactions on Intelligent Systems and Technology, vol. 3, no. 3, May 2012

work page 2012

[3] [3]

Aljanaki, y.-h

A. Aljanaki, y.-h. Yang, and M. Soleymani, ``Developing a benchmark for emotional analysis of music,'' PLOS ONE, vol. 12, p. e0173392, 03 2017

work page 2017

[4] [4]

Aljanaki, F

A. Aljanaki, F. Wiering, and R. C. Veltkamp, ``Studying emotion induced by music through a crowdsourcing game,'' Information Processing & Management, vol. 52, no. 1, p. 115–128, Jan. 2016

work page 2016

[5] [5]

L. N. Ferreira and J. Whitehead, ``Learning to generate music with sentiment,'' in Proceedings of the Conference of the International Society for Music Information Retrieval, Delft, Netherlands, 2019, pp. 384--390

work page 2019

[6] [6]

H.-T. Hung, J. Ching, S. Doh, N. Kim, J. Nam, and Y.-H. Yang, ``Emopia: A multi-modal pop piano dataset for emotion recognition and emotion-based music generation,'' in International Society for Music Information Retrieval Conference, 2021

work page 2021

[7] [7]

Barthet, G

M. Barthet, G. Fazekas, and M. Sandler, ``Music emotion recognition: From content- to context-based models,'' in From Sounds to Music and Emotions, M. Aramaki, M. Barthet, R. Kronland-Martinet, and S. Ystad, Eds. 1em plus 0.5em minus 0.4em Berlin, Heidelberg: Springer Berlin Heidelberg, 2013, pp. 228--252

work page 2013

[8] [8]

Delbouys, R

R. Delbouys, R. Hennequin, F. Piccoli, J. Royo-Letelier, and M. Moussallam, ``Music mood detection based on audio and lyrics with deep neural net,'' ArXiv, vol. abs/1809.07276, 2018

arXiv 2018