Pith. sign in

REVIEW 2 major objections 5 minor 1 cited by

SMART: Tuning a symbolic music generation system with an audio domain aesthetic reward

T0 review · 2 major / 5 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read An audio-domain aesthetic reward can fine-tune symbolic music generation: tuned piano MIDI scored higher with listeners, but over-optimization collapses diversity.

desk verdict A genuinely new combination of audio reward and symbolic MIDI, but the listening-study significance is fragile due to an incompletely reported mixed model. read the letter →

arxiv 2504.16839 v1 pith:JGIQ5XKF submitted 2025-04-23 cs.SD

Nicolas Jonason , Luca Casini , Bob L. T. Sturm This is my paper · ORCID
classification cs.SD
keywords symbolicmusicgenerationreinforcementlearningaudioaestheticrewardGRPOMetaAudioboxAestheticspianoMIDIevaluationover-optimization
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks whether a model trained to predict human aesthetic ratings of music audio can be used as a reward signal to improve a symbolic music generator that outputs piano MIDI. The authors fine-tune a pretrained piano Transformer with Group Relative Policy Optimization, rendering each generated MIDI to 10 seconds of audio, scoring it with the Meta Audiobox Aesthetics content-enjoyment model, and using those scores as rewards. They report that the tuned model produces more notes, more polyphony, fewer empty beats, a wider pitch range, and more dynamic variation, and that a 14-participant listening study rated its outputs about 1.22 points higher on a 10-point enjoyability scale (p = 0.002). They also show that longer, less-regularized optimization keeps raising the proxy score while destroying output diversity. The contribution is a working demonstration that an audio-domain aesthetic reward can be propagated through a renderer into symbolic music generation, with the caveat that the reward is a proxy for genuine aesthetic experience.

What carries the argument

The load-bearing machinery is a closed loop from symbolic tokens to audio and back: a causal Transformer generates MIDI with a REMI+-style tokenizer; a soundfont renderer converts the MIDI to 10-second audio; Meta Audiobox Aesthetics assigns a content-enjoyment score to that audio; and Group Relative Policy Optimization (GRPO) turns the scores into group-normalized advantages used to update the policy, while a KL divergence term keeps the policy close to the frozen pretrained reference. The renderer is the critical bridge, because it lets an audio-trained reward model shape decisions that are made in symbolic space.

What would settle it

Render the same set of base and SMART MIDI outputs with at least two different soundfonts and collect enjoyability ratings from a larger, more diverse panel; if the post-intervention gain disappears or reverses for one renderer, the reward signal was partly an artifact of the specific synth rather than a property of the musical structure.

Watch

Extended reading notes

Core claim

The central claim is that audio-domain aesthetic preference models can act as effective rewards for tuning symbolic music generation. Concretely, the paper shows that optimizing a piano MIDI model toward Meta Audiobox Aesthetics content-enjoyment ratings of audio-rendered outputs, with a KL constraint keeping the policy near its pretrained reference, yields a model whose outputs receive higher predicted aesthetic scores and are rated as more enjoyable by a convenience-sample listening panel. The 200-step intervention shifts measurable musical features, such as more notes, more polyphony, fewer rests, wider pitch range, and lower and more varied velocities, and the linear mixed-effects analysis estimates a 1.22-point gain in enjoyability (p = 0.002). The paper also demonstrates the known over-optimization failure mode: pushing the same objective for 1000 steps, or with no KL penalty, increases the reward but visibly collapses the variety of outputs.

Load-bearing premise

The training loop assumes that Meta Audiobox Aesthetics content-enjoyment scores on 10-second audio-rendered piano excerpts are a reliable reward signal that transfers to true human enjoyment of generated music; if the reward mostly captures renderer-specific timbre or short-horizon surface features, the reported improvement may not replicate with other renderers, longer excerpts, or broader listener populations.

Editorial extensions

If this is right

  • The same SMART pipeline can be applied to other symbolic generators and other audio aesthetic preference models, since the renderer is the only audio-specific component in the loop.
  • Aesthetic-reward tuning changes musical surface features in predictable directions, such as denser, more polyphonic, and more dynamic output, which suggests the proxy reward shapes perceived musical content rather than only low-level audio quality.
  • Audio-domain aesthetic prediction offers a reference-free way to evaluate symbolic music generators, using a scalar enjoyment score instead of comparisons against a reference dataset.
  • Over-optimization of the proxy reward produces high predicted aesthetic scores but low output diversity, so practical deployment needs a cap on optimization or a diversity-preserving regularizer.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A stronger test than the paper's own would be a cross-renderer listening study: the same MIDI rendered by a different soundfont might not show the 1.22-point gain if the reward scores are partly timbre-specific.
  • Because the reward model's receptive field is 10 seconds, the method rewards local surface qualities and cannot capture long-form musical development; a natural extension is to score longer contexts or combine the reward with structure-sensitive features.
  • The 14-participant panel was drawn from one computer-science department, so the effect size is best read as a proof-of-concept; a preregistered multi-demographic study could estimate a more general effect.
  • The over-optimization curve suggests a reward-diversity trade-off frontier; a future system could tune the KL penalty or add diversity bonuses to navigate it, analogous to entropy bonuses used elsewhere in reinforcement learning.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

2 major / 5 minor

Summary. The paper investigates whether an audio-domain aesthetic reward model can be used to fine-tune a symbolic music generation system. The authors train a small Transformer on piano MIDI from MetaMIDI, then apply GRPO with the Meta Audiobox Aesthetics (MAA) 'content enjoyment' score of a 10-second audio rendering as the reward, calling the intervention SMART. They report that SMART training increases the MAA scores of generated outputs, changes several MIDI-level features (more notes, more polyphony, fewer empty beats, lower velocities), and, in a listening study with 14 participants, yields significantly higher enjoyability ratings (mixed-effects model estimate +1.22, p = 0.002). They also show that longer or unregularized optimization collapses output diversity. The main evidence for the subjective improvement is a small, convenience-sampled listening study analyzed with a linear mixed-effects model.

Significance. If the subjective-effect claim holds, the paper provides a useful, relatively simple recipe for aligning a symbolic music generator with an audio-based aesthetic model, and it is one of the few works in this space that validates the reward against an external human listener benchmark rather than only the reward model itself. The over-optimization analysis is a clear illustration of Goodhart's law in this setting, and the authors are transparent about several limitations (MAA's 10-second window, unknown rater demographics, small convenience sample). However, because the central claim depends on the statistical model, the current analysis needs correction before the result can be considered established.

major comments (2)
  1. [Section 4.3, Table 5] The text in Section 4.3 states that random intercepts and slopes were included for the stimulus group and the subject group, but Table 5 reports only a single grouping factor ('id', 14 groups) with random effects for the intercept and the SMART slope. No prompt-level random effect appears. Because each participant rated matched base/SMART excerpts from the same 15 prompts, observations within a prompt are correlated; omitting prompt as a random effect can inflate the precision of the fixed effect. Please re-fit the model with a prompt-level random intercept and slope (e.g., (1 + system | prompt) + (1 + system | id)), report the variance components, and indicate whether the SMART effect remains significant. If the prompt random effects were omitted deliberately, justify that choice; if they were included but not shown, correct the table.
  2. [Section 4.3] The exclusion of 8 observations from 4 prompts that produced silent base-model outputs is disclosed, but the paper does not consider its effect on the conclusion. The resulting analysis supports only an effect conditional on outputs being non-silent; it does not establish that SMART improves the full distribution of base-model outputs, including the failure cases one might expect the intervention to address. Please report the number of silent outputs per condition, run a sensitivity analysis that retains the excluded observations (e.g., assigning a low rating to silence) or fits a model with an exclusion term, and discuss how the estimate and p-value change.
minor comments (5)
  1. [Section 4.2] 'Using 1000 procedurally, we generate' is missing a noun; it should read 'Using 1000 procedurally generated prompts, we generate'.
  2. [Section 3.1] The sentence '10% of the dataset is left as a holdout test set, while the remainder is used for test and validation with a 90-10 ratio' is ambiguous; presumably the remainder is split into training and validation sets.
  3. [Section 5.2, Figure 6] The labels '=0.04' and '=0.00' should be written as 'β = 0.04' and 'β = 0.00'.
  4. [Section 4.3] The listening-study description should state whether all participants rated the same audio files or different samples from the same prompts, since this affects the correlation structure and the choice of random effects.
  5. [Section 4.3] The paper would benefit from reporting residual diagnostics for the mixed-effects model, for example a QQ plot or a plot of residuals versus fitted values.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the SMART claim is tested against external human ratings, not against the reward used for optimization.

full rationale

Walking the derivation chain, no load-bearing step reduces to its own input by construction, and no self-citation chain is invoked to force the result. The base model is trained on MetaMIDI piano data; SMART finetuning uses GRPO with Meta Audiobox Aesthetics content enjoyment as reward (Sec. 3.2-3.5). The MAA-score increase in Sec. 4.1 is an expected consequence of optimizing that exact reward, and the paper explicitly labels it as such ('SMART training is able to optimize the Content Enjoyment reward'), so it is not presented as an independent prediction. The paper's main claim, improved subjective enjoyability, is evaluated by a listening study with human participants (Sec. 4.3), which is external to the MAA reward and to the training objective. The linear mixed-effects model is fit after data collection to estimate the intervention effect; its coefficients are not used to construct the method or to define the reward, so the p = 0.002 result is not a fitted input masquerading as a prediction. The paper's own discussion acknowledges MAA limitations, and the ethics statement is transparent about rater and cultural bias. No uniqueness theorem or prior work by the same authors is invoked to forbid alternatives. The skeptic's concerns about omitted prompt-level random effects and silent-output exclusions are statistical robustness issues, not circularity: even if the analysis were re-run with different exclusions or a fuller model, the claim would not become definitionally equivalent to its inputs. Overall, the derivation is self-contained with respect to its central external benchmark, so circularity is absent.

Assumptions & free parameters 6 free parameters · 4 assumptions · 0 invented entities

The central claim rests on the proxy validity of MAA, the rendering pipeline, and standard RL and statistical assumptions. No invented entities are introduced. Hyperparameters and design choices (KL beta, training length, prompt distribution, soundfont, exclusion rule, temperature) are listed as free parameters because they shape the reported outcomes.

free parameters (6)
  • KL regularization coefficient beta = 0.04
    Chosen by hand to balance reward optimization against drift from the reference model; also run at 0.0 in the over-optimization experiment (Sections 3.5 and 5.2).
  • Number of SMART training iterations = 200 (and 1000 for aggressive runs)
    Training length is a design choice; the paper shows longer runs increase MAA scores but collapse diversity (Section 5.2).
  • Prompt generation procedure = Random tempo/time-signature tokens or prompts sampled from the pretraining set
    The prompt distribution drives the domain shift discussed in Section 5.3 and can affect the optimization trajectory.
  • Soundfont selection = Yamaha C5 Salamander-JNv5_1
    Chosen among five soundfonts partly because it produced high MAA ratings (Table 2); this choice affects the audio render and therefore the reward signal.
  • Listening study exclusion rule = 8 observations from 4 silent-output prompts removed
    The exclusion is disclosed and likely conservative, but it is a data-dependent choice that changes the estimated effect size (Section 4.3).
  • Sampling temperature = 1.0
    Sampling temperature affects generation diversity and reward optimization; fixed at 1.0 for all runs.
assumptions (4)
  • domain assumption Meta Audiobox Aesthetics provides a reward signal that correlates with human aesthetic judgment of music.
    The entire SMART training loop relies on MAA ratings as a suitable proxy for how much listeners enjoy generated music. The paper acknowledges MAA limitations in Section 3.3.
  • domain assumption TinySoundFont rendering with the selected Yamaha soundfont preserves the musical qualities that matter for aesthetic judgment.
    MIDI is converted to audio with a fixed soundfont before reward scoring; if the renderer injects artifacts or masks expression, the reward targets rendering artifacts rather than musical content. See Section 3.4.
  • standard math GRPO and the pretrained causal Transformer provide a stable enough optimization that observed improvements are due to the reward rather than training instability.
    The paper treats GRPO as reliable prior machinery (Shao et al., 2024) and does not test convergence robustness across seeds.
  • standard math Linear mixed-effects model assumptions hold for the listening-study ratings.
    The 1-10 enjoyment ratings are treated as a continuous response with normal residuals in lmer (Section 4.3, Table 5); no residual diagnostics are shown.

how reviews work

0 comments
Cite this review

Pith. "Pith review of SMART: Tuning a symbolic music generation system with an audio domain aesthetic reward." pith.science (2026). https://pith.science/paper/JGIQ5XKF

@misc{pith2026250416839,
  author       = {Pith},
  title        = {Pith review of: SMART: Tuning a symbolic music generation system with an audio domain aesthetic reward},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/JGIQ5XKF}},
  note         = {Machine review of arXiv:2504.16839}
}
abstract

Recent work has proposed training machine learning models to predict aesthetic ratings for music audio. Our work explores whether such models can be used to finetune a symbolic music generation system with reinforcement learning, and what effect this has on the system outputs. To test this, we use group relative policy optimization to finetune a piano MIDI model with Meta Audiobox Aesthetics ratings of audio-rendered outputs as the reward. We find that this optimization has effects on multiple low-level features of the generated outputs, and improves the average subjective ratings in a preliminary listening study with $14$ participants. We also find that over-optimization dramatically reduces diversity of model outputs.

Figures

Figures reproduced from arXiv: 2504.16839 by the authors.

Figure 1
Figure 1. [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Predicted content enjoyment rating from MAA across SMART training iterations. [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. shows that SMART training is able to optimize the Content Enjoyment reward. We also notice increases in the MAA ratings we did not optimize for. 2 4 6 8 10 Predicted Content Enjoyment 0 250 500 750 1000 Number of Tracks base SMART 2 4 6 8 10 Predicted Content Usefulness 0 250 500 750 1000 Number of Tracks base SMART 2 4 6 8 10 Predicted Production Complexity 0 250 500 750 1000 Number of Tracks base SMART 2 4 6 8 10 … view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: Optimizing towards the aesthetic reward affects distribution of low-level features in model outputs. Histograms of various track and note features from 1000 generations with random procedural prompts for the piano model before and after SMART training. Velocity range: …
Figure 5
Figure 5. Figure 5: On the left: Overall distribution of the ratings. Red indicates the base model, blue indicates the post-intervention model. On the right: Distribution of the ratings from each subject. To further consolidate the results, we fit a linear mixed effect model y = Xβ + Zu +…
Figure 6
Figure 6. Figure 6: Aggressive optimization of models towards the aesthetic reward results in high rewards but destroys diversity in outputs. The distribution of Content Enjoyment scores (top) and average piano rolls from first 16 beats (bottom) computed on 1000 samples from the base mode…
Figure 7
Figure 7. Figure 7: MAA ratings and MIDI features for 1000 outputs from the base model and after 200 iterations of SMART training. *Prompts extracted from the pretraining dataset. 5.3 What if we use prompts from the pretraining set instead? One consequence of the procedural prompts used d…

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. From Aesthetics to Human Preferences: Comparative Perspectives of Evaluating Text-to-Music Systems

    eess.AS 2025-04 conditional novelty 4.0 of 10

    Across five text-to-music models, aesthetic predictor scores, pairwise human preferences, and reference-based distribution metrics produce inconsistent rankings, so the choice of evaluation metric changes the winner.

Reference graph

Works this paper leans on

25 extracted references · 15 canonical work pages · cited by 1 Pith paper

  1. [1]

    Bao, J., Behl, H., et al. (2024). Phi-3 technical report: A highly capable language model locally on your phone. arXiv preprint arXiv:2404.14219

    arXiv 2024

  2. [2]

    Bruford, F., McDonald, S., and Sandler, M. (2020). jaki: user-controllable generation of drum patterns using an lstm encoder-decoder and deep reinforcement learning. In The 2020 Joint Conference on AI Music Creativity

    work page 2020

  3. [3]

    Cideron, G., Girgin, S., Verzetti, M., Vincent, D., Kastelic, M., Borsos, Z., McWilliams, B., Ungure- anu, V ., Bachem, O., Pietquin, O., et al. (2024). Musicrl: Aligning music generation to human preferences. arXiv preprint arXiv:2402.04229

    arXiv 2024

  4. [4]

    Colton, S., Bradshaw, L., Banar, B., and Bhandari, K. (2024). Automatic generation of expressive piano miniatures. In Grace, K., Llano, M. T., Martins, P., and Hedblom, M. M., editors,Proceedings of the 15th International Conference on Computational Creativity, ICCC 2024, Jönköping, Sweden, June 17-21, 2024, pages 59–63. Association for Computational Crea...

    work page 2024

  5. [5]

    Dong, H.-W., Chen, K., McAuley, J., and Berg-Kirkpatrick, T. (2020). Muspy: A toolkit for symbolic music generation. In Proceedings of the 21st International Society for Music Information Retrieval Conference (ISMIR)

    work page 2020

  6. [6]

    and Pasquier, P

    Ens, J. and Pasquier, P. (2021). Building the metamidi dataset: Linking symbolic and audio musical data. In ISMIR, pages 182–188

    work page 2021

  7. [7]

    Fradet, N., Briot, J.-P., Chhel, F., El Fallah Seghrouchni, A., and Gutowski, N. (2021). MidiTok: A python package for MIDI file tokenization. In Extended Abstracts for the Late-Breaking Demo Session of the 22nd International Society for Music Information Retrieval Conference

    work page 2021

  8. [8]

    and Crestel, L

    Hadjeres, G. and Crestel, L. (2021). The piano inpainting application. arXiv preprint arXiv:2107.05944

    arXiv 2021

Show all 25 references
  1. [9]

    Hawthorne, C., Huang, A., Ippolito, D., and Eck, D. (2018). Transformer-nade for piano performances. In NIPS 2nd workshop on machine learning for creativity and design

    2018

  2. [10]

    D., Dinculescu, M., and Eck, D

    Hoffman, M. D., Dinculescu, M., and Eck, D. (2019). Music transformer. In International Conference on Learning Representations. 9

    2019

  3. [11]

    E., and Eck, D

    Jaques, N., Gu, S., Turner, R. E., and Eck, D. (2017). Tuning recurrent neural networks with reinforcement learning. 5th International Conference on Learning Representations ICLR2017

    2017

  4. [12]

    Karwowski, J., Hayman, O., Bai, X., Kiendlhofer, K., Griffin, C., and Skalse, J. M. V . (2024). Goodhart’s law in reinforcement learning. In The Twelfth International Conference on Learning Representations

    2024

  5. [13]

    Kilgour, K., Zuluaga, M., Roblek, D., and Sharifi, M. (2019). Fréchet audio distance: A reference-free metric for evaluating music enhancement algorithms. pages 2350–2354

    2019

  6. [14]

    Kirk, R., Mediratta, I., Nalmpantis, C., Luketina, J., Hambro, E., Grefenstette, E., and Raileanu, R. (2024). Understanding the effects of RLHF on LLM generalisation and diversity. In The Twelfth International Conference on Learning Representations

    2024

  7. [15]

    Kotecha, N. (2018). Bach2bach: generating music using a deep reinforcement learning approach. arXiv preprint arXiv:1812.01060

    2018 arXiv

  8. [16]

    Lacoste, A., Luccioni, A., Schmidt, V ., and Dandres, T. (2019). Quantifying the carbon emissions of machine learning. arXiv preprint arXiv:1910.09700

    2019 arXiv

  9. [17]

    Chen, H., Feng, T., Zhao, T., Liu, C.-H., Song, X., Li, Y ., and Zhou, Y . (2025). Analyzable chain-of-musical-thought prompting for high-fidelity music generation

    2025

  10. [18]

    Lan, Q., Tørresen, J., and Jensenius, A. R. (2019). Raveforce: A deep reinforcement learning environment for music. In Proceedings of the SMC conferences, pages 217–222. Society for Sound and Music Computing

    2019

  11. [19]

    D., Ermon, S., and Finn, C

    Rafailov, R., Sharma, A., Mitchell, E., Manning, C. D., Ermon, S., and Finn, C. (2023). Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36:53728–53741

    2023

  12. [20]

    Schulman, J., Wolski, F., Dhariwal, P., Radford, A., and Klimov, O. (2017). Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347

    2017 arXiv

  13. [21]

    Shao, Z., Wang, P., Zhu, Q., Xu, R., Song, J., Bi, X., Zhang, H., Zhang, M., Li, Y ., Wu, Y ., et al. (2024). Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300

    2024 arXiv

  14. [22]

    and Oore, S

    Simon, I. and Oore, S. (2017). Performance rnn: Generating music with expressive timing and dynamics. https://magenta.tensorflow.org/performance-rnn

    2017

  15. [23]

    Tjandra, A., Wu, Y .-C., Guo, B., Hoffman, J., Ellis, B., Vyas, A., Shi, B., Chen, S., Le, M., Zacharov, N., et al. (2025). Meta audiobox aesthetics: Unified automatic quality assessment for speech, music, and sound. arXiv preprint arXiv:2502.05139. von Rütte, D., Biggio, L., ...

    2025 arXiv

  16. [24]

    Wang, Y ., Wu, S., Hu, J., Du, X., Peng, Y ., Huang, Y ., Fan, S., Li, X., Yu, F., and Sun, M. (2025). Notagen: Advancing musicality in symbolic music generation with large language model training paradigms. arXiv preprint arXiv:2502.18008

    2025 arXiv

  17. [25]

    Wu, S., Guo, Z., Yuan, R., Jiang, J., Doh, S., Xia, G., Nam, J., Li, X., Yu, F., and Sun, M. (2025). Clamp 3: Universal music information retrieval across unaligned modalities and unseen languages. arXiv preprint arXiv:2502.10362. 7 Appendix 10 Table 3: Soundfonts used in expe...

    2025 arXiv

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.