{"id":"e145a353-fad1-4f93-8341-52b60e3a29fe","arxiv_id":"2504.16839","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"SMART uses an audio aesthetic reward to fine-tune a symbolic MIDI piano model, raising human enjoyment ratings by about 1.2 points in a small listening study while over-optimization collapses output diversity.","lead":"The authors use an audio aesthetic-rating model to fine-tune a piano MIDI generator with reinforcement learning, then test the output with 14 human listeners. The tuned system's music was rated more enjoyable on average, but over-optimization produced repetitive outputs.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Listening-test significance rests on an incomplete mixed model: prompt-level random effects are absent from Table 5 and silent-output exclusions may inflate the SMART effect.","rationale":"The reader's weakest assumption focuses on MAA reward transfer, but the listening test is the direct empirical check of that transfer. The more immediate load-bearing issue is whether the listening-test analysis itself supports the claimed improvement. The reported lmer model appears to omit the prompt/stimulus random effects that the text says were included, and the analysis excludes silent base-model outputs before fitting. These are not accusations of misconduct; the paper discloses the exclusions and the small convenience sample. However, the disclosed analysis as written does not yet establish that the 1.22-point effect is robust to prompt-level clustering or to inclusion of silence as an outcome. A re-analysis with the full data and a correct mixed-effects structure would settle this. If the effect survives, the conditional verdict is appropriate; if not, the central claim would need to be substantially weakened. The reader's conditional verdict already flags missing code/data, so the concern reinforces rather than changes the verdict.","tokens_in":10254,"tokens_out":6063,"duration_ms":59629,"concrete_test":"Re-fit the listening-study model using the full 407 ratings (or with silence coded as a separate category) as rating ~ system + (system|participant) + (system|prompt), and also run a prompt-level cluster bootstrap or permutation test. If the SMART effect remains significant with prompt random effects and with silent outputs included, the concern is resolved; if the p-value crosses 0.05 or the effect shrinks materially, the paper's central claim should be weakened.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim ('We see a significant (p = 0.002) positive effect...' Section 4.3) rests entirely on the linear mixed-effects model summarized in Table 5, but the reported model is rating ~ system + (system|id), with 14 subject groups. The surrounding text says random intercepts and slopes were included for 'the stimulus group and for the subject group,' yet no prompt/stimulus random effect appears in the table. Because each of the 15 prompts contributes multiple ratings (and the same prompt is rated under both systems), omitting prompt as a random effect treats correlated residuals as independent, likely understating the SE of the 1.221 SMART coefficient. In addition, 8 observations from 4 prompts are excluded because the base model produced silent outputs. If those are base-model failure modes, dropping them before fitting removes the very cases the intervention is meant to fix, so the reported improvement is conditional not only on the convenience sample but on deleting a class of systematic base-model failures. Either issue alone would warrant re-analysis; together they make the p=0.002 headline more fragile than the paper suggests.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper investigates whether an audio-domain aesthetic reward model can be used to fine-tune a symbolic music generation system. The authors train a small Transformer on piano MIDI from MetaMIDI, then apply GRPO with the Meta Audiobox Aesthetics (MAA) 'content enjoyment' score of a 10-second audio rendering as the reward, calling the intervention SMART. They report that SMART training increases the MAA scores of generated outputs, changes several MIDI-level features (more notes, more polyphony, fewer empty beats, lower velocities), and, in a listening study with 14 participants, yields significantly higher enjoyability ratings (mixed-effects model estimate +1.22, p = 0.002). They also show that longer or unregularized optimization collapses output diversity. The main evidence for the subjective improvement is a small, convenience-sampled listening study analyzed with a linear mixed-effects model.","tokens_in":10464,"tokens_out":7191,"duration_ms":64693,"significance":"If the subjective-effect claim holds, the paper provides a useful, relatively simple recipe for aligning a symbolic music generator with an audio-based aesthetic model, and it is one of the few works in this space that validates the reward against an external human listener benchmark rather than only the reward model itself. The over-optimization analysis is a clear illustration of Goodhart's law in this setting, and the authors are transparent about several limitations (MAA's 10-second window, unknown rater demographics, small convenience sample). However, because the central claim depends on the statistical model, the current analysis needs correction before the result can be considered established.","major_comments":[{"comment":"The text in Section 4.3 states that random intercepts and slopes were included for the stimulus group and the subject group, but Table 5 reports only a single grouping factor ('id', 14 groups) with random effects for the intercept and the SMART slope. No prompt-level random effect appears. Because each participant rated matched base/SMART excerpts from the same 15 prompts, observations within a prompt are correlated; omitting prompt as a random effect can inflate the precision of the fixed effect. Please re-fit the model with a prompt-level random intercept and slope (e.g., (1 + system | prompt) + (1 + system | id)), report the variance components, and indicate whether the SMART effect remains significant. If the prompt random effects were omitted deliberately, justify that choice; if they were included but not shown, correct the table.","section":"Section 4.3, Table 5"},{"comment":"The exclusion of 8 observations from 4 prompts that produced silent base-model outputs is disclosed, but the paper does not consider its effect on the conclusion. The resulting analysis supports only an effect conditional on outputs being non-silent; it does not establish that SMART improves the full distribution of base-model outputs, including the failure cases one might expect the intervention to address. Please report the number of silent outputs per condition, run a sensitivity analysis that retains the excluded observations (e.g., assigning a low rating to silence) or fits a model with an exclusion term, and discuss how the estimate and p-value change.","section":"Section 4.3"}],"minor_comments":[{"comment":"'Using 1000 procedurally, we generate' is missing a noun; it should read 'Using 1000 procedurally generated prompts, we generate'.","section":"Section 4.2"},{"comment":"The sentence '10% of the dataset is left as a holdout test set, while the remainder is used for test and validation with a 90-10 ratio' is ambiguous; presumably the remainder is split into training and validation sets.","section":"Section 3.1"},{"comment":"The labels '=0.04' and '=0.00' should be written as 'β = 0.04' and 'β = 0.00'.","section":"Section 5.2, Figure 6"},{"comment":"The listening-study description should state whether all participants rated the same audio files or different samples from the same prompts, since this affects the correlation structure and the choice of random effects.","section":"Section 4.3"},{"comment":"The paper would benefit from reporting residual diagnostics for the mixed-effects model, for example a QQ plot or a plot of residuals versus fitted values.","section":"Section 4.3"}],"recommendation":"major_revision","confidential_remarks":"The paper does not appear to provide code, raw rating data, or the generated audio files; making these available would strengthen verifiability. The missing prompt random effect in Table 5 is a reporting inconsistency that must be resolved; if the actual model is as reported, the p-value may be optimistic."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The short version: this paper is a genuinely new combination—using an audio-domain aesthetic reward (Meta Audiobox Aesthetics) to fine-tune a symbolic piano MIDI model with GRPO—and it documents a real diversity-collapse effect under aggressive optimization. But the headline listening-study result (p=0.002, ~1.2-point improvement) is fragile because the mixed model is incompletely reported. The text says random intercepts and slopes were included for both stimulus group and subject group, yet Table 5 only shows the subject-group random effects. If prompt-level random effects are missing, the standard error on the SMART coefficient is likely understated. The disclosed exclusion of 8 silent-output observations from the base model also drops exactly the failure mode the intervention is meant to fix; conditioning on non-silent base outputs may inflate the apparent effect.\n\nWhat's new and good: the related-work section convincingly distinguishes this from RaveForce, MusicRL, and NotaGen. Applying an audio aesthetic reward directly to symbolic MIDI generation is novel, and the paper is honest about MAA's limitations and the convenience sample. The low-level feature histograms are a clear, useful description of what changed.\n\nSoft spots: no code or raw data, which blocks verification of the lmer specification and the exact exclusions. The listening study is small and convenience-sampled. No alternative reward baseline (e.g., spectral or symbolic) is included, so we don't know how much of the effect is specific to MAA. The over-optimization analysis is qualitative—mean piano-roll plots only—and would be stronger with a quantitative diversity metric.\n\nWho this is for: people working on RL for generative music, especially symbolic-to-audio systems. The core idea is methodologically sound and deserves referee time, but the statistical evidence needs re-analysis.\n\nRecommendation: send to peer review, with a request for (a) a corrected model table including prompt random effects or a justification for their absence, (b) code/data release, and (c) a sensitivity analysis around the silent-output exclusions. If the p-value survives those, the paper is a solid contribution; if not, the diversity-collapse finding and the method description still carry value.","headline":"A genuinely new combination of audio reward and symbolic MIDI, but the listening-study significance is fragile due to an incompletely reported mixed model.","tokens_in":10982,"tokens_out":3645,"would_cite":false,"duration_ms":31993,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"An audio-domain aesthetic reward can fine-tune symbolic music generation: tuned piano MIDI scored higher with listeners, but over-optimization collapses diversity.","keywords":["symbolic music generation","reinforcement learning","audio aesthetic reward","GRPO","Meta Audiobox Aesthetics","piano MIDI","music evaluation","reward over-optimization"],"falsifier":"Render the same set of base and SMART MIDI outputs with at least two different soundfonts and collect enjoyability ratings from a larger, more diverse panel; if the post-intervention gain disappears or reverses for one renderer, the reward signal was partly an artifact of the specific synth rather than a property of the musical structure.","tokens_in":10041,"feed_emoji":"🎹","tokens_out":6785,"duration_ms":61106,"temperature":0.7,"pith_summary":"This paper asks whether a model trained to predict human aesthetic ratings of music audio can be used as a reward signal to improve a symbolic music generator that outputs piano MIDI. The authors fine-tune a pretrained piano Transformer with Group Relative Policy Optimization, rendering each generated MIDI to 10 seconds of audio, scoring it with the Meta Audiobox Aesthetics content-enjoyment model, and using those scores as rewards. They report that the tuned model produces more notes, more polyphony, fewer empty beats, a wider pitch range, and more dynamic variation, and that a 14-participant listening study rated its outputs about 1.22 points higher on a 10-point enjoyability scale (p = 0.002). They also show that longer, less-regularized optimization keeps raising the proxy score while destroying output diversity. The contribution is a working demonstration that an audio-domain aesthetic reward can be propagated through a renderer into symbolic music generation, with the caveat that the reward is a proxy for genuine aesthetic experience.","feed_headline":"Audio reward tuning lifts listener ratings by 1.22 points","feed_subtitle":"MIDI piano outputs fine-tuned on 10-second audio aesthetics were rated more enjoyable—until over-optimization.","key_machinery":"The load-bearing machinery is a closed loop from symbolic tokens to audio and back: a causal Transformer generates MIDI with a REMI+-style tokenizer; a soundfont renderer converts the MIDI to 10-second audio; Meta Audiobox Aesthetics assigns a content-enjoyment score to that audio; and Group Relative Policy Optimization (GRPO) turns the scores into group-normalized advantages used to update the policy, while a KL divergence term keeps the policy close to the frozen pretrained reference. The renderer is the critical bridge, because it lets an audio-trained reward model shape decisions that are made in symbolic space.","core_discovery":"The central claim is that audio-domain aesthetic preference models can act as effective rewards for tuning symbolic music generation. Concretely, the paper shows that optimizing a piano MIDI model toward Meta Audiobox Aesthetics content-enjoyment ratings of audio-rendered outputs, with a KL constraint keeping the policy near its pretrained reference, yields a model whose outputs receive higher predicted aesthetic scores and are rated as more enjoyable by a convenience-sample listening panel. The 200-step intervention shifts measurable musical features, such as more notes, more polyphony, fewer rests, wider pitch range, and lower and more varied velocities, and the linear mixed-effects analysis estimates a 1.22-point gain in enjoyability (p = 0.002). The paper also demonstrates the known over-optimization failure mode: pushing the same objective for 1000 steps, or with no KL penalty, increases the reward but visibly collapses the variety of outputs.","pith_inferences":["A stronger test than the paper's own would be a cross-renderer listening study: the same MIDI rendered by a different soundfont might not show the 1.22-point gain if the reward scores are partly timbre-specific.","Because the reward model's receptive field is 10 seconds, the method rewards local surface qualities and cannot capture long-form musical development; a natural extension is to score longer contexts or combine the reward with structure-sensitive features.","The 14-participant panel was drawn from one computer-science department, so the effect size is best read as a proof-of-concept; a preregistered multi-demographic study could estimate a more general effect.","The over-optimization curve suggests a reward-diversity trade-off frontier; a future system could tune the KL penalty or add diversity bonuses to navigate it, analogous to entropy bonuses used elsewhere in reinforcement learning."],"forward_implications":["The same SMART pipeline can be applied to other symbolic generators and other audio aesthetic preference models, since the renderer is the only audio-specific component in the loop.","Aesthetic-reward tuning changes musical surface features in predictable directions, such as denser, more polyphonic, and more dynamic output, which suggests the proxy reward shapes perceived musical content rather than only low-level audio quality.","Audio-domain aesthetic prediction offers a reference-free way to evaluate symbolic music generators, using a scalar enjoyment score instead of comparisons against a reference dataset.","Over-optimization of the proxy reward produces high predicted aesthetic scores but low output diversity, so practical deployment needs a cap on optimization or a diversity-preserving regularizer."],"supporting_citations":[{"why":"Supplies the Meta Audiobox Aesthetics model whose content-enjoyment score is the reward signal in SMART.","marker":"Tjandra et al., 2025"},{"why":"Supplies the GRPO objective and training procedure used to fine-tune the policy.","marker":"Shao et al., 2024"},{"why":"Prior demonstration of using an aesthetic preference model as an RL reward for music generation, the audio-generation baseline this work contrasts with by operating on symbolic MIDI.","marker":"Cideron et al., 2024"},{"why":"Provides the architecture of the base causal Transformer model that is fine-tuned.","marker":"Abdin et al., 2024"},{"why":"Defines the REMI+ tokenization and prompt format used for the symbolic music tokens.","marker":"von Rütte et al., 2023"},{"why":"Provides the MetaMIDI dataset used for pretraining the piano model.","marker":"Ens and Pasquier, 2021"},{"why":"Establishes the over-optimization phenomenon that the paper invokes to explain why aggressive proxy-reward pursuit harms the true objective.","marker":"Karwowski et al., 2024"},{"why":"Documents diversity loss under RLHF-style optimization, supporting the paper's over-optimization finding.","marker":"Kirk et al., 2024"}],"fun_headline_variants":["Audio aesthetic reward boosts music enjoyment scores","RL with audio taste improves symbolic music ratings","Tuning MIDI with audio reward raises appeal 1.22","Over-optimizing audio reward collapses music diversity","Audio reward tune-up lifts ratings then kills variety"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The training loop assumes that Meta Audiobox Aesthetics content-enjoyment scores on 10-second audio-rendered piano excerpts are a reliable reward signal that transfers to true human enjoyment of generated music; if the reward mostly captures renderer-specific timbre or short-horizon surface features, the reported improvement may not replicate with other renderers, longer excerpts, or broader listener populations.","fun_headline_variants_meta":{"raw":{"variants":["Audio aesthetic reward boosts music enjoyment scores","RL with audio taste improves symbolic music ratings","Tuning MIDI with audio reward raises appeal 1.22","Over-optimizing audio reward collapses music diversity","Audio reward tune-up lifts ratings then kills variety"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.0002,"raw_usage":{"total_tokens":1320,"prompt_tokens":836,"completion_tokens":484,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":452,"completion_tokens_details":{"reasoning_tokens":412}},"tokens_in":452,"tokens_out":484,"duration_ms":5025,"temperature":1.0,"reasoning_tokens":412,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T10:53:54.902910+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Render the same set of base and SMART MIDI outputs with at least two different soundfonts and collect enjoyability ratings from a larger, more diverse panel; if the post-intervention gain disappears or reverses for one renderer, the reward signal was partly an artifact of the specific synth rather than a property of the musical structure.","supporting_citations":[{"cited_title":"and Pasquier, P","cited_arxiv_id":null,"evidence_quote":"Provides the MetaMIDI dataset used for pretraining the piano model."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Establishes the over-optimization phenomenon that the paper invokes to explain why aggressive proxy-reward pursuit harms the true objective."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Documents diversity loss under RLHF-style optimization, supporting the paper's over-optimization finding."}],"review_version":1}