{"id":"f53e7958-60a4-408e-8b8c-0a24ab7b936f","arxiv_id":"2608.02831","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":8,"one_line_summary":"AudioRubrics uses evolving, audio-grounded rubric rewards from a powerful judge model to improve reinforcement learning for audio reasoning, beating baselines on MMAU, MMAR, and MMSU.","lead":"AudioRubrics trains audio-language models with reinforcement learning rewards that are generated on the fly by an external judge model, using rubrics that adapt to each audio clip and to the model's current weaknesses. The method reports gains over strong baselines on three audio reasoning benchmarks, but the improvements are modest and depend on having a very capable judge model.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Rubric judge is shown each rollout's correctness before scoring, so the process reward may simply leak the outcome; without a blinded-judge control, the central mechanism claim is unproven.","rationale":"The paper is otherwise coherent: the ablations are consistent, the reward decomposition is clearly specified, and the reported gains are plausible. The reader's concern about judge quality is real but secondary; even with a strong judge, the judge's access to per-rollout correctness labels means the process reward could encode the outcome rather than audio-grounded reasoning quality. This is a specific, testable confound that directly targets the claimed mechanism rather than the general reliability of LLM judges. The missing control is cheap and decisive, so conditional acceptance remains appropriate until the blinded-judge experiment is run. Secondary issues that also support conditionality include the absence of error bars or multiple seeds, selection of gamma and delta on the evaluation benchmarks, and the unexplained reference_score < 0.5 constraint in the static rubric prompt; these reinforce the need for caution but are not the main attack.","tokens_in":20296,"tokens_out":10138,"duration_ms":99336,"concrete_test":"Run the full AudioRubrics pipeline with a blinded judge: in the Call-1 prompt (Figure 11), remove all correctness labels and final-answer letters from the candidate response blocks and from the per-rollout correctness summary, or split judging into a separate call that receives only the audio, question, rubrics, and anonymized reasoning traces. Keep all other hyperparameters fixed and evaluate on MMAU Test-mini, MMAR, and MMSU. If the blinded variant retains most of the reported gains (78.00/65.80/65.86 vs GRPO 75.20/62.20/63.14), the process-reward claim survives; if it collapses toward or below GRPO, the reported advantage is attributable to outcome leakage rather than evolving rubrics.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The most load-bearing weakness is that the rubric reward is not independent of the outcome reward. In the evolving-rubric judging prompt (Figure 11), the judge receives, for every rollout, its extracted final answer and a correctness label ('T1 [final answer ... -> CORRECT / WRONG]' plus a 'Per-rollout correctness summary'), and the same call then returns binary judgments for all rubrics. The rubric judgments are therefore made with full knowledge of which rollouts are correct. Since the variance filter then keeps rubrics whose verdicts best separate the rollouts, the retained rubrics are likely to be those that track outcome correctness, so R_rub can act as a soft correctness classifier rather than an audio-grounded process signal. This undermines the central claim that the gains come from evolving, audio-grounded process supervision: the improvement could be an artifact of leaking outcome information into the process reward. Figure 3 already shows the method is fragile to the judge; the unblinded judging is a more specific confound because it does not require judge incompetence.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes AudioRubrics, a reinforcement-learning framework for audio reasoning in which a frozen audio-capable model (Gemini-3.1-Pro) first generates per-question rubrics from the raw waveform and then, at each GRPO step, proposes new rubrics from the current policy rollouts, judges all rubrics on those rollouts, discards nondiscriminative criteria, and reweights the survivors. The resulting rubric score is combined with accuracy/format rewards and a length penalty. Experiments on MMAU, MMAR, and MMSU with a Qwen2.5-Omni-7B base report accuracy above all similar-size baselines, with ablations attributing the gains to RL training, static rubrics, rubric evolution, and the overthinking penalty, plus an analysis of response-length stability and rubric evolution over training.","tokens_in":20547,"tokens_out":5796,"duration_ms":54108,"significance":"If the effects are real and the mechanism is as claimed, the paper offers a practical way to supply dense, adaptive process supervision for audio reasoning without hand-designing criteria, and the qualitative examples of rubric co-evolution are instructive. The paper's strengths include evaluation across three benchmarks with a wide baseline set, transparent appendix prompts, an ablation table, sensitivity tables for reward weights, and a training-length analysis showing that the full method avoids both collapse and explosion. However, the central attribution of the gains to audio-grounded process supervision is not yet supported by the evidence presented.","major_comments":[{"comment":"The judge call that elicits new rubrics and returns binary judgments also receives the ground-truth answer, each rollout's extracted final answer with a CORRECT/WRONG label, and a per-rollout correctness summary. Because the variance filter then keeps rubrics whose verdicts best separate the rollouts, the retained rubric reward can act as a soft proxy for final-answer correctness rather than as an independent audio-grounded process signal. Table 6 only ablates the presence of the rubric reward, not the judge's access to outcome labels; a blinded control in which the judge scores the same rubrics without any correctness information, or in a separate label-free call, is needed to support the paper's central claim that the gains come from evolving, audio-grounded process supervision. As written, the 'audio-grounded process reward' interpretation is confounded with outcome leakage.","section":"§3.2, Figure 11, Algorithm 1"},{"comment":"The final hyperparameters γ=0.5 and δ=0.15 are selected from sensitivity sweeps run on the same three evaluation benchmarks used for the headline results, and Table 6 reports single-run accuracies without error bars or multiple seeds. Since several reported gains are small (e.g., 65.86 vs 65.44 on MMSU in the ablation), the ranking could be affected by seed noise, and tuning on the test benchmarks can inflate absolute numbers. Please report selection on a held-out split and means/standard deviations over at least three seeds, or otherwise justify that the reported differences exceed run-to-run variance.","section":"§4.3, Tables 4–5, Table 7"},{"comment":"The static rubric generation prompt imposes an unexplained constraint that the reference answer's satisfied rubric weight be less than 0.5. This is not described in the method section, and since the reference answer is the ground-truth response y* used in Eq. (5), the constraint may bias the initial rubrics toward criteria that the correct answer does not satisfy. Please state the motivation for this constraint, report its effect, and test sensitivity to the threshold.","section":"Figure 9 and §3.2"}],"minor_comments":[{"comment":"As printed, Eq. (1) defines R(o) as a sum over all G rollouts of importance-weighted rewards, then subtracts the group mean and divides by the group standard deviation; this is not the GRPO objective, and the per-response reward and advantage notation is inconsistent. Please rewrite so that r(o_i) is the per-rollout reward and A_i = (r(o_i) - mean)/std.","section":"§2.1, Eq. (1)"},{"comment":"The horizontal axis is labeled only 'values' with tick marks 0, 0.1, 0.3, 0.5, 0.7, 0.9; it should be labeled γ, and the caption should state whether the y-axis is averaged over the three benchmarks or refers to a selected one.","section":"Figure 3(a)"},{"comment":"The 'ratio of newly evolved rubrics adopted' is not defined; please state whether it is the fraction of kept rubrics coming from R_new after the variance filter and the top-M selection.","section":"§4.5"},{"comment":"The column headers for MMSU and MMAU appear to be misaligned, with the 'Seman./Phono./Para./Avg' group repeated without a clear separator between the two benchmarks; please reformat the table.","section":"Table 2"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is positioned as an audio-specific contribution, but the evolving-rubric mechanism is largely adapted from recent text- and vision-domain rubric-RL work. The distinguishing audio-grounded aspect is exactly what the unblinded-judge confound leaves unproven; if the authors cannot show that blinding does not remove the gains, the novelty relative to that literature is substantially diminished. I would also ask the editor to ensure that code, checkpoints, and seed-level results are provided, since the paper's reproducibility claims are prominent."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a competent transfer of the evolving-rubric idea to audio reasoning, with sensible design and consistent benchmark gains. The soft spot is not minor: the same call that produces rubric verdicts also reveals each rollout's correctness to the judge. The supposedly process-level reward may therefore be a soft, learned re-expression of the outcome reward, not an independent audio-grounded signal.\n\nWhat is genuinely new: the first audio-grounded, self-evolving rubric reward, with raw-waveform input to the generator and a fixed taxonomy of evaluation dimensions. The adaptation of variance filtering, re-weighting, and overthinking penalty from the text/vision rubric literature is done thoughtfully, and the ablations in Table 6 support each component's contribution. Using Gemini-3.1-Pro as the judge is also handled honestly: Figure 3 shows performance drops below the GRPO baseline with GPT-audio-1.5, which is a useful fragility result.\n\nThe paper's own prompts undermine the central claim. Figure 11 shows the evolving-rubric judging prompt includes the ground-truth answer, per-rollout final answers, a CORRECT/WRONG label, and a per-rollout correctness summary. The judge is then asked to generate new rubrics and judge all rubrics against all rollouts in that same prompt. Since the variance filter keeps the rubrics whose verdicts best separate rollouts, the retained rubrics are likely the ones that track correctness. The instruction that correctness is \"context only\" does not remove the confound. A blinded-judge control, where rubrics are generated or judged without correctness labels, is needed before the paper can claim audio-grounded process supervision rather than outcome leakage.\n\nOther weaknesses are proportionate. There are no error bars or multiple seeds; gamma, delta, and L are tuned on the evaluation benchmarks; and the reference_score < 0.5 constraint in Figure 9 is unexplained. These are fixable. The dependence on a proprietary judge is a practical limitation but not a rejection-level flaw.\n\nWho is this for? People working on RLVR for audio reasoning and rubric-based rewards generally. It is a solid experimental paper with a clear confound in its central mechanism. I would send it to serious peer review, but the reviewers should ask for a blinded-judge ablation or an analysis showing rubric judgments are not explained by outcome correctness. If that control fails, the paper still has value as a cautionary result about judge leakage in rubric rewards.","headline":"The audio-domain adaptation is real and well-executed, but the rubric judge sees ground-truth correctness before scoring, so the central 'process-level supervision' claim is not yet established.","tokens_in":21070,"tokens_out":1857,"would_cite":false,"duration_ms":18288,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"AudioRubrics claims that self-evolving, audio-grounded rubric rewards let a 7B audio-language model beat every comparable baseline on three audio-reasoning benchmarks.","keywords":["audio reasoning","reinforcement learning","evolving rubrics","process rewards","large audio-language models","GRPO","audio-grounded evaluation","reward design"],"falsifier":"Retrain AudioRubrics with the same judge and rubric loop but feed the judge only a text transcript, not the raw waveform; if the reported MMAU/MMAR/MMSU accuracies (78.00/65.80/65.86) do not drop, the audio-grounding story is falsified. Alternatively, on a held-out set of rollouts, compare judge verdicts on the rubrics against human expert verdicts: near-chance agreement would show the reward signal is noise.","tokens_in":20114,"feed_emoji":"🎧","tokens_out":7562,"duration_ms":59148,"temperature":0.7,"pith_summary":"AudioRubrics claims that the reward signal for training audio-language models can be made far more informative by replacing a single outcome check with a set of audio-grounded rubrics—binary criteria with weights—that are regenerated from the model's own rollouts as training proceeds. The paper argues that outcome-only reinforcement learning lets the model reach correct answers without genuinely listening, while fixed hand-written process criteria saturate once the policy passes them and never adapt to the question at hand. It reports that a 7B model trained this way outperforms all comparable open and training-based baselines on MMAU, MMAR, and MMSU, and that the advantage grows with the capability of the rubric generator and judge. A careful reader would care because it suggests a concrete recipe for keeping RL supervision dense and non-saturating in a sensory domain.","feed_headline":"Evolving audio rubrics push 7B model past audio-reasoning baselines","feed_subtitle":"Self-updating, audio-grounded rubrics keep the RL signal from saturating; results beat every similar-size baseline on three benchmarks.","key_machinery":"Rubric-as-reward: a per-sample set of binary criteria $r_k$ with weights $w_k$ summing to 1, where a judge model $\\Phi$ that hears the raw audio returns $b_{k,i}\\in\\{0,1\\}$ for whether rollout $i$ satisfies criterion $k$, producing the rubric reward $R^{\\mathrm{rub}}_i=\\sum_{k\\in\\mathcal{K}} w_k b_{k,i}$. The distinctive machinery is the evolution step: $\\Phi$ sees the group of $G=8$ rollouts, proposes up to $N_{\\mathrm{new}}=3$ new positive or negative rubrics that separate strong from weak answers, prunes rubrics with zero variance across the group, keeps the top $M=5$ most discriminative, and reweights them. The final per-rollout reward is $R_i=R^{\\mathrm{out}}_i+\\gamma R^{\\mathrm{rub}}_i+\\delta R^{\\mathrm{over}}_i$, where $R^{\\mathrm{out}}$ is accuracy plus format and $R^{\\mathrm{over}}=1-|o_i|/L$ penalizes overthinking; GRPO then optimizes the policy.","core_discovery":"On the paper's own terms, the discovery is that self-evolving, audio-grounded rubric rewards make a 7B audio-language model reach state-of-the-art results among similar-size models: 78.00% on MMAU Test-mini, 65.80% on MMAR, and 65.86% on MMSU average, against 75.20%, 62.20%, and 63.14% for the GRPO baseline trained with outcome rewards alone. The rubrics are initialized per question from the raw waveform, then at each RL step an audio-capable judge both scores the current rollouts and elicits new discriminative criteria, dropping any criterion all rollouts either pass or fail, and re-weighting the survivors. The paper also shows that the reward signal is what drives the gain: ablating evolution, static rubrics, or the length penalty each lowers accuracy, and swapping the judge for a weaker model drops performance below the vanilla GRPO baseline.","pith_inferences":["This suggests the same evolving-rubric loop could be dropped onto other sensory domains—video, or audio-plus-video—wherever a strong multimodal judge can ground criteria in the raw input; the audio-specific taxonomy is a design choice, not a prerequisite.","A natural extension the paper does not test is ensembling several judges or training a small reward model on judge verdicts, which would reduce the single-judge failure mode visible in its Figure 3(b).","The unexplained constraint in the static prompt that the reference answer must satisfy less than half the rubric weight probably biases early rubrics upward in difficulty; varying that threshold could change early-training dynamics and is worth a targeted ablation.","Because the variance filter keeps only rubrics that discriminate among just 8 rollouts, some kept criteria may separate by chance early in training; increasing the group size or applying a statistical significance threshold could sharpen the reward signal."],"forward_implications":["If the central claim holds, outcome-only RLVR for audio reasoning can be improved by process supervision without hand-designing reward functions per question: the rubrics are generated automatically from the audio and the rollouts.","The reported ablation quantifies each component: evolving rubrics add roughly two points over static rubrics on MMAU (76.20 vs 78.00 full), and removing the length penalty costs about one point, so the gains are not an artifact of a single term.","The judge-capability dependence means practitioners adopting the method should expect gains only when the judge/generator is strong; with a weaker judge the method can underperform plain GRPO.","The convergence to a stable reasoning length—contrasting with GRPO's collapse and the unbounded growth without the penalty—suggests the reward design also controls inference cost at deployment.","The improvement transfers to the perception-oriented MMSU split (66.14 vs 60.16 for CESAR, the best comparable baseline), which the paper reads as evidence that the model is actually listening rather than pattern-matching text."],"supporting_citations":[{"why":"Defines the MMAU benchmark used for evaluation.","marker":"(Sakshi et al., 2024)"},{"why":"Defines the MMAR benchmark used for evaluation.","marker":"(Ma et al., 2025b)"},{"why":"Defines the MMSU benchmark used for evaluation.","marker":"(Wang et al., 2025)"},{"why":"Supplies the GRPO algorithm and KL regularization that AudioRubrics builds on.","marker":"(Guo et al., 2025)"},{"why":"Provides the R1-AQA outcome-only RLVR baseline and the AVQA-derived training-data recipe.","marker":"(Li et al., 2025)"},{"why":"Provides the Omni-R1 baseline that supervises only the final answer.","marker":"(Rouditchenko et al., 2025)"},{"why":"Provides the CESAR process-reward baseline with fixed criteria that AudioRubrics extends.","marker":"(Fan et al., 2025)"},{"why":"Supplies the Qwen2.5-Omni-7B base model used as the policy.","marker":"(Xu et al., 2025)"},{"why":"Supplies the AVQA dataset from which the 40,176 training samples are constructed.","marker":"(Yang et al., 2022)"}],"fun_headline_variants":["Self-evolving audio rubrics beat static rewards in audio reasoning","Evolving audio-grounded rubrics push 7B model to top scores","RL with adaptive rubrics keeps rewards fresh, beats fixed baselines","Audio rubrics that adapt per question improve reasoning beyond GRPO"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that one external judge model (Gemini-3.1-Pro) can be trusted, throughout training, to hear the audio, generate meaningful rubrics, and grade every rollout correctly; the entire process reward flows through this judge without verification.","fun_headline_variants_meta":{"raw":{"variants":["Self-evolving audio rubrics beat static rewards in audio reasoning","Evolving audio-grounded rubrics push 7B model to top scores","RL with adaptive rubrics keeps rewards fresh, beats fixed baselines","Audio rubrics that adapt per question improve reasoning beyond GRPO"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000521,"raw_usage":{"total_tokens":2580,"prompt_tokens":1059,"completion_tokens":1521,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":675,"completion_tokens_details":{"reasoning_tokens":1447}},"tokens_in":675,"tokens_out":1521,"duration_ms":11154,"temperature":1.0,"reasoning_tokens":1447,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T14:57:53.096620+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Retrain AudioRubrics with the same judge and rubric loop but feed the judge only a text transcript, not the raw waveform; if the reported MMAU/MMAR/MMSU accuracies (78.00/65.80/65.86) do not drop, the audio-grounding story is falsified. Alternatively, on a held-out set of rollouts, compare judge verdicts on the rubrics against human expert verdicts: near-chance agreement would show the reward signal is noise.","supporting_citations":[],"review_version":2}