{"id":"fe3a722f-153e-4153-8017-736b9b68cffa","arxiv_id":"2603.09714","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"LALMs degrade sharply with more concurrent audio inputs on the new MUGEN benchmark; Audio-Permutational Self-Consistency plus CoT recovers up to 6.74% accuracy without training.","lead":"Current large audio-language models get much worse when they must reason over several audio clips at once; a new benchmark called MUGEN measures this failure across speech, sound, and music. A simple training-free trick—trying many orderings of the same clips and aggregating answers—recovers several points of accuracy.","discovery_kind":"new_method","skeptic_critique":{"model":"grok-4.5","headline":"No significant objection identified beyond the reader's already-flagged confound risk; APSC gains remain the softest empirical claim under corrupted text.","rationale":"The Reader's strongest claim matches the abstract exactly, and the weakest assumption (task design may not isolate multi-audio fusion) is precisely the condition that must hold for both the bottleneck diagnosis and the APSC gains to be attributed correctly. Recoverable text does not reveal a deeper flaw (e.g., circular metric, unstated train/test leakage, or contradictory numbers). Encoding corruption prevents table-level audit, which the Reader already treats as medium risk and CONDITIONAL acceptance pending artifacts. No new load-bearing attack is warranted; the concrete length-matched + text-only permutation control is the natural next verification the Reader's concern already implies. Verdict and confidence stay as stated.","tokens_in":12500,"tokens_out":462,"duration_ms":4833,"concrete_test":"On a clean release of MUGEN + code: (1) re-run the main multi-audio accuracy curve while holding total audio tokens fixed (pad/truncate single-audio baselines to match multi-audio length) and (2) compare APSC aggregation against a non-audio multi-choice control that only permutes text options. If degradation vanishes under length-matched controls or APSC gains appear equally on pure-text multi-choice, the multi-audio attribution weakens; otherwise the claim holds.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim (sharp multi-audio degradation + APSC/CoT gains of 6.28–6.74%) rests on MUGEN isolating multi-audio fusion rather than confounds (context length, multi-choice format, order bias). The Reader already names this weakest assumption correctly. The supplied manuscript is heavily encoding-corrupted, so tables, exact model list, statistical tests, and ablations cannot be audited; no stronger internal contradiction or hidden assumption is visible in the recoverable abstract/claims. APSC is a plausible order-diversification fix for known LALM position sensitivity, and the gains are modest. Thus the load-bearing concern does not escalate beyond the Reader's medium correctness risk from incomplete verifiability.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.5","summary":"The paper introduces MUGEN, a benchmark for multi-audio understanding in large audio-language models (LALMs) spanning speech, general audio, and music. Experiments indicate consistent weaknesses under multi-audio inputs, with accuracy degrading sharply as the number of concurrent audio streams increases, which the authors interpret as an input-scaling bottleneck. As a training-free mitigation they propose Audio-Permutational Self-Consistency (APSC), which aggregates model answers over diversified orderings of audio candidates and reports gains of up to 6.28% accuracy; combining APSC with Chain-of-Thought raises the reported improvement to 6.74%. The work positions MUGEN as a foundation for evaluating complex auditory comprehension and APSC as a practical inference-time remedy for order sensitivity.","tokens_in":12631,"tokens_out":1092,"duration_ms":16875,"significance":"Multi-audio fusion (e.g., multi-speaker, multi-source, or mixed speech–music scenes) is practically important and comparatively under-benchmarked relative to multi-image or multi-video settings. A domain-spanning evaluation suite plus a simple, training-free aggregation heuristic would be useful community resources if the measured degradation and APSC gains can be attributed to multi-audio understanding rather than generic context-length or multi-choice artifacts. The reported gains are modest but the identification of input scaling as a bottleneck is a clear, falsifiable claim that can guide architecture and training work. Strengths include an explicit training-free protocol and a concrete accuracy delta that can be re-checked once full tables and code are available.","major_comments":[{"comment":"Central attribution claim (input scaling as multi-audio bottleneck; APSC gains of 6.28–6.74%): the manuscript must demonstrate that MUGEN isolates multi-audio fusion rather than confounds such as total audio duration / token budget, multi-choice format, or prompt-order bias. Without matched single-stream controls that hold context length fixed while varying the number of distinct sources, and without ablations that randomize or fix candidate order outside APSC, the degradation curve and APSC improvements cannot be cleanly attributed to multi-audio understanding. Please add (or clearly report) these controls in the experimental section and main results tables.","section":"Benchmark construction / Experiments (MUGEN protocol & APSC)"},{"comment":"Reported accuracy gains (up to 6.28% APSC; 6.74% APSC+CoT) need statistical grounding: number of seeds or bootstrap intervals, per-model and per-domain breakdowns, and whether the aggregation rule (majority vote vs. other) is fixed a priori. Absent error bars or significance tests, it is unclear whether the gains are robust or driven by a subset of models/tasks. Please report variance and the exact aggregation rule used for the headline numbers.","section":"Results (APSC / APSC+CoT)"},{"comment":"How concurrent audio is encoded and presented to the LALM (separate audio tokens with explicit separators, temporal concatenation, interleaved embeddings, etc.) is load-bearing for both the degradation claim and APSC. The evaluation protocol should state this encoding, any length truncation policy, and whether models that natively support multi-audio differ systematically from those that do not. Without this, input-scaling results are hard to interpret or reproduce.","section":"Evaluation protocol / Model setup"}],"minor_comments":[{"comment":"Define APSC formally (permutation set size, sampling vs. full enumeration, aggregation function) in one place so the free parameters of the method are unambiguous.","section":"Method (Audio-Permutational Self-Consistency)"},{"comment":"Clarify the maximum concurrent-audio schedule and task difficulty construction for MUGEN so that “sharp degradation” can be plotted against a well-specified x-axis (number of streams, total duration, or both).","section":"MUGEN benchmark"},{"comment":"Ensure model list, checkpoint versions, and decoding settings (temperature, max tokens) are fully specified for reproducibility of the multi-audio and APSC runs.","section":"Experimental setup"},{"comment":"Abstract and conclusion use “fundamental bottleneck”; consider softer language unless the controlled ablations above establish that the failure is not primarily context-length or formatting.","section":"Abstract / Conclusion"}],"recommendation":"major_revision","confidential_remarks":"The supplied full-text encoding in the review packet is heavily corrupted, so tables, exact model lists, and statistical tests could not be audited line-by-line; the report therefore rests on the abstract, recoverable structure, and the stated claims. If the camera-ready manuscript already contains the requested context-length and order controls with error bars, the revision burden may be lighter than it appears from the garbled source. Scope fit for a sound/ML venue is appropriate; novelty is empirical (benchmark + inference heuristic) rather than architectural."},"author_rebuttal":null,"desk_editor":{"model":"grok-4.5","letter":"This is a useful empirical paper for people working on large audio-language models. The punchline is simple: current LALMs get worse as you feed them more concurrent audio clips, and a cheap order-permutation aggregation (APSC), alone or with CoT, recovers a few points of accuracy on their new multi-domain suite.\n\nWhat is actually new is MUGEN itself—a multi-audio benchmark spanning speech, general audio, and music—and the explicit framing of input scaling as a bottleneck rather than a side note. That diagnosis matches what many of us have seen informally and is worth having in a shared eval. APSC is not a deep algorithmic invention; it is a natural audio-side adaptation of self-consistency / order diversification already used in text and multi-choice settings. The reported gains (up to ~6.3%, ~6.7% with CoT) are modest and therefore more believable than dramatic claims would be. The paper does the right thing by staying training-free and product-relevant.\n\nSoft spots, in proportion: the load-bearing assumption is that MUGEN’s concurrent-candidate design isolates multi-audio fusion rather than longer context, multi-choice format, or prompt-order bias. The reader and stress-test both flag this correctly; it does not kill the paper, but any referee will want ablations that hold total context roughly fixed and that separate order sensitivity from true multi-source reasoning. Free knobs (how many permutations, how you aggregate, how hard the multi-input schedule is) should be reported cleanly. The source we have is badly encoding-corrupted, so tables, model lists, and significance cannot be fully audited here—that is a review-process problem, not an internal contradiction in the claims.\n\nWho this is for: LALM evaluation and audio multimodal people who need a shared multi-clip stress test and a baseline inference trick. It will not reorganize core ML theory. Math is not the point; data and protocol are. Citation pattern looks standard for the subfield.\n\nI would send it to peer review rather than desk-reject. Engage if multi-audio robustness is on your roadmap; skim the abstract and APSC section if it is not. Clean artifact release and confound checks would make the contribution tighter, but the central story already holds up as a solid benchmark-plus-mitigation paper.","headline":"Solid multi-audio LALM benchmark with a clear input-scaling diagnosis and a modest, usable permutation fix; main risk is confounds, not the core story.","tokens_in":13353,"tokens_out":587,"would_cite":true,"duration_ms":11106,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.5","headline":"Large audio-language models weaken sharply as concurrent audio inputs increase; reordering candidates and aggregating recovers up to 6.74% accuracy.","keywords":["multi-audio understanding","large audio-language models","MUGEN","Audio-Permutational Self-Consistency","input scaling","speech","music","general audio"],"falsifier":"Evaluate the same models on single-audio MUGEN items whose context length is padded with silence or noise to match the multi-audio setting; if the sharp accuracy drop disappears, the claimed multi-audio bottleneck is not isolated.","tokens_in":13349,"feed_emoji":"🎧","tokens_out":739,"duration_ms":15642,"temperature":0.7,"pith_summary":"The paper argues that multi-audio understanding is essential for large audio-language models yet remains a clear weak spot. It introduces MUGEN, a benchmark spanning speech, general audio, and music, and shows that accuracy falls sharply as the number of simultaneous audio streams grows, treating input scaling as a fundamental bottleneck. Training-free Audio-Permutational Self-Consistency, which presents the same candidates in different orders and aggregates the answers, raises accuracy by as much as 6.28 percent; pairing it with chain-of-thought reasoning reaches 6.74 percent. A sympathetic reader cares because real listening almost never involves a single isolated stream, so models that cannot fuse multiple sources cannot support reliable scene understanding or multi-talker applications.","feed_headline":"Audio models lose accuracy as concurrent inputs rise","feed_subtitle":"MUGEN shows the drop; reordering candidates and aggregating recovers up to 6.74%.","key_machinery":"Audio-Permutational Self-Consistency (APSC): a training-free procedure that varies the presentation order of the same audio candidates, collects the model’s answers under each order, and aggregates them into a more stable final prediction; the paper uses this mechanism to obtain the reported accuracy gains.","core_discovery":"Current large audio-language models show consistent multi-audio weaknesses on the MUGEN benchmark across speech, general audio, and music. Performance degrades sharply as the number of concurrent audio inputs rises, identifying input scaling as a fundamental bottleneck. Audio-Permutational Self-Consistency, which diversifies the order of audio candidates and aggregates predictions, yields up to 6.28% accuracy gains; combining it with Chain-of-Thought further improves performance to 6.74%. These results expose blind spots in present models and supply a foundation for evaluating complex auditory comprehension.","pith_inferences":["Architectures that explicitly model relations among concurrent streams, rather than simple concatenation, may be needed before scaling alone closes the gap.","The same order sensitivity could appear in multi-image or multi-video language models, pointing to a broader multi-modal fusion issue.","Benchmark construction should systematically control pure context-length effects to confirm that measured failures are truly multi-stream."],"forward_implications":["Future LALM evaluations must include multi-stream stress tests rather than single-audio suites alone.","Applications that require simultaneous speech, environmental sound, and music will remain limited until input-scaling behavior improves.","Training-free order permutation and aggregation can be adopted immediately to raise multi-choice multi-audio accuracy without retraining.","Pairing order diversification with explicit reasoning steps further mitigates the observed bottleneck."],"fun_headline_variants":["MUGEN shows LALMs lose accuracy as concurrent audio inputs rise","Multi-audio performance drops sharply with more simultaneous inputs","Permuting audio candidates recovers up to 6.74% multi-audio accuracy","Input scaling is a core bottleneck for large audio-language models","Audio-Permutational Self-Consistency lifts multi-audio scores 6.28%"],"cache_read_input_tokens":128,"weakest_assumption_plain":"MUGEN’s task design and multi-candidate construction truly isolate multi-audio fusion rather than confounds such as longer context, label leakage, or order-sensitive prompting.","fun_headline_variants_meta":{"raw":{"variants":["MUGEN shows LALMs lose accuracy as concurrent audio inputs rise","Multi-audio performance drops sharply with more simultaneous inputs","Permuting audio candidates recovers up to 6.74% multi-audio accuracy","Input scaling is a core bottleneck for large audio-language models","Audio-Permutational Self-Consistency lifts multi-audio scores 6.28%"]},"model":"grok-4.5","effort":"low","cost_usd":0.004704,"raw_usage":{"total_tokens":1334,"prompt_tokens":729,"num_sources_used":0,"completion_tokens":98,"cost_in_usd_ticks":47040000,"prompt_tokens_details":{"text_tokens":729,"audio_tokens":0,"image_tokens":0,"cached_tokens":256},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":507,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":729,"tokens_out":98,"duration_ms":3891,"temperature":1.0,"reasoning_tokens":507,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-07-15T00:04:19.918369+00:00","model_set":{"reader":"grok-4.5"},"falsifier":"Evaluate the same models on single-audio MUGEN items whose context length is padded with silence or noise to match the multi-audio setting; if the sharp accuracy drop disappears, the claimed multi-audio bottleneck is not isolated.","supporting_citations":[],"review_version":1}