REVIEW 5 major objections 6 minor 54 references
Music Source Restoration
T0 review · 5 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read The paper establishes Music Source Restoration as a well-posed conditional-generation task: recover unprocessed stems from a degraded sum, constrain them by priors over unprocessed instruments, and evaluate them on the new RawStems dataset.
desk verdict Useful new task and dataset for source restoration, but the 'unprocessed' premise is unverified and the evaluation is closed-loop; worth reviewing with revisions required. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the degraded-mixture model $y = \sum_{i=1}^{n} f_i(s_i)$, where each $f_i$ is drawn from a degradation set of five families (spectral filtering, dynamic range compression, harmonic distortion, reverb, lossy codec) plus gain. The neutrality constraint, enforced by requiring restored stems to match distributions of unprocessed instruments, turns the ill-posed inverse problem into conditional generation. RawStems supplies the priors as training data, and the U-Former baseline—a U-Net with rotary-position attention blocks in the bottleneck—carries the experiments, trained first with L1 reconstruction and then with adversarial and feature-matching losses.
What would settle it
Profile a random sample of RawStems files for traces of prior processing (lossy-codec fingerprints, intersample peaks, EQ-shaped noise floors, or phase correlation between stems) and compare against the original multitracks; finding a substantial share already processed would invalidate the training targets.
Extended reading notes
Core claim
The central claim is that MSR is well-posed once each recovered source is required to be both plausible (some degradation functions from the allowed set can re-produce the mixture) and neutral (each source lies under its prior distribution of unprocessed instruments). The paper supports this by releasing RawStems, a dataset of unprocessed stems with hierarchical instrument categories, and by showing that a single U-Former model trained per stem group can restore degraded mixtures: mel-SSIM scores between 0.40 and 0.60 with SI-SDR often below 0 dB. The authors interpret the gap as task difficulty, not a failure of training, because a stronger pretrained restoration system performs far worse on their vocal test set and extra training steps barely change the guitar models.
Load-bearing premise
The entire setup depends on the RawStems audio files being truly unprocessed original stems and on the five hand-chosen degradation families and their parameter ranges approximating real production chains; if either fails, the training targets or the task itself lose validity.
Editorial extensions
If this is right
- MSR becomes a trainable and evaluable research area with a public dataset, so future systems can be compared on the same degradations and test splits.
- Perceptual metrics such as mel-SSIM become the meaningful success measure for restoration; SI-SDR near or below 0 dB does not by itself disqualify a restored stem.
- Fine-grained per-instrument models outperform a generic group model, since acoustic and electric guitar both beat the general guitar model, supporting class-specific or hierarchical modeling.
- The five-family degradation pipeline with random parameters is enough to produce useful training data, but the paper's convergence and comparison results suggest the remaining gap is inherent task difficulty, not architecture.
- Evaluating restoration through remixing or effect re-application, rather than waveform distance alone, is a plausible next benchmark direction.
Reading between the lines
- A reader can infer that the reported SSIM/SI-SDR gap means waveform-faithful restoration may be the wrong target for creative tools; perceptual and timbral fidelity is what carries musical value, which argues for developing evaluation around listening or downstream mixing tasks rather than dB.
- Because the degradation parameters were chosen heuristically and sampled independently, real production chains with correlated effects are likely harder; a natural extension is to profile actual sessions and train on those chains, or test the current models on third-party stems to see how much the synthetic pipeline transfers.
- The hierarchical taxonomy invites a single conditional model with instrument prompts (for example "guitar:acoustic") instead of nine separate models, and the paper's fine-grained results hint that such conditioning would pay off.
- The fact that even a strong general restoration system fails on the vocal test set suggests this task is not a special case of speech restoration; dedicated MSR architectures that exploit instrument-specific priors may be needed.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The manuscript introduces Music Source Restoration (MSR), defined as recovering original, undegraded instrument signals s_i from mixtures y = sum f_i(s_i) under neutrality priors p_i. It presents RawStems, annotations for 578 songs from the Mixing Secrets multitrack library claimed to contain unprocessed, hierarchically categorized stems; defines five degradation families; and evaluates a U-Former baseline trained on synthetic mixtures generated by applying these degradations. Results show mel-SSIM 0.40-0.60 and SI-SDR near or below 0 dB, which the authors interpret as demonstrating feasibility of the task on their dataset, with code, annotations, and checkpoints released.
Significance. If RawStems indeed contains unprocessed stems, the dataset and task address a real gap in source separation research, and the public release of annotations, the degradation simulation pipeline, and trained models is a useful community asset. The explicit formulation of the many-to-one inverse problem and the discussion of neutrality are conceptually valuable. However, the validity of the central benchmark depends on two unverified premises: the 'unprocessed' status of the library audio and the representativeness of the synthetic degradation pipeline; the evaluation is in-distribution with respect to the training pipeline. These caveats do not destroy the paper's value but make the feasibility claim provisional.
major comments (5)
- [Sec 3.1-3.2] The claim that RawStems contains 'unprocessed source signals' is load-bearing and is asserted rather than verified. The data-collection description only covers LLM-assisted labeling and manual verification of taxonomy; no audio provenance audit, listening check for tracking-stage processing, or metadata inspection is described. Since Sec 4.2 treats the stems as clean targets and applies synthetic degradations on top, any pre-existing EQ, compression, distortion, or room reverb in the Mixing Secrets files would make the training targets themselves 'degraded' under the paper's own definition (Sec 2.1). The closed-loop evaluation cannot detect this. Please provide evidence of the raw status (e.g., checks against known processed artifacts, producer documentation for a sample, or a clear statement that rawness is assumed), or substantially soften the dataset claim.
- [Sec 4.2-4.3] The baseline evaluation is closed-loop: the test mixtures are generated by the same degradation simulation used for training, so the reported SSIM/SI-SDR show only that the model can partially invert its own synthetic degradation distribution. This does not support the broader 'feasibility ... in music production settings' claim without external validation on real recording chains or at least a held-out set generated with different parameter distributions. The authors acknowledge the heuristic nature of the pipeline in Sec 5, but the evaluation section should either include such a generalization test or explicitly limit the feasibility claim to the synthetic setup.
- [Sec 2.3] The evaluation metrics do not implement the task definition. The plausibility condition in Sec 2.1 requires existence of degradation functions f_i in F with sum f_i(s_i) = y, but SI-SDR measures waveform distance to the original source, not satisfaction of this constraint; conversely, mel-SSIM does not verify that the output lies under the neutrality prior p_i. Either the metrics should be re-derived from the two conditions (e.g., a reconstruction check through the pipeline and a prior-likelihood estimate), or the text should state that SI-SDR and SSIM are proxy metrics rather than direct measurements of plausibility and neutrality.
- [Sec 3.1] The count of instrument categories is internally inconsistent: the abstract, Sec 1, and Sec 6 state 8 primary and 17 secondary groups; Sec 3.1 text says 8 first-level and 18 second-level groups; and the bullet list actually enumerates nine primary-level entries (Vocals, Rhythm, Guitars, Keyboards, Orchestra, Synth, Bass, Misc, MiscRoom). Please correct the taxonomy counting and align the abstract and conclusion with the actual taxonomy.
- [Sec 4.2] The degradation pipeline used for training includes elements absent from the formal degradation set F in Sec 2.2: downsampling/resampling, Gaussian noise, and limiter on the mixture are applied in Sec 4.2 but not defined as members of F. Also, the codec degradation is applied only to mixtures, not to target stems, while the task definition allows any f_i from F. Please align the implementation with the formal definition or extend F accordingly.
minor comments (6)
- [Sec 3.1] The phrase 'yielding 17 subcategories' does not match the enumerated list; please provide an explicit count table for the second-level categories.
- [Sec 1] 'BS-Roformers' should be 'BS-RoFormer', and the citation [5] is a mel-band RoFormer, which should be identified precisely when introducing U-Former's RoFormer blocks.
- [Footnote 1] 'correspondance' should be 'correspondence'.
- [Sec 3.1] The sentence beginning 'our findings' should begin with a capital 'O'.
- [Sec 4.4] The VoiceFixer comparison is a speech-restoration model not designed for source separation; its poor SI-SDR is expected and should be interpreted only as a sanity check, not as a competitive baseline.
- [Sec 1 and Sec 6] The claims 'first dataset' and 'largest dataset' should be reconciled and supported with a citation to prior multitrack datasets.
Circularity Check
No significant circularity: the task, dataset, and baseline are defined independently; the synthetic train/test pipeline is a benchmark limitation, not a reduction.
full rationale
The derivation chain is self-contained: Sec 2.1 defines MSR as recovering original sources si from y = sum fi(si) under neutrality priors; Sec 2.2 specifies a concrete degradation set F with independent parameter ranges; Sec 3 collects external multitrack audio from the Mixing Secrets library and annotates it; Sec 4 trains U-Former on synthetically degraded mixtures and evaluates on test mixtures generated by the same pipeline. The reported SSIM and SI-SDR values are empirical outcomes, not quantities forced by the definitions—the model could have failed, and the paper explicitly reports low SI-SDR as evidence of task difficulty. Using the same degradation simulation for training and testing is a standard supervised benchmark design that makes the feasibility claim in-distribution, but this limitation is acknowledged in Sec 5 ('parameter distributions were selected heuristically' and 'Real-world sessions exhibit correlations ... that our independent sampling ignores'). The load-bearing assumption that RawStems tracks are unprocessed is an external data-provenance assertion, not a definitional equivalence: the paper defines unprocessed independently ('similar to direct recordings from a live performance') and then labels the library files as such. If those files contain tracking-stage processing, the dataset would be invalid, but that is an empirical validity risk, not a circular argument. There are no load-bearing self-citations; the architecture combines independent cited components (U-Net, RoFormer, HiFi-GAN) and the only external baseline (VoiceFixer) is an independent model. No step reduces to its own inputs by construction.
Assumptions & free parameters
free parameters (5)
- Degradation parameter ranges =
EQ: -6 to +6 dB; compression threshold -20 to 0 dB, ratio 1.5 to 10; drive 0 to 5 dB; room size 0.1 to 1; wet level…
- Per-effect application probabilities =
0.5 for each effect; 50% Gaussian noise augmentation
- Mixture SNR range =
-5 to 20 dB
- Loss weights lambda_g and lambda_f =
0.001 and 0.01
- Training schedule =
50000 reconstruction steps + 35000 adversarial steps; batch sizes 16 and 4
assumptions (5)
- domain assumption Music production mixtures are sums of individually degraded sources: y = sum of fi(si).
- domain assumption RawStems source files from Mixing Secrets are unprocessed originals.
- domain assumption LLM-assisted annotation plus human verification yields correct hierarchical labels.
- ad hoc to paper A neutrality prior p_i exists and sufficiently constrains the many-to-one inverse mapping.
- domain assumption Mel-SSIM is a valid perceptual or neutrality metric.
invented entities (1)
-
Neutrality prior p_i
Cite this review
Pith. "Pith review of Music Source Restoration." pith.science (2026). https://pith.science/paper/SDGAUF3A
@misc{pith2026250521827,
author = {Pith},
title = {Pith review of: Music Source Restoration},
year = {2026},
howpublished = {\url{https://pith.science/paper/SDGAUF3A}},
note = {Machine review of arXiv:2505.21827}
}
read the original abstract
We introduce Music Source Restoration (MSR), a novel task addressing the gap between idealized source separation and real-world music production. Current Music Source Separation (MSS) approaches assume mixtures are simple sums of sources, ignoring signal degradations employed during music production like equalization, compression, and reverb. MSR models mixtures as degraded sums of individually degraded sources, with the goal of recovering original, undegraded signals. Due to the lack of data for MSR, we present RawStems, a dataset annotation of 578 songs with unprocessed source signals organized into 8 primary and 17 secondary instrument groups, totaling 354.13 hours. To the best of our knowledge, RawStems is the first dataset that contains unprocessed music stems with hierarchical categories. We consider spectral filtering, dynamic range compression, harmonic distortion, reverb and lossy codec as possible degradations, and establish U-Former as a baseline method, demonstrating the feasibility of MSR on our dataset. We release the RawStems dataset annotations, degradation simulation pipeline, training code and pre-trained models to be publicly available.
Reference graph
Works this paper leans on
-
[1]
Koffka, Principles of Gestalt psychology
K. Koffka, Principles of Gestalt psychology . routledge, 2013
work page 2013
-
[2]
Open-Unmix-a reference implementation for music source separation,
F.-R. St¨oter et al., “Open-Unmix-a reference implementation for music source separation,” JOSS, vol. 4, no. 41, p. 1667, 2019
work page 2019
-
[3]
Music source separation in the waveform domain,
A. D ´efossez et al., “Music source separation in the waveform domain,” arXiv, 2019
work page 2019
-
[4]
Music source separation with band-split RoPE transformer,
W.-T. Lu et al., “Music source separation with band-split RoPE transformer,” in IEEE ICASSP, 2024, pp. 481–485
work page 2024
-
[5]
Mel-band roformer for music source separa- tion,
J.-C. Wang et al., “Mel-band roformer for music source separa- tion,” arXiv, 2023
work page 2023
-
[6]
Wave-U-Net: A multi-scale neural network for end-to-end audio source separation,
D. Stoller et al. , “Wave-U-Net: A multi-scale neural network for end-to-end audio source separation,” in ISMIR, 2018
work page 2018
-
[7]
Conv-TasNet: Surpassing ideal time–frequency magnitude masking for speech separation,
Y . Luoet al., “Conv-TasNet: Surpassing ideal time–frequency magnitude masking for speech separation,” TASLP, vol. 27, no. 8, pp. 1256–1266, 2019
work page 2019
-
[8]
Time-domain audio source separation with neural networks based on multiresolution analysis,
T. Nakamura et al., “Time-domain audio source separation with neural networks based on multiresolution analysis,” TASLP, vol. 29, pp. 1687–1701, 2021
work page 2021
Show all 54 references
-
[9]
Improving music source separation based on deep neural networks through data augmentation and network blending,
S. Uhlich et al., “Improving music source separation based on deep neural networks through data augmentation and network blending,” in IEEE ICASSP, 2017, pp. 261–265
2017
-
[10]
Joint optimization of masks and deep recurrent neural networks for monaural source separation,
P.-S. Huang et al. , “Joint optimization of masks and deep recurrent neural networks for monaural source separation,” TASLP, vol. 23, no. 12, pp. 2136–2147, 2015
2015
-
[11]
MMDenseLSTM: An efficient combination of convolutional and recurrent neural networks for audio source separation,
N. Takahashi et al., “MMDenseLSTM: An efficient combination of convolutional and recurrent neural networks for audio source separation,” in IWAENC, 2018, pp. 106–110
2018
-
[12]
Singing voice separation with deep U-Net convolutional networks,
A. Jansson et al., “Singing voice separation with deep U-Net convolutional networks,” in ISMIR, 2017
2017
-
[13]
Spleeter: a fast and efficient music source separation tool with pre-trained models,
R. Hennequin et al., “Spleeter: a fast and efficient music source separation tool with pre-trained models,” JOSS, vol. 5, no. 50, p. 2154, 2020
2020
-
[14]
Music source separation with band-split rnn,
Y . Luo et al. , “Music source separation with band-split rnn,” IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 31, pp. 1893–1901, 2023
1901
-
[15]
Hybrid spectrogram and waveform source separa- tion,
A. D ´efossez, “Hybrid spectrogram and waveform source separa- tion,” in ISMIR Workshop, 2021
2021
-
[16]
Hybrid transformers for music source separa- tion,
S. Rouard et al., “Hybrid transformers for music source separa- tion,” in ICASSP, 2023
2023
-
[17]
Watkinson, Art of Digital Audio
J. Watkinson, Art of Digital Audio . Routledge, 2013
2013
-
[18]
Z ¨olzer, Digital Audio Signal Processing
U. Z ¨olzer, Digital Audio Signal Processing . John Wiley & Sons, 2022
2022
-
[19]
The acoustics of sound recording rooms,
P. E. Sabine, “The acoustics of sound recording rooms,” Transac- tions of the Society of Motion Picture Engineers , vol. 12, no. 35, pp. 809–822, 1928
1928
-
[20]
Reproducibility of voice parameters: The effect of room acoustics and microphones,
P. Bottalico et al. , “Reproducibility of voice parameters: The effect of room acoustics and microphones,” Journal of Voice , vol. 34, no. 3, pp. 320–334, 2020
2020
-
[21]
Corbett, Mic it!: Microphones, Microphone techniques, and Their Impact on the Final Mix
I. Corbett, Mic it!: Microphones, Microphone techniques, and Their Impact on the Final Mix . Routledge, 2020
2020
-
[22]
Comparison of acoustic voice features derived from mobile devices and studio microphone recordings,
V . S. Fahedet al., “Comparison of acoustic voice features derived from mobile devices and studio microphone recordings,” Journal of Voice, 2022
2022
-
[23]
Senior, Mixing Secrets for the Small Studio
M. Senior, Mixing Secrets for the Small Studio . Routledge, 2018
2018
-
[24]
Shelvock, Audio Mastering as Musical Practice
M. Shelvock, Audio Mastering as Musical Practice . The University of Western Ontario (Canada), 2012
2012
-
[25]
Communication protocols and technologies for multimedia transmission: A comprehensive study,
M. R. Islam, “Communication protocols and technologies for multimedia transmission: A comprehensive study,” Journal of Innovative Technology Convergence, vol. 4, no. 1, 2022
2022
-
[26]
Image quality assessment: from Error Visibility to Structural Similarity,
Z. Wang et al., “Image quality assessment: from Error Visibility to Structural Similarity,” IEEE Transactions on Image Processing, vol. 13, no. 4, pp. 600–612, 2004
2004
-
[27]
What you hear is what you see: Audio quality metrics from image quality metrics,
T. Namgyal et al. , “What you hear is what you see: Audio quality metrics from image quality metrics,” arXiv, 2023
2023
-
[28]
SDR–half-baked or well done?
J. Le Roux et al., “SDR–half-baked or well done?” in ICASSP. IEEE, 2019, pp. 626–630
2019
-
[29]
MUSDB18-HQ-an uncompressed version of MUSDB18,
Z. Rafii et al. , “MUSDB18-HQ-an uncompressed version of MUSDB18,” 2019
2019
-
[30]
MoisesDB: A dataset for source separation beyond 4-stems,
I. Pereira et al. , “MoisesDB: A dataset for source separation beyond 4-stems,” arXiv, 2023
2023
-
[31]
Izhaki, Mixing Audio: Concepts, Practices, and Tools
R. Izhaki, Mixing Audio: Concepts, Practices, and Tools . Rout- ledge, 2017
2017
-
[32]
Design aspects of graphic equalizers,
R. A. Greiner et al. , “Design aspects of graphic equalizers,” JAES, vol. 31, no. 6, pp. 394–407, 1983
1983
-
[33]
Parametric equalization,
G. Massenburg, “Parametric equalization,” in AES Convention
-
[34]
J. O. Smith, Introduction to Digital Filters: with Audio Applica- tions. Julius Smith, 2007, vol. 2
2007
-
[35]
scipy/scipy: Scipy 1.15. 0,
R. Gommers et al., “scipy/scipy: Scipy 1.15. 0,” Zenodo, 2024
2024
-
[36]
Digital dynamic range compressor de- sign—a tutorial and analysis,
D. Giannoulis et al. , “Digital dynamic range compressor de- sign—a tutorial and analysis,” JAES, vol. 60, no. 6, pp. 399–408, 2012
2012
-
[37]
Dynamic range control of digital audio signals,
G. W. McNally, “Dynamic range control of digital audio signals,” JAES, vol. 32, no. 5, pp. 316–327, 1984
1984
-
[38]
Pedalboard,
P. Sobot, “Pedalboard,” Jul. 2021. [Online]. Available: https://doi.org/10.5281/zenodo.7817838
2021 doi
-
[39]
Audio nonlinear modeling through hyper- bolic tangent functionals,
A. Schuck Jr et al., “Audio nonlinear modeling through hyper- bolic tangent functionals,” in DAFx, 2016, pp. 5–9
2016
-
[40]
The development of audio software with distortion,
E. Tarr, “The development of audio software with distortion,” in Distortion in Music Production . Focal Press, 2023, pp. 13–27
2023
-
[41]
McGuire et al., “Mixing,” in The Art of Digital Orchestration
S. McGuire et al., “Mixing,” in The Art of Digital Orchestration . Focal Press, 2020, pp. 166–208
2020
-
[42]
Physical audio signal processing: For virtual musical instruments and audio effects,
J. O. Smith III, “Physical audio signal processing: For virtual musical instruments and audio effects,” 2010
2010
-
[43]
Apollo: Band-sequence modeling for high-quality audio restoration,
K. Li et al., “Apollo: Band-sequence modeling for high-quality audio restoration,” in IEEE ICASSP, 2025
2025
-
[44]
MP3 decoder in theory and practice,
P. Sripada, “MP3 decoder in theory and practice,” 2006
2006
-
[45]
The 2018 signal separation evaluation campaign,
F.-R. St ¨oter et al. , “The 2018 signal separation evaluation campaign,” in LVA/ICA. Springer, 2018, pp. 293–305
2018
-
[46]
Large-scale contrastive language-audio pretraining with feature fusion and keyword-to-caption augmentation,
Y . Wuet al., “Large-scale contrastive language-audio pretraining with feature fusion and keyword-to-caption augmentation,” in ICASSP, 2023
2023
-
[47]
Learning audio concepts from counterfactual natural language,
A. V osoughiet al., “Learning audio concepts from counterfactual natural language,” in IEEE ICASSP, 2024, pp. 366–370
2024
-
[48]
Towards efficient models for real-time deep noise suppression,
S. Braun et al. , “Towards efficient models for real-time deep noise suppression,” in IEEE ICASSP, 2021, pp. 656–660
2021
-
[49]
DTT-Net: Dual-domain translation trans- former for semi-supervised image deraining,
Z.-B. Chen et al. , “DTT-Net: Dual-domain translation trans- former for semi-supervised image deraining,” in IEEE ICIP , 2022, pp. 1621–1625
2022
-
[50]
Benchmarking music demixing models for deep drum source separation,
A. I. Mezza et al., “Benchmarking music demixing models for deep drum source separation,” in IEEE International Symposium on the Internet of Sounds (IS2) , 2024, pp. 1–6
2024
-
[51]
High-fidelity audio compression with improved rvqgan,
R. Kumar et al., “High-fidelity audio compression with improved rvqgan,” NeurIPS, vol. 36, pp. 27 980–27 993, 2023
2023
-
[52]
V oicefixer: Toward general speech restoration with neural vocoder,
H. Liu et al. , “V oicefixer: Toward general speech restoration with neural vocoder,” arXiv preprint arXiv:2109.13731 , 2021
2021 arXiv
-
[53]
Godsill et al., Digital audio restoration
S. Godsill et al., Digital audio restoration . Springer, 2002
2002
-
[54]
Separate what you describe: Language-queried audio source separation,
X. Liu et al., “Separate what you describe: Language-queried audio source separation,” arXiv preprint arXiv:2203.15147, 2022
2022 arXiv
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.