Pith. sign in

REVIEW 3 major objections 5 minor 27 references

Interleaved Multitask Learning for Audio Source Separation with Independent Databases

T0 review · 3 major / 5 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A single shared-encoder network, trained by alternating mini-batches from per-source databases, separates vocals, drums, and bass with less interference than simultaneous multitask training, using half the parameters.

desk verdict A useful shared-encoder/interleaved-training idea whose headline SIR comparison is confounded by different data splits and order; worth a careful control experiment. read the letter →

arxiv 1908.05182 v1 pith:E3PJUU2C submitted 2019-08-14 cs.SD cs.LGeess.AS

Clement S. J. Doire , Olumide Okubadejo This is my paper · ORCID
classification cs.SDcs.LGeess.AS
keywords audiosourceseparationmultitasklearninginterleavedtrainingsharedencoderindependentdatabasesmusicdeepneuralnetworksMUSDB18
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper proposes a way to train one audio source separation network when the training data for each source lives in separate databases that do not all contain every stem. Instead of optimizing all four separation objectives at once, it alternates mini-batches from each per-source database through a shared encoder and source-specific decoders. The central quantitative claim is that this interleaved procedure yields higher Source-to-Interference Ratios than simultaneous multitask training on three of four MUSDB18 sources, while using roughly half the learnable parameters of four independently trained networks. If the claim holds, it matters because the constraint that every training mixture have ground truth for all sources disappears, so separation training data can be enlarged source by source.

What carries the argument

The load-bearing mechanism is the interleaved training loop, not a new layer type. At each epoch, for the number of batches in the smallest source database, the loop samples one mini-batch from vocals, drums, bass, and other, back-propagates each sub-task's L1 spectrogram loss, and updates the shared encoder plus the corresponding decoder. A variant, accumulated-encoder interleaved training, sums the encoder gradients over the whole four-source group before updating the encoder once. The shared encoder is a U-Net-style stack with five strided-convolution downsampling stages widening from 16 to 256 feature maps, and each of the four decoders upsamples by nearest-neighbour interpolation and skip-concatenates encoder features before feature-map averaging and a ReLU. The alternation is meant to disentangle the shared distribution from task-specific distributions, while the accumulation variant tests whether removing per-mini-batch stochasticity at the encoder hurts that disentangling.

What would settle it

Train the simultaneous method on the same per-source databases with the same shuffled order and the same 75/15 train/validation split as the interleaved runs, holding all other settings fixed; if the interleaved SIR advantage over simultaneous training on vocals, drums, and bass disappears or reverses, the claimed benefit is an artifact of data ordering rather than of interleaving.

Watch

Extended reading notes

Core claim

The paper's central claim is that the shared-encoder architecture $\hat{S}_i = D_{S_i}(E(Y;\theta_E); \theta_{D_{S_i}})$ can be trained by interleaving per-source mini-batches rather than by minimizing the summed loss of all four objectives at once, and that doing so changes what the shared representation learns. On the MUSDB18 test set, interleaved training gives SIR 8.92 dB for vocals, 7.25 dB for drums, 3.80 dB for bass, and 2.04 dB for other, against 7.62, 6.74, 3.65, and 2.48 for simultaneous training, while SDR scores stay comparable. Widening the encoder's top layers from 16 to 32 feature maps (about 50,000 extra parameters) lifts SIR to 9.28, 7.43, 4.37, and 2.25, beating independent networks on drums, bass, and other. The authors interpret this as evidence that interleaving forces the encoder to keep a joint representation that serves all sub-tasks, while each decoder learns to disentangle its own source; the wider encoder helps most on the ill-defined 'other' stem.

Load-bearing premise

The load-bearing premise is that the interleaving procedure itself, not the different random train/validation splits and shuffled data order used to build the independent databases (stated in Section 4.2), is what produces the reported SIR improvements.

Editorial extensions

If this is right

  • Per-source training databases no longer have to be aligned: any mixture with a labelled vocals, drums, bass, or other stem can contribute to the corresponding sub-task, so the data pool can grow independently for each source.
  • Interleaved training beats simultaneous multitask training on SIR for vocals, drums, and bass even when all four sources are available, so the benefit is not only about missing labels.
  • The shared-encoder model needs roughly 3.5 million parameters against about 8 million for four separate networks, and a modest width increase closes the gap with independent models on SIR.
  • Because the encoder is shared, a trained model can be extended to a new source objective by adding a new decoder and fine-tuning, reusing the learned representation as a starting point.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • An editor's inference: the paper's equal-data setup probably understates the practical advantage of independent databases; if per-source databases are enlarged, interleaving should benefit more than simultaneous training, which cannot use such data.
  • The paper notes in Section 4.2 that the independent and simultaneous databases use different random train/validation splits and shuffled orders; a control run with identical splits is needed before attributing the SIR differences to interleaving itself.
  • A direct test of the mechanism would vary the interleaving schedule or the per-source batch ratio: if SIR tracks the schedule, interleaving is doing the work; if not, the random subset sampling is the active ingredient.
  • The 'other' stem behaves differently in every comparison, which suggests the shared encoder helps least for a target defined by exclusion; a natural extension is to test interleaving with more granular targets such as guitar or piano.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes a shared-encoder, per-source-decoder architecture for music source separation and an interleaved training procedure in which mini-batches from source-specific databases are alternated, so that each training sample need not have ground truth for all sources. The authors compare four protocols on MUSDB18: independent per-source networks, simultaneous multitask training, interleaved training, and interleaved training with accumulated encoder gradients. The main reported findings are that interleaved training yields higher Source-to-Interference Ratios than simultaneous training on three of four sources (Table 2), and that modestly increasing encoder width further closes the gap to independently trained models (Table 3), while using fewer total parameters. The paper frames the contribution as enabling training on independent databases and a transferable shared encoder.

Significance. If the main comparison were clean, the paper would make a useful engineering contribution: it shows a practical way to train a multitask separation model without requiring all source stems for every mixture, and it documents a concrete training-procedure effect on SIR. The architecture and pseudocode are clearly described, and the evaluation uses a standard public dataset with standard BSS metrics. However, the load-bearing quantitative claim, that interleaved training outperforms simultaneous training, is confounded by differences in the data setup, and no uncertainty estimates are provided. These issues do not invalidate the feasibility of the shared-encoder architecture or the potential value of interleaved training, but they mean the central empirical comparison needs to be re-established under controlled conditions before the paper's conclusions can be accepted.

major comments (3)
  1. [Section 4.2 / Table 2] The comparison between Interleaved/Interleaved_acc and Simultaneous does not hold the data-generation procedure constant. Section 4.2 states that the four independent databases use 'a different random training/validation split' and a shuffled order of data pairs, while the simultaneous database is built from the same MUSDB18 set without those per-source re-splits and reorderings. Consequently, the reported SIR differences (e.g., vocals 8.92 vs. 7.62) conflate the training procedure with differences in the train/validation partition and the sample order seen by the optimizer. To support the claim that interleaving itself improves SIR, the authors should compare Interleaved and Simultaneous using identical train/validation splits and identical data ordering, or otherwise run both procedures on each setup and verify that the advantage is robust.
  2. [Section 4.3 / Table 2] No repeated runs, error bars, or significance tests are reported, yet Section 4.3 describes the interleaved training as leading to 'significantly higher SIR scores.' Several of the differences in Table 2 are small (e.g., bass SIR 3.80 vs. 3.65, other SIR 2.04 vs. 2.48), and without variance estimates across seeds or train/validation splits it is not possible to tell which differences are meaningful. The authors should report mean and standard deviation over multiple training runs, or at least provide per-song score distributions, for the central comparisons in Tables 2 and 3.
  3. [Section 5] The conclusion states that the proposed method achieves 'state-of-the-art performance compared to separate networks trained independently on each task,' but no external baseline architectures or published methods are evaluated in the paper. Section 4.3 explicitly says the authors 'did not compare it against other baseline architectures.' The 'state-of-the-art' phrasing is unsupported; at most the experiments show results competitive with the paper's own independent and simultaneous baselines. The claim should be removed or replaced with a claim that is scoped to the internal comparisons actually performed.
minor comments (5)
  1. [Section 4.2] The four per-source databases are re-splits and re-shuffles of the same MUSDB18 training set, not genuinely independent collections. The text acknowledges this with 'This is projected to simulate the case of unavailability,' but the term 'independent databases' is used throughout and could be misleading; I suggest consistently referring to them as 'simulated independent databases' or stating explicitly that true independence is not tested.
  2. [Section 4.3] The displayed simultaneous loss is written as 'L = ∑ L_Si' but the subscript on L_Si is missing in the text; it should read L = sum over S_i in S of L_{S_i}.
  3. [Table 3] The table caption reads 'Average song SDR score in dB' but the table reports both SDR and SIR; the caption should be updated to mention both metrics.
  4. [Section 4.4 / Table 3] The paper reports that encoder+ has approximately 50,000 additional parameters but does not give the exact total parameter count for encoder+ or for the independent baselines. Since the efficiency claim depends on parameter counts, providing exact figures would make the comparison more transparent.
  5. [Section 4.2] The early-stopping criterion is described as 30 epochs without improvement on the validation set, but the total number of epochs or the number of training batches is not reported. Reporting training length would help reproducibility and would clarify how the interleaved and simultaneous procedures compare in wall-clock cost.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the paper reports an empirical comparison of training procedures, with no fitted-input-as-prediction loop and no self-citation chain.

full rationale

The paper's claims are empirical: a shared-encoder architecture is trained under independent, simultaneous, and interleaved protocols and scored on the MUSDB18 test set. There is no derivation in which an input quantity is defined in terms of the predicted quantity, and no parameter fitted to a subset is later reported as a prediction of a closely related target. The architecture (Section 2) and losses (Eq. 5) are stated independently of the results they produce, and the comparison in Tables 2 and 3 evaluates held-out test performance. The only identifiable weakness is experimental confounding, not circularity: Section 4.2 states that the independent databases use a different random train/validation split and shuffled order than the simultaneous database, so the Interleaved versus Simultaneous comparison varies more than the training rule alone. This is a methodological confound and a threat to the causal reading of Table 2, not a circular reduction. No load-bearing self-citation or imported uniqueness theorem appears; references are standard prior art. The 'state-of-the-art' wording in Section 5 is unsupported by external baselines, but that is an evidentiary overstatement, not a circular step. Accordingly, the appropriate circularity score is 0.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

The paper introduces no new theoretical entities or fitted constants. Its free parameters are standard deep learning hyperparameters. The main unstated assumptions are about the evaluation protocol, the use of mixture phase, and the representativeness of the deliberately re-split 'independent' databases. The invented_entities list is empty because the paper proposes no new physical or mathematical objects; the shared encoder and interleaved procedure are algorithmic constructs, not entities requiring independent evidence.

free parameters (5)
  • Encoder width (number of feature maps) = 16 starting feature maps; 32 for the first 3 layers in encoder+
    The encoder capacity is chosen by the authors and then tuned (encoder+ adds about 50k parameters) to close the gap with independent networks. This is a manual architectural choice, not fitted from data in a formal sense, but it is a free design parameter that affects the central results.
  • Learning rate = 0.0002
    A fixed ADAM learning rate chosen by the authors without a reported sensitivity analysis.
  • Batch size = 32 samples per GPU (or max that fits)
    Chosen by memory constraints; not systematically varied.
  • STFT window size and overlap = 2048 samples, 75% overlap
    Standard choices for music source separation, but they are fixed hyperparameters of the input representation.
  • Number of downsampling stages and feature map progression = 5 stages, 16 to 256 channels
    Architecture design choice, not compared against alternatives.
assumptions (5)
  • domain assumption The L1 spectrogram reconstruction loss is an appropriate proxy for perceptual source separation quality.
    The paper optimizes L1 distance between magnitude spectrograms (Eq. 5) and then evaluates with BSS metrics. The mismatch between training objective and evaluation metric is a known issue in source separation, and the paper does not demonstrate a strong correlation.
  • domain assumption Using the mixture phase for reconstruction is acceptable.
    The paper reconstructs time-domain signals using the input mixture phase, a common but lossy assumption that limits achievable separation quality.
  • domain assumption The random train/validation split of MUSDB18 (75/15/10) is representative.
    The paper uses one fixed split and reports no variance across splits, so the results could be sensitive to this particular split.
  • domain assumption The independent databases, built by re-splitting and shuffling the same MUSDB18 training set, faithfully represent the scenario of independent databases from different sources.
    This is the load-bearing premise discussed in weakest_assumption: the sources still come from the same recordings, so the 'independent databases' are not truly independent in content.
  • standard math BSSEval v4 with one-second segments and median aggregation gives reliable scores.
    This is a standard evaluation protocol from SiSec 2018, but BSSEval has known artifacts; the paper does not use the newer museval with the recommended settings, though it follows [1].

how reviews work

0 comments
Cite this review

Pith. "Pith review of Interleaved Multitask Learning for Audio Source Separation with Independent Databases." pith.science (2026). https://pith.science/paper/E3PJUU2C

@misc{pith2026190805182,
  author       = {Pith},
  title        = {Pith review of: Interleaved Multitask Learning for Audio Source Separation with Independent Databases},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/E3PJUU2C}},
  note         = {Machine review of arXiv:1908.05182}
}
read the original abstract

Deep Neural Network-based source separation methods usually train independent models to optimize for the separation of individual sources. Although this can lead to good performance for well-defined targets, it can also be computationally expensive. The multitask alternative of a single network jointly optimizing for all targets simultaneously usually requires the availability of all target sources for each input. This requirement hampers the ability to create large training databases. In this paper, we present a model that decomposes the learnable parameters into a shared parametric model (encoder) and independent components (decoders) specific to each source. We propose an interleaved training procedure that optimizes the sub-task decoders independently and thus does not require each sample to possess a ground truth for all of its composing sources. Experimental results on MUSDB18 with the proposed method show comparable performance to independently trained models, with less trainable parameters, more efficient inference, and an encoder transferable to future target objectives. The results also show that using the proposed interleaved training procedure leads to better Source-to-Interference energy ratios when compared to the simultaneous optimization of all training objectives, even when all composing sources are available.

Figures

Figures reproduced from arXiv: 1908.05182 by the authors.

Figure 1
Figure 1. Model diagrams for the shared encoder and the task specific decoders. Conv refers to convolution, NN Interp. to nearest-neighbour interpolation and FM Avg. to feature map averaging. Layer Type Parameters Array Size E Input - (2, 128, 1025) Conv k(5x6), s1, p2 (16, 128, 1024) Downsample k(4x4), s2, p1 (16, 64, 512) Conv k(3x3), s1, p1 (16, 64, 512) Downsample k(4x4), s2, p1 (32, 32, 256) Conv k(3x3), s1, p1 (32, 32, … view at source ↗
Figure 2
Figure 2. Interleaved training. Combining mini-batches from the Vocals, Drums, Bass and Other independent databases to train our shared encoder model. These randomly sampled subsets vary across epochs. Subsequently, the encoder is trained by alternating the sub-task mini-batches of samples passed. The layers of the encoder are chained to the corresponding decoder, whose loss optimizes both decoder and encoder as per the sub-t… view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

27 extracted references · 23 canonical work pages

  1. [1]

    Stöter, A

    F.-R. Stöter, A. Liutkus, and N. Ito. The 2018 signal sepa ration evaluation campaign. International Conference on Latent V ariable Analysis and Signal Separation (LVA/ICA), 2018

    work page 2018

  2. [2]

    Ozerov and C

    A. Ozerov and C. Févotte. Multichannel nonnegative matr ix factorization in convolutive mixtures for audio source separation. IEEE Transactions on Audio, Speech, and Language Processin g, 18(3), 2010

    work page 2010

  3. [3]

    Févotte, E

    C. Févotte, E. Vincent, and A. Ozerov. Audio Source Separation, chapter Single-channel audio source separation with NMF: divergences, constraints and algorithms. Spring er, 2018

    work page 2018

  4. [4]

    Virtanen, A

    T. Virtanen, A. Mesaros, and M. Ryynänen. Combining pitc h-based inference and non-negative spectrogram factorization in separating vocals from polyphonic music. ISCA Tutorial and Research W orkshop on Statistical and Perceptual Audition (SAPA), 2008

    work page 2008

  5. [5]

    Hennequin, J

    R. Hennequin, J. J. Burred, S. Maller, and P . Leveau. Spee ch-guided source separation using a pitch-adaptive guide signal model. IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2014

    work page 2014

  6. [6]

    Weninger, J

    F. Weninger, J. R. Hershey, J. Le Roux, and B. Schuller. Di scriminatively trained recurrent neural networks for single-channel speech separation. IEEE Global Conference on Signal and Information Processin g (GlobalSIP) , 2014

    work page 2014

  7. [7]

    Hochreiter and J

    S. Hochreiter and J. Schmidhuber. Long short-term memor y. Neural Computation, 9(8):1735–1780, 1997

    work page 1997

  8. [8]

    Uhlich, M

    S. Uhlich, M. Porcu, F. Giron, M. Enenkl, T. Kemp, N. Takah ashi, and Y . Mitsufuji. Improving music source separation based on deep neural networks through data augme ntation and network blending. IEEE International Conference on Acoustics, Speech and Signal Processing (ICA SSP), 2017

    work page 2017

Show all 27 references
  1. [9]

    Huang, M

    P .-S. Huang, M. Kim, M. Hasegawa-Johnson, and P . Smaragdis. Singing-voice separation from monaural record- ings using deep recurrent neural networks. 15th International Society for Music Information Retrieval (IS MIR) Conference, 2014. 8

    2014

  2. [10]

    S. I. Mimilakis, K. Drossos, J. Santos, G. Schuller, T. V irtanen, and Y . Bengio. Monaural singing voice separation with skip-filtering connections and recurrent i nference of time-frequency mask. arXiv preprint arXiv:1711.01437v2, 2018

    2018 arXiv

  3. [11]

    LeCun, L

    Y . LeCun, L. Bottou, Y . Bengio, and P . Haffner. Gradient-based learning applied to document recognition. Proc. of the IEEE , 86(11):2278–2324, 1998

    1998

  4. [12]

    Simonyan and A

    K. Simonyan and A. Zisserman. V ery deep convolutional n etworks for large-scale image recognition. arXiv preprint arXiv:1409.1556v6, 2014

    2014 arXiv

  5. [13]

    Chandna, M

    P . Chandna, M. Miron, J. Janer, and E. Gómez. Monoaural a udio source separation using deep convolutional neural networks. International Conference on Latent V ariable Analysis and Signal Separation (LVA/ICA), 2017

    2017

  6. [14]

    Jansson, E

    A. Jansson, E. Humphrey, N. Montecchio, R. Bittner, A. K umar, and T. Weyde. Singing voice separation with deep u-net convolutional networks. 18th International Society for Music Information Retrieval (IS MIR) Confer- ence, 2017

    2017

  7. [15]

    Takahashi and Y

    N. Takahashi and Y . Mitsufuji. Multi-scale multi-band densenets for audio source separation. IEEE W orkshop on Applications of Signal Processing to Audio and Acoustics (WASPAA), 2017

    2017

  8. [16]

    S. Park, T. Kim, K. Lee, and N. Kwak. Music source separat ion using stacked hourglass networks. 19th Interna- tional Society for Music Information Retrieval (ISMIR) Con ference, 2018

    2018

  9. [17]

    Stoller, S

    D. Stoller, S. Ewert, and S. Dixon. Wave-u-net: A multi- scale neural network for end-to-end audio source separation. 19th International Society for Music Information Retrieval (IS MIR) Conference, 2018

    2018

  10. [18]

    Takahashi, N

    N. Takahashi, N. Goswami, and Y . Mitsufuji. MMDenseLST M: an efficient combination of convolutional and recurrent neural networks for audio source separation. arXiv preprint arXiv:1805.02410v2, 2018

    2018 arXiv

  11. [19]

    R. Caruana. Multitask learning. Machine Learning, 28(1):41–75, 1997

    1997

  12. [20]

    Z. Chao, V . Badrinarayanan, C.-Y . Lee, and A. Rabinovich. Gradnorm : Gradient normalization for adaptive loss balancing in deep multitask networks. International Conference on Machine Learning (ICML) , 2018

    2018

  13. [21]

    Kirkpatrick, R

    J. Kirkpatrick, R. Pascanu, N. Rabinowitz, J. V eness, G . Desjardins, A. A. Rusu, K. Milan, J. Quan, T. Ra- malho, A. Grabska-Barwinska, D. Hassabis, C. Clopath, D. Ku maran, and R. Hadsell. Overcoming catastrophic forgetting in neural networks. arXiv preprint arXiv:1612.0079...

    2017 arXiv

  14. [22]

    Stoller, S

    D. Stoller, S. Ewert, and S. Dixon. Jointly detecting an d separating singing voice: A multi-task approach. International Conference on Latent V ariable Analysis and Signal Separation (LVA/ICA), 2018

    2018

  15. [23]

    Ioffe and C

    S. Ioffe and C. Szegedy. Batch normalization: accelera ting deep network training by reducing internal covariate shift. International Conference on Machine Learning (ICML) , pages 448–456, 2015

    2015

  16. [24]

    A. L. Maas, A. Y . Hannun, and A. Y . Ng. Rectifier nonlinear ities improve neural network acoustic models. International Conference on Machine Learning (ICML) , 2013

    2013

  17. [25]

    Odena, V

    A. Odena, V . Dumoulin, and C. Olah. Deconvolution and ch eckerboard artifacts. Distill, 2016

    2016

  18. [26]

    Vincent, R

    E. Vincent, R. Gribonval, and C. Févotte. Performance m easurement in blind audio source separation. IEEE Transactions on Audio, Speech, and Language Processing , 14(4):1462–1469, 2006

    2006

  19. [27]

    D. P . Kingma and J. Ba. Adam: A method for stochastic opti mization. arXiv preprint arXiv:1412.6980, 2014. 9

    2014 arXiv

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.