REVIEW 2 major objections 6 minor 31 references
Speaker Embeddings to Improve Tracking of Intermittent and Moving Speakers
T0 review · 2 major / 6 minor · reviewed 2026-08-15 · deepseek-v4-flash
Pith's one-line read Speaker embeddings fix identity swaps in moving-speaker tracking
desk verdict Solid, useful post-tracking identity reassignment with a real confound in the enrollment-count experiment; the core M=J result holds. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is fragment-level identity reassignment. A fragment is the continuous active period within one tracker trajectory. For each fragment, a beamformer (ideal, delay-and-sum, or MVDR) is steered toward the fragment's direction of arrival to suppress noise and interfering speakers; a pretrained ECAPA-TDNN speaker-embedding network converts the beamformed audio into a 192-dimensional vector; and a first-in-first-out cosine-similarity match against enrollment embeddings assigns a new identity, discarding enrollments already used by temporally overlapping fragments. This machinery converts identity from a spatial continuity assumption into a voice-match decision, and its quality determines whether the system gains over the tracker baseline.
What would settle it
Run the proposed reassignment on 100 ms fragments from a tracker with a high swap rate using MVDR beamforming; the paper's universal claim predicts an AssA improvement over the baseline in every condition, so any such condition where AssA does not improve would disprove the claim.
Extended reading notes
Core claim
The paper claims that a post-tracking identity reassignment stage, built on beamformed speaker embeddings, consistently improves the identity assignment performance of tracking systems on scenes with intermittent and moving speakers. Concretely, each tracker output trajectory is cut into fragments, the multichannel signal is beamformed toward each fragment's direction of arrival, a pretrained speaker-embedding model encodes the enhanced audio, and each fragment is reassigned to the enrollment embedding with the highest cosine similarity. On the simulated two-speaker evaluation, the best configuration (whole fragments, ideal beamforming, enrollment size equal to the true number of speakers) raises tracking association accuracy from 40.5% before reassignment to 92.1% after. The paper reports gains for every beamformer, input duration, and enrollment count tested, with the size of the gain controlled by embedding quality and tracker trajectory quality.
Load-bearing premise
The load-bearing premise is that each beamformed fragment contains mostly one speaker's voice, so its embedding is closer to that speaker's enrollment than to any other; short, noisy, or overlapping fragments break this premise.
Editorial extensions
If this is right
- Any existing tracking system can be upgraded by adding this reassignment stage, without retraining the underlying tracker.
- The gain is largest when fragments are long and spatially separated; low-latency use with 250 ms fragments roughly halves the best-case accuracy gain.
- Beamformer choice matters: MVDR outperforms delay-and-sum, especially for closer speakers and shorter fragments.
- Enlarging the enrollment pool hurts the tracker baseline, but the reassignment stage becomes resilient to this when fragment embeddings are clean.
- The method helps both Bayesian and neural trackers, with the magnitude of the gain tied to the tracker's spatial quality and its track swap rate.
Reading between the lines
- The paper does not test a verification threshold; replacing top-1 cosine assignment with a similarity threshold could let the system reject unknown speakers and handle open enrollment sets.
- Because the dominant failure mode is short-input embedding quality, fine-tuning a speaker-embedding model on short, noisy, beamformed fragments could recover much of the low-latency loss the paper reports.
- The same fragment-level idea could be combined with a track-swap trigger, running embedding checks only after spatial discontinuities, which would reduce computation in long sessions.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This paper addresses identity assignment in multi-speaker tracking when speakers move during inactivity, producing discontinuous spatial trajectories. The authors propose a post-tracking module that splits each estimated trajectory into activity fragments, beamforms the multichannel mixture toward each fragment's direction of arrival, extracts ECAPA-TDNN speaker embeddings, and reassigns fragment identities by cosine similarity against a pool of enrollment embeddings using a first-in-first-out rule. Experiments on the LibriJump-2spk dataset with two speakers compare three trackers (ground-truth DoA particle filter, estimated-DoA particle filter, and a neural tracker), three beamformers (ideal, delay-and-sum, MVDR), fragment durations from 250 ms to full length, and enrollment pool sizes M=2, 10, 20, 30. The reported metric is AssA (tracking association accuracy). The central result is that reassignment improves AssA over the tracker-alone baseline in all reported conditions, with the best case rising from 40.5% to 92.1%.
Significance. If the result holds, the paper makes a modest but useful contribution: it shows that a simple post-tracking identity-reassignment scheme based on speaker embeddings can correct identity swaps in intermittent/moving-speaker scenarios. A notable strength is that the embedding extractor and the neural trackers are pretrained or classical, and enrollment embeddings come from separate clean recordings, so the reported improvements are not obtained by fitting to the test set. The bootstrap standard deviations below 1% give some confidence in the comparisons. The main significance is as a proof-of-concept that identity-related observations can complement spatial observations for tracking; however, the generality of the headline claim is limited by the enrollment-pool/track-capacity confound discussed below and by the lack of comparison with non-embedding identity features.
major comments (2)
- [Section IV-A and Section III-B] The claim that the proposed reassignment improves performance 'for any ... number of enrollments' is confounded by the GT tracker design. In Section III-B, the Bayesian tracker's maximum number of predicted identities is fixed to the number of enrollments M. Since the scenes contain J=2 speakers, increasing M to 10, 20, or 30 forces the tracker to maintain up to M tracks, generating spurious identities and lowering the no-reassignment AssA from 40.5% (M=2) to 12.5% (M=30), as shown in Fig. 2. The reassignment then merges these spurious tracks via enrollment embeddings. Part of the measured improvement at large M is therefore an artifact of coupling enrollment-pool size to tracker track capacity rather than a demonstration of the identity-reassignment module's robustness. The neural tracker, whose output count is fixed at two, is only evaluated at M=2. To support the 'any number of enrollments' statement, the authors should either evaluate with a tracker whose track capacity is independent of M (e.g., the NN tracker at M>2), or restrict the claim to the case where the tracker's capacity equals the enrollment pool size.
- [Section II-C and Section IV] The paper motivates speaker embeddings by their 'superior ability to distinguish between speakers, compared to other speaker-related features' and claims the experiments prove the usefulness of embeddings as identity-related observations. However, the only baseline is the tracker without reassignment; no comparison is made against an equivalent reassignment system using spectral signatures or fundamental-frequency features, which are the alternatives cited in [6], [7]. As a result, the experiments show that embedding-based reassignment improves over no reassignment, but they do not establish that speaker embeddings are better than previously proposed identity-related features for this task. The authors should either add such a comparison or soften the claims accordingly.
minor comments (6)
- [Section II-B] The title 'Fragment-level identity reassignment using speaker emebddings' contains a typo: 'emebddings' should be 'embeddings'.
- [Section III-A] The term 'wet speech signals' is used without definition; please clarify that it means reverberant signals, and specify the exact angular ranges (e.g., 25 degrees to 60 degrees inclusive) for the two datasets.
- [Section III-C] The text contains a spacing typo, 'V oxceleb1', and the Hugging Face model link should include an access date for reproducibility.
- [Section IV-A] The sentence 'This assesses the usefulness of speaker embeddings...' should be reworded to 'This demonstrates the usefulness...' or 'This supports the usefulness...' to avoid an odd collocation.
- [Section V] The conclusion states 'the unsuitability of such general pretrained models to extract robust speaker embeddings', which is too strong given that the same model yields substantial improvements in several conditions; 'the limited robustness' would be more accurate.
- [Table I] Tables report only point estimates; since the text states bootstrap standard deviations are below 1%, reporting those values or confidence intervals in the tables would strengthen the quantitative comparisons.
Circularity Check
Independent empirical evaluation; no circular derivation found.
full rationale
The proposed reassignment pipeline is not circular: it takes tracker-defined fragments, beamforms toward the tracker's DoA estimates, extracts ECAPA-TDNN embeddings from a pretrained VoxCeleb model, and matches them by cosine similarity to enrollment embeddings computed from separate clean utterances. No parameter is fitted to the evaluation set, and the reported metrics (AssA, TSR, TFR, LE) are computed from reassigned trajectories against ground truth. The only self-citation is [12], which supplies the LibriJump-2spk dataset and the AssA adaptation; this is a disclosed experimental resource rather than a load-bearing derivation. The claim that reassignment 'improves performances over the baseline red scores before reassignment' is an empirical generalization from Figs. 2-3 and Table III, and the core comparison holds the tracker settings fixed while toggling reassignment. The M-variation experiment does couple the enrollment count to the GT tracker's maximum track count, which is a possible confound in the magnitude of the reported gain, but it is not a definitional reduction of the reassignment output to its inputs, and no fitted parameter is renamed as a prediction. No uniqueness theorem or ansatz is imported from prior work by the same authors, so no circular step can be exhibited by quoting a specific equation or construction.
Assumptions & free parameters
assumptions (3)
- domain assumption The synthetic SRIR-based simulation of movement during silence faithfully represents the real problem of intermittent and moving speakers.
- domain assumption A tracker-defined fragment, after beamforming toward its DoA, is acoustically dominated by one speaker so that a pretrained embedding model yields identity-discriminative representations.
- domain assumption The VoxCeleb-pretrained ECAPA-TDNN embedding model transfers to the simulated LibriSpeech FOA mixtures (25-180 degree separation, 15 dB SNR) without adaptation.
Cite this review
Pith. "Pith review of Speaker Embeddings to Improve Tracking of Intermittent and Moving Speakers." pith.science (2026). https://pith.science/paper/ORE3OT2R
@misc{pith2026250619875,
author = {Pith},
title = {Pith review of: Speaker Embeddings to Improve Tracking of Intermittent and Moving Speakers},
year = {2026},
howpublished = {\url{https://pith.science/paper/ORE3OT2R}},
note = {Machine review of arXiv:2506.19875}
}
read the original abstract
Speaker tracking methods often rely on spatial observations to assign coherent track identities over time. This raises limits in scenarios with intermittent and moving speakers, i.e., speakers that may change position when they are inactive, thus leading to discontinuous spatial trajectories. This paper proposes to investigate the use of speaker embeddings, in a simple solution to this issue. We propose to perform identity reassignment post-tracking, using speaker embeddings. We leverage trajectory-related information provided by an initial tracking step and multichannel audio signal. Beamforming is used to enhance the signal towards the speakers' positions in order to compute speaker embeddings. These are then used to assign new track identities based on an enrollment pool. We evaluate the performance of the proposed speaker embedding-based identity reassignment method on a dataset where speakers change position during inactivity periods. Results show that it consistently improves the identity assignment performance of neural and standard tracking systems. In particular, we study the impact of beamforming and input duration for embedding extraction.
Figures
Reference graph
Works this paper leans on
-
[6]
Audio Tracking in Noisy Environments by Acoustic Map and Spec- tral Signature,
M. Crocco, S. Martelli, A. Trucco, A. Zunino, and V . Murino, “Audio Tracking in Noisy Environments by Acoustic Map and Spec- tral Signature,” IEEE Transactions on Cybernetics , vol. 48, no. 5, pp. 1619–1632, May 2018
work page 2018
-
[7]
A. O. T. Hogg, C. Evers, and P. A. Naylor, “Multichannel Overlapping Speaker Segmentation Using Multiple Hypothesis Tracking Of Acous- tic And Spatial Features,” en, in Proc. ICASSP, Toronto, ON, Canada: IEEE, Jun. 2021, pp. 26–30
work page 2021
-
[1]
Sound source localization and speech enhancement with sparse Bayesian learning beamforming,
A. Xenaki, J. B ¨unsow Boldt, and M. Græsbøll Christensen, “Sound source localization and speech enhancement with sparse Bayesian learning beamforming,” The Journal of the Acoustical Society of America, vol. 143, no. 6, pp. 3912–3921, Jun. 2018
work page 2018
-
[2]
A. S. Subramanian, C. Weng, S. Watanabe, M. Yu, and D. Yu, “Deep learning based multi-source localization with source splitting and its effectiveness in multi-talker speech recognition,” Comput. Speech Lang., vol. 75, p. 101 360, Sep. 2022
work page 2022
-
[3]
Tracking Intermittently Speaking Multiple Speakers Using a Particle Filter,
A. Quinlan, M. Kawamoto, Y . Matsusaka, H. Asoh, and F. Asano, “Tracking Intermittently Speaking Multiple Speakers Using a Particle Filter,” en, EURASIP Journal on Audio, Speech, and Music Process- ing, vol. 2009, no. 1, pp. 1–11, Dec. 2009, Number: 1 Publisher: SpringerOpen
work page 2009
-
[4]
Acoustic Source Localization and Tracking of a Time-Varying Number of Speakers,
M. F. Fallon and S. J. Godsill, “Acoustic Source Localization and Tracking of a Time-Varying Number of Speakers,” IEEE Transactions on Audio, Speech, and Language Processing , vol. 20, no. 4, pp. 1409– 1415, May 2012, Conference Name: IEEE Transactions on Audio, Speech, and Language Processing
work page 2012
-
[5]
Online Localization and Tracking of Multiple Moving Speakers in Reverberant Environments,
X. Li, Y . Ban, L. Girin, X. Alameda-Pineda, and R. Horaud, “Online Localization and Tracking of Multiple Moving Speakers in Reverberant Environments,” en, IEEE Journal of Selected Topics in Signal Process- ing, vol. 13, no. 1, pp. 88–103, Mar. 2019
work page 2019
-
[8]
B.-n. V o, M. Mallick, Y . Bar-shalom, S. Coraluppi, R. Osborne III, R. Mahler, and B.-t. V o, “Multitarget Tracking,” en, inWiley Encyclopedia of Electrical and Electronics Engineering , John Wiley & Sons, Ltd, 2015, pp. 1–15
work page 2015
Show all 31 references
-
[9]
Differentiable Tracking- Based Training of Deep Learning Sound Source Localizers,
S. Adavanne, A. Politis, and T. Virtanen, “Differentiable Tracking- Based Training of Deep Learning Sound Source Localizers,” in Proc. WASPAA, ISSN: 1947-1629, Oct. 2021, pp. 211–215
1947
-
[10]
Position Tracking of a Varying Number of Sound Sources with Sliding Permutation Invariant Training,
D. Diaz-Guerra, A. Politis, and T. Virtanen, “Position Tracking of a Varying Number of Sound Sources with Sliding Permutation Invariant Training,” in Proc. EUSIPCO, Sep. 2023, pp. 251–255
2023
-
[11]
The Neural-SRP Method for Universal Robust Multi-Source Tracking,
E. Grinstein, C. M. Hicks, T. van Waterschoot, M. Brookes, and P. A. Naylor, “The Neural-SRP Method for Universal Robust Multi-Source Tracking,” IEEE Open Journal of Signal Processing, vol. 5, pp. 19–28, 2024
2024
-
[12]
Tracking of Intermittent and Moving Speakers : Dataset and Metrics,
T. Iatariene, A. Gu ´erin, and R. Serizel, “Tracking of Intermittent and Moving Speakers : Dataset and Metrics,” in Proceedings of the 11th Convention of the European Acoustics Association Forum Acusticum 2025, Malaga, Espagne, Spain, Jun. 2025
2025
-
[13]
X-Vectors: Robust DNN Embeddings for Speaker Recognition,
D. Snyder, D. Garcia-Romero, G. Sell, D. Povey, and S. Khudanpur, “X-Vectors: Robust DNN Embeddings for Speaker Recognition,” en, in Proc. ICASSP, Calgary, AB: IEEE, Apr. 2018, pp. 5329–5333
2018
-
[14]
ECAPA-TDNN: Emphasized Channel Attention, Propagation and Aggregation in TDNN Based Speaker Verification,
B. Desplanques, J. Thienpondt, and K. Demuynck, “ECAPA-TDNN: Emphasized Channel Attention, Propagation and Aggregation in TDNN Based Speaker Verification,” en, in Proc. Interspeech , Oct. 2020, pp. 3830–3834
2020
-
[15]
Deep speaker em- beddings for Speaker Verification: Review and experimental compari- son,
M. Jakubec, R. Jarina, E. Lieskovska, and P. Kasak, “Deep speaker em- beddings for Speaker Verification: Review and experimental compari- son,” en, Engineering Applications of Artificial Intelligence , vol. 127, p. 107 232, Jan. 2024
2024
-
[16]
A review of speaker diarization: Recent advances with deep learning,
T. J. Park, N. Kanda, D. Dimitriadis, K. J. Han, S. Watanabe, and S. Narayanan, “A review of speaker diarization: Recent advances with deep learning,” Computer Speech & Language , vol. 72, p. 101 317, Mar. 2022
2022
-
[17]
Librispeech: An ASR corpus based on public domain audio books,
V . Panayotov, G. Chen, D. Povey, and S. Khudanpur, “Librispeech: An ASR corpus based on public domain audio books,” in Proc. ICASSP, Apr. 2015, pp. 5206–5210
2015
-
[18]
Improved feature extraction for CRNN-based multiple sound source localization,
P.-A. Grumiaux, S. Kitic, L. Girin, and A. Guerin, “Improved feature extraction for CRNN-based multiple sound source localization,” en, in Proc. EUSIPCO, Dublin, Ireland: IEEE, Aug. 2021, pp. 231–235
2021
-
[19]
TRAMP: Tracking by a Real-time AMbisonic- based Particle filter,
S. Kiti ´c and A. Gu´erin, “TRAMP: Tracking by a Real-time AMbisonic- based Particle filter,” in IEEE-AASP Challenge on Acoustic Source Localization and Tracking - LOCATA , Tokyo, Japan, Sep. 2018
2018
-
[20]
Ravanelli, T
M. Ravanelli, T. Parcollet, P. Plantinga, et al., SpeechBrain: A General- Purpose Speech Toolkit, Jun. 2021
2021
-
[21]
V oxCeleb: A Large-Scale Speaker Identification Dataset,
A. Nagrani, J. S. Chung, and A. Zisserman, “V oxCeleb: A Large-Scale Speaker Identification Dataset,” in Proc. Interspeech, 2017, pp. 2616– 2620
2017
-
[22]
Dowerah, R
S. Dowerah, R. Serizel, D. Jouvet, M. Mohammadamini, and D. Matrouf, How to Leverage DNN-based speech enhancement for multi- channel speaker verification? arXiv:2210.08834 [cs], Oct. 2022
2022 arXiv
-
[23]
Analyse de sc `ene sonore multi-capteurs : Un front-end temps-r´eel pour la manipulation de sc `ene,
M. Baque, “Analyse de sc `ene sonore multi-capteurs : Un front-end temps-r´eel pour la manipulation de sc `ene,” fr, Ph.D. dissertation, Universit´e du Maine, Jun. 2017
2017
-
[24]
High-resolution frequency-wavenumber spectrum analysis,
J. Capon, “High-resolution frequency-wavenumber spectrum analysis,” Proceedings of the IEEE , vol. 57, no. 8, pp. 1408–1418, Aug. 1969, Conference Name: Proceedings of the IEEE
1969
-
[25]
Dilated U-net based approach for multichannel speech enhancement from First-Order Ambisonics recordings,
A. Bosca, A. Gu ´erin, L. Perotin, and S. Kiti ´c, “Dilated U-net based approach for multichannel speech enhancement from First-Order Ambisonics recordings,” in 2020 28th European Signal Processing Conference (EUSIPCO), ISSN: 2076-1465, Jan. 2021, pp. 216–220
2020
-
[26]
HOTA: A Higher Order Metric for Evaluating Multi-Object Tracking,
J. Luiten, A. Osep, P. Dendorfer, P. Torr, A. Geiger, L. Leal-Taixe, and B. Leibe, “HOTA: A Higher Order Metric for Evaluating Multi-Object Tracking,” International Journal of Computer Vision , vol. 129, no. 2, pp. 548–578, Feb. 2021
2021
-
[27]
The LOCATA Challenge: Acoustic Source Localization and Tracking,
C. Evers, H. Loellmann, H. Mellmann, A. Schmidt, H. Barfuss, P. Naylor, and W. Kellermann, “The LOCATA Challenge: Acoustic Source Localization and Tracking,” en, IEEE/ACM Trans. Audio, Speech, Lang. Process., vol. 28, pp. 1620–1643, 2020
2020
-
[28]
CRNN-Based Multi- ple DoA Estimation Using Acoustic Intensity Features for Ambisonics Recordings,
L. Perotin, R. Serizel, E. Vincent, and A. Gu ´erin, “CRNN-Based Multi- ple DoA Estimation Using Acoustic Intensity Features for Ambisonics Recordings,” IEEE Journal of Selected Topics in Signal Processing , vol. 13, no. 1, pp. 22–33, Mar. 2019, Conference Name: IEEE Journal o...
2019
-
[29]
C. Cui, I. A. Sheikh, M. Sadeghi, and E. Vincent, Improving Speaker Assignment in Speaker-Attributed ASR for Real Meeting Applications , arXiv:2403.06570 [cs], Sep. 2024
2024 arXiv
-
[30]
Z. Chen, T. Yoshioka, L. Lu, T. Zhou, Z. Meng, Y . Luo, J. Wu, X. Xiao, and J. Li, Continuous speech separation: Dataset and analysis , en, arXiv:2001.11482 [cs, eess], May 2020
2001 arXiv
-
[31]
Integrating End-to-End Neural and Clustering-Based Diarization: Getting the Best of Both Worlds,
K. Kinoshita, M. Delcroix, and N. Tawara, “Integrating End-to-End Neural and Clustering-Based Diarization: Getting the Best of Both Worlds,” in Proc. ICASSP , ISSN: 2379-190X, Jun. 2021, pp. 7198– 7202
2021
Reviewed August 15, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.