{"id":"32aab8c6-77b4-40fc-90ed-89ceb857c729","arxiv_id":"2411.11514","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":7,"one_line_summary":"A self-supervised training method using a neural Kalman filter and Sinkhorn assignment learns to associate detections across frames without identity labels, reaching state-of-the-art self-supervised scores on MOT17 and MOT20.","lead":"This paper trains a neural network to link detections of the same object across video frames without identity labels, using a Kalman filter and a differentiable assignment step. It is much faster to train than prior self-supervised trackers and matches or beats them on standard multi-object tracking benchmarks.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Eq. (8) maximizes a smoothed density that conditions on the same observation z_t, so it is not a marginal likelihood; the objective can be high for degenerate associations, and the paper provides no analysis or control ruling this out.","rationale":"The reader's weakest assumption correctly identifies the self-conditioned nature of eq. (8). I sharpen the concern: even treating P_t as fixed, p(z_t | z_{1:T}, P_{1:T}) is not a valid likelihood contribution because z_t is in the conditioning set; the correct likelihood is the predictive density from the filter. Because the supplied noise settings make the smoother trust observations heavily, the smoothed mean can absorb any locally consistent permutation, so the objective can be maximized by degenerate associations. This does not prove the reported MOT results are wrong, but it does mean the central theoretical claim, that the network is trained by maximizing a marginal likelihood via EM, is unsupported. A synthetic permutation-recovery experiment would settle whether the objective carries any association signal. This keeps the reader's CONDITIONAL verdict: the empirical contribution may survive, but the paper should either provide the missing analysis or reframe the method as a heuristic self-supervised loss.","tokens_in":13178,"tokens_out":9701,"duration_ms":114670,"concrete_test":"Train the exact Algorithm 1 on a synthetic dataset of K=4 objects with known linear tracks, randomly permuting the detection order at every frame so the correct association is a nontrivial random permutation, and measure whether the network's predicted P_t matches the ground-truth permutation. If eq. (8) decreases while permutation-recovery accuracy stays at chance, the objective is self-confirmatory; if recovery accuracy is high, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing issue is in the training objective, eqs. (6) and (8). In a state-space model the likelihood of the observations is the predictive density product p(z_t | z_{1:t-1}, P_{1:t-1}) computed by the filter in eq. (3)-(5). The paper instead maximizes sum_t log p(z_t | z_{1:T}, P_{1:T}), where the conditioning set z_{1:T} already includes z_t. This is not a likelihood for z_t; it is a smoothed reconstruction density evaluated at the very observation used to compute it. With the supplementary training noise settings (R=5, Q=150), the smoother trusts observations heavily, so for any locally consistent permutation sequence the smoothed mean approximately satisfies H_t P_t tilde_mu_t approx z_t and the loss is high whether or not P_t is the true association. Since P_t enters both the observation model and the state estimate, gradient descent on eq. (8) can reduce the loss by making the association copy current detections into the state rather than by recovering correct correspondences. Algorithm 1 is therefore direct gradient descent on a self-confirmatory objective, not EM: there is no E-step over P and no lower bound on the marginal likelihood. The strong MOT results may still hold as a heuristic, but the paper does not report any experiment, such as association accuracy against known training correspondences or comparison with a proper predictive-likelihood objective, that excludes the degenerate solution.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a self-supervised multi-object tracking method. Detections in short video clips are associated into tracks by an MLP that outputs score matrices, followed by Sinkhorn normalization to obtain doubly stochastic association matrices. These matrices parameterize the observation model of a Kalman filter with a random-walk motion model, and Kalman smoothing is used to form the training objective in Eq. (8), which the paper describes as maximizing the marginal probability of the observations. An appearance model is then fine-tuned against the learned associations via a KL divergence. At inference, the learned motion and appearance affinities are combined in an online Hungarian matching procedure. Experiments on MOT17, MOT20, and BDD100K report state-of-the-art results among self-supervised trackers using public detections, together with ablations over tracking parameters, the amount of unlabeled data, and Sinkhorn iterations.","tokens_in":13494,"tokens_out":4837,"duration_ms":48623,"significance":"If the probabilistic derivation were sound, this would be an attractive contribution: a fast, fully differentiable, self-supervised tracker with a principled objective and strong empirical results. The paper has clear strengths: the pipeline is carefully described, the ablations cover the main hyperparameters, the training time is remarkably short compared with prior self-supervised trackers, and the final evaluation is against external MOT benchmarks. However, the central theoretical claim is not established. The quantity in Eq. (6), and therefore the objective in Eq. (8), is not a marginal likelihood for the observations because the conditioning set includes the observation being scored. Algorithm 1 is also not an expectation-maximization procedure, since no expectation over the latent association variables is taken. The stress-test concern about a self-confirmatory objective lands directly on Eqs. (6) and (8) and is reinforced by the training noise settings in Appendix 6.","major_comments":[{"comment":"The quantity p(zt | z1:T, P1:T) is not a likelihood for zt, because zt is included in the conditioning set. Once z1:T is conditioned on, the conditional distribution of zt is point mass at the observed value; the Gaussian expression in Eq. (6) is a reconstruction density, not a predictive or marginal likelihood. The correct filtering likelihood is p(zt | z1:t-1, P1:t-1) from Eq. (5), evaluated at the observed zt. Maximizing Eq. (8) is therefore not maximum likelihood estimation, and the abstract's and Section 1's claim of maximizing 'the marginal probability of observations' is unsupported.","section":"§3.1, Eqs. (6) and (8)"},{"comment":"The procedure is not an EM algorithm. There is no E-step over the latent association variables P_t: the permutation matrices are deterministic functions of the observations through g_theta and Sinkhorn normalization, and the algorithm alternates only between computing smoothed densities and taking gradient steps on theta. An EM treatment would require a complete-data likelihood, a posterior over P_t given the observations, and an expected complete-data log-likelihood (or a variational lower bound). The repeated use of 'expectation maximisation' in the abstract, Section 1, Section 3.1, and the conclusion is not justified by the presented algorithm, which is direct gradient descent on a self-conditioned reconstruction loss.","section":"§3.1, Algorithm 1"},{"comment":"The training configuration Q=150, R=5 makes the smoother rely heavily on the observations. In that regime, for any locally consistent P_t, the smoothed mean approximately satisfies H_t P_t tilde_mu_t ≈ z_t, so the objective in Eq. (8) is high regardless of whether P_t recovers the true correspondences. Because P_t enters both the observation operator and the state estimate, gradient descent can reduce the loss by making P_t copy current detections into the state rather than by identifying correct associations. The paper reports no experiment that would expose this failure mode: there is no evaluation of association accuracy against known correspondences, no baseline with random or identity P_t, and no comparison of Eq. (8) with the predictive-likelihood objective of Eq. (5). Without such a control, the benchmark gains could be driven by the geometric and appearance features and by the KCF preprocessing rather than by the training objective.","section":"Appendix 6 and Eq. (8)"},{"comment":"The training clips are generated with the KCF tracker, which is initialized independently for each detection in the first frame and is used to replace missing detections and to discard detections with low IoU against tracked boxes. KCF is a single-object tracker, so its outputs carry temporal identity information that is injected into the supposedly label-free training signal. This should be disclosed and discussed as a form of pseudo-labeling; the self-supervision claim should be qualified, and an ablation without KCF (for example, using raw detections with varying K per clip) should be reported.","section":"Appendix 6, Preprocessing Detections"}],"minor_comments":[{"comment":"The denominator 'p(zt | z1:t−1P1:t)' is missing a comma; it should read p(zt | z1:t-1, P1:t).","section":"Eq. (4)"},{"comment":"The paper uses 'permutation matrix' for the output of the Sinkhorn layer, but that output is a doubly stochastic matrix, not a hard permutation. The distinction matters because P_t is defined as a product of soft matrices; the terminology should be made precise.","section":"Eq. (7) and surrounding text"},{"comment":"The notation 'Pt:t+2' in the caption is confusing, since P_t is defined per frame in Section 3; please clarify what the subscripts range over.","section":"Figure 2 caption"},{"comment":"The reference to 'Favyen et al.' should be 'Bastani et al.' when referring to citation [1].","section":"Section 2"},{"comment":"The explanation that UnsupTrack uses CenterNet detections appears in a footnote after the table; it should be moved into the table caption or the main text for clarity.","section":"Table 2"},{"comment":"The label 'marginal probability of observations' is inconsistent with Eq. (6), whose conditioning set includes those observations; consider renaming this quantity a 'reconstruction density' if the objective is retained as a heuristic.","section":"Algorithm 1, line 10"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The short version: this is a genuinely useful self-supervised MOT method with strong empirical results, but the paper oversells its theoretical foundation. The training objective is not actually EM, and the likelihood it maximizes conditions on the same observations it is trying to explain; that is a real gap, not a nitpick.\n\nWhat's new: combining a neural association network, Sinkhorn normalization, and Kalman smoothing into one differentiable self-supervised objective is a new combination, and it trains in minutes instead of the roughly 24 hours needed by prior self-supervised methods like Bastani et al. and Lu et al. The results on MOT17 and MOT20 with public detections are state-of-the-art among self-supervised trackers, and the method also beats several fully-supervised baselines. The ablations are reasonably thorough—the studies on the termination threshold, amount of unlabeled data, and hyperparameter robustness are useful. Fine-tuning the appearance model from learned associations clearly helps, cutting IDSW by 11% on the MOT17 training set.\n\nWhere it's soft: the EM framing does not hold up. Algorithm 1 is direct gradient descent on eq. (8); there is no E-step over the permutation matrices, and no lower bound on a marginal likelihood is constructed. More importantly, eq. (6) conditions on z_{1:T}, which includes z_t, so the \"marginal probability of observations\" is a smoothed reconstruction density evaluated at the very observations used to compute it. With the training noise settings (R=5, Q=150), the smoother trusts observations heavily, so the loss can be high even when the association matrix is wrong—the network can reduce the loss by copying current detections into the state rather than recovering true correspondences. The paper gives no experiment, such as measuring association accuracy against ground-truth correspondences or comparing with a proper predictive-likelihood objective, that would rule out this degenerate solution. That said, the benchmark results are external and strong; if the objective were collapsing to degeneracy, you'd expect worse MOTA/HOTA. So the self-referential objective probably works as a rich heuristic, but the paper should stop calling it a marginal likelihood and EM.\n\nMinor issues: no code release, which makes the KCF-based clip preprocessing hard to reproduce, and the BDD100K comparison is thin—only SORT as a self-supervised baseline. These are fixable.\n\nBottom line: this deserves peer review, but the revision should reframe the objective as a heuristic loss, add a control experiment to test for degenerate associations, and release code. I would cite the empirical recipe if I were working on self-supervised MOT.","headline":"A useful self-supervised MOT method with strong benchmark results, but the paper overstates its theory: the objective is not a true marginal likelihood and the EM framing is misleading.","tokens_in":14026,"tokens_out":1812,"would_cite":true,"duration_ms":18430,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper shows that data association for multi-object tracking can be learned from unlabeled detections alone by maximizing the Kalman-smoothed likelihood of the observations under Sinkhorn-normalized soft permutations.","keywords":["multi-object tracking","self-supervised learning","data association","neural Kalman filter","Kalman smoothing","Sinkhorn normalization","expectation maximization","appearance model"],"falsifier":"On a synthetic clip of two identical objects moving at constant velocity and crossing paths, the method should preserve each identity through the crossing; if the learned association network swaps them, the smoothed-likelihood objective is not rewarding the correct association.","tokens_in":12944,"feed_emoji":"🎯","tokens_out":8444,"duration_ms":75107,"temperature":0.7,"pith_summary":"This paper attempts to establish that the association step of multi-object tracking—deciding which detection in one frame belongs to which track—can be learned without any identity annotations. The authors build a fully differentiable system in which a small network predicts a soft permutation matrix linking detections in adjacent frames, a Sinkhorn layer makes that matrix doubly stochastic, and a Kalman smoother then scores how well the implied state trajectories explain the observed boxes. Training maximizes that smoothed likelihood, and an appearance model is fine-tuned afterwards using only the learned associations. If the claim holds, self-supervised trackers can match or beat several fully-supervised methods on standard benchmarks, and training takes minutes rather than days.","feed_headline":"Self-supervised tracker tops label-free baselines on MOT17 and MOT20","feed_subtitle":"A neural Kalman filter plus Sinkhorn normalization learns data association in minutes, no identity labels needed.","key_machinery":"The load-bearing object is the neural association network gθ together with the Sinkhorn layer that turns its pairwise score matrix into a doubly stochastic permutation matrix A_t, and the Kalman filter/smoother whose observation model p(z_t|x_t,P_t)=N(z_t; H_t P_t x_t, R_t) uses that permutation to assign detections to states. The training objective is the negative smoothed log-likelihood, eq. (8), which is minimized by gradient descent. Kalman smoothing propagates information through the whole batch of frames in both directions, so the objective rewards associations that produce smooth, globally consistent trajectories. The appearance model φθ is then fine-tuned by minimizing the KL divergence between the learned permutation matrix and the cosine-similarity matrix of appearance features.","core_discovery":"The central claim is that a maximum-likelihood objective over a Kalman filter, with the observation model conditioned on a network-predicted association matrix, provides enough training signal to learn data association from unlabeled detections. The permutations are made differentiable by Sinkhorn normalization, so the whole pipeline can be trained end-to-end by back-propagating the negative log marginal likelihood computed by Kalman smoothing. On the paper's own terms, this is an expectation-maximisation procedure that alternates between inferring state trajectories and identifying permutations mapping observations to trajectories. On MOT17 and MOT20 with public detections, the method achieves state-of-the-art results among self-supervised trackers, and it also improves over SORT on BDD100K.","pith_inferences":["A decisive test of the objective would use synthetic videos with known ground-truth associations: if the learned permutations recover the ground truth even when appearance is ambiguous, the motion prior is doing the expected work; if not, the loss may be a heuristic that happens to succeed on the evaluated benchmarks.","The same EM-plus-Sinkhorn recipe could transfer to other assignment problems with a Markov structure, such as cell tracking in microscopy, where identity labels are even harder to obtain.","The paper leaves open the regime where the random-walk motion assumption is violated, such as abrupt camera cuts or erratic motion; there the smoothed-likelihood objective may degrade, and a learned or nonlinear motion model would be needed.","One extension is to iterate appearance fine-tuning and association training in a fully self-supervised loop, which could further reduce identity switches in crowded scenes."],"forward_implications":["Self-supervised MOT becomes a single differentiable maximum-likelihood problem, removing the need for hand-designed pseudo labels or path-consistency heuristics.","Because the association network trains in minutes rather than hours, the practical cost of building a tracker for a new video domain drops sharply.","The learned motion and appearance affinities combine naturally in an online Kalman-filter tracking loop, supporting near-real-time inference.","The same label-free training procedure transfers across MOT17, MOT20, and BDD100K, suggesting it can scale to new datasets without annotation effort.","On public detections the method outperforms prior self-supervised trackers on HOTA and IDF1 on MOT17 and on all reported metrics on MOT20."],"supporting_citations":[{"why":"Supplies the linear Gaussian Kalman filter whose prediction and update equations form the backbone of the association model.","marker":"[21]"},{"why":"Provides the Kalman smoothing recursion used to compute the marginal likelihood over the full frame batch in eq. (6).","marker":"[34]"},{"why":"Establishes the doubly stochastic normalization that converts the network's score matrix into a soft permutation matrix.","marker":"[37]"},{"why":"Gives the differentiable Sinkhorn-layer construction that lets gradients flow through the permutation matrix during end-to-end training.","marker":"[29]"},{"why":"Supplies the Tracktor-based preprocessing that converts raw public detections into the track-consistent inputs used across training and inference.","marker":"[2]"},{"why":"Serves as the leading self-supervised baseline with path-consistency training; the paper's EM objective is positioned against it and compared on MOT17.","marker":"[26]"},{"why":"Provides the cross-input consistency self-supervised baseline whose results the method improves on in HOTA and IDF1.","marker":"[1]"},{"why":"Supplies the YOLOX detections used for the private-detection MOT17 experiments and the BDD100K evaluation.","marker":"[43]"}],"fun_headline_variants":["Neural Kalman filter learns tracking without identity labels","EM-based self-supervised tracker with Sinkhorn normalization","Self-supervised MOT: no labels, state-of-the-art among label-free","Learning data association from unlabeled detections via Kalman smoothing"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The approach assumes that maximizing the likelihood of the observed detections under the network's own predicted associations—rather than averaging over all possible associations—still steers the network toward correct matches rather than degenerate ones.","fun_headline_variants_meta":{"raw":{"variants":["Neural Kalman filter learns tracking without identity labels","EM-based self-supervised tracker with Sinkhorn normalization","Self-supervised MOT: no labels, state-of-the-art among label-free","Learning data association from unlabeled detections via Kalman smoothing"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000382,"raw_usage":{"total_tokens":2014,"prompt_tokens":923,"completion_tokens":1091,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":539,"completion_tokens_details":{"reasoning_tokens":1020}},"tokens_in":539,"tokens_out":1091,"duration_ms":10245,"temperature":1.0,"reasoning_tokens":1020,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T18:26:12.743275+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"On a synthetic clip of two identical objects moving at constant velocity and crossing paths, the method should preserve each identity through the crossing; if the learned association network swaps them, the smoothed-likelihood objective is not rewarding the correct association.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the linear Gaussian Kalman filter whose prediction and update equations form the backbone of the association model."},{"cited_title":"Maximum likelihood estimates of linear dynamic systems","cited_arxiv_id":null,"evidence_quote":"Provides the Kalman smoothing recursion used to compute the marginal likelihood over the full frame batch in eq. (6)."},{"cited_title":"Learning latent permutations with gumbel-sinkhorn networks","cited_arxiv_id":null,"evidence_quote":"Gives the differentiable Sinkhorn-layer construction that lets gradients flow through the permutation matrix during end-to-end training."},{"cited_title":"Tracking without bells and whistles","cited_arxiv_id":null,"evidence_quote":"Supplies the Tracktor-based preprocessing that converts raw public detections into the track-consistent inputs used across training and inference."},{"cited_title":"Self-supervised multi-object tracking with path consistency","cited_arxiv_id":null,"evidence_quote":"Serves as the leading self-supervised baseline with path-consistency training; the paper's EM objective is positioned against it and compared on MOT17."},{"cited_title":"Self-supervised multi-object track- ing with cross-input consistency","cited_arxiv_id":null,"evidence_quote":"Provides the cross-input consistency self-supervised baseline whose results the method improves on in HOTA and IDF1."}],"review_version":1}