REVIEW 2 major objections 6 minor 1 cited by
3D Multi-Object Tracking with Semi-Supervised GRU-Kalman Filter
T0 review · 2 major / 6 minor · reviewed 2026-08-12 · deepseek-v4-flash
Pith's one-line read A learnable GRU-Kalman filter can replace hand-designed motion models in 3D multi-object tracking and match or beat traditional trackers on nuScenes and Argoverse2.
desk verdict Reasonable KalmanNet-for-3D-MOT idea, but the results are oversold and the EKF pseudo-label supervision undermines the central claim. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is the GRU-Kalman filter, a neural approximation of recursive Bayesian filtering in which three gated recurrent units replace the hand-set covariance updates of a Kalman filter: the first GRU tracks process noise covariance $Q_n$, the second tracks state error covariance $\hat{P}_{n|n-1}$, and the third tracks observation error covariance $\hat{S}_{n|n-1}$, with a fully connected output layer producing the Kalman gain $K_n$. A parallel extended Kalman filter supplies pseudo-labels $\tilde{X}_n$ for trajectories without annotation matches, and the training loss combines squared errors to ground-truth states $X_n$ and to these pseudo-labels. This design is what allows the system to avoid linearizing the state-transition function $f$ and observation function $h$ and to avoid assuming Gaussian noise, while keeping the interpretable predict-update structure of a Kalman filter.
What would settle it
Train the same GRU-Kalman filter twice on data with complete ground truth, such as synthetic trajectories with known nonlinear dynamics: once with the semi-supervised pseudo-label loss and once with only true labels. If the pseudo-label variant performs worse on the segments where the parallel EKF's predicted position differs most from the observed position, the semi-supervised strategy is propagating EKF errors rather than contributing useful signal.
Extended reading notes
Core claim
The central claim is that a learnable Kalman filter built from three GRUs can replace the manually specified state-space model and Gaussian noise assumptions in the motion module of a tracking-by-detection pipeline and achieve competitive 3D MOT accuracy. The filter ingests state and observation differences, infers the process noise covariance, state-error covariance, and observation-error covariance through separate GRUs, and produces the Kalman gain through an output layer; since the loss is differentiable with respect to the gain, the whole filter trains end-to-end on state error. To handle sparse annotations and imperfect annotation-trajectory association, the authors run a parallel EKF and use its state estimates as pseudo-labels for trajectories without ground-truth matches. On the nuScenes test set the method ties Poly-MOT's AMOTA of 70.0 with a better AMOTP (50.4 vs 50.9) and the best AMOTP among listed methods, and on Argoverse2 it raises average HOTA from 46.0 (greedy LT3D) and 42.7 (AB3DMOT LT3D) to 47.3. The ablation shows the GRU motion module is insensitive to which hand-designed state space it is paired with, while conventional filters change accuracy with that model choice.
Load-bearing premise
The load-bearing premise is that, for trajectories without ground-truth annotations, the parallel EKF's state estimates are accurate enough to act as training labels; if the EKF is wrong on exactly the hard or ambiguous cases, the GRU filter is trained to imitate those errors rather than learn the true motion.
Editorial extensions
If this is right
- Object classes can be added to a tracking system without designing a new motion model or retuning filter parameters for each class.
- The semi-supervised pseudo-label strategy converges in roughly 1,700 steps (about two epochs) versus three epochs for fully supervised training, and reaches a higher final AMOTA in the reported ablation.
- A model trained only on nuScenes transfers to Argoverse2 and beats the LT3D greedy and AB3DMOT baselines on average HOTA, indicating the learned motion dynamics generalize.
- Because the loss is differentiable with respect to the Kalman gain, the filter can be trained end-to-end from state errors without needing labels for intermediate covariance quantities.
Reading between the lines
- If the learned motion model truly removes per-class tuning, extending to rare or entirely new object classes should require only detection labels, not kinematic model engineering; this could be tested by training on a subset of Argoverse2 classes and measuring HOTA on held-out classes.
- The semi-supervised strategy could double as a diagnostic: comparing GRU-Kalman filter accuracy on trajectories with ground-truth labels versus EKF pseudo-labels would quantify how much EKF model error leaks into the learned filter.
- The same three-GRU Kalman architecture could plausibly replace hand-tuned Kalman filters in other perception modules such as multi-sensor fusion or trajectory prediction, since the machinery is not specific to 3D bounding boxes.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes a 3D multi-object tracking pipeline in which the motion module is a GRU-based learnable Kalman filter (following KalmanNet), intended to replace hand-designed per-class state-space models and manually tuned noise parameters. To cope with the partial annotation of MOT datasets, the authors add a semi-supervised loss that uses a parallel EKF's state estimates as pseudo-labels for trajectories without ground-truth associations. The method is evaluated on the nuScenes test and validation sets and on the Argoverse2 validation set, reporting AMOTA values comparable to Poly-MOT on nuScenes and improved HOTA over weak baselines on Argoverse2. An ablation indicates that the GRU motion module is robust to the choice of state-space model and that semi-supervised training converges faster than supervised-only training.
Significance. If the claims are fully supported, the paper offers a useful engineering contribution: a parameter-efficient way to avoid per-class motion-model design in tracking-by-detection systems, with released code. The nuScenes validation results are competitive with Poly-MOT, and the zero-shot application to Argoverse2 is interesting. However, the abstract's 'superior performance' claim is not supported by the strongest baselines in the paper's own Table I, and the pseudo-label mechanism using EKF outputs is not validated against its accuracy, which is central to the claimed benefit of avoiding model mismatch.
major comments (2)
- [§IV.C.1, Table I] The abstract's claim that the system exhibits superior performance compared to traditional TBD methods is contradicted by Table I. With the CenterPoint detector, Ours matches Poly-MOT on AMOTA (70.0) but has more ID switches (353 vs. 331). With LargeKernel3D, Ours is lower than the reported Poly-MOT on AMOTA (74.4 vs. 75.4), AMOTP (42.9 vs. 42.2), and IDS (352 vs. 292). Since Poly-MOT is a TBD method, the claim must be revised to 'comparable' or the experiments must isolate a setting where the advantage is consistent.
- [§III.B, Eq. (10), Fig. 4] The semi-supervised loss trains the GRU-Kalman filter to imitate a parallel EKF for trajectories without annotations. The paper motivates the GRU-KF precisely by the failure of linear-Gaussian models such as the EKF, so using EKF outputs as pseudo-labels risks teaching the GRU to reproduce the model mismatch it is meant to avoid. No evidence is provided that the EKF pseudo-labels are accurate on the unlabeled portion of the data; Fig. 4 shows only a single convergence curve with no variance and no comparison to alternative pseudo-label sources. Please add a pseudo-label accuracy analysis against keyframe ground truth and an ablation varying the pseudo-label source.
minor comments (6)
- [Table I footnote] The footnote says parentheses indicate the authors' reproduced results, but the table does not explain whether the comparison of 'Ours' against Poly-MOT should use the published numbers or the reproduced numbers; please clarify the convention in the caption.
- [Table IV] The last row under the Car section is labeled 'Bicycle + Ours'; it should presumably be 'Car + Ours'. Also, 'Bicycle CA + KF' would be clearer as 'Bicycle + CA + KF'.
- [§III.B, Eq. (10)] Equation (10) alternates between 'Kalman filter' and 'EKF'; define \tilde{X}_n explicitly as the output of the parallel EKF used as a pseudo-label and keep the terminology consistent throughout the section.
- [§IV.B] The implementation details omit the GRU hidden dimension, number of layers, and the exact dimension of the state vector fed to the GRU-KF; please add these for reproducibility.
- [Fig. 4] A single convergence trace is insufficient to support the claim of improved robustness; report the final AMOTA values, training steps, and ideally multiple runs with variance.
- [§IV.C.2] The conclusion that 'changing SS does not affect the tracking accuracy' of Ours is based on only two object categories; either add more categories or soften the claim.
Circularity Check
No significant circularity: the learnable filter is evaluated on external labeled benchmarks, and the EKF pseudo-label term is a transparent training choice, not a reduction of the central claim.
full rationale
The paper is an empirical tracking pipeline, not a derivation, and its central claim is supported by external evaluation: Tables I-III report AMOTA/HOTA/IDS on the nuScenes test/val and Argoverse2 val sets, which are labeled benchmarks outside the semi-supervised training loop. The GRU-Kalman filter architecture is adapted from KalmanNet [25], an external prior work, and the two-stage association from Poly-MOT [7], another external baseline; there are no load-bearing self-citations. The only step that could raise a circularity concern is Eq. (10), where unassociated trajectories are supervised by pseudo-labels from a parallel EKF (\tilde X_n). This is not a circular reduction: the GRU-KF output \hat X_n is not defined as the EKF output, the supervised term ||X_n - \hat X_n||^2 still anchors training to ground truth, and the final evaluation is on annotated frames that were not used to generate pseudo-labels. The EKF-imitation issue is a real limitation (the pseudo-label source embodies linear-Gaussian assumptions), but it is a modeling weakness, not a case where the claimed prediction is equivalent to its input by construction. Thus score 0.
Assumptions & free parameters
free parameters (4)
- EKF noise covariance matrices Q and R =
not reported
- GRU-Kalman architecture hyperparameters =
not reported
- Association thresholds for two-stage GIoU matching =
not reported
- Detector preprocessing parameters (score filter, NMS) =
not reported
assumptions (4)
- domain assumption A GRU network can emulate the Kalman gain computation well enough to capture nonlinear object motion in 3D MOT.
- domain assumption EKF pseudo-labels for trajectories without annotations are sufficiently reliable to supervise training.
- domain assumption The two-stage association correctly matches detections to trajectories so that the loss in Eqs. (8)-(10) is computed on corresponding states.
- domain assumption The GRU-KF trained on one detector (CenterPoint) transfers to another detector (LT3D) and another dataset (Argoverse2) without adaptation.
Cite this review
Pith. "Pith review of 3D Multi-Object Tracking with Semi-Supervised GRU-Kalman Filter." pith.science (2026). https://pith.science/paper/DWGRWVJ7
@misc{pith2026241108433,
author = {Pith},
title = {Pith review of: 3D Multi-Object Tracking with Semi-Supervised GRU-Kalman Filter},
year = {2026},
howpublished = {\url{https://pith.science/paper/DWGRWVJ7}},
note = {Machine review of arXiv:2411.08433}
}
read the original abstract
3D Multi-Object Tracking (MOT), a fundamental component of environmental perception, is essential for intelligent systems like autonomous driving and robotic sensing. Although Tracking-by-Detection frameworks have demonstrated excellent performance in recent years, their application in real-world scenarios faces significant challenges. Object movement in complex environments is often highly nonlinear, while existing methods typically rely on linear approximations of motion. Furthermore, system noise is frequently modeled as a Gaussian distribution, which fails to capture the true complexity of the noise dynamics. These oversimplified modeling assumptions can lead to significant reductions in tracking precision. To address this, we propose a GRU-based MOT method, which introduces a learnable Kalman filter into the motion module. This approach is able to learn object motion characteristics through data-driven learning, thereby avoiding the need for manual model design and model error. At the same time, to avoid abnormal supervision caused by the wrong association between annotations and trajectories, we design a semi-supervised learning strategy to accelerate the convergence speed and improve the robustness of the model. Evaluation experiment on the nuScenes and Argoverse2 datasets demonstrates that our system exhibits superior performance and significant potential compared to traditional TBD methods.
Figures
Forward citations
Cited by 1 Pith paper
-
Leveraging Consistent Spatio-Temporal Correspondence for Robust Visual Odometry
STVO improves visual odometry by combining temporal motion propagation and depth-based spatial attention to make multi-frame optical flow matching more consistent, setting state-of-the-art ATE on TUM-RGBD, EuRoC, ETH3...
Reference graph
Works this paper leans on
-
[1]
3d multi-object tracking: A baseline and new evaluation metrics,
X. Weng, J. Wang, D. Held, and K. Kitani, “3d multi-object tracking: A baseline and new evaluation metrics,” in 2020 IEEE/RSJ Interna- tional Conference on Intelligent Robots and Systems (IROS) . IEEE, 2020, pp. 10 359–10 366
2020
-
[2]
Center-based 3d object detec- tion and tracking,
T. Yin, X. Zhou, and P. Krahenbuhl, “Center-based 3d object detec- tion and tracking,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , 2021, pp. 11 784–11 793
2021
-
[3]
You only look once: Unified, real-time object detection,
J. Redmon, S. Divvala, R. Girshick, and A. Farhadi, “You only look once: Unified, real-time object detection,” in Proceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 779– 788
2016
-
[4]
End-to-end object detection with transformers,
N. Carion, F. Massa, G. Synnaeve, N. Usunier, A. Kirillov, and S. Zagoruyko, “End-to-end object detection with transformers,” in European conference on computer vision . Springer, 2020, pp. 213– 229
2020
-
[5]
V oxelnext: Fully sparse voxelnet for 3d object detection and tracking,
Y . Chen, J. Liu, X. Zhang, X. Qi, and J. Jia, “V oxelnext: Fully sparse voxelnet for 3d object detection and tracking,” pp. 21 674–21 683, 2023
work page 2023
-
[6]
Simpletrack: Understanding and rethinking 3d multi-object tracking,
Z. Pang, Z. Li, and N. Wang, “Simpletrack: Understanding and rethinking 3d multi-object tracking,” in European Conference on Computer Vision. Springer, 2022, pp. 680–696
2022
-
[7]
Poly-mot: A polyhedral framework for 3d multi-object tracking,
X. Li, T. Xie, D. Liu, J. Gao, K. Dai, Z. Jiang, L. Zhao, and K. Wang, “Poly-mot: A polyhedral framework for 3d multi-object tracking,” in 2023 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, 2023, pp. 9391–9398
2023
-
[8]
Joint multi-object detection and tracking with camera-lidar fusion for autonomous driving,
K. Huang and Q. Hao, “Joint multi-object detection and tracking with camera-lidar fusion for autonomous driving,” in 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) . IEEE, 2021, pp. 6983–6989
2021
Show all 31 references
-
[9]
Learnable online graph representations for 3d multi-object tracking,
J.-N. Zaech, A. Liniger, D. Dai, M. Danelljan, and L. Van Gool, “Learnable online graph representations for 3d multi-object tracking,” IEEE Robotics and Automation Letters , vol. 7, no. 2, pp. 5103–5110, 2022
2022
-
[10]
Eagermot: 3d multi-object tracking via sensor fusion,
A. Kim, A. O ˇsep, and L. Leal-Taix ´e, “Eagermot: 3d multi-object tracking via sensor fusion,” in 2021 IEEE International conference on Robotics and Automation (ICRA) . IEEE, 2021, pp. 11 315–11 321
2021
-
[11]
Camo-mot: Combined appearance-motion optimization for 3d multi-object tracking with camera-lidar fusion,
L. Wang, X. Zhang, W. Qin, X. Li, J. Gao, L. Yang, Z. Li, J. Li, L. Zhu, H. Wang, et al. , “Camo-mot: Combined appearance-motion optimization for 3d multi-object tracking with camera-lidar fusion,” IEEE Transactions on Intelligent Transportation Systems , vol. 24, no. 11, pp. ...
2023
-
[12]
A new approach to linear filtering and prediction problems,
R. E. Kalman, “A new approach to linear filtering and prediction problems,” Journal of Basic Engineering , p. 35–45. [Online]. Available: http://dx.doi.org/10.1115/1.3662552
-
[13]
Gruber, AN APPROACH TO TARGET TRACKING
M. Gruber, AN APPROACH TO TARGET TRACKING . [Online]. Available: http://dx.doi.org/10.21236/ad0654272
-
[14]
R. E. Larson, R. M. Dressler, and R. S. Ratner, Application of the extended kalman filter to ballistic trajectory estimation . Stanford Research Institute Menlo Park, 1967
1967
-
[15]
Empirical evaluation of gated recurrent neural networks on sequence modeling,
J. Chung, C. Gulcehre, K. Cho, and Y . Bengio, “Empirical evaluation of gated recurrent neural networks on sequence modeling,” arXiv preprint arXiv:1412.3555, 2014
2014 arXiv
-
[16]
Long short-term memory,
A. Graves and A. Graves, “Long short-term memory,” Supervised sequence labelling with recurrent neural networks , pp. 37–45, 2012
2012
-
[17]
nuscenes: A multimodal dataset for autonomous driving,
H. Caesar, V . Bankiti, A. H. Lang, S. V ora, V . E. Liong, Q. Xu, A. Krishnan, Y . Pan, G. Baldan, and O. Beijbom, “nuscenes: A multimodal dataset for autonomous driving,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , 2020, pp. 11 621–11 631
2020
-
[18]
Argoverse 2: Next generation datasets for self-driving perception and forecasting,
B. Wilson, W. Qi, T. Agarwal, J. Lambert, J. Singh, S. Khandelwal, B. Pan, R. Kumar, A. Hartnett, J. K. Pontes, et al., “Argoverse 2: Next generation datasets for self-driving perception and forecasting,” arXiv preprint arXiv:2301.00493, 2023
2023 arXiv
-
[19]
Motiontrack: end-to-end transformer-based multi-object tracking with lidar-camera fusion,
C. Zhang, C. Zhang, Y . Guo, L. Chen, and M. Happold, “Motiontrack: end-to-end transformer-based multi-object tracking with lidar-camera fusion,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2023, pp. 151–160
2023
-
[20]
Attention is all you need,
A. Vaswani, “Attention is all you need,” Advances in Neural Informa- tion Processing Systems , 2017
2017
-
[21]
Transfusion: Robust lidar-camera fusion for 3d object detection with transformers,
X. Bai, Z. Hu, X. Zhu, Q. Huang, Y . Chen, H. Fu, and C.-L. Tai, “Transfusion: Robust lidar-camera fusion for 3d object detection with transformers,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , 2022, pp. 1090–1099
2022
-
[22]
Learning a neural solver for multiple object tracking,
G. Bras ´o and L. Leal-Taix ´e, “Learning a neural solver for multiple object tracking,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , 2020, pp. 6247–6257
2020
-
[23]
Kfnet: Learning temporal camera relocalization using kalman filtering,
L. Zhou, Z. Luo, T. Shen, J. Zhang, M. Zhen, Y . Yao, T. Fang, and L. Quan, “Kfnet: Learning temporal camera relocalization using kalman filtering,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , 2020, pp. 4919–4928
2020
-
[24]
Combining generative and discriminative models for hybrid inference,
V . Satorras, Z. Akata, and M. Welling, “Combining generative and discriminative models for hybrid inference,” Neural Information Pro- cessing Systems,Neural Information Processing Systems , Jun 2019
2019
-
[25]
Kalmannet: Neural network aided kalman filtering for partially known dynamics,
G. Revach, N. Shlezinger, X. Ni, A. L. Escoriza, R. J. Van Sloun, and Y . C. Eldar, “Kalmannet: Neural network aided kalman filtering for partially known dynamics,” IEEE Transactions on Signal Processing , vol. 70, pp. 1532–1547, 2022
2022
-
[26]
Score refinement for confidence-based 3d multi-object tracking,
N. Benbarka, J. Schr ¨oder, and A. Zell, “Score refinement for confidence-based 3d multi-object tracking,” in 2021 IEEE/RSJ Inter- national Conference on Intelligent Robots and Systems (IROS). IEEE, 2021, pp. 8083–8090
2021
-
[27]
Tracking objects as points,
X. Zhou, V . Koltun, and P. Kr ¨ahenb¨uhl, “Tracking objects as points,” in European conference on computer vision . Springer, 2020, pp. 474–490
2020
-
[28]
Cascade r-cnn: Delving into high quality object detection,
Z. Cai and N. Vasconcelos, “Cascade r-cnn: Delving into high quality object detection,” in Proceedings of the IEEE conference on computer vision and pattern recognition , 2018, pp. 6154–6162
2018
-
[29]
Scaling up kernels in 3d cnns,
Y . Chen, J. Liu, X. Qi, X. Zhang, J. Sun, and J. Jia, “Scaling up kernels in 3d cnns,” arXiv preprint arXiv:2206.10555 , vol. 1, no. 2, p. 5, 2022
2022 arXiv
-
[30]
Towards long-tailed 3d detection,
N. Peri, A. Dave, D. Ramanan, and S. Kong, “Towards long-tailed 3d detection,” in Conference on Robot Learning . PMLR, 2023, pp. 1904–1915
2023
-
[31]
Argoverse: 3d tracking and forecasting with rich maps,
M.-F. Chang, J. W. Lambert, P. Sangkloy, J. Singh, S. Bak, A. Hartnett, D. Wang, P. Carr, S. Lucey, D. Ramanan, and J. Hays, “Argoverse: 3d tracking and forecasting with rich maps,” in Conference on Computer Vision and Pattern Recognition (CVPR) , 2019
2019
Reviewed August 12, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.