REVIEW 4 major objections 6 minor 42 references
HybridTrack: A Hybrid Approach for Robust Multi-Object Tracking
T0 review · 4 major / 6 minor · reviewed 2026-08-10 · deepseek-v4-flash
Pith's one-line read This paper claims that a learnable Kalman filter can replace manually designed motion and noise models in 3D multi-object tracking, reaching 82.72% HOTA on the KITTI dataset while running at 112 FPS.
desk verdict Solid learnable-Kalman 3D MOT with a real contribution but an overreaching abstract and one unbacked robustness claim. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the learnable Kalman filter (LKF). Its two learned components are the Transition Residual Predictor (TRP), an encoder-decoder that consumes the current trajectory plus recent state differences and the previous Kalman correction, and outputs residual updates $\Delta xyz$, $\Delta whl$, and $\Delta\theta$; and the Kalman Gain Estimation Module, an RNN that outputs the gain $K^j_k$ used in the update equation, replacing the classical computation from process and measurement noise covariances. A dynamic scaling factor $\alpha_k$ multiplies the learned residual before it is added to the previous state, limiting overshoot during initial noisy detections and during missed-detection gaps. This design keeps the recursive update of a Kalman filter intact, so the filter still produces a posterior state from a prediction and a measurement, but the motion and noise models are now functions learned from data.
What would settle it
Take the released HybridTrack model, replace the KITTI detector outputs with detections whose positions, sizes, and headings have added Gaussian noise of increasing variance, and measure HOTA and association accuracy. If accuracy collapses or tracks fragment well before the noise level that a hand-tuned Kalman filter tolerates, the clean-training transfer assumption is falsified.
Extended reading notes
Core claim
The central claim is that a Kalman filter whose transition residual and Kalman gain are learned from data can match or exceed hand-tuned model-based trackers in accuracy while running faster and generalizing without scene-specific tuning. In HybridTrack, the prior state is computed as $\hat{x}^j_k = \alpha_k S^j_k + x^j_{k-1}$, where $S^j_k$ is the output of the Transition Residual Predictor (a small encoder-decoder with three MLPs producing residuals for position, box dimensions, and heading) and $\alpha_k$ is a dynamic scaling factor that ramps up over the first frames and shrinks after missed detections. The update step is $x^j_k = \hat{x}^j_k + K^j_k (r^i_k - H \hat{x}^j_k)$, with the gain $K^j_k$ produced by an RNN-based module rather than computed from manually set covariances. Trained end-to-end with L1, temporal-smoothness, and direction-consistency losses on clean 20-frame ground-truth tracklets, the system is reported to reach 82.72% HOTA on the KITTI test set, 86.92% association accuracy, and 112 FPS, outperforming all but one of the compared methods while using only LiDAR, not camera-LiDAR fusion.
Load-bearing premise
The learned motion and gain networks are trained on clean ground-truth tracklets and are assumed to work when fed noisy detections from an external 3D detector at inference; the paper states that noise augmentation had no significant impact but does not show that experiment.
Editorial extensions
If this is right
- If the central claim holds, porting a tracker to a new sensor suite or a new class of road users reduces to collecting tracklets and retraining the two small networks, rather than hand-deriving motion models and tuning noise covariances.
- The one-step greedy association with a 3D CIoU cost reaches 86.92% association accuracy, a level comparable to two-step association methods, suggesting that accurate learned predictions simplify the matching problem.
- The reported data efficiency (about 81% HOTA from only 320 training timesteps) implies the approach can be applied in domains where large annotated tracking datasets do not exist, such as accident scenes.
- Because the tracker runs at roughly 99-112 FPS on a single desktop GPU and is LiDAR-only, it is positioned for real-time deployment in ADAS without requiring camera-LiDAR synchronization or fusion.
Reading between the lines
- A direct test of the transfer assumption would be to train the TRP and gain modules on ground-truth tracklets, then feed them detections with artificially increased localization noise and measure HOTA; the paper states that noise augmentation had no significant impact but does not report the experiment, so this test could settle whether the learned components are miscalibrated under real detector
- The dynamic scaling factor $\alpha$ is hand-scheduled with empirically chosen bounds; one could infer that a learned confidence or uncertainty estimator could replace this schedule, making the method fully parameter-free and potentially improving early-track stability.
- If the residual predictor is class-agnostic, the same architecture should transfer to pedestrians and cyclists by retraining on their tracklets, which would extend the claim to mixed-traffic ADAS; this is an extrapolation beyond the paper's vehicle-only experiments.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. HybridTrack proposes a 3D multi-object tracker for vehicles that embeds two learned components into a Kalman-filter tracking-by-detection pipeline: a Transition Residual Predictor (TRP) that replaces the hand-designed motion model, and a Kalman Gain Estimation Module that replaces manually set noise covariances. The method is trained on clean ground-truth tracklets from KITTI and evaluated on the KITTI test set using detections from an external 3D detector (VirConv or CasA). The paper reports 82.72% HOTA, claims to significantly outperform state-of-the-art methods, reports 112 FPS in one configuration, and includes ablations on detector choice, cost functions, learned components, and dataset size.
Significance. If the central claims held, the contribution would be practically valuable: a lightweight, data-driven Kalman filter that retains the recursive structure of classical tracking while removing manual motion-model and noise-covariance design. The paper includes several strengths: the code is publicly released, the method is evaluated on the standard KITTI benchmark with standard metrics, and the ablation study on training-set size is a useful data-efficiency analysis. However, the significance is currently overstated. The HOTA result is statistically indistinguishable from the two top published methods, the 112 FPS figure comes from a non-default cost configuration, and the claimed robustness to detection noise rests on an experiment that is mentioned but not reported. These issues are fixable, but they affect the paper's main claims.
major comments (4)
- [Abstract and Table I] The abstract and contribution list claim that HybridTrack 'significantly outperforms state-of-the-art methods' with 82.72% HOTA, but Table I shows MCtrack at 82.75 and BiTrack at 82.70. A 0.03-point gap is not a significant improvement in any statistical sense, and the paper should either revise the claim to 'competitive with state-of-the-art' or provide significance testing. This is load-bearing because the headline contribution is the performance claim.
- [Abstract and Table IV] The 112 FPS speed claim in the abstract and introduction corresponds to the L2-error cost configuration (112.83 FPS in Table IV), not to the main 3D-CIoU configuration, which runs at 98.91 FPS. The paper should attribute the speed to the specific configuration and should not present the fastest configuration as the speed of the overall proposed method without qualification.
- [Section III-H and Table V] The training procedure uses clean ground-truth pose sequences, while inference consumes detections from an external detector, yet the paper states that 'no significant impact on performance is observed when augmenting model sequences input with noise' without reporting the experiment. This is load-bearing because the TRP and Kalman gain networks never see detection noise, missed detections, or association errors during training, and Table V shows that the learned update step alone collapses to 54.32 HOTA when prediction quality is poor. The authors should report the noise-augmentation experiment or an evaluation with pseudo-tracklets built from VirConv detections; without it, the claimed robustness to detection noise is unverified.
- [Section IV, Implementation paragraph] The claim that HybridTrack 'eliminates manual motion and stochastic parameter modeling' is contradicted by the hand-set inference parameters listed in the implementation: Tmax = 8, amin = 0.1, amax = 0.9, mu_max = 22, sigma = 5, and the CIoU threshold 1.20, in addition to the training hyperparameters (learning rate, weight decay, batch size, iterations). These are exactly the kind of empirically tuned choices the paper claims to remove. The authors should either provide a sensitivity analysis for these parameters or soften the claim to 'reduces manual motion and noise modeling' rather than 'eliminates' it.
minor comments (6)
- [Section III-D, Eq. (2)] The scaling factor alpha_k is defined in Eq. (1) and then redefined in Eq. (2); please clarify whether Eq. (2) multiplicatively modifies the alpha from Eq. (1) and specify the exact update when both conditions apply.
- [Section III-C] The initialization of prior states as x_{k0} - epsilon needs a definition of epsilon and an explanation of how it avoids duplicate states without distorting the first prediction.
- [Section III-D] The parameter P is described as 'any integer' in a range, but the TRP architecture presumably uses a fixed P in the experiments; please state the value used.
- [Section IV, Table IV] The caption says 'The results are reported in %', but the FPS column is in frames per second; please fix the caption or label the columns accordingly.
- [Section V, Figure 3] The text refers to '16 sequences (320 timesteps)' and '16,000 sequences', which is confusing; please specify whether the x-axis is in sequences, frames, or tracklets, and label the axes consistently.
- [Table I] Several rows have missing entries (e.g., UCMCTrack lacks DetA, DetRe, DetPr, LoCA); please state whether these values are unavailable or use placeholder symbols, and unify the abbreviation LocA/LoCA.
Circularity Check
No significant circularity: HybridTrack's learned Kalman components are trained on ground-truth tracklets and evaluated on the external KITTI benchmark, with no load-bearing self-citation or fitted-input-as-prediction step.
full rationale
The paper's derivation chain is self-contained with respect to the central claim. The transition residual predictor is trained with explicit supervised losses (Eqs. 5-8) on ground-truth tracklets, and the Kalman gain is inferred by an RNN rather than solved from the test set; the prior state in Eq. (3) is a learned residual added to the previous posterior, not a rearrangement of the target HOTA metric. The update in Eq. (4) is a standard Kalman correction with a learned gain. All hyperparameters (Tmax, amin, amax, mu_max, sigma, tau_3D) are hand-set on the validation set in a conventional manner and do not amount to fitting the test-set HOTA. The comparison against state-of-the-art methods is made on the external KITTI test set, and the reported HOTA values are not derived from the method's own training objective. The one self-citation, MonoKalman [18], appears only in the related-work discussion of extended Kalman filters and is not load-bearing. The paper's unsupported assertion that adding noise during training has 'no significant impact' is a generalizability/correctness concern, not a circularity: the claim is unverified but the evaluation itself remains an external benchmark. Therefore, no specific circular step can be quoted and exhibited, and the appropriate finding is no significant circularity.
Assumptions & free parameters
free parameters (8)
- Tmax =
8
- amin / amax =
0.1 / 0.9
- mu_max =
22
- sigma =
5
- tau_3D =
1.20
- lambda_states, lambda_temp, lambda_dir =
1, 1, 1
- learning_rate, weight_decay, batch_size, iterations =
0.001, 1e-5, 128, 1500
- epsilon (initial state perturbation) =
unspecified
assumptions (4)
- standard math Kalman filter update with identity observation model
- domain assumption CIoU is a suitable association cost for 3D boxes
- domain assumption The 7-element state [x, y, z, w, l, h, theta] is sufficient to represent a vehicle for tracking
- domain assumption Motion models trained on clean ground truth generalize to noisy detections
Cite this review
Pith. "Pith review of HybridTrack: A Hybrid Approach for Robust Multi-Object Tracking." pith.science (2026). https://pith.science/paper/ABL752WM
@misc{pith2026250101275,
author = {Pith},
title = {Pith review of: HybridTrack: A Hybrid Approach for Robust Multi-Object Tracking},
year = {2026},
howpublished = {\url{https://pith.science/paper/ABL752WM}},
note = {Machine review of arXiv:2501.01275}
}
read the original abstract
The evolution of Advanced Driver Assistance Systems (ADAS) has increased the need for robust and generalizable algorithms for multi-object tracking. Traditional statistical model-based tracking methods rely on predefined motion models and assumptions about system noise distributions. Although computationally efficient, they often lack adaptability to varying traffic scenarios and require extensive manual design and parameter tuning. To address these issues, we propose a novel 3D multi-object tracking approach for vehicles, HybridTrack, which integrates a data-driven Kalman Filter (KF) within a tracking-by-detection paradigm. In particular, it learns the transition residual and Kalman gain directly from data, which eliminates the need for manual motion and stochastic parameter modeling. Validated on the real-world KITTI dataset, HybridTrack achieves 82.72% HOTA accuracy, significantly outperforming state-of-the-art methods. We also evaluate our method under different configurations, achieving the fastest processing speed of 112 FPS. Consequently, HybridTrack eliminates the dependency on scene-specific designs while improving performance and maintaining real-time efficiency. The code is publicly available at: https://github.com/leandro-svg/HybridTrack.
Figures
Reference graph
Works this paper leans on
-
[1]
Learning to track with object permanence,
P. Tokmakov, J. Li, W. Burgard, and A. Gaidon, “Learning to track with object permanence,” in ICCV, 2021, pp. 10 860–10 869
work page 2021
-
[2]
Object Permanence Emerges in a Random Walk along Memory
P. Tokmakov, A. Jabri, J. Li, and A. Gaidon, “Object perma- nence emerges in a random walk along memory,” arXiv preprint arXiv:2204.01784, 2022
work page Pith review arXiv 2022
-
[3]
Tracklet proposal network for multi-object tracking on point clouds
H. Wu, Q. Li, C. Wen, X. Li, X. Fan, and C. Wang, “Tracklet proposal network for multi-object tracking on point clouds.” in IJCAI, 2021, pp. 1165–1171
work page 2021
-
[4]
A multi-modal fusion-based 3d multi-object tracking framework with joint detection,
X. Wang et al. , “A multi-modal fusion-based 3d multi-object tracking framework with joint detection,” IEEE RA-L , pp. 1–8, 2024
work page 2024
-
[5]
Finding and tracking people from the bottom up,
D. Ramanan and D. Forsyth, “Finding and tracking people from the bottom up,” in CVPR, vol. 2, 2003, pp. II–II
work page 2003
-
[6]
Pv-rcnn: Point-voxel feature set abstraction for 3d object detection,
S. Shi, C. Guo, L. Jiang, Z. Wang, J. Shi, X. Wang, and H. Li, “Pv-rcnn: Point-voxel feature set abstraction for 3d object detection,” in CVPR, 2020, pp. 10 529–10 538
work page 2020
-
[7]
Virtual sparse convolution for multimodal 3d object detection,
H. Wu, C. Wen, S. Shi, X. Li, and C. Wang, “Virtual sparse convolution for multimodal 3d object detection,” in CVPR, 2023, pp. 21 653–21 662
work page 2023
-
[8]
Casa: A cascade attention network for 3-d object detection from lidar point clouds,
H. Wu, J. Deng, C. Wen, X. Li, C. Wang, and J. Li, “Casa: A cascade attention network for 3-d object detection from lidar point clouds,” IEEE T-GRS, vol. 60, pp. 1–11, 2022
work page 2022
Show all 42 references
-
[9]
An introduction to the kalman filter,
G. Welch, G. Bishop et al., “An introduction to the kalman filter,” 1995
1995
-
[10]
Bayesian filtering: From kalman filters to particle filters, and beyond,
Z. Chen et al., “Bayesian filtering: From kalman filters to particle filters, and beyond,” Statistics, vol. 182, no. 1, pp. 1–69, 2003
2003
-
[11]
3d multi-object tracking: A baseline and new evaluation metrics,
X. Weng, J. Wang, D. Held, and K. Kitani, “3d multi-object tracking: A baseline and new evaluation metrics,” in 2020 IEEE/RSJ IROS . IEEE, 2020, pp. 10 359–10 366
2020
-
[12]
3d multi-object tracking in point clouds based on prediction confidence-guided data association,
H. Wu, W. Han, C. Wen, X. Li, and C. Wang, “3d multi-object tracking in point clouds based on prediction confidence-guided data association,” IEEE T-ITS, vol. 23, no. 6, pp. 5668–5677, 2021
2021
-
[13]
Observation- centric sort: Rethinking sort for robust multi-object tracking,
J. Cao, J. Pang, X. Weng, R. Khirodkar, and K. Kitani, “Observation- centric sort: Rethinking sort for robust multi-object tracking,” in CVPR, 2023, pp. 9686–9696
2023
-
[14]
Camo-mot: Combined appearance-motion optimization for 3d multi-object tracking with camera-lidar fusion,
L. Wang et al., “Camo-mot: Combined appearance-motion optimization for 3d multi-object tracking with camera-lidar fusion,” IEEE T-ITS , 2023
2023
-
[15]
A novel adaptive noise covariance matrix estimation and filtering method: Application to mul- tiobject tracking,
C. Jiang, Z. Wang, H. Liang, and Y . Wang, “A novel adaptive noise covariance matrix estimation and filtering method: Application to mul- tiobject tracking,” IEEE T-IV, vol. 9, no. 1, pp. 626–641, 2023
2023
-
[16]
Qdtrack: Quasi-dense similarity learning for appearance-only multiple object tracking,
T. Fischer et al. , “Qdtrack: Quasi-dense similarity learning for appearance-only multiple object tracking,” IEEE T-PAMI, 2023
2023
-
[17]
Simple online and realtime tracking with a deep association metric,
N. Wojke, A. Bewley, and D. Paulus, “Simple online and realtime tracking with a deep association metric,” in 2017 IEEE ICIP . IEEE, 2017, pp. 3645–3649
2017
-
[18]
Monokalman: Monocular vehicle pose estimation with kalman filter-based temporal consistency,
L. Di Bella, Y . Lyu, B. Cornelis, and A. Munteanu, “Monokalman: Monocular vehicle pose estimation with kalman filter-based temporal consistency,” in 2024 25th IEEE MDM . IEEE, 2024, pp. 247–250
2024
-
[19]
Mono-camera 3d multi-object tracking using deep learning detections and pmbm filtering,
S. Scheidegger, J. Benjaminsson, E. Rosenberg, A. Krishnan, and K. Granstr ¨om, “Mono-camera 3d multi-object tracking using deep learning detections and pmbm filtering,” in 2018 IEEE IV Symp. IEEE, 2018, pp. 433–440
2018
-
[20]
Ucmctrack: Multi-object tracking with uniform camera motion compensation,
K. Yi et al. , “Ucmctrack: Multi-object tracking with uniform camera motion compensation,” in AAAI, vol. 38, no. 7, 2024, pp. 6702–6710
2024
-
[21]
Pmtrack: Multi-object tracking with motion-aware,
X. Guo, Y . Zheng, and D. Wang, “Pmtrack: Multi-object tracking with motion-aware,” in ACCV, 2024, pp. 3091–3106
2024
-
[22]
Exploiting multi- modal synergies for enhancing 3d multi-object tracking,
X. Xu, W. Ren, X. Chen, H. Fan, Z. Han, and H. Liu, “Exploiting multi- modal synergies for enhancing 3d multi-object tracking,” IEEE RA-L , 2024
2024
-
[23]
Bitrack: Bidirectional offline 3d multi-object tracking using camera-lidar data,
K. Huang, M. Zhang, and Q. Hao, “Bitrack: Bidirectional offline 3d multi-object tracking using camera-lidar data,” arXiv preprint arXiv:2406.18414, 2024
2024 arXiv
-
[24]
Fast-poly: A fast polyhedral framework for 3d multi-object tracking,
X. Li, D. Liu, Y . Wu, X. Wu, L. Zhao, and J. Gao, “Fast-poly: A fast polyhedral framework for 3d multi-object tracking,” arXiv preprint arXiv:2403.13443, 2024
2024 arXiv
-
[25]
3d multi-object tracking based on informatic divergence-guided data association,
J. He, C. Fu, X. Wang, and J. Wang, “3d multi-object tracking based on informatic divergence-guided data association,” Signal Processing, vol. 222, p. 109544, 2024
2024
-
[26]
Deepfusionmot: A 3d multi-object tracking framework based on camera-lidar fusion with deep association,
X. Wang, C. Fu, Z. Li, Y . Lai, and J. He, “Deepfusionmot: A 3d multi-object tracking framework based on camera-lidar fusion with deep association,” IEEE RA-L , vol. 7, no. 3, pp. 8260–8267, 2022
2022
-
[27]
Triplettrack: 3d object tracking using triplet embeddings and lstm,
N. Marinello, M. Proesmans, and L. Van Gool, “Triplettrack: 3d object tracking using triplet embeddings and lstm,” in CVPR, 2022, pp. 4500– 4510
2022
-
[28]
Apptracker+: Dis- placement uncertainty for occlusion handling in low-frame-rate multiple object tracking,
T. Zhou, Q. Ye, W. Luo, H. Ran, Z. Shi, and J. Chen, “Apptracker+: Dis- placement uncertainty for occlusion handling in low-frame-rate multiple object tracking,” IJCV, pp. 1–26, 2024
2024
-
[29]
Dino-mot: 3d multi-object tracking with visual foundation model for pedestrian re- identification using visual memory mechanism,
M. Y . Lee, C. D. W. Lee, J. Li, and M. H. Ang Jr, “Dino-mot: 3d multi-object tracking with visual foundation model for pedestrian re- identification using visual memory mechanism,” IEEE RA-L , 2024
2024
-
[30]
Polarmot: How far can geometric relations take us in 3d multi-object tracking?
A. Kim, G. Bras ´o, A. O ˇsep, and L. Leal-Taix ´e, “Polarmot: How far can geometric relations take us in 3d multi-object tracking?” in ECCV. Springer, 2022, pp. 41–58
2022
-
[31]
Towards robust reference system for autonomous driving: Rethinking 3d mot,
L. Wang, J. Zhang, P. Cai, and X. Lil, “Towards robust reference system for autonomous driving: Rethinking 3d mot,” in 2023 IEEE ICRA . IEEE, 2023, pp. 8319–8325
2023
-
[32]
Enhancing geometric factors in model learning and inference for object detection and instance segmentation,
Z. Zheng et al. , “Enhancing geometric factors in model learning and inference for object detection and instance segmentation,” IEEE Trans. Cybern., vol. 52, no. 8, pp. 8574–8586, 2021
2021
-
[33]
Kalmannet: Neural network aided kalman filtering for partially known dynamics,
G. Revach, N. Shlezinger, X. Ni, A. L. Escoriza, R. J. Van Sloun, and Y . C. Eldar, “Kalmannet: Neural network aided kalman filtering for partially known dynamics,” IEEE T-SP, vol. 70, pp. 1532–1547, 2022
2022
-
[34]
Finding structure in time,
J. L. Elman, “Finding structure in time,” Cognitive science , vol. 14, no. 2, pp. 179–211, 1990
1990
-
[35]
Pnas-mot: Multi-modal object tracking with pareto neural architecture search,
C. Peng et al. , “Pnas-mot: Multi-modal object tracking with pareto neural architecture search,” IEEE RA-L , 2024
2024
-
[36]
Mctrack: A unified 3d multi-object tracking framework for autonomous driving,
X. Wang et al., “Mctrack: A unified 3d multi-object tracking framework for autonomous driving,” arXiv preprint arXiv:2409.16149 , 2024
2024 arXiv
-
[37]
Are we ready for autonomous driving? the kitti vision benchmark suite,
A. Geiger, P. Lenz, and R. Urtasun, “Are we ready for autonomous driving? the kitti vision benchmark suite,” in CVPR. IEEE, 2012, pp. 3354–3361
2012
-
[38]
Adam: A method for stochastic optimization,
D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” CoRR, vol. abs/1412.6980, 2014
2014 arXiv
-
[39]
Hota: A higher order metric for evaluating multi-object tracking,
J. Luiten et al., “Hota: A higher order metric for evaluating multi-object tracking,” IJCV, vol. 129, pp. 548–578, 2021
2021
-
[40]
Evaluating multiple object tracking performance: The clear mot metrics,
K. Bernardin and R. Stiefelhagen, “Evaluating multiple object tracking performance: The clear mot metrics,” EURASIP J. Image Video Process., vol. 2008, pp. 1–10, 2008
2008
-
[41]
Performance measures and a data set for multi-target, multi-camera tracking,
E. Ristani, F. Solera, R. Zou, R. Cucchiara, and C. Tomasi, “Performance measures and a data set for multi-target, multi-camera tracking,” in ECCV. Springer, 2016, pp. 17–35
2016
-
[42]
Second: Sparsely embedded convolutional detection,
Y . Yan, Y . Mao, and B. Li, “Second: Sparsely embedded convolutional detection,” Sensors, vol. 18, no. 10, p. 3337, 2018
2018
Reviewed August 10, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.