REVIEW 4 major objections 6 minor 26 references
IMM-MOT: A Novel 3D Multi-object Tracking Framework with Interacting Multiple Model Filter
T0 review · 4 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read This paper claims 3D multi-object tracking improves by blending four motion models with an Interacting Multiple Model filter, reaching 73.8% AMOTA on NuScenes Val.
desk verdict Central claim is plausible but unverifiable from the text: the IMM module's reported gain depends on a state-vector unification and mixing step that Algorithm 1 never defines. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the Interacting Multiple Model (IMM) filter, an estimation scheme that runs several motion models in parallel and outputs a probability-weighted mixture of their predictions. Here it carries the argument because it lets the tracker switch between constant velocity (CV), constant acceleration (CA), constant turn rate and velocity (CTRV), and constant turn rate and acceleration (CTRA) at runtime, rather than committing to one model per object class. The paper unifies the four state vectors into one common state and fuses the predictions as $\hat{X}_t = \sum_i \mu^i_{t-1} \hat{X}^i_t$, updating the mixing probabilities $\mu^i$ from association residuals via a Markov transition matrix. Two supporting mechanisms complete the framework: the Damping Window score function $s(t)$ judges a trajectory's life-cycle state from its whole association history, and the Distance-Based Score Enhancement rescales detector scores by distance so the score filter separates true from false detections more cleanly.
What would settle it
Open the released code and locate the state-vector unification between the CV/CA models and the CTRV/CTRA models, then run the IMM fusion on a NuScenes sequence with a clear left turn. If the fused trajectory jumps or becomes physically impossible during the turn, or if replacing the IMM with a single CTRA model does not reduce AMOTA by roughly the reported 0.5 points, the central claim is contradicted.
Extended reading notes
Core claim
The central claim is that a single motion model, even one chosen per object class, cannot track a maneuvering object well because the same target changes regime over its lifetime. IMM-MOT therefore runs constant velocity, constant acceleration, constant turn rate and velocity, and constant turn rate and acceleration models in parallel for each target, fuses their predictions by model probability, and updates those probabilities from association residuals through a Markov transition matrix. Two supporting modules, the Damping Window and Distance-Based Score Enhancement, handle trajectory birth and death and detection-score calibration. The paper reports this design as the main source of improvement: 73.8% AMOTA on NuScenes Val, the highest value it knows for methods built on the CenterPoint detector, with the IMM module contributing +0.5% over the Poly-MOT baseline and a larger +0.7% gain on test.
Load-bearing premise
The load-bearing premise is that the four motion models' different state vectors can be converted into one common state so that their probability-weighted average is a physically meaningful prediction; the paper states that this unification is done but does not give the transformation.
Editorial extensions
If this is right
- On the NuScenes validation set, IMM-MOT reports 73.8% AMOTA with CenterPoint detections, above Fast-Poly's 73.7% and Poly-MOT's 73.1%.
- The IMM module itself contributes +0.5% AMOTA on validation and +0.7% on test when it replaces the per-class single motion model, with gains spread across bicycle, bus, motorcycle, and truck classes.
- The Damping Window mechanism reduces false negatives for cars by about 14.5%, keeping low-confidence true targets alive longer.
- The Distance-Based Score Enhancement reduces false positives by roughly 4% by amplifying the score gap between nearby reliable detections and distant unreliable ones.
- With all three modules, the framework also improves Most Tracked and Most Lost counts compared with the Poly-MOT baseline.
Reading between the lines
- Beyond the paper, the IMM prediction module is detector-agnostic in principle, so it could replace the per-class single-motion predictor in other tracking-by-detection systems and likely transfer its AMOTA gain without retraining the detector.
- The DBSE design assumes LiDAR confidence declines with distance; on camera or radar inputs, or on a detector with different score calibration, the same weighting could hurt, and a learned distance-to-score mapping would be a natural extension.
- A direct comparison the paper does not make is against a learned motion predictor fed the same CenterPoint detections; if a learned model matched or beat the IMM gain, the improvement would be attributed to adaptive prediction generally rather than to the specific IMM mechanism.
- The Damping Window's tolerance for fragmented tracks suggests it could help pedestrian tracking under occlusion, but the paper tunes it only for bus, car, pedestrian, and trailer on NuScenes, so its behavior in dense crowds remains untested.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper presents IMM-MOT, a tracking-by-detection 3D multi-object tracking framework evaluated on the nuScenes Val split. Three modules are added on top of the Poly-MOT baseline: an Interacting Multiple Model (IMM) filter that fuses CV, CA, CTRV, and CTRA motion models; a Damping Window (DW) trajectory lifecycle mechanism; and a Distance-Based Score Enhancement (DBSE) preprocessing module. The paper reports 73.8% AMOTA with CenterPoint detections, claims the highest known value for CenterPoint-based single-modal methods, and presents ablations showing positive AMOTA increments for each module.
Significance. If the method is fully and correctly specified, this is a useful engineering contribution to LiDAR-only 3D MOT: the idea of replacing a single per-class motion model with an adaptive multiple-model filter is well motivated, and the reported gains on nuScenes Val, though modest, are consistent across the ablation rows. The release of code is a strength, as is the per-module ablation structure. The central risk is that the IMM component, which accounts for the largest reported improvement, is not described at the level of detail needed for reproduction or even for verification that the algorithm is an IMM. The DW module also depends on an unspecified damping function. The manuscript is therefore not currently self-contained.
major comments (4)
- [III.A, Algorithm 1, Table I] The IMM state fusion is not defined. Table I lists state vectors of different dimensions and different kinematic representations: CV/CA use Cartesian velocity/acceleration components, while CTRV/CTRA use speed, heading, and turn rate. The paper states that the state vectors are unified but gives no transformation, no inverse mapping, and no covariance transformation. Consequently, the fused prediction in Algorithm 1, Xhat_t = sum_i mu_i^{t-1} Xhat_i^t, is dimensionally undefined unless an explicit conversion is inserted. The abstract and Section IV.D attribute the main gain to IMM, so this gap directly affects the central claim. The authors must provide the exact state transformation and the corresponding covariance transformation, or otherwise show how the weighted average is computed.
- [III.A, Algorithm 1] Algorithm 1 is not the standard IMM algorithm and is incomplete as written. A standard IMM has an interaction/mixing step before prediction, in which each mode filter is reinitialized with a mode-conditioned mixed state and mixed covariance using the Markov transition probabilities; Algorithm 1 instead predicts directly from a single X_{t-1} and only weight-fuses the predicted states. The algorithm also does not maintain model-conditioned states and covariances, does not define the innovation covariance or the likelihood distribution used in Step 5, and does not give the EKF linearization for the nonlinear CTRV/CTRA models. The definition of c_i in Step 6 uses the index k-1 instead of t-1. Please replace Algorithm 1 with the complete IMM recursion, including mixing, mode-matched prediction and update, and covariance combination, and justify any deliberate deviation from the standard IMM.
- [III.B, Eq. (4)-(8)] The damping function f(x) used in the Damping Window score is never specified. Equations (5)-(8) only state that f is positive, decays to zero as x approaches -infinity, is increasing on (-infinity,0], and satisfies f(0)=c. These constraints define a class of functions, not a concrete function, and the DW score s(t) in Eq. (4), the behavior shown in Fig. 3, and the thresholds reported in Section IV.B all depend on the actual f. Without an explicit f, Tables V and VI, which attribute FN reduction to the DW module, are not reproducible. Please provide the exact damping function and its parameter values.
- [IV.B, IV.C, V] The headline AMOTA of 73.8% is measured on the nuScenes Val split, and the module parameters (DBSE alpha and beta, DW thresholds, IMM transition matrix, initial model probabilities) are selected on that same split, as the conclusion acknowledges. This is a self-tuning evaluation setting, and the phrase 'highest known value for methods using the CenterPoint point cloud detector' therefore needs qualification: no independent test-set result is reported in Table II, and no sensitivity analysis is given. Please clarify the tuning protocol, add a test-set result or parameter-sensitivity study, or soften the claim accordingly.
minor comments (6)
- [IV.A, Table II] There are typos: 'Dateset' should be 'Dataset' in Section IV.A, and 'Methed' should be 'Method' in the Table II heading.
- [III.B, Eq. (7)] Equation (7) writes d f(x)/x; this should be d f(x)/dx.
- [III.A, Algorithm 1] In Step 6, the mixing coefficient c_i is defined as sum_j pi_ji mu_j^{k-1}; the subscript should be t-1 to match the recursion.
- [IV.D] The text says the IMM tracker is compared with other models in Fig. 4, but the figure showing this comparison is numbered Fig. 5; please correct the cross-reference.
- [Tables IV and II] Table IV reports a Val overall AMOTA of 73.6 for 'Ours', while Table II reports 73.8 for the full system. Please state explicitly whether Table IV is the IMM-only variant, so that the two numbers are not read as inconsistent.
- [III.B, Tables V and VII] The notation X1 and X2 is introduced only in the surrounding text; define these quantities in the table captions or in the main text at first use.
Circularity Check
Reported AMOTA is partly a fitted quantity: DBSE/IMM parameters were tuned on the nuScenes Val split, and the same split is used to report the headline result.
-
fitted input called prediction
[Section IV.B (Implementation Details), Section IV.C (Comparative Evaluation), Section V (Conclusion)]
"we use (10) for the car and trailer, with (α, β) set to (0.01, 0.1). For the bus and bicycle, we apply (11), with ( α, β) set to (70, 0.1), (90, 0.2) respectively. ... limitations such as the DBSE function selection and parameter tuning relying on the detector, and the IMM transition matrix and initial model probabilities being set empirically."
The headline 73.8% AMOTA and the module contributions in Tables III and IV are measured on the same nuScenes Val split on which the DBSE functions/parameters, DW thresholds, and IMM transition matrix and initial model probabilities were tuned. The paper states that the parameter values match those used in the experimental section and explicitly acknowledges that DBSE function selection and parameter tuning rely on the detector, and that IMM priors are set empirically. Thus the reported gain is the outcome of selecting parameters to maximize the evaluation metric on the evaluation set itself; the numerical result is a selected maximum rather than an independent prediction.
full rationale
No circularity appears in the mathematical derivation chain: the IMM filter equations are standard Kalman/EKF forms, and Algorithm 1, while under-specified (it omits the standard interaction/mixing step and the state-vector unification transformation), is not defined in terms of the claimed output. There are no load-bearing self-citations: Poly-MOT is an external baseline, and the cited IMM references are standard. The more serious issue is evaluation circularity: the per-class DBSE parameters, damping-window thresholds, and IMM priors are tuned on nuScenes Val, and the same split is then used for the main benchmark and ablation claims. The paper's own conclusion admits this tuning. This does not invalidate the central algorithmic idea, and the external test-set comparison in Table IV gives some independent evidence for the IMM gain, so the circularity is partial rather than total.
Assumptions & free parameters
free parameters (16)
- DBSE alpha for car/trailer =
0.01
- DBSE beta for car/trailer =
0.1
- DBSE alpha for bus =
70
- DBSE beta for bus =
0.1
- DBSE alpha for bicycle =
90
- DBSE beta for bicycle =
0.2
- DW active threshold for bus =
0.4
- DW tentative threshold for bus =
0.05
- DW active threshold for car =
0.3
- DW tentative threshold for car =
0.05
- DW active threshold for pedestrian =
0.3
- DW tentative threshold for pedestrian =
0.1
- DW active threshold for trailer =
0.6
- DW tentative threshold for trailer =
0.1
- IMM Markov transition matrix =
not disclosed
- Damping function f(x) =
not specified
assumptions (4)
- ad hoc to paper The four motion models can be mixed after 'unifying' their state vectors.
- domain assumption Point-cloud density is a reliable proxy for detection score reliability.
- domain assumption The damping function f(x) satisfying constraints (5)-(8) yields a valid trajectory score.
- standard math Standard Kalman and EKF assumptions (Gaussian noise, linearized transition for CTRV/CTRA) apply.
Cite this review
Pith. "Pith review of IMM-MOT: A Novel 3D Multi-object Tracking Framework with Interacting Multiple Model Filter." pith.science (2026). https://pith.science/paper/AGEJMEGK
@misc{pith2026250209672,
author = {Pith},
title = {Pith review of: IMM-MOT: A Novel 3D Multi-object Tracking Framework with Interacting Multiple Model Filter},
year = {2026},
howpublished = {\url{https://pith.science/paper/AGEJMEGK}},
note = {Machine review of arXiv:2502.09672}
}
read the original abstract
3D Multi-Object Tracking (MOT) provides the trajectories of surrounding objects, assisting robots or vehicles in smarter path planning and obstacle avoidance. Existing 3D MOT methods based on the Tracking-by-Detection framework typically use a single motion model to track an object throughout its entire tracking process. However, objects may change their motion patterns due to variations in the surrounding environment. In this paper, we introduce the Interacting Multiple Model filter in IMM-MOT, which accurately fits the complex motion patterns of individual objects, overcoming the limitation of single-model tracking in existing approaches. In addition, we incorporate a Damping Window mechanism into the trajectory lifecycle management, leveraging the continuous association status of trajectories to control their creation and termination, reducing the occurrence of overlooked low-confidence true targets. Furthermore, we propose the Distance-Based Score Enhancement module, which enhances the differentiation between false positives and true positives by adjusting detection scores, thereby improving the effectiveness of the Score Filter. On the NuScenes Val dataset, IMM-MOT outperforms most other single-modal models using 3D point clouds, achieving an AMOTA of 73.8%. Our project is available at https://github.com/Ap01lo/IMM-MOT.
Figures
Reference graph
Works this paper leans on
-
[1]
3d multi-object tracking: A baseline and new evaluation metrics,
X. Weng, J. Wang, D. Held, and K. Kitani, “3d multi-object tracking: A baseline and new evaluation metrics,” in 2020 IEEE/RSJ Interna- tional Conference on Intelligent Robots and Systems (IROS) , 2020, pp. 10 359–10 366
work page 2020
-
[2]
M. Nagy, N. Werghi, B. Hassan, J. Dias, and M. Khonji, “Rob- mot: Robust 3d multi-object tracking by observational noise and state estimation drift mitigation on lidar pointcloud,” arXiv preprint arXiv:2405.11536, 2024
arXiv 2024
-
[3]
Fast-poly: A fast polyhedral algorithm for 3d multi-object tracking,
X. Li, D. Liu, Y . Wu, X. Wu, L. Zhao, and J. Gao, “Fast-poly: A fast polyhedral algorithm for 3d multi-object tracking,” IEEE Robotics and Automation Letters, 2024
work page 2024
-
[4]
Joint multi-object detection and tracking with camera-lidar fusion for autonomous driving,
K. Huang and Q. Hao, “Joint multi-object detection and tracking with camera-lidar fusion for autonomous driving,” in 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS) , 2021, pp. 6983–6989
work page 2021
-
[5]
W. E. Ahmar, D. Kolhatkar, F. Nowruzi, and R. Laganiere, “Enhancing thermal mot: A novel box association method leveraging thermal identity and motion similarity,” arXiv preprint arXiv:2411.12943 , 2024
work page Pith review arXiv 2024
-
[6]
Mctrack: A unified 3d multi-object tracking framework for autonomous driving,
X. Wang, S. Qi, J. Zhao, H. Zhou, S. Zhang, G. Wang, K. Tu, S. Guo, J. Zhao, J. Li et al. , “Mctrack: A unified 3d multi-object tracking framework for autonomous driving,” arXiv preprint arXiv:2409.16149, 2024
arXiv 2024
-
[7]
Boosttrack++: using tracklet informa- tion to detect more objects in multiple object tracking,
V . Stanojevi´c and B. Todorovi´c, “Boosttrack++: using tracklet informa- tion to detect more objects in multiple object tracking,” arXiv preprint arXiv:2408.13003, 2024
arXiv 2024
-
[8]
ShaSTA: Modeling Shape and Spatio-Temporal Affinities for 3D Multi-Object Tracking,
T. Sadjadpour, J. Li, R. Ambrus, and J. Bohg, “ShaSTA: Modeling Shape and Spatio-Temporal Affinities for 3D Multi-Object Tracking,” IEEE Robotics and Automation Letters , vol. 9, no. 5, pp. 4273–4280, May 2024
work page 2024
Show all 26 references
-
[9]
Enhanced kalman with adaptive appearance motion sort for grounded generic multiple object tracking,
D. L. D. Anh, K. H. Tran, Q.-T. Nguyen, and N. H. Le, “Enhanced kalman with adaptive appearance motion sort for grounded generic multiple object tracking,” in Asian Conference on Computer Vision . Springer, 2025, pp. 310–328
2025
-
[10]
Pkf: Probabilistic data association kalman filter for multi-object tracking,
H. Cao, G. J. Pappas, and N. Atanasov, “Pkf: Probabilistic data association kalman filter for multi-object tracking,” arXiv preprint arXiv:2411.06378, 2024
2024 arXiv
-
[11]
Poly-mot: A polyhedral framework for 3d multi-object tracking,
X. Li, T. Xie, D. Liu, J. Gao, K. Dai, Z. Jiang, L. Zhao, and K. Wang, “Poly-mot: A polyhedral framework for 3d multi-object tracking,” in 2023 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, 2023, pp. 9391–9398
2023
-
[12]
Eagermot: 3d multi-object tracking via sensor fusion,
A. Kim, A. O ˇsep, and L. Leal-Taix ´e, “Eagermot: 3d multi-object tracking via sensor fusion,” in 2021 IEEE International Conference on Robotics and Automation (ICRA) , 2021, pp. 11 315–11 321
2021
-
[13]
Score refinement for confidence-based 3D multi-object tracking,
N. Benbarka, J. Schr ¨oder, and A. Zell, “Score refinement for confidence-based 3D multi-object tracking,” in 2021 IEEE/RSJ Inter- national Conference on Intelligent Robots and Systems (IROS) , Sep. 2021, pp. 8083–8090
2021
-
[14]
Largekernel3d: Scaling up kernels in 3d sparse cnns,
Y . Chen, J. Liu, X. Zhang, X. Qi, and J. Jia, “Largekernel3d: Scaling up kernels in 3d sparse cnns,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2023, pp. 13 488–13 498
2023
-
[15]
Center-based 3d object detec- tion and tracking,
T. Yin, X. Zhou, and P. Krahenbuhl, “Center-based 3d object detec- tion and tracking,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , June 2021, pp. 11 784–11 793
2021
-
[16]
Poly-pc: A polyhedral network for multiple point cloud tasks at once,
T. Xie, S. Wang, K. Wang, L. Yang, Z. Jiang, X. Zhang, K. Dai, R. Li, and J. Cheng, “Poly-pc: A polyhedral network for multiple point cloud tasks at once,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , June 2023, pp. 1233–1243
2023
-
[17]
Simple online and realtime tracking with a deep association metric,
N. Wojke, A. Bewley, and D. Paulus, “Simple online and realtime tracking with a deep association metric,” in 2017 IEEE International Conference on Image Processing (ICIP) , 2017, pp. 3645–3649
2017
-
[18]
Bytetrack: Multi-object tracking by associating every detection box,
Y . Zhang, P. Sun, Y . Jiang, D. Yu, F. Weng, Z. Yuan, P. Luo, W. Liu, and X. Wang, “Bytetrack: Multi-object tracking by associating every detection box,” in Computer Vision – ECCV 2022 , S. Avidan, G. Brostow, M. Ciss ´e, G. M. Farinella, and T. Hassner, Eds. Cham: Springer N...
2022
-
[19]
Simpletrack: Understanding and rethinking 3d multi-object tracking,
Z. Pang, Z. Li, and N. Wang, “Simpletrack: Understanding and rethinking 3d multi-object tracking,” in European Conference on Computer Vision. Springer, 2022, pp. 680–696
2022
-
[20]
Soft-NMS – Improving Object Detection With One Line of Code,
N. Bodla, B. Singh, R. Chellappa, and L. S. Davis, “Soft-NMS – Improving Object Detection With One Line of Code,” in Proceedings of the IEEE International Conference on Computer Vision , 2017, pp. 5561–5569
2017
-
[21]
Adaptive NMS: Refining Pedestrian Detection in a Crowd,
S. Liu, D. Huang, and Y . Wang, “Adaptive NMS: Refining Pedestrian Detection in a Crowd,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , 2019, pp. 6459–6468
2019
-
[22]
Swtrack: Multiple hy- pothesis sliding window 3d multi-object tracking,
S. Papais, R. Ren, and S. Waslander, “Swtrack: Multiple hy- pothesis sliding window 3d multi-object tracking,” arXiv preprint arXiv:2402.17892, 2024
2024 arXiv
-
[23]
Error performance analysis of imm-kalman filter for maneuvering target tracking application,
M. Yunita, J. Suryana, and A. Izzuddin, “Error performance analysis of imm-kalman filter for maneuvering target tracking application,” in 2020 6th International Conference on Wireless and Telematics (ICWT), 2020, pp. 1–6
2020
-
[24]
Performance prediction of the interacting multiple model algorithm,
X. Li and Y . Bar-Shalom, “Performance prediction of the interacting multiple model algorithm,” IEEE Transactions on Aerospace and Electronic Systems, vol. 29, no. 3, pp. 755–771, Jul. 1993
1993
-
[25]
Learnable Online Graph Representations for 3D Multi-Object Track- ing,
J.-N. Zaech, A. Liniger, D. Dai, M. Danelljan, and L. Van Gool, “Learnable Online Graph Representations for 3D Multi-Object Track- ing,” IEEE Robotics and Automation Letters , vol. 7, no. 2, pp. 5103– 5110, Apr. 2022
2022
-
[26]
nuScenes: A Multimodal Dataset for Autonomous Driving,
H. Caesar, V . Bankiti, A. H. Lang, S. V ora, V . E. Liong, Q. Xu, A. Krishnan, Y . Pan, G. Baldan, and O. Beijbom, “nuScenes: A Multimodal Dataset for Autonomous Driving,” in 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) . Seattle, W A, USA: IEEE,...
2020
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.