MeMoSORT: Memory-Assisted Filtering and Motion-Adaptive Association Metric for Multi-Person Tracking

Le Zheng; Roujing Li; Tianxiao Liu; Xueyao Hu; Yingjie Wang; Zhixing Wang

REVIEW 2 major objections 4 minor 2 references

MeMoSORT: Memory-Assisted Filtering and Motion-Adaptive Association Metric for Multi-Person Tracking

T0 review · 2 major / 4 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read MeMoSORT claims memory-assisted filtering and motion-adaptive matching push multi-person tracking to new benchmark highs.

desk verdict Strong benchmark numbers but a load-bearing contradiction in the core filter equations and no code or appendix; worth refereeing, not worth accepting as-is. read the letter →

arxiv 2508.09796 v1 pith:UV7XPVJW submitted 2025-08-13 cs.CV

Yingjie Wang , Zhixing Wang , Le Zheng , Tianxiao Liu , Roujing Li , Xueyao Hu This is my paper · ORCID

classification cs.CV

keywords multi-objecttrackingtracking-by-detectionKalmanfiltermemory-augmentedneuralnetworkIoUassociationmotionadaptivityDanceTrackSportsMOT

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper proposes MeMoSORT, an online, real-time multi-person tracker built on the tracking-by-detection pipeline. Its central claim is that the two usual failure points—the Kalman filter's linear first-order Markov motion assumption and fixed IoU-based association—can be corrected with learned memory and motion-adaptive matching, producing state-of-the-art tracking quality on human-dominant benchmarks. The authors report HOTA scores of 67.9% on DanceTrack and 82.1% on SportsMOT, surpassing previous trackers including larger transformer- and diffusion-based ones. If correct, this shows that a lightweight, physics-grounded filter plus an adaptive geometric association metric is sufficient to handle complex motion and severe occlusion without expensive deep motion predictors.

What carries the argument

MeKF is a memory-assisted Kalman filter: an LSTM updates a memory representation of trajectory history, and four MLPs generate compensation terms for the state transition and measurement processes, leaving the standard Kalman prediction-update cycle intact. Mo-IoU is a multiplicative association metric, Mo-IoU = EIoU × HIoU, where EIoU expands both boxes by (2p+1) before computing IoU and HIoU measures vertical alignment as (l_t / (h'_t + e h_t − l_t))^{q_t}; the Motion-Adaptive Technique sets p and q from normalized velocity thresholds derived from training-set percentiles.

What would settle it

Run MeMoSORT on DanceTrack and SportsMOT with the motion-adaptive thresholds replaced by extreme fixed percentiles (e.g., 10th and 90th) or by per-sequence online-estimated percentiles; if HOTA stays within noise of 67.9% and 82.1%, the adaptive component is not what drives the gains. A stronger test would re-split the benchmarks so that training and test velocity distributions are deliberately mismatched.

Watch

Extended reading notes

Core claim

MeMoSORT's first component, MeKF, replaces the strict first-order Markov update with a non-Markovian formulation in which an LSTM memory module tracks history and MLP gates emit compensation terms for both the state transition and the measurement model; the Kalman update remains the backbone, so even if the learned gates fail, the physical model still provides a baseline prediction. The second component, Mo-IoU, is the product of Expansion IoU (which symmetrically expands both boxes by a factor (2p+1) before overlap computation) and Height IoU (a vertical 1D-IoU with exponent q), with p and q chosen by a Motion-Adaptive Technique that thresholds normalized center speed and height speed into

Load-bearing premise

The motion-adaptive thresholds are fixed at the 50th and 70th percentiles of the normalized velocity distribution computed on the training set; if test videos have different velocity statistics, the adaptive expansion and height weighting can become mis-calibrated and the Mo-IoU advantage may vanish.

Editorial extensions

If this is right

If the reported results replicate, lightweight Kalman-based trackers can beat transformer- and diffusion-based motion models on DanceTrack and SportsMOT, with reported HOTA of 67.9% and 82.1%.
Ablations attribute about +10.5 HOTA to MeKF and about +10.1 HOTA to Mo-IoU over the baseline on the DanceTrack validation set, so each innovation carries substantial weight.
The largest gains appear in association metrics (AssA and IDF1), implying the method primarily reduces identity switches and occlusion-driven track losses.
The discrete, motion-adaptive parameter choice keeps the pipeline real-time (about 29 FPS with ReID), suggesting accuracy gains are compatible with online deployment.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

A natural extension would replace the fixed training-set velocity percentiles with per-target online estimates of velocity statistics, removing the need for dataset-specific calibration.
The MeKF compensation gates are architecture-agnostic and could be dropped into other SORT-style trackers, though the paper only demonstrates them inside MeMoSORT.
The height-similarity term is likely most valuable in densely occluded scenes; testing on overhead or crowd-heavy benchmarks would show whether height weighting rather than expansion alone drives the AssA gains.
If the motion-adaptive gains generalize, they offer a cheaper real-time alternative to learned motion predictors for robotics or autonomous driving contexts with similar motion statistics.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

MeMoSORT is a tracking-by-detection system that bolts two ideas onto Deep OC-SORT: a memory-compensated Kalman filter (MeKF) taken from Yan et al. 2024, and a motion-adaptive IoU (Mo-IoU) that switches expansion and height-IoU parameters based on normalized velocity. The benchmark results are genuinely good: 67.9 HOTA on DanceTrack and 82.1 on SportsMOT, both ahead of prior SOTA, and the ablations on DanceTrack val show large, plausible component effects. The paper is clearly written, the experimental protocol is standard, and the authors cite the prior work their components are built on.

The soft spots are real. First, there is a direct contradiction between Eq. 6 and Eq. 11: Eq. 6 says the measurement compensation Delta_H_t is a function of the detection ebt, but Eq. 11 defines it as F^3_MLP(hat b'_t) (the predicted state). The SUG residual in Eq. 11 uses the predicted state, so as written the measurement compensation is not what the text describes. That matters because the ablation attributes +10.5 HOTA to MeKF, so the headline result depends on this exact module. Second, Appendix A, which supposedly contains the theoretical derivations, is missing, and the MeKF training setup is under-specified: no loss function, no training data details, no sequence length, no code. This makes the central claim impossible to verify independently as submitted. Third, the Mo-IoU thresholds are fixed at the 50th/70th percentiles of training-set velocities; the paper gives no sensitivity analysis, so we don't know how fragile that choice is on test videos with different motion statistics.

None of this means the result is fake. The benchmark tables are internally coherent, and the ablation story is consistent. But the paper, as packaged, is not reproducible, and the internal inconsistency in the core equations needs to be resolved before the numbers can be trusted. A serious referee should be given the chance to ask for the appendix, the training details, and ideally a code release. With those, this could be a useful incremental contribution; without them, it's a claim that can't be checked.

Recommendation: send to peer review, but be prepared for major revision. The authors need to fix the equations, provide the missing appendix/training details, and either release code or provide enough detail to reimplement.

Referee Report

2 major / 4 minor

Summary. The paper proposes MeMoSORT, an online tracking-by-detection MOT method with two main innovations. The Memory-assisted Kalman filter (MeKF) augments a standard Kalman filter with an LSTM-based memory module and MLP-generated compensation terms for state prediction and measurement update. The Motion-adaptive IoU (Mo-IoU) association metric adaptively expands bounding boxes and weights vertical overlap based on normalized velocity thresholds. On DanceTrack and SportsMOT, the authors report state-of-the-art HOTA scores of 67.9% and 82.1%, respectively. Ablations on the DanceTrack validation set attribute a 10.47 HOTA improvement to MeKF and a 10.13 HOTA improvement to Mo-IoU.

Significance. If the reported results are reproducible, the paper would make a solid contribution: it achieves SOTA on two challenging human-tracking benchmarks with a lightweight, online, real-time tracker, and the ablations show large, directionally consistent component effects. The paper's strength is its clear motivation for replacing the first-order Markovian Kalman filter and rigid IoU association, and the benchmark comparison is comprehensive. However, the central claim is currently not verifiable: the MeKF update equations are internally inconsistent, the referenced Appendix A is missing, the MeKF training protocol is not specified, and no code is released. The Mo-IoU adaptive thresholds are calibrated on training-set statistics, and no sensitivity analysis is provided.

major comments (2)

[Motion-Adaptive Technique, Eqs. (16)-(17); Implementation Details] The MAT thresholds are calibrated to training-set statistics: Theta_center and Theta_height are set to the 50th and 70th percentiles of the normalized velocity distribution from the training split. This makes the method adaptive only in the sense that the thresholds are fixed after training; they are not necessarily calibrated for test videos with different velocity distributions. No sensitivity analysis over Theta_center and Theta_height is reported, and Table 5 compares Mo-IoU to fixed-parameter variants on the same validation set used to select the thresholds. As the SOTA claim depends on Mo-IoU, the authors should show that the advantage is robust across threshold choices and across test sequences with different motion statistics.
[Implementation Details; missing Appendix A] The MeKF is a learned component, but the paper does not specify its training data, loss function, input/output representation, sequence length, or training/inference protocol. The only training detail is 'AdamW optimizer with learning rate set to 10^-4'. The sentence about unspecified hyper-parameters being consistent with Deep OC-SORT cannot cover the MeKF networks, which are new. No code is released. Without this information, the 10.47 HOTA contribution of MeKF and the headline results cannot be reproduced or checked. Please provide the missing derivation (Appendix A), a complete training specification, and ideally code.

minor comments (4)

[Implementation Details] The hyperparameter sentence appears to swap roles: Eq. (16) uses Mslow/Mfast for expansion parameters, while Eq. (17) uses Nslow/Nfast for height parameters, but the text says 'height modulation parameter are set to Mslow=2' and 'expansion scaling parameters uses Nslow=0.5'. Please correct the labels.
[Table 1] Hybrid-SORT is listed twice with HOTA 62.2 and 65.7, contrary to the stated policy of reporting only the highest-HOTA variant for methods with multiple models. Please clarify the versions or remove the duplicate.
[Experiments] There are several typographical and grammatical errors, including 'motin model' instead of 'motion model', 'the boxes is mixed to a mess', and 'we evaluates MeMoSORT'. These should be corrected throughout.
[Table 4] MeKF achieves the best HOTA, AssA, and IDF1, but its DetA (91.69) is lower than several of the compared filters, including Diffusion (92.93). The trade-off between detection accuracy and association quality should be discussed.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the SOTA claim is benchmarked externally and component gains are ablated; the self-citation of Yan et al. 2024 and missing Appendix A are completeness/disclosure issues, not circular reductions.

full rationale

The central derivation chain is not circular. MeMoSORT's headline HOTA numbers (67.9/82.1 on DanceTrack/SportsMOT) are measured on public test sets against published methods; no metric is defined in terms of MeMoSORT's own outputs or fitted constants. The two innovations are isolated in Table 3: adding MeKF to the baseline changes HOTA from 56.94 to 67.41, and adding Mo-IoU changes it to 77.54 on the validation set; these are empirical comparisons, not consequences of the definitions. Table 4 compares MeKF with KF/LSTM/Transformer/Diffusion under the same framework, and Table 5 compares Mo-IoU with EIoU/HMIoU/HA-EIoU; each comparison is a separate ablation, not a self-fulfilling identity. The MAT thresholds (Eqs. 16-17) are set from training-set velocity percentiles, a standard hyperparameter choice; test-set evaluation is external, so this is not 'fitted input called prediction'. The MeKF section does cite Yan et al. 2024, which shares corresponding author Le Zheng, and says the gates are 'supported by theoretical derivations, which are detailed in Appendix A'; Appendix A is absent in this version. That is a missing-proof and self-citation-disclosure concern, and the inconsistency between Eq. 6 (compensation from ebt) and Eq. 11 (compensation from ˆb′_t) is a load-bearing specification error for the MeKF ablation. However, neither issue makes the derivation equivalent to its input by construction, so no circular step meets the evidentiary bar.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central claim rests on learned neural compensations whose training is unspecified, on hand-fitted speed thresholds, and on the transfer of training-set velocity statistics to test sets. These are the main hidden costs the reader pays for the reported improvement. No new physical entities are introduced; MeKF and Mo-IoU are algorithmic components.

free parameters (4)

Velocity thresholds Theta_center and Theta_height = DanceTrack: 0.0406 and 0.0090; SportsMOT: 0.1172 and 0.0062
Set to the 50th and 70th percentiles of the normalized velocity distribution on the training set (Implementation Details). They switch p_t and q_t in Eqs. 16 and 17, so the claimed Mo-IoU gains depend on these values.
Mo-IoU quantization levels Mslow, Mfast, Nslow, Nfast = Mslow=2, Mfast=1, Nslow=0.5, Nfast=0.6
Hand-chosen hyperparameters controlling expansion scale and height-weight exponent; reported as fixed in Implementation Details.
Association confidence thresholds = 0.6 and 0.1
Confidence split for high-score and low-score association stages; assumed from Deep OC-SORT but still a manual setting.
MeKF LSTM and MLP network weights = Learned, training set and loss not specified
The memory and compensation functions are neural networks fitted to data; without the training protocol the learned values are not reproducible.

assumptions (4)

ad hoc to paper The non-Markovian dynamics in Eq. 3 can be approximated by a first-order linear transition plus an additive memory-dependent compensation, Eq. 5.
This representational assumption is introduced specifically for MeKF; if the residual is not learnable from LSTM memory, the filter gain disappears.
domain assumption The measurement-compensation functions and noise covariances can be learned while preserving the Kalman update equations as correct Bayesian updates.
The paper asserts Bayesian-principled gates and points to a missing Appendix A; the validity of treating MLP outputs as additive compensation and covariance terms is assumed.
domain assumption Training-set velocity percentile statistics transfer to the test videos.
Mo-IoU thresholds are estimated from training-set normalized velocity distributions; test sequences are assumed to have comparable distributions.
domain assumption Standard Kalman filter noise assumptions, zero-mean Gaussian w_t and v_t with covariances Q_t and R_t, remain approximately valid for the compensated model.
Eqs. 1 and 2 and the update equations rely on these assumptions; the paper does not re-derive or justify them after adding learned compensation.

how reviews work

0 comments

Cite this review

Pith. "Pith review of MeMoSORT: Memory-Assisted Filtering and Motion-Adaptive Association Metric for Multi-Person Tracking." pith.science (2026). https://pith.science/paper/UV7XPVJW

@misc{pith2026250809796,
  author       = {Pith},
  title        = {Pith review of: MeMoSORT: Memory-Assisted Filtering and Motion-Adaptive Association Metric for Multi-Person Tracking},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/UV7XPVJW}},
  note         = {Machine review of arXiv:2508.09796}
}

read the original abstract

Multi-object tracking (MOT) in human-dominant scenarios, which involves continuously tracking multiple people within video sequences, remains a significant challenge in computer vision due to targets' complex motion and severe occlusions. Conventional tracking-by-detection methods are fundamentally limited by their reliance on Kalman filter (KF) and rigid Intersection over Union (IoU)-based association. The motion model in KF often mismatches real-world object dynamics, causing filtering errors, while rigid association struggles under occlusions, leading to identity switches or target loss. To address these issues, we propose MeMoSORT, a simple, online, and real-time MOT tracker with two key innovations. First, the Memory-assisted Kalman filter (MeKF) uses memory-augmented neural networks to compensate for mismatches between assumed and actual object motion. Second, the Motion-adaptive IoU (Mo-IoU) adaptively expands the matching space and incorporates height similarity to reduce the influence of detection errors and association failures, while remaining lightweight. Experiments on DanceTrack and SportsMOT show that MeMoSORT achieves state-of-the-art performance, with HOTA scores of 67.9\% and 82.1\%, respectively.

Figures

Figures reproduced from arXiv: 2508.09796 by the authors.

**Figure 2.** Comparison between (a) the conventional Tracking-by-Detection framework and (b) our proposed MeMoSORT framework. MeMoSORT introduces two key enhancements: it leverages a memory mechanism to guide motion estimation for more accurate state prediction and update, and it applies a Motion-adaptive IoU to achieve robust data association. Together, these improvements enhance tracking robustness in complex scenarios. with … view at source ↗

**Figure 3.** Framework of proposed MeKF. In both SPG and [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗

Figures from the paper (1 more)

**Figure 4.** Figure 4: Qualitative analysis of DiffMOT (a, c) and MeMoSORT (b, d) in challenging scenarios from the DanceTrack valida [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

2 extracted references · 2 linked inside Pith

[2023]

In Proceedings of the IEEE/CVF con- ference on computer vision and pattern recognition , 9686– 9696

Observation-centric sort: Rethinking sort for robust multi-object tracking. In Proceedings of the IEEE/CVF con- ference on computer vision and pattern recognition , 9686– 9696. Cioppa, A.; Giancola, S.; Deliege, A.; Kang, L.; Zhou, X.; Cheng, Z.; Ghanem, B.; and Van Droogenbroeck, M. 2022. SoccerNet-Tracking: Multiple Object Tracking Dataset and Benchmark...

arXiv 2022
[2025]

Beyond Kalman Filters: Deep Learning-Based Filters for Improved Object Tracking. 36(1). Aharon, N.; Orfaig, R.; and Bobrovsky, B.-Z. 2022. Bot- sort: Robust associations multi-pedestrian tracking. arXiv preprint arXiv:2206.14651. Bernardin, K.; and Stiefelhagen, R. 2008. Evaluating Multi- ple Object Tracking Performance: The CLEAR MOT Met- rics. 2008: 1–1...

arXiv 2022

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.

[1] [2023]

In Proceedings of the IEEE/CVF con- ference on computer vision and pattern recognition , 9686– 9696

Observation-centric sort: Rethinking sort for robust multi-object tracking. In Proceedings of the IEEE/CVF con- ference on computer vision and pattern recognition , 9686– 9696. Cioppa, A.; Giancola, S.; Deliege, A.; Kang, L.; Zhou, X.; Cheng, Z.; Ghanem, B.; and Van Droogenbroeck, M. 2022. SoccerNet-Tracking: Multiple Object Tracking Dataset and Benchmark...

arXiv 2022

[2] [2025]

Beyond Kalman Filters: Deep Learning-Based Filters for Improved Object Tracking. 36(1). Aharon, N.; Orfaig, R.; and Bobrovsky, B.-Z. 2022. Bot- sort: Robust associations multi-pedestrian tracking. arXiv preprint arXiv:2206.14651. Bernardin, K.; and Stiefelhagen, R. 2008. Evaluating Multi- ple Object Tracking Performance: The CLEAR MOT Met- rics. 2008: 1–1...

arXiv 2022