Pith. sign in

REVIEW 3 major objections 6 minor 60 references

Diff-MM: Exploring Pre-trained Text-to-Image Generation Model for Unified Multi-modal Object Tracking

T0 review · 3 major / 6 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A pre-trained text-to-image generation model's UNet can be repurposed as a unified multi-modal object tracker, with one parameter set covering RGB-N/D/T/E tracking.

desk verdict A well-executed, strongly benchmarked unified multi-modal tracker built on a frozen SD UNet; the causal claim that SD priors drive the gains is plausible but not isolated, so the paper needs one more baseline before the conclusion is as strong as the numbers. read the letter →

arxiv 2505.12606 v1 pith:5LFBVW5J submitted 2025-05-19 cs.CV

Shiyu Xuan , Zechao Li , Jinhui Tang This is my paper · ORCID
classification cs.CV
keywords multi-modalobjecttrackingdiffusionmodelsStableunifiedtrackerRGB-N/D/T/Etransferlearningparallelfeatureextractionmodalitysub-moduletuning
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that a pre-trained text-to-image generation model can be converted into a unified multi-modal object tracker, replacing the usual recipe of training an RGB tracker and then prompt-tuning it on small multi-modal datasets. If true, this matters because the main bottleneck of multi-modal tracking—the scarcity of paired RGB-depth, RGB-thermal, RGB-event, and RGB-language training data—could be bypassed by borrowing priors learned from web-scale image-text pairs. Diff-MM reports benchmark numbers that beat earlier unified and modality-specific trackers, including 66.3 versus 58.0 AUC on TNL2K and 68.7 versus 60.7 F-score on DepthTrack when compared with OneTracker.

What carries the argument

The load-bearing machinery is a pair of modifications to a frozen Stable Diffusion UNet. Parallel Feature Extraction (PFE) concatenates the template and search token sequences at the self-attention layer of every Basic Block, runs attention jointly, then de-concatenates, so the UNet handles pairwise inputs while its architecture and weights stay intact. Multi-modal Sub-module Tuning (MST) clones the UNet encoder and middle block into a modality sub-module, feeds the auxiliary modality's noisy VAE latents through it, and adds the result to the RGB branch through zero-initialized convolutions; starting at zero means the auxiliary signal is learned gradually and does not disturb the pretrained RGB features at initialization.

What would settle it

Run Diff-MM's full two-stage training with the UNet and VAE re-initialized from random weights, keeping the same architecture, PFE/MST, data, and loss, and compare DepthTrack F-score and TNL2K AUC; if the randomly initialized model matches the reported numbers, the claim that pre-trained generative priors are the source of the gains is refuted. A complementary check is to shuffle the auxiliary modality's pixels at test time: a tracker that truly exploits depth, thermal, or event data should degrade sharply, whereas one that learned benchmark statistics would not.

Watch

Extended reading notes

Core claim

The central claim is that the UNet of Stable Diffusion—normally a denoiser—can be turned into a tracking feature extractor without altering its architecture. The parallel feature extraction pipeline feeds template and search frames through the same frozen UNet, exchanging information only at the existing self-attention layers; a cloned encoder-and-middle-block sub-module, injected through zero-initialized lateral connections, brings in depth, thermal, or event data; language descriptions enter through the already-present cross-attention text condition. With one parameter set shared across all four modality types, Diff-MM outperforms earlier unified trackers and modality-specific trackers on RGB-N/D/T/E benchmarks, for example 66.3 AUC on TNL2K and 68.7 F-score on DepthTrack in the single-parameter-set configuration. On the paper's own account, the generative model's prior knowledge of depth, thermal, event, and language is what makes small multi-modal training sets sufficient.

Load-bearing premise

The load-bearing premise is that depth, thermal, and event frames, once converted to RGB-like images and passed through the frozen VAE and diffusion noising pipeline, still carry the modality-specific information a tracker needs; if that premise fails, the reported gains would come from the cloned sub-module and lateral connections rather than from Stable Diffusion's priors.

Editorial extensions

If this is right

  • Training a multi-modal tracker reduces to fine-tuning a frozen generative backbone, so limited multi-modal data no longer limits model capacity.
  • The same PFE recipe can turn other SD-family diffusion backbones into tracking extractors, as demonstrated by the SD v3-5 variant.
  • One set of weights handles RGB-N, RGB-D, RGB-T, and RGB-E tracking, and switching modalities amounts to attaching the same sub-module.
  • Freezing the UNet avoids forgetting, so adding new auxiliary modalities does not degrade RGB or language-guided tracking performance.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • An inference from the PFE design: any pairwise-input visual task, such as stereo correspondence, frame-to-frame local feature matching, or few-shot segmentation, could reuse the concat/de-concat self-attention trick to inherit generative priors without retraining the backbone.
  • An inference from the MST design: new sensing modalities could be added to an existing tracker by cloning the encoder and middle block and training only the new sub-module, leaving the RGB branch untouched, which would make a practical recipe for sensor-agnostic trackers.
  • A testable extension: evaluate the same pipeline with an untrained UNet of identical architecture; if performance stays high, the gains come from scale and architecture rather than the pretrained generative prior, which would reframe the paper's contribution.
  • A natural question raised by the unified-parameter results is whether mixing all modalities in one second-stage sub-module, rather than one sub-module per modality, is what preserves per-modality accuracy; the paper reports both variants but does not isolate that effect.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 6 minor

Summary. The paper proposes Diff-MM, a unified multi-modal object tracker covering RGB-N (natural language), RGB-D, RGB-T, and RGB-E modalities with a single parameter set. It builds on Stable Diffusion v1-5, using the frozen VAE and denoising UNet as a feature extractor. A parallel feature extraction pipeline (PFE) processes template and search frames through the UNet in parallel, concatenating them at each self-attention layer to model their relationship without architectural modification. A multi-modal sub-module tuning method (MST) clones UNet encoder/middle-block weights into a sub-module for auxiliary modalities, injecting features via zero-initialized lateral convolutions while freezing the RGB UNet. A two-stage training recipe first fine-tunes on RGB/RGB-N tracking data, then tunes the sub-module on RGB-D/T/E data. Experiments on TNL2K, OTB99, DepthTrack, VOT-RGBD2022, LasHeR, RGBT234, and VisEvent report consistent improvements over prior dedicated and unified trackers, e.g., 66.3 vs 58.0 AUC on TNL2K and 68.7 vs 60.7 F-score on DepthTrack versus OneTracker (Table 1). Ablations (Table 2) show PFE outperforms prompt-token and cross-attention alternatives, and MST outperforms early fusion and extra-encoder designs.

Significance. If the causal claim holds, the paper makes a strong contribution: it demonstrates that a text-to-image generative model's pretrained representations can be transferred to multi-modal tracking, reducing the need for large multi-modal training data and enabling one parameter set across four modality pairs. The ablations are carefully designed and support the specific design choices of PFE and MST over several alternatives. The reported gains over strong baselines are large and consistent across benchmarks, which makes the method practically interesting. However, the paper's central attribution of these gains to Stable Diffusion's 'extensive prior knowledge' is not isolated experimentally; the missing same-architecture non-SD baseline leaves open alternative explanations based on the UNet architecture, parameter count, and training recipe. The SD v3-5 variant, used to claim generality, is insufficiently specified.

major comments (3)
  1. [§4.3, Table 2] The central claim that gains arise from 'extensive prior knowledge in the generation model' is not supported by a same-architecture control without SD pretraining. Every ablation in Table 2 varies both the interaction design and the backbone simultaneously: PFE uses the SD UNet for both template and search, while Cross-Attn/MAE/CLIP variants introduce a different template encoder; similarly, MST is compared to early fusion and extra encoders that modify the network path. A random-init SD-UNet (or a non-generative pretrained UNet of comparable capacity) trained with the identical PFE+MST pipeline and two-stage recipe would isolate the contribution of SD pretraining. Without such a baseline, the reported improvements on TNL2K, DepthTrack, LasHeR, and VisEvent could stem from the UNet architecture, the large fine-tuned parameter count, or the training recipe rather than from generative priors. Please add this control and report the comparison in Table 2.
  2. [§4.1, 'Diff-MM 256†'] The SD v3-5 variant is reported as evidence that the design transfers to other diffusion architectures, but no implementation details are given. It is unclear how PFE's self-attention concatenation and MST's sub-module are adapted to the MM-DiT architecture, what text conditioning and VAE/noising scheme are used, whether the same two-stage training recipe and hyperparameters apply, and how the modality inputs are converted. Without these details, the Diff-MM 256† results in Table 1 cannot be reproduced or interpreted, and the generality claim is weakened. Please provide a full description or remove the variant.
  3. [§4.3, Table 2 right] The ablation labeled 'W/ TNL2K vs. W/O TNL2K' claims that the performance improvement is not due to the extra TNL2K training data, but the reported numbers (DepthTrack, LasHeR, VisEvent) do not include any RGB-N tracking result. Since TNL2K is both a first-stage training dataset and a headline test set in Table 1, the claim is unsupported for the RGB-N results. Please report RGB-N results (e.g., on OTB99) obtained without TNL2K in first-stage training, or otherwise clarify the scope of the claim.
minor comments (6)
  1. [§4.1] The use of a fixed diffusion timestep t=1 is mentioned but not justified; please state whether the same timestep is used for template and search latents and how it was chosen.
  2. [§3.3, Eq. (4)] The index N in \hat{s}_t[N-i] is not defined; clarify that N is the number of lateral-connection injection points or the layer index.
  3. [Figure 2] The 'Data Flow in UNet Basic Block' inset is too small to read the ConcatL/DeConcatL operations; a larger separate diagram would help.
  4. [Table 1] The column headers 'Modality-specific Model', 'Unified Model with Modality-specific Parameters', and 'Unified Model with a Single Set of Parameters' are easy to misread because Diff-MM columns appear in multiple groups; consider using separate sub-tables or a clearer grouping.
  5. [§4.2] The statement that Diff-Tracker's poor performance is due to lack of fine-grained detail is plausible but the re-implementation details (training data, epochs, prompt token length) are not given; please add them for reproducibility.
  6. [General] No inference speed or parameter count is reported; since tracking methods are compared also on efficiency, adding FPS or MACs for the 256 and 384 variants would be useful.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the paper reports empirical benchmark comparisons with no test-set fitting and no self-citation chain that forces its conclusions.

full rationale

Diff-MM is an empirical method paper, not a derivational one. The central claim is that a Stable Diffusion UNet, adapted via the parallel feature extraction pipeline (PFE) and multi-modal sub-module tuning (MST), achieves strong results on RGB-N/D/T/E tracking benchmarks. There is no step in Section 3 where an output quantity is defined in terms of the quantity it purports to predict, and no fitted parameter is renamed as a prediction. The training procedure is standard two-stage supervised learning, and the benchmark numbers in Table 1 are obtained by evaluating a trained model on held-out test splits, not by optimizing constants on those splits. The one near-issue is that TNL2K appears both in the first-stage training set and as a reported RGB-N benchmark; however, the paper explicitly includes an ablation 'W/ TNL2K v.s. W/O TNL2K' in Table 2, showing only marginal differences on the multi-modal benchmarks, and using a benchmark's training split while reporting its test split is a standard protocol rather than circular reasoning. The paper's reliance on prior work (Marigold, GeoWizard, E-Motion, thermal translation, VPD, DIFT) is external evidence for the transferability of Stable Diffusion representations, not a self-citation that carries the argument. The absence of a same-architecture non-Stable-Diffusion baseline is a legitimate causal-attribution or correctness concern, but it is not circularity: the paper does not define its success metric in terms of that baseline, and the empirical comparison remains meaningful as reported. No circular step can be exhibited by quoting an equation where Eq. X equals Eq. Y by construction or where a fitted parameter is relabeled as a prediction. Therefore the appropriate finding is no significant circularity.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The central empirical claims rest on the transferability of Stable Diffusion representations and on two architectural design assumptions, PFE and MST. The listed free parameters are standard training hyperparameters rather than fitted scientific constants. No new physical or theoretical entities are introduced.

free parameters (2)
  • Diffusion forward-process timestep t = 1
    Set to 1 in Section 4.1; hand-chosen noise level applied to template and search latents before entering the UNet, and not swept in the paper.
  • Loss weights λ_giou and λ_L1 = 2 and 5
    Inherited from OSTrack (Section 4.1); not fitted on these benchmarks and not central to the methodological novelty, but they are hand-set constants the training depends on.
assumptions (4)
  • domain assumption Stable Diffusion v1-5, trained on web-scale image-text pairs, encodes transferable visual priors useful for tracking features.
    Invoked throughout Sections 1 and 3.1; the paper cites Marigold, GeoWizard, VPD, and DIFT as evidence that SD features transfer to perception tasks.
  • domain assumption Depth, thermal, and event frames can be converted to RGB-like images and encoded by the SD VAE without losing the information needed for tracking.
    Stated in Section 4.1 ('transform the depth map, thermal-infrared image, and event flow into an RGB-like form') and used in Section 3.3; no direct analysis of information loss is provided.
  • ad hoc to paper Concatenating template and search latents at every self-attention layer preserves the pre-trained UNet's knowledge while enabling cross-frame interaction.
    This is the core design hypothesis of Eq. 3 in Section 3.2; the authors support it by ablations against cross-attention and prompt-token designs, but it remains an unproven architectural assumption.
  • ad hoc to paper Auxiliary modality features can be injected into the frozen RGB UNet via zero-initialized lateral-connection convolutions without degrading RGB tracking.
    Used in Eq. 4 in Section 3.3; the 'w/o Zero Init' ablation in Table 2 suggests zero-init matters, but the mechanism is assumed rather than derived.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Diff-MM: Exploring Pre-trained Text-to-Image Generation Model for Unified Multi-modal Object Tracking." pith.science (2026). https://pith.science/paper/5LFBVW5J

@misc{pith2026250512606,
  author       = {Pith},
  title        = {Pith review of: Diff-MM: Exploring Pre-trained Text-to-Image Generation Model for Unified Multi-modal Object Tracking},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/5LFBVW5J}},
  note         = {Machine review of arXiv:2505.12606}
}
read the original abstract

Multi-modal object tracking integrates auxiliary modalities such as depth, thermal infrared, event flow, and language to provide additional information beyond RGB images, showing great potential in improving tracking stabilization in complex scenarios. Existing methods typically start from an RGB-based tracker and learn to understand auxiliary modalities only from training data. Constrained by the limited multi-modal training data, the performance of these methods is unsatisfactory. To alleviate this limitation, this work proposes a unified multi-modal tracker Diff-MM by exploiting the multi-modal understanding capability of the pre-trained text-to-image generation model. Diff-MM leverages the UNet of pre-trained Stable Diffusion as a tracking feature extractor through the proposed parallel feature extraction pipeline, which enables pairwise image inputs for object tracking. We further introduce a multi-modal sub-module tuning method that learns to gain complementary information between different modalities. By harnessing the extensive prior knowledge in the generation model, we achieve a unified tracker with uniform parameters for RGB-N/D/T/E tracking. Experimental results demonstrate the promising performance of our method compared with recently proposed trackers, e.g., its AUC outperforms OneTracker by 8.3% on TNL2K.

Figures

Figures reproduced from arXiv: 2505.12606 by the authors.

Figure 1
Figure 1. (a) Recent multi-modal trackers typically start from a foundation RGB tracker trained [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Illustrations of the overall framework and the parallel feature extraction pipeline of Diff [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Visualization of tracking results. Effectiveness of Multi-modal Sub-module Tuning. We further validate the effectiveness of MST in [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

60 extracted references · 44 canonical work pages

  1. [1]

    Fully-convolutional siamese networks for object tracking

    Luca Bertinetto, Jack Valmadre, João F Henriques, Andrea Vedaldi, and Philip H S Torr. Fully-convolutional siamese networks for object tracking. In ECCVW, 2016

    work page 2016

  2. [2]

    Transformer tracking

    Xin Chen, Bin Yan, Jiawen Zhu, Dong Wang, Xiaoyun Yang, and Huchuan Lu. Transformer tracking. In CVPR, 2021

    work page 2021

  3. [3]

    ECO: Efficient convolution operators for tracking

    Martin Danelljan, Goutam Bhat, Fahad Shahbaz Khan, and Michael Felsberg. ECO: Efficient convolution operators for tracking. In CVPR, 2017

    work page 2017

  4. [4]

    Probabilistic regression for visual tracking

    Martin Danelljan, Luc Van Gool, and Radu Timofte. Probabilistic regression for visual tracking. In CVPR, 2020

    work page 2020

  5. [5]

    An image is worth 16x16 words: Transformers for image recognition at scale

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. ICLR, 2021

    2021

  6. [6]

    LaSOT: A high-quality benchmark for large-scale single object tracking

    Heng Fan, Liting Lin, Fan Yang, Peng Chu, Ge Deng, Sijia Yu, Hexin Bai, Yong Xu, Chunyuan Liao, and Haibin Ling. LaSOT: A high-quality benchmark for large-scale single object tracking. In CVPR, 2019

    work page 2019

  7. [7]

    Siamese natural language tracker: Tracking by natural language descriptions with siamese trackers

    Qi Feng, Vitaly Ablavsky, Qinxun Bai, and Stan Sclaroff. Siamese natural language tracker: Tracking by natural language descriptions with siamese trackers. In CVPR, pages 5851–5860, 2021

    work page 2021

  8. [8]

    Geowizard: Unleashing the diffusion priors for 3d geometry estimation from a single image

    Xiao Fu, Wei Yin, Mu Hu, Kaixuan Wang, Yuexin Ma, Ping Tan, Shaojie Shen, Dahua Lin, and Xiaoxiao Long. Geowizard: Unleashing the diffusion priors for 3d geometry estimation from a single image. In ECCV, pages 241–258. Springer, 2024

    work page 2024

Show all 60 references
  1. [9]

    Deep adaptive fusion network for high performance RGBT tracking

    Yuan Gao, Chenglong Li, Yabin Zhu, Jin Tang, Tao He, and Futian Wang. Deep adaptive fusion network for high performance RGBT tracking. In ICCVW, pages 0–0, 2019

    2019

  2. [10]

    Are we ready for autonomous driving? the kitti vision benchmark suite

    Andreas Geiger, Philip Lenz, and Raquel Urtasun. Are we ready for autonomous driving? the kitti vision benchmark suite. In CVPR, pages 3354–3361. IEEE, 2012

    2012

  3. [11]

    Generative adversarial nets

    Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. NeurIPS, 27, 2014

    2014

  4. [12]

    High-speed tracking with kernelized correlation filters

    João F Henriques, Rui Caseiro, Pedro Martins, and Jorge Batista. High-speed tracking with kernelized correlation filters. In ICVS, 2008

    2008

  5. [13]

    Denoising diffusion probabilistic models

    Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. NeurIPS, 33: 6840–6851, 2020

    2020

  6. [14]

    Onetracker: Unifying visual object tracking with foundation models and efficient tuning

    Lingyi Hong, Shilin Yan, Renrui Zhang, Wanyun Li, Xinyu Zhou, Pinxue Guo, Kaixun Jiang, Yiting Chen, Jinglun Li, Zhaoyu Chen, et al. Onetracker: Unifying visual object tracking with foundation models and efficient tuning. In CVPR, pages 19079–19091, 2024

    2024

  7. [15]

    Sdstrack: Self-distillation symmetric adapter learning for multi-modal visual object tracking

    Xiaojun Hou, Jiazheng Xing, Yijie Qian, Yaowei Guo, Shuo Xin, Junhao Chen, Kai Tang, Mengmeng Wang, Zhengkai Jiang, Liang Liu, et al. Sdstrack: Self-distillation symmetric adapter learning for multi-modal visual object tracking. In CVPR, pages 26551–26561, 2024

    2024

  8. [16]

    Lora: Low-rank adaptation of large language models

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. In ICLR, 2022

    2022

  9. [17]

    Got-10k: A large high-diversity benchmark for generic object tracking in the wild

    Lianghua Huang, Xin Zhao, and Kaiqi Huang. Got-10k: A large high-diversity benchmark for generic object tracking in the wild. TPAMI, 2019

    2019

  10. [18]

    Visual prompt tuning

    Menglin Jia, Luming Tang, Bor-Chun Chen, Claire Cardie, Serge Belongie, Bharath Hariharan, and Ser-Nam Lim. Visual prompt tuning. In ECCV, 2022

    2022

  11. [19]

    Repurposing diffusion-based image generators for monocular depth estimation

    Bingxin Ke, Anton Obukhov, Shengyu Huang, Nando Metzger, Rodrigo Caye Daudt, and Konrad Schindler. Repurposing diffusion-based image generators for monocular depth estimation. InCVPR, pages 9492–9502, 2024

    2024

  12. [20]

    The sixth visual object tracking VOT2018 challenge results

    Matej Kristan, Ales Leonardis, Jiri Matas, Michael Felsberg, Roman Pfugfelder, Luka Cehovin Zajc, Tomas V ojir, Goutam Bhat, Alan Lukezic, Abdelrahman Eldesokey, Gustavo Fernandez, and et al. The sixth visual object tracking VOT2018 challenge results. In ECCV, 2018. 10

    2018

  13. [21]

    The tenth visual object tracking vot2022 challenge results

    Matej Kristan, Aleš Leonardis, Jiˇrí Matas, Michael Felsberg, Roman Pflugfelder, Joni-Kristian Kämäräinen, Hyung Jin Chang, Martin Danelljan, Luka ˇCehovin Zajc, Alan Lukeži ˇc, et al. The tenth visual object tracking vot2022 challenge results. In ECCVW, pages 431–460. Springer, 2023

    2023

  14. [22]

    Stable diffusion image variations

    Lambdalabs. Stable diffusion image variations. https://huggingface.co/lambdalabs/ sd-image-variations-diffusers , 2022

    2022

  15. [23]

    Cornernet: Detecting objects as paired keypoints

    Hei Law and Jia Deng. Cornernet: Detecting objects as paired keypoints. In ECCV, 2018

    2018

  16. [24]

    SiamRPN++: Evolution of siamese visual tracking with very deep networks

    Bo Li, Wei Wu, Qiang Wang, Fangyi Zhang, Junliang Xing, and Junjie Yan. SiamRPN++: Evolution of siamese visual tracking with very deep networks. In CVPR, 2019

    2019

  17. [25]

    RGB-T object tracking: Benchmark and baseline

    Chenglong Li, Xinyan Liang, Yijuan Lu, Nan Zhao, and Jin Tang. RGB-T object tracking: Benchmark and baseline. Pattern Recognition, 96:106977, 2019

    2019

  18. [26]

    Lasher: A large-scale high-diversity benchmark for RGBT tracking

    Chenglong Li, Wanlin Xue, Yaqing Jia, Zhichen Qu, Bin Luo, Jin Tang, and Dengdi Sun. Lasher: A large-scale high-diversity benchmark for RGBT tracking. TIP, 31:392–404, 2021

    2021

  19. [27]

    Tracking by natural language specification

    Zhenyang Li, Ran Tao, Efstratios Gavves, Cees GM Snoek, and Arnold WM Smeulders. Tracking by natural language specification. In CVPR, pages 6495–6503, 2017

    2017

  20. [28]

    Belongie, Lubomir D

    Tsung-Yi Lin, Michael Maire, Serge J. Belongie, Lubomir D. Bourdev, Ross B. Girshick, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. Microsoft COCO: Common objects in context. In ECCV, 2014

    2014

  21. [29]

    Decoupled weight decay regularization

    I Loshchilov. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101, 2017

    2017 arXiv

  22. [30]

    Narrowing the Synthetic-to- Real Gap for Thermal Infrared Semantic Image Segmentation Using Diffusion-based Conditional Image Synthesis

    Christian Mayr, Christian Kuebler, Norbert Haala, and Michael Teutsch. Narrowing the Synthetic-to- Real Gap for Thermal Infrared Semantic Image Segmentation Using Diffusion-based Conditional Image Synthesis. In CVPRW, pages 3131–3141, 2024

    2024

  23. [31]

    T2i- adapter: Learning adapters to dig out more controllable ability for text-to-image diffusion models

    Chong Mou, Xintao Wang, Liangbin Xie, Yanze Wu, Jian Zhang, Zhongang Qi, and Ying Shan. T2i- adapter: Learning adapters to dig out more controllable ability for text-to-image diffusion models. In AAAI, pages 4296–4304, 2024

    2024

  24. [32]

    TrackingNet: A large-scale dataset and benchmark for object tracking in the wild

    Matthias Muller, Adel Bibi, Silvio Giancola, Salman Alsubaihi, and Bernard Ghanem. TrackingNet: A large-scale dataset and benchmark for object tracking in the wild. In ECCV, 2018

    2018

  25. [33]

    Learning multi–domain convolutional neural networks for visual tracking

    Hyeonseob Nam and Bohyung Han. Learning multi–domain convolutional neural networks for visual tracking. In CVPR, 2016

    2016

  26. [34]

    Scalable diffusion models with transformers

    William Peebles and Saining Xie. Scalable diffusion models with transformers. InICCV, pages 4195–4205, 2023

    2023

  27. [35]

    Zero: Memory optimizations toward training trillion parameter models

    Samyam Rajbhandari, Jeff Rasley, Olatunji Ruwase, and Yuxiong He. Zero: Memory optimizations toward training trillion parameter models. In SC20: International Conference for High Performance Computing, Networking, Storage and Analysis, pages 1–16. IEEE, 2020

    2020

  28. [36]

    Generalized intersection over union: A metric and a loss for bounding box regression

    Hamid Rezatofighi, Nathan Tsoi, JunYoung Gwak, Amir Sadeghian, Ian Reid, and Silvio Savarese. Generalized intersection over union: A metric and a loss for bounding box regression. In CVPR, 2019

    2019

  29. [37]

    High-resolution image synthesis with latent diffusion models

    Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent diffusion models. In CVPR, pages 10684–10695, 2022

    2022

  30. [38]

    Laion-5b: An open large-scale dataset for training next generation image-text models

    Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion-5b: An open large-scale dataset for training next generation image-text models. NeurIPS, 35:25278–25294, 2022

    2022

  31. [39]

    Transformer rgbt tracking with spatio- temporal multimodal tokens

    Dengdi Sun, Yajie Pan, Andong Lu, Chenglong Li, and Bin Luo. Transformer rgbt tracking with spatio- temporal multimodal tokens. TCSVT, 34(11):12059–12072, 2024

    2024

  32. [40]

    Towards a generalist and blind rgb-x tracker.arXiv preprint arXiv:2405.17773, 2024

    Yuedong Tan, Zongwei Wu, Yuqian Fu, Zhuyun Zhou, Guolei Sun, Chao Ma, Danda Pani Paudel, Luc Van Gool, and Radu Timofte. Towards a generalist and blind rgb-x tracker.arXiv preprint arXiv:2405.17773, 2024

    2024 arXiv

  33. [41]

    Emergent correspondence from image diffusion

    Luming Tang, Menglin Jia, Qianqian Wang, Cheng Perng Phoo, and Bharath Hariharan. Emergent correspondence from image diffusion. NeurIPS, 36:1363–1389, 2023. 11

    2023

  34. [42]

    Embodiedscan: A holistic multi-modal 3d perception suite towards embodied ai

    Tai Wang, Xiaohan Mao, Chenming Zhu, Runsen Xu, Ruiyuan Lyu, Peisen Li, Xiao Chen, Wenwei Zhang, Kai Chen, Tianfan Xue, et al. Embodiedscan: A holistic multi-modal 3d perception suite towards embodied ai. In CVPR, pages 19757–19767, 2024

    2024

  35. [43]

    Visevent: Reliable object tracking via collaboration of frame and event flows

    Xiao Wang, Jianing Li, Lin Zhu, Zhipeng Zhang, Zhe Chen, Xin Li, Yaowei Wang, Yonghong Tian, and Feng Wu. Visevent: Reliable object tracking via collaboration of frame and event flows. arXiv preprint arXiv:2108.05015, 2021

    2021 arXiv

  36. [44]

    Towards more flexible and accurate object tracking with natural language: Algorithms and benchmark

    Xiao Wang, Xiujun Shu, Zhipeng Zhang, Bo Jiang, Yaowei Wang, Yonghong Tian, and Feng Wu. Towards more flexible and accurate object tracking with natural language: Algorithms and benchmark. In CVPR, pages 13763–13773, 2021

    2021

  37. [45]

    E-motion: Future motion simulation via event sequence diffusion

    Song Wu, Zhiyu Zhu, Junhui Hou, Guangming Shi, and Jinjian Wu. E-motion: Future motion simulation via event sequence diffusion. NeurIPS, 37:105552–105582, 2025

    2025

  38. [46]

    Object tracking benchmark

    Yi Wu, Jongwoo Lim, and Ming Hsuan Yang. Object tracking benchmark. TPAMI, 2015

    2015

  39. [47]

    Single-model and any-modality for video object tracking

    Zongwei Wu, Jilai Zheng, Xiangxuan Ren, Florin-Alexandru Vasluianu, Chao Ma, Danda Pani Paudel, Luc Van Gool, and Radu Timofte. Single-model and any-modality for video object tracking. In CVPR, pages 19156–19166, 2024

    2024

  40. [48]

    Multiple human tracking based on multi-view upper-body detection and discriminative learning

    Junliang Xing, Haizhou Ai, and Shihong Lao. Multiple human tracking based on multi-view upper-body detection and discriminative learning. In ICPR, pages 1698–1701. IEEE, 2010

    2010

  41. [49]

    Learning spatio-temporal transformer for visual tracking

    Bin Yan, Houwen Peng, Jianlong Fu, Dong Wang, and Huchuan Lu. Learning spatio-temporal transformer for visual tracking. In ICCV, 2021

    2021

  42. [50]

    Depth- track: Unveiling the power of RGBD tracking

    Song Yan, Jinyu Yang, Jani Käpylä, Feng Zheng, Aleš Leonardis, and Joni-Kristian Kämäräinen. Depth- track: Unveiling the power of RGBD tracking. In ICCV, pages 10725–10733, 2021

    2021

  43. [51]

    Prompting for multi-modal tracking

    Jinyu Yang, Zhe Li, Feng Zheng, Ales Leonardis, and Jingkuan Song. Prompting for multi-modal tracking. In ACMMM, pages 3492–3500, 2022

    2022

  44. [52]

    Joint feature learning and relation modeling for tracking: A one-stream framework

    Botao Ye, Hong Chang, Bingpeng Ma, Shiguang Shan, and Xilin Chen. Joint feature learning and relation modeling for tracking: A one-stream framework. In ECCV, pages 341–357. Springer, 2022

    2022

  45. [53]

    Adding conditional control to text-to-image diffusion models

    Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. In ICCV, pages 3836–3847, 2023

    2023

  46. [54]

    Paste, inpaint and harmonize via denoising: Subject-driven image editing with pre-trained diffusion model.arXiv preprint arXiv:2306.07596, 2023

    Xin Zhang, Jiaxian Guo, Paul Yoo, Yutaka Matsuo, and Yusuke Iwasawa. Paste, inpaint and harmonize via denoising: Subject-driven image editing with pre-trained diffusion model.arXiv preprint arXiv:2306.07596, 2023

    2023 arXiv

  47. [55]

    Diff-tracker: Text-to-image diffusion models are unsupervised trackers

    Zhengbo Zhang, Li Xu, Duo Peng, Hossein Rahmani, and Jun Liu. Diff-tracker: Text-to-image diffusion models are unsupervised trackers. In ECCV. Springer, 2024

    2024

  48. [56]

    Unleashing text-to-image diffusion models for visual perception

    Wenliang Zhao, Yongming Rao, Zuyan Liu, Benlin Liu, Jie Zhou, and Jiwen Lu. Unleashing text-to-image diffusion models for visual perception. In ICCV, pages 5729–5739, 2023

    2023

  49. [57]

    Joint visual grounding and tracking with natural language specification

    Li Zhou, Zikun Zhou, Kaige Mao, and Zhenyu He. Joint visual grounding and tracking with natural language specification. In CVPR, pages 23151–23160, 2023

    2023

  50. [58]

    Visual prompt multi-modal tracking

    Jiawen Zhu, Simiao Lai, Xin Chen, Dong Wang, and Huchuan Lu. Visual prompt multi-modal tracking. In CVPR, 2023

    2023

  51. [59]

    RGBD1K: A large-scale dataset and benchmark for RGB-D object tracking

    Xue-Feng Zhu, Tianyang Xu, Zhangyong Tang, Zucheng Wu, Haodong Liu, Xiao Yang, Xiao-Jun Wu, and Josef Kittler. RGBD1K: A large-scale dataset and benchmark for RGB-D object tracking. AAAI, 2023

    2023

  52. [60]

    Exploring pre-trained text-to-video diffusion models for referring video object segmentation

    Zixin Zhu, Xuelu Feng, Dongdong Chen, Junsong Yuan, Chunming Qiao, and Gang Hua. Exploring pre-trained text-to-video diffusion models for referring video object segmentation. In ECCV, 2024. 12

    2024

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.