Pith. sign in

REVIEW 3 major objections 4 minor 53 references

VideoMolmo: Spatio-Temporal Grounding Meets Pointing

T0 review · 3 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read VideoMolmo is a large multimodal model that grounds natural-language queries in videos by outputting precise point coordinates, and it reports consistent gains over prior video-grounding models on point, mask, counting, and reasoning…

desk verdict Solid video pointing model with a valuable dataset and credible external results, but the headline benchmark numbers are partly circular and the point-level claim lacks video baselines. read the letter →

arxiv 2506.05336 v2 pith:2ME2GKP3 submitted 2025-06-05 cs.CV

Ghazi Shazan Ahmad , Ahmed Heakl , Hanan Gani , Abdelrahman Shaker , Zhiqiang Shen , Fahad Shahbaz Khan , Salman Khan This is my paper
classification cs.CV
keywords spatio-temporalgroundingvideopointinglargemultimodalmodelreferringobjectsegmentationSAM2temporalattentionpointsupervisionreasoning
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

VideoMolmo is a large multimodal model that answers natural-language queries about videos by outputting precise point coordinates for the objects mentioned in the query, rather than dense masks. The paper argues that splitting video grounding into two stages—an LLM that emits points and a post-processing module that converts those points into temporally consistent masks with SAM2—makes the problem easier for the model and more interpretable, and it reports consistent accuracy gains over existing video-grounding models. To support this, the authors built a 72k-video-caption dataset with 100k object points using a semi-automatic SAM2-based annotation pipeline, and they introduced VPoS-Bench, a five-scenario benchmark spanning cell tracking, egocentric video, autonomous driving, video-GUI interaction, and robotics. Their central claim is that this system improves spatio-temporal pointing accuracy and reasoning across point grounding, counting, referring video segmentation, and reasoning segmentation, with the largest reported gains being 5.4 percentage points average on VPoS-Bench and 9.5 points on the J&F metric (the average of region Jaccard and boundary F-measure) on MeViS. If the claim holds, fine-grained video interaction in fields such as robotics, assistive technology, and autonomous driving can be driven by text instructions with point-level outputs that are cheaper to annotate and easier to debug than mask supervision.

What carries the argument

The load-bearing machinery is the point-then-mask decomposition. A temporal module $M$ runs multi-head cross-attention over local $2\times2$ window features, using the current frame's windows as queries and the mean of the previous $l$ frames' windows as keys and values, then adds the attended features back to the current frame to preserve spatial detail while injecting temporal context. The bidirectional temporal mask fusion module samples frames at rate $k$, obtains SAM2 masks for those frames from predicted points, propagates masks forward and backward to intermediate frames, and fuses the two propagations by IoU threshold $\tau$, taking the intersection when they agree and the union otherwise, with a one-sided fallback if a propagation is empty. Eq. (8) couples the training supervision to SAM2's point-to-mask behavior, since each training point is selected as the candidate whose SAM2 mask best matches the ground-truth mask.

What would settle it

Score the models directly on VPoS-Bench using the human-annotated point references and a point-distance metric such as F1 at a fixed radius, without converting predictions to masks. If VideoMolmo's point predictions are not clearly more accurate than Molmo+SAM2's under this direct point metric, the reported 5.4 pp mask gain is inflated by SAM2's fusion and propagation rather than by better spatio-temporal grounding.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is that a video large multimodal model can perform fine-grained spatio-temporal pointing from text by decomposing the task: the LLM autoregressively emits normalized $(x,y)$ coordinates for each queried object, and a new bidirectional temporal mask fusion module uses SAM2 to propagate those points into coherent masks across the video. The architecture conditions each frame on the mean of the previous $l$ frames through a local multi-head cross-attention module over $2\times2$ patch windows, and ablations show this beats single-frame, addition, and concatenation alternatives. Training points are chosen by the annotation pipeline of Eq. (8): sample candidate points inside a ground-truth mask, run SAM2 on each, and keep the candidate whose SAM2 mask has the highest Intersection-over-Union with the ground-truth mask. The same SAM2-based conversion is used at evaluation, and the authors show the mask-fusion post-processing improves not only VideoMolmo itself but also the Molmo+SAM2 baseline.

Load-bearing premise

The load-bearing premise is that SAM2 turns a point into a faithful mask of the intended object, because the same converter is used to build training labels, to convert predicted points into evaluation masks, and to construct the VPoS-Bench reference masks.

Editorial extensions

If this is right

  • If the central claim is correct, video grounding can be trained with point supervision instead of dense mask supervision, making annotation substantially cheaper while leaving reasoning to the language model.
  • The VPoS-Bench results imply that the model transfers to out-of-distribution domains such as cell tracking and autonomous driving without task-specific fine-tuning.
  • The bidirectional mask fusion is a plug-in post-processor: the paper shows it improves the Molmo+SAM2 baseline as well, so other point-predicting video models could adopt it directly.
  • Because the same point-output interface also improves counting and reasoning segmentation, one decoder can serve multiple downstream tasks that need object-level localization rather than full masks.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper, if SAM2 systematically prefers certain point placements, the Eq. (8) training labels and the VPoS-Bench reference masks both inherit that bias, so the 5.4 pp mask gain could partly measure alignment with SAM2's preferences rather than better grounding.
  • Beyond the paper, the fixed IoU threshold $\tau=0.7$ in the fusion rule could be replaced by a learned or flow-aware fusion for occlusion and fast-motion cases, where the two propagated masks are likely to disagree spatially.
  • Beyond the paper, the single-point-per-object output limits mask quality for elongated or multi-part objects, so predicting multiple points per query and prompting SAM2 with all of them is a natural, testable extension.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper introduces VideoMolmo, a large multimodal model for text-conditioned spatio-temporal pointing in videos. It extends Molmo with a temporal module that uses per-window cross-attention over the mean of preceding frames, and it adds a post-processing pipeline that converts predicted points into temporally coherent masks via SAM2 with bidirectional propagation and IoU-based fusion. The authors also contribute a training dataset of 72k video-caption pairs with 100k object points, generated by a semi-automatic SAM2-based annotation pipeline, and introduce VPoS-Bench, a five-scenario out-of-distribution benchmark with 100 video-caption pairs and 1k manual point annotations. Experiments report point-grounding results against Molmo, mask-level results on VPoS-Bench, referring video segmentation on Refer-DAVIS-17, Refer-YouTube-VOS, and MeViS, reasoning segmentation on ReasonVOS, and a counting task. The central claims are that VideoMolmo substantially improves spatio-temporal pointing accuracy and reasoning capability.

Significance. If the central claims hold, the paper makes a useful contribution by decomposing video grounding into point prediction followed by mask lifting, which is a simpler and more interpretable training target than dense mask prediction. The public release of code, models, the curated dataset, and the VPoS-Bench benchmark are concrete assets. The strongest evidence in the paper is the consistent improvement on external, human-GT mask benchmarks: MeViS J&F improves by about 7 points over the strongest prior baseline, and VideoMolmo also leads on Refer-DAVIS-17, Refer-YouTube-VOS, and ReasonVOS. However, the headline claim about spatio-temporal pointing accuracy is not yet directly supported against video-capable baselines, and the VPoS-Bench mask evaluation shares a SAM2 component across training-label selection, predicted-mask generation, and ground-truth construction, which makes the reported 5.4 pp average improvement difficult to attribute cleanly to better point grounding.

major comments (3)
  1. [§5.1, Fig. 5, Table 4] The point-level evaluation that underlies the central claim compares VideoMolmo only with Molmo, a static-image model, in Fig. 5. The video-capable baselines VideoLISA and VideoGLaMM appear in the VPoS-Bench results only through mask-level J/F/J&F scores in Table 4, after VideoMolmo's points have been lifted to masks by SAM2. Since these baselines output masks, a direct point-level comparison is feasible by converting their predicted masks to point coordinates (e.g., centroids or extreme points) and evaluating all models on the 1k manually annotated point locations in VPoS-Bench with the same Precision/Recall/F1 protocol. Without such a control, the claim that VideoMolmo 'substantially improves spatio-temporal pointing accuracy' over existing video models is underdetermined by the reported evidence.
  2. [Eq. (8), Sec. 3.3, Appendix A.3] The VPoS-Bench mask evaluation is partially self-referential with respect to SAM2. Training labels are selected as the point whose SAM2-prompted mask best matches the original ground-truth mask (Eq. 8); predicted points are converted to masks with SAM2 (Sec. 3.3); and the VPoS-Bench ground-truth masks are themselves SAM2 segmentations of manually annotated points (Appendix A.3). If SAM2 has systematic point preferences, the training signal and the evaluation metric jointly reward models that mimic those preferences rather than models that locate the semantically correct object position. The reported 5.4 pp average improvement on VPoS-Bench (Table 4) could therefore be inflated. To support the mask-level claim, the authors should report point-level accuracy on the manual VPoS-Bench point annotations for all models, and ideally also compare mask metrics on subsets where independent human-drawn masks are available, or compare SAM2 mask quality from predicted points against masks from random or centrally sampled points.
  3. [Table 4, Appendix A.3] The VPoS-Bench subtasks are small: the five categories contain 12, 18, 13, 13, and 14 videos respectively, with 100 video-caption pairs in total. Table 4 reports no confidence intervals, error bars, or significance tests, and per-subtask differences vary widely (e.g., 2.9 vs 57.4 J&F on the Robotics and Autonomous subsets for VideoLISA). Given the small sample sizes, the headline 5.4 pp average improvement over the strongest baseline needs a variance estimate or per-video analysis before it can be read as a stable generalization result.
minor comments (4)
  1. [Sec. 5.1, referring segmentation paragraph] The sentence 'which contrasts with recent methods like VideoGLaMM [23] and VideoMolmo [3]' appears to contain a citation typo: [3] is VideoLISA, not VideoMolmo, and the text should name VideoLISA here.
  2. [Appendix A.1.2, Table 10] The Molmo+SAM2 baseline is tuned per dataset for the sampling rate k (Appendix A.1.2), while VideoMolmo uses k=5 on Refer-YouTube-VOS even though the ablation in Table 10 shows k=20 gives J&F 68.14 versus 67.33. Reporting the best k for each method, or a unified protocol with justification, would make the comparisons fairer and the main-table numbers more representative.
  3. [Fig. 5] The point-grounding figure reports Precision, Recall, and F1 but does not state the number of points or videos used, and it shows no error bars; please add the sample size and, where feasible, variability across VPoS-Bench subtasks.
  4. [Abstract and Sec. 5] The abstract claims 'substantially improves spatio-temporal pointing accuracy and reasoning capability,' but the reasoning evaluation is currently limited to mask-level J&F on ReasonVOS; consider adding a reasoning-specific point-level metric or tempering the claim to match the evidence.

Circularity Check

1 steps flagged · score 4.0 of 10

VPoS-Bench mask evaluation is partially self-referential through SAM2: training labels are SAM2-IoU-optimal points and test masks are SAM2 segmentations of both predicted and manual points.

  1. fitted input called prediction [Sec. 4, Eq. (8); Sec. 3.3; Appendix A.3; Table 4]
    "The point coordinate whose predicted mask achieves the highest IoU is selected as the representative ground truth point for the object: p∗ = arg max (x,y) IoU (SAM2(x, y), mj), where SAM2(x, y) denotes the predicted mask obtained using point (x, y) as a prompt to SAM2. ... we use the predicted points as prompts to SAM2 to obtain the segmentation masks of the objects. ... For mask-based evaluations, we employ the SAM2 model to convert these point annotations into segmentation masks."

    The training label is defined as the point that maximizes the same SAM2 mask-IoU function later used for evaluation. During training (Eq. 8), the model is taught to emit points for which SAM2(x,y) best matches the ground-truth mask. At test time on VPoS-Bench, the predicted point is fed to SAM2 to form the predicted mask (Sec. 3.3), and the manual point is fed to SAM2 to form the 'ground-truth' mask (Appendix A.3); mask J&F then measures agreement between two SAM2-derived masks. A model trained with Eq. 8 is therefore rewarded exactly for reproducing SAM2's preferred point-to-mask behavior, so part of the 5.4 pp gain over Molmo+SAM2, and much of the gain over direct-mask baselines, is built into the training/evaluation loop rather than demonstrated by independent point localization.

full rationale

VideoMolmo's central claim is empirical, not a derivation, and most of the architecture (Molmo, Qwen2, SAM2) is externally supplied. No load-bearing uniqueness theorem or ansatz is imported from the authors' own prior work; self-citations (e.g., VideoGLaMM) appear only as baselines and are not load-bearing. The point-grounding result in Fig. 5 is an independent comparison against Molmo using point-level Precision/Recall/F1, which does not pass through SAM2. The circular concern is confined to the mask-based VPoS-Bench numbers: the training target in Eq. 8 is literally the point that maximizes IoU(SAM2(x,y), GT mask), so the model is optimized to produce SAM2-preferred points, and Appendix A.3 converts both manual and predicted points to masks with the same SAM2 function for evaluation. The reported 5.4 pp gain over Molmo+SAM2 therefore conflates genuine pointing improvements with learned alignment to SAM2's point-to-mask mapping; against direct-mask baselines (VideoLISA, VideoGLaMM) the comparison is also asymmetric. The paper itself acknowledges this dependence in the Limitations section. Because the human-annotated points and the point-level metric remain independent, the circularity is partial, not total.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

The central claim rests on three external pillars: pretrained Molmo/SAM2 components, the assumption that SAM2 is a faithful oracle for points-to-masks, and a diverse training mixture. The SAM2 oracle assumption is the most exposed because it appears in both training label selection and in VPoS-Bench mask evaluation, creating a closed loop for that benchmark. No invented physical or conceptual entities are introduced.

free parameters (4)
  • Temporal module context length l = 4 (best in Fig. 6)
    Number of preceding frames used in cross-attention; selected by ablation on Refer-DAVIS, with performance peaking at l=4 and dropping at l=5.
  • Mask fusion IoU threshold tau = 0.7
    Ablated in Table 9 on Refer-DAVIS; the authors choose 0.7 as best, though differences across values are small.
  • Frame sampling rate k for point prediction = 5 for main results (k=20 gives 68.14 vs 67.33 on Ref-YouTube-VOS)
    k controls how often points are predicted; Table 10 shows sensitivity, and main reported results use a consistent but not optimal setting.
  • Candidate point count k in the annotation pipeline = not specified
    The semi-automatic annotation pipeline samples k candidate points per object (Sec. 4) and picks the one with highest SAM2 IoU; k is not reported, so the annotation quality dependence on k is unknown.
assumptions (5)
  • domain assumption Pretrained CLIP ViT-L/14 and Qwen2-7B weights transfer to video pointing after fine-tuning with the temporal module.
    The model is initialized from Molmo components and no isolated transfer study is run; end-task metrics are the only evidence.
  • domain assumption SAM2 point-to-mask conversion is accurate enough to serve as an oracle for both training label selection and evaluation masks.
    Eq. (8) selects training points by SAM2 IoU with GT masks; Sec. 3.3 converts predicted points to masks via SAM2; Appendix A.3 creates VPoS-Bench GT masks via SAM2 from manual points.
  • ad hoc to paper Cross-attention over 2x2 window features from the mean of the previous l frames captures the temporal cues needed for grounding.
    Sec. 3.2 defines the temporal module; if pooling to a mean loses motion information, the proposed temporal advantage would not materialize.
  • domain assumption Manual point annotations in VPoS-Bench are unambiguous across the five scenarios.
    No inter-annotator agreement or annotation protocol details are reported in Appendix A.3.
  • domain assumption The seven source datasets used to build the 72k training set provide enough diversity for out-of-distribution generalization to VPoS-Bench.
    Section 4 lists the sources; the OOD benchmark claim rests on this mixture being representative enough.

how reviews work

0 comments
Cite this review

Pith. "Pith review of VideoMolmo: Spatio-Temporal Grounding Meets Pointing." pith.science (2026). https://pith.science/paper/2ME2GKP3

@misc{pith2026250605336,
  author       = {Pith},
  title        = {Pith review of: VideoMolmo: Spatio-Temporal Grounding Meets Pointing},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/2ME2GKP3}},
  note         = {Machine review of arXiv:2506.05336}
}
read the original abstract

Spatio-temporal localization is vital for precise interactions across diverse domains, from biological research to autonomous navigation and interactive interfaces. Current video-based approaches, while proficient in tracking, lack the sophisticated reasoning capabilities of large language models, limiting their contextual understanding and generalization. We introduce VideoMolmo, a large multimodal model tailored for fine-grained spatio-temporal pointing conditioned on textual descriptions. Building upon the Molmo architecture, VideoMolmo incorporates a temporal module utilizing an attention mechanism to condition each frame on preceding frames, ensuring temporal consistency. Additionally, our novel temporal mask fusion pipeline employs SAM2 for bidirectional point propagation, significantly enhancing coherence across video sequences. This two-step decomposition, i.e., first using the LLM to generate precise pointing coordinates, then relying on a sequential mask-fusion module to produce coherent segmentation, not only simplifies the task for the language model but also enhances interpretability. Due to the lack of suitable datasets, we curate a comprehensive dataset comprising 72k video-caption pairs annotated with 100k object points. To evaluate the generalization of VideoMolmo, we introduce VPoS-Bench, a challenging out-of-distribution benchmark spanning five real-world scenarios: Cell Tracking, Egocentric Vision, Autonomous Driving, Video-GUI Interaction, and Robotics. We also evaluate our model on Referring Video Object Segmentation (Refer-VOS) and Reasoning VOS tasks. In comparison to existing models, VideoMolmo substantially improves spatio-temporal pointing accuracy and reasoning capability. Our code and models are publicly available at https://github.com/mbzuai-oryx/VideoMolmo.

Figures

Figures reproduced from arXiv: 2506.05336 by the authors.

Figure 1
Figure 1. Given complex referring expressions in natural language, [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. VIDEOMOLMO Architecture. The visual encoder extracts multi-crop features from the current frame and the past l frames. These temporal features provide contextual cues and are processed by the Temporal Module M via multi-head cross-attention, where the query comes from the current frame, and key and value from the mean of previous frames. The output is fused with the original features to enrich temporal cues while pr… view at source ↗
Figure 3
Figure 3. VIDEOMOLMO annotation pipeline: We construct point-level supervision from frame￾level masks using a semi-automatic process. For each frame, k points are sampled on the mask and passed to SAM2 to generate candidate masks. The point with the highest-IoU candidate mask (w.r.t. ground truth) is selected as the optimal annotation. This bidirectional temporal fusion ensures temporally consistent masks with reduced compute… view at source ↗
Figures from the paper (9 more)
Figure 4
Figure 4. Figure 4: VIDEOMOLMO demonstrates robust generalization and fine-grained spatio-temporal grounding across diverse out-of-distribution scenarios from our proposed benchmark, for instance, correctly pointing to traffic lights (2nd row) in challenging driving scenes despite never e…
Figure 5
Figure 5. Figure 5: Performance comparison of Video￾Molmo on point grounding. 0 1 2 3 4 5 l 69.5 70.0 70.5 71.0 71.5 72.0 72.5 J&F [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 7
Figure 7. Figure 7: Effect of Bidirectional temporal mask fusion on Molmo+SAM2 baseline. we follow a cosine learning-rate schedule; we also clip gradients to a maximum norm of 1.0 to guard against unstable updates. For all inference and reported results, we use 4-bit precision. A.3 VPoS B…
Figure 8
Figure 8. Figure 8: VPoS-Bench qualitative examples. 4 [PITH_FULL_IMAGE:figures/full_fig_p017_8.png]
Figure 9
Figure 9. Figure 9: MeVis qualitative examples. 5 [PITH_FULL_IMAGE:figures/full_fig_p018_9.png]
Figure 10
Figure 10. Figure 10: Refer-YouTube-VOS qualitative examples. Point to person at the back of the go-cart without a helmet. Point to the helmet worn by the biker. Point to a rope which the guy is hanging on [PITH_FULL_IMAGE:figures/full_fig_p019_10.png]
Figure 11
Figure 11. Figure 11: Refer-DAVIS qualitative examples. 6 [PITH_FULL_IMAGE:figures/full_fig_p019_11.png]
Figure 12
Figure 12. Figure 12: ReasonVOS qualitative examples. Point to a black harness with an airbag. Point to the paraglider lines [PITH_FULL_IMAGE:figures/full_fig_p020_12.png]
Figure 13
Figure 13. Figure 13: Qualitative failure cases of VIDEOMOLMO. 7 [PITH_FULL_IMAGE:figures/full_fig_p020_13.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

53 extracted references · 20 canonical work pages

  1. [1]

    Vicas: A dataset for combining holistic and pixel-level video understanding using captions with grounded segmentation.arXiv preprint arXiv:2412.09754, 2024

    Ali Athar, Xueqing Deng, and Liang-Chieh Chen. Vicas: A dataset for combining holistic and pixel-level video understanding using captions with grounded segmentation.arXiv preprint arXiv:2412.09754, 2024

    arXiv 2024

  2. [2]

    Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al. Qwen2. 5-vl technical report.arXiv preprint arXiv:2502.13923, 2025

    arXiv 2025

  3. [3]

    One token to seg them all: Language instructed reasoning segmentation in videos.Advances in Neural Information Processing Systems, 37:6833–6859, 2024

    Zechen Bai, Tong He, Haiyang Mei, Pichao Wang, Ziteng Gao, Joya Chen, Zheng Zhang, and Mike Zheng Shou. One token to seg them all: Language instructed reasoning segmentation in videos.Advances in Neural Information Processing Systems, 37:6833–6859, 2024

    2024

  4. [4]

    nuscenes: A multimodal dataset for autonomous driving

    Holger Caesar, Varun Bankiti, Alex H Lang, Sourabh V ora, Venice Erin Liong, Qiang Xu, Anush Krishnan, Yu Pan, Giancarlo Baldan, and Oscar Beijbom. nuscenes: A multimodal dataset for autonomous driving. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11621–11631, 2020

    2020

  5. [5]

    Molmo and pixmo: Open weights and open data for state-of-the-art multimodal models.arXiv preprint arXiv:2409.17146, 2024

    Matt Deitke, Christopher Clark, Sangho Lee, Rohun Tripathi, Yue Yang, Jae Sung Park, Mohammadreza Salehi, Niklas Muennighoff, Kyle Lo, Luca Soldaini, et al. Molmo and pixmo: Open weights and open data for state-of-the-art multimodal models.arXiv preprint arXiv:2409.17146, 2024

    arXiv 2024

  6. [6]

    MeViS: A large-scale benchmark for video segmentation with motion expressions

    Henghui Ding, Chang Liu, Shuting He, Xudong Jiang, and Chen Change Loy. MeViS: A large-scale benchmark for video segmentation with motion expressions. InICCV, 2023

    2023

  7. [7]

    Lasot: A high-quality benchmark for large-scale single object tracking

    Heng Fan, Liting Lin, Fan Yang, Peng Chu, Ge Deng, Sijia Yu, Hexin Bai, Yong Xu, Chunyuan Liao, and Haibin Ling. Lasot: A high-quality benchmark for large-scale single object tracking. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 5374–5383, 2019

    work page 2019

  8. [8]

    Ego4d: Around the world in 3,000 hours of egocentric video

    Kristen Grauman, Andrew Westbury, Eugene Byrne, Zachary Chavis, Antonino Furnari, Rohit Girdhar, Jackson Hamburger, Hao Jiang, Miao Liu, Xingyu Liu, et al. Ego4d: Around the world in 3,000 hours of egocentric video. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 18995–19012, 2022

    2022

Show all 53 references
  1. [9]

    The kinetics human action video dataset.arXiv preprint arXiv:1705.06950, 2017

    Will Kay, Joao Carreira, Karen Simonyan, Brian Zhang, Chloe Hillier, Sudheendra Vijaya- narasimhan, Fabio Viola, Tim Green, Trevor Back, Paul Natsev, et al. The kinetics human action video dataset.arXiv preprint arXiv:1705.06950, 2017. 10

    2017 arXiv

  2. [10]

    Video object segmentation with language referring expressions

    Anna Khoreva, Anna Rohrbach, and Bernt Schiele. Video object segmentation with language referring expressions. InComputer Vision–ACCV 2018: 14th Asian Conference on Computer Vision, Perth, Australia, December 2–6, 2018, Revised Selected Papers, Part IV 14, pages 123–141. Sprin...

    2018

  3. [11]

    Lisa: Reasoning segmentation via large language model.arXiv preprint arXiv:2308.00692, 2023

    Xin Lai, Zhuotao Tian, Yukang Chen, Yanwei Li, Yuhui Yuan, Shu Liu, and Jiaya Jia. Lisa: Reasoning segmentation via large language model.arXiv preprint arXiv:2308.00692, 2023

    2023 arXiv

  4. [12]

    Mast: A memory-augmented self-supervised tracker

    Zihang Lai, Erika Lu, and Weidi Xie. Mast: A memory-augmented self-supervised tracker. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6479–6488, 2020

    2020

  5. [13]

    Videochat: Chat-centric video understanding.arXiv:2305.06355, 2023

    Kunchang Li, Yinan He, Yi Wang, Yizhuo Li, Wenhai Wang, Ping Luo, Yali Wang, Limin Wang, and Yu Qiao. Videochat: Chat-centric video understanding.arXiv:2305.06355, 2023

    2023 arXiv

  6. [14]

    Cross-modal target retrieval for tracking by natural language

    Yihao Li, Jun Yu, Zhongpeng Cai, and Yuwen Pan. Cross-modal target retrieval for tracking by natural language. In2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pages 4927–4936, 2022

    2022

  7. [15]

    Video-llava: Learning united visual representation by alignment before projection.arXiv preprint arXiv:2311.10122, 2023

    Bin Lin, Bin Zhu, Yang Ye, Munan Ning, Peng Jin, and Li Yuan. Video-llava: Learning united visual representation by alignment before projection.arXiv preprint arXiv:2311.10122, 2023

    2023 arXiv

  8. [16]

    Videogui: A benchmark for gui automation from instructional videos.arXiv preprint arXiv:2406.10227, 2024

    Kevin Qinghong Lin, Linjie Li, Difei Gao, Qinchen Wu, Mingyi Yan, Zhengyuan Yang, Lijuan Wang, and Mike Zheng Shou. Videogui: A benchmark for gui automation from instructional videos.arXiv preprint arXiv:2406.10227, 2024

    2024 arXiv

  9. [17]

    Dimbert: Learning vision-language grounded representations with disentangled multimodal-attention

    Fenglin Liu, Xian Wu, Shen Ge, Xuancheng Ren, Wei Fan, Xu Sun, and Yuexian Zou. Dimbert: Learning vision-language grounded representations with disentangled multimodal-attention. ACM Transactions on Knowledge Discovery from Data (TKDD), 16(1):1–19, 2021

    2021

  10. [18]

    Visual instruction tuning.Advances in neural information processing systems, 36:34892–34916, 2023

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning.Advances in neural information processing systems, 36:34892–34916, 2023

    2023

  11. [19]

    Decoupled weight decay regularization

    Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. InInternational Conference on Learning Representations (ICLR), 2019

    2019

  12. [20]

    Videogpt+: Integrating image and video encoders for enhanced video understanding.arXiv preprint arXiv:2406.09418, 2024

    Muhammad Maaz, Hanoona Rasheed, Salman Khan, and Fahad Khan. Videogpt+: Integrating image and video encoders for enhanced video understanding.arXiv preprint arXiv:2406.09418, 2024

    2024 arXiv

  13. [21]

    Maška, V

    M. Maška, V . Ulman, P. Delgado-Rodriguez, E. Gómez-de Mariscal, T. Neˇcasová, F.A. Guer- rero Peña, T.I. Ren, E.M. Meyerowitz, T. Scherr, K. Löffler, R. Mikut, T. Guo, Y . Wang, J.P. Allebach, R. Bao, N.M. Al-Shakarji, G. Rahmon, I.E. Toubal, K. Palaniappan, F. Lux, P. Matula...

    2023

  14. [22]

    Methods for cell and particle tracking

    Erik Meijering, Oleksiy Dzyubachyk, and Ihor Smal. Methods for cell and particle tracking. In Methods in Enzymology, volume 504, pages 183–200. Academic Press, 2012

    2012

  15. [23]

    Videoglamm: A large multimodal model for pixel-level visual grounding in videos.arXiv preprint arXiv:2411.04923, 2024

    Shehan Munasinghe, Hanan Gani, Wenqi Zhu, Jiale Cao, Eric Xing, Fahad Shahbaz Khan, and Salman Khan. Videoglamm: A large multimodal model for pixel-level visual grounding in videos.arXiv preprint arXiv:2411.04923, 2024

    2024 arXiv

  16. [24]

    Type-to-track: Retrieve any object via prompt-based tracking.Advances in Neural Information Processing Systems, 36:3205–3219, 2023

    Pha Nguyen, Kha Gia Quach, Kris Kitani, and Khoa Luu. Type-to-track: Retrieve any object via prompt-based tracking.Advances in Neural Information Processing Systems, 36:3205–3219, 2023

    2023

  17. [25]

    Chatgpt: Large language model for human-style conversation

    OpenAI. Chatgpt: Large language model for human-style conversation. https://chat. openai.com, 2023

    2023

  18. [26]

    Gpt-4o system card.arXiv preprint arXiv:2410.21276, 2024

    OpenAI. Gpt-4o system card.arXiv preprint arXiv:2410.21276, 2024. 11

    2024 arXiv

  19. [27]

    Perazzi, J

    F. Perazzi, J. Pont-Tuset, B. McWilliams, L. Van Gool, M. Gross, and A. Sorkine-Hornung. A benchmark dataset and evaluation methodology for video object segmentation. InComputer Vision and Pattern Recognition, 2016

    2016

  20. [28]

    Learning transferable visual models from natural language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. InInternational conference on machine learning, pag...

    2021

  21. [29]

    Segment anything meets point tracking

    Frano Rajiˇc, Lei Ke, Yu-Wing Tai, Chi-Keung Tang, Martin Danelljan, and Fisher Yu. Segment anything meets point tracking. In2025 IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pages 9302–9311. IEEE, 2025

    2025

  22. [30]

    Glamm: Pixel grounding large multimodal model.arXiv preprint arXiv:2311.03356, 2023

    Hanoona Rasheed, Muhammad Maaz, Sahal Shaji, Abdelrahman Shaker, Salman Khan, Hisham Cholakkal, Rao M Anwer, Erix Xing, Ming-Hsuan Yang, and Fahad S Khan. Glamm: Pixel grounding large multimodal model.arXiv preprint arXiv:2311.03356, 2023

    2023 arXiv

  23. [31]

    Sam 2: Segment anything in images and videos.arXiv preprint arXiv:2408.00714, 2024

    Nikhila Ravi, Valentin Gabeur, Yuan-Ting Hu, Ronghang Hu, Chaitanya Ryali, Tengyu Ma, Haitham Khedr, Roman Rädle, Chloe Rolland, Laura Gustafson, et al. Sam 2: Segment anything in images and videos.arXiv preprint arXiv:2408.00714, 2024

    2024 arXiv

  24. [32]

    Urvos: Unified referring video object segmentation network with a large-scale benchmark

    Seonguk Seo, Joon-Young Lee, and Bohyung Han. Urvos: Unified referring video object segmentation network with a large-scale benchmark. InComputer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XV 16, pages 208–223. Springer, 2020

    2020

  25. [33]

    Urvos: Unified referring video object segmentation network with a large-scale benchmark

    Seonguk Seo, Joon-Young Lee, and Bohyung Han. Urvos: Unified referring video object segmentation network with a large-scale benchmark. In Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan-Michael Frahm, editors,Computer Vision – ECCV 2020, pages 208–223, Cham,

    2020

  26. [34]

    Malmm: Multi-agent large language models for zero-shot robotics manipulation.arXiv preprint arXiv:2411.17636, 2024

    Harsh Singh, Rocktim Jyoti Das, Mingfei Han, Preslav Nakov, and Ivan Laptev. Malmm: Multi-agent large language models for zero-shot robotics manipulation.arXiv preprint arXiv:2411.17636, 2024

    2024 arXiv

  27. [35]

    Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, Pouya Tafti, Léonard Hussenot, Pier Giuseppe Sessa, Aakanksha Chowdhery, Adam Roberts, Aditya Barua, Alex Botev, Alex...

    2024

  28. [36]

    Towards more flexible and accurate object tracking with natural language: Algorithms and benchmark

    Xiao Wang, Xiujun Shu, Zhipeng Zhang, Bo Jiang, Yaowei Wang, Yonghong Tian, and Feng Wu. Towards more flexible and accurate object tracking with natural language: Algorithms and benchmark. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pag...

    2021

  29. [37]

    Internvideo2: Scaling video foundation models for multimodal video understanding.arXiv preprint arXiv:2403.15377, 2024

    Yi Wang, Kunchang Li, Xinhao Li, Jiashuo Yu, Yinan He, Guo Chen, Baoqi Pei, Rongkun Zheng, Jilan Xu, Zun Wang, et al. Internvideo2: Scaling video foundation models for multimodal video understanding.arXiv preprint arXiv:2403.15377, 2024

    2024 arXiv

  30. [38]

    Visa: Reasoning video object segmentation via large language models

    Cilin Yan, Haochen Wang, Shilin Yan, Xiaolong Jiang, Yao Hu, Guoliang Kang, Weidi Xie, and Efstratios Gavves. Visa: Reasoning video object segmentation via large language models. In European Conference on Computer Vision, pages 98–115. Springer, 2024

    2024

  31. [39]

    An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, et al. Qwen2. 5 technical report.arXiv preprint arXiv:2412.15115, 2024

    2024 arXiv

  32. [40]

    Grounding- tracking-integration.IEEE Transactions on Circuits and Systems for Video Technology, 31(9):3433–3443, 2020

    Zhengyuan Yang, Tushar Kumar, Tianlang Chen, Jingsong Su, and Jiebo Luo. Grounding- tracking-integration.IEEE Transactions on Circuits and Systems for Video Technology, 31(9):3433–3443, 2020

    2020

  33. [41]

    Robopoint: A vision-language model for spatial affordance prediction for robotics.arXiv preprint arXiv:2406.10721, 2024

    Wentao Yuan, Jiafei Duan, Valts Blukis, Wilbert Pumacay, Ranjay Krishna, Adithyavairavan Murali, Arsalan Mousavian, and Dieter Fox. Robopoint: A vision-language model for spatial affordance prediction for robotics.arXiv preprint arXiv:2406.10721, 2024

    2024 arXiv

  34. [42]

    Videollama 3: Frontier multimodal foundation models for image and video understanding.arXiv preprint arXiv:2501.13106, 2025

    Boqiang Zhang, Kehan Li, Zesen Cheng, Zhiqiang Hu, Yuqian Yuan, Guanzheng Chen, Sicong Leng, Yuming Jiang, Hang Zhang, Xin Li, et al. Videollama 3: Frontier multimodal foundation models for image and video understanding.arXiv preprint arXiv:2501.13106, 2025

    2025 arXiv

  35. [43]

    Video-llama: An instruction-tuned audio-visual language model for video understanding.arXiv:2306.02858, 2023

    Hang Zhang, Xin Li, and Lidong Bing. Video-llama: An instruction-tuned audio-visual language model for video understanding.arXiv:2306.02858, 2023

    2023 arXiv

  36. [44]

    Transformer vision- language tracking via proxy token guided cross-modal fusion.Pattern Recognit

    Haojie Zhao, Xiao Wang, Dong Wang, Huchuan Lu, and Xiang Ruan. Transformer vision- language tracking via proxy token guided cross-modal fusion.Pattern Recognit. Lett., 168:10– 16, April 2023

    2023

  37. [45]

    Minigpt-4: En- hancing vision-language understanding with advanced large language models.arXiv preprint arXiv:2304.10592, 2023

    Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. Minigpt-4: En- hancing vision-language understanding with advanced large language models.arXiv preprint arXiv:2304.10592, 2023

    2023 arXiv

  38. [46]

    Tracking with human-intent reasoning.arXiv preprint arXiv:2312.17448, 2023

    Jiawen Zhu, Zhi-Qi Cheng, Jun-Yan He, Chenyang Li, Bin Luo, Huchuan Lu, Yifeng Geng, and Xuansong Xie. Tracking with human-intent reasoning.arXiv preprint arXiv:2312.17448, 2023

    2023 arXiv

  39. [47]

    Internvl3: Exploring advanced training and test-time recipes for open-source multimodal models.arXiv preprint arXiv:2504.10479, 2025

    Jinguo Zhu, Weiyun Wang, Zhe Chen, Zhaoyang Liu, Shenglong Ye, Lixin Gu, Yuchen Duan, Hao Tian, Weijie Su, Jie Shao, et al. Internvl3: Exploring advanced training and test-time recipes for open-source multimodal models.arXiv preprint arXiv:2504.10479, 2025. 13 A Appendix A.1 A...

    2025 arXiv

  40. [49]

    These videos are partially sourced from [21] and the remaining are requested internally

    Cell Tracking:Features internally sourced 12 microscopic videos with dynamic cellular structures, where precise localization of individual cells is essential for tasks like tracking cell division or counting. These videos are partially sourced from [21] and the remaining are r...

  41. [50]

    The egocentric videos in our test benchmark are derived from [8] dataset

    Egocentric Videos:Comprises 18 first-person videos capturing daily human-object interactions, enabling the assessment of grounded pointing in scenarios such as object manipulation and activity recognition. The egocentric videos in our test benchmark are derived from [8] dataset

  42. [51]

    Autonomous Driving:Includes 13 urban driving scenes from nuScenes’s dataset [ 4], with complex environments, requiring accurate identification of specific road elements (e.g., traffic signals) to support navigation and safety systems

  43. [52]

    The VideoGUI videos are sampled from VideoGUI dataset [16]

    Video-GUI:Consists of 13 screen recordings from software applications, focusing on tasks like identifying and interacting with user interface elements based on textual instructions. The VideoGUI videos are sampled from VideoGUI dataset [16]

  44. [53]

    pick up the red block

    Robotics:Encompasses 14 videos of robotic operations, emphasizing the need for precise object localization to execute commands such as "pick up the red block" or "press the top button." Few of the robotic videos in our benchmark are sourced from [34], and the remaining are sou...

  45. [2020]

    Springer International Publishing

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.