Pith. sign in

REVIEW 4 major objections 3 minor 47 references

Memory-Augmented Multimodal Large Language Models for Small Object Understanding in Streaming Aerial Videos

T0 review · 4 major / 3 minor · reviewed 2026-08-01 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read SkyAnchor, a 3-billion-parameter model that routes visual tokens by salience and uses a two-layer memory, claims state-of-the-art tiny-object segmentation in live drone video streams.

desk verdict Genuinely new dataset and a plausible method, but the headline numbers rest on SAM3-generated labels and an unclear SkyFind protocol—worth serious review, not yet worth quoting. read the letter →

arxiv 2607.19857 v1 pith:62Q6NPCO submitted 2026-07-22 cs.CV cs.AI

Penglei Sun , Yehua Huang , Zhuoli Tao , Xiang Li , Runwei Guan , Yaoxian Song , Kaiyong Zhao , Henghui Ding
show 3 more authors
Bo Han Yang Yang Xiaowen Chu
This is my paper · ORCID
classification cs.CVcs.AI
keywords aerialvideounderstandingreferringsegmentationsmallobjectmultimodallargelanguagemodelsstreamingmemorybanktokencompressiondroneperception
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

SkyAnchor is a proposal for making language-guided drone perception work on the fly: a drone flying over a scene should be able to follow a natural-language query ("find the white truck") and segment the tiny target in each arriving frame without peeking at future frames. The paper argues that existing multimodal large language models fail here for two reasons: they compress visual tokens uniformly, which blends tiny targets into background, and they have no affordable way to carry context across a long stream. It introduces DroneEyes, the first pixel-level open-vocabulary referring-segmentation dataset for tiny aerial targets (2,140 high-definition videos, 176,623 description/segmentation pairs), and SkyAnchor, a model that preserves small-target details through a salience-weighted token router and maintains long-term identity through a hierarchical memory bank that separates slow-changing semantic anchors from fast-changing tracking states. On DroneEyes, SkyAnchor reports a 2x improvement in object-description score over the strongest general-purpose baselines and a 15.3% gain in region-and-contour accuracy over the best segmentation MLLM despite using only 3 billion parameters. On an unseen maritime benchmark, it reports gains of 25.9% (IoU@0.5) and 95.5% (IoU@mean) over the previous domain-specific best without any fine-tuning, and an optimized edge deployment reaches about 5.8 frames per second.

What carries the argument

The central mechanism is the augmentation of a vision-language MLLM with two components. The Semantics-Aware Token Router partitions visual tokens into k×k blocks, learns a per-token importance score, and merges each block into one token via softmax-weighted aggregation, supervised by an auxiliary routing loss from the ground-truth mask; this preserves fine-grained target details under a fixed token budget. The Hierarchical Memory Bank separates long-term identity from short-term position: a Semantic Memory layer stores object-level appearance tokens from anchor frames, while a Tracking Memory layer keeps the most recent spatial states in a bounded FIFO window. The VLM backbone runs only eve

What would settle it

Take a random sample of, say, 200 DroneEyes frames stratified by object size; have human annotators draw ground-truth masks from scratch; re-run SkyAnchor and the strongest baselines on those frames and compare J&F against both the human masks and the dataset masks. If the performance gap between SkyAnchor and the best baseline shrinks or disappears when measured against human masks, the claimed advantage would be an artifact of the machine-label pipeline. A second check: feed the router test frames with no mask supervision and inspect whether its salience map still concentrates on the referre

Watch

Extended reading notes

Core claim

SkyAnchor's central claim: small-object understanding in streaming aerial video improves when the visual representation itself focuses on the target, not when the language model is made larger. A Semantics-Aware Token Router scores each visual token's salience and merges blocks by softmax-weighted aggregation, keeping tiny targets from being averaged into background. A Hierarchical Memory Bank stores persistent appearance anchors ("what it is") plus a fixed-size sliding window of tracking states ("where it is"). The backbone runs only on anchor frames; the mask decoder runs every frame, giving online causal inference with bounded memory. The 3B model reports state-of-the-art results on the n

Load-bearing premise

The load-bearing premise is that the DroneEyes ground-truth segmentation masks — produced by a foundation tracker seeded with detector proposals and only sampled manually — are accurate enough to serve as pixel-level labels for tiny aerial objects; if those masks carry the tracker's systematic errors, training and evaluation measure agreement with the tracker's output distribution rather than true boundaries.

Editorial extensions

If this is right

  • Flying drones can follow open-vocabulary natural-language queries and segment tiny targets frame-by-frame without access to future frames or unbounded memory, making onboard, cloud-free operator assistance feasible.
  • Model scale is not the decisive factor for tiny-object aerial segmentation: a 3B model with targeted token preservation and memory beats much larger offline models, redirecting effort toward representation and memory design.
  • The reported zero-shot transfer to an unseen maritime domain suggests that the router and memory induce a general notion of tracking the referred target that survives domain shifts, potentially reducing the need for per-scene fine-tuning.
  • The bounded-memory design (a 16-frame sliding tracking window plus persistent semantic anchors) offers a practical template for long-running streaming perception where context cannot grow with time.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper evaluates the Token Router with ground-truth masks as supervision for salience scores; at deployment no such mask exists for a new query. A natural extension is to test how router salience degrades for unseen categories or when the query is compositional, and to add self-supervised or weakly supervised salience objectives.
  • Because DroneEyes masks are machine-generated by a foundation tracker seeded with detector proposals, a portion of the claimed quality advantage may reflect agreement with that tracker's output rather than with human-annotated boundaries. Re-annotating a random sample by hand and recomputing the comparison would show how much of the 15.3% J&F gain is real.
  • The hierarchical memory idea — separating stable identity from transient position — is not specific to video segmentation; it could transfer to streaming tasks such as referring multi-object tracking or compositional query understanding, where the semantic layer stores per-target attribute tokens and the tracking layer keeps spatial proposals.
  • Given the reported insensitivity to anchor interval, further speedups might come from sparser anchoring or quantized memory, though the paper does not explore these; the deployment numbers already show the backbone is only a small fraction of per-frame latency once optimized.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 3 minor

Summary. The paper introduces DroneEyes, claimed to be the first pixel-level, open-vocabulary referring-segmentation dataset for tiny aerial targets in streaming drone videos, and SkyAnchor, a 3B-parameter MLLM designed for online referring video object segmentation. SkyAnchor combines a Semantics-Aware Token Router for token compression under a fixed budget and a Hierarchical Memory Bank with semantic and tracking layers. Experiments on DroneEyes report large gains over general and segmentation MLLMs on object description and referring expression tasks. On the unseen SkyFind benchmark, the authors report large cross-domain improvements over domain-specific SOTA without fine-tuning. The paper also describes an edge deployment on Jetson AGX Orin with a 3.05× speedup over vanilla PyTorch. The central claims are the dataset contribution and the method's ability to outperform larger offline specialists on tiny-object referring segmentation while transferring to unseen maritime scenes.

Significance. If the DroneEyes annotations were verified at the pixel level and the SkyFind evaluation protocol were clearly defined, this would be a substantial contribution: it fills a real gap in aerial streaming referring-segmentation benchmarks, and the proposed architecture is well motivated by the tiny-object and online-streaming challenges. The ablations of the token router and memory bank are informative, and the edge deployment results give practical context. However, the significance is conditional on resolving the benchmark-validity issues below: the primary in-domain numbers are measured against machine-generated masks, and the cross-domain numbers lack a stated metric protocol.

major comments (4)
  1. [Section III-B1, Eq. (7), Table II] The DroneEyes pixel-level ground truth is produced by SAM3-based tracking from YOLO proposals with only 'final manual sampling inspection' and confidence filtering. No per-pixel human verification, inter-annotator agreement, or correction rate is reported. Because Eq. (7) trains the router with L_route = BCE(s, M_gt) and Table II evaluates J&F against the same SAM3-derived masks, a high J&F can largely reflect fidelity to SAM3's output distribution rather than accurate segmentation of true tiny-object boundaries. This directly affects the dataset contribution and the main in-domain claim. Please provide quantitative label-quality evidence: e.g., a human-corrected subset with per-pixel agreement, correction statistics, and/or an evaluation on a human-verified test split.
  2. [Section V-E, Table VI] The SkyFind evaluation protocol is not stated. SkyAnchor outputs segmentation masks, while the comparison methods (FAOA, RSC, RefTR, etc.) are bounding-box visual grounding models trained on SkyFind. The paper reports IoU@0.5 and IoU@mean but never explains how masks are converted to boxes (or boxes to masks) before IoU is computed. Without this protocol, the claimed +25.9% IoU@0.5 and +95.5% IoU@mean on the unseen test split are not interpretable. Please specify the exact conversion and matching rule, and if possible report mask-to-mask IoU against any available ground-truth masks.
  3. [Section III-B2/3, Section V-B1] The L1 object-description ground truth is generated by a VLM/LLM with no reported human verification beyond the object-extraction stage. SkyAnchor is trained on this machine-generated text, and the L1 evaluation uses a GPT-5.2 judge with video frames as context. This creates a plausible distribution-fit effect: high GPT scores may partly reward reproducing the same VLM/LLM's annotation style rather than objectively correct descriptions. Please report a human evaluation on a sampled subset, or at least agreement between the GPT judge and human raters, for the L1 claim of 2× improvement.
  4. [Section V-C, Figure 6] The text states that in the two smallest-target intervals (ultra-tiny and very-tiny) SkyAnchor leads the strongest baseline by 9.5 and 10.8 percentage points, but Figure 6's caption and bar labels report improvements of +8.9% and +4.8%. These numbers are inconsistent, and they are load-bearing for the claim that SkyAnchor's advantage grows with decreasing target size. Please reconcile the text with the figure and clarify whether the reported values are percentage points or relative percentages.
minor comments (3)
  1. [Fig. 10(d)] The query text reads 'sliver car' but should be 'silver car'.
  2. [Table VII] The caption says 'Ablation study on progressive optimization', but the table reports the full deployment pipeline, not an ablation. Suggest renaming to 'Progressive optimization results'.
  3. [Algorithm 1, line 6] The notation 'H←F_vlm(...)' is fine, but in Eq. (2) the same object H is defined as an assignment from the backbone; align the notation for clarity.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant formal circularity; DroneEyes label provenance is a benchmark-validity concern, not a derivation-equivalence issue.

full rationale

The paper's derivation chain is self-contained: SkyAnchor maps online video frames and a language query to a mask via a VLM backbone, a semantics-aware token router, a hierarchical memory bank, and a SAM2 decoder. The routing loss (Eq. 7) is supervised by the provided ground-truth masks, but that is standard supervised training rather than a hidden equivalence; the model is not defined in terms of the evaluation labels, and the DroneEyes splits are disjoint train/test portions of the same dataset. The strongest cross-domain claim is evaluated on the external SkyFind benchmark with no training on it, which breaks any self-referential loop. The paper does cite the authors' own prior works [4], [13], but only as contextual references; no load-bearing claim reduces to a self-citation, and no uniqueness theorem or imported ansatz is invoked to force the architecture. The main caveat is that DroneEyes masks are produced by a SAM3 tracker from YOLO proposals with only sampled human inspection (Section III-B1), and SkyAnchor is trained on those same machine-generated labels; this could make DroneEyes scores partly reflect agreement with the annotation engine rather than true object boundaries. That is a legitimate data-quality and external-validity concern, but it is not a circular derivation: the predictions are not equal to the inputs by construction, and the external SkyFind results provide independent grounding. Accordingly, no specific circular step meeting the stated evidentiary standard can be exhibited.

Assumptions & free parameters 5 free parameters · 5 assumptions · 0 invented entities

The central empirical claims rest primarily on the quality of automatically generated DroneEyes labels and on the comparability of the SkyFind evaluation; these are the main unpaid premises. No new physical entities are introduced; SkyAnchor's components are architectural modules.

free parameters (5)
  • Anchor interval N = 16
    Chosen by ablation on DroneEyes (Fig. 9); controls how often the VLM runs.
  • Memory FIFO capacity W = 16 frames
    Chosen by ablation (Fig. 7); J&F peaks at 16 frames and degrades with more memory.
  • Token-router block size k = 2
    Chosen by ablation (Fig. 8); 2x2 merging is the default speed/quality trade-off.
  • Routing loss weights lambda1, lambda2 = 1.0
    Set by hand in the implementation details; not ablated.
  • LoRA rank and alpha = r=128, alpha=256
    Hyperparameters chosen for backbone fine-tuning; no sensitivity study is given.
assumptions (5)
  • domain assumption SAM3 propagation outputs are valid pixel-level ground truth labels for tiny aerial objects.
    Section III-B1 uses SAM3 as the foundation tracker and retains only high-confidence outputs, with a final manual sampling inspection. No per-mask human validation is reported.
  • domain assumption VLM-generated semantic descriptions and LLM-refined expressions are valid ground-truth text for evaluating object description.
    Section III-B2 and III-B3 generate attributes with a VLM and refine with an LLM; the L1 evaluation then compares models against these machine-generated references.
  • domain assumption Comparing SkyAnchor in online streaming mode with baselines in their native offline mode is a fair comparison.
    Section V-A4 states that segment MLLMs are evaluated offline while SkyAnchor runs streaming; the paper does not control for frame budget or protocol differences.
  • domain assumption SkyAnchor's mask outputs can be scored against SkyFind's bounding-box ground truth without a described conversion or matching procedure.
    Section V-E and Table VI report IoU on SkyFind, a bbox-level benchmark, but the paper does not state how masks are converted to boxes or matched to the bbox ground truth.
  • domain assumption Skipping the VLM between anchor frames preserves the causal streaming requirement and produces the same semantic understanding.
    Section IV-D re-uses cached tokens for non-anchor frames; the paper assumes this does not change the answer semantics in the online setting.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Memory-Augmented Multimodal Large Language Models for Small Object Understanding in Streaming Aerial Videos." pith.science (2026). https://pith.science/paper/62Q6NPCO

@misc{pith2026260719857,
  author       = {Pith},
  title        = {Pith review of: Memory-Augmented Multimodal Large Language Models for Small Object Understanding in Streaming Aerial Videos},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/62Q6NPCO}},
  note         = {Machine review of arXiv:2607.19857}
}
abstract

Language-guided aerial perception aims to understand user-specified tiny targets in complex unmanned aerial vehicle (UAV) scenes. In real UAV deployment, the UAV must respond while it flies, so such perception runs in an online streaming manner, where frames arrive sequentially and the model responds to each one without access to future frames. However, applying current Multimodal Large Language Models (MLLMs) to this setting raises two challenges. First, targets viewed from the air are often tiny, yet the visual compression in existing MLLMs treats all regions equally and discards their fine-grained details. Second, understanding a continuous stream requires past-frame context, yet retaining the entire history is infeasible on resource-constrained onboard hardware, whereas discarding it causes the target to drift or disappear. We address the tiny object and streaming challenges from both data and method perspectives. From the data perspective, we present \textbf{DroneEyes}, the \textbf{first} pixel-level and open-vocabulary referring-segmentation dataset for tiny aerial targets, comprising $2,140$ high-definition videos and $176,623$ pairs across Object Description and Referring Expression tasks, with dense per-frame masks. From the method perspective, we propose \textbf{SkyAnchor}, an MLLM with two designs to the above challenges: a Semantics-Aware Token Router that preserves small-target under a reduced visual-token budget, and a Hierarchical Memory Bank that keeps the target consistently understood on streams.

Figures

Figures reproduced from arXiv: 2607.19857 by the authors.

Figure 1
Figure 1. Comparison of DroneEyes with existing daily-life and aerial offline [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. The data label pipeline for DroneEyes. (1) Object Extraction. The candidate objects are detected with human verification, followed by mask propagation. (2) Semantic Enrichment. We use VLM to enrich attribute descriptions via context-aware visual prompting. (3) QA Generation. Two￾level QA pairs are generated with LLM-based linguistic refinement. ground-level datasets, aerial visual understanding depends on fine-grain… view at source ↗
Figure 3
Figure 3. The statistic of DroneEyes. (a) Distribution of video frame counts across the dataset. (b) Distribution of object visibility ratios across annotation [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (8 more)
Figure 5
Figure 5. Figure 5: Edge inference schedule of SkyAnchor. The top row is the backbone, which runs only on anchor frames, re-anchoring every N −th frame. Dashed boxes mark frames where it is skipped and the cached token is reused. The bottom row is the SAM 2 decoder (orange), which runs on…
Figure 4
Figure 4. Figure 4: Overview of SkyAnchor. (A) Model Overview: Q, optional visual prompt, and {I1, . . . , It} are encoded into Zq, Zr, Zv; together with Zm, the VLM produces H, from which the Language Head outputs A and the Seg Head outputs eseg to drive SAM2 for Mt. (B) Hierarchical Mem…
Figure 6
Figure 6. Figure 6: Per-size J &F comparison. All three models are of comparable size (3B–4B) and fine-tuned on the same DroneEyes dataset. SkyAnchor outperforms three finetuned models across all size buckets, with a more margin on tiny objects. Numbers above bars denote the improvement o…
Figure 7
Figure 7. Figure 7: Ablation study on memory capacity. (a) Segmentation quality ( [PITH_FULL_IMAGE:figures/full_fig_p010_7.png]
Figure 8
Figure 8. Figure 8: Ablation study on Token Router. (a) Segmentation quality ( [PITH_FULL_IMAGE:figures/full_fig_p010_8.png]
Figure 9
Figure 9. Figure 9: Ablation study of the anchor interval N on SkyAnchor. with the highest score obtained at N = 16. We use N = 16 as the default setting because it gives the highest measured score while avoiding frequent VLM inference. E. Cross-Domain Generalization to Unseen Scene We be…
Figure 10
Figure 10. Figure 10: Qualitative results of SkyAnchor on four challenging video sequences. The input queries are shown in the green boxes. For each case, we display [PITH_FULL_IMAGE:figures/full_fig_p011_10.png]
Figure 11
Figure 11. Figure 11: Qualitative results of SkyAnchor for the real world deployment. The six queries are absent in the DroneEyes dataset. For privacy protection purposes, [PITH_FULL_IMAGE:figures/full_fig_p012_11.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

47 extracted references · 8 linked inside Pith

  1. [1]

    The small-drone revolution is coming—scientists need to ensure it will be safe,

    X. Huang, “The small-drone revolution is coming—scientists need to ensure it will be safe,”Nature, vol. 637, no. 8044, pp. 29–30, 2025

    2025

  2. [2]

    Champion-level drone racing using deep reinforcement learning,

    E. Kaufmann, L. Bauersfeld, A. Loquercio, M. M ¨uller, V . Koltun, and D. Scaramuzza, “Champion-level drone racing using deep reinforcement learning,”Nature, vol. 620, no. 7976, pp. 982–987, 2023

    2023

  3. [3]

    Video object segmentation without tem- poral information,

    K.-K. Maninis, S. Caelles, Y . Chen, J. Pont-Tuset, L. Leal-Taix ´e, D. Cremers, and L. Van Gool, “Video object segmentation without tem- poral information,”IEEE transactions on pattern analysis and machine intelligence, vol. 41, no. 6, pp. 1515–1530, 2018

    2018

  4. [4]

    3d question answering for city scene understanding,

    P. Sun, Y . Song, X. Liu, X. Yang, Q. Wang, T. Li, Y . Yang, and X. Chu, “3d question answering for city scene understanding,” inProceedings of the 32nd ACM International Conference on Multimedia, 2024, pp. 2156–2165

    2024

  5. [5]

    Phenobench: A large dataset and benchmarks for semantic image interpretation in the agricultural do- main,

    J. Weyler, F. Magistri, E. Marks, Y . L. Chong, M. Sodano, G. Roggiolani, N. Chebrolu, C. Stachniss, and J. Behley, “Phenobench: A large dataset and benchmarks for semantic image interpretation in the agricultural do- main,”IEEE transactions on pattern analysis and machine intelligence, vol. 46, no. 12, pp. 9583–9594, 2024

    2024

  6. [6]

    Detecting flying objects using a single moving camera,

    A. Rozantsev, V . Lepetit, and P. Fua, “Detecting flying objects using a single moving camera,”IEEE transactions on pattern analysis and machine intelligence, vol. 39, no. 5, pp. 879–892, 2016

    2016

  7. [7]

    Revisiting image-language networks for open-ended phrase detection,

    B. A. Plummer, K. J. Shih, Y . Li, K. Xu, S. Lazebnik, S. Sclaroff, and K. Saenko, “Revisiting image-language networks for open-ended phrase detection,”IEEE transactions on pattern analysis and machine intelligence, vol. 44, no. 4, pp. 2155–2167, 2020

    2020

  8. [8]

    Mevis: A large- scale benchmark for video segmentation with motion expressions,

    H. Ding, C. Liu, S. He, X. Jiang, and C. C. Loy, “Mevis: A large- scale benchmark for video segmentation with motion expressions,” in Proceedings of the IEEE/CVF international conference on computer vision, 2023, pp. 2694–2703

    2023

Show all 47 references
  1. [9]

    Lamot: Language- guided multi-object tracking,

    Y . Li, X. Liu, L. Liu, H. Fan, and L. Zhang, “Lamot: Language- guided multi-object tracking,” in2025 IEEE International Conference on Robotics and Automation (ICRA). IEEE, 2025, pp. 6816–6822

    2025

  2. [10]

    Skyfind: A large-scale benchmark unveiling referring expres- sion comprehension for uav

    K. Wang, G. Wu, X. Fu, X. Wang, K. Liu, X. Lu, C. Ge, W. Zhai, and Z. Zha, “Skyfind: A large-scale benchmark unveiling referring expres- sion comprehension for uav.”IEEE Transactions on Pattern Analysis and Machine Intelligence, 2026

    2026

  3. [11]

    Aerialmind: Towards referring multi-object tracking in uav scenarios,

    C. Chen, S. Liang, R. Guan, X. Sun, H. Zhao, H. Jiang, T. Huang, H. Ding, and Q.-L. Han, “Aerialmind: Towards referring multi-object tracking in uav scenarios,” inProceedings of the AAAI Conference on Artificial Intelligence, vol. 40, no. 4, 2026, pp. 2805–2813

    2026

  4. [12]

    Event-aware instructed assistant for referring video segmentation,

    J. Liu, H. Ding, S. He, and Y .-G. Jiang, “Event-aware instructed assistant for referring video segmentation,”IEEE Transactions on Image Processing, 2026

    2026

  5. [13]

    City-vlm: Towards multidomain perception scene understanding via multimodal incomplete learning,

    P. Sun, Y . Song, X. Zhu, X. Liu, Q. Wang, Y . Liu, C. Xia, T. Li, Y . Yang, and X. Chu, “City-vlm: Towards multidomain perception scene understanding via multimodal incomplete learning,” inProceedings of the 33rd ACM International Conference on Multimedia, 2025, pp. 3448– 3457

    2025

  6. [14]

    Qwen2.5 technical report,

    Qwen, :, A. Yang, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Li, D. Liu, F. Huang, H. Wei, H. Lin, J. Yang, J. Tu, J. Zhang, J. Yang, J. Yang, J. Zhou, J. Lin, K. Dang, K. Lu, K. Bao, K. Yang, L. Yu, M. Li, M. Xue, P. Zhang, Q. Zhu, R. Men, R. Lin, T. Li, T. Tang, T. Xia, ...

  7. [15]

    Sam 3: Segment anything with concepts,

    N. Carion, L. Gustafson, Y .-T. Hu, S. Debnath, R. Hu, D. Suris, C. Ryali, K. V . Alwala, H. Khedr, A. Huanget al., “Sam 3: Segment anything with concepts,”arXiv preprint arXiv:2511.16719, 2025

    2025 arXiv

  8. [16]

    React: Streaming video analytics on the edge with asynchronous cloud support,

    A. Ghosh, S. Iyengar, S. Lee, A. Rathore, and V . N. Padmanabhan, “React: Streaming video analytics on the edge with asynchronous cloud support,” inProceedings of the 8th ACM/IEEE Conference on Internet of Things Design and Implementation, 2023, pp. 222–235

    2023

  9. [17]

    Multiple-object-tracking algorithm based on dense trajectory voting in aerial videos,

    T. Yang, D. Li, Y . Bai, F. Zhang, S. Li, M. Wang, Z. Zhang, and J. Li, “Multiple-object-tracking algorithm based on dense trajectory voting in aerial videos,”Remote Sensing, vol. 11, no. 19, p. 2278, 2019

    2019

  10. [18]

    Urvos: Unified referring video object segmentation network with a large-scale benchmark,

    S. Seo, J.-Y . Lee, and B. Han, “Urvos: Unified referring video object segmentation network with a large-scale benchmark,” inEuropean conference on computer vision. Springer, 2020, pp. 208–223

    2020

  11. [19]

    Sharegpt4video: Improving video understanding and generation with better captions,

    L. Chen, X. Wei, J. Li, X. Dong, P. Zhang, Y . Zang, Z. Chen, H. Duan, B. Lin, Z. Tanget al., “Sharegpt4video: Improving video understanding and generation with better captions,”Advances in Neural Information Processing Systems, vol. 37, pp. 19 472–19 495, 2024

    2024

  12. [20]

    Mevis: A multi-modal dataset for referring motion expression video segmentation,

    H. Ding, C. Liu, S. He, K. Ying, X. Jiang, C. C. Loy, and Y .-G. Jiang, “Mevis: A multi-modal dataset for referring motion expression video segmentation,”IEEE Transactions on Pattern Analysis and Machine Intelligence, 2025

    2025

  13. [21]

    Jtd-uav: Mllm-enhanced joint tracking and description framework for anti-uav systems,

    Y . Wang, J. Zhao, Z. Fan, X. Zhang, X. Wu, Y . Zhang, L. Jin, X. Li, G. Wang, M. Jiaet al., “Jtd-uav: Mllm-enhanced joint tracking and description framework for anti-uav systems,” inProceedings of the Computer Vision and Pattern Recognition Conference, 2025, pp. 1633– 1644

    2025

  14. [22]

    Visa: Reasoning video object segmentation via large lan- guage models,

    C. Yan, H. Wang, S. Yan, X. Jiang, Y . Hu, G. Kang, W. Xie, and E. Gavves, “Visa: Reasoning video object segmentation via large lan- guage models,” inEuropean Conference on Computer Vision. Springer, 2024, pp. 98–115

    2024

  15. [23]

    Sa2va: Marrying sam2 with llava for dense grounded understanding of images and videos,

    H. Yuan, X. Li, T. Zhang, Y . Sun, Z. Huang, S. Xu, S. Ji, Y . Tong, L. Qi, J. Fenget al., “Sa2va: Marrying sam2 with llava for dense grounded understanding of images and videos,”arXiv preprint arXiv:2501.04001, 2025

    2025 arXiv

  16. [24]

    Unipixel: Unified object referring and segmentation for pixel-level visual reason- ing,

    Y . Liu, Z. Ma, J. Pu, Z. Qi, Y . Wu, Y . Shan, and C. Chen, “Unipixel: Unified object referring and segmentation for pixel-level visual reason- ing,”Advances in Neural Information Processing Systems, vol. 38, pp. 126 078–126 108, 2026

    2026

  17. [25]

    Sam 2: Segment anything in images and videos,

    N. Ravi, V . Gabeur, Y .-T. Hu, R. Hu, C. Ryali, T. Ma, H. Khedr, R. R¨adle, C. Rolland, L. Gustafsonet al., “Sam 2: Segment anything in images and videos,”arXiv preprint arXiv:2408.00714, 2024

    2024 arXiv

  18. [26]

    Videoglamm: A large multimodal model for pixel-level visual JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2021 14 grounding in videos,

    S. Munasinghe, H. Gani, W. Zhu, J. Cao, E. Xing, F. S. Khan, and S. Khan, “Videoglamm: A large multimodal model for pixel-level visual JOURNAL OF LATEX CLASS FILES, VOL. 14, NO. 8, AUGUST 2021 14 grounding in videos,” inProceedings of the Computer Vision and Pattern Recognitio...

    2021

  19. [27]

    Glus: Global-local reasoning unified into a single large language model for video segmentation,

    L. Lin, X. Yu, Z. Pang, and Y .-X. Wang, “Glus: Global-local reasoning unified into a single large language model for video segmentation,” in Proceedings of the Computer Vision and Pattern Recognition Confer- ence, 2025, pp. 8658–8667

    2025

  20. [28]

    The devil is in temporal token: High quality video reasoning segmentation,

    S. Gong, Y . Zhuge, L. Zhang, Z. Yang, P. Zhang, and H. Lu, “The devil is in temporal token: High quality video reasoning segmentation,” inProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2025, pp. 29 183–29 192

    2025

  21. [29]

    Instructseg: Unifying instructed visual segmentation with multi-modal large language models,

    C. Wei, Y . Zhong, H. Tan, Y . Zeng, Y . Liu, H. Wang, and Y . Yang, “Instructseg: Unifying instructed visual segmentation with multi-modal large language models,” inProceedings of the IEEE/CVF International Conference on Computer Vision, 2025, pp. 20 193–20 203

    2025

  22. [30]

    Geochat: Grounded large vision-language model for remote sensing,

    K. Kuckreja, M. S. Danish, M. Naseer, A. Das, S. Khan, and F. S. Khan, “Geochat: Grounded large vision-language model for remote sensing,” inProceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2024, pp. 27 831–27 840

    2024

  23. [31]

    Skysensegpt: A fine-grained instruction tuning dataset and model for remote sensing vision-language understanding,

    J. Luo, Z. Pang, Y . Zhang, T. Wang, L. Wang, B. Dang, J. Lao, J. Wang, J. Chen, Y . Tanet al., “Skysensegpt: A fine-grained instruction tuning dataset and model for remote sensing vision-language understanding,” arXiv preprint arXiv:2406.10100, 2024

    2024 arXiv

  24. [32]

    Earthgpt: A universal multimodal large language model for multisensor image comprehension in remote sensing domain,

    W. Zhang, M. Cai, T. Zhang, Y . Zhuang, and X. Mao, “Earthgpt: A universal multimodal large language model for multisensor image comprehension in remote sensing domain,”IEEE Transactions on Geo- science and Remote Sensing, vol. 62, pp. 1–20, 2024

    2024

  25. [33]

    Kimi k2. 5: Visual agentic intelligence,

    K. Team, T. Bai, Y . Bai, Y . Bao, S. Cai, Y . Cao, Y . Charles, H. Che, C. Chen, G. Chenet al., “Kimi k2. 5: Visual agentic intelligence,”arXiv preprint arXiv:2602.02276, 2026

    2026 arXiv

  26. [34]

    Qwen3.6-Plus: Towards real world agents,

    Qwen Team, “Qwen3.6-Plus: Towards real world agents,” April 2026. [Online]. Available: https://qwen.ai/blog?id=qwen3.6

    2026

  27. [35]

    Visual instruction tuning,

    H. Liu, C. Li, Q. Wu, and Y . J. Lee, “Visual instruction tuning,” Advances in neural information processing systems, vol. 36, pp. 34 892– 34 916, 2023

    2023

  28. [36]

    Qwen2.5-vl technical report,

    S. Bai, K. Chen, X. Liu, J. Wang, W. Ge, S. Song, K. Dang, P. Wang, S. Wang, J. Tang, H. Zhong, Y . Zhu, M. Yang, Z. Li, J. Wan, P. Wang, W. Ding, Z. Fu, Y . Xu, J. Ye, X. Zhang, T. Xie, Z. Cheng, H. Zhang, Z. Yang, H. Xu, and J. Lin, “Qwen2.5-vl technical report,” 2025. [Onli...

    2025 arXiv

  29. [37]

    Streamingvlm: Real-time understanding for infinite video streams,

    R. Xu, G. Xiao, Y . Chen, L. He, K. Peng, Y . Lu, and S. Han, “Streamingvlm: Real-time understanding for infinite video streams,” arXiv preprint arXiv:2510.09608, 2025

    2025 arXiv

  30. [38]

    A fast and accurate one-stage approach to visual grounding,

    Z. Yang, B. Gong, L. Wang, W. Huang, D. Yu, and J. Luo, “A fast and accurate one-stage approach to visual grounding,” inProceedings of the IEEE/CVF international conference on computer vision, 2019, pp. 4683–4693

    2019

  31. [39]

    Improving one-stage visual grounding by recursive sub-query construction,

    Z. Yang, T. Chen, L. Wang, and J. Luo, “Improving one-stage visual grounding by recursive sub-query construction,” inEuropean conference on computer vision. Springer, 2020, pp. 387–404

    2020

  32. [40]

    Referring transformer: A one-step approach to multi-task visual grounding,

    M. Li and L. Sigal, “Referring transformer: A one-step approach to multi-task visual grounding,”Advances in neural information processing systems, vol. 34, pp. 19 652–19 664, 2021

    2021

  33. [41]

    Transvg: End-to-end visual grounding with transformers,

    J. Deng, Z. Yang, T. Chen, W. Zhou, and H. Li, “Transvg: End-to-end visual grounding with transformers,” inProceedings of the IEEE/CVF international conference on computer vision, 2021, pp. 1769–1779

    2021

  34. [42]

    Improving visual grounding with visual-linguistic verification and iterative reasoning,

    L. Yang, Y . Xu, C. Yuan, W. Liu, B. Li, and W. Hu, “Improving visual grounding with visual-linguistic verification and iterative reasoning,” in proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2022, pp. 9499–9508

    2022

  35. [43]

    Seqtr: A simple yet universal network for visual grounding,

    C. Zhu, Y . Zhou, Y . Shen, G. Luo, X. Pan, M. Lin, C. Chen, L. Cao, X. Sun, and R. Ji, “Seqtr: A simple yet universal network for visual grounding,” inEuropean Conference on Computer Vision. Springer, 2022, pp. 598–615

    2022

  36. [44]

    Shifting more attention to visual backbone: Query-modulated refinement networks for end-to-end visual grounding,

    J. Ye, J. Tian, M. Yan, X. Yang, X. Wang, J. Zhang, L. He, and X. Lin, “Shifting more attention to visual backbone: Query-modulated refinement networks for end-to-end visual grounding,” inproceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2022, ...

    2022

  37. [45]

    A survivor in the era of large- scale pretraining: An empirical study of one-stage referring expression comprehension,

    G. Luo, Y . Zhou, J. Sun, X. Sun, and R. Ji, “A survivor in the era of large- scale pretraining: An empirical study of one-stage referring expression comprehension,”IEEE Transactions on Multimedia, vol. 26, pp. 3689– 3700, 2023

    2023

  38. [46]

    Polyformer: Referring image segmentation as sequential polygon generation,

    J. Liu, H. Ding, Z. Cai, Y . Zhang, R. K. Satzoda, V . Mahadevan, and R. Manmatha, “Polyformer: Referring image segmentation as sequential polygon generation,” inProceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2023, pp. 18 653–18 663

    2023

  39. [2025]

    Available: https://arxiv.org/abs/2412.15115

    [Online]. Available: https://arxiv.org/abs/2412.15115

    arXiv

Pith tools

Reviewed August 1, 2026 · model on record in the stance chip above.