Pith. sign in

REVIEW 3 major objections 5 minor 57 references

FRAME: Pre-Training Video Feature Representations via Anticipation and Memory

T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read FRAME is a self-supervised video encoder that distills spatial features from DINO and semantic features from CLIP into a compact vision transformer, then adds memory and anticipation; the paper claims that to its knowledge it is the first…

desk verdict A solid and well-ablated video SSL recipe whose main soft spot is disclosed-but-real tuning on downstream subsets and an under-tested priority claim; worth peer review with revisions. read the letter →

arxiv 2506.05543 v1 pith:EIPAM3NB submitted 2025-06-05 cs.CV

Sethuraman TV , Savya Khosla , Vignesh Srinivasakumar , Jiahui Huang , Seoung Wug Oh , Simon Jenni , Derek Hoiem , Joon-Young Lee This is my paper
classification cs.CV
keywords self-supervisedvideorepresentationlearningfeaturedistillationdensepredictiontemporalmemoryfutureanticipationobjectsegmentationsemanticpartpropagationactionclassification
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

FRAME (Feature Representation and Anticipation with MEmory) is a self-supervised video encoder whose training goal is to make a small vision transformer produce per-frame features that are spatially precise like DINO's and semantically organized like CLIP's, while also carrying temporal context from the past and a short look ahead. The paper claims that distilling these image-model features and then teaching a memory-and-anticipation module to predict current and future DINO and CLIP features yields the first video encoder that beats both the original image encoders and prior self-supervised video encoders on frame-level dense prediction tasks such as object segmentation, part and pose propagation, and semantic segmentation. If true, this matters because it offers a cheaper path to strong video backbones: instead of pretraining on massive video datasets from scratch, one transfers the visual knowledge of large image models and adds lightweight temporal modules. The paper reports consistent gains across six dense prediction tasks on seven datasets, with a compact model that also matches CLIP on zero-shot and linear video action classification.

What carries the argument

The load-bearing machinery is a two-stage student-teacher pipeline. Stage 1 trains a ViT encoder so that a linear head reproduces CLIP's [CLS] token and a one-block Transformer head reproduces DINO's patch tokens, transferring dense spatial and semantic knowledge into a compact encoder. Stage 2 freezes that encoder, stores projections of the past five frames' patch tokens (reduced to dimension 64) in a FIFO memory bank, and runs cross-attention in which current-frame features serve as queries and memory-plus-current features as keys and values, followed by a self-attention block. Four lightweight decoders supervise this temporal module by predicting current CLIP features, current DINO patches, CLIP features four frames ahead, and DINO patches two frames ahead; the cosine and MSE losses are combined with weights (0.2, 0.1, 2.0, 0.4) that the authors chose empirically on a DAVIS subset. At inference the decoders are removed and only the memory-augmented encoder's patch and [CLS] outputs are used.

What would settle it

Train FRAME with every Stage 2 hyperparameter fixed before any downstream benchmark is inspected, then run the same frozen-feature evaluations on DAVIS, VIP, JHMDB, CamVid, and VSPW; if FRAME no longer beats both DINO and SiamMAE on those tasks, the central claim fails.

Watch

Extended reading notes

Core claim

The paper's central claim is that a compact vision transformer, trained in two stages, becomes a general-purpose video frame encoder for dense prediction. In Stage 1, the student encoder is trained to output CLIP's [CLS] semantic features and DINO's patch-level spatial features for single frames. In Stage 2, the frozen encoder is augmented with a FIFO memory bank of the past five frames plus a cross-attention module, and four lightweight decoders are trained to predict current CLIP and DINO features, CLIP semantics four frames ahead, and DINO patch features two frames ahead. The authors claim that this is the first student video encoder distilled from image teachers that outperforms both the image-only teachers (DINO, DINOv2) and prior self-supervised video encoders (VideoMAE, SiamMAE, CropMAE) on dense frame-level tasks. They show gains on DAVIS, VIP, JHMDB, CamVid, and VSPW while keeping the model compact, and they show that the [CLS] token, projected through the semantic decoder, performs comparably to CLIP on zero-shot and linear video action classification.

Load-bearing premise

The reported margins assume that the Stage 2 loss weights, anticipation deltas, and other settings chosen on subsets of the evaluation benchmarks did not give FRAME an unfair advantage over the baselines.

Editorial extensions

If this is right

  • Because the reported gains come from frozen features under a simple k-NN propagation protocol, the improvements reside in the video representation itself rather than in task-specific decoders.
  • Removing either memory or anticipation lowers performance on DAVIS, VIP, JHMDB, and CamVid, so the temporal modules are what convert image-level spatial fidelity into video-level consistency.
  • The compact ViT-S/8 FRAME (38M parameters) beats larger DINO ViT-B/8 (85M) on the correspondence benchmarks, implying a favorable accuracy-to-compute trade-off for the distillation-plus-temporal route.
  • The [CLS] token, projected through the semantic decoder, reaches CLIP-level zero-shot and linear action classification, so a single backbone can cover dense prediction and language-driven video tasks.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural next step, which the paper lists as future work, is to use FRAME as a drop-in backbone in video-language models; the CLIP-aligned [CLS] token gives such models a ready-made text-vision interface.
  • The paper evaluates only frozen features and explicitly notes that fine-tuning is untested; whether fine-tuning preserves or erodes the temporal-consistency gains remains an open question.
  • The memory bank holds only five past frames and the anticipation targets are two to four frames ahead, so the demonstrated advantage is short-term temporal coherence; longer-range motion understanding would likely need a hierarchical or recurrent memory design.
  • Because the approach inherits its spatial and semantic knowledge from DINO and CLIP, its ceiling is partly set by those teachers; a teacher with degraded spatial correspondence would presumably shrink the gains.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper introduces FRAME, a two-stage self-supervised video frame encoder. Stage 1 distills dense patch features from DINO and class-token features from CLIP into a compact ViT encoder with two lightweight decoders. Stage 2 freezes this encoder and adds a FIFO memory bank plus cross-attention, trained to predict current and future DINO/CLIP features (Eq. 2). The authors evaluate the frozen encoder on video object segmentation (DAVIS), part propagation (VIP), pose propagation (JHMDB), video semantic segmentation (CamVid, VSPW), and action classification (HMDB-51, UCF-101). They report consistent improvements over DINO, SiamMAE, VideoMAE, and other self-supervised baselines, and claim to be the first student video encoder distilled from image teachers that outperforms both the original image-only models and prior self-supervised video encoders on dense prediction tasks.

Significance. If the claims hold, FRAME is a practically valuable result: it is compact (e.g., 38M parameters for ViT-S/8 while outperforming DINO ViT-B/8 on several tasks), it shows consistent gains across seven datasets and three dense-prediction families, and its two-stage distillation recipe is clearly described. The paper also provides unusually transparent ablations of encoder depth, decoder depth, memory length, resolution, epochs, and data fraction, and promises code and checkpoints. The central caveat is that the reported configuration appears to have been selected using the same downstream benchmarks that are then used as evidence of superiority. This does not make the method circular in an algorithmic sense, since the same evaluation protocol is applied to baselines, but it does make the reported margins, which are often small (e.g., +3.7 J&F over SiamMAE on DAVIS S/16), difficult to separate from selection-induced inflation. The 'first' claim is also not directly tested against the most relevant image-to-video adaptation baselines, DINO-Tracker and Time Does Tell.

major comments (3)
  1. [§3.2, Eq. (2); Supplementary Tables 6–10, 15] The final model configuration was selected using downstream benchmark performance on DAVIS, VIP, JHMDB, and CamVid, and those same datasets are then reported as the evidence that FRAME outperforms prior methods. Specifically, the Stage 2 loss weights (α1=0.2, α2=0.1, α3=2.0, α4=0.4) are stated in §3.2 to be 'empirically determined based on results from a subset of DAVIS'; Supplementary Table C.1 selects the anticipation deltas (+4 semantic, +2 spatial) using feature variability on Kinetics and DAVIS subsets; and Supplementary Tables 6–10 and 15 select encoder depth (12), epochs (70), data fraction (40%), resolution (400×400), memory length (5), and decoder depth (1) from DAVIS/VIP/JHMDB/CamVid scores. No held-out validation split or pre-registered configuration protocol is described. Because the reported margins are modest relative to the amount of configuration search, the paper needs either an independent validation-based selection procedure or an explicit statement of how much of the reported advantage survives under a random/held-out configuration choice.
  2. [Abstract and §2; Table 1] The paper's central priority claim—that FRAME is the first student video encoder distilled from image teachers to outperform image-only models and prior self-supervised video encoders on dense prediction—is not directly tested against the two most relevant image-to-video adaptation baselines discussed in §2: DINO-Tracker [47] and Time Does Tell [41]. Neither method appears in Table 1 or any other comparison table. Since those methods also adapt pre-trained DINO features for video correspondence, the 'first' claim cannot be evaluated without including them (or explicitly restricting the claim to the specific training paradigm of full-encoder distillation rather than fine-tuning or time-tuning). Please add these baselines to the comparisons, or soften the claim to match the evaluated scope.
  3. [Tables 1, 4, 5 and Supplementary Table 20] No error bars or statistical significance measures are reported for the main results, even though the authors themselves report variability across training-data subsets in Supplementary Table 20: four Ego4D subsets give DAVIS J&F values of 66.1, 66.4, 66.0, and 65.8, spanning a range comparable to some of the reported margins over baselines. Since several headline improvements are small (e.g., +1.8 J&F for FRAME S/8 over SiamMAE on DAVIS; +1.5 mIoU for FRAME ViT-L/14 over DINOv2 on CamVid current-frame), the paper should report the variance across training runs or at least state that the numbers are single runs and indicate which differences are stable across the Ego4D-subset repeat in Table 20.
minor comments (5)
  1. [§3.1] There is a typo in the first sentence of the ViT Encoder paragraph: 'We the input image' should be 'We tokenize the input image' or similar.
  2. [§3.1] The phrase 'yielding a strong strong frame encoder' contains a duplicated word.
  3. [§3.3 and §3.2] The inference protocol is internally inconsistent: §3.3 says 'we discard all decoders' and then immediately says that zero-shot classification projects the [CLS] token into CLIP space 'using the semantic decoder.' Please clarify which decoders are retained at inference time and whether the semantic decoder is used only for the zero-shot classification mode.
  4. [Table 1 and Related Work] The spacing and formatting of 'DAVIS' is inconsistent ('DAVIS' vs. 'DA VIS'), and the caption of Figure 1 has a misplaced '(left)' / '(Right)' annotation.
  5. [References] References [20] and [21] are duplicates of the same paper (Space-Time Correspondence as a Contrastive Random Walk); please merge them.

Circularity Check

1 steps flagged · score 4.0 of 10

DAVIS-subset hyperparameter tuning creates a partial benchmark-selection loop; the core distillation-plus-memory derivation is not circular.

  1. fitted input called prediction [Sec. 3.2 (Eq. 2), Sec. 4 Table 1, and Sec. C.1]
    "These weights (α1, α2, α3, α4) were empirically determined based on results from a subset of DA VIS [33] dataset."

    The same benchmark (DAVIS) used to select the Stage-2 loss weights (Eq. 2) and, in Sec. C.1, the anticipation frame deltas is then reported in Table 1 as the headline evidence that FRAME 'outperforms SiamMAE' (e.g., 65.7 vs 62.0 J&F). Since the configuration was chosen on a DAVIS subset before evaluation on DAVIS, the reported margin is the result of a selection loop on the evaluation benchmark rather than an independent test. This is not an equation-level identity, but it is a fitted-input-then-prediction loop that directly bears on the 'first to outperform' priority claim.

full rationale

FRAME's core derivation is not circular: Stage 1 distills frozen DINO/CLIP features into a compact ViT, and Stage 2 adds memory and anticipation; the central comparisons (FRAME vs DINO/SiamMAE on DAVIS/VIP/JHMDB/CamVid) are empirical evaluations of features from the same encoder, not identities forced by the loss. In fact, the student is trained to match DINO and still beats DINO, which is a meaningful external result. The self-citations (refs [10, 25, 44]) appear only in related work and the optional region-tracking appendix; they are not load-bearing and do not by themselves raise the circularity score. The substantial caveat is that several hyperparameters were selected using DAVIS/VIP/JHMDB/CamVid subsets (Sec. 3.2, Sec. B, Sec. C.1) and the same datasets are then reported as results. This introduces a selection-on-test-set loop that can inflate the headline margins, but it does not make the method's predictions equivalent to its training inputs by construction. The score of 4 reflects this partial benchmark-selection circularity while acknowledging that the central distillation-memory derivation has independent content.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

No new physical entities, forces, or conserved quantities are introduced. The memory bank and anticipation heads are standard architectural components; the hyperparameters are the main free choices and were tuned on downstream benchmarks.

free parameters (4)
  • Stage 2 loss weights alpha1..alpha4 = 0.2, 0.1, 2.0, 0.4
    Empirically determined on a subset of DAVIS (Sec. 3.2); used in final model and reported DAVIS results.
  • Anticipation frame deltas = t+4 for CLIP/CLS, t+2 for DINO/patch
    Chosen from feature variability analysis on Kinetics and DAVIS subsets (Sec. C.1).
  • Memory length m = 5 frames
    Selected via ablation on DAVIS, VIP, JHMDB, CamVid (Table 10).
  • Training schedule and data fraction = 70 epochs, 400x400, 40% Kinetics, 700 Ego4D videos
    Selected via ablations on downstream benchmarks (Tables 7-9, 14).
assumptions (4)
  • domain assumption DINO patch features and CLIP class tokens are suitable and sufficient targets for learning video dense representations.
    Core of Stage 1 loss (Eq. 1).
  • domain assumption Predicting future teacher features from current and past frames improves temporal consistency and correspondence.
    Core of Stage 2 loss (Eq. 2); supported only by downstream ablations.
  • domain assumption The student encoder can surpass its frozen teachers on downstream tasks when augmented with temporal modules.
    Claimed in Sec. 4; depends on evaluation protocol.
  • domain assumption Simple kNN patch propagation and linear probes are faithful measures of feature quality.
    Evaluation protocol in Sec. 4, following prior work.

how reviews work

0 comments
Cite this review

Pith. "Pith review of FRAME: Pre-Training Video Feature Representations via Anticipation and Memory." pith.science (2026). https://pith.science/paper/EIPAM3NB

@misc{pith2026250605543,
  author       = {Pith},
  title        = {Pith review of: FRAME: Pre-Training Video Feature Representations via Anticipation and Memory},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/EIPAM3NB}},
  note         = {Machine review of arXiv:2506.05543}
}
read the original abstract

Dense video prediction tasks, such as object tracking and semantic segmentation, require video encoders that generate temporally consistent, spatially dense features for every frame. However, existing approaches fall short: image encoders like DINO or CLIP lack temporal awareness, while video models such as VideoMAE underperform compared to image encoders on dense prediction tasks. We address this gap with FRAME, a self-supervised video frame encoder tailored for dense video understanding. FRAME learns to predict current and future DINO patch features from past and present RGB frames, leading to spatially precise and temporally coherent representations. To our knowledge, FRAME is the first video encoder to leverage image-based models for dense prediction while outperforming them on tasks requiring fine-grained visual correspondence. As an auxiliary capability, FRAME aligns its class token with CLIP's semantic space, supporting language-driven tasks such as video classification. We evaluate FRAME across six dense prediction tasks on seven datasets, where it consistently outperforms image encoders and existing self-supervised video models. Despite its versatility, FRAME maintains a compact architecture suitable for a range of downstream applications.

Figures

Figures reproduced from arXiv: 2506.05543 by the authors.

Figure 1
Figure 1. FRAME outperforms state-of-the-art self-supervised models (DINO, SiamMAE) on [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Overview of FRAME Architecture and Two-Stage Training Process. In Stage 1, the encoder is trained to jointly distill CLIP features (providing semantic understanding) and DINO features (providing spatial understanding). In Stage 2, this pre-trained encoder processes the past and current frames and the model is optimized for memory integration and future anticipation. performance gap motivates our approach: rather tha… view at source ↗
Figure 3
Figure 3. Examples of correspondence and segmentation tasks (a) Video object segmentation: initial (top) and propagated (bottom) frames. (b) Semantic part propagation: initial (top) and propagated (bottom) frames. (c) Pose propagation: initial (top) and propagated (bottom) poses. (d) Semantic segmentation: original image (top) and segmentation overlay (bottom). randomly sampled videos from Ego4D [14]. We repeat training with … view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: Comparison of FRAME and DINO on feature propagation across video frames. FRAME demonstrates greater robustness to viewpoint changes, occlusions, and object reappearances, making it a more suitable video frame encoder. drawn from a similar table in [15]. In the suppleme…
Figure 5
Figure 5. Figure 5: Comparison of FRAME and DINO across model scales. FRAME outperforms DINO with fewer parameters. results suggest that FRAME + SAM 2 could enable a more memory-efficient many-object tracking. See supplementary for additional details of the experiment. Semantic Part Propa…
Figure 6
Figure 6. Figure 6: (a) Semantic segmentation on cur￾rent and future frames. FRAME outperforms DINO on CamVid. Removing memory or an￾ticipation reduces performance, showing their complementary role in temporal reasoning [PITH_FULL_IMAGE:figures/full_fig_p009_6.png]
Figure 7
Figure 7. Figure 7: Region-based tracking with FRAME. We track different objects—eg. camel, fence, and floor—across time using FRAME-pooled region features. Each row shows the temporal propagation of a single object, demonstrating the ability of FRAME to support accurate and consistent tr…

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

57 extracted references · 30 canonical work pages

  1. [47]

    DINO-Tracker: Taming DINO for Self-Supervised Point Tracking in a Single Video

    N. Tumanyan, A. Singer, S. Bagon, and T. Dekel. Dino-tracker: Taming dino for self-supervised point tracking in a single video, 2024. URLhttps://arxiv.org/abs/2403.14548

    work page Pith review arXiv 2024

  2. [41]

    Salehi, E

    M. Salehi, E. Gavves, C. G. M. Snoek, and Y . M. Asano. Time does tell: Self-supervised time-tuning of dense image representations.2023 IEEE/CVF International Conference on Computer Vision (ICCV), pages 16490–16501, 2023. URLhttps://api.semanticscholar.org/CorpusID:261076544

    work page 2023

  3. [1]

    Self-supervised Object-Centric Learning for Videos

    G. Aydemir, W. Xie, and F. Güney. Self-supervised object-centric learning for videos.ArXiv, abs/2310.06907, 2023. URLhttps://api.semanticscholar.org/CorpusID:263835208

    work page Pith review arXiv 2023

  4. [2]

    Fully-Convolutional Siamese Networks for Object Tracking

    L. Bertinetto, J. Valmadre, J. F. Henriques, A. Vedaldi, and P. H. S. Torr. Fully-convolutional siamese networks for object tracking, 2021. URLhttps://arxiv.org/abs/1606.09549

    work page Pith review arXiv 2021

  5. [3]

    G. J. Brostow, J. Shotton, J. Fauqueur, and R. Cipolla. Segmentation and recognition using structure from motion point clouds. InECCV (1), pages 44–57, 2008

    work page 2008

  6. [4]

    Caron, H

    M. Caron, H. Touvron, I. Misra, H. Jégou, J. Mairal, P. Bojanowski, and A. Joulin. Emerging properties in self-supervised vision transformers, 2021. URLhttps://arxiv.org/abs/2104.14294

    arXiv 2021

  7. [5]

    T. Chen, S. Kornblith, M. Norouzi, and G. Hinton. A simple framework for contrastive learning of visual representations, 2020. URLhttps://arxiv.org/abs/2002.05709

    arXiv 2020

  8. [7]

    Chen and K

    X. Chen and K. He. Exploring simple siamese representation learning.arXiv preprint arXiv:2011.10566, 2020

    arXiv 2011

Show all 57 references
  1. [8]

    H. K. Cheng, S. W. Oh, B. Price, A. Schwing, and J.-Y . Lee. Tracking anything with decoupled video segmentation, 2023. URLhttps://arxiv.org/abs/2309.03903

    2023 arXiv

  2. [9]

    H. K. Cheng, S. W. Oh, B. Price, J.-Y . Lee, and A. Schwing. Putting the object back into video object segmentation, 2024. URLhttps://arxiv.org/abs/2310.12982

    2024 arXiv

  3. [10]

    I. R. Dave, S. Jenni, and M. Shah. No more shortcuts: Realizing the potential of temporal self-supervision. InProceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 1481–1491, 2024

    2024

  4. [11]

    Doersch, A

    C. Doersch, A. Gupta, and A. A. Efros. Unsupervised visual representation learning by context prediction. InProceedings of the IEEE international conference on computer vision, pages 1422–1430, 2015

    2015

  5. [12]

    Eymaël, R

    A. Eymaël, R. Vandeghen, A. Cioppa, S. Giancola, B. Ghanem, and M. V . Droogenbroeck. Efficient image pre-training with siamese cropped masked autoencoders, 2024. URL https://arxiv.org/abs/2403. 17823

    2024

  6. [13]

    Feichtenhofer, H

    C. Feichtenhofer, H. Fan, B. Xiong, R. Girshick, and K. He. A large-scale study on unsupervised spatiotemporal representation learning, 2021. URLhttps://arxiv.org/abs/2104.14558

    2021 arXiv

  7. [14]

    Grauman, A

    K. Grauman, A. Westbury, E. Byrne, Z. Chavis, A. Furnari, R. Girdhar, J. Hamburger, H. Jiang, M. Liu, X. Liu, M. Martin, T. Nagarajan, I. Radosavovic, S. K. Ramakrishnan, F. Ryan, J. Sharma, M. Wray, M. Xu, E. Z. Xu, C. Zhao, S. Bansal, D. Batra, V . Cartillier, S. Crane, T. D...

    2021

  8. [15]

    Gupta, J

    A. Gupta, J. Wu, J. Deng, and L. Fei-Fei. Siamese masked autoencoders. InNeurIPS, 2023

    2023

  9. [17]

    K. He, H. Fan, Y . Wu, S. Xie, and R. Girshick. Momentum contrast for unsupervised visual representation learning, 2020. URLhttps://arxiv.org/abs/1911.05722. 10

    2020 arXiv

  10. [18]

    K. He, H. Fan, Y . Wu, S. Xie, and R. Girshick. Momentum contrast for unsupervised visual representation learning. InProceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 9729–9738, 2020

    2020

  11. [19]

    K. He, X. Chen, S. Xie, Y . Li, P. Dollár, and R. Girshick. Masked autoencoders are scalable vision learners,

  12. [21]

    Jabri, A

    A. Jabri, A. Owens, and A. A. Efros. Space-time correspondence as a contrastive random walk, 2020. URLhttps://arxiv.org/abs/2006.14613

    2020 arXiv

  13. [22]

    Jhuang, J

    H. Jhuang, J. Gall, S. Zuffi, C. Schmid, and M. J. Black. Towards understanding action recognition. In2013 IEEE International Conference on Computer Vision, pages 3192–3199, 2013. doi: 10.1109/ICCV .2013.396

    2013 doi

  14. [23]

    Karaev, I

    N. Karaev, I. Rocco, B. Graham, N. Neverova, A. Vedaldi, and C. Rupprecht. Cotracker: It is better to track together, 2024. URLhttps://arxiv.org/abs/2307.07635

    2024 arXiv

  15. [24]

    W. Kay, J. Carreira, K. Simonyan, B. Zhang, C. Hillier, S. Vijayanarasimhan, F. Viola, T. Green, T. Back, P. Natsev, M. Suleyman, and A. Zisserman. The kinetics human action video dataset, 2017. URL https://arxiv.org/abs/1705.06950

    2017 arXiv

  16. [25]

    Khosla, S

    S. Khosla, S. T. V , A. Schwing, and D. Hoiem. Relocate: A simple training-free baseline for visual query localization using region-based representations, 2024. URLhttps://arxiv.org/abs/2412.01826

    2024

  17. [26]

    Kirillov, E

    A. Kirillov, E. Mintun, N. Ravi, H. Mao, C. Rolland, L. Gustafson, T. Xiao, S. Whitehead, A. C. Berg, W.-Y . Lo, P. Dollár, and R. Girshick. Segment anything, 2023. URLhttps://arxiv.org/abs/2304.02643

    2023 arXiv

  18. [27]

    Kuehne, H

    H. Kuehne, H. Jhuang, E. Garrote, T. Poggio, and T. Serre. Hmdb: A large video database for human motion recognition. In2011 International Conference on Computer Vision, pages 2556–2563, 2011. doi: 10.1109/ICCV .2011.6126543

    2011

  19. [28]

    X. Li, S. Liu, S. D. Mello, X. Wang, J. Kautz, and M.-H. Yang. Joint-task self-supervised learning for temporal correspondence, 2019. URLhttps://arxiv.org/abs/1909.11895

    2019 arXiv

  20. [29]

    X. Li, S. Liu, S. D. Mello, X. Wang, J. Kautz, and M.-H. Yang. Joint-task self-supervised learning for temporal correspondence. InNeurIPS, 2019

    2019

  21. [30]

    J. Miao, Y . Wei, Y . Wu, C. Liang, G. Li, and Y . Yang. Vspw: A large-scale dataset for video scene parsing in the wild. In2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 4131–4141, 2021. doi: 10.1109/CVPR46437.2021.00412

    2021

  22. [31]

    Misra, C

    I. Misra, C. L. Zitnick, and M. Hebert. Shuffle and learn: unsupervised learning using temporal order verification. InComputer Vision–ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11–14, 2016, Proceedings, Part I 14, pages 527–544. Springer, 2016

    2016

  23. [32]

    Oquab, T

    M. Oquab, T. Darcet, T. Moutakanni, H. V o, M. Szafraniec, V . Khalidov, P. Fernandez, D. Haziza, F. Massa, A. El-Nouby, M. Assran, N. Ballas, W. Galuba, R. Howes, P.-Y . Huang, S.-W. Li, I. Misra, M. Rabbat, V . Sharma, G. Synnaeve, H. Xu, H. Jegou, J. Mairal, P. Labatut, A. ...

    2024 arXiv

  24. [33]

    Pont-Tuset, F

    J. Pont-Tuset, F. Perazzi, S. Caelles, P. Arbeláez, A. Sorkine-Hornung, and L. V . Gool. The 2017 davis challenge on video object segmentation, 2018. URLhttps://arxiv.org/abs/1704.00675

    2017 arXiv

  25. [34]

    R. Qian, S. Ding, X. Liu, and D. Lin. Semantics meets temporal correspondence: Self-supervised object- centric learning in videos.2023 IEEE/CVF International Conference on Computer Vision (ICCV), pages 16629–16641, 2023. URLhttps://api.semanticscholar.org/CorpusID:261049053

    2023

  26. [35]

    Radford, J

    A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, G. Krueger, and I. Sutskever. Learning transferable visual models from natural language supervision, 2021. URLhttps://arxiv.org/abs/2103.00020

    2021 arXiv

  27. [36]

    Ranasinghe, M

    K. Ranasinghe, M. Naseer, S. Khan, F. S. Khan, and M. S. Ryoo. Self-supervised video transformer. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2874–2884, 2022

    2022

  28. [37]

    Ranzinger, G

    M. Ranzinger, G. Heinrich, J. Kautz, and P. Molchanov. Am-radio: Agglomerative vision foundation model reduce all domains into one. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12490–12500, 2024

    2024

  29. [38]

    Rasheed, M

    H. Rasheed, M. U. Khattak, M. Maaz, S. Khan, and F. S. Khan. Fine-tuned clip models are efficient video learners, 2023. URLhttps://arxiv.org/abs/2212.03640

    2023 arXiv

  30. [39]

    H. A. Rasheed, M. U. Khattak, M. Maaz, S. H. Khan, and F. S. Khan. Fine-tuned clip models are efficient video learners.2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 6545–6554, 2022. URLhttps://api.semanticscholar.org/CorpusID:254366626. 11

    2023

  31. [40]

    N. Ravi, V . Gabeur, Y .-T. Hu, R. Hu, C. Ryali, T. Ma, H. Khedr, R. Rädle, C. Rolland, L. Gustafson, E. Mintun, J. Pan, K. V . Alwala, N. Carion, C.-Y . Wu, R. Girshick, P. Dollár, and C. Feichtenhofer. Sam 2: Segment anything in images and videos, 2024. URLhttps://arxiv.org/...

    2024 arXiv

  32. [42]

    Sameni, K

    S. Sameni, K. Kafle, H. Tan, and S. Jenni. Building vision-language models on solid foundations with masked distillation. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14216–14226, 2024

    2024

  33. [43]

    Sermanet, C

    P. Sermanet, C. Lynch, Y . Chebotar, J. Hsu, E. Jang, S. Schaal, and S. Levine. Time-contrastive networks: Self-supervised learning from video, 2018. URLhttps://arxiv.org/abs/1704.06888

    2018 arXiv

  34. [44]

    Shlapentokh-Rothman, A

    M. Shlapentokh-Rothman, A. Blume, Y . Xiao, Y . Wu, S. T. V , H. Tao, J. Y . Lee, W. Torres, Y .-X. Wang, and D. Hoiem. Region-based representations revisited, 2024. URL https://arxiv.org/abs/2402.02352

    2024 arXiv

  35. [45]

    Soomro, A

    K. Soomro, A. R. Zamir, and M. Shah. Ucf101: A dataset of 101 human actions classes from videos in the wild, 2012. URLhttps://arxiv.org/abs/1212.0402

    2012 arXiv

  36. [46]

    Z. Tong, Y . Song, J. Wang, and L. Wang. Videomae: Masked autoencoders are data-efficient learners for self-supervised video pre-training, 2022. URLhttps://arxiv.org/abs/2203.12602

    2022 arXiv

  37. [48]

    Valmadre, L

    J. Valmadre, L. Bertinetto, J. Henriques, A. Vedaldi, and P. H. S. Torr. End-to-end representation learning for correlation filter based tracking. In2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 5000–5008, 2017. doi: 10.1109/CVPR.2017.531

    2017 doi

  38. [49]

    M. Wang, J. Xing, and Y . Liu. Actionclip: A new paradigm for video action recognition.ArXiv, abs/2109.08472, 2021. URLhttps://api.semanticscholar.org/CorpusID:237563206

    2021 arXiv

  39. [50]

    Q. Wang, J. Du, K. Yan, and S. Ding. Seeing in flowing: Adapting clip for action recognition with motion prompts learning.Proceedings of the 31st ACM International Conference on Multimedia, 2023. URL https://api.semanticscholar.org/CorpusID:260735882

    2023

  40. [51]

    X. Wang, A. Jabri, and A. A. Efros. Learning correspondence from the cycle-consistency of time. In CVPR, 2019

    2019

  41. [52]

    Y . Wang, X. Shen, Y . Yuan, Y . Du, M. Li, S. X. Hu, J. L. Crowley, and D. Vaufreydaz. Tokencut: Segmenting objects in images and videos with self-supervised transformer and normalized cut.IEEE Transactions on Pattern Analysis and Machine Intelligence, 45:15790–15801, 2022. U...

    2022

  42. [53]

    Y . Weng, M. Han, H. He, M. Li, L. Yao, X. Chang, and B. Zhuang. Mask propagation for efficient video semantic segmentation, 2023. URLhttps://arxiv.org/abs/2310.18954

    2023 arXiv

  43. [54]

    T. Xiao, X. Wang, A. A. Efros, and T. Darrell. What should not be contrastive in contrastive learning, 2021. URLhttps://arxiv.org/abs/2008.05659

    2021 arXiv

  44. [55]

    Xu and X

    J. Xu and X. Wang. Rethinking self-supervised correspondence learning: A video frame-level similarity perspective, 2021. URLhttps://arxiv.org/abs/2103.17263

    2021 arXiv

  45. [56]

    J. Xu, Z. Xiong, and S. P. Bhattacharyya. Pidnet: A real-time semantic segmentation network inspired by pid controllers, 2023. URLhttps://arxiv.org/abs/2206.02066

    2023 arXiv

  46. [57]

    N. Xu, L. Yang, Y . Fan, D. Yue, Y . Liang, J. Yang, and T. Huang. Youtube-vos: A large-scale video object segmentation benchmark, 2018. URLhttps://arxiv.org/abs/1809.03327

    2018 arXiv

  47. [58]

    Zhang, X

    T. Zhang, X. Tian, Y . Zhou, S. Ji, X. Wang, X. Tao, Y . Zhang, P. Wan, Z. Wang, and Y . Wu. Dvis++: Improved decoupled framework for universal video segmentation, 2023. URL https://arxiv.org/ abs/2312.13305

    2023 arXiv

  48. [59]

    Q. Zhou, X. Liang, K. Gong, and L. Lin. Adaptive temporal encoding network for video instance-level human parsing, 2018. URLhttps://arxiv.org/abs/1808.00661. 12 A Supplementary This section is structured as follows. In section B, we analyze the sensitivity of FRAME to various ...

    2018 arXiv

  49. [2021]

    URLhttps://arxiv.org/abs/2111.06377

    arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.