pith. machine review for the scientific record. sign in

arxiv: 2605.08810 · v1 · submitted 2026-05-09 · 💻 cs.LG · cs.AI

Recognition: no theorem link

Compressed Video Aggregator: Content-driven Module for Efficient Micro-Video Recommendation

Yang Xiao , Huiyuan Chen , Kaiyuan Deng , Chao Jiang , Zinan Ling , Ruimeng Ye , Xiaolong Ma , Bo Hui
Authors on Pith no claims yet

Pith reviewed 2026-05-12 02:05 UTC · model grok-4.3

classification 💻 cs.LG cs.AI
keywords micro-video recommendationvideo aggregatorfrozen embeddingslatent reasoningkeyframe selectionCLIPtraining efficiencyrecommender systems
0
0 comments X

The pith

The Compressed Video Aggregator decouples video content from preference learning by combining frozen VFM embeddings through latent reasoning to create compact representations that improve micro-video recommendation accuracy while cutting训练

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper introduces CVA as a lightweight module for micro-video recommendation that processes video information separately from learning user preferences. It does this by taking embeddings already computed by frozen video foundation models and combining them in a simple latent space without using cross-attention. This produces short video vectors that recommendation models can use efficiently. Tests on two datasets reveal better results along with huge savings in training time and GPU memory. Selecting better key frames using video titles and CLIP also boosts every tested method, and the system handles some title errors reasonably well.

Core claim

The central claim is that aggregating embeddings from frozen video foundation models using latent reasoning without cross-attention projection yields compact video embeddings sufficient for effective micro-video recommendation. This approach decouples the video content extraction from the preference learning task, leading to substantial efficiency gains. Re-selecting key frames based on titles via CLIP addresses redundancy in standard benchmarks and further improves outcomes across methods.

What carries the argument

Compressed Video Aggregator (CVA) that aggregates frozen VFM embeddings via latent reasoning without cross-attention projection to produce compact video embeddings for recommenders.

Load-bearing premise

Frozen embeddings from video foundation models plus title-guided keyframe selection provide enough video content information to support effective preference learning without any fine-tuning or full video processing.

What would settle it

An experiment training an end-to-end fine-tuned video recommendation model on the same datasets and showing higher accuracy than CVA while using comparable or lower resources would challenge the sufficiency and efficiency claims.

Figures

Figures reproduced from arXiv: 2605.08810 by Bo Hui, Chao Jiang, Huiyuan Chen, Kaiyuan Deng, Ruimeng Ye, Xiaolong Ma, Yang Xiao, Zinan Ling.

Figure 1
Figure 1. Figure 1: Data size comparison. Video Compression. (Main) While resampling reduces temporal redun￾dancy, it does not address the com￾putational cost of high-dimensional visual features. Standard visual foun￾dation models (e.g., DINOv3, CLIP) produce dense features that remain too costly for sequential recommendation, even with few sampled frames. Previ￾ous studies [14] showed that directly using frozen visual backbo… view at source ↗
Figure 2
Figure 2. Figure 2: The overall pipeline of our proposed method. The framework is decoupled into two distinct [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Compressed Video Aggregator (CVA) adaptively encodes multiple frame embeddings based on the number of frames N into a compact video embedding. 3.2.2 Frozen Feature Extraction We employ pre-trained Visual Foundation Models (VFM), such as DINOv3, as the visual encoder VFM(·). This encoder is frozen to avoid the massive computational cost of back-propagation through the vision backbone. For each selected fram… view at source ↗
Figure 5
Figure 5. Figure 5: Comparison of parameters and performance. [PITH_FULL_IMAGE:figures/full_fig_p006_5.png] view at source ↗
Figure 4
Figure 4. Figure 4: Statistics of different tags of video. The blue bars represent the corresponding HIT@10 [PITH_FULL_IMAGE:figures/full_fig_p007_4.png] view at source ↗
Figure 6
Figure 6. Figure 6: Performance in different frames More frames, more performance [PITH_FULL_IMAGE:figures/full_fig_p009_6.png] view at source ↗
read the original abstract

We propose Compressed Video Aggregator (CVA), a lightweight micro-video recommendation module that decouples video information from preference learning. It aggregates frozen VFM embeddings, and uses latent reasoning without cross-attention projection, producing compact video embeddings for recommenders. Due to the redundancy in the frame count of the original benchmark and its overly coarse sampling, we used titles to re-select key frames based on CLIP. Experiments on MicroLens and Short-Video show consistent gains with orders-of-magnitude reductions in training time and GPU memory, and re-selected frames can further enhance the performance of all methods, including CVA. Furthermore, we also discussed the impact of several scenarios involving erroneous titles on our method. Code will be released soon.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

3 major / 2 minor

Summary. The paper proposes Compressed Video Aggregator (CVA), a lightweight module for micro-video recommendation that decouples video content modeling from preference learning. CVA aggregates embeddings from frozen Video Foundation Models (VFMs) via latent reasoning that avoids cross-attention projections, yielding compact video representations. To address frame redundancy and coarse sampling in existing benchmarks, the authors introduce title-driven key-frame re-selection using CLIP. Experiments on the MicroLens and Short-Video datasets report consistent accuracy gains for CVA and baselines, together with orders-of-magnitude reductions in training time and GPU memory; the paper also examines the effect of erroneous titles on the pipeline.

Significance. If the efficiency and accuracy claims are substantiated by the full experimental results, the work could meaningfully advance practical micro-video recommenders by demonstrating that precomputed frozen VFM features plus lightweight aggregation suffice for competitive performance. The explicit discussion of title errors and the promise of code release are positive elements. The significance hinges on whether the frozen-representation assumption holds for preference signals that may depend on motion or audio-visual alignment not captured by title-aligned frames.

major comments (3)
  1. [Abstract and §4] The central claim that frozen VFM embeddings plus title-based CLIP key-frame selection already encode sufficient content for preference learning (Abstract and §4) lacks a direct test against end-to-end fine-tuning of the VFM or full-video processing. Without such a comparison, it is unclear whether the reported gains reflect a principled decoupling or simply an incomplete content proxy; this is load-bearing for the efficiency narrative.
  2. [Method description] The latent-reasoning aggregation mechanism is described as operating 'without cross-attention projection' (Abstract), yet the precise formulation, number of parameters, and how it differs from standard attention-based pooling are not specified with equations or pseudocode. This omission prevents verification that the method is indeed parameter-light and projection-free.
  3. [Experiments section] Table results on MicroLens and Short-Video are cited as showing 'consistent gains' and 'orders-of-magnitude' efficiency improvements, but the manuscript provides no ablation isolating the contribution of the re-selected frames versus the original sampling, nor error bars or statistical significance tests. This weakens the claim that re-selection 'further enhance[s] the performance of all methods'.
minor comments (2)
  1. [Method] Notation for the VFM embedding dimensions and the output embedding size of CVA should be introduced explicitly in the method section rather than only in the experimental setup.
  2. [Discussion] The discussion of erroneous titles would benefit from a quantitative breakdown (e.g., percentage of titles affected and corresponding performance delta) rather than a purely qualitative treatment.

Simulated Author's Rebuttal

3 responses · 0 unresolved

We thank the referee for the constructive feedback, which highlights important areas for clarification and strengthening. We address each major comment below and indicate the revisions planned for the next version of the manuscript.

read point-by-point responses

  1. Referee: [Abstract and §4] The central claim that frozen VFM embeddings plus title-based CLIP key-frame selection already encode sufficient content for preference learning (Abstract and §4) lacks a direct test against end-to-end fine-tuning of the VFM or full-video processing. Without such a comparison, it is unclear whether the reported gains reflect a principled decoupling or simply an incomplete content proxy; this is load-bearing for the efficiency narrative.

    Authors: Our work centers on the efficiency advantages of a decoupled approach using frozen VFMs, which is the core practical contribution for resource-constrained micro-video recommenders. All baselines in our experiments operate under the same frozen-embedding constraint, so the reported gains isolate the effect of the CVA aggregator rather than differences in content modeling. A comprehensive end-to-end fine-tuning comparison would require prohibitive compute and contradict the lightweight premise. In the revision we will expand the discussion in §4 to explicitly note this scope limitation, reference related literature on frozen versus fine-tuned video representations, and, where feasible, include a small-scale proxy experiment on a data subset. revision: partial

  2. Referee: [Method description] The latent-reasoning aggregation mechanism is described as operating 'without cross-attention projection' (Abstract), yet the precise formulation, number of parameters, and how it differs from standard attention-based pooling are not specified with equations or pseudocode. This omission prevents verification that the method is indeed parameter-light and projection-free.

    Authors: We agree that the current description is insufficient for verification. The revised manuscript will add the full mathematical formulation of the latent-reasoning aggregator, its exact parameter count, and pseudocode. These additions will make explicit how the mechanism avoids cross-attention projections and remains lighter than standard attention-based pooling. revision: yes

  3. Referee: [Experiments section] Table results on MicroLens and Short-Video are cited as showing 'consistent gains' and 'orders-of-magnitude' efficiency improvements, but the manuscript provides no ablation isolating the contribution of the re-selected frames versus the original sampling, nor error bars or statistical significance tests. This weakens the claim that re-selection 'further enhance[s] the performance of all methods'.

    Authors: The referee correctly identifies gaps in experimental rigor. We will insert a new ablation table that directly compares original sampling against title-driven key-frame re-selection for every method, including CVA. We will also report means and standard deviations across multiple random seeds and add paired statistical significance tests to support the performance claims. revision: yes

Circularity Check

0 steps flagged

No circularity: derivation relies on independent frozen models and empirical evaluation.

full rationale

The paper introduces CVA as a lightweight aggregator of embeddings from externally pre-trained frozen VFMs, combined with CLIP-based title-driven key-frame re-selection. All reported gains (efficiency, performance on MicroLens/Short-Video) are empirical outcomes of applying this module to standard recommendation pipelines. No equations or claims reduce by construction to fitted parameters from the evaluation data, no self-citation chains justify core premises, and no ansatz or uniqueness result is smuggled in. The method is self-contained against external benchmarks (pre-trained VFMs and CLIP) whose training is independent of the present experiments.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 0 invented entities

Abstract-only review provides minimal detail on internal parameters or assumptions. Main unstated premises are sufficiency of frozen embeddings and reliability of title-based frame selection.

axioms (1)
  • domain assumption Frozen VFM embeddings plus title-driven CLIP selection capture enough video semantics for downstream preference learning
    Central to the decoupling claim and efficiency argument.

pith-pipeline@v0.9.0 · 5437 in / 1301 out tokens · 55566 ms · 2026-05-12T02:05:09.875128+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

44 extracted references · 44 canonical work pages · 2 internal anchors

  1. [1]

    The 55th statistical report on internet development in china,

    China Internet Network Information Center, “The 55th statistical report on internet development in china,” China Internet Network Information Center (CNNIC), Beijing, Tech. Rep., jan 2025, accessed: 2025-01-17. [Online]. Available: https://www.cnnic.com.cn/IDR/ReportDownloads/202505/ P020250514564119130448.pdf

    work page 2025

  2. [2]

    Real-time short video recom- mendation on mobile devices,

    X. Gong, Q. Feng, Y . Zhang, J. Qin, W. Ding, B. Li, P. Jiang, and K. Gai, “Real-time short video recom- mendation on mobile devices,” inProceedings of the 31st ACM international conference on information & knowledge management, 2022, pp. 3103–3112

    work page 2022

  3. [3]

    Dvr: micro-video recommendation optimizing watch-time-gain under duration bias,

    Y . Zheng, C. Gao, J. Ding, L. Yi, D. Jin, Y . Li, and M. Wang, “Dvr: micro-video recommendation optimizing watch-time-gain under duration bias,” inProceedings of the 30th ACM International Conference on Multimedia, 2022, pp. 334–345

    work page 2022

  4. [4]

    Improving micro-video recommendation by controlling position bias,

    Y . Yu, B. Jin, J. Song, B. Li, Y . Zheng, and W. Zhuo, “Improving micro-video recommendation by controlling position bias,” inJoint European Conference on Machine Learning and Knowledge Discovery in Databases. Springer, 2022, pp. 508–523

    work page 2022

  5. [5]

    Dancelets mining for video recommendation based on dance styles,

    T. Han, H. Yao, C. Xu, X. Sun, Y . Zhang, and J. J. Corso, “Dancelets mining for video recommendation based on dance styles,”IEEE Transactions on Multimedia, vol. 19, no. 4, pp. 712–724, 2016

    work page 2016

  6. [6]

    Concept-aware denoising graph neural network for micro-video recommendation,

    Y . Liu, Q. Liu, Y . Tian, C. Wang, Y . Niu, Y . Song, and C. Li, “Concept-aware denoising graph neural network for micro-video recommendation,” inProceedings of the 30th ACM international conference on information & knowledge management, 2021, pp. 1099–1108

    work page 2021

  7. [7]

    Kuairec: A fully-observed dataset and insights for evaluating recommender systems

    C. Gao, S. Li, W. Lei, J. Chen, B. Li, P. Jiang, X. He, J. Mao, and T.-S. Chua, “Kuairec: A fully-observed dataset and insights for evaluating recommender systems,” inProceedings of the 31st ACM International Conference on Information & Knowledge Management, ser. CIKM ’22, 2022, p. 540–550. [Online]. Available: https://doi.org/10.1145/3511808.3557220

    work page doi:10.1145/3511808.3557220 2022

  8. [8]

    Tenrec: A large-scale multipurpose benchmark dataset for recommender systems,

    G. Yuan, F. Yuan, Y . Li, B. Kong, S. Li, L. Chen, M. Yang, C. Yu, B. Hu, Z. Liet al., “Tenrec: A large-scale multipurpose benchmark dataset for recommender systems,”Advances in Neural Information Processing Systems, vol. 35, pp. 11 480–11 493, 2022

    work page 2022

  9. [9]

    User-video co-attention network for personalized micro-video recommendation,

    S. Liu, Z. Chen, H. Liu, and X. Hu, “User-video co-attention network for personalized micro-video recommendation,” inThe world wide web conference, 2019, pp. 3020–3026

    work page 2019

  10. [10]

    What aspect do you like: Multi-scale time-aware user interest modeling for micro-video recommendation,

    H. Jiang, W. Wang, Y . Wei, Z. Gao, Y . Wang, and L. Nie, “What aspect do you like: Multi-scale time-aware user interest modeling for micro-video recommendation,” inProceedings of the 28th ACM International conference on Multimedia, 2020, pp. 3487–3495

    work page 2020

  11. [11]

    Semi: A sequential multi-modal information transfer network for e-commerce micro-video recommendations,

    C. Lei, Y . Liu, L. Zhang, G. Wang, H. Tang, H. Li, and C. Miao, “Semi: A sequential multi-modal information transfer network for e-commerce micro-video recommendations,” inProceedings of the 27th ACM SIGKDD Conference on knowledge discovery & data mining, 2021, pp. 3161–3171

    work page 2021

  12. [12]

    Short video segment-level user dynamic interests modeling in personalized recommendation,

    Z. He, Z. Ling, J. Li, Z. Guo, W. Ma, X. Luo, M. Zhang, and G. Zhou, “Short video segment-level user dynamic interests modeling in personalized recommendation,” inProceedings of the 48th International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 2025, Padua, Italy, July 13-18, 2025, N. Ferro, M. Maistro, G. Pasi, O. Alo...

    work page doi:10.1145/3726302.3730083 2025

  13. [13]

    Mutual information-aware knowledge distillation for short video recommendation,

    H. Xu, T. Pan, Z. Liu, and X. Xu, “Mutual information-aware knowledge distillation for short video recommendation,” inProceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining, V .1, KDD 2025, Toronto, ON, Canada, August 3-7, 2025, Y . Sun, F. Chierichetti, H. W. Lauw, C. Perlich, W. H. Tok, and A. Tomkins, Eds. ACM, 2025, pp. 2...

    work page doi:10.1145/3690624.3709403 2025

  14. [14]

    A content-driven micro-video recommendation dataset at scale,

    Y . Ni, Y . Cheng, X. Liu, J. Fu, Y . Li, X. He, Y . Zhang, and F. Yuan, “A content-driven micro-video recommendation dataset at scale,” inProceedings of the 34th ACM International Conference on Information and Knowledge Management, 2025, pp. 6486–6491

    work page 2025

  15. [15]

    Describe what you see with multimodal large language models to enhance video recommendations,

    M. De Nadai, A. Damianou, and M. Lalmas, “Describe what you see with multimodal large language models to enhance video recommendations,” inProceedings of the Nineteenth ACM Conference on Recommender Systems, 2025, pp. 1159–1163

    work page 2025

  16. [16]

    Exploring the design space of visual context representation in video mllms,

    Y . Du, Y . Huo, K. Zhou, Z. Zhao, H. Lu, H. Huang, X. Zhao, B. Wang, W. Chen, and J. Wen, “Exploring the design space of visual context representation in video mllms,” inThe Thirteenth International Conference on Learning Representations, ICLR 2025, Singapore, April 24-28, 2025. OpenReview.net,

    work page 2025

  17. [17]

    Available: https://openreview.net/forum?id=UN6Ik6OCx8

    [Online]. Available: https://openreview.net/forum?id=UN6Ik6OCx8

    work page

  18. [18]

    A large-scale dataset with behavior, attributes, and content of mobile short-video platform,

    Y . Shang, C. Gao, N. Li, and Y . Li, “A large-scale dataset with behavior, attributes, and content of mobile short-video platform,” inCompanion Proceedings of the ACM on Web Conference 2025, 2025, pp. 793–796

    work page 2025

  19. [19]

    Video swin transformer,

    Z. Liu, J. Ning, Y . Cao, Y . Wei, Z. Zhang, S. Lin, and H. Hu, “Video swin transformer,” inProceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2022, pp. 3202–3211

    work page 2022

  20. [20]

    Slowfast networks for video recognition,

    C. Feichtenhofer, H. Fan, J. Malik, and K. He, “Slowfast networks for video recognition,” inProceedings of the IEEE/CVF international conference on computer vision, 2019, pp. 6202–6211

    work page 2019

  21. [21]

    Is space-time attention all you need for video understanding?

    G. Bertasius, H. Wang, and L. Torresani, “Is space-time attention all you need for video understanding?” inIcml, vol. 2, no. 3, 2021, p. 4

    work page 2021

  22. [22]

    Videomae: Masked autoencoders are data-efficient learners for self-supervised video pre-training,

    Z. Tong, Y . Song, J. Wang, and L. Wang, “Videomae: Masked autoencoders are data-efficient learners for self-supervised video pre-training,”Advances in neural information processing systems, vol. 35, pp. 10 078–10 093, 2022

    work page 2022

  23. [23]

    Learning transferable visual models from natural language supervision,

    A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clarket al., “Learning transferable visual models from natural language supervision,” inInternational conference on machine learning. PmLR, 2021, pp. 8748–8763

    work page 2021

  24. [24]

    Perceiver io: A general architecture for structured inputs & outputs,

    A. Jaegle, S. Borgeaud, J.-B. Alayrac, C. Doersch, C. Ionescu, D. Ding, S. Koppula, A. Brock, E. Shelhamer, O. Hénaff, M. M. Botvinick, A. Zisserman, O. Vinyals, and J. Carreira, “Perceiver io: A general architecture for structured inputs & outputs,” 2021

    work page 2021

  25. [25]

    Deep neural networks for youtube recommendations,

    P. Covington, J. Adams, and E. Sargin, “Deep neural networks for youtube recommendations,” inProceed- ings of the 10th ACM conference on recommender systems, 2016, pp. 191–198

    work page 2016

  26. [26]

    Self-attentive sequential recommendation,

    W.-C. Kang and J. McAuley, “Self-attentive sequential recommendation,” in2018 IEEE international conference on data mining (ICDM). IEEE, 2018, pp. 197–206

    work page 2018

  27. [27]

    Session-based Recommendations with Recurrent Neural Networks

    B. Hidasi, A. Karatzoglou, L. Baltrunas, and D. Tikk, “Session-based recommendations with recurrent neural networks,”arXiv preprint arXiv:1511.06939, 2015

    work page internal anchor Pith review Pith/arXiv arXiv 2015

  28. [28]

    Mmgcn: Multi-modal graph convolution network for personalized recommendation of micro-video,

    Y . Wei, X. Wang, L. Nie, X. He, R. Hong, and T.-S. Chua, “Mmgcn: Multi-modal graph convolution network for personalized recommendation of micro-video,” inProceedings of the 27th ACM international conference on multimedia, 2019, pp. 1437–1445

    work page 2019

  29. [29]

    Where to go next for recommender systems? id-vs. modality-based recommender models revisited,

    Z. Yuan, F. Yuan, Y . Song, Y . Li, J. Fu, F. Yang, Y . Pan, and Y . Ni, “Where to go next for recommender systems? id-vs. modality-based recommender models revisited,” inProceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval, 2023, pp. 2639–2649

    work page 2023

  30. [30]

    Content-based video recommendation system based on stylistic visual features,

    Y . Deldjoo, M. Elahi, P. Cremonesi, F. Garzotto, P. Piazzolla, and M. Quadrana, “Content-based video recommendation system based on stylistic visual features,”Journal on Data Semantics, vol. 5, no. 2, pp. 99–113, 2016

    work page 2016

  31. [31]

    Ups and downs: Modeling the visual evolution of fashion trends with one-class collaborative filtering,

    R. He and J. McAuley, “Ups and downs: Modeling the visual evolution of fashion trends with one-class collaborative filtering,” inproceedings of the 25th international conference on world wide web, 2016, pp. 507–517

    work page 2016

  32. [32]

    Kuairand: An unbiased sequential recommendation dataset with randomly exposed videos,

    C. Gao, S. Li, Y . Zhang, J. Chen, B. Li, W. Lei, P. Jiang, and X. He, “Kuairand: An unbiased sequential recommendation dataset with randomly exposed videos,” inProceedings of the 31st ACM International Conference on Information and Knowledge Management, ser. CIKM ’22, 2022, p. 3953–3957. [Online]. Available: https://doi.org/10.1145/3511808.3557624

    work page doi:10.1145/3511808.3557624 2022

  33. [33]

    Kuaisar: A unified search and recommendation dataset,

    Z. Sun, Z. Si, X. Zang, D. Leng, Y . Niu, Y . Song, X. Zhang, and J. Xu, “Kuaisar: A unified search and recommendation dataset,” 2023. [Online]. Available: https://doi.org/10.1145/3583780.3615123 11

    work page doi:10.1145/3583780.3615123 2023

  34. [34]

    Large-scale content-only video recommendation,

    J. Lee and S. Abu-El-Haija, “Large-scale content-only video recommendation,” inProceedings of the IEEE International Conference on Computer Vision Workshops, 2017, pp. 987–995

    work page 2017

  35. [35]

    Rethinking the inception architecture for computer vision,

    C. Szegedy, V . Vanhoucke, S. Ioffe, J. Shlens, and Z. Wojna, “Rethinking the inception architecture for computer vision,” inProceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 2818–2826

    work page 2016

  36. [36]

    Tiktok-10m: A large-scale short video dataset for video understanding,

    T. D. Company, “Tiktok-10m: A large-scale short video dataset for video understanding,” 2025, a dataset of 10 million TikTok posts for multimodal learning and social media analysis. [Online]. Available: https://huggingface.co/datasets/The-data-company/TikTok-10M

    work page 2025

  37. [37]

    Visual instruction tuning,

    H. Liu, C. Li, Q. Wu, and Y . J. Lee, “Visual instruction tuning,”Advances in neural information processing systems, vol. 36, pp. 34 892–34 916, 2023

    work page 2023

  38. [38]

    Flamingo: a visual language model for few-shot learning,

    J.-B. Alayrac, J. Donahue, P. Luc, A. Miech, I. Barr, Y . Hasson, K. Lenc, A. Mensch, K. Millican, M. Reynoldset al., “Flamingo: a visual language model for few-shot learning,”Advances in neural information processing systems, vol. 35, pp. 23 716–23 736, 2022

    work page 2022

  39. [39]

    Otter: A multi-modal model with in-context instruction tuning,

    B. Li, Y . Zhang, L. Chen, J. Wang, F. Pu, J. A. Cahyono, J. Yang, C. Li, and Z. Liu, “Otter: A multi-modal model with in-context instruction tuning,”IEEE Transactions on Pattern Analysis and Machine Intelligence, 2025

    work page 2025

  40. [40]

    Optimal transport for brain-image alignment: Unveiling redundancy and synergy in neural information processing,

    Y . Xiao, W. Lu, J. Ji, R. Ye, G. Li, X. Ma, and B. Hui, “Optimal transport for brain-image alignment: Unveiling redundancy and synergy in neural information processing,”arXiv preprint arXiv:2503.10663, 2025

    work page arXiv 2025

  41. [41]

    Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models,

    J. Li, D. Li, S. Savarese, and S. Hoi, “Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models,” inInternational conference on machine learning. PMLR, 2023, pp. 19 730–19 742

    work page 2023

  42. [42]

    Instructblip: To- wards general-purpose vision-language models with instruction tuning,

    W. Dai, J. Li, D. Li, A. Tiong, J. Zhao, W. Wang, B. Li, P. N. Fung, and S. Hoi, “Instructblip: To- wards general-purpose vision-language models with instruction tuning,”Advances in neural information processing systems, vol. 36, pp. 49 250–49 267, 2023

    work page 2023

  43. [43]

    Gaussian Error Linear Units (GELUs)

    D. Hendrycks, “Gaussian error linear units (gelus),”arXiv preprint arXiv:1606.08415, 2016

    work page internal anchor Pith review Pith/arXiv arXiv 2016

  44. [44]

    Revisiting 3d resnets for video recognition,

    X. Du, Y . Li, Y . Cui, R. Qian, J. Li, and I. Bello, “Revisiting 3d resnets for video recognition,”arXiv preprint arXiv:2109.01696, 2021. A Detailed Experiment Setups Implementation.All models are implemented in PyTorch. Unless otherwise stated, we use the AdamW optimizer with weight decay 1e−1. We set the random seed to 42 and report the result of a sin...

    work page arXiv 2021