Pith. sign in

REVIEW 4 major objections 5 minor 97 references

RAVEN: Query-Guided Representation Alignment for Question Answering over Audio, Video, Embedded Sensors, and Natural Language

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A query-conditioned gating module, QuART, aligns audio, video, and sensor tokens for multimodal QA.

desk verdict Substantial dataset and training recipe, but the core QuART equations don't type-check as written and the robustness results are in-distribution. read the letter →

arxiv 2505.17114 v3 pith:2WZAUXOH submitted 2025-05-21 cs.CL cs.CVcs.LGcs.MM

Subrata Biswas , Mohammad Nur Hossain Khan , Bashima Islam This is my paper
classification cs.CLcs.CVcs.LGcs.MM
keywords multimodalquestionansweringquery-conditionedgatingsensordataaudio-videofusionegocentricvideorepresentationalignmentlargelanguagemodelsdatasetcuration
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper presents RAVEN, a question-answering system that combines video, audio, and inertial sensor (IMU) streams with a large language model. The central claim is that conditioning token selection on the question itself—through a module called QuART—lets the model amplify informative signals and suppress distractors before fusion, and that this improves accuracy on seven QA benchmarks. The paper also introduces AVS-QA, a 300K-pair dataset of synchronized audio-video-sensor streams with automatically generated questions and answers, and reports that adding sensor data gives an additional 16.4% accuracy boost while improving robustness under modality corruption. If true, this is a practical recipe for building multimodal QA systems that stay reliable when some modalities are noisy or misleading.

What carries the argument

The core mechanism is QuART (Query-Aligned Representation of Tokens), a query-conditioned cross-modal gating module that assigns a scalar relevance score to each token in the concatenated video-audio-sensor token matrix. Given query embedding $z_q$ and token matrix $Z$, it computes attention output $M = \mathrm{softmax}(QK^\top/\sqrt{d_k})V$ and then a relevance projection $\alpha = \mathrm{softmax}(M W_R)$ with learned $W_R \in \mathbb{R}^{E\times L}$, aggregating tokens into a context vector $C = \sum_j \alpha_j Z_j$ that conditions the LLM decoder. The training pipeline adds an entropy regularizer $\sum_j \alpha_j \log \alpha_j$ to encourage sparse selection, and a three-stage schedule: unimodal pretraining, query-aligned joint training, and perturbation-aware fine-tuning.

What would settle it

Implement the published equations with the reported hyperparameters: $M \in \mathbb{R}^{L_q \times 448}$ and $W_R \in \mathbb{R}^{3584 \times 2968}$. Attempting to compute $\mathrm{softmax}(M W_R)$ will fail because the inner dimensions (448 and 3584) do not match. A working implementation must reshape or replace $W_R$, which would mean the paper's description does not correspond to the actual model.

Watch

Extended reading notes

Core claim

RAVEN's central claim is that query-conditioned token-level gating, implemented as QuART, outperforms raw attention and existing fusion methods for multimodal question answering. QuART computes a relevance score for every token across the video, audio, and sensor streams, conditioned on the query embedding, and uses these scores to form a context vector that feeds the language model. Trained through three stages—unimodal pretraining, query-aligned fusion joint-training, and disagreement-oriented fine-tuning with an entropy regularizer—RAVEN achieves what the paper reports as state-of-the-art accuracy on seven benchmarks, with gains up to 14.5% over VideoLLaVA and 8.0% over AVicuna. Incorporating sensor data adds a further 16.4% improvement, and under synthetic modality corruption the model retains a 50.23% advantage over prior systems. The paper further claims that the AVS-QA dataset is the first large-scale QA benchmark with synchronized audio, video, and sensor supervision.

Load-bearing premise

QuART's relevance scoring relies on the product $M W_R$ being computable; with the paper's stated dimensions ($M$ has 448 columns, $W_R$ is $3584 \times 2968$) that product is undefined, so the mechanism as written cannot be implemented without an unstated shape correction.

Editorial extensions

If this is right

  • Query-conditioned token gating can replace or augment raw attention in multimodal fusion, improving accuracy on egocentric and exocentric video QA.
  • Sensor streams such as IMU contribute information that is absent from audio and video, and adding them yields a measurable accuracy gain.
  • The three-stage training recipe—unimodal pretraining, query-aligned fusion, and disagreement-oriented fine-tuning—produces a model that degrades gracefully when modalities are corrupted.
  • The AVS-QA dataset provides a large-scale resource (300K pairs, four QA formats) for training and benchmarking audio-video-sensor QA systems.
  • Sparse token selection via entropy regularization may lower inference cost by filtering tokens early.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper's printed equations for QuART contain a shape inconsistency: $M$ has $d_k=448$ columns while $W_R\in\mathbb{R}^{3584\times 2968}$, so $M W_R$ is undefined. The actual implementation must differ from the text, meaning the reported gains may come from a different operation than the one described.
  • The sensor benefit is likely most pronounced when questions target physical actions or motion; on tasks where all relevant information is visible or audible, adding IMU streams may not help and could even add noise.
  • Because AVS-QA is filtered by LLM judges, the dataset may be biased toward question types that LLMs consider answerable, potentially overstating the advantage of query-conditioned gating.
  • The gating idea generalizes: the same query-conditioned token weighting could be applied to other modality combinations (e.g., tactile, GPS, or multi-camera views) or to long-form video with adaptive frame selection.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes RAVEN, a multimodal question-answering architecture that fuses video, audio, and embedded-sensor streams guided by a natural-language query. The central contribution is QuART, a query-conditioned token-gating module intended to assign scalar relevance scores to tokens across modalities before fusion. The authors introduce AVS-QA, a 300K-example dataset of synchronized audio, video, and sensor streams with automatically generated QA pairs, and train RAVEN in three stages: unimodal pretraining, query-aligned fusion, and disagreement-oriented fine-tuning. They report state-of-the-art results on several egocentric and exocentric benchmarks, a 16.4% gain from adding sensor data, and a 50.23% robustness improvement under modality corruption.

Significance. If the claims held, the paper would make a useful contribution: query-conditioned token gating with sensor inputs addresses a real gap in multimodal QA, and the released dataset and code would support follow-up work. The paper also contains useful ablations (modality encoders, frame sampling, fusion modules) and qualitative examples that illustrate the value of sensor signals. However, the central QuART equations do not type-check as written, and the headline robustness and AVS-QA results are evaluated largely on distributions the authors generated and tuned on. The significance therefore depends on whether the architectural specification can be corrected and whether the evaluation can be made convincingly out-of-distribution.

major comments (4)
  1. [Section 4, QuART; Table 9] The definitions M = softmax(QK^T/√d_k)V and α = softmax(M W_R) do not type-check as written. With Q ∈ R^{L_q×d_k} and V ∈ R^{L×d_k}, the attention output M has shape L_q×d_k, where d_k = 448 (Table 9). W_R is declared to be in R^{E×L} with E = 3584, so M W_R is undefined unless an additional multi-head concatenation and output projection is applied to M, which is never specified. The subsequent equation C = Σ_j α_j Z_j is also ambiguous: if α is an L_q×L matrix, the reduction over query tokens is not stated. Because Table 7 attributes QuART's gains over raw attention to this learned relevance head, the exact tensor shapes and reductions must be given.
  2. [Section 5 Stage III, Appendix D Algorithm 1, Table 5] The robustness claim is trained and evaluated on the same synthetic mismatch distribution. Stage III fine-tunes RAVEN on perturbed inputs generated by the procedure in Algorithm 1 (add noise, temporal reversal, modal replacement, sensor jitter), and Table 5 evaluates under the same kinds of perturbation. The reported 50.23% improvement over SOTA baselines is therefore in-distribution. Please evaluate on held-out perturbation types or on a separately collected corruption benchmark, or explicitly qualify the robustness claim as in-distribution.
  3. [Section 3, Tables 3 and 4, Appendix E.3] AVS-QA is generated by the authors' Actor–Evaluator–Critic LLM pipeline, and RAVEN is fine-tuned on AVS-QA during Stages II and III; the AVS-QA test split is held out from the same pipeline. This means the AVS-QA results in Tables 3 and 4 are in-distribution with respect to the dataset's generation bias. The 300-sample human evaluation in Appendix B.3 is too small to resolve this concern. Please report transfer to an external egocentric QA benchmark with synchronized multi-modal streams, or show that AVS-QA test performance correlates with external benchmarks for the same system.
  4. [Table 6, Figure 14, Appendix G] Key hyperparameters appear to be selected on the evaluation benchmarks. λ is chosen by comparing AVSD, MUSIC-QA, AVSSD, MSVD-QA, ActivityNet-QA, and AVS-QA scores in Table 6, and the LoRA rank is ablated on the same test benchmarks in Figure 14. No variance estimates or confidence intervals are reported anywhere. This makes it difficult to determine whether the reported margins over baselines are significant and whether the selected hyperparameters transfer. Please move hyperparameter selection to a validation split and report error bars or confidence intervals.
minor comments (5)
  1. [Table 3] The LLaVA-1.5 row appears to contain a formatting error: the EgoThink Avg column shows '34.7' instead of a value in [0,1], and the table layout is otherwise inconsistent with adjacent columns.
  2. [Appendix F.1] There are several typos in baseline names: 'ShareGPT4V' is misspelled as 'SahreGPT4V', and 'VALLEY' is written as 'V ALLEY'.
  3. [Appendix B.2 and Appendix G] 'Gemeni Pro' should be 'Gemini Pro', and 'audio pertaining' in Appendix G should be 'audio pretraining'.
  4. [Figure 3] The legend contains unrendered placeholder icons (described as '' and '') for trainable versus frozen components; these should be replaced with actual symbols or textual labels.
  5. [Abstract and Section 1] The phrase 'state-of-the-art' should be qualified: the comparisons are against a fixed set of open-source baselines, and several recent proprietary or larger models are not included, so 'state-of-the-art' overstates the scope of the comparison.

Circularity Check

2 steps flagged · score 6.0 of 10

Robustness and sensor-benefit claims are in-distribution by construction; the A/V fusion results on external benchmarks are independent.

  1. fitted input called prediction [Section 5 (Stage III), Appendix D (Algorithm 1), and Section 6.1 Table 5]
    "To improve robustness under real-world conditions, we fine-tune RAVEN using perturbed multi-modal inputs that simulate modality mismatch–such as dropped sensor packets or off-screen audio. We apply stochastic transformations independently to each modality: video undergoes frame jitter, dropout, or temporal inversion; audio is corrupted with Gaussian noise, reversed, or replaced with unrelated samples; sensor signals are perturbed with zero-centered Gaussian noise based on empirical variance (see Appendix D). ..."

    Stage III fine-tunes RAVEN on exactly the perturbation family defined in Algorithm 1 (add noise, reverse, replace with irrelevant samples, add jitter), and the same Appendix D procedure is the paper's stated mechanism for generating the cross-modal mismatch evaluation reported in Table 5 and summarized as a 50.23% robustness gain. The model is therefore fit to the synthetic mismatch distribution and then evaluated on that same distribution; the robustness number is an in-distribution result, not a prediction about unseen or naturally occurring modality corruption.

  2. self definitional [Section 3 (Evaluator: Modality-Consistency Filtering), Section 6.1 (Sensor-Aware Evaluation on AVS-QA), and Abstract]
    "Given a candidate QA pair (q, A) generated from meta-information M, the Evaluator verifies that the referenced modality or modalities are supported by the corresponding input triplet (v, a, s) in D. For instance, motion-related questions require significant activity in the sensor stream (e.g., variance spike), while visual or auditory references must align with detected objects or acoustic summaries. ... Incorporating sensor data provides an additional 16.4% boost."

    AVS-QA is curated so that sensor-grounded questions are deliberately retained and verified against sensor-stream activity, and the Actor prompt includes IMU statistics as a required input for generating QA pairs. The held-out AVS-QA test set therefore has a label distribution that encodes sensor relevance by construction. The reported 16.4% gain from adding sensor tokens (Table 4) is measured on this benchmark, so the 'sensor integration benefit' is substantially a property of the dataset curation rule rather than an independent empirical discovery.

full rationale

RAVEN's core cross-modal fusion results on external benchmarks (AVSD, MUSIC-QA, MSVD-QA, MSRVTT-QA, ActivityNet-QA, EgoThink) are not circular: those datasets are independent, baselines use official checkpoints, and the QuART-vs-raw-attention ablation is a genuine architecture comparison under the same training/evaluation protocol. The AVS-QA held-out split is session-disjoint per Appendix E.3, so training and testing on AVS-QA is not by itself leakage. The unresolved M-W_R shape mismatch in Section 4 (M has d_k=448 columns while W_R is in R^{E x L}, E=3584, making M W_R undefined as written) is a serious reproducibility defect but not a circularity. Two headline claims are, however, partly circular. First, the modality-corruption robustness number is obtained after Stage III fine-tunes on perturbations generated by the same Algorithm 1 that defines the corruption evaluation, making the 50.23% figure in-distribution. Second, the 16.4% sensor gain is measured only on AVS-QA, whose curation pipeline actively filters for sensor-grounded questions and feeds IMU statistics to the QA generator, so the benchmark's label distribution encodes the sensor benefit that the paper presents as an empirical result. These artifacts are partial: the A/V-only gains on external benchmarks and the QuART ablation retain independent content, so the paper is not wholly circular.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

The central claim rests on a small number of tuned hyperparameters, a dimension-compatibility assumption for the core module, and the assumption that synthetic perturbations and LLM-generated QA pairs are trustworthy. No new physical entities are introduced.

free parameters (3)
  • Entropy regularizer weight lambda = 0.001
    Chosen via ablation on the evaluation benchmarks themselves (Table 6) without a separate held-out tuning set; reported as final configuration.
  • LoRA rank = 256
    Selected via ablation on the same evaluation benchmarks (Figure 14); text states 256 while Table 9 lists 4256.
  • Critic rejection threshold = >=2 axes score <3 from at least 3 of 5 LLMs
    Dataset curation threshold calibrated on 300 human-annotated samples (Appendix B.3).
assumptions (3)
  • ad hoc to paper Matrix dimensions in the QuART relevance projection are compatible as written (M W_R is defined).
    Section 4 defines M as softmax(QK^T/sqrt(d_k)) V with d_k=448, and W_R in R^{E x L} with E=3584. The product is undefined unless the hidden dimension of M equals E, which is not stated and contradicts Table 9. The central mechanism depends on this.
  • domain assumption Synthetic cross-modal mismatch perturbations in Algorithm 1 are a valid proxy for real-world modality mismatch.
    Stage III fine-tuning uses these perturbations and the robustness evaluation (Table 5) uses the same algorithm, so gains may not transfer to natural mismatch.
  • domain assumption LLM-generated QA pairs filtered by critic LLMs are reliable ground truth.
    AVS-QA is fully auto-generated; only 300 samples were human checked (81% satisfied by two annotators). The benchmark's validity rests on this.

how reviews work

0 comments
Cite this review

Pith. "Pith review of RAVEN: Query-Guided Representation Alignment for Question Answering over Audio, Video, Embedded Sensors, and Natural Language." pith.science (2026). https://pith.science/paper/2WZAUXOH

@misc{pith2026250517114,
  author       = {Pith},
  title        = {Pith review of: RAVEN: Query-Guided Representation Alignment for Question Answering over Audio, Video, Embedded Sensors, and Natural Language},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/2WZAUXOH}},
  note         = {Machine review of arXiv:2505.17114}
}
read the original abstract

Multimodal question answering (QA) often requires identifying which video, audio, or sensor tokens are relevant to the question. Yet modality disagreements are common: off-camera speech, background noise, or motion outside the field of view often mislead fusion models that weight all streams equally. We present RAVEN, a unified QA architecture whose core is QuART, a query-conditioned cross-modal gating module that assigns scalar relevance scores to each token across modalities, enabling the model to amplify informative signals and suppress distractors before fusion. RAVEN is trained through a three-stage pipeline comprising unimodal pretraining, query-aligned fusion, and disagreement-oriented fine-tuning -- each stage targeting a distinct challenge in multi-modal reasoning: representation quality, cross-modal relevance, and robustness to modality mismatch. To support training and evaluation, we release AVS-QA, a dataset of 300K synchronized Audio--Video-Sensor streams paired with automatically generated question-answer pairs. Experimental results on seven multi-modal QA benchmarks -- including egocentric and exocentric tasks -- show that RAVEN achieves up to 14.5\% and 8.0\% gains in accuracy compared to state-of-the-art multi-modal large language models, respectively. Incorporating sensor data provides an additional 16.4\% boost, and the model remains robust under modality corruption, outperforming SOTA baselines by 50.23\%. Our code and dataset are available at https://github.com/BASHLab/RAVEN.

Figures

Figures reproduced from arXiv: 2505.17114 by the authors.

Figure 1
Figure 1. RAVEN jointly interprets video, audio, and sensor signals (e.g., inertial measurement unit or IMU) to answer fine-grained, context-aware questions. It outperforms existing MLLMs across six QA benchmarks, demonstrating robust generalization through multi-modal alignment. modalities that are critical in domains like AR/VR, robotics, and mobile health. Moreover, they of￾ten assume clean, synchronized inputs and rely on… view at source ↗
Figure 2
Figure 2. Overview of the AVS-QA dataset pipeline. Given synchronized audio–video–sensor input, the Actor generates metadata and QA pairs, the Evaluator filters weakly grounded examples, and the Critic ranks quality across five axes. The process is fully automated and yields 300K high-quality QA examples across four types. 2 Related Work Large and Multi-modal Language Mod￾els. Large language models (LLMs) such as LLaMA (Touvr… view at source ↗
Figure 3
Figure 3. Overview of RAVEN. Each modality (video, audio, sensor) is encoded using pretrained encoders and projected into a shared space. The QuART module performs query-conditioned token relevance scoring to align informative tokens across modalities. The figure also highlights the three-stage training pipeline for alignment-aware multi-modal reasoning. Here, and represent trainable and frozen components, respectively. {v, a… view at source ↗
Figures from the paper (13 more)
Figure 4
Figure 4. Figure 4: Distribution of question types in AVS-QA. The dataset includes a diverse mix of open-ended, close￾ended, true/false, multiple choice, and other formats, supporting comprehensive evaluation settings. 0 10 20 30 40 Number of Words in a Question 0 10000 20000 30000 40000 …
Figure 5
Figure 5. Figure 5: Length of questions has some variation due to [PITH_FULL_IMAGE:figures/full_fig_p015_5.png]
Figure 6
Figure 6. Figure 6: True/false and multiple choice questions often [PITH_FULL_IMAGE:figures/full_fig_p016_6.png]
Figure 7
Figure 7. Figure 7: System prompt used for generating ques￾tions and answers in Actor phase. Please generate two question answers of each type of open-ended, close-ended, multiple choice and True-False. Generate five answers for each open-ended question and single answer for other type of…
Figure 11
Figure 11. Figure 11: System prompt used for generating ques￾tions and answers in Critic phase. Please rate the quality of questions and answers considering the relevance, correctness, clarity, and depth based on the following five information: Entire Scene Narration: {} Objects Present: {…
Figure 12
Figure 12. Figure 12: User prompt used for generating questions and answers in Critic phase. human-in-the-loop authoring. C Additional Model Architecture Details C.1 LIMU-BERT Pre-Training As our sensor encoder, we employ LIMU￾BERT (Xu et al., 2021), a multi-head attention￾based encoder-de…
Figure 13
Figure 13. Figure 13: System and user prompt used to evaluate the [PITH_FULL_IMAGE:figures/full_fig_p023_13.png]
Figure 14
Figure 14. Figure 14: Impact of LoRA rank on QA accuracy across five benchmarks. Accuracy improves steadily with higher ranks, saturating near 256, indicating that moderate-rank adapters suffice for effective multimodal alignment and reasoning. G Ablation Study Effect of Modality Encoder. …
Figure 15
Figure 15. Figure 15: Example illustrating the value of sensor input for activity disambiguation. Given the question [PITH_FULL_IMAGE:figures/full_fig_p025_15.png]
Figure 16
Figure 16. Figure 16: Example illustrating subtle activity disambiguation using multimodal reasoning. Given the question [PITH_FULL_IMAGE:figures/full_fig_p025_16.png]
Figure 17
Figure 17. Figure 17: Example demonstrating the added value of sensor data in identifying subtle concurrent actions. Given [PITH_FULL_IMAGE:figures/full_fig_p026_17.png]
Figure 18
Figure 18. Figure 18: Example showcasing multimodal reasoning for fine-grained activity understanding. Given the question [PITH_FULL_IMAGE:figures/full_fig_p026_18.png]
Figure 19
Figure 19. Figure 19: Example illustrating confirmatory reasoning across modalities. Given the question [PITH_FULL_IMAGE:figures/full_fig_p027_19.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

97 extracted references · 26 canonical work pages

  1. [1]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Marah Abdin, Jyoti Aneja, Hany Awadalla, Ahmed Awadallah, Ammar Ahmad Awan, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Jianmin Bao, Harkirat Behl, and 1 others. 2024. Phi-3 technical report: A highly capable language model locally on your phone. arXiv preprint arXiv:2404.14219

    arXiv 2024

  4. [4]

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, and 1 others. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774

    arXiv 2023

  5. [5]

    Huda Alamri, Vincent Cartillier, Abhishek Das, Jue Wang, Anoop Cherian, Irfan Essa, Dhruv Batra, Tim K Marks, Chiori Hori, Peter Anderson, and 1 others. 2019. Audio visual scene-aware dialog. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7558--7567

    2019

  6. [6]

    Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, and 1 others. 2022. Flamingo: a visual language model for few-shot learning. Advances in neural information processing systems, 35:23716--23736

    2022

  7. [7]

    Anas Awadalla, Irena Gao, Josh Gardner, Jack Hessel, Yusuf Hanafy, Wanrong Zhu, Kalyani Marathe, Yonatan Bitton, Samir Gadre, Shiori Sagawa, and 1 others. 2023. Openflamingo: An open-source framework for training large autoregressive vision-language models. arXiv preprint arXiv:2308.01390

    arXiv 2023

  8. [8]

    Shehreen Azad, Vibhav Vineet, and Yogesh Singh Rawat. 2025. Hierarq: Task-aware hierarchical q-former for enhanced video understanding. arXiv preprint arXiv:2503.08585

    arXiv 2025

Show all 97 references
  1. [9]

    Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, and 1 others. 2023. Qwen technical report. arXiv preprint arXiv:2309.16609

    2023 arXiv

  2. [10]

    Max Bain, Arsha Nagrani, G \"u l Varol, and Andrew Zisserman. 2021. Frozen in time: A joint video and image encoder for end-to-end retrieval. In Proceedings of the IEEE/CVF international conference on computer vision, pages 1728--1738

    2021

  3. [11]

    Hangbo Bao, Wenhui Wang, Li Dong, Qiang Liu, Owais Khan Mohammed, Kriti Aggarwal, Subhojit Som, Songhao Piao, and Furu Wei. 2022. Vlmo: Unified vision-language pre-training with mixture-of-modality-experts. Advances in Neural Information Processing Systems, 35:32897--32912

    2022

  4. [12]

    Soravit Changpinyo, Piyush Sharma, Nan Ding, and Radu Soricut. 2021. Conceptual 12m: Pushing web-scale image-text pre-training to recognize long-tail visual concepts. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 3558--3568

    2021

  5. [13]

    Honglie Chen, Weidi Xie, Andrea Vedaldi, and Andrew Zisserman. 2020. Vggsound: A large-scale audio-visual dataset. In ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 721--725. IEEE

    2020

  6. [14]

    Lin Chen, Jinsong Li, Xiaoyi Dong, Pan Zhang, Conghui He, Jiaqi Wang, Feng Zhao, and Dahua Lin. 2024 a . Sharegpt4v: Improving large multi-modal models with better captions. In European Conference on Computer Vision, pages 370--387. Springer

    2024

  7. [15]

    Sanyuan Chen, Yu Wu, Chengyi Wang, Shujie Liu, Daniel Tompkins, Zhuo Chen, and Furu Wei. 2022. Beats: Audio pre-training with acoustic tokenizers. arXiv preprint arXiv:2212.09058

    2022 arXiv

  8. [16]

    Tsai-Shien Chen, Aliaksandr Siarohin, Willi Menapace, Ekaterina Deyneka, Hsiang-wei Chao, Byung Eun Jeon, Yuwei Fang, Hsin-Ying Lee, Jian Ren, Ming-Hsuan Yang, and 1 others. 2024 b . Panda-70m: Captioning 70m videos with multiple cross-modality teachers. In Proceedings of the ...

    2024

  9. [17]

    Wenqiang Chen, Jiaxuan Cheng, Leyao Wang, Wei Zhao, and Wojciech Matusik. 2024 c . https://doi.org/10.1145/3699747 Sensor2text: Enabling natural language interactions for daily activity tracking using wearable sensors . Proc. ACM Interact. Mob. Wearable Ubiquitous Technol., 8(4)

    2024 doi

  10. [18]

    Sijie Cheng, Zhicheng Guo, Jingwen Wu, Kechen Fang, Peng Li, Huaping Liu, and Yang Liu. 2024 a . Egothink: Evaluating first-person perspective thinking capability of vision-language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, p...

    2024

  11. [19]

    Zesen Cheng, Sicong Leng, Hang Zhang, Yifei Xin, Xin Li, Guanzheng Chen, Yongxin Zhu, Wenqi Zhang, Ziyang Luo, Deli Zhao, and 1 others. 2024 b . Videollama 2: Advancing spatial-temporal modeling and audio understanding in video-llms. arXiv preprint arXiv:2406.07476

    2024 arXiv

  12. [20]

    Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, and 1 others. 2023. Palm: Scaling language modeling with pathways. Journal of Machine Learning Research, 24(240):1--113

    2023

  13. [21]

    Yunfei Chu, Jin Xu, Qian Yang, Haojie Wei, Xipin Wei, Zhifang Guo, Yichong Leng, Yuanjun Lv, Jinzheng He, Junyang Lin, and 1 others. 2024. Qwen2-audio technical report. arXiv preprint arXiv:2407.10759

    2024 arXiv

  14. [22]

    Yunfei Chu, Jin Xu, Xiaohuan Zhou, Qian Yang, Shiliang Zhang, Zhijie Yan, Chang Zhou, and Jingren Zhou. 2023. Qwen-audio: Advancing universal audio understanding via unified large-scale audio-language models. arXiv preprint arXiv:2311.07919

    2023 arXiv

  15. [23]

    Justin Cosentino, Anastasiya Belyaeva, Xin Liu, Nicholas A Furlotte, Zhun Yang, Chace Lee, Erik Schenck, Yojan Patel, Jian Cui, Logan Douglas Schneider, and 1 others. 2024. Towards a personal health large language model. arXiv preprint arXiv:2406.06474

    2024 arXiv

  16. [24]

    Dima Damen, Hazel Doughty, Giovanni Maria Farinella, Sanja Fidler, Antonino Furnari, Evangelos Kazakos, Davide Moltisanti, Jonathan Munro, Toby Perrett, Will Price, and Michael Wray. 2018. Scaling egocentric vision: The epic-kitchens dataset. In European Conference on Computer...

    2018

  17. [25]

    Benjamin Elizalde, Soham Deshmukh, Mahmoud Al Ismail, and Huaming Wang. 2023. Clap learning audio concepts from natural language supervision. In ICASSP 2023-2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1--5. IEEE

    2023

  18. [26]

    Jinlan Fu, See-Kiong Ng, Zhengbao Jiang, and Pengfei Liu. 2023. Gptscore: Evaluate as you desire. arXiv preprint arXiv:2302.04166

    2023 arXiv

  19. [27]

    Rohit Girdhar, Alaaeldin El-Nouby, Zhuang Liu, Mannat Singh, Kalyan Vasudev Alwala, Armand Joulin, and Ishan Misra. 2023. Imagebind: One embedding space to bind them all. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 15180--15190

    2023

  20. [28]

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, and 1 others. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783

    2024 arXiv

  21. [29]

    Kristen Grauman, Andrew Westbury, Eugene Byrne, Zachary Chavis, Antonino Furnari, Rohit Girdhar, Jackson Hamburger, Hao Jiang, Miao Liu, Xingyu Liu, and 1 others. 2022. Ego4d: Around the world in 3,000 hours of egocentric video. In Proceedings of the IEEE/CVF conference on com...

    2022

  22. [30]

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, and 1 others. 2022. Lora: Low-rank adaptation of large language models. ICLR, 1(2):3

    2022

  23. [31]

    Bin Huang, Xin Wang, Hong Chen, Zihan Song, and Wenwu Zhu. 2024. Vtimellm: Empower llm to grasp video moments. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14271--14280

    2024

  24. [32]

    Sheikh Asif Imran, Mohammad Nur Hossain Khan, Subrata Biswas, and Bashima Islam. 2024. Llasa: A multimodal llm for human activity analysis through wearable and smartphone sensors. arXiv preprint arXiv:2406.14498

    2024

  25. [33]

    Baoxiong Jia, Ting Lei, Song-Chun Zhu, and Siyuan Huang. 2022. Egotaskqa: Understanding human tasks in egocentric videos. Advances in Neural Information Processing Systems, 35:3343--3360

    2022

  26. [34]

    Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, and 1 others. 2024. Mixtral of experts. arXiv preprint arXiv:2401.04088

    2024 arXiv

  27. [35]

    Rahima Khanam and Muhammad Hussain. 2024. Yolov11: An overview of the key architectural enhancements. arXiv preprint arXiv:2410.17725

    2024 arXiv

  28. [36]

    Chris Dongjoo Kim, Byeongchang Kim, Hyunmin Lee, and Gunhee Kim. 2019. Audiocaps: Generating captions for audios in the wild. In NAACL-HLT

    2019

  29. [37]

    Zhifeng Kong, Arushi Goel, Rohan Badlani, Wei Ping, Rafael Valle, and Bryan Catanzaro. 2024. Audio flamingo: A novel audio language model with few-shot learning and dialogue abilities. arXiv preprint arXiv:2402.01831

    2024 arXiv

  30. [38]

    Alexandre Lacoste, Sasha Luccioni, Victor Schmidt, and Thomas Dandres. 2019. Quantifying the carbon emissions of machine learning. arXiv preprint arXiv:1910.09700

    2019 arXiv

  31. [39]

    Zikang Leng, Amitrajit Bhattacharjee, Hrudhai Rajasekhar, Lizhe Zhang, Elizabeth Bruda, Hyeokhyen Kwon, and Thomas Pl \"o tz. 2024. Imugpt 2.0: Language-based cross modality transfer for sensor-based human activity recognition. Proceedings of the ACM on Interactive, Mobile, We...

    2024

  32. [40]

    Bohao Li, Rui Wang, Guangzhi Wang, Yuying Ge, Yixiao Ge, and Ying Shan. 2023 a . Seed-bench: Benchmarking multimodal llms with generative comprehension. arXiv preprint arXiv:2307.16125

    2023 arXiv

  33. [41]

    Guangyao Li, Yake Wei, Yapeng Tian, Chenliang Xu, Ji-Rong Wen, and Di Hu. 2022. Learning to answer questions in dynamic audio-visual scenarios. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 19108--19118

    2022

  34. [42]

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. 2023 b . Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In International conference on machine learning, pages 19730--19742. PMLR

    2023

  35. [43]

    Junnan Li, Ramprasaath Selvaraju, Akhilesh Gotmare, Shafiq Joty, Caiming Xiong, and Steven Chu Hong Hoi. 2021. Align before fuse: Vision and language representation learning with momentum distillation. Advances in neural information processing systems, 34:9694--9705

    2021

  36. [44]

    KunChang Li, Yinan He, Yi Wang, Yizhuo Li, Wenhai Wang, Ping Luo, Yali Wang, Limin Wang, and Yu Qiao. 2023 c . Videochat: Chat-centric video understanding. arXiv preprint arXiv:2305.06355

    2023 arXiv

  37. [45]

    Bin Lin, Yang Ye, Bin Zhu, Jiaxi Cui, Munan Ning, Peng Jin, and Li Yuan. 2023 a . Video-llava: Learning united visual representation by alignment before projection. arXiv preprint arXiv:2311.10122

    2023 arXiv

  38. [46]

    Yuanze Lin, Chen Wei, Huiyu Wang, Alan Yuille, and Cihang Xie. 2023 b . Smaug: Sparse masked autoencoder for efficient video-language pre-training. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 2459--2469

    2023

  39. [47]

    Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. 2024 a . Improved baselines with visual instruction tuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 26296--26306

    2024

  40. [48]

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2023 a . Visual instruction tuning. Advances in neural information processing systems, 36:34892--34916

    2023

  41. [49]

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2023 b . Visual instruction tuning. Advances in neural information processing systems, 36:34892--34916

    2023

  42. [50]

    Yuan Liu, Haodong Duan, Yuanhan Zhang, Bo Li, Songyang Zhang, Wangbo Zhao, Yike Yuan, Jiaqi Wang, Conghui He, Ziwei Liu, and 1 others. 2024 b . Mmbench: Is your multi-modal model an all-around player? In European conference on computer vision, pages 216--233. Springer

    2024

  43. [51]

    Ruipu Luo, Ziwang Zhao, Min Yang, Junwei Dong, Da Li, Pengcheng Lu, Tao Wang, Linmei Hu, Minghui Qiu, and Zhongyu Wei. 2023. Valley: Video assistant with large language model enhanced ability. arXiv preprint arXiv:2306.07207

    2023 arXiv

  44. [52]

    Chenyang Lyu, Minghao Wu, Longyue Wang, Xinting Huang, Bingshuai Liu, Zefeng Du, Shuming Shi, and Zhaopeng Tu. 2023. Macaw-llm: Multi-modal language modeling with image, audio, video, and text integration. arXiv preprint arXiv:2306.09093

    2023 arXiv

  45. [53]

    Muhammad Maaz, Hanoona Rasheed, Salman Khan, and Fahad Shahbaz Khan. 2023. Video-chatgpt: Towards detailed video understanding via large vision and language models. arXiv preprint arXiv:2306.05424

    2023 arXiv

  46. [54]

    Mohammad Malekzadeh, Richard G Clegg, Andrea Cavallaro, and Hamed Haddadi. 2019. Mobile sensor data anonymization. In Proceedings of the international conference on internet of things design and implementation, pages 49--58

    2019

  47. [55]

    Xinhao Mei, Chutong Meng, Haohe Liu, Qiuqiang Kong, Tom Ko, Chengqi Zhao, Mark D Plumbley, Yuexian Zou, and Wenwu Wang. 2024. Wavcaps: A chatgpt-assisted weakly-labelled audio captioning dataset for audio-language multimodal research. IEEE/ACM Transactions on Audio, Speech, an...

    2024

  48. [56]

    Mike A Merrill, Akshay Paruchuri, Naghmeh Rezaei, Geza Kovacs, Javier Perez, Yun Liu, Erik Schenck, Nova Hammerquist, Jake Sunshine, Shyam Tailor, and 1 others. 2024. Transforming wearable data into health insights using large language model agents. arXiv preprint arXiv:2406.06464

    2024 arXiv

  49. [57]

    Arsha Nagrani, Paul Hongsuck Seo, Bryan Seybold, Anja Hauth, Santiago Manen, Chen Sun, and Cordelia Schmid. 2022. Learning audio-video modalities from image captions. In European Conference on Computer Vision, pages 407--426. Springer

    2022

  50. [58]

    Arsha Nagrani, Shan Yang, Anurag Arnab, Aren Jansen, Cordelia Schmid, and Chen Sun. 2021. Attention bottlenecks for multimodal fusion. Advances in neural information processing systems, 34:14200--14213

    2021

  51. [59]

    Xiaomin Ouyang and Mani Srivastava. 2024. Llmsense: Harnessing llms for high-level reasoning over spatiotemporal sensor traces. In 2024 IEEE 3rd Workshop on Machine Learning on Edge in Sensor Systems (SenSys-ML), pages 9--14. IEEE

    2024

  52. [60]

    Daniel Povey, Arnab Ghoshal, Gilles Boulianne, Lukas Burget, Ondrej Glembek, Nagendra Goel, Mirko Hannemann, Petr Motlicek, Yanmin Qian, Petr Schwarz, and 1 others. 2011. The kaldi speech recognition toolkit. In IEEE 2011 workshop on automatic speech recognition and understand...

    2011

  53. [61]

    Shraman Pramanick, Yale Song, Sayan Nag, Kevin Qinghong Lin, Hardik Shah, Mike Zheng Shou, Rama Chellappa, and Pengchuan Zhang. 2023. Egovlpv2: Egocentric video-language pre-training with fusion in the backbone. In Proceedings of the IEEE/CVF International Conference on Comput...

    2023

  54. [62]

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, and 1 others. 2021. Learning transferable visual models from natural language supervision. In International conference on machine...

    2021

  55. [63]

    Ilija Radosavovic, Raj Prateek Kosaraju, Ross Girshick, Kaiming He, and Piotr Doll \'a r. 2020. Designing network design spaces. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 10428--10436

    2020

  56. [64]

    Jorge-L Reyes-Ortiz, Luca Oneto, Albert Sam \`a , Xavier Parra, and Davide Anguita. 2016. Transition-aware human activity recognition using smartphones. Neurocomputing, 171:754--767

    2016

  57. [65]

    o rster, Gerhard Tr \

    Daniel Roggen, Alberto Calatroni, Mirco Rossi, Thomas Holleczek, Kilian F \"o rster, Gerhard Tr \"o ster, Paul Lukowicz, David Bannach, Gerald Pirkl, Alois Ferscha, and 1 others. 2010. Collecting complex activity datasets in highly rich networked sensor environments. In 2010 S...

    2010

  58. [66]

    Muhammad Shoaib, Stephan Bosch, Ozlem Durmaz Incel, Hans Scholten, and Paul JM Havinga. 2014. Fusion of smartphone motion sensors for physical activity recognition. Sensors, 14(6):10146--10176

    2014

  59. [67]

    Fangxun Shu, Lei Zhang, Hao Jiang, and Cihang Xie. 2023. Audio-visual llm for video understanding. arXiv preprint arXiv:2312.06720

    2023 arXiv

  60. [68]

    Allan Stisen, Henrik Blunck, Sourav Bhattacharya, Thor Siiger Prentow, Mikkel Baun Kj rgaard, Anind Dey, Tobias Sonne, and Mads M ller Jensen. 2015. Smart devices are different: Assessing and mitigatingmobile sensing heterogeneities for activity recognition. In Proceedings of ...

    2015

  61. [69]

    Yixuan Su, Tian Lan, Huayang Li, Jialu Xu, Yan Wang, and Deng Cai. 2023. Pandagpt: One model to instruction-follow them all. arXiv preprint arXiv:2305.16355

    2023 arXiv

  62. [70]

    Yunlong Tang, Daiki Shimada, Jing Bi, Mingqian Feng, Hang Hua, and Chenliang Xu. 2024. Empowering llms with pseudo-untrimmed videos for audio-visual temporal understanding. arXiv preprint arXiv:2403.16276

    2024

  63. [71]

    Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, and 1 others. 2023. Gemini: a family of highly capable multimodal models. arXiv preprint arXiv:2312.11805

    2023 arXiv

  64. [72]

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timoth \'e e Lacroix, Baptiste Rozi \`e re, Naman Goyal, Eric Hambro, Faisal Azhar, and 1 others. 2023. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971

    2023 arXiv

  65. [73]

    Jack Urbanek, Florian Bordes, Pietro Astolfi, Mary Williamson, Vasu Sharma, and Adriana Romero-Soriano. 2024. A picture is worth more than 77 text tokens: Evaluating clip-style models on dense captions. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern R...

    2024

  66. [74]

    Junke Wang, Dongdong Chen, Chong Luo, Bo He, Lu Yuan, Zuxuan Wu, and Yu-Gang Jiang. 2024. Omnivid: A generative framework for universal video understanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18209--18220

    2024

  67. [75]

    Yi Wang, Yinan He, Yizhuo Li, Kunchang Li, Jiashuo Yu, Xin Ma, Xinhao Li, Guo Chen, Xinyuan Chen, Yaohui Wang, and 1 others. 2023. Internvid: A large-scale video-text dataset for multimodal understanding and generation. arXiv preprint arXiv:2307.06942

    2023 arXiv

  68. [76]

    Dhanuja Wanniarachchi and Archan Misra. 2025. Mimic: Ai and ar-enhanced multi-modal, immersive, relative instruction comprehension. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, 9(1):1--34

    2025

  69. [77]

    Shengqiong Wu, Hao Fei, Leigang Qu, Wei Ji, and Tat-Seng Chua. 2024. Next-gpt: Any-to-any multimodal llm. In Forty-first International Conference on Machine Learning

    2024

  70. [78]

    Huatao Xu, Liying Han, Qirui Yang, Mo Li, and Mani Srivastava. 2024 a . Penetrative ai: Making llms comprehend the physical world. In Proceedings of the 25th International Workshop on Mobile Computing Systems and Applications, pages 1--7

    2024

  71. [79]

    Huatao Xu, Pengfei Zhou, Rui Tan, Mo Li, and Guobin Shen. 2021. Limu-bert: Unleashing the potential of unlabeled data for imu sensing applications. In Proceedings of the 19th ACM Conference on Embedded Networked Sensor Systems, pages 220--233

    2021

  72. [80]

    Jun Xu, Tao Mei, Ting Yao, and Yong Rui. 2016. Msr-vtt: A large video description dataset for bridging video and language. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5288--5296

    2016

  73. [81]

    Xuhai Xu, Bingsheng Yao, Yuanzhe Dong, Saadia Gabriel, Hong Yu, James Hendler, Marzyeh Ghassemi, Anind K Dey, and Dakuo Wang. 2024 b . Mental-llm: Leveraging large language models for mental health prediction via online text data. Proceedings of the ACM on Interactive, Mobile,...

    2024

  74. [82]

    An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, and 1 others. 2024. Qwen2. 5 technical report. arXiv preprint arXiv:2412.15115

    2024 arXiv

  75. [83]

    Pinci Yang, Xin Wang, Xuguang Duan, Hong Chen, Runze Hou, Cong Jin, and Wenwu Zhu. 2022. Avqa: A dataset for audio-visual question answering on videos. In Proceedings of the 30th ACM international conference on multimedia, pages 3480--3491

    2022

  76. [84]

    Lewei Yao, Runhui Huang, Lu Hou, Guansong Lu, Minzhe Niu, Hang Xu, Xiaodan Liang, Zhenguo Li, Xin Jiang, and Chunjing Xu. 2021. Filip: Fine-grained interactive language-image pre-training. arXiv preprint arXiv:2111.07783

    2021 arXiv

  77. [85]

    Hanrong Ye, De-An Huang, Yao Lu, Zhiding Yu, Wei Ping, Andrew Tao, Jan Kautz, Song Han, Dan Xu, Pavlo Molchanov, and 1 others. 2024. X-vila: Cross-modality alignment for large language model. arXiv preprint arXiv:2405.19335

    2024 arXiv

  78. [86]

    Qinghao Ye, Guohai Xu, Ming Yan, Haiyang Xu, Qi Qian, Ji Zhang, and Fei Huang. 2023. Hitea: Hierarchical temporal-aware video-language pre-training. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 15405--15416

    2023

  79. [87]

    Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung, Mojtaba Seyedhosseini, and Yonghui Wu. 2022. Coca: Contrastive captioners are image-text foundation models. arXiv preprint arXiv:2205.01917

    2022 arXiv

  80. [88]

    Weihao Yu, Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Zicheng Liu, Xinchao Wang, and Lijuan Wang. 2023. Mm-vet: Evaluating large multimodal models for integrated capabilities. arXiv preprint arXiv:2308.02490

    2023 arXiv

  81. [89]

    Zhou Yu, Dejing Xu, Jun Yu, Ting Yu, Zhou Zhao, Yueting Zhuang, and Dacheng Tao. 2019. Activitynet-qa: A dataset for understanding complex web videos via question answering. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 9127--9134

    2019

  82. [90]

    Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. 2023. Sigmoid loss for language image pre-training. In Proceedings of the IEEE/CVF international conference on computer vision, pages 11975--11986

    2023

  83. [91]

    Hang Zhang, Xin Li, and Lidong Bing. 2023 a . Video-llama: An instruction-tuned audio-visual language model for video understanding. arXiv preprint arXiv:2306.02858

    2023 arXiv

  84. [92]

    Renrui Zhang, Jiaming Han, Chris Liu, Peng Gao, Aojun Zhou, Xiangfei Hu, Shilin Yan, Pan Lu, Hongsheng Li, and Yu Qiao. 2023 b . Llama-adapter: Efficient fine-tuning of language models with zero-init attention. arXiv preprint arXiv:2303.16199

    2023 arXiv

  85. [93]

    Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, and 1 others. 2022. Opt: Open pre-trained transformer language models. arXiv preprint arXiv:2205.01068

    2022 arXiv

  86. [94]

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, and 1 others. 2023 a . Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36:46595--46623

    2023

  87. [95]

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, and 1 others. 2023 b . Judging llm-as-a-judge with mt-bench and chatbot arena. Advances in Neural Information Processing Systems, 36:46595--46623

    2023

  88. [96]

    Bin Zhu, Bin Lin, Munan Ning, Yang Yan, Jiaxi Cui, HongFa Wang, Yatian Pang, Wenhao Jiang, Junwu Zhang, Zongwei Li, and 1 others. 2023 a . Languagebind: Extending video-language pretraining to n-modality by language-based semantic alignment. arXiv preprint arXiv:2310.01852

    2023 arXiv

  89. [97]

    Deyao Zhu, Jun Chen, Xiaoqian Shen, Xiang Li, and Mohamed Elhoseiny. 2023 b . Minigpt-4: Enhancing vision-language understanding with advanced large language models. arXiv preprint arXiv:2304.10592

    2023 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.