REVIEW 10 cited by
AuroraCap: Efficient, Performant Video Detailed Captioning and a New Benchmark
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
read the original abstract
Video detailed captioning is a key task which aims to generate comprehensive and coherent textual descriptions of video content, benefiting both video understanding and generation. In this paper, we propose AuroraCap, a video captioner based on a large multimodal model. We follow the simplest architecture design without additional parameters for temporal modeling. To address the overhead caused by lengthy video sequences, we implement the token merging strategy, reducing the number of input visual tokens. Surprisingly, we found that this strategy results in little performance loss. AuroraCap shows superior performance on various video and image captioning benchmarks, for example, obtaining a CIDEr of 88.9 on Flickr30k, beating GPT-4V (55.3) and Gemini-1.5 Pro (82.2). However, existing video caption benchmarks only include simple descriptions, consisting of a few dozen words, which limits research in this field. Therefore, we develop VDC, a video detailed captioning benchmark with over one thousand carefully annotated structured captions. In addition, we propose a new LLM-assisted metric VDCscore for bettering evaluation, which adopts a divide-and-conquer strategy to transform long caption evaluation into multiple short question-answer pairs. With the help of human Elo ranking, our experiments show that this benchmark better correlates with human judgments of video detailed captioning quality.
Forward citations
Cited by 10 Pith papers
-
ARGUS: Hallucination and Omission Evaluation in Video-LLMs
ARGUS measures hallucination and omission in free-form video captions using LLM-based entailment and temporal alignment, finding that even the best video-LLM still produces roughly 40% hallucinated content.
-
Vid-SME: Membership Inference Attacks against Large Video Understanding Models
Vid-SME computes Sharma-Mittal entropy differences between natural and reversed video frame sequences to infer training membership in video understanding LLMs, but its effectiveness is confounded by member/non-member ...
-
Visual Token Compression Enhances Robustness of MLLMs
Pruning visual tokens farthest from the text feature space at selected 'robust' layers improves MLLM jailbreak defense (average +13.29% RAR) and slightly reduces hallucination.
-
AuroraLong: Bringing RNNs Back to Efficient Open-Ended Video Understanding
A 2B-parameter video-language model using an RWKV linear-RNN backbone and sorted token merging achieves competitive long-video QA accuracy with far lower memory cost than transformer-based models.
-
AVC-DPO: Aligned Video Captioning via Direct Preference Optimization
Using preference pairs synthesized from the model's own prompt-varied outputs, DPO fine-tuning improves Qwen2.5-VL-7B's video captioning on the VDC benchmark from 43.9 to 51.1 average VDCSCORE.
-
ToSA: Token Merging with Spatial Awareness
A training-free token merging method that adds depth-derived spatial similarity to ToMe's bipartite soft matching, improving VQA accuracy at high token reduction rates.
-
TUNA: Comprehensive Fine-grained Temporal Understanding Evaluation on Dense Dynamic Videos
TUNA introduces a 1,000-video benchmark with dense temporal captions and 1,432 multiple-choice questions, and finds that current video LMMs are weakest at camera motion, action sequences, and multi-subject scenes.
-
Modality Curation: Building Universal Embeddings for Advanced Multimodal Information Retrieval
UNITE combines curated multimodal training data and a modality-masked contrastive loss to achieve strong retrieval performance across text, image, and video tasks.
-
PercepCap: Video Captioner with Structured Spatio-Temporal Perception
Explicitly generating object trajectories and event timestamps before the final caption improves detailed video captioning on multiple benchmarks, though the improvement is largely driven by an external perception oracle.
-
Empowering Nanoscale Connectivity through Molecular Communication: A Case Study of Virus Infection
A position paper proposing molecular communication as the link layer for epidemic-control bio-nano networks, with an ORF3a-based mutation identification simulation; the provided manuscript body does not match this abstract.
Discussion (0). Sign in to comment.