Pith. sign in

REVIEW 12 cited by

Tarsier2: Advancing Large Vision-Language Models from Detailed Video Description to Comprehensive Video Understanding

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2501.07888 v3 pith:WDZOQVVN submitted 2025-01-14 cs.CV cs.AI

Liping Yuan , Jiawei Wang , Haomiao Sun , Yuchen Zhang , Yuan Lin This is my paper · ORCID
classification cs.CVcs.AI
keywords videotarsier2-7bdetailedgpt-4otarsier2vision-languagedatadescription
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

We introduce Tarsier2, a state-of-the-art large vision-language model (LVLM) designed for generating detailed and accurate video descriptions, while also exhibiting superior general video understanding capabilities. Tarsier2 achieves significant advancements through three key upgrades: (1) Scaling pre-training data from 11M to 40M video-text pairs, enriching both volume and diversity; (2) Performing fine-grained temporal alignment during supervised fine-tuning; (3) Using model-based sampling to automatically construct preference data and applying DPO training for optimization. Extensive experiments show that Tarsier2-7B consistently outperforms leading proprietary models, including GPT-4o and Gemini 1.5 Pro, in detailed video description tasks. On the DREAM-1K benchmark, Tarsier2-7B improves F1 by 2.8% over GPT-4o and 5.8% over Gemini-1.5-Pro. In human side-by-side evaluations, Tarsier2-7B shows a +8.6% performance advantage over GPT-4o and +24.9% over Gemini-1.5-Pro. Tarsier2-7B also sets new state-of-the-art results across 15 public benchmarks, spanning tasks such as video question-answering, video grounding, hallucination test, and embodied question-answering, demonstrating its versatility as a robust generalist vision-language model.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 12 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. VideoChat3: Fully Open Video MLLM for Efficient and Generalist Video Understanding

    cs.CV 2026-07 conditional novelty 6.0 of 10

    An open 4B video MLLM with inflated-3D ViT tokenization and adaptive streaming perception outperforms comparable open models on general, long-video, and streaming benchmarks while using fewer visual tokens.

  2. LookME: Lookup-Based Multimodal Embeddings for Layer Injection in Vision-Language Models

    cs.CV 2026-07 conditional novelty 6.0 of 10

    LookME extends per-layer embedding lookup from text to continuous image and cross-modal embeddings and reports improved VLM scores on most benchmarks.

  3. Syn4D: A Multiview Synthetic 4D Dataset

    cs.CV 2026-05 unverdicted novelty 6.0 of 10

    Syn4D supplies multiview synthetic dynamic scenes with dense geometric, tracking and pose ground truth that lets any pixel be unprojected to any time and camera.

  4. Kwai Keye-VL 1.5 Technical Report

    cs.CV 2025-09 conditional novelty 6.0 of 10

    Keye-VL-1.5 combines similarity-based Slow-Fast video token allocation with progressive context extension and iterative RL, reporting leading video-understanding results among 8B-scale multimodal models.

  5. Kwai Keye-VL Technical Report

    cs.CV 2025-07 conditional novelty 6.0 of 10

    Kwai Keye-VL shows that a five-mode chain-of-thought cold-start plus mix-mode reinforcement learning can push an 8B multimodal model to strong short-video and general vision-language performance.

  6. How Important are Videos for Training Video LLMs?

    cs.CV 2025-06 conditional novelty 6.0 of 10

    Training video LLMs on ordered static images with temporal questions nearly matches real video training on TVBench, suggesting video data is underused.

  7. VideoCap-R1: Enhancing MLLMs for Video Captioning via Structured Thinking

    cs.CV 2025-06 conditional novelty 6.0 of 10

    GRPO post-training with structured thinking and dual think/caption rewards improves Qwen2-VL-7B video captioning over the base model and SFT on DREAM-1K, VDC, and CAREBENCH.

  8. Zooming from Context to Cue: Hierarchical Preference Optimization for Multi-Image MLLMs

    cs.CV 2025-05 conditional novelty 6.0 of 10

    Context-to-Cue Direct Preference Optimization (CcDPO) reduces multi-image hallucinations in 7B multimodal LLMs by training on perturbed full-sequence captions and region-focused visual prompts, improving average multi...

  9. Modality Curation: Building Universal Embeddings for Advanced Multimodal Information Retrieval

    cs.CV 2025-05 conditional novelty 6.0 of 10

    UNITE combines curated multimodal training data and a modality-masked contrastive loss to achieve strong retrieval performance across text, image, and video tasks.

  10. JoVA: Unified Multimodal Learning for Joint Video-Audio Generation and Editing

    cs.CV 2025-12 conditional novelty 5.0 of 10

    A dual-branch diffusion transformer with joint video-audio self-attention and a keypoint-based mouth-area loss reports top lip-sync and speech metrics on two benchmarks.

  11. NeMo: Needle in a Montage for Video-Language Understanding

    cs.CV 2025-09 conditional novelty 5.0 of 10

    NeMoBench, an automatically generated benchmark with 31,378 QA pairs, shows that video LLMs struggle with temporal grounding of relevant clips hidden in long montages.

  12. Goku: Flow Based Video Generative Foundation Models

    cs.CV 2025-02 conditional novelty 5.0 of 10

    A joint image-video generation model family reports state-of-the-art benchmark scores using rectified flow transformers, with all key evidence self-reported and no artifacts released.

Pith tools