Pith. sign in

REVIEW 18 cited by

Koala-36M: A Large-scale Video Dataset Improving Consistency between Fine-grained Conditions and Video Content

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2410.08260 v2 pith:KAE7LVGK submitted 2024-10-10 cs.CV cs.AI

Qiuheng Wang , Yukai Shi , Jiarong Ou , Rui Chen , Ke Lin , Jiahao Wang , Boyuan Jiang , Haotian Yang
show 5 more authors
Mingwu Zheng Xin Tao Fei Yang Pengfei Wan Di Zhang
This is my paper · ORCID
classification cs.CVcs.AI
keywords videodatasetqualitycaptionsconditionsconsistencydatasetsfine-grained
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

With the continuous progress of visual generation technologies, the scale of video datasets has grown exponentially. The quality of these datasets plays a pivotal role in the performance of video generation models. We assert that temporal splitting, detailed captions, and video quality filtering are three crucial determinants of dataset quality. However, existing datasets exhibit various limitations in these areas. To address these challenges, we introduce Koala-36M, a large-scale, high-quality video dataset featuring accurate temporal splitting, detailed captions, and superior video quality. The essence of our approach lies in improving the consistency between fine-grained conditions and video content. Specifically, we employ a linear classifier on probability distributions to enhance the accuracy of transition detection, ensuring better temporal consistency. We then provide structured captions for the splitted videos, with an average length of 200 words, to improve text-video alignment. Additionally, we develop a Video Training Suitability Score (VTSS) that integrates multiple sub-metrics, allowing us to filter high-quality videos from the original corpus. Finally, we incorporate several metrics into the training process of the generation model, further refining the fine-grained conditions. Our experiments demonstrate the effectiveness of our data processing pipeline and the quality of the proposed Koala-36M dataset. Our dataset and code have been released at https://koala36m.github.io/.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 18 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Interspatial Attention for Efficient 4D Human Video Generation

    cs.CV 2025-05 conditional novelty 7.0 of 10

    A new symmetric 3D-to-2D attention mechanism with relative positional encodings, plus a motion-tuned video VAE, improves controllable 4D human video generation.

  2. MSC: A Marine Wildlife Video Dataset with Grounded Segmentation and Clip-Level Captioning

    cs.CV 2025-08 reject novelty 6.0 of 10

    A new marine wildlife video dataset with clip-level captions and grounded segmentation masks, benchmarked on captioning, grounding, and video generation.

  3. VMoBA: Mixture-of-Block Attention for Video Diffusion Models

    cs.CV 2025-06 conditional novelty 6.0 of 10

    VMoBA is a sparse attention mechanism for video diffusion models that combines cyclic 1D-2D-3D block partitioning with global and threshold-based block selection to reduce training FLOPs while keeping generation quality.

  4. Phantom-Data : Towards a General Subject-Consistent Video Generation Dataset

    cs.CV 2025-06 conditional novelty 6.0 of 10

    Phantom-Data provides around one million cross-context, identity-consistent reference-video pairs for subject-to-video generation, and training on it improves prompt following and visual quality.

  5. UltraVideo: High-Quality UHD Video Dataset with Comprehensive Captions

    cs.CV 2025-06 conditional novelty 6.0 of 10

    UltraVideo provides the first UHD 4K/8K text-to-video dataset with 10 structured captions per clip, and LoRA fine-tuning on it produces native 1K/4K video generation with improved visual quality.

  6. PolyVivid: Vivid Multi-Subject Video Generation with Cross-Modal Interaction and Enhancement

    cs.CV 2025-06 conditional novelty 6.0 of 10

    PolyVivid combines VLLM-based grounding, 3D-RoPE positional encoding, and attention-inherited identity injection to generate customized videos with multiple consistent subjects and text-specified interactions.

  7. UNIC: Unified In-Context Video Editing

    cs.CV 2025-06 conditional novelty 6.0 of 10

    One diffusion transformer handles ID insert, swap, delete, stylization, propagation, and re-camera control in a single model using in-context token concatenation with task-aware positional encoding and bias.

  8. VideoREPA: Learning Physics for Video Generation through Relational Alignment with Foundation Models

    cs.CV 2025-05 conditional novelty 6.0 of 10

    VideoREPA adds a token-relation distillation loss that aligns a text-to-video diffusion model's internal features with VideoMAEv2, boosting physical commonsense scores on VideoPhy and VideoPhy2.

  9. OpenS2V-Nexus: A Detailed Benchmark and Million-Scale Dataset for Subject-to-Video Generation

    cs.CV 2025-05 conditional novelty 6.0 of 10

    A benchmark and five-million-clip dataset for evaluating and training subject-to-video generation models, with three new metrics for subject consistency, naturalness, and text alignment.

  10. HunyuanVideo-Avatar: High-Fidelity Audio-Driven Human Animation for Multiple Characters

    cs.CV 2025-05 conditional novelty 6.0 of 10

    HunyuanVideo-Avatar is an audio-driven video generator that enables emotion-controllable and multi-character animation by injecting character images, routing audio via face masks, and transferring emotion from referen...

  11. Semantic Frame Interpolation

    cs.CV 2025-07 reject novelty 5.0 of 10

    The authors define Semantic Frame Interpolation, build a 300k-clip dataset and benchmark, and propose a Mixture-of-LoRA adaptation of Wan2.1 that improves temporal smoothness but does not preserve the given start and ...

  12. GigaVideo-1: Advancing Video Generation via Automatic Feedback with 4 GPU-Hours Fine-Tuning

    cs.CV 2025-06 conditional novelty 5.0 of 10

    GigaVideo-1 fine-tunes Wan2.1 on synthetic weakness-targeted prompts with VLM reward reweighting and reports ~4% average VBench-2.0 gains per dimension at 4 GPU-hours each, though joint training gains less.

  13. FullDiT2: Efficient In-Context Conditioning for Video Diffusion Transformers

    cs.CV 2025-06 conditional novelty 5.0 of 10

    FullDiT2 accelerates FullDiT-style in-context conditioning for video by dynamic token selection and selective context caching, cutting per-step time by 2-3x with minimal quality loss.

  14. OmniV2V: Versatile Video Generation and Editing via Dynamic Content Manipulation

    cs.CV 2025-06 conditional novelty 5.0 of 10

    OmniV2V is one diffusion-transformer model that performs eight video generation and editing tasks by combining mask, pose, image, and text-instruction conditions.

  15. Frame-Level Captions for Long Video Generation with Complex Multi Scenes

    cs.CV 2025-05 conditional novelty 5.0 of 10

    Frame-level captions with per-frame cross-attention and parallel multi-window denoising reduce semantic confusion in long multi-scene video generation in the authors' internal evaluation.

  16. RealCam-I2V: Real-World Image-to-Video Generation with Interactive Complex Camera Control

    cs.CV 2025-02 conditional novelty 5.0 of 10

    Metric-scale depth alignment plus scene-constrained noise shaping improves camera controllability and video quality for image-to-video generation on RealEstate10K.

  17. Wan-S2V: Audio-Driven Cinematic Video Generation

    cs.CV 2025-08 reject novelty 4.0 of 10

    Wan-S2V is an audio-driven video generator built on Wan, claiming better cinematic character animation than prior systems, though the evaluation is limited.

  18. A Survey on Long-Video Storytelling Generation: Architectures, Consistency, and Cinematic Quality

    cs.CV 2025-07 conditional novelty 3.0 of 10

    A survey of 32 long-video generation papers, presenting a taxonomy and component recommendations for backbones, text encoders, objectives, and positional encodings.

Pith tools