FocusedAD: Character-centric Movie Audio Description

Chun Wang; Jiajun Bu; Liangcheng Li; Sheng Zhou; Xiaojun Ye; Yiren Song

REVIEW 3 major objections 5 minor 61 references

FocusedAD: Character-centric Movie Audio Description

T0 review · 3 major / 5 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read FocusedAD claims that naming and tracking the active characters in a movie clip, then conditioning narration on story context, is enough to produce character-centric audio descriptions that outperform prior systems on zero-shot benchmarks.

desk verdict A plausible systems paper whose headline zero-shot SOTA claim currently rests on an unverified train/test split; the architecture and ablations deserve a serious look, but the empirical claim needs a data-disjointness check before it should be cited. read the letter →

arxiv 2504.12157 v3 pith:MBMALURU submitted 2025-04-16 cs.CV

Xiaojun Ye , Chun Wang , Yiren Song , Sheng Zhou , Liangcheng Li , Jiajun Bu This is my paper · ORCID

classification cs.CV

keywords movieaudiodescriptioncharacter-centriccaptioningmulti-modallargelanguagemodelscharacterrecognitionfaceclusteringsoftpromptingzero-shotvideoaccessibilityforblindandvisuallyimpaired

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper is trying to establish that movie audio description can be improved by making it character-centric: instead of describing entire scenes, a model should first find the active main characters, name them, and then narrate their actions and interactions. The authors propose FocusedAD, which combines a character perception module that tracks faces and links them to names, a dynamic prior module that feeds previous descriptions and subtitles into the generator through learnable soft prompts, and a focused caption module that pools visual features inside detected character regions. Training deliberately uses a proxy video dataset built from movie frames and character annotations rather than the target AD benchmark, so the reported gains on MAD-eval-Named and the new Cinepile-AD set are presented as zero-shot evidence. If the claim holds, automatic AD for blind and visually impaired audiences could be built from data that is easier to obtain, and generated narrations would carry the character names that AD guidelines require.

What carries the argument

The load-bearing object is the character best query bank: for each main character, a set of face crops is clustered and a representative embedding is chosen by maximizing the ratio of inter-class distance to intra-class distance, so that characters remain distinguishable across lighting, makeup, and camera changes. Around that bank, the Character Perception Module performs face matching and bi-directional region propagation to mark which named characters are active in each clip; the Dynamic Prior Module converts previous ADs and subtitles into text priors and adds trainable soft prompts that adapt when zero, one, or many characters are present; and the Focused Caption Module pools image features inside each character mask to create character tokens, merges them temporally, and lets the LLM reason jointly over scene tokens, character tokens, and text tokens. The full pipeline is what the paper credits for shifting attention from generic objects to narrative-salient character actions.

What would settle it

Compare the list of film titles (and their clips) in the Storyboard20K-derived Storyboard-v2 training set against the ten movies in MAD-eval-Named and the clips in Cinepile-AD; any overlap would undermine the zero-shot interpretation. A second check: retrain or re-run the system with the ground-truth previous ADs replaced by the model's own previously generated ADs, and measure the score drop; if the drop is large, the benchmark's text-prior setting is doing much of the work.

Watch

Extended reading notes

Core claim

FocusedAD claims that explicit character grounding is the missing ingredient in current automated audio description. The model detects faces in key frames, associates each face with a character name using a clustered 'best query bank' built from portraits and movie frames, propagates the resulting regions forward and backward in time, and then feeds both scene-level and character-level visual tokens into an instruction-tuned language model together with prior ADs and subtitles. On the MAD-eval-Named benchmark the full system reports SPICE 7.4, METEOR 8.1, and BertScore 57.7, and on the proposed Cinepile-AD set SPICE 15.4, METEOR 14.7, and BertScore 64.5, exceeding dedicated AD systems such as AutoAD-I and MM-Narrator as well as general video models. The paper states that the model was fine-tuned only on its own Storyboard-v2 training set, not on the target AD datasets, making the comparison a zero-shot evaluation.

Load-bearing premise

The claim that the results are zero-shot depends on the assumption that no clip from the evaluation movies appears in the Storyboard20K-derived training data, even though both sides draw from the same family of movie datasets; if training and evaluation films overlap, the reported numbers would measure memorization rather than generalization.

Editorial extensions

If this is right

Automated AD can be trained on proxy video data rather than on scarce AD-annotated film, lowering the data barrier for accessibility tools.
Character names, not just pronouns or generic labels, become a normal output of the captioning model, which matches professional AD style.
The dynamic soft prompt makes a single model handle scenes with any number of active characters without re-training.
The redundancy-aware metric introduced in the paper offers a way to quantify whether generated AD is concise enough for BVI listeners.
The Cinepile-AD test set provides a public-style benchmark derived from online clips, giving future AD systems a second zero-shot target.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The same character query-bank construction could be applied to other identity-grounded video tasks, such as person-specific video question answering or long-form character tracking in TV series, though the paper does not test those settings.
A stricter test of the zero-shot claim would be to hold out every film whose clips or annotations appear in the Storyboard20K-derived training data, since the reported independence rests on dataset lineage.
One could test whether the soft prompt is actually carrying the narrative context by replacing the previous ADs with shuffled or wrong-context ADs; if scores drop sharply, the prompt is not just a style token.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Punchline: FocusedAD is a well-engineered character-centric AD pipeline that does something genuinely useful: it clusters face embeddings into per-character "best queries," tracks those characters through clips, and injects both character regions and prior AD/subtitle context as soft prompts into a 7B video LLM. The system-level integration is sensible, and the ablations are internally coherent. The best-query clustering is the most novel piece and it earns its keep in Table 1 (C2 vs B2). The application matters, too: automatic AD for BVI audiences is a real and underserved need.

What I think is soft, in proportion:

First, the zero-shot SOTA claim is load-bearing and not yet verifiable. Storyboard-v2 is built from MovieNet/LSMDC-derived Storyboard20K, and MAD-eval-Named is also LSMDC-derived. The paper says the test split excluded films from LSMDC train/test splits, but it never shows that the 10 retained movies are absent from Storyboard20K or from the 11,250 training triplets. Since the training pipeline extracts complete movie clips from full films using Storyboard20K timestamps, a single overlapping movie would put training clips into the evaluation set. Code and data are not released, and no overlap check is reported. That is a verifiability problem, not an architecture problem. The stress-test note is right.

Second, the comparison table is missing AutoAD-II even though the text describes it as a fine-tuning-based SOTA. If it is omitted because it uses MAD training data, that needs to be stated. Also, the AutoAD-I BertScore of 24.2 looks inconsistent with the other numbers in the table; I would want the source or a recalculation.

Third, the new redundancy metric R is defined with a threshold theta, but theta is never reported. Without it, the R comparisons in Table 2 are not reproducible. This is minor and fixable, but it should be a one-line addition.

Smaller items: the free parameters u, mu, and theta are all selected with some tuning, which is acceptable for a systems paper, but theta should be reported and the u tuning should be described as calibration rather than as part of the zero-shot claim. The qualitative examples are cherry-picked, as is usual.

None of these are fatal. The architecture is plausible, the ablation logic is coherent, and the method is a legitimate extension of known components. But the headline empirical claim cannot be accepted as stated until the train/test disjointness is demonstrated. This is a conditional for me: add an overlap check and release code/data, and I would take the SOTA numbers seriously. Without that, the contribution reduces to a solid engineering report with a promising method rather than a verified SOTA result.

Who it is for: people working on movie understanding, accessible media, and multimodal captioning. Worth sending to a serious referee, with the split check and the missing baseline as required revisions.

Referee Report

3 major / 5 minor

Summary. The paper proposes FocusedAD, a framework for automatic movie audio description (AD) that combines a Character Perception Module (CPM), a Dynamic Prior Module (DPM) with learnable soft prompts, and a Focused Caption Module (FCM) built on a 7B MLLM. The authors also introduce an automated pipeline that constructs a character query bank via clustering and use it to build a training dataset, Storyboard-v2, from Storyboard20K. They evaluate the model zero-shot on MAD-eval-Named and a newly introduced Cinepile-AD test set, reporting state-of-the-art results on SPICE, METEOR, and BertScore, alongside a newly proposed redundancy metric R. The central claim is that a model trained on proxy movie data can outperform dedicated AD systems, including AutoAD-I, MM-Narrator, and general video MLLMs, without training on the target AD datasets.

Significance. If the empirical claims hold, FocusedAD would be a meaningful step toward practical automatic AD, particularly for character-centric narration and for the blind and visually impaired community. The proposed pipeline for constructing character query banks is sensible and addresses a real bottleneck in character identification. The ablations in Table 1 are internally coherent and provide evidence for the contribution of each module. However, the headline zero-shot SOTA claim rests on assumptions about data disjointness that are not verified, and the comparison table omits a key baseline, so the significance is conditional on resolving these issues.

major comments (3)

[§4.1, §4.2] The zero-shot claim requires that the Storyboard-v2 training clips, constructed from Storyboard20K (sourced from MovieNet and LSMDC), do not overlap with the evaluation movies in MAD-eval-Named (derived from LSMDC, with 10 retained movies) or in Cinepile-AD. No disjointness check is reported, and given that both the training and evaluation corpora draw from the same movie families, even a single overlapping movie would place training clips in the evaluation set and invalidate the zero-shot interpretation of Table 2. The authors should provide an explicit overlap analysis at the movie and clip level.
[§5.4, Table 2] AutoAD-II is mentioned in the text as a fine-tuning-based state-of-the-art model, but its results are not included in Table 2. Since the paper claims SOTA over previous AD methods, omitting a strong and directly comparable baseline leaves the comparison incomplete and weakens the central empirical claim.
[§5.4, Table 2] The reported BertScore of 24.2 for AutoAD-I is far below the range 50-60 reported for all other methods in the same table. This is likely an artifact of using a different evaluation configuration (e.g., tokenization, model variant, or rescaling). The authors should clarify the exact evaluation settings and, if necessary, re-run the AutoAD-I evaluation under the same protocol to ensure fair comparison.

minor comments (5)

[Abstract] There is a missing space in 'understanding.To identify' in the abstract.
[§5.2] The section header contains a typo: 'FocesedAD' should be 'FocusedAD'.
[§3.1, Eq. (6)] Equation (6) uses y_k^a for the detected face but the text defines x_k^a as the detected face image; please unify the notation.
[Fig. 5] In Figure 5, 'con’idence' is a typo for 'confidence'.
[§5.1] The redundancy metric R relies on a threshold theta and a 'rigorously validated vocabulary', but the paper does not specify how theta is chosen, whether it is tuned on a validation set, or whether R correlates with human judgments of conciseness. Since R is used to support the model in Table 2, a sentence of justification or a reference to a validation study would be helpful.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation: the model is trained on an independently constructed video dataset and evaluated with standard metrics; the proposed metric and test set are self-validating but not definitionally tied to the model's outputs.

full rationale

The paper's central chain is: build Storyboard-v2 from Storyboard20K, train FocusedAD's CPM/DPM/FCM components, then evaluate on MAD-eval-Named and Cinepile-AD using SPICE, METEOR, BertScore, and the authors' redundancy metric R. None of these steps reduces by construction to the model's own inputs. The character query clustering (Eqs. 1-3) is a preprocessing step for character recognition, not a quantity that is later 'predicted' as an evaluation result. The soft prompts and dynamic instruction templates are trained components whose ablations (Table 1) are compared against baseline configurations, so the claimed improvements are empirically grounded rather than assumed. The self-defined metric R (Eqs. 9-10) is a post-hoc evaluation score, and the headline SOTA claims rest primarily on the standard captioning metrics SPICE, METEOR, and BertScore, which are independent of the authors' design choices. The only substantive concern is dataset hygiene: Storyboard-v2 is sourced from MovieNet/LSMDC and MAD-eval-Named is derived from LSMDC, and the paper does not report an explicit overlap check between the 10 evaluation movies and the 11,250 training triplets. That is a verifiability and possible leakage concern, not circularity, because no equation or fitted parameter is definitionally tied to the evaluation outcome. Self-citations such as MMAD [50] appear only as related-work context and are not load-bearing. Overall, the derivation is self-contained and no circular step can be exhibited.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

The method relies on several domain assumptions about face-based character identification, dataset quality, and teacher-forced context. The main free parameters are the recognition threshold u, the token merge count mu, and the unreported redundancy threshold theta. No physically new entities are introduced.

free parameters (3)

character recognition threshold u = 1.3
Tuned on the film Les Misérables (Section 5.2, Fig. 7) to maximize character recognition accuracy, then applied to all test clips.
temporal token merge count mu = 4
Set as a constant in Section 3.3 (Eq. 8) to merge K character tokens into 4 representative tokens.
redundancy metric threshold theta = not reported
Used in the proposed redundancy metric R (Eq. 9, Section 5.1); the threshold theta is not specified in the paper.

assumptions (5)

domain assumption Faces detected by MTCNN and embedded by FaceNet are sufficient to identify main characters in movie frames
The entire Character Perception Module relies on face detection and face embedding similarity; character appearances with occlusions, heavy makeup, or off-screen faces may break this.
domain assumption Storyboard20K annotations provide reliable character bounding boxes and movie clips for constructing training data
The training set Storyboard-v2 is auto-constructed from Storyboard20K, and the quality of the character query bank depends on the quality of these source annotations.
domain assumption Prior ADs and subtitles as text context improve current AD generation and are available at inference
The Dynamic Prior Module feeds previous ADs and subtitles; if previous ADs are from the ground-truth dataset rather than the model's own outputs, evaluation is teacher-forced and not deployment-realistic.
ad hoc to paper The newly proposed redundancy metric R with threshold theta is a valid measure of conciseness
The metric is introduced in this paper and used in the main comparison table; it has no external validation or published threshold.
ad hoc to paper The clustering objective in Eq. 3 can be optimized to produce character best queries
The paper states K-means is used, but Eq. 3 defines a different objective; the actual clustering algorithm is unspecified.

how reviews work

0 comments

Cite this review

Pith. "Pith review of FocusedAD: Character-centric Movie Audio Description." pith.science (2026). https://pith.science/paper/MBMALURU

@misc{pith2026250412157,
  author       = {Pith},
  title        = {Pith review of: FocusedAD: Character-centric Movie Audio Description},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/MBMALURU}},
  note         = {Machine review of arXiv:2504.12157}
}

read the original abstract

Movie Audio Description (AD) aims to narrate visual content during dialogue-free segments, particularly benefiting blind and visually impaired (BVI) audiences. Compared with general video captioning, AD demands plot-relevant narration with explicit character name references, posing unique challenges in movie understanding.To identify active main characters and focus on storyline-relevant regions, we propose FocusedAD, a novel framework that delivers character-centric movie audio descriptions. It includes: (i) a Character Perception Module(CPM) for tracking character regions and linking them to names; (ii) a Dynamic Prior Module(DPM) that injects contextual cues from prior ADs and subtitles via learnable soft prompts; and (iii) a Focused Caption Module(FCM) that generates narrations enriched with plot-relevant details and named characters. To overcome limitations in character identification, we also introduce an automated pipeline for building character query banks. FocusedAD achieves state-of-the-art performance on multiple benchmarks, including strong zero-shot results on MAD-eval-Named and our newly proposed Cinepile-AD dataset. Code and data will be released at https://github.com/Thorin215/FocusedAD .

Figures

Figures reproduced from arXiv: 2504.12157 by the authors.

**Figure 2.** Overview of FocusedAD: FocusedAD takes movie clips as input and captures the character best query bank through [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗

**Figure 3.** Character Perception Module traverses the key [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗

Figures from the paper (5 more)

**Figure 4.** Figure 4: Instruction template with soft prompt. We use [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]

**Figure 5.** Figure 5: Samples of Storyboard-v2. Our dataset involves [PITH_FULL_IMAGE:figures/full_fig_p005_5.png]

**Figure 6.** Figure 6: The clustering results for obtaining the best query [PITH_FULL_IMAGE:figures/full_fig_p006_6.png]

**Figure 7.** Figure 7: Ablation study on the film Les Misérables to evaluate changes in FocusedAD indicators under varying thresholds. This film is selected for its representative nature, as its metrics closely align with the average of MAD-eval-Named. Effect of Dynamic Prior Module. In the…

**Figure 8.** Figure 8: Qualitative results of our method. The top two movie clips demos are from MAD-eval-Named and the bottom two [PITH_FULL_IMAGE:figures/full_fig_p008_8.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

61 extracted references · 25 canonical work pages

[1]

[n. d.]. American Council of the Blind. https://adp.acb.org/

work page
[2]

Qwen2 Technical Report

2024. Qwen2 Technical Report. (2024)

2024
[3]

Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Floren- cia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 (2023)

arXiv 2023
[4]

Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al
[5]

Peter Anderson, Basura Fernando, Mark Johnson, and Stephen Gould. 2016. Spice: Semantic propositional image caption evaluation. In Computer Vision– ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part V 14 . Springer, 382–398

2016
[6]

Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al . 2025. Qwen2. 5-vl technical report. arXiv preprint arXiv:2502.13923 (2025)

arXiv 2025
[7]

Max Bain, Jaesung Huh, Tengda Han, and Andrew Zisserman. 2023. Whis- perx: Time-accurate speech transcription of long-form audio. arXiv preprint arXiv:2303.00747 (2023)

arXiv 2023
[8]

Satanjeev Banerjee and Alon Lavie. 2005. METEOR: An automatic metric for MT evaluation with improved correlation with human judgments. InProceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization. 65–72

2005

Show all 61 references

[9]

Zesen Cheng Zhiqiang Hu Yuqian Yuan Guanzheng Chen Sicong Leng Yuming Jiang Hang Zhang Xin Li Peng Jin Wenqi Zhang Fan Wang Lidong Bing Deli Zhao Boqiang Zhang, Kehan Li. 2025. VideoLLaMA 3: Frontier Multimodal Foundation Models for Image and Video Understanding. arXiv preprin...

2025 arXiv
[10]

Andrew Brown, Ernesto Coto, and Andrew Zisserman. 2021. Automated video labelling: Identifying faces by corroborative evidence. In 2021 IEEE 4th Inter- national Conference on Multimedia Information Processing and Retrieval (MIPR) . IEEE, 77–83

2021
[11]

Andrew Brown, Vicky Kalogeiton, and Andrew Zisserman. 2021. Face, body, voice: Video person-clustering with multiple modalities. In Proceedings of the IEEE/CVF International Conference on Computer Vision . 3184–3194

2021
[12]

Virgínia P Campos, Luiz MG Gonçalves, Wesnydy L Ribeiro, Tiago MU Araújo, Thaís G Do Rego, Pedro HV Figueiredo, Suanny FS Vieira, Thiago FS Costa, Caio C Moraes, Alexandre CS Cruz, et al. 2023. Machine generation of audio description for blind and visually impaired people.ACM ...

2023
[13]

Aman Chadha, Gurneet Arora, and Navpreet Kaloty. 2020. iPerceive: Applying common-sense reasoning to multi-modal dense video captioning and video question answering. arXiv preprint arXiv:2011.07735 (2020)

2020 arXiv
[14]

David Chen and William B Dolan. 2011. Collecting highly parallel data for paraphrase evaluation. In Proceedings of the 49th annual meeting of the association for computational linguistics: human language technologies . 190–200

2011
[15]

Lin Chen, Xilin Wei, Jinsong Li, Xiaoyi Dong, Pan Zhang, Yuhang Zang, Zehui Chen, Haodong Duan, Zhenyu Tang, Li Yuan, et al. 2024. Sharegpt4video: Im- proving video understanding and generation with better captions. Advances in Neural Information Processing Systems 37 (2024), ...

2024
[16]

Shaoxiang Chen and Yu-Gang Jiang. 2021. Towards bridging event captioner and sentence localizer for weakly supervised dense event captioning. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition . 8425–8435

2021
[17]

Tengda Han, Max Bain, Arsha Nagrani, Gul Varol, Weidi Xie, and Andrew Zis- serman. 2023. Autoad ii: The sequel-who, when, and what in movie audio description. In Proceedings of the IEEE/CVF International Conference on Computer Vision. 13645–13655

2023
[18]

Tengda Han, Max Bain, Arsha Nagrani, Gül Varol, Weidi Xie, and Andrew Zisser- man. 2023. Autoad: Movie description in context. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition . 18930–18940

2023
[19]

Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick
[20]

Qingqiu Huang, Wentao Liu, and Dahua Lin. 2018. Person search in videos with one portrait through visual and temporal links. In Proceedings of the European conference on computer vision (ECCV) . 425–441

2018
[21]

In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition

Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition . 16000–16009
[22]

Ranjay Krishna, Kenji Hata, Frederic Ren, Li Fei-Fei, and Juan Carlos Niebles
[23]

Qingqiu Huang, Yu Xiong, Anyi Rao, Jiaze Wang, and Dahua Lin. 2020. Movienet: A holistic dataset for movie understanding. In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part IV 16 . Springer, 709–727

2020
[24]

KunChang Li, Yinan He, Yi Wang, Yizhuo Li, Wenhai Wang, Ping Luo, Yali Wang, Limin Wang, and Yu Qiao. 2023. Videochat: Chat-centric video understanding. arXiv preprint arXiv:2305.06355 (2023)

2023 arXiv
[25]

Siyuan Li, Li Sun, and Qingli Li. 2023. Clip-reid: exploiting vision-language model for image re-identification without concrete text labels. In Proceedings of the AAAI conference on artificial intelligence , Vol. 37. 1405–1413

2023
[26]

Anna Kukleva, Makarand Tapaswi, and Ivan Laptev. 2020. Learning interactions and relationships between movie characters. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition . 9849–9858

2020
[27]

Kevin Lin, Faisal Ahmed, Linjie Li, Chung-Ching Lin, Ehsan Azarnasab, Zhengyuan Yang, Jianfeng Wang, Lin Liang, Zicheng Liu, Yumao Lu, et al. 2023. Mm-vid: Advancing video understanding with gpt-4v (ision). arXiv preprint arXiv:2310.19773 (2023)

2023 arXiv
[28]

Kevin Lin, Linjie Li, Chung-Ching Lin, Faisal Ahmed, Zhe Gan, Zicheng Liu, Yumao Lu, and Lijuan Wang. 2022. Swinbert: End-to-end transformers with sparse attention for video captioning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition . 17949–17958

2022
[29]

Yehao Li, Ting Yao, Yingwei Pan, Hongyang Chao, and Tao Mei. 2018. Jointly localizing and describing events for dense video captioning. In Proceedings of the IEEE conference on computer vision and pattern recognition . 7492–7500

2018
[30]

Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. 2023. Visual In- struction Tuning

2023
[31]

Ilya Loshchilov and Frank Hutter. 2016. Sgdr: Stochastic gradient descent with warm restarts. arXiv preprint arXiv:1608.03983 (2016)

2016 arXiv
[32]

Tsung-Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie. 2017. Feature pyramid networks for object detection. In Proceed- ings of the IEEE conference on computer vision and pattern recognition . 2117–2125

2017
[33]

Huaishao Luo, Lei Ji, Botian Shi, Haoyang Huang, Nan Duan, Tianrui Li, Jason Li, Taroon Bharti, and Ming Zhou. 2020. Univl: A unified video and language pre-training model for multimodal understanding and generation. arXiv preprint arXiv:2002.06353 (2020)

2020 arXiv
[34]

Jonghwan Mun, Linjie Yang, Zhou Ren, Ning Xu, and Bohyung Han. 2019. Stream- lined dense video captioning. In Proceedings of the IEEE/CVF conference on com- puter vision and pattern recognition . 6588–6597

2019
[35]

Ilya Loshchilov and Frank Hutter. 2017. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101 (2017)

2017 arXiv
[36]

Elisa Perego. 2016. Gains and losses of watching audio described films for sighted viewers. Target 28, 3 (2016), 424–444

2016
[37]

Nikhila Ravi, Valentin Gabeur, Yuan-Ting Hu, Ronghang Hu, Chaitanya Ryali, Tengyu Ma, Haitham Khedr, Roman Rädle, Chloe Rolland, Laura Gustafson, et al. 2024. Sam 2: Segment anything in images and videos. arXiv preprint arXiv:2408.00714 (2024)

2024 arXiv
[38]

Arsha Nagrani and Andrew Zisserman. 2018. From benedict cumberbatch to sherlock holmes: Character identification in tv series without a script. arXiv preprint arXiv:1801.10442 (2018)

2018 arXiv
[39]

Anna Rohrbach, Atousa Torabi, Marcus Rohrbach, Niket Tandon, Christopher Pal, Hugo Larochelle, Aaron Courville, and Bernt Schiele. 2017. Movie description. International Journal of Computer Vision 123 (2017), 94–120

2017
[40]

Chaitanya Ryali, Yuan-Ting Hu, Daniel Bolya, Chen Wei, Haoqi Fan, Po-Yao Huang, Vaibhav Aggarwal, Arkabandhu Chowdhury, Omid Poursaeed, Judy Hoffman, et al . 2023. Hiera: A hierarchical vision transformer without the bells-and-whistles. In International conference on machine l...

2023
[41]

Ruchit Rawal, Khalid Saifullah, Miquel Farré, Ronen Basri, David Jacobs, Gowthami Somepalli, and Tom Goldstein. 2024. Cinepile: A long video question answering dataset and benchmark. arXiv preprint arXiv:2405.08813 (2024)

2024 arXiv
[42]

Paul Hongsuck Seo, Arsha Nagrani, Anurag Arnab, and Cordelia Schmid. 2022. End-to-end generative pretraining for multimodal video captioning. In Pro- ceedings of the IEEE/CVF conference on computer vision and pattern recognition . 17959–17968

2022
[43]

Shikhar Sharma, Layla El Asri, Hannes Schulz, and Jeremie Zumer. 2017. Rele- vance of Unsupervised Metrics in Task-Oriented Dialogue for Evaluating Natural Xiaojun Ye, Chun Wang et al. Language Generation. CoRR abs/1706.09799 (2017). http://arxiv.org/abs/1706. 09799

2017 arXiv
[44]

Florian Schroff, Dmitry Kalenichenko, and James Philbin. 2015. Facenet: A unified embedding for face recognition and clustering. In Proceedings of the IEEE conference on computer vision and pattern recognition . 815–823

2015
[45]

Atousa Torabi, Christopher Pal, Hugo Larochelle, and Aaron Courville. 2015. Using descriptive video services to create a large data source for video annotation research. arXiv preprint arXiv:1503.01070 (2015)

2015 arXiv
[46]

Yujia Wang, Wei Liang, Haikun Huang, Yongqi Zhang, Dingzeyu Li, and Lap-Fai Yu. 2021. Toward automatic audio description generation for accessible videos. In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems . 1–12

2021
[47]

Mattia Soldan, Alejandro Pardo, Juan León Alcázar, Fabian Caba, Chen Zhao, Silvio Giancola, and Bernard Ghanem. 2022. Mad: A scalable dataset for language grounding in videos from movie audio descriptions. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern...

2022
[48]

Jinheng Xie, Jiajun Feng, Zhaoxu Tian, Kevin Qinghong Lin, Yawen Huang, Xi Xia, Nanxu Gong, Xu Zuo, Jiaqi Yang, Yefeng Zheng, et al. 2024. Learning long- form video prior via generative pre-training. arXiv preprint arXiv:2404.15909 (2024)

2024 arXiv
[49]

Antoine Yang, Arsha Nagrani, Paul Hongsuck Seo, Antoine Miech, Jordi Pont- Tuset, Ivan Laptev, Josef Sivic, and Cordelia Schmid. 2023. Vid2seq: Large-scale pretraining of a visual language model for dense video captioning. InProceedings of the IEEE/CVF Conference on Computer V...

2023
[50]

Tong Xiao, Shuang Li, Bochao Wang, Liang Lin, and Xiaogang Wang. 2017. Joint detection and identification feature learning for person search. In Proceedings of the IEEE conference on computer vision and pattern recognition . 3415–3424

2017
[51]

Youngjae Yu, Jiwan Chung, Heeseung Yun, Jongseok Kim, and Gunhee Kim
[52]

Yuqian Yuan, Hang Zhang, Wentong Li, Zesen Cheng, Boqiang Zhang, Long Li, Xin Li, Deli Zhao, Wenqiao Zhang, Yueting Zhuang, et al. 2024. Videorefer suite: Advancing spatial-temporal object understanding with video llm. arXiv preprint arXiv:2501.00599 (2024)

2024 arXiv
[53]

Xiaojun Ye, Junhao Chen, Xiang Li, Haidong Xin, Chao Li, Sheng Zhou, and Jiajun Bu. 2024. MMAD: Multi-modal Movie Audio Description. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024) . 1...

2024
[54]

Chaoyi Zhang, Kevin Lin, Zhengyuan Yang, Jianfeng Wang, Linjie Li, Chung- Ching Lin, Zicheng Liu, and Lijuan Wang. 2024. Mm-narrator: Narrating long- form videos with multimodal in-context learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recog...

2024
[55]

Kaipeng Zhang, Zhanpeng Zhang, Zhifeng Li, and Yu Qiao. 2016. Joint face detection and alignment using multitask cascaded convolutional networks. IEEE signal processing letters 23, 10 (2016), 1499–1503

2016
[56]

Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. 2019. Bertscore: Evaluating text generation with bert. arXiv preprint arXiv:1904.09675 (2019)

2019 arXiv
[57]

Zihao Yue, Qi Zhang, Anwen Hu, Liang Zhang, Ziheng Wang, and Qin Jin. 2023. Movie101: A New Movie Understanding Benchmark. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 4669–4684. doi:10.18653/v1/2023.acl-long.257

2023 doi
[61]

Xingyi Zhou, Anurag Arnab, Shyamal Buch, Shen Yan, Austin Myers, Xuehan Xiong, Arsha Nagrani, and Cordelia Schmid. 2024. Streaming dense video cap- tioning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 18243–18252

2024
[2017]

In Proceedings of the IEEE international conference on computer vision

Dense-captioning events in videos. In Proceedings of the IEEE international conference on computer vision . 706–715
[2021]

In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

Transitional adaptation of pretrained models for visual storytelling. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition . 12658–12668
[2022]

Advances in neural information processing systems 35 (2022), 23716–23736

Flamingo: a visual language model for few-shot learning. Advances in neural information processing systems 35 (2022), 23716–23736

2022

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.

[1] [1]

[n. d.]. American Council of the Blind. https://adp.acb.org/

work page

[2] [2]

Qwen2 Technical Report

2024. Qwen2 Technical Report. (2024)

2024

[3] [3]

Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Floren- cia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 (2023)

arXiv 2023

[4] [4]

Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al

[5] [5]

Peter Anderson, Basura Fernando, Mark Johnson, and Stephen Gould. 2016. Spice: Semantic propositional image caption evaluation. In Computer Vision– ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part V 14 . Springer, 382–398

2016

[6] [6]

Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, et al . 2025. Qwen2. 5-vl technical report. arXiv preprint arXiv:2502.13923 (2025)

arXiv 2025

[7] [7]

Max Bain, Jaesung Huh, Tengda Han, and Andrew Zisserman. 2023. Whis- perx: Time-accurate speech transcription of long-form audio. arXiv preprint arXiv:2303.00747 (2023)

arXiv 2023

[8] [8]

Satanjeev Banerjee and Alon Lavie. 2005. METEOR: An automatic metric for MT evaluation with improved correlation with human judgments. InProceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization. 65–72

2005