Pith. sign in

REVIEW 4 major objections 6 minor 65 references

HierSum: A Global and Local Attention Mechanism for Video Summarization

T0 review · 4 major / 6 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Hierarchical attention over subtitles and global instructions, supervised by YouTube replay statistics, improves instructional video summarization.

desk verdict A clearly described, honestly written method paper whose abstract overclaims: the own tables show HierSum is not consistently SOTA, so the central claim needs rewriting before it can be published. read the letter →

arxiv 2504.18689 v1 pith:3AO2GECY submitted 2025-04-25 cs.CV cs.AIcs.LG

Apoorva Beedu , Irfan Essa This is my paper · ORCID
classification cs.CVcs.AIcs.LG
keywords videosummarizationinstructionalvideoshierarchicalattentionmostreplayedstatisticsmulti-modallearningparent-childtrainingsubtitlealignmentglobalinstructions
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that video summarization for instructional videos improves when a model learns from two levels of text at once: fine-grained subtitle sentences aligned to moments of the video, and coarse global instructions describing the whole task. The proposed HierSum model alternates between these two levels in a parent-child training loop and uses YouTube's "most replayed" statistics as the supervisory signal that tells it which segments matter. If this works, summaries of how-to videos would better preserve the actual steps of a task rather than just the visually salient or frequently repeated moments. The paper also contributes a new multi-modal dataset of WikiHow and EHow videos paired with step-by-step articles. Reported gains include roughly 2 percent on TVSum rank correlation, 2 percent F1 on Mr.HiSum, and 1 percent cosine similarity on BLiSS, though F1 on TVSum trails the best baseline.

What carries the argument

The machinery is a parent-child hierarchical training protocol over a shared multimodal transformer. In child steps the input sequence interleaves video frame features with sentence embeddings of subtitles; in parent steps the subtitles are replaced by one global description and only video importance is predicted. An alignment-guided self-attention mask forces cross-modal attention between frames and their time-aligned subtitles, and the final objective combines focal loss on importance labels, mean-squared error on most-replayed scores, and inter- and intra-modality contrastive losses. This two-level loop is what carries the claim: it is the mechanism by which local step cues and global task context jointly shape the importance scores.

What would settle it

Take the Mr.HiSum or curated dataset's most-replayed labels and compare them, shot by shot, with human importance ratings collected for the same videos; if agreement is low, or if a model trained on human ratings outperforms HierSum on human-judged summarization metrics while HierSum only wins on replay-derived ground truth, the central claim is refuted.

Watch

Extended reading notes

Core claim

HierSum's central claim is that a shared transformer trained with an alternating protocol, using m batches of clip-level subtitle-video pairs followed by one batch of global instruction-video pairs, learns representations that capture both the local, step-by-step execution of a task and its overall goal. The model predicts frame importance and sentence importance, with an alignment-guided self-attention mask that lets video frames attend to their corresponding subtitles, and it is trained with a loss combining focal classification, mean-squared error on most-replayed scores, and inter- and intra-modality contrastive terms. Using most-replayed statistics as the relevance label, the paper curates a large instructional-video dataset and shows that pre-training on it transfers to TVSum, BLiSS, Mr.HiSum, and the WikiHow test set. The intended consequence is that summaries align better with human judgment, as reflected in rank correlation gains, while remaining competitive on F1.

Load-bearing premise

Everything rests on the assumption that YouTube's "most replayed" statistic marks the same segments a good summary should keep; if viewers rewatch entertaining moments rather than instructive steps, the training labels drift away from what a summary needs, and the paper does not validate this correlation against human importance scores.

Editorial extensions

If this is right

  • Training on the curated WikiHow-EHow dataset, using most-replayed labels plus both text levels, transfers to existing benchmarks and yields roughly 1 to 2 percent gains after fine-tuning.
  • The parent-child protocol is crucial: using only subtitles or only global instructions hurts performance, and the best results appear around one global batch per five local batches.
  • Most-replayed supervision transfers to summarization: a model trained to predict replay scores also produces competitive summaries on F1 and mean average precision.
  • The model improves rank-order agreement with human importance scores on TVSum, raising Kendall's tau and Spearman's rho over the compared baselines.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If most-replayed statistics are a valid proxy for instructional importance, the same two-level text supervision could apply to other procedural media, such as repair or medical videos, where step descriptions and global protocols coexist.
  • The released dataset, pairing videos with both ASR subtitles and step-by-step articles, could be reused for step grounding or temporal action segmentation, not just summarization.
  • The optimal ratio of local to global training steps is likely dataset-dependent, so future work might adaptively schedule global steps instead of fixing them.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes HierSum, a hierarchical multimodal attention model for instructional video summarization. The model combines frame-level video features with subtitle-level local text and video-level global instruction text, and it is trained with an alternating parent-child protocol in which several local subtitle batches are followed by one global description batch. Supervision comes from YouTube "most replayed" statistics converted to binary importance labels and regression targets, combined with focal loss, an MSE loss on replay scores, and inter- and intra-modality contrastive losses. The authors evaluate on TVSum, BLiSS, Mr.HiSum, and WikiHow, and they additionally curate a new multi-modal instructional dataset from WikiHow and EHow videos. The central claim, stated in the abstract and conclusion, is that HierSum consistently outperforms existing methods on F1-score and rank correlation, with roughly 1-2% gains from pre-training on the curated dataset.

Significance. If the central claims were fully supported, the paper would make a useful contribution by demonstrating that hierarchical alignment of fine-grained subtitle cues with global instructional text, combined with replay-statistic supervision, improves video summarization. The proposed parent-child training protocol is a reasonable and timely idea, and the cross-dataset ablations in Tables 6 and 7, together with the global-step frequency study in Figure 3, provide useful design insights. The authors also report qualitative failure cases, which is a strength. However, the evidence as presented supports only a narrow set of claims: rank-correlation gains on TVSum, F1/MAP gains on Mr.HiSum, and cosine-similarity gains on BLiSS. The broad "consistently outperforms" statement is not supported by the paper's own tables.

major comments (4)
  1. [Abstract and Section 7 (Conclusion) vs. Tables 2 and 5] The abstract and conclusion claim that HierSum "consistently outperforms existing methods in key metrics such as F1-score and rank correlation," but the paper's own numbers contradict this. On TVSum (Table 2), HierSum's F1 of 62.5 is below CLIP-It (64.2), iPTNet (63.4), and A2Summ (63.4); only the rank correlations are best among the comparable methods. On WikiHow (Table 5), HierSum trained on WikiHow achieves F1 48.17 and rank correlations 0.08/0.10, far below IV-Sum (67.3, 0.101, 0.212) and CLIP-It with ASR (62.5, 0.093, 0.191). The abstract and conclusion must be revised to specify exactly which datasets and metrics show improvements, or additional experiments must be provided to support the unqualified claim.
  2. [Section 4.5, Table 5 (WikiHow)] The WikiHow comparison is not apples-to-apples. The text states that the grayed-out baselines use segments of 32 frames sampled at 8 FPS and convert frame-level scores to shot-level scores via 0/1 knapsack, while HierSum selects the top 55% of frames. Under these different protocols, the IV-Sum and CLIP-It numbers in the same table cannot be directly compared with HierSum's numbers. To support any claim of superiority on WikiHow, the authors must either rerun the baselines under the same segment-and-selection protocol or explicitly present the comparison as protocol-different and refrain from drawing comparative conclusions from it. As written, the only valid same-protocol comparison is HierSum trained on pseudo summaries versus HierSum trained on the curated WikiHow data, which supports the pre-training contribution but not state-of-the-art performance.
  3. [Section 4.5, Tables 2-5] All reported results are single numbers without variance, confidence intervals, or significance tests. The claimed improvements are small, e.g., Kendall's tau 0.172 versus A2Summ's 0.150 on TVSum and F1 58.16 versus PGL-SUM's 56.89 on Mr.HiSum, and these differences could easily be within run-to-run noise. The paper should report results over multiple random seeds or provide bootstrap confidence intervals for the main metrics, and the word "significantly" in the abstract and contribution list should be avoided unless a statistical test supports it.
  4. [Section 3 (Most Replayed Statistics) and Section 4.1 (Mr.HiSum), Tables 4 and 7] The Mr.HiSum evaluation is partly circular. The Mr.HiSum ground-truth importance scores are themselves constructed from YouTube most-replayed statistics, which is the same signal used as HierSum's supervision. Table 4 therefore largely measures how well the model fits its training signal rather than how well the summaries agree with independent human judgments. The zero-shot transfer results in Table 7 (F1 44.7 versus 57.7 after fine-tuning) also show that the learned scores do not transfer cleanly to Mr.HiSum's Inception-feature domain. The paper should explicitly acknowledge this circularity and provide a validation that does not rely on the same statistic, such as a small human study or an evaluation on a human-annotated benchmark like SumMe.
minor comments (6)
  1. [Section 4.5] The text contains the typo "HierSumm" in the TVSum paragraph; it should read "HierSum."
  2. [Table 2] The DSNet-AF row reports rho as ".0138", which appears to be a typo for "0.138" or similar; please correct.
  3. [Equation (4) and surrounding text] The text says hyperparameters α, β, and γ control the loss trade-off, but Equation (4) also includes λ for the intra-modality contrastive loss; the list of hyperparameters should be completed and their values given consistently with Table 1, which currently reports only β and λ.
  4. [Section 4.2] The dataset curation description uses terms such as "scrape" and "crawl" but does not report the final number of videos, the filtering statistics, or a train/validation/test split; please add these details for reproducibility.
  5. [Throughout] The dataset name is spelled inconsistently as both "BLISS" and "BLiSS"; please standardize to "BLiSS" as used in the dataset references and tables.
  6. [Figure 3] The caption mentions "ASR only" but the figure legend and axis labels are not fully defined; please add a clear legend and describe the meaning of step = 0 and step = 1 in the caption.

Circularity Check

1 steps flagged · score 2.0 of 10

Minor circularity: Mr.HiSum ground truth is the same most-replayed statistic used as training supervision, so the Mr.HiSum improvement cannot independently validate the most-replayed-to-summarization assumption; no equation-level circularity elsewhere.

  1. fitted input called prediction [Section 4.1, Datasets (Mr.HiSum); see also Section 3, Most Replayed Statistics]
    "This dataset leverages the most replayed statistics as importance scores, which are then aggregated to shot-level using KTS [42] boundary information."

    The Mr.HiSum benchmark's ground-truth importance scores are the YouTube most-replayed statistic, and Section 3 defines the model's training supervision as the same statistic: the loss includes L_mse on the 'ground-truth relevancy score', with frames whose relevancy score is at least 0.15 labeled important. Table 4 then reports F1/MAP on Mr.HiSum as evidence that the most-replayed supervision transfers to summarization.

full rationale

No equation-level circularity is present: the model is trained with a well-defined loss (focal classification loss, MSE on relevance scores, and contrastive losses) and evaluated on external video summarization benchmarks. The abstract's overstatement relative to Tables 2 and 5 is an internal-consistency/correctness issue, not a circular-derivation issue, so it is not scored here. The self-citations in the reference list are not load-bearing: they support auxiliary design choices (contrastive learning, action anticipation) and the main architecture follows A2Summ and Mr.HiSum, which are external works. The one genuine circularity concern is the Mr.HiSum evaluation: the dataset's ground truth is itself derived from the most-replayed statistic used as the training signal, so the Mr.HiSum numbers cannot independently validate the hypothesized correlation between replay statistics and human-judged summarization importance. The TVSum, BLiSS, and WikiHow results use different annotation sources (human importance scores, thumbnail-derived keyframes, and wikiHow article image-frame matching, respectively) and therefore provide independent evidence. Overall, the circularity is partial and localized to the Mr.HiSum benchmark interpretation, yielding a low score of 2.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

No genuinely new entities are introduced; the "most replayed" signal is an existing YouTube statistic. The ledger is dominated by the unvalidated assumption that this statistic marks summary-worthy segments, plus per-dataset hyperparameters and the unverified transferability of pretraining.

free parameters (3)
  • Relevance threshold for binary importance labels = 0.15
    Frames with most-replayed relevance score >= 0.15 are labeled as important for the focal loss; no sensitivity analysis is provided. Section 3, Loss Function.
  • Global step interval m = 2 for TVSum/BLiSS/WikiHow, 5 for Mr.HiSum
    Number of local subtitle batches per one global instruction batch; tuned per dataset via Figure 3, affecting the central training protocol.
  • Loss weights alpha, beta, lambda = alpha not reported; beta=0.1/0.01/1/1, lambda=1/0.001/1/1 per dataset (Table 1)
    Hyperparameters controlling MSE, inter and intra contrastive terms; set per dataset without a stated selection procedure; gamma (focal) also not reported.
assumptions (4)
  • domain assumption Most-replayed statistics correlate with summary importance.
    Stated as "We posit..." in Section 3; it is the supervision signal and is not validated against human importance scores in the paper.
  • domain assumption ASR subtitles and global written instructions provide complementary and temporally usable information.
    The whole hierarchical training relies on subtitle timestamps for alignment and on article descriptions matching the video; Figure 1 illustrates the complementarity but no systematic check is done.
  • domain assumption Pretraining on the curated WikiHow/EHow dataset transfers to TVSum, BLiSS, Mr.HiSum, and WikiHow test sets.
    The claimed gains come from fine-tuning after pretraining; cross-dataset experiments in Tables 6-7 show zero-shot performance is much weaker, so transfer is only demonstrated through fine-tuning.
  • standard math Transformer architectures and contrastive losses behave as standard.
    Borrowed from prior work (A2Summ, HierVL) without modification.

how reviews work

0 comments
Cite this review

Pith. "Pith review of HierSum: A Global and Local Attention Mechanism for Video Summarization." pith.science (2026). https://pith.science/paper/3AO2GECY

@misc{pith2026250418689,
  author       = {Pith},
  title        = {Pith review of: HierSum: A Global and Local Attention Mechanism for Video Summarization},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/3AO2GECY}},
  note         = {Machine review of arXiv:2504.18689}
}
read the original abstract

Video summarization creates an abridged version (i.e., a summary) that provides a quick overview of the video while retaining pertinent information. In this work, we focus on summarizing instructional videos and propose a method for breaking down a video into meaningful segments, each corresponding to essential steps in the video. We propose \textbf{HierSum}, a hierarchical approach that integrates fine-grained local cues from subtitles with global contextual information provided by video-level instructions. Our approach utilizes the ``most replayed" statistic as a supervisory signal to identify critical segments, thereby improving the effectiveness of the summary. We evaluate on benchmark datasets such as TVSum, BLiSS, Mr.HiSum, and the WikiHow test set, and show that HierSum consistently outperforms existing methods in key metrics such as F1-score and rank correlation. We also curate a new multi-modal dataset using WikiHow and EHow videos and associated articles containing step-by-step instructions. Through extensive ablation studies, we demonstrate that training on this dataset significantly enhances summarization on the target datasets.

Figures

Figures reproduced from arXiv: 2504.18689 by the authors.

Figure 1
Figure 1. Instructional video on “How to Bake Salmon” from [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Overview of HierSum: in the sub-clip level (child level) training, with N frames and M subtitles as input, the model predicts the important frames and sentences as summaries. During the parent-level training, the subtitles are replaced with global descriptions and the model is trained only to predict important video frames. Note that the model F is common and is trained in both stages. When training with most replay… view at source ↗
Figure 3
Figure 3. Performance comparison for different global and local [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Qualitative Results. We show summaries from our method HierSum trained on Pseduo Summaries [38] and our dataset. HierSum, when trained on our dataset assigns higher scores to all the frames relevant, and lower scores that aren’t crucial to the task, e.g WikiHow logo. W…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

65 extracted references · 45 canonical work pages

  1. [1]

    End-to-End Multimodal Representation Learning for Video Dialog

    Huda Alamri, Anthony Bilic, Michael Hu, Apoorva Beedu, and Irfan Essa. End-to-end multimodal representation learn- ing for video dialog.arXiv preprint arXiv:2210.14512, 2022. 2

    work page Pith review arXiv 2022

  2. [2]

    Combining global and local attention with positional encoding for video summarization

    Evlampios Apostolidis, Georgios Balaouras, Vasileios Mezaris, and Ioannis Patras. Combining global and local attention with positional encoding for video summarization. In 2021 IEEE international symposium on multimedia (ISM), pages 226–234. IEEE, 2021. 2, 6

    work page 2021

  3. [3]

    Summarizing videos using con- centrated attention and considering the uniqueness and diver- sity of the video frames

    Evlampios Apostolidis, Georgios Balaouras, Vasileios Mezaris, and Ioannis Patras. Summarizing videos using con- centrated attention and considering the uniqueness and diver- sity of the video frames. In Proceedings of the 2022 inter- national conference on multimedia retrieval, pages 407–415,

    work page 2022

  4. [4]

    Hiervl: Learning hierarchical video- language embeddings

    Kumar Ashutosh, Rohit Girdhar, Lorenzo Torresani, and Kristen Grauman. Hiervl: Learning hierarchical video- language embeddings. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 23066–23078, 2023. 2

    2023

  5. [5]

    Raikwar, and Charul Bhatnagar

    Neeraj Baghel, Suresh C. Raikwar, and Charul Bhatnagar. Image conditioned keyframe-based video summarization us- ing object detection, 2020. 1

    work page 2020

  6. [6]

    Text descriptions of actions and objects improve action anticipa- tion

    Apoorva Beedu, Harish Haresamudram, and Irfan Essa. Text descriptions of actions and objects improve action anticipa- tion. In ICASSP 2025-2025 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1–5. IEEE, 2025. 2, 4

    work page 2025

  7. [7]

    Leveraging contextual cues for generating basketball high- lights

    Vinay Bettadapura, Caroline Pantofaru, and Irfan Essa. Leveraging contextual cues for generating basketball high- lights. In Proceedings of the 24th ACM international confer- ence on Multimedia, pages 908–917, 2016. 2

    work page 2016

  8. [8]

    Videosage: Video summarization with graph representation learning

    Jose M Rojas Chaves and Subarna Tripathi. Videosage: Video summarization with graph representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vi- sion and Pattern Recognition , pages 2527–2534, 2024. 2, 5

    work page 2024

Show all 65 references
  1. [9]

    Multimodal contrastive learning with hard negative sampling for human activity recognition

    Hyeongju Choi, Apoorva Beedu, and Irfan Essa. Multimodal contrastive learning with hard negative sampling for human activity recognition. arXiv preprint arXiv:2309.01262, 2023. 4

    2023 arXiv

  2. [10]

    Vsumm: A mechanism designed to produce static video summaries and a novel evaluation method

    Sandra Eliza Fontes De Avila, Ana Paula Brandao Lopes, Antonio da Luz Jr, and Arnaldo de Albuquerque Ara ´ujo. Vsumm: A mechanism designed to produce static video summaries and a novel evaluation method. Pattern recog- nition letters, 32(1):56–68, 2011. 2

    2011

  3. [11]

    Video summarization via segments summary graphs

    Mahmut Demir and H Isil Bozma. Video summarization via segments summary graphs. InProceedings of the IEEE Inter- national Conference on Computer Vision Workshops, pages 19–25, 2015. 1

    2015

  4. [12]

    Bert: Pre-training of deep bidirectional trans- formers for language understanding

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional trans- formers for language understanding. In Proceedings of the 2019 conference of the North American chapter of the asso- ciation for computational linguistics: human l...

    2019

  5. [13]

    Mamba fusion: Learning actions through questioning

    Zhikang Dong, Apoorva Beedu, Jason Sheinkopf, and Irfan Essa. Mamba fusion: Learning actions through questioning. arXiv preprint arXiv:2409.11513, 2024. 2

    2024 arXiv

  6. [14]

    Summarizing videos with attention

    Jiri Fajtl, Hajar Sadeghi Sokeh, Vasileios Argyriou, Dorothy Monekosso, and Paolo Remagnino. Summarizing videos with attention. In Computer Vision–ACCV 2018 Workshops: 14th Asian Conference on Computer Vision, Perth, Australia, December 2–6, 2018, Revised Selected Papers 14, pages 39–

    2018

  7. [15]

    Slowfast networks for video recognition

    Christoph Feichtenhofer, Haoqi Fan, Jitendra Malik, and Kaiming He. Slowfast networks for video recognition. In Proceedings of the IEEE/CVF international conference on computer vision, pages 6202–6211, 2019. 2

    2019

  8. [16]

    Youtube video player updates: Most replayed, video chapters, single loop & more

    Google. Youtube video player updates: Most replayed, video chapters, single loop & more. https : / / support . google . com / youtube / thread / 164099151 / youtube - video - player - updates - most - replayed - video - chapters - single-loop-more, 2022. 4

    2022

  9. [17]

    Vtg-llm: Integrating timestamp knowledge into video llms for enhanced video temporal grounding

    Yongxin Guo, Jingyu Liu, Mingda Li, Dingxin Cheng, Xi- aoying Tang, Dianbo Sui, Qingbin Liu, Xi Chen, and Kevin Zhao. Vtg-llm: Integrating timestamp knowledge into video llms for enhanced video temporal grounding. arXiv preprint arXiv:2405.13382, 2024. 2

    2024 arXiv

  10. [18]

    Creating summaries from user videos

    Michael Gygli, Helmut Grabner, Hayko Riemenschneider, and Luc Van Gool. Creating summaries from user videos. In Computer Vision–ECCV 2014: 13th European Confer- ence, Zurich, Switzerland, September 6-12, 2014, Proceed- ings, Part VII 13, pages 505–520. Springer, 2014. 1, 2

    2014

  11. [19]

    Shot2story20k: A new benchmark for comprehen- sive understanding of multi-shot videos

    Mingfei Han, Xiaojun Chang, Heng Wang, and Linjie Yang. Shot2story20k: A new benchmark for comprehen- sive understanding of multi-shot videos. arXiv preprint arXiv:2312.10300, 2023. 1, 2

    2023 arXiv

  12. [20]

    Limitations in employing natural language supervision for sensor-based human activity recognition-and ways to overcome them

    Harish Haresamudram, Apoorva Beedu, Mashfiqui Rabbi, Sankalita Saha, Irfan Essa, and Thomas Ploetz. Limitations in employing natural language supervision for sensor-based human activity recognition-and ways to overcome them. In Proceedings of the AAAI Conference on Artificial ...

    2025

  13. [21]

    Align and attend: Multimodal summarization with dual contrastive losses

    Bo He, Jun Wang, Jielin Qiu, Trung Bui, Abhinav Shrivas- tava, and Zhaowen Wang. Align and attend: Multimodal summarization with dual contrastive losses. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pat- tern Recognition, pages 14867–14878, 2023. 1, 2, 3, 4, 5, 6

    2023

  14. [22]

    V2xum-llm: Cross-modal video summarization with tem- poral prompt instruction tuning

    Hang Hua, Yunlong Tang, Chenliang Xu, and Jiebo Luo. V2xum-llm: Cross-modal video summarization with tem- poral prompt instruction tuning. In Proceedings of the AAAI Conference on Artificial Intelligence , pages 3599– 3607, 2025. 2, 5

    2025

  15. [23]

    Cluster-based video summa- rization with temporal context awareness

    Hai-Dang Huynh-Lam, Ngoc-Phuong Ho-Thi, Minh-Triet Tran, and Trung-Nghia Le. Cluster-based video summa- rization with temporal context awareness. In Pacific-Rim Symposium on Image and Video Technology , pages 15–28. Springer, 2023. 2

    2023

  16. [24]

    Video summarization with attention-based encoder–decoder networks

    Zhong Ji, Kailin Xiong, Yanwei Pang, and Xuelong Li. Video summarization with attention-based encoder–decoder networks. IEEE Transactions on Circuits and Systems for Video Technology, 30(6):1709–1717, 2019. 2

    2019

  17. [25]

    Joint video summarization and moment localization by cross-task sample transfer

    Hao Jiang and Yadong Mu. Joint video summarization and moment localization by cross-task sample transfer. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16388–16398, 2022. 6

    2022

  18. [26]

    Discriminative feature learning for un- supervised video summarization

    Yunjae Jung, Donghyeon Cho, Dahun Kim, Sanghyun Woo, and In So Kweon. Discriminative feature learning for un- supervised video summarization. In Proceedings of the AAAI Conference on artificial intelligence , pages 8537– 8544, 2019. 2

    2019

  19. [27]

    Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension

    Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvinine- jad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. arXiv preprint arXiv:1910.13461, 2019. 6

    1910 arXiv

  20. [28]

    Multi-modal summarization for asyn- chronous collection of text, image, audio and video

    Haoran Li, Junnan Zhu, Cong Ma, Jiajun Zhang, and Chengqing Zong. Multi-modal summarization for asyn- chronous collection of text, image, audio and video. In Pro- ceedings of the 2017 conference on empirical methods in nat- ural language processing, pages 1092–1102, 2017. 2

    2017

  21. [29]

    Read, watch, listen, and summarize: Multi-modal summarization for asynchronous text, image, audio and video

    Haoran Li, Junnan Zhu, Cong Ma, Jiajun Zhang, and Chengqing Zong. Read, watch, listen, and summarize: Multi-modal summarization for asynchronous text, image, audio and video. IEEE Transactions on Knowledge and Data Engineering, 31(5):996–1009, 2018. 2

    2018

  22. [30]

    Hero: Hierarchical encoder for video+ language omni-representation pre-training

    Linjie Li, Yen-Chun Chen, Yu Cheng, Zhe Gan, Licheng Yu, and Jingjing Liu. Hero: Hierarchical encoder for video+ language omni-representation pre-training. arXiv preprint arXiv:2005.00200, 2020. 2

    2005 arXiv

  23. [31]

    Rouge: A package for automatic evaluation of summaries

    Chin-Yew Lin. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out , pages 74–81, 2004. 5

    2004

  24. [32]

    Videoxum: Cross- modal visual and textural summarization of videos

    Jingyang Lin, Hang Hua, Ming Chen, Yikang Li, Jenhao Hsiao, Chiuman Ho, and Jiebo Luo. Videoxum: Cross- modal visual and textural summarization of videos. IEEE Transactions on Multimedia, 26:5548–5560, 2023. 2

    2023

  25. [33]

    Univtg: Towards unified video- language temporal grounding

    Kevin Qinghong Lin, Pengchuan Zhang, Joya Chen, Shra- man Pramanick, Difei Gao, Alex Jinpeng Wang, Rui Yan, and Mike Zheng Shou. Univtg: Towards unified video- language temporal grounding. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 2794–...

    2023

  26. [34]

    Focal loss for dense object detection

    Tsung-Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Doll´ar. Focal loss for dense object detection. In Pro- ceedings of the IEEE international conference on computer vision, pages 2980–2988, 2017. 4

    2017

  27. [35]

    Ubiss: A unified frame- work for bimodal semantic summarization of videos

    Yuting Mei, Linli Yao, and Qin Jin. Ubiss: A unified frame- work for bimodal semantic summarization of videos. InPro- ceedings of the 2024 International Conference on Multime- dia Retrieval, pages 1034–1042, 2024. 2

    2024

  28. [36]

    Leveraging bert for extractive text summariza- tion on lectures

    Derek Miller. Leveraging bert for extractive text summariza- tion on lectures. arXiv preprint arXiv:1906.04165, 2019. 6

    1906 arXiv

  29. [37]

    Clip-it! language-guided video summarization

    Medhini Narasimhan, Anna Rohrbach, and Trevor Darrell. Clip-it! language-guided video summarization. Advances in neural information processing systems, 34:13988–14000,

  30. [38]

    Tl; dw? summarizing instructional videos with task relevance and cross-modal saliency

    Medhini Narasimhan, Arsha Nagrani, Chen Sun, Michael Rubinstein, Trevor Darrell, Anna Rohrbach, and Cordelia Schmid. Tl; dw? summarizing instructional videos with task relevance and cross-modal saliency. InEuropean Conference on Computer Vision, pages 540–557. Springer, 2022. ...

    2022

  31. [39]

    Video summarization using deep se- mantic features

    Mayu Otani, Yuta Nakashima, Esa Rahtu, Janne Heikkil ¨a, and Naokazu Yokoya. Video summarization using deep se- mantic features. In Computer Vision–ACCV 2016: 13th Asian Conference on Computer Vision, Taipei, Taiwan, November 20-24, 2016, Revised Selected Papers, Part V 13, pa...

    2016

  32. [40]

    Sumgraph: Video summarization via recursive graph modeling

    Jungin Park, Jiyoung Lee, Ig-Jae Kim, and Kwanghoon Sohn. Sumgraph: Video summarization via recursive graph modeling. In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceed- ings, Part XXV 16, pages 647–663. Springer, 2020. 2

    2020

  33. [41]

    Enhancing video summarization via vision-language embed- ding

    Bryan A Plummer, Matthew Brown, and Svetlana Lazebnik. Enhancing video summarization via vision-language embed- ding. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5781–5789, 2017. 2

    2017

  34. [42]

    Category-specific video summarization

    Danila Potapov, Matthijs Douze, Zaid Harchaoui, and Cordelia Schmid. Category-specific video summarization. In Computer Vision–ECCV 2014: 13th European Confer- ence, Zurich, Switzerland, September 6-12, 2014, Proceed- ings, Part VI 13, pages 540–555. Springer, 2014. 5

    2014

  35. [43]

    Liveseg: Unsupervised multi- modal temporal segmentation of long livestream videos

    Jielin Qiu, Franck Dernoncourt, Trung Bui, Zhaowen Wang, Ding Zhao, and Hailin Jin. Liveseg: Unsupervised multi- modal temporal segmentation of long livestream videos. In Proceedings of the IEEE/CVF Winter Conference on Appli- cations of Computer Vision, pages 5188–5198, 2023. 2

    2023

  36. [44]

    Learning transferable visual models from natural language supervi- sion

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervi- sion. In International conference on machine learning, ...

    2021

  37. [45]

    Robust speech recognition via large-scale weak supervision

    Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey, and Ilya Sutskever. Robust speech recognition via large-scale weak supervision. In Interna- tional conference on machine learning, pages 28492–28518. PMLR, 2023. 5

    2023

  38. [46]

    Sentence-bert: Sentence embeddings using siamese bert-networks

    Nils Reimers and Iryna Gurevych. Sentence-bert: Sentence embeddings using siamese bert-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Lan- guage Processing. Association for Computational Linguis- tics, 2019. 3, 4, 5

    2019

  39. [47]

    Everything at once-multi- modal fusion transformer for video retrieval

    Nina Shvetsova, Brian Chen, Andrew Rouditchenko, Samuel Thomas, Brian Kingsbury, Rogerio S Feris, David Harwath, James Glass, and Hilde Kuehne. Everything at once-multi- modal fusion transformer for video retrieval. In Proceedings of the ieee/cvf conference on computer vision ...

    2022

  40. [48]

    Tvsum: Summarizing web videos using titles

    Yale Song, Jordi Vallmitjana, Amanda Stent, and Alejandro Jaimes. Tvsum: Summarizing web videos using titles. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5179–5187, 2015. 1, 2, 4, 5

    2015

  41. [49]

    Jinhwan Sul, Jihoon Han, and Joonseok Lee. Mr. hisum: a large-scale dataset for video highlight detection and summa- rization. Advances in Neural Information Processing Sys- tems, 36, 2024. 1, 2, 3, 4, 5

    2024

  42. [50]

    Videobert: A joint model for video and language representation learning

    Chen Sun, Austin Myers, Carl V ondrick, Kevin Murphy, and Cordelia Schmid. Videobert: A joint model for video and language representation learning. InProceedings of the IEEE International Conference on Computer Vision, pages 7464– 7473, 2019. 2

    2019

  43. [51]

    Rethinking the inception archi- tecture for computer vision

    Christian Szegedy, Vincent Vanhoucke, Sergey Ioffe, Jon Shlens, and Zbigniew Wojna. Rethinking the inception archi- tecture for computer vision. In Proceedings of the IEEE con- ference on computer vision and pattern recognition , pages 2818–2826, 2016. 5

    2016

  44. [52]

    Large model based sequential keyframe extrac- tion for video summarization, 2024

    Kailong Tan, Yuxiang Zhou, Qianchen Xia, Rui Liu, and Yong Chen. Large model based sequential keyframe extrac- tion for video summarization, 2024. 1

    2024

  45. [53]

    Gomez, Lukasz Kaiser, and Illia Polosukhin

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszko- reit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need, 2023. 2

    2023

  46. [54]

    Springer, 2019. 2, 6

    2019

  47. [55]

    Intentvizor: Towards generic query guided interactive video summariza- tion

    Guande Wu, Jianzhe Lin, and Claudio T Silva. Intentvizor: Towards generic query guided interactive video summariza- tion. In Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition , pages 10503–10512,

  48. [56]

    Video repurposing from user generated con- tent: A large-scale dataset and benchmark

    Yongliang Wu, Wenbo Zhu, Jiawang Cao, Yi Lu, Bozheng Li, Weiheng Chi, Zihan Qiu, Lirian Su, Haolin Zheng, Jay Wu, et al. Video repurposing from user generated con- tent: A large-scale dataset and benchmark. arXiv preprint arXiv:2412.08879, 2024. 2

    2024 arXiv

  49. [57]

    Cross-category video high- light detection via set-based learning

    Minghao Xu, Hang Wang, Bingbing Ni, Riheng Zhu, Zhen- bang Sun, and Changhu Wang. Cross-category video high- light detection via set-based learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 7970–7979, 2021. 6

    2021

  50. [58]

    Unloc: A unified framework for video localiza- tion tasks

    Shen Yan, Xuehan Xiong, Arsha Nagrani, Anurag Arnab, Zhonghao Wang, Weina Ge, David Ross, and Cordelia Schmid. Unloc: A unified framework for video localiza- tion tasks. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 13623–13633, 2023. 1

    2023

  51. [59]

    Hierarchical self-supervised augmented knowledge dis- tillation

    Chuanguang Yang, Zhulin An, Linhang Cai, and Yongjun Xu. Hierarchical self-supervised augmented knowledge dis- tillation. arXiv preprint arXiv:2107.13715, 2021. 2

    2021 arXiv

  52. [60]

    Bert representations for video question answering

    Zekun Yang, Noa Garcia, Chenhui Chu, Mayu Otani, Yuta Nakashima, and Haruo Takemura. Bert representations for video question answering. In The IEEE Winter Confer- ence on Applications of Computer Vision, pages 1556–1565,

  53. [61]

    Hierarchical multi-modal video summariza- tion with dynamic sampling

    Lingjian Yu, Xing Zhao, Liang Xie, Haoran Liang, and Ronghua Liang. Hierarchical multi-modal video summariza- tion with dynamic sampling. IET Image Processing, 2024. 2

    2024

  54. [62]

    Hierarchical video-moment retrieval and step-captioning

    Abhay Zala, Jaemin Cho, Satwik Kottur, Xilun Chen, Bar- las Oguz, Yashar Mehdad, and Mohit Bansal. Hierarchical video-moment retrieval and step-captioning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pat- tern Recognition, pages 23056–23065, 2023. 2

    2023

  55. [63]

    Hsa-rnn: Hier- archical structure-adaptive rnn for video summarization

    Bin Zhao, Xuelong Li, and Xiaoqiang Lu. Hsa-rnn: Hier- archical structure-adaptive rnn for video summarization. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 7405–7414, 2018. 2

    2018

  56. [64]

    Deep reinforce- ment learning for unsupervised video summarization with diversity-representativeness reward

    Kaiyang Zhou, Yu Qiao, and Tao Xiang. Deep reinforce- ment learning for unsupervised video summarization with diversity-representativeness reward. In Proceedings of the AAAI conference on artificial intelligence, 2018. 2

    2018

  57. [65]

    Dsnet: A flexible detect-to-summarize network for video summariza- tion

    Wencheng Zhu, Jiwen Lu, Jiahao Li, and Jie Zhou. Dsnet: A flexible detect-to-summarize network for video summariza- tion. IEEE Transactions on Image Processing, 30:948–962,

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.