SST-EM: Advanced Metrics for Evaluating Semantic, Spatial and Temporal Aspects in Video Editing

Bharat Chanderprakash Kathuria; Jialu Li; Varun Biyyala; Youshan Zhang

REVIEW 4 major objections 5 minor 35 references

SST-EM: Advanced Metrics for Evaluating Semantic, Spatial and Temporal Aspects in Video Editing

T0 review · 4 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read SST-EM claims an automated score can rank edited videos in near-perfect agreement with human quality judgments, reporting a 0.962 Pearson correlation.

desk verdict Sensible component choices and public code, but the headline correlation is an in-sample fit and the temporal formula contradicts itself; needs major revision, not desk rejection. read the letter →

arxiv 2501.07554 v1 pith:Y33XWNX2 submitted 2025-01-13 cs.CV cs.CL

Varun Biyyala , Bharat Chanderprakash Kathuria , Jialu Li , Youshan Zhang This is my paper · ORCID

classification cs.CVcs.CL

keywords videoeditingevaluationsemanticsimilaritytemporalconsistencyobjectdetectionvision-languagemodelshumancorrelationregressionweightingSST-EM

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper proposes SST-EM, an automated score for judging edited videos, and claims it tracks human-quality judgments more closely than existing CLIP-text or EditBoard-style metrics. The metric combines three signals: how well frames semantically match the editing prompt via a vision-language model, whether the primary object stays present via text-conditioned object detection guided by an LLM, and how smoothly consecutive frames change via vision-transformer embedding similarity. These three sub-scores are combined into one weighted formula, with the weights fit by linear regression to human ratings on a small optimization set. The paper reports a Pearson correlation of 0.962 between the final score and human evaluation, with perfect Spearman and Kendall rank agreement, and concludes that a fully automated pipeline can stand in for human assessment of video edits.

What carries the argument

The load-bearing object is the final score formula $$S_{\mathrm{final}} = w_1 S_{\mathrm{similarity}} + w_2 S_{\mathrm{object\ detection}} + w_3 (1 - S_{\mathrm{temporal}}),$$ with weights $w_1,w_2,w_3$ obtained by least-squares regression against human evaluation scores. Each component is itself a frame-averaged similarity: PaliGemma captions compared with the editing prompt by cosine similarity, Grounding DINO confidence for the LLM-selected primary object averaged over frames, and cosine similarity between Vision Transformer embeddings of consecutive frames. The $1 - S_{\mathrm{temporal}}$ term converts high frame-to-frame similarity into a high final score, and the regression step is what turns three generic model outputs into a calibrated approximation of human preference.

What would settle it

Re-run the evaluation on a larger, independently labeled set of edited videos, say at least 100 videos per model with documented inter-rater agreement, and recompute the Pearson, Spearman, and Kendall correlations; if the SST-EM correlations drop substantially or if the fitted weights change dramatically across random halves of the data, the claim that the metric reproduces human judgment at 0.962 would be falsified. A cheaper check is to compute correlations within the 40-video optimization set and see whether the reported 1.000 rank correlations are an artifact of near-tied scores across the handful of editing models.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that a composite of three per-frame computations—caption-prompt cosine similarity, text-conditioned object-detection confidence, and consecutive-frame embedding similarity—can be weight-fitted to reproduce human judgments of video-edit quality. The reported headline numbers are a Pearson correlation of 0.962 between the SST-EM final score and human evaluation scores, alongside Spearman and Kendall correlations of 1.000, which the authors read as evidence that the metric ranks edited videos essentially as humans do. The paper also claims the component scores illuminate why the whole works: temporal consistency shows the strongest individual correlation with human judgment, with a 0.927 Pearson correlation and perfect rank agreement, followed by object-detection score at 0.835, while raw context similarity alone is weak at 0.072 Pearson, showing that the temporal term carries much of the load.

Load-bearing premise

The reliability of the human evaluation scores used as ground truth: the paper reports no rater count, annotation instructions, or inter-rater agreement, and the weights are fitted on only 40 videos, so if those human scores are noisy or biased, every reported correlation inherits that noise.

Editorial extensions

If this is right

SST-EM can serve as a drop-in automated replacement for human evaluation in routine video-editing benchmarks, giving a single scalar that ranks models much as humans do.
Because the weights are fitted once on human data, new video-editing models can be compared without rerunning costly human studies, provided the new videos resemble the optimization set.
Researchers can decompose a model's SST-EM score into semantic, object, and temporal components to diagnose why a model ranks low, such as a low temporal sub-score flagging jittery or abrupt edits.
The component-wise correlations suggest future metrics should weight temporal consistency heavily rather than relying only on CLIP-style text-image alignment.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Editorial inference: the same three-component recipe could be adapted to other video-understanding tasks, such as evaluating text-to-video generation or video summarization, by re-running the regression against whatever human judgments are available for that task.
Editorial inference: weights fitted on 40 videos are likely unstable; a testable extension is to report bootstrap confidence intervals for $w_1,w_2,w_3$ and for the final correlation, which would show how much of the 0.962 reflects the specific optimization sample.
Editorial inference: because the temporal sub-score alone reaches a 0.927 Pearson correlation, a simpler metric built only from consecutive-frame embedding similarity might capture most of the signal, and SST-EM's added complexity would need to be justified on larger datasets with head-to-head comparisons.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Quick take: SST-EM is a plausible engineering contribution — a weighted combination of PaliGemma caption similarity, Grounding DINO object confidence, and ViT frame-embedding similarity, with the object prompt refined by an LLM agent — and the code is public. The problem is that the paper's central claim, a 0.962 Pearson and 1.0 Spearman/Kendall correlation with human scores, is not shown to be out-of-sample. Section 4.5 fits w1–w3 by least squares against human scores on the optimization set, and Section 4.6 computes Pearson on the same M videos. The validation set introduced in Section 3.1 never appears in the results. With 40 videos and 3 fitted weights, a high in-sample correlation is exactly what you'd expect; it doesn't demonstrate alignment with human judgment.

What's genuinely new is the integration itself: nobody else, as far as I can tell, has chained these specific components into a single video-editing metric, and the LLM-driven object refinement is a nice touch. The Table 3 breakdown, with each component's correlation, is a useful diagnostic. But the numbers as reported can't be trusted. The fitted weights are never given, so I can't even check whether the sign of w3 compensates for the temporal term. Speaking of which, Section 4.4 writes S_final = ... + w3·(S1−temporal), and the text says subtracting from 1 ensures higher temporal consistency yields a higher final score. As written, it does the opposite unless w3 is negative. That's a load-bearing inconsistency, not a typo — the entire claim about temporal handling rests on it.

There are also smaller problems: the human evaluation protocol is undocumented (no rater count, no inter-rater agreement, despite the related work calling for Krippendorff's alpha), Table 5 gives per-model human scores but Table 3 correlations presumably use per-video data with no sample size stated, and the 'R1 score' in 4.6 is presumably R² and is never reported. The citations are sloppy — [22] and [11] are mis-attributed — but that's minor.

Who's this for? Anyone building automated video-editing benchmarks. The paper is useful as a cautionary example of fitting-then-claiming, and the component choices are worth copying. But it needs a major revision: report the weights, run a true held-out validation split, correct the temporal formula, and document the human rating protocol. I'd send it to review with the expectation of heavy revision, not desk-reject it — the underlying idea has merit and the code is there for the referees to check.

Referee Report

4 major / 5 minor

Summary. The paper proposes SST-EM, a composite metric for evaluating text-based video editing, combining a PaliGemma-based semantic similarity score, a Grounding DINO object-detection score, and a ViT-based temporal consistency score. The three components are combined as a weighted sum, with weights fit by least-squares regression to human evaluation scores on a 40-video optimization set. The central claim is that SST-EM achieves the highest correlations with human evaluation scores among compared metrics, with a Pearson correlation of 0.962 (Table 3). The paper also reports ablation-style comparisons across several video editing models.

Significance. If the central claim were rigorously established, SST-EM would be a useful contribution to video-editing evaluation, where automated metrics that align with human judgment are genuinely needed. The component choices are reasonable and draw on modern tools (PaliGemma, Grounding DINO, ViT). However, the paper's headline evidence is currently in-sample and circular: the weights are fit to human scores and then correlated with those same scores, with no reported validation-set result. The temporal term in the final formula also has an apparent sign error. These issues make the current manuscript unsuitable for publication without substantial revision.

major comments (4)

[§4.5–4.6, §5.1, Table 3] The headline result, Pearson ρ = 0.962 in Table 3, is not shown to be out-of-sample. Section 4.5 fits w1, w2, w3 by minimizing squared error against human scores on the optimization set, and Section 4.6 computes the Pearson correlation over the same M videos used for training. The Validation Set introduced in Section 3.1 is never tied to the correlations in Table 3, and the fitted weights are never reported. With only 40 videos and 3 free parameters, a high in-sample correlation is expected and does not substantiate the claim in Section 5.5 that SST-EM 'exhibits the highest correlations' with human judgment. The authors must report the validation-set correlation, the sample size for that set, and the fitted weights.
[§4.4, Eq. (4)] The final-score formula S_final = w1·S_similarity + w2·S_object + w3·(1−S_temporal) is presented with the statement that subtracting S_temporal from 1 'ensures that a higher temporal consistency score yields a higher final score.' This is directionally wrong for positive w3: higher S_temporal makes (1−S_temporal) smaller, so the temporal term decreases. Unless the fitted w3 is negative, which is never stated or reported, the formula contradicts the paper's own description. This must be corrected and the sign of w3 disclosed.
[§3.1, §2] The human evaluation scores that serve as regression targets are described only as 'subjective ratings collected from multiple individuals.' The paper gives no rater count, annotation instructions, judgment scale, or inter-rater reliability, despite Section 2 explicitly noting that Krippendorff's alpha or Cohen's kappa are needed to ensure reliability. With only 40 videos in the optimization set, noisy or biased ground truth would propagate directly into the fitted weights and every subsequent correlation. This missing information is load-bearing for the validity of the entire evaluation.
[§5.3, §5.4] The text states that SST-EM achieves Spearman and Kendall correlations of 1.000 'with both Imaging Quality and Aesthetic Quality,' but Table 3 reports those perfect correlations against Human Evaluation scores, not against Imaging Quality or Aesthetic Quality. This conflation of the reference variable obscures what is actually being compared and should be corrected.

minor comments (5)

[§4.4] The notation 'w3 ·(S1−temporal)' appears to be a typo for 'w3 ·(1−S_temporal)'; please fix the formula formatting.
[§4.6] The paper refers to an 'R1 score' without defining it; presumably this is R², and if so the symbol should be corrected and its computation described.
[§3, Table 2] The table reports 40 videos and 640 frame-prompt pairs for Weights-Optimization, but 40 videos and 900 pairs for Evaluation; the relationship between these counts and the optimization/validation split is unclear and should be clarified.
[§5.1, Table 3] The Context Similarity Score has Pearson 0.072 and negative Spearman/Kendall correlations with human evaluation; the claim that 'individual components... also show notable correlation' is not supported for this component.
[§1, §2] Some references appear misaligned with their cited claims, e.g., [22] is cited for regression-based metric optimization in video editing but the reference is about gaze-guided cinematic editing; please verify all citations.

Circularity Check

1 steps flagged · score 6.0 of 10

Headline correlation 0.962 is not shown to be out-of-sample: weights are fit to human scores in §4.5, and §4.6 correlates the same fitted scores with the same human scores over the training videos, so the central outperformance claim reduces to a fit unless held-out results are reported.

fitted input called prediction [§4.5–4.6, §5.1, Table 3; cf. §5]
"L = 1/M Σ_j (S_final^(j) − S_human^(j))^2, where ... M is the number of videos in the dataset used for training (§4.5); We calculate the Pearson correlation coefficient ρ between the predicted final scores and the human evaluation scores: ρ = Σ_{i=1}^M ... (§4.6); including SST-EM, which is based on the Human Evaluation Score (§5)."

The weights w1, w2, w3 are least-squares fitted to human scores by minimizing L over the training set, so S_final is constructed to approximate S_human on those videos. §4.6 then evaluates agreement by Pearson correlation between the same S_final and S_human over the same M training videos. Table 3 reports SST-EM Pearson 0.962, Spearman 1.000, Kendall 1.000 without stating that these are computed on the Validation Set defined in §3.1/§4.5, and no held-out correlations are reported anywhere. In-sample correlation of a regressed weighted sum with its own target is expected by construction, so the headline claim that SST-EM 'exhibits the highest correlations with the Human Evaluation scores' is, as presented, a fitted-input-called-prediction.

full rationale

The derivation chain is largely self-contained: the semantic, object-detection, and temporal components are computed from external models (PaliGemma, Grounding DINO, ViT) and are not circular. No load-bearing self-citation or imported uniqueness theorem was found. The circular step is confined to the final-score validation: weights are fit to human evaluation scores, and the paper's headline evidence is the correlation of that fitted score with the same human scores, with no reported held-out result. The paper defines Optimization and Validation sets but never states which set produced Table 3; §4.6's correlation formula sums over M training videos, making the central outperformance claim partially circular as written. Human-evaluation reliability (no rater count or inter-rater agreement reported) is a separate data-quality risk, not a circularity.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

The metric's central claim depends on three fitted weights and several domain assumptions about what each off-the-shelf model measures. The fitted weights are not reported, so the free-parameter count cannot be fully audited, and no new physical or theoretical entities are introduced.

free parameters (3)

w1 = not reported
Weight on the Context Similarity Score, fit by linear regression to human evaluation scores in Section 4.5.
w2 = not reported
Weight on the Object Detection Score, fit by linear regression to human evaluation scores in Section 4.5.
w3 = not reported
Weight on the temporal term (1 - Stemporal), fit by linear regression to human evaluation scores in Section 4.5.

assumptions (5)

domain assumption Human Evaluation Scores are a reliable, unbiased ground-truth for video editing quality.
Section 3.1 treats human scores as regression targets and as the baseline for all correlations; no rater count, annotation instructions, or inter-rater agreement is reported.
ad hoc to paper Cosine similarity between PaliGemma frame captions and the editing prompt measures semantic fidelity.
Section 4.1 defines Ssimilarity without specifying the text-embedding model or validating that caption-prompt cosine similarity tracks human semantic judgment.
domain assumption Average Grounding DINO confidence over frames measures object-edit fidelity.
Section 4.2 assumes detection confidence for the primary object reflects whether the edited object matches the prompt.
domain assumption Cosine similarity of ViT frame embeddings across consecutive frames measures temporal smoothness.
Section 4.3 defines Stemporal as the mean cosine similarity between consecutive frame embeddings, with no comparison to any temporal-quality ground truth.
ad hoc to paper A linear weighted combination of these three scores can reproduce human judgment.
Section 4.5 fits only linear weights; no evidence is given that the relationship is linear or that these three components are sufficient.

how reviews work

0 comments

Cite this review

Pith. "Pith review of SST-EM: Advanced Metrics for Evaluating Semantic, Spatial and Temporal Aspects in Video Editing." pith.science (2026). https://pith.science/paper/Y33XWNX2

@misc{pith2026250107554,
  author       = {Pith},
  title        = {Pith review of: SST-EM: Advanced Metrics for Evaluating Semantic, Spatial and Temporal Aspects in Video Editing},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/Y33XWNX2}},
  note         = {Machine review of arXiv:2501.07554}
}

read the original abstract

Video editing models have advanced significantly, but evaluating their performance remains challenging. Traditional metrics, such as CLIP text and image scores, often fall short: text scores are limited by inadequate training data and hierarchical dependencies, while image scores fail to assess temporal consistency. We present SST-EM (Semantic, Spatial, and Temporal Evaluation Metric), a novel evaluation framework that leverages modern Vision-Language Models (VLMs), Object Detection, and Temporal Consistency checks. SST-EM comprises four components: (1) semantic extraction from frames using a VLM, (2) primary object tracking with Object Detection, (3) focused object refinement via an LLM agent, and (4) temporal consistency assessment using a Vision Transformer (ViT). These components are integrated into a unified metric with weights derived from human evaluations and regression analysis. The name SST-EM reflects its focus on Semantic, Spatial, and Temporal aspects of video evaluation. SST-EM provides a comprehensive evaluation of semantic fidelity and temporal smoothness in video editing. The source code is available in the \textbf{\href{https://github.com/custommetrics-sst/SST_CustomEvaluationMetrics.git}{GitHub Repository}}.

Figures

Figures reproduced from arXiv: 2501.07554 by the authors.

**Figure 2.** Comparison between Final and Human Eval Results [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗

**Figure 3.** Pipeline for evaluating video editing models using custom metrics: semantic analysis, object detection, and temporal consistency [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗

Figures from the paper (1 more)

**Figure 4.** Figure 4: Comparison of results between different video editing models. The primary object of interest and the original video are high [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

35 extracted references · 24 canonical work pages

[22]

Gazed–gaze-guided cinematic editing of wide-angle monocular video recordings

KL Bhanu Moorthy, Moneish Kumar, Ramanathan Subra- manian, and Vineet Gandhi. Gazed–gaze-guided cinematic editing of wide-angle monocular video recordings. In Pro- ceedings of the 2020 CHI Conference on Human Factors in Computing Systems, pages 1–11, 2020. 2, 3

work page 2020
[11]

Text-based Talking Video Editing with Cascaded Conditional Diffusion

Bo Han, Heqing Zou, Haoyang Li, Guangcong Wang, and Chng Eng Siong. Text-based talking video edit- ing with cascaded conditional diffusion. arXiv preprint arXiv:2407.14841, 2024. 2, 3

work page Pith review arXiv 2024
[1]

Detectron2 object detection & manipulating images using cartooniza- tion

Allena Venkata Sai Abhishek and Sonali Kotni. Detectron2 object detection & manipulating images using cartooniza- tion. Int. J. Eng. Res. Technol.(IJERT), 10:1–5, 2021. 2

work page 2021
[2]

A deep learning framework for quality assessment and restoration in video endoscopy

Sharib Ali, Felix Zhou, Adam Bailey, Barbara Braden, James E East, Xin Lu, and Jens Rittscher. A deep learning framework for quality assessment and restoration in video endoscopy. Medical image analysis, 68:101900, 2021. 3

work page 2021
[3]

Text2live: Text-driven layered image and video editing

Omer Bar-Tal, Dolev Ofri-Amar, Rafail Fridman, Yoni Kas- ten, and Tali Dekel. Text2live: Text-driven layered image and video editing. In European conference on computer vi- sion, pages 707–723. Springer, 2022. 3

work page 2022
[4]

Paligemma: A versatile 3b vlm for trans- fer

Lucas Beyer, Andreas Steiner, Andr ´e Susano Pinto, Alexan- der Kolesnikov, Xiao Wang, Daniel Salz, Maxim Neumann, Ibrahim Alabdulmohsin, Michael Tschannen, Emanuele Bugliarello, et al. Paligemma: A versatile 3b vlm for trans- fer. arXiv preprint arXiv:2407.07726, 2024. 4

arXiv 2024
[5]

Stream- ing video diffusion: Online video editing with diffusion models

Feng Chen, Zhen Yang, Bohan Zhuang, and Qi Wu. Stream- ing video diffusion: Online video editing with diffusion models. arXiv preprint arXiv:2405.19726, 2024. 2

arXiv 2024
[6]

Control-a-video: Controllable text-to-video generation with diffusion models

Weifeng Chen, Yatai Ji, Jie Wu, Hefeng Wu, Pan Xie, Jiashi Li, Xin Xia, Xuefeng Xiao, and Liang Lin. Control-a-video: Controllable text-to-video generation with diffusion models. arXiv preprint arXiv:2305.13840, 2023. 7, 8

arXiv 2023

Show all 35 references

[7]

Editboard: Towards a comprehensive evalu- ation benchmark for text-based video editing models

Yupeng Chen, Penglin Chen, Xiaoyu Zhang, Yixian Huang, and Qian Xie. Editboard: Towards a comprehensive evalu- ation benchmark for text-based video editing models. arXiv preprint arXiv:2409.09668, 2024. 6

2024 arXiv
[8]

Clip-adapter: Better vision-language models with fea- ture adapters

Peng Gao, Shijie Geng, Renrui Zhang, Teli Ma, Rongyao Fang, Yongfeng Zhang, Hongsheng Li, and Yu Qiao. Clip-adapter: Better vision-language models with fea- ture adapters. International Journal of Computer Vision , 132(2):581–595, 2024. 2

2024
[9]

Tokenflow: Consistent diffusion features for consistent video editing

Michal Geyer, Omer Bar-Tal, Shai Bagon, and Tali Dekel. Tokenflow: Consistent diffusion features for consistent video editing. arXiv preprint arXiv:2307.10373, 2023. 7, 8

2023 arXiv
[10]

Enhancing the video editing capabilities of text-to-video generators using ddpm inversion

W Wilfred Godfrey and Abhinav Ratna. Enhancing the video editing capabilities of text-to-video generators using ddpm inversion. In 2023 IEEE International Conference on Com- puter Vision and Machine Intelligence (CVMI) , pages 1–5. IEEE, 2023. 2

2023
[12]

ultralytics/yolov5: v6

Glenn Jocher, Ayush Chaurasia, Alex Stoken, Jirka Borovec, Yonghye Kwon, Kalen Michael, Jiacong Fang, Colin Wong, Zeng Yifu, Diego Montes, et al. ultralytics/yolov5: v6. 2-yolov5 classification models, apple m1, reproducibility, clearml and deci. ai integrations. Zenodo, 2022. 2

2022
[13]

Vilt: Vision- and-language transformer without convolution or region su- pervision

Wonjae Kim, Bokyung Son, and Ildoo Kim. Vilt: Vision- and-language transformer without convolution or region su- pervision. In International conference on machine learning, pages 5583–5594. PMLR, 2021. 2

2021
[14]

Segment any- thing

Alexander Kirillov, Eric Mintun, Nikhila Ravi, Hanzi Mao, Chloe Rolland, Laura Gustafson, Tete Xiao, Spencer White- head, Alexander C Berg, Wan-Yen Lo, et al. Segment any- thing. In Proceedings of the IEEE/CVF International Con- ference on Computer Vision, pages 4015–4026, 2023. 2

2023
[15]

Anyv2v: A plug-and-play framework for any video- to-video editing tasks

Max Ku, Cong Wei, Weiming Ren, Huan Yang, and Wenhu Chen. Anyv2v: A plug-and-play framework for any video- to-video editing tasks. arXiv preprint arXiv:2403.14468 ,

arXiv
[16]

Shape-aware text-driven lay- ered video editing

Yao-Chih Lee, Ji-Ze Genevieve Jang, Yi-Ting Chen, Eliza- beth Qiu, and Jia-Bin Huang. Shape-aware text-driven lay- ered video editing. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition, pages 14317–14326, 2023. 2

2023
[17]

Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation

Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation. In Interna- tional conference on machine learning, pages 12888–12900. PMLR, 2022. 2

2022
[18]

Vidtome: Video token merging for zero-shot video editing

Xirui Li, Chao Ma, Xiaokang Yang, and Ming-Hsuan Yang. Vidtome: Video token merging for zero-shot video editing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7486–7495, 2024. 7

2024
[19]

Grounding dino: Marrying dino with grounded pre-training for open-set object detection

Shilong Liu, Zhaoyang Zeng, Tianhe Ren, Feng Li, Hao Zhang, Jie Yang, Qing Jiang, Chunyuan Li, Jianwei Yang, Hang Su, et al. Grounding dino: Marrying dino with grounded pre-training for open-set object detection. In European Conference on Computer Vision , pages 38–55. Springe...

2025
[20]

Video-p2p: Video editing with cross-attention control

Shaoteng Liu, Yuechen Zhang, Wenbo Li, Zhe Lin, and Jiaya Jia. Video-p2p: Video editing with cross-attention control. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8599–8608, 2024. 7, 8

2024
[21]

Trailblazer: Trajectory control for diffusion-based video generation

Wan-Duo Kurt Ma, John P Lewis, and W Bastiaan Kleijn. Trailblazer: Trajectory control for diffusion-based video generation. arXiv preprint arXiv:2401.00896, 2023. 3

2023 arXiv
[23]

Fatezero: Fus- ing attentions for zero-shot text-based video editing

Chenyang Qi, Xiaodong Cun, Yong Zhang, Chenyang Lei, Xintao Wang, Ying Shan, and Qifeng Chen. Fatezero: Fus- ing attentions for zero-shot text-based video editing. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 15932–15942, 2023. 7, 8

2023
[24]

Enhanced end-to-end video editing: Adaptive customization of path, object, and motion dynam- ics

Lakshmi Priya Ramisetty, Namrata Patel, Hiep Dang, and Aditya Singh Parmar. Enhanced end-to-end video editing: Adaptive customization of path, object, and motion dynam- ics. 3
[25]

Pro- cedural crowd generation for semantically augmented virtual cities

Otger Rogla, Gustavo A Patow, and Nuria Pelechano. Pro- cedural crowd generation for semantically augmented virtual cities. Computers & Graphics, 99:83–99, 2021. 3

2021
[26]

Yolov7: Trainable bag-of-freebies sets new state-of-the-art for real-time object detectors

Chien-Yao Wang, Alexey Bochkovskiy, and Hong- Yuan Mark Liao. Yolov7: Trainable bag-of-freebies sets new state-of-the-art for real-time object detectors. In Pro- ceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 7464–7475, 2023. 2

2023
[27]

Actionclip: Adapting language-image pretrained models for video action recognition

Mengmeng Wang, Jiazheng Xing, Jianbiao Mei, Yong Liu, and Yunliang Jiang. Actionclip: Adapting language-image pretrained models for video action recognition. IEEE Trans- actions on Neural Networks and Learning Systems, 2023. 2

2023
[28]

Medclip: Contrastive learning from unpaired medical images and text

Zifeng Wang, Zhenbang Wu, Dinesh Agarwal, and Jimeng Sun. Medclip: Contrastive learning from unpaired medical images and text. arXiv preprint arXiv:2210.10163, 2022. 2

2022 arXiv
[29]

Tune-a-video: One-shot tuning of image diffusion models for text-to-video generation

Jay Zhangjie Wu, Yixiao Ge, Xintao Wang, Stan Weixian Lei, Yuchao Gu, Yufei Shi, Wynne Hsu, Ying Shan, Xiaohu Qie, and Mike Zheng Shou. Tune-a-video: One-shot tuning of image diffusion models for text-to-video generation. In Proceedings of the IEEE/CVF International Conference...

2023
[30]

Florence-2: Advancing a unified representation for a variety of vision tasks

Bin Xiao, Haiping Wu, Weijian Xu, Xiyang Dai, Houdong Hu, Yumao Lu, Michael Zeng, Ce Liu, and Lu Yuan. Florence-2: Advancing a unified representation for a variety of vision tasks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 481...

2024
[31]

Temporally consistent semantic video editing

Yiran Xu, Badour AlBahar, and Jia-Bin Huang. Temporally consistent semantic video editing. In European Conference on Computer Vision, pages 357–374. Springer, 2022. 2

2022
[32]

Context-aware talking-head video editing

Songlin Yang, Wei Wang, Jun Ling, Bo Peng, Xu Tan, and Jing Dong. Context-aware talking-head video editing. In Proceedings of the 31st ACM International Conference on Multimedia, pages 7718–7727, 2023. 2

2023
[33]

Rethinking human evaluation protocol for text- to-video models: Enhancing reliability, reproducibility, and practicality

Tianle Zhang, Langtian Ma, Yuchen Yan, Yuchen Zhang, Kai Wang, Yue Yang, Ziyao Guo, Wenqi Shao, Yang You, Yu Qiao, et al. Rethinking human evaluation protocol for text- to-video models: Enhancing reliability, reproducibility, and practicality. arXiv preprint arXiv:2406.08845, 2024. 2

2024 arXiv
[34]

Motiondirector: Motion customization of text-to-video diffusion models

Rui Zhao, Yuchao Gu, Jay Zhangjie Wu, David Jun- hao Zhang, Jia-Wei Liu, Weijia Wu, Jussi Keppo, and Mike Zheng Shou. Motiondirector: Motion customization of text-to-video diffusion models. In European Conference on Computer Vision, pages 273–290. Springer, 2025. 3

2025
[35]

Deformable detr: Deformable trans- formers for end-to-end object detection

Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, and Jifeng Dai. Deformable detr: Deformable trans- formers for end-to-end object detection. arXiv preprint arXiv:2010.04159, 2020. 2

2010 arXiv

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.

[1] [22]

Gazed–gaze-guided cinematic editing of wide-angle monocular video recordings

KL Bhanu Moorthy, Moneish Kumar, Ramanathan Subra- manian, and Vineet Gandhi. Gazed–gaze-guided cinematic editing of wide-angle monocular video recordings. In Pro- ceedings of the 2020 CHI Conference on Human Factors in Computing Systems, pages 1–11, 2020. 2, 3

work page 2020

[2] [11]

Text-based Talking Video Editing with Cascaded Conditional Diffusion

Bo Han, Heqing Zou, Haoyang Li, Guangcong Wang, and Chng Eng Siong. Text-based talking video edit- ing with cascaded conditional diffusion. arXiv preprint arXiv:2407.14841, 2024. 2, 3

work page Pith review arXiv 2024

[3] [1]

Detectron2 object detection & manipulating images using cartooniza- tion

Allena Venkata Sai Abhishek and Sonali Kotni. Detectron2 object detection & manipulating images using cartooniza- tion. Int. J. Eng. Res. Technol.(IJERT), 10:1–5, 2021. 2

work page 2021

[4] [2]

A deep learning framework for quality assessment and restoration in video endoscopy

Sharib Ali, Felix Zhou, Adam Bailey, Barbara Braden, James E East, Xin Lu, and Jens Rittscher. A deep learning framework for quality assessment and restoration in video endoscopy. Medical image analysis, 68:101900, 2021. 3

work page 2021

[5] [3]

Text2live: Text-driven layered image and video editing

Omer Bar-Tal, Dolev Ofri-Amar, Rafail Fridman, Yoni Kas- ten, and Tali Dekel. Text2live: Text-driven layered image and video editing. In European conference on computer vi- sion, pages 707–723. Springer, 2022. 3

work page 2022

[6] [4]

Paligemma: A versatile 3b vlm for trans- fer

Lucas Beyer, Andreas Steiner, Andr ´e Susano Pinto, Alexan- der Kolesnikov, Xiao Wang, Daniel Salz, Maxim Neumann, Ibrahim Alabdulmohsin, Michael Tschannen, Emanuele Bugliarello, et al. Paligemma: A versatile 3b vlm for trans- fer. arXiv preprint arXiv:2407.07726, 2024. 4

arXiv 2024

[7] [5]

Stream- ing video diffusion: Online video editing with diffusion models

Feng Chen, Zhen Yang, Bohan Zhuang, and Qi Wu. Stream- ing video diffusion: Online video editing with diffusion models. arXiv preprint arXiv:2405.19726, 2024. 2

arXiv 2024

[8] [6]

Control-a-video: Controllable text-to-video generation with diffusion models

Weifeng Chen, Yatai Ji, Jie Wu, Hefeng Wu, Pan Xie, Jiashi Li, Xin Xia, Xuefeng Xiao, and Liang Lin. Control-a-video: Controllable text-to-video generation with diffusion models. arXiv preprint arXiv:2305.13840, 2023. 7, 8

arXiv 2023