TimeChat-Captioner: Scripting Multi-Scene Videos with Time-Aware and Structural Audio-Visual Captions

Feifan Song; Kun Gai; Kun Ouyang; Lei Li; Lingpeng Kong; Linli Yao; Pengfei Wan; Qi Liu; Xinlong Chen; Xu Sun

REVIEW 3 major objections 4 minor 1 cited by

TimeChat-Captioner: Scripting Multi-Scene Videos with Time-Aware and Structural Audio-Visual Captions

T0 review · 3 major / 4 minor · reviewed 2026-08-03 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A 7B open model beats Gemini-2.5-Pro at dense, timestamped video captions.

desk verdict Task framing and benchmark are genuinely useful; the SodaM-based SOTA claim is not trustworthy because the training reward and evaluation metric are the same video-blind judge. read the letter →

arxiv 2602.08711 v3 pith:2DMLWPZO submitted 2026-02-09 cs.CV

Linli Yao , Yuancheng Wei , Yaojie Zhang , Lei Li , Xinlong Chen , Feifan Song , Ziyue Wang , Kun Ouyang

show 7 more authors

Yuanxin Liu Lingpeng Kong Qi Liu Pengfei Wan Kun Gai Yuanxing Zhang Xu Sun

This is my paper

classification cs.CV

keywords OmniDenseCaptioningaudio-visualtemporalsegmentationSodaMGRPOvideounderstandingmultimodalLLMbenchmark

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper introduces Omni Dense Captioning, a task that turns video clips into continuous, timestamped, scene-by-scene scripts covering six dimensions: events, background, camera movement, shot editing, dialogue, and acoustic cues. To support the task, the authors construct OmniDCBench, a human-annotated benchmark of 1,122 videos, and propose SodaM, a metric that aligns predicted and ground-truth scene boundaries before scoring caption completeness via a checklist judge. They then train TimeChat-Captioner-7B on 42K synthetic examples using supervised fine-tuning followed by GRPO with rewards for format, length, timestamp accuracy, and SodaM score. The paper claims this 7B model reaches state-of-the-art on OmniDCBench, surpassing Gemini-2.5-Pro on SodaM (35.0 vs 33.7), and that its dense captions transfer to audio-visual reasoning and temporal grounding tasks. A sympathetic reader would care because it suggests dense, script-like captions are a tractable and powerful supervision signal for multimodal models, and that an open 7B model can rival a frontier closed model on a demanding captioning task.

What carries the argument

The load-bearing mechanism is the six-dimensional structural schema that forces captions to cover Events, Background, Camera State, Shot Editing, Dialogue, and Acoustic Cues, producing script-like descriptions that are simultaneously temporally dense and semantically dense. The evaluation and training hinge on SodaM, a metric that first aligns predicted to ground-truth scenes via IoU-based dynamic programming (merging many-to-one predictions) and then scores caption coverage using a checklist judge model; remarkably, the same SodaM score is used as the GRPO caption reward, making the metric both the yardstick and the training objective.

What would settle it

Run a human side-by-side preference test between TimeChat-Captioner-7B-GRPO and Gemini-2.5-Pro captions on a random subset of OmniDCBench; if human raters do not prefer the model's captions at least as often as Gemini's, or if injecting known hallucinated sentences into captions does not lower SodaM scores, the headline comparison is unsupported.

Watch

Extended reading notes

Core claim

The central claim is that a 7B open-weights model, trained with synthetic data and reinforcement learning, can generate fine-grained, timestamped, six-dimensional audio-visual captions that score higher on the paper's SodaM metric than Gemini-2.5-Pro (35.0 vs 33.7), while also improving downstream audio-visual reasoning (Daily-Omni 52.8, WorldSense 22.6) and temporal grounding (Charades-STA R1@0.7 48.3). On the authors' own terms, this establishes Omni Dense Captioning as a viable foundational task: dense, structured captions provide richer supervision than sparse event captions, and optimizing for time-aware caption quality is an effective proxy for general audiovisual comprehension.

Load-bearing premise

The central claim rests on SodaM being a valid, unbiased measure of caption quality—specifically, that the checklist judge does not reward verbosity or hallucination—since SodaM is both the evaluation metric and the reinforcement-learning reward.

Editorial extensions

If this is right

If correct, dense timestamped captions can serve as a supervision source for multimodal LLMs, improving audio-visual reasoning and temporal grounding without task-specific training data.
The open-sourced model, benchmark, and metric give the research community a reproducible alternative to proprietary captioning systems.
The GRPO result (2K samples beat scaling SFT from 20K to 40K) suggests reward-guided optimization is sample-efficient for this task.
The segment-then-caption strategy for hour-long videos points to a practical path for long-form dense captioning, though the paper acknowledges context-window limits.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The paper's headline comparison is entangled with its own evaluation metric: because the GRPO caption reward is SodaM, and SodaM uses a judge model, the claimed superiority over Gemini-2.5-Pro may partly reflect reward-hacking toward judge preferences rather than genuine caption quality; a human evaluation or a precision-penalized variant of SodaM would be needed to disentangle this.
The six-dimensional schema could be directly reused as conditioning inputs for video generation models, giving a natural text-to-video interface that reads like a screenplay.
The checklist-judge approach could be extended to other fine-grained generation tasks, such as image captioning or audio description, where n-gram metrics fail.
Training on synthetic captions from a proprietary model and then evaluating with a judge from the same model family may understate domain-shift issues; a judge from a different model family would test the metric's robustness.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

First thing you should know: this is a genuinely useful benchmark/task paper, and the headline SOTA claim should not be believed as stated. The task formulation is new enough: continuous scene-level timestamps plus six-dimensional structured captions (events, background, camera, editing, dialogue, acoustics), with a human-annotated benchmark and a DP-aligned metric to handle scene boundary ambiguity. That combination is not in the prior dense captioning or audiovisual captioning literature, and OmniDCBench looks carefully built—1,122 videos, 995-word reference captions, expert annotation per dimension. The authors also give a concrete training pipeline and a 7B model; the SFT/GRPO recipe is reproducible in principle.

The soft spot is the one the skeptical report flags, and it is load-bearing. In Section 4.3.2 (Eq. 8), the caption reward R_C is SodaM, and SodaM (Section 3.3) is computed by Gemini-2.5-Flash judging whether ground-truth keypoints are “explicitly mentioned or clearly implied.” The same judge and the same metric then produce the Table 1 headline. So the 35.0 vs 33.7 “surpasses Gemini-2.5-Pro” line mostly measures how well the model was optimized to satisfy that judge. The judge has no video access, so it cannot tell a factual caption from fluent hallucination; the checklist is recall of keypoints with no explicit precision penalty, so verbose paraphrases can inflate it. Gemini-2.5-Pro was not trained on this reward, which makes the comparison asymmetric. I would need an independent judge model or human evaluation before treating the SOTA claim as real.

Also minor: the “training data entirely independent from benchmark” statement is overstated. Training uses Movie101 and MMTrail-2M; the benchmark includes Movie101 clips plus YT-Temporal-1B. Sources overlap, even if the actual videos may differ. And no error bars are reported, with only 1,122 benchmark samples.

What survives: the benchmark, the schema, and the SodaM alignment procedure are useful even if the current metric needs validation. The downstream results on DailyOmni, WorldSense, and Charades-STA are independent of the SodaM loop and are the strongest evidence that dense structured captions transfer. I would cite the paper for the task and data, and I would send it to peer review—but with a requirement that the SodaM headline be re-evaluated with a separate judge or human ratings, not a metric the model was trained against.

Referee Report

3 major / 4 minor

Summary. This paper proposes Omni Dense Captioning, a task in which a video is segmented into scenes and each scene is annotated with timestamps and a six-dimensional structured description (events, background, camera state, shot editing, dialogue, acoustics). It introduces OmniDCBench, a 1,122-video benchmark with entirely human annotations; SodaM, a metric that first aligns predicted and ground-truth scenes by dynamic programming over temporal IoU and then scores caption coverage using a Gemini-2.5-Flash checklist; and TimeChat-Captioner-7B, built on Qwen2.5-Omni and trained with SFT on 40K synthetic samples followed by GRPO on 2K samples. The central empirical claim is that TimeChat-Captioner-7B-GRPO attains 35.0 SodaM, surpassing Gemini-2.5-Pro (33.7), and that its captions improve downstream audio-visual QA (Daily-Omni, WorldSense) and temporal grounding (Charades-STA).

Significance. The task definition and the effort to construct a human-annotated benchmark are useful contributions, and a 7B open model that could rival a frontier closed model for structured dense video captioning would be significant. Strengths of the paper include the concrete six-dimensional schema, the fully manual annotation pipeline for the benchmark, the detailed two-stage training recipe, and the promise of public release of data, code, and models. However, the headline result is not currently supported because the caption reward used in the RL stage is the same judge-based, recall-only checklist that is later reported as the evaluation metric. The reported SOTA gap therefore may reflect optimization against the metric rather than superior factual caption quality. The paper can be made convincing only with an independent evaluation protocol.

major comments (3)

[Section 4.3.2, Eq. (8); Section 3.3, Eq. (2)] The time-aware caption reward R_C in Eq. (8) is defined as the unified SodaM metric, and SodaM is the Gemini-2.5-Flash checklist score of Eq. (2). Thus the GRPO stage is trained to maximize the same scoring function that is later reported as the headline metric in Table 1. The comparison against Gemini-2.5-Pro is asymmetric: TimeChat-Captioner is directly optimized against a specific judge and checklist, while Gemini-2.5-Pro is not. No experiment in Section 5 reports SodaM computed with a different judge, held-out human ratings, or a precision-aware variant. This is load-bearing for the abstract's 'surpassing Gemini-2.5-Pro' claim. Please re-evaluate the final model with an independent judge model and human annotation, or restrict the claim to 'higher SodaM under the training reward'.
[Section 3.3, Eq. (2); Table 9] Eq. (2) computes recall only: the fraction of ground-truth keypoints the judge considers explicitly mentioned or clearly implied. There is no precision term and no check that the predicted caption is true to the video. The judge prompt in Table 9 forbids the judge from generating new keypoints but does not require video-grounded verification of the candidate caption. Since the model is rewarded to maximize this recall (Table 4: SodaM rises from 32.5 without R_C to 35.0 with it), verbose, semantically broad, or partially hallucinated captions can inflate the score. Provide precision, human agreement, and an independent judge; without these, the 35.0 vs 33.7 gap is not a reliable measure of factual caption quality.
[Section 4.2 vs Section 3.2] Section 4.2 lists MMTrail-2M and Movie101 as training sources, while Section 3.2 lists Movie101 and YT-Temporal-1B as benchmark sources. Yet the Summary paragraph in Section 4.2 states that the training data is 'entirely independent from the benchmark in terms of video sources'. Movie101 appears in both lists. If disjoint splits or different subsets of Movie101 are used, this should be stated explicitly; if the same Movie101 videos are used, the claimed independence and the resulting generalization conclusions are not valid.

minor comments (4)

[Figure 4 / Appendix E] The output labeled 'Ours (7B Model)' in Figure 4 contains identity and event details ('a woman driving', 'her girlfriend's 60th birthday') that are not present in the ground-truth annotation printed immediately above it, while Appendix E states that the model correctly identifies 'Xia Luo' and the back-seat passenger. The qualitative example as shown does not support the 'Fine-grained Alignment' label; please correct the example or reconcile the appendix.
[Section 3.3] The text says SodaM computes 'F1 score for all pairs', but Eq. (2) defines only a recall fraction. Define the precision term and how the F1 is aggregated across pairs; otherwise the score is uninterpretable.
[Table 1] No standard errors, confidence intervals, or significance tests are reported. With 1,122 benchmark videos, the 1.3-point SodaM gap between TimeChat-Captioner and Gemini-2.5-Pro may be within noise; reporting uncertainty would strengthen the claim.
[Section 5.2] The text calls the result 'state-of-the-art' without noting that on segmentation F1 the model (61.2) is below Gemini-2.5-Pro (68.5). SOTA holds only for the caption-quality aggregate; please qualify the wording.

Circularity Check

1 steps flagged · score 6.0 of 10

Headline SodaM score is the training reward itself: GRPO optimizes Eq. 2's judge-based CheckList via R_C (Eq. 8), so the reported 35.0 vs 33.7 'surpasses Gemini-2.5-Pro' is not an independent assessment of caption quality.

fitted input called prediction [Section 4.3.2 (Eq. 8), Section 3.3 (Eq. 2), Table 4]
"Time-aware Caption Reward RC: We adopt the unified SodaM metric as the reward to encourage comprehensive and temporally-aligned structural captions. ... R=α f ·R F +α l ·R L +α t ·R T +α c ·R C (8) ... Here, Judge(ˆc, ei)∈ {0,1} indicates if ˆc covers element ei from a judge model Gemini-2.5-Flash. ... GRPO (w/o RC) 32.5 ... GRPO (w/ RC) 35.0"

R_C is defined as SodaM, and SodaM is defined (Eq. 2) as the Gemini-2.5-Flash judge's recall of ground-truth keypoints. The model is therefore directly optimized to maximize the exact score that is later reported as the headline metric. The ablation's conclusion that R_C 'significantly improves ... caption completeness' is circular: caption completeness is measured by SodaM itself, so adding R_C trivially raises the score it defines. Comparing against Gemini-2.5-Pro is asymmetric because the baseline was not trained to maximize SodaM. The claimed 35.0 vs 33.7 gap is thus a measure of reward optimization, not independent caption quality.

full rationale

Most of the paper is a self-contained empirical contribution: task definition, human-annotated benchmark, architecture, SFT, and downstream evaluations are independent of each other. The only load-bearing circular step is the identity between the training reward and the headline evaluation metric. Because the GRPO reward R_C is SodaM, and SodaM is the checklist score produced by a Gemini-2.5-Flash judge with no video access and no precision penalty, the central 'SOTA on OmniDCBench' claim reduces to the model being optimized for that metric. The training/eval data split prevents a full 8-10, but the metric identity means the headline comparison is not an external validation of caption quality. Downstream tasks (DailyOmni, WorldSense, Charades-STA) are external and help support the method, which keeps the score at 6 rather than higher.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The central claim depends on the validity of the SodaM metric (used as both reward and evaluation), on the faithfulness of synthetic training captions, and on the quality of human benchmark annotations. The paper provides no precision measure, no inter-annotator agreement, and the metric's judge is shared between training and evaluation.

free parameters (3)

GRPO reward weights (alpha_f, alpha_l, alpha_t, alpha_c) = (0.5, 0.5, 1.0, 1.0)
Hand-chosen; ablation only varies alpha_c (1.0 vs 1.5), no systematic search. The balance between format, length, temporal, and caption rewards directly shapes the final SodaM score.
KL penalty coefficient beta = 0.04
Set by hand in Section D; not justified by experiments, and it controls how far the policy can drift from the reference.
Training data scales (SFT 40K, GRPO 2K) = 40K / 2K
Chosen based on availability and the ablation in Table 4; not derived from a principled criterion.

assumptions (4)

domain assumption Gemini-2.5-Flash's checklist judgments (Eq. 2) are a valid, unbiased measure of caption coverage.
SodaM relies on an LLM judge to decide whether predicted captions cover ground-truth keypoints; the same judge provides the R_C training reward, so its biases propagate into both training and evaluation.
domain assumption Recall-only SodaM (no precision penalty) reflects overall caption quality.
The metric has no hallucination penalty; verbose or unfaithful captions can inflate coverage. This assumption is load-bearing because SodaM is both primary metric and reward.
domain assumption Gemini-2.5-Pro-generated synthetic captions in TimeChatCap-42K are faithful enough to train a specialist model.
Training data is entirely LLM-synthesized; no human verification is reported beyond JSON, duration, and audio filters.
domain assumption Human annotations in OmniDCBench are accurate and scene boundaries are well-defined.
Benchmark quality rests on crowd annotators with double-checking, but no inter-annotator agreement or boundary consistency measure is reported.

how reviews work

0 comments

Cite this review

Pith. "Pith review of TimeChat-Captioner: Scripting Multi-Scene Videos with Time-Aware and Structural Audio-Visual Captions." pith.science (2026). https://pith.science/paper/2DMLWPZO

@misc{pith2026260208711,
  author       = {Pith},
  title        = {Pith review of: TimeChat-Captioner: Scripting Multi-Scene Videos with Time-Aware and Structural Audio-Visual Captions},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/2DMLWPZO}},
  note         = {Machine review of arXiv:2602.08711}
}

read the original abstract

This paper proposes Omni Dense Captioning, a novel task designed to generate continuous, fine-grained, and structured audio-visual narratives with explicit timestamps. To ensure dense semantic coverage, we introduce a six-dimensional structural schema to create "script-like" captions, enabling readers to vividly imagine the video content scene by scene, akin to a cinematographic screenplay. To facilitate research, we construct OmniDCBench, a high-quality, human-annotated benchmark, and propose SodaM, a unified metric that evaluates time-aware detailed descriptions while mitigating scene boundary ambiguity. Furthermore, we construct a training dataset, TimeChatCap-42K, and present TimeChat-Captioner-7B, a strong baseline trained via SFT and GRPO with task-specific rewards. Extensive experiments demonstrate that TimeChat-Captioner-7B achieves state-of-the-art performance, surpassing Gemini-2.5-Pro, while its generated dense descriptions significantly boost downstream capabilities in audio-visual reasoning (DailyOmni and WorldSense) and temporal grounding (Charades-STA). All datasets, models, and code are available at https://github.com/yaolinli/TimeChat-Captioner.

Figures

Figures reproduced from arXiv: 2602.08711 by the authors.

**Figure 1.** Illustration of the OmniDenseCaptioning task. This paper introduces Omni Dense Captioning task, which generates fine-grained, temporally grounded descriptions for comprehensive audio-visual understanding. The term “dense” reflects two key properties: (1) temporally-dense: continuous scene segmentation with explicit timestamps, and (2) description-dense: structured captions spanning six dimensions: Audio-Visual Event… view at source ↗

**Figure 2.** Statistics of human-annotated OmniDCBench. (a) Video duration distribution. (b) Caption length distribution with per-dimension. The benchmark features comprehensive annotations averaging 995 words per video. (c) Scene duration distribution (in seconds), compared against MLLM-generated outputs to highlight the granularity gap between human and model segmentations. distinct dimensions. This approach enables the captur… view at source ↗

**Figure 3.** Overview of TimeChat-Captioner Architecture. (Left) This model leverages Qwen2.5-Omni (Xu et al., 2025a) with interleaved audio-visual tokens to generate multi-scene timestamps and six-dimensional captions. (Right) Two-stage training: SFT for task format learning, followed by GRPO with rewards for format, length, timestamp accuracy, and time-aware fine-grained caption quality. then assessed against each of these ele… view at source ↗

Figures from the paper (4 more)

**Figure 4.** Figure 4: Qualitative case analysis. We compare TimeChat-Captioner with Gemini-2.5-Pro and Qwen-3-Omni on a sample from OmniDCBench. Our model achieves fine-grained alignment with the ground truth across all six annotation dimensions: detailed events, visual background, acoustic…

**Figure 5.** Figure 5: Statistics of the training dataset TimeChatCap-42K. (a) Video duration distribution; most videos (73.9%) fall within 50-60 seconds. (b) Caption length distribution with per-dimension average word counts; annotations average 877 words per video across six dimensions. (c…

**Figure 6.** Figure 6: Overview of the synthetic training data construction pipeline for the training dataset TimeChatCap-42K [PITH_FULL_IMAGE:figures/full_fig_p015_6.png]

**Figure 7.** Figure 7: Interface page used for manual annotation during the construction of OmniDCBench. ambient sounds (“the hum of the car’s engine”). • Dialogue Content: Speaker attribution is precise, with each utterance correctly assigned to the corresponding character (“Xia Luo: ...”),…

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

PercepCap: Video Captioner with Structured Spatio-Temporal Perception
cs.CV 2026-07 conditional novelty 5.0 of 10

Explicitly generating object trajectories and event timestamps before the final caption improves detailed video captioning on multiple benchmarks, though the improvement is largely driven by an external perception oracle.

Reference graph

Works this paper leans on

9 extracted references · 1 linked inside Pith · cited by 1 Pith paper

[1]

• Each minute usually contains4–5 segments, but prioritize the video’s logic over strict numbers

Segmentation Logic: • Split the video into natural segments according toscene changes, shot transitions, events, character actions, or core content shifts. • Each minute usually contains4–5 segments, but prioritize the video’s logic over strict numbers
[2]

• Includecharacters, actions, objects, emotions, and scene detailswhere relevant

Caption Requirements: • For each segment, provide atime rangein the format:start time(0:00) - end time(0:05): caption • The caption should beconcise but descriptive, summarizing what happens in that segment. • Includecharacters, actions, objects, emotions, and scene detailswhere relevant. • Avoid redundancy, but ensure that important visual and narrative ...
[3]

• Timestamp format:minutes:seconds(e.g.,0:00 - 0:11)

Output Format: • A clean list of captions, each starting with the time range followed by the description. • Timestamp format:minutes:seconds(e.g.,0:00 - 0:11). • Segment boundaries should beclear and non-overlapping—the end time of one segment and start time of the next should be at least 1 second apart. • Example: if one segment ends at0:11, the next one...
[5]

Each GT caption defines the rough boundaries of a segment

Segmentation: • Follow thetimestamps provided in the GT captions. Each GT caption defines the rough boundaries of a segment. • Timestamp format:minutes:seconds(e.g.,0:00 - 0:11). • Segment boundaries should beclear and non-overlapping—at least 1 second apart between consecutive segments
[6]

Use the video itself toexpand with details: •Characters: actions, gestures, facial expressions, emotions

Generation Logic: Donotsimply extract or paraphrase the GT caption. Use the video itself toexpand with details: •Characters: actions, gestures, facial expressions, emotions. •Objects & Setting: relevant items, props, environment. •Camera: framing, movement, zoom, transitions. •Storyline: how the segment advances or changes the plot. •Speech: actual dialog...
[7]

timestamp

Output Format (JSON Schema): { "timestamp": "start_time - end_time", "segment_detail_caption": "Detailed description of what happens (gestures, expressions, setting details, etc.).", "camera_state": "Camera angle, framing, zoom, and movement.", "video_background": "Setting, environment, or background elements.", "storyline": "How this segment fits into th...
[8]

A list ofground-truth keypointsalready organized in 6 dimensions
[9]

by_dim": {

Onemodel-generated captionto evaluate. The ground-truth keypoints are alreadyatomic and accurate. You only need to check whether each keypoint isexplicitly mentioned or clearly impliedin the model’s caption. Rules: • Mark a keypoint as correct if its meaning appears in the model’s caption with the same or equivalent semantics. • Ignore differences in phra...

Show all 9 references

[2023]

timestamp

URL https://api.semanticscholar. org/CorpusID:257232853. Yang, Q., Yao, S., Chen, W., Fu, S., Bai, D., Zhao, J., Sun, B., Yin, B., Wei, X., and Zhou, J. Humanomniv2: From understanding to omni-modal reasoning with con- text.arXiv preprint arXiv:2506.21277, 2025a. Yang, Z., Yu,...

2022 arXiv

Pith tools

Reviewed August 3, 2026 · model on record in the stance chip above.

[1] [1]

• Each minute usually contains4–5 segments, but prioritize the video’s logic over strict numbers

Segmentation Logic: • Split the video into natural segments according toscene changes, shot transitions, events, character actions, or core content shifts. • Each minute usually contains4–5 segments, but prioritize the video’s logic over strict numbers

[2] [2]

• Includecharacters, actions, objects, emotions, and scene detailswhere relevant

Caption Requirements: • For each segment, provide atime rangein the format:start time(0:00) - end time(0:05): caption • The caption should beconcise but descriptive, summarizing what happens in that segment. • Includecharacters, actions, objects, emotions, and scene detailswhere relevant. • Avoid redundancy, but ensure that important visual and narrative ...

[3] [3]

• Timestamp format:minutes:seconds(e.g.,0:00 - 0:11)

Output Format: • A clean list of captions, each starting with the time range followed by the description. • Timestamp format:minutes:seconds(e.g.,0:00 - 0:11). • Segment boundaries should beclear and non-overlapping—the end time of one segment and start time of the next should be at least 1 second apart. • Example: if one segment ends at0:11, the next one...

[4] [5]

Each GT caption defines the rough boundaries of a segment

Segmentation: • Follow thetimestamps provided in the GT captions. Each GT caption defines the rough boundaries of a segment. • Timestamp format:minutes:seconds(e.g.,0:00 - 0:11). • Segment boundaries should beclear and non-overlapping—at least 1 second apart between consecutive segments

[5] [6]

Use the video itself toexpand with details: •Characters: actions, gestures, facial expressions, emotions

Generation Logic: Donotsimply extract or paraphrase the GT caption. Use the video itself toexpand with details: •Characters: actions, gestures, facial expressions, emotions. •Objects & Setting: relevant items, props, environment. •Camera: framing, movement, zoom, transitions. •Storyline: how the segment advances or changes the plot. •Speech: actual dialog...

[6] [7]

timestamp

Output Format (JSON Schema): { "timestamp": "start_time - end_time", "segment_detail_caption": "Detailed description of what happens (gestures, expressions, setting details, etc.).", "camera_state": "Camera angle, framing, zoom, and movement.", "video_background": "Setting, environment, or background elements.", "storyline": "How this segment fits into th...

[7] [8]

A list ofground-truth keypointsalready organized in 6 dimensions

[8] [9]

by_dim": {

Onemodel-generated captionto evaluate. The ground-truth keypoints are alreadyatomic and accurate. You only need to check whether each keypoint isexplicitly mentioned or clearly impliedin the model’s caption. Rules: • Mark a keypoint as correct if its meaning appears in the model’s caption with the same or equivalent semantics. • Ignore differences in phra...