Pith. sign in

REVIEW 4 major objections 4 minor 1 cited by

From Vision To Language through Graph of Events in Space and Time: An Explainable Self-supervised Approach

T0 review · 4 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A graph of events in space and time, built automatically from off-the-shelf vision models, can serve as an explainable bridge from video to natural language and as a teacher for end-to-end captioning models.

desk verdict Solid extension of their ICCV 2023 GEST with a credible video-to-paragraph result, but the explainability claim leaks at the final LLM refinement, which is allowed to add, rename, or delete content that the graph never supported. read the letter →

arxiv 2507.04815 v1 pith:6EZ2RMKD submitted 2025-07-07 cs.CV cs.AIcs.CL

Mihai Masala , Marius Leordeanu This is my paper
classification cs.CVcs.AIcs.CL
keywords videocaptioninggraphofeventsinspaceandtimevision-languagerepresentationexplainabledescriptionself-supervisedteacher-studentlong-formnarrationVLM-as-a-jury
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper proposes that the bridge between video and natural language should be an explicit graph of events in space and time (GEST), built automatically from several off-the-shelf vision models, rather than a direct black-box mapping. From this graph the method derives a proto-language description, which a text-only LLM then rewrites into fluent, rich prose. The authors argue this makes video description explainable and trustworthy, and that the same procedure can act as a fully automatic teacher for training end-to-end vision-to-language student models. On the Videos-to-Paragraphs dataset, which they argue is the only one requiring genuinely story-like descriptions, their pipeline is preferred by humans and by a jury of large vision-language models alike.

What carries the argument

The Graph of Events in Space and Time (GEST): nodes are events, defined by an actor, an action, possibly involved objects, a location, and a start and end frame; edges encode spatial proximity and temporal relations (next, same time, meanwhile). The machinery that carries the argument is the automatic construction of GEST from four vision modules — action detection (VideoMAE on AVA-Kinetics), object detection and tracking (YOLO), semantic segmentation (Mask2Former), and monocular depth (Marigold) — followed by person re-identification via HSV-histogram matching, action filtering and voting, and event aggregation. GEST is then converted to a proto-language by mapping the graph into a 3D volume and traversing it with a breadth-first search, and the proto-language is handed to a text-only LLM, which is explicitly allowed to pick among candidate objects, rename actions, or delete events. A small vision-language model supplies a scene label that is prepended as context. The same teacher pathway (video to GEST to text) is then used to supervise an end-to-end neural student in a self-supervised pretraining step.

What would settle it

Take a video whose key action is absent from the detector's vocabulary, such as someone mopping a floor, and run the full pipeline; if the final LLM-refined description still cannot name the action or invents a wrong one, the claim that GEST plus LLM refinement grounds language in the story fails.

Watch

Extended reading notes

Core claim

The central claim is that a Graph of Events in Space and Time — nodes are events (an actor, an action, involved objects, location, start/end frame), edges are spatial and temporal relations such as next, same time, meanwhile, and close in space — is a shared representation that can be obtained analytically from video and converted into natural language. The method aggregates per-frame outputs of action detection, object detection and tracking, semantic segmentation, and depth estimation into video-level events, then renders the graph as a proto-language through a breadth-first traversal, and finally feeds that proto-language to an LLM that selects the most probable objects, renames or replaces actions, and produces a fluent paragraph. The paper further claims that these generated descriptions can serve as a teacher: pre-training an end-to-end model on GEST-generated descriptions before finetuning on human captions improves the model on every text metric they report. A jury of four large VLMs agrees with human rankings over 80 percent of the time, which the paper presents as validation that its descriptions are grounded rather than hallucinated.

Load-bearing premise

The pipeline assumes that the fixed, small action vocabulary of the action detector is enough to capture the events that matter in a story, and that the LLM refinement stage can repair or rename missing or too-low-level actions whenever the detector falls short; the paper concedes in Section 6.5 that when the detector falls short, descriptions become low-level and inflexible.

Editorial extensions

If this is right

  • Rich, story-like video descriptions can be produced without any video-text training data, using only off-the-shelf vision modules and a text-only LLM.
  • The fixed action vocabulary of current detectors is the main ceiling on description flexibility; richer detectors would directly improve the final text.
  • Standard n-gram and embedding metrics underrate explainable methods, and a panel of VLMs is a more reliable proxy for human preferences.
  • GEST-generated descriptions can be used as synthetic training data to improve end-to-end captioners, pointing to a self-supervised loop that needs no new human annotation.
  • The approach singles out Videos-to-Paragraphs as the dataset that actually tests story-level description; other popular datasets are closer to category recognition.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Inference: the same GEST representation could serve as an interface for other sensory modalities (audio, touch), since the event vocabulary is modality-agnostic; audio events like a door slam could become nodes.
  • Inference: GEST's hierarchy (any event can be expanded into a sub-GEST, any GEST collapsed into an event) suggests a path to hierarchical, long-horizon video understanding beyond the seconds-long clips tested here.
  • Inference: a testable extension is to measure how much of the gain comes from the LLM refinement vs. the graph itself, by ablating the LLM step and comparing descriptions directly.
  • Inference: the VLM-jury result implies a practical evaluation protocol that could be reused by other groups, though its reliability on less curated videos is untested.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper proposes GEST (Graph of Events in Space and Time), a structured spatio-temporal representation built automatically from multiple vision tasks (action detection, object detection/tracking, semantic segmentation, depth estimation). Events are aggregated into a graph, converted into a 'proto-language', and then refined by a text-only LLM into fluent natural-language video descriptions. The authors further propose a self-supervised teacher-student scheme in which GEST-generated descriptions pretrain an end-to-end video captioning model. They validate the approach on five datasets using standard text-similarity metrics, human rankings, and a VLM-as-a-Jury protocol, reporting that GEST is competitive overall and best on Videos-to-Paragraphs. The paper claims GEST is a shared representation between vision and language and that the pipeline is explainable and trustworthy.

Significance. If the central claim holds, this is a valuable contribution: an explicit, multi-task visual representation that can be converted into rich language, with a self-supervised training signal for end-to-end models. The paper merits credit for the breadth of the evaluation (multiple datasets, humans, and a four-model VLM jury), for the zero/few-shot comparisons, and for demonstrating that GEST-generated captions can improve a finetuned video captioning model (VALOR). However, the significance is conditional on the faithfulness of the final LLM-refined text to the GEST. Because the refinement stage is explicitly allowed to introduce objects and rename or delete actions, and because no experiment verifies that the final description is entailed by the graph, the 'trustworthy and explainable' half of the central claim is currently unverified. The shared-representation claim is also only demonstrated in the vision-to-language direction.

major comments (4)
  1. [Section 4.2, Figure 6] The refinement LLM is explicitly permitted to 'pick a new object that is not present in the list', to 'change the name of an action', and to delete an action 'entirely if it does not fit the context'. This means the final natural-language description is not guaranteed to be entailed by the GEST. Since the paper's central claim is that the pipeline is explainable and trustworthy, the faithfulness of the final text to the graph is a load-bearing property, but no experiment in Section 6 measures it. The evaluations compare generated text against human captions or against the video, not against the GEST or the proto-language. I recommend adding a direct faithfulness evaluation (e.g., fraction of final-text objects/actions that appear in the corresponding GEST event lists, or human/VLM judgments of whether each final claim is supported by the graph), and/or constraining the refinement prompt so that unsupported content is prohibited.
  2. [Section 6.5] The paper itself concedes that the fixed action vocabulary of the VideoMAE/AVA-Kinetics detector yields lower-level descriptions, e.g., mopping described as 'holding an object while walking around'. Because every event node in the GEST is derived from that detector, and because the refinement LLM can rename actions without any verification, the high-level semantics in the final text may come from the LLM's priors rather than from the GEST. This is not an equation-level circularity, but it is a grounding gap. I would like to see a quantitative decomposition: in the final descriptions, what fraction of action names and object mentions are present in the proto-language, and what fraction are introduced by the refinement step? Without such numbers, the explainable-grounding claim is not established.
  3. [Abstract, Section 3] The paper describes GEST as 'a shared representation between vision and language', but the experiments only demonstrate the vision-to-language direction: video to events to text. There is no experiment or mechanism that maps natural language back to GEST, nor any demonstration that language input can produce or be aligned with the same graph representation. I recommend either providing evidence for the reverse direction (or at least a formal definition of bidirectionality) or softening the claim to 'a visual representation that can be converted into language'.
  4. [Section 6.8, Table 8] The self-supervised teacher-student claim is supported by finetuning a single end-to-end model, VALOR, on Videos-to-Paragraphs. The text says 'we finetune video captioning methods' and concludes that the approach 'boost[s] the performance of end-to-end vision-to-language models', but no other architecture or dataset is tested. This is a generalization beyond the evidence. Either test additional models (e.g., VidIL, GIT2, or a transformer-based captioner) or restrict the claim to VALOR and one dataset.
minor comments (4)
  1. [Section 4.1, Table 1] Several hyperparameters (action confidence, top-K actions, IOU threshold, depth threshold, person re-id threshold, voting window) are set empirically on small validation samples, e.g., 20-25 examples for person-id unification and 40 videos for the HSV threshold. A sensitivity analysis over these thresholds, or at least a range check, would improve reproducibility and reduce concerns about dataset-specific tuning.
  2. [Figure 3 caption] The caption contains a duplicated word: 'we we automatically build the associated GEST'.
  3. [Algorithm 2] The pseudo-code uses inconsistent variable names ('event' vs. 'en', 'crtgroup' vs. 'crt group'), and lines 12 and 24 leave 'linked event' undefined in places; this makes the algorithm harder to follow.
  4. [Section 5.3.2, Table 3] The human evaluation subset contains only 75-93 videos for COIN, WebVid, VidOR, and VidVRD; the cross-dataset conclusions in Table 3 should be interpreted with these small sample sizes in mind, and the text could state this limitation explicitly.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity found: the GEST pipeline is algorithmic, its thresholds are engineering calibrations, and the cited prior GEST paper is restated in full rather than load-bearing.

full rationale

The paper's derivation chain starts from off-the-shelf vision modules (VideoMAE on AVA-Kinetics, YOLO, Mask2Former, Marigold), aggregates frame-level outputs through explicit rules into events, builds GEST edges from temporal/spatial criteria, and renders proto-language with a fixed grammar before LLM refinement. No equation in Sections 3-4 solves for a quantity that is defined by the final description, and no fitted parameter is renamed as a prediction: the empirically set thresholds (IoU 0.1, depth 25%, HSV cosine 0.3, confidence 75%) are preprocessing choices, not outputs claimed as predictions. The self-citations to [16] (prior GEST formalism) and [61] (Videos-to-Paragraphs dataset) are not load-bearing: Section 3 restates GEST completely, and the dataset is used as external test data. The VLM-as-a-Jury is validated against human rankings on a human-annotated subset and then applied at larger scale, which is an inductive evaluation choice rather than a reduction to the paper's own output. The concern that the refining LLM can add objects or actions absent from the GEST is a faithfulness limitation explicitly acknowledged in Section 6.5, not a circular dependence; it weakens the explainability claim but does not make the derivation equivalent to its input. The teacher-student experiment compares finetuning with and without GEST-generated pretraining data, so its outcome is not fixed by construction. Overall, the central derivation is self-contained and externally benchmarked; no step reduces by definition or by self-citation.

Assumptions & free parameters 7 free parameters · 5 assumptions · 1 invented entities

The central claim rests on the reliability of several pre-trained component models and on the sufficiency of a fixed action vocabulary; these are domain assumptions, not standard math. The GEST representation is defined algorithmically, with no formal semantic grounding. The free parameters are thresholds calibrated on small validation sets; the most consequential is the action vocabulary itself, which is not a numeric parameter but a modeling choice. No new physical entities are introduced; GEST is a formal data structure with falsifiable outputs through generated descriptions.

free parameters (7)
  • Action confidence threshold = >= 0.75
    The threshold is used in frame-level action filtering (Section 4.1, Table 1) and was set empirically.
  • Top-K actions per frame = 2
    Only the two most confident actions per frame are kept (Section 4.1, Table 1).
  • Actor-object interaction IOU threshold = >= 0.1
    Objects are kept only if their bounding box overlaps the person box with IoU at least 0.1 (Table 1).
  • Depth difference threshold = <= 25%
    Objects with average depth differing from the person by more than the threshold are discarded (Section 4.1, Table 1).
  • Person re-identification threshold = cosine similarity >= 0.3
    Calibrated on a curated set of 40 videos with about 2500 positive and 143k negative person pairs (Section 4.1, Figure 4).
  • Action voting window and voting threshold = 11 frames, >= 45% presence
    An action is kept only if it appears at least five times in an 11-frame window (Section 4.1, Table 1).
  • Event unification distance threshold = <= 10%
    Events with the same actor and action that are close in time are unified (Section 4.1, Table 1).
assumptions (5)
  • domain assumption Off-the-shelf vision models (VideoMAE, YOLO, Mask2Former, Marigold) provide sufficiently accurate frame-level action, object, segmentation and depth information.
    Invoked throughout Section 4.1; the entire event-graph construction depends on these pre-trained models being correct.
  • domain assumption The fixed action vocabulary of the action detector (AVA-Kinetics classes) is sufficient to describe the events in a video.
    The paper acknowledges in Section 6.5 that this vocabulary limits the flexibility and precision of generated descriptions.
  • domain assumption A text-only LLM can faithfully rewrite proto-language into natural language while preserving factual content and can be instructed to select correct objects and rename actions.
    Section 4.2 gives the LLM license to change action names, delete actions, or pick objects not in the list; an unfaithful LLM would break the grounding claim.
  • ad hoc to paper GEST is a bidirectional shared representation between vision and language.
    Postulated in Section 3 and the abstract; the paper demonstrates only the vision-to-language direction, and Figure 2 manually illustrates a text-derived GEST without an automatic parser.
  • domain assumption The thresholds for event and person unification generalize beyond the small validation sets used to set them.
    Thresholds in Table 1 and the person re-id threshold were calibrated on 20-40 manually inspected videos (Section 4.1).
invented entities (1)
  • Graph of Events in Space and Time (GEST) independent evidence
    purpose: Explicit intermediate representation of a video story as a graph of event nodes with spatio-temporal edges, used to generate proto-language and to act as a teacher for neural students.
    GEST is a formal, algorithmically defined data structure. It was introduced in the authors' ICCV 2023 paper, so it is not new here, but in this paper it is the central construct. Its outputs are falsifiable through video-description evaluation.

how reviews work

0 comments
Cite this review

Pith. "Pith review of From Vision To Language through Graph of Events in Space and Time: An Explainable Self-supervised Approach." pith.science (2026). https://pith.science/paper/6EZ2RMKD

@misc{pith2026250704815,
  author       = {Pith},
  title        = {Pith review of: From Vision To Language through Graph of Events in Space and Time: An Explainable Self-supervised Approach},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/6EZ2RMKD}},
  note         = {Machine review of arXiv:2507.04815}
}
read the original abstract

The task of describing video content in natural language is commonly referred to as video captioning. Unlike conventional video captions, which are typically brief and widely available, long-form paragraph descriptions in natural language are scarce. This limitation of current datasets is due to the expensive human manual annotation required and to the highly challenging task of explaining the language formation process from the perspective of the underlying story, as a complex system of interconnected events in space and time. Through a thorough analysis of recently published methods and available datasets, we identify a general lack of published resources dedicated to the problem of describing videos in complex language, beyond the level of descriptions in the form of enumerations of simple captions. Furthermore, while state-of-the-art methods produce impressive results on the task of generating shorter captions from videos by direct end-to-end learning between the videos and text, the problem of explaining the relationship between vision and language is still beyond our reach. In this work, we propose a shared representation between vision and language, based on graphs of events in space and time, which can be obtained in an explainable and analytical way, to integrate and connect multiple vision tasks to produce the final natural language description. Moreover, we also demonstrate how our automated and explainable video description generation process can function as a fully automatic teacher to effectively train direct, end-to-end neural student pathways, within a self-supervised neuro-analytical system. We validate that our explainable neuro-analytical approach generates coherent, rich and relevant textual descriptions on videos collected from multiple varied datasets, using both standard evaluation metrics, human annotations and consensus from ensembles of state-of-the-art VLMs.

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Authoring for Living Worlds: Tool-Constrained LLM Agents for Executable Multi-Actor Scenarios

    cs.CV 2026-04 unverdicted novelty 7.0 of 10

    Tool-validated LLM agents author formal event-graph story specifications that a deterministic game engine executes into multi-actor videos with perfect annotations, reaching 80% seeded end-to-end success versus 0% for...

Reference graph

Works this paper leans on

79 extracted references · 59 canonical work pages · cited by 1 Pith paper

  1. [1]

    In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp

    Wang, B., Ma, L., Zhang, W., Liu, W.: Reconstruction network for video captioning. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 7622–7631 (2018)

    work page 2018

  2. [2]

    In: Proceedings of the 26th ACM International Conference on Multimedia, pp

    Liu, S., Ren, Z., Yuan, J.: Sibnet: Sibling con- volutional encoder for video captioning. In: Proceedings of the 26th ACM International Conference on Multimedia, pp. 1425–1434 (2018)

    work page 2018

  3. [3]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp

    Aafaq, N., Akhtar, N., Liu, W., Gilani, S.Z., Mian, A.: Spatio-temporal dynamics and semantic attribute enriched visual encoding for video captioning. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 12487–12496 (2019)

    work page 2019

  4. [4]

    arXiv preprint 22 Fig

    Li, L., Lei, J., Gan, Z., Yu, L., Chen, Y.-C., Pillai, R., Cheng, Y., Zhou, L., Wang, X.E., Wang, W.Y., et al.: Value: A multi-task benchmark for video-and-language understanding evaluation. arXiv preprint 22 Fig. A1 Annotation instructions given to both humans and VLMs. Fig. A2 Screenshot of the annotation tool, containing in the top part a short list of...

    arXiv 2021

  5. [5]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp

    Lin, K., Li, L., Lin, C.-C., Ahmed, F., Gan, Z., Liu, Z., Lu, Y., Wang, L.: Swinbert: End- to-end transformers with sparse attention for video captioning. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 17949–17958 (2022)

    work page 2022

  6. [6]

    arXiv preprint arXiv:2304.08345 (2023)

    Chen, S., He, X., Guo, L., Zhu, X., Wang, W., Tang, J., Liu, J.: Valor: Vision-audio- language omni-perception pretraining model and dataset. arXiv preprint arXiv:2304.08345 (2023)

    arXiv 2023

  7. [7]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogni- tion, pp

    Wang, C.-Y., Bochkovskiy, A., Liao, H.-Y.M.: Yolov7: Trainable bag-of-freebies sets new state-of-the-art for real-time object detectors. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogni- tion, pp. 7464–7475 (2023)

    work page 2023

  8. [8]

    arXiv preprint arXiv:2304.06718 (2023)

    Zou, X., Yang, J., Zhang, H., Li, F., Li, L., Gao, J., Lee, Y.J.: Segment every- thing everywhere all at once. arXiv preprint arXiv:2304.06718 (2023)

    arXiv 2023

Show all 79 references
  1. [9]

    Cheng, B., Misra, I., Schwing, A.G., Kir- illov, A., Girdhar, R.: Masked-attention mask transformer for universal image segmenta- tion. (2022)

    2022

  2. [10]

    Advances in neural information processing systems 35, 10078–10093 (2022)

    Tong, Z., Song, Y., Wang, J., Wang, L.: Videomae: Masked autoencoders are data- efficient learners for self-supervised video pre-training. Advances in neural information processing systems 35, 10078–10093 (2022)

    2022

  3. [11]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp

    Wang, L., Huang, B., Zhao, Z., Tong, Z., He, Y., Wang, Y., Wang, Y., Qiao, Y.: Video- mae v2: Scaling video masked autoencoders with dual masking. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14549–14560 (2023)

    2023

  4. [12]

    Advances in neural information processing systems 33, 1877–1901 (2020)

    Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J.D., Dhariwal, P., Neelakan- tan, A., Shyam, P., Sastry, G., Askell, A., et al.: Language models are few-shot learners. Advances in neural information processing systems 33, 1877–1901 (2020)

    2020

  5. [13]

    : Learn- ing transferable visual models from natural language supervision

    Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al. : Learn- ing transferable visual models from natural language supervision. In: International Con- ference on Machine Learning, pp. 8748–8763 (2021). PMLR

    2021

  6. [14]

    In: International Conference on Machine Learning, pp

    Li, J., Li, D., Xiong, C., Hoi, S.: Blip: Boot- strapping language-image pre-training for unified vision-language understanding and generation. In: International Conference on Machine Learning, pp. 12888–12900 (2022). PMLR

    2022

  7. [15]

    In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp

    Zhai, X., Mustafa, B., Kolesnikov, A., Beyer, L.: Sigmoid loss for language image pre- training. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 11975–11986 (2023)

    2023

  8. [16]

    In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp

    Masala, M., Cudlenco, N., Rebedea, T., Leordeanu, M.: Explaining vision and lan- guage through graphs of events in space and time. In: Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 2826–2831 (2023)

    2023

  9. [17]

    In: Pro- ceedings of the IEEE Conference on Com- puter Vision and Pattern Recognition, pp

    Johnson, J., Krishna, R., Stark, M., Li, L.- J., Shamma, D., Bernstein, M., Fei-Fei, L.: Image retrieval using scene graphs. In: Pro- ceedings of the IEEE Conference on Com- puter Vision and Pattern Recognition, pp. 3668–3678 (2015)

    2015

  10. [18]

    In: Proceedings of the IEEE Conference on Com- puter Vision and Pattern Recognition, pp

    He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Com- puter Vision and Pattern Recognition, pp. 770–778 (2016)

    2016

  11. [19]

    6546–6555 (2018)

    Hara, K., Kataoka, H., Satoh, Y.: Can spa- tiotemporal 3d cnns retrace the history of 2d cnns and imagenet? In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 6546–6555 (2018)

    2018

  12. [20]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp

    Miech, A., Alayrac, J.-B., Smaira, L., Laptev, 24 I., Sivic, J., Zisserman, A.: End-to-end learn- ing of visual representations from uncu- rated instructional videos. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9879– 9889 (2020)

    2020

  13. [21]

    arXiv preprint arXiv:2303.15389 (2023)

    Sun, Q., Fang, Y., Wu, L., Wang, X., Cao, Y.: Eva-clip: Improved training techniques for clip at scale. arXiv preprint arXiv:2303.15389 (2023)

    2023 arXiv

  14. [22]

    arXiv preprint arXiv:2010.11929 (2020)

    Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., et al.: An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929 (2020)

    2020 arXiv

  15. [23]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp

    Liu, Z., Ning, J., Cao, Y., Wei, Y., Zhang, Z., Lin, S., Hu, H.: Video swin transformer. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3202–3211 (2022)

    2022

  16. [24]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp

    Fu, T.-J., Li, L., Gan, Z., Lin, K., Wang, W.Y., Wang, L., Liu, Z.: An empirical study of end-to-end video-language transformers with masked visual modeling. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 22898– 22909 (2023)

    2023

  17. [25]

    arXiv preprint arXiv:2306.09085 (2023)

    Chen, S., He, X., Li, H., Jin, X., Feng, J., Liu, J.: Cosa: Concatenated sample pre- trained vision-language foundation model. arXiv preprint arXiv:2306.09085 (2023)

    2023 arXiv

  18. [26]

    In: Pro- ceedings of the IEEE/CVF International Conference on Computer Vision, pp

    Wang, T., Zhang, R., Lu, Z., Zheng, F., Cheng, R., Luo, P.: End-to-end dense video captioning with parallel decoding. In: Pro- ceedings of the IEEE/CVF International Conference on Computer Vision, pp. 6847– 6857 (2021)

    2021

  19. [27]

    Advances in Neural Information Processing Systems 35, 23716–23736 (2022)

    Alayrac, J.-B., Donahue, J., Luc, P., Miech, A., Barr, I., Hasson, Y., Lenc, K., Mensch, A., Millican, K., Reynolds, M., et al.: Flamingo: a visual language model for few-shot learning. Advances in Neural Information Processing Systems 35, 23716–23736 (2022)

    2022

  20. [28]

    arXiv preprint arXiv:2108.10904 (2021)

    Wang, Z., Yu, J., Yu, A.W., Dai, Z., Tsvetkov, Y., Cao, Y.: Simvlm: Simple visual language model pretraining with weak supervision. arXiv preprint arXiv:2108.10904 (2021)

    2021 arXiv

  21. [29]

    arXiv preprint arXiv:2205.01917 (2022)

    Yu, J., Wang, Z., Vasudevan, V., Yeung, L., Seyedhosseini, M., Wu, Y.: Coca: Contrastive captioners are image-text foundation models. arXiv preprint arXiv:2205.01917 (2022)

    2022 arXiv

  22. [30]

    Advances in Neural Information Processing Systems 36, 72842– 72866 (2023)

    Chen, S., Li, H., Wang, Q., Zhao, Z., Sun, M., Zhu, X., Liu, J.: Vast: A vision- audio-subtitle-text omni-modality founda- tion model and dataset. Advances in Neural Information Processing Systems 36, 72842– 72866 (2023)

    2023

  23. [31]

    Devlin, J., Chang, M.-W., Lee, K., Toutanova, K.: BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding (2019)

    2019

  24. [32]

    arXiv preprint arXiv:2205.14100 (2022)

    Wang, J., Yang, Z., Hu, X., Li, L., Lin, K., Gan, Z., Liu, Z., Liu, C., Wang, L.: Git: A generative image-to-text transformer for vision and language. arXiv preprint arXiv:2205.14100 (2022)

    2022 arXiv

  25. [33]

    ArXiv abs/2302.00402 (2023)

    Xu, H., Ye, Q., Yan, M., Shi, Y., Ye, J., Xu, Y., Li, C., Bi, B., Qian, Q., Wang, W., Xu, G., Zhang, J., Huang, S., Huang, F., Zhou, J.: mplug-2: A modularized multi-modal foun- dation model across text, image and video. ArXiv abs/2302.00402 (2023)

    2023 arXiv

  26. [34]

    : Language models with image descriptors are strong few-shot video- language learners

    Wang, Z., Li, M., Xu, R., Zhou, L., Lei, J., Lin, X., Wang, S., Yang, Z., Zhu, C., Hoiem, D., et al. : Language models with image descriptors are strong few-shot video- language learners. Advances in Neural Infor- mation Processing Systems 35, 8483–8497 (2022)

    2022

  27. [35]

    In: Proceedings of the AAAI Conference on Artificial Intelligence, vol

    Wang, R., Wei, Z., Li, P., Zhang, Q., Huang, X.: Storytelling from an image stream using scene graphs. In: Proceedings of the AAAI Conference on Artificial Intelligence, vol. 34, pp. 9185–9192 (2020)

    2020

  28. [36]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogni- tion, pp

    Chen, S., Jin, Q., Wang, P., Wu, Q.: Say as 25 you wish: Fine-grained control of image cap- tion generation with abstract scene graphs. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recogni- tion, pp. 9962–9971 (2020)

    2020

  29. [37]

    Journal of Visual Communication and Image Representation 58, 477–485 (2019)

    Xu, N., Liu, A.-A., Liu, J., Nie, W., Su, Y.: Scene graph captioner: Image caption- ing based on structural visual representation. Journal of Visual Communication and Image Representation 58, 477–485 (2019)

    2019

  30. [38]

    In: Proceed- ings of the IEEE Conference on Computer Vision and Pattern Recognition, pp

    Johnson, J., Gupta, A., Fei-Fei, L.: Image generation from scene graphs. In: Proceed- ings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1219– 1228 (2018)

    2018

  31. [39]

    Advances in Neural Information Processing Systems 32 (2019)

    Li, Y., Ma, T., Bai, Y., Duan, N., Wei, S., Wang, X.: Pastegan: A semi-parametric method to generate image from scene graph. Advances in Neural Information Processing Systems 32 (2019)

    2019

  32. [40]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp

    Dhamo, H., Farshad, A., Laina, I., Navab, N., Hager, G.D., Tombari, F., Rupprecht, C.: Semantic image manipulation using scene graphs. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5213–5222 (2020)

    2020

  33. [41]

    arXiv preprint arXiv:1902.05715 (2019)

    Ghosh, S., Burachas, G., Ray, A., Ziskind, A.: Generating natural language explanations for visual question answering using scene graphs and visual attention. arXiv preprint arXiv:1902.05715 (2019)

    2019 arXiv

  34. [42]

    arXiv preprint arXiv:1907.12133 (2019)

    Zhang, C., Chao, W.-L., Xuan, D.: An empir- ical study on leveraging scene graphs for visual question answering. arXiv preprint arXiv:1907.12133 (2019)

    2019 arXiv

  35. [43]

    In: ACM International Conference on Multi- media, Mountain View, CA USA (2017)

    Shang, X., Ren, T., Guo, J., Zhang, H., Chua, T.-S.: Video visual relation detection. In: ACM International Conference on Multi- media, Mountain View, CA USA (2017)

    2017

  36. [44]

    In: Proceed- ings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition, pp

    Ji, J., Krishna, R., Fei-Fei, L., Niebles, J.C.: Action genome: Actions as compositions of spatio-temporal scene graphs. In: Proceed- ings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition, pp. 10236–10247 (2020)

    2020

  37. [45]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp

    Zheng, S., Chen, S., Jin, Q.: Vrdformer: End-to-end video visual relation detection with transformers. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 18836–18846 (2022)

    2022

  38. [46]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp

    Yang, J., Peng, W., Li, X., Guo, Z., Chen, L., Li, B., Ma, Z., Zhou, K., Zhang, W., Loy, C.C., et al.: Panoptic video scene graph gen- eration. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 18675–18685 (2023)

    2023

  39. [47]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp

    Rodin, I., Furnari, A., Min, K., Tripathi, S., Farinella, G.M.: Action scene graphs for long- form understanding of egocentric videos. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 18622–18632 (2024)

    2024

  40. [48]

    In: 2016 IEEE Inter- national Conference on Image Processing (ICIP), pp

    Shin, A., Ohnishi, K., Harada, T.: Beyond caption to narrative: Video captioning with multiple sentences. In: 2016 IEEE Inter- national Conference on Image Processing (ICIP), pp. 3364–3368 (2016). IEEE

    2016

  41. [49]

    In: Proceed- ings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition, pp

    Park, J.S., Rohrbach, M., Darrell, T., Rohrbach, A.: Adversarial inference for multi-sentence video description. In: Proceed- ings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition, pp. 6598–6608 (2019)

    2019

  42. [50]

    In: Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XXI 16, pp

    Park, J.S., Darrell, T., Rohrbach, A.: Identity-aware multi-sentence video descrip- tion. In: Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XXI 16, pp. 360–378 (2020). Springer

    2020

  43. [51]

    Machine Vision and Applica- tions 35(4), 64 (2024)

    Ziaeetabar, F., Safabakhsh, R., Momtazi, S., Tamosiunaite, M., W¨ org¨ otter, F.: Multi sen- tence description of complex manipulation action videos. Machine Vision and Applica- tions 35(4), 64 (2024)

    2024

  44. [52]

    arXiv preprint arXiv:1804.02767 (2018) 26

    Redmon, J., Farhadi, A.: Yolov3: An incremental improvement. arXiv preprint arXiv:1804.02767 (2018) 26

    2018 arXiv

  45. [53]

    In: Proceed- ings of the IEEE Conference on Computer Vision and Pattern Recognition, pp

    Cao, Z., Simon, T., Wei, S.-E., Sheikh, Y.: Realtime multi-person 2d pose estima- tion using part affinity fields. In: Proceed- ings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 7291– 7299 (2017)

    2017

  46. [54]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp

    Islam, M.M., Ho, N., Yang, X., Nagarajan, T., Torresani, L., Bertasius, G.: Video recap: Recursive captioning of hour-long videos. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 18198–18208 (2024)

    2024

  47. [55]

    arXiv preprint arXiv:2406.06499 (2024)

    Nadeem, A., Sardari, F., Dawes, R., Husain, S.S., Hilton, A., Mustafa, A.: Narrative- bridge: Enhancing video captioning with causal-temporal narrative. arXiv preprint arXiv:2406.06499 (2024)

    2024 arXiv

  48. [56]

    In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp

    Yu, H., Cheng, S., Ni, B., Wang, M., Zhang, J., Yang, X.: Fine-grained video captioning for sports narrative. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 6006–6015 (2018)

    2018

  49. [57]

    In: Advances in Neural Information Processing Systems, vol

    Vaswani, A., Shazeer, N., Parmar, N., Uszko- reit, J., Jones, L., Gomez, A.N., Kaiser, L.u., Polosukhin, I.: Attention is all you need. In: Advances in Neural Information Processing Systems, vol. 30 (2017)

    2017

  50. [58]

    arXiv preprint arXiv:2005.00214 (2020)

    Li, A., Thotakuri, M., Ross, D.A., Car- reira, J., Vostrikov, A., Zisserman, A.: The ava-kinetics localized human actions video dataset. arXiv preprint arXiv:2005.00214 (2020)

    2020 arXiv

  51. [59]

    https://github.com/ultralytics/ ultralytics

    Jocher, G., Chaurasia, A., Qiu, J.: Ultralytics YOLOv8. https://github.com/ultralytics/ ultralytics

  52. [60]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2024)

    Ke, B., Obukhov, A., Huang, S., Metzger, N., Daudt, R.C., Schindler, K.: Repurposing diffusion-based image generators for monoc- ular depth estimation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2024)

    2024

  53. [61]

    In: Proceedings of the 28th International Conference on Com- putational Linguistics, pp

    Bogolin, S.-V., Croitoru, I., Leordeanu, M.: A hierarchical approach to vision-based lan- guage generation: from simple sentences to complex natural language. In: Proceedings of the 28th International Conference on Com- putational Linguistics, pp. 2436–2447 (2020)

    2020

  54. [62]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp

    Tang, Y., Ding, D., Rao, Y., Zheng, Y., Zhang, D., Zhao, L., Lu, J., Zhou, J.: Coin: A large-scale dataset for comprehensive instructional video analysis. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1207– 1216 (2019)

    2019

  55. [63]

    In: Proceed- ings of the IEEE/CVF International Con- ference on Computer Vision, pp

    Bain, M., Nagrani, A., Varol, G., Zisserman, A.: Frozen in time: A joint video and image encoder for end-to-end retrieval. In: Proceed- ings of the IEEE/CVF International Con- ference on Computer Vision, pp. 1728–1738 (2021)

    2021

  56. [64]

    In: Proceed- ings of the 2019 on International Conference on Multimedia Retrieval, pp

    Shang, X., Di, D., Xiao, J., Cao, Y., Yang, X., Chua, T.-S.: Annotating objects and rela- tions in user-generated videos. In: Proceed- ings of the 2019 on International Conference on Multimedia Retrieval, pp. 279–287 (2019). ACM

    2019

  57. [65]

    In: Proceedings of the 40th Annual Meeting of the Associa- tion for Computational Linguistics, pp

    Papineni, K., Roukos, S., Ward, T., Zhu, W.-J.: Bleu: a method for automatic evalua- tion of machine translation. In: Proceedings of the 40th Annual Meeting of the Associa- tion for Computational Linguistics, pp. 311–

  58. [66]

    In: Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation And/or Summa- rization, pp

    Banerjee, S., Lavie, A.: METEOR: An automatic metric for MT evaluation with improved correlation with human judgments. In: Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation And/or Summa- rization, pp. 65–72. Association for ...

    2005

  59. [67]

    In: Text Summarization Branches Out, pp

    Lin, C.-Y.: ROUGE: A package for automatic evaluation of summaries. In: Text Summarization Branches Out, pp. 74–81. Association for Computational Linguistics, Barcelona, Spain (2004). https://aclanthology.org/W04-1013 27

    2004

  60. [68]

    In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp

    Vedantam, R., Lawrence Zitnick, C., Parikh, D.: Cider: Consensus-based image descrip- tion evaluation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4566–4575 (2015)

    2015

  61. [69]

    In: Computer Vision–ECCV 2016: 14th European Confer- ence, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part V 14, pp

    Anderson, P., Fernando, B., Johnson, M., Gould, S.: Spice: Semantic propositional image caption evaluation. In: Computer Vision–ECCV 2016: 14th European Confer- ence, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part V 14, pp. 382–398 (2016). Springer

    2016

  62. [70]

    arXiv preprint arXiv:1904.09675 (2019)

    Zhang, T., Kishore, V., Wu, F., Wein- berger, K.Q., Artzi, Y.: Bertscore: Evaluating text generation with bert. arXiv preprint arXiv:1904.09675 (2019)

    2019 arXiv

  63. [71]

    In: Proceedings of the 58th Annual Meeting of the Association for Computational Lin- guistics, pp

    Sellam, T., Das, D., Parikh, A.: BLEURT: Learning robust metrics for text generation. In: Proceedings of the 58th Annual Meeting of the Association for Computational Lin- guistics, pp. 7881–7892. Association for Com- putational Linguistics, Online (2020). https: //doi.org/10.1...

    2020 doi

  64. [72]

    arXiv preprint arXiv:2404.18796 (2024)

    Verga, P., Hofstatter, S., Althammer, S., Su, Y., Piktus, A., Arkhangorodsky, A., Xu, M., White, N., Lewis, P.: Replacing judges with juries: Evaluating llm generations with a panel of diverse models. arXiv preprint arXiv:2404.18796 (2024)

    2024 arXiv

  65. [73]

    arXiv preprint arXiv:2410.21276 (2024)

    Hurst, A., Lerer, A., Goucher, A.P., Perel- man, A., Ramesh, A., Clark, A., Ostrow, A., Welihinda, A., Hayes, A., Radford, A., et al.: Gpt-4o system card. arXiv preprint arXiv:2410.21276 (2024)

    2024 arXiv

  66. [74]

    arXiv preprint arXiv:2409.12191 (2024)

    Wang, P., Bai, S., Tan, S., Wang, S., Fan, Z., Bai, J., Chen, K., Liu, X., Wang, J., Ge, W., Fan, Y., Dang, K., Du, M., Ren, X., Men, R., Liu, D., Zhou, C., Zhou, J., Lin, J.: Qwen2-vl: Enhancing vision-language model’s percep- tion of the world at any resolution. arXiv prepri...

    2024 arXiv

  67. [75]

    Computational Linguistics 44(3), 393–401 (2018)

    Reiter, E.: A structured review of the valid- ity of bleu. Computational Linguistics 44(3), 393–401 (2018)

    2018

  68. [76]

    Computational Linguistics 35(4), 529–558 (2009)

    Reiter, E., Belz, A.: An investigation into the validity of some metrics for automati- cally evaluating natural language generation systems. Computational Linguistics 35(4), 529–558 (2009)

    2009

  69. [77]

    arXiv preprint arXiv:2408.09169 (2024)

    Schmidtov´ a, P., Mahamood, S., Balloccu, S., Duˇ sek, O., Gatt, A., Gkatzia, D., Howcroft, D.M., Pl´ atek, O., Sivaprasad, A.: Automatic metrics in natural language generation: A survey of current evaluation practices. arXiv preprint arXiv:2408.09169 (2024)

    2024 arXiv

  70. [78]

    Advances in Neural Information Processing Systems 36, 53728– 53741 (2023) 28

    Rafailov, R., Sharma, A., Mitchell, E., Man- ning, C.D., Ermon, S., Finn, C.: Direct pref- erence optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems 36, 53728– 53741 (2023) 28

    2023

  71. [318]

    https://doi.org/10.3115/1073083.1073135

    Association for Computational Linguis- tics, Philadelphia, Pennsylvania, USA (2002). https://doi.org/10.3115/1073083.1073135

    2002

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.