Pith. sign in

REVIEW 4 major objections 6 minor 1 cited by

TV-Dialogue: Crafting Theme-Aware Video Dialogues with Immersive Interaction

T0 review · 4 major / 6 minor · reviewed 2026-08-09 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read TV-Dialogue generates theme-aware video dialogue zero-shot by assigning each video character an LLM sub-agent that perceives its own visual cues in turn and self-corrects, beating GPT-4o, GPT-4V, and PLLaVA on a six-metric benchmark.

desk verdict New task and framework are genuinely useful; the headline performance claim rests on an unvalidated LLM judge and needs major evaluation work. read the letter →

arxiv 2501.18940 v1 pith:JL5W5VX4 submitted 2025-01-31 cs.CV

Sai Wang , Fan Ma , Xinyi Li , Hehe Fan , Yu Wu This is my paper · ORCID
classification cs.CV
keywords theme-awarevideodialoguecraftingmulti-agentgenerationzero-shotunderstandingvisualconsistencyself-correctionmultimodalevaluationbenchmarkvideo-textretrievalmulti-themedataset
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper introduces Theme-aware Video Dialogue Crafting (TVDC), a task in which a model invents a fresh dialogue for the characters in an existing video such that the lines follow a user-supplied theme while still matching what the characters are visibly doing and feeling. The proposed TV-Dialogue framework treats each video character as an independent LLM sub-agent that perceives its own emotional and behavioral cues frame by frame, speaks in turn, and accepts revision suggestions from a central agent. The paper reports that on its Multi-Theme Video Dialogue dataset, TV-Dialogue with GPT-4o as backbone outperformed GPT-4o, GPT-4V, and PLLaVA on every evaluation dimension, without any training, and that dialogues it generated improved downstream video-text retrieval. The practical stakes are that video re-creation, dubbing, and synthetic training-data creation could be automated for any theme.

What carries the argument

The load-bearing mechanism is a central-agent–sub-agent conversation loop. The central agent $A_0$ creates a new plot and roles from the theme, the first video frame, and ASR-transcribed original dialogue; each sub-agent $A_i$ keeps a state $s_t^i = [role_i; memory_t^i]$; at turn $t$ it obtains its own behavior $a_t^i$ and emotion $e_t^i$ from a vision-language model, then produces $d_t = A_i(s_{t-1}^i, a_t^i, e_t^i, d_{t-1})$. The central agent evaluates each line from local to global coherence and returns revision suggestion $o_t$, triggering regeneration $d'_t = A_i(s_{t-1}^i, a_t^i, e_t^i, d_{t-1}, o_t)$. This turn-by-turn, first-person generation is what keeps the dialogue both theme-aligned and visually consistent while avoiding the information loss of generating all lines at once.

What would settle it

A much larger and more diverse human preference study that fails to confirm the 72.5% preference for TV-Dialogue, or an audit showing the GPT-4o-mini judge assigns high scores to off-theme or visually mismatched lines, would undermine the central performance claim. For example, asking human raters to judge a 'presidential election' dialogue placed over a serious meeting scene would settle whether the benchmark's theme-relevance and scenario-consistency scores track actual quality.

Watch

Extended reading notes

Core claim

On the paper's own terms, the discovery is that theme-aware video dialogue does not require a specialized trained model; it can be assembled from an LLM plus a VLM through a staged multi-agent loop. A central agent first builds a theme-specific plot and assigns each character a role; each sub-agent then reads its own current behavior and emotion from the video, consults its memory of prior dialogue, generates exactly one line, and sends it to the other agents. A self-correction pass checks each line locally and globally, and the speaker regenerates it if needed. The paper shows this loop outperforms end-to-end text, image, and video LLMs across all six reported metrics, and that 72.5% of human preference judgments favored TV-Dialogue over GPT-4o in its user study.

Load-bearing premise

The central claim depends on trusting the automated scoring model as a fair judge of dialogue quality, since there is no ground-truth dialogue for the main comparison and the human check covers only 20 participants and 50 videos.

Editorial extensions

If this is right

  • Any existing LLM can be turned into a themed video-dialogue generator by orchestrating it as TV-Dialogue; even an 8B model outperforms text-only GPT-4o on the reported metrics.
  • Since no training or per-video ground truth is required, creators can re-dub or re-voice arbitrary videos on arbitrary themes as a zero-shot service.
  • The generated dialogues carry enough video–text correspondence to serve as synthetic training data, improving R@5 by more than 6% in video-text retrieval when added to original dialogue.
  • A reference-free multi-granularity evaluation protocol (scores plus comments, text- and video-oriented) becomes available for dialogue tasks that lack ground truth.
  • Because each line is tied to a specific moment's facial expression and body movement, the framework promises finer temporal alignment between speech and on-screen action than one-shot generation.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The same three-stage recipe—role assignment, perception-driven turn prediction, self-correction—could be carried over to time-aligned text generation beyond dialogue, such as narrated silent films, sports commentary, or archival-footage captioning, whenever a theme constrains the text.
  • The 20-participant, 50-video human validation leaves room to test whether the GPT-4o-mini judge tracks human preference on strongly theme-conflicting videos; Figure 5 suggests those are exactly the cases where all methods degrade.
  • The retrieval result hints that generated dialogue can act as a free caption-augmentation signal; a natural extension is checking whether the same gain appears on standard large-scale retrieval benchmarks, not just the 351-video MVD split.
  • The paper's impact statement acknowledges that generated dialogue can misrepresent original video content under some themes; that admission points to a testable boundary, namely that theme-scene conflict should measurably lower both theme relevance and scenario consistency.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper introduces Theme-aware Video Dialogue Crafting (TVDC), a task in which a model must generate new character dialogue for a given video clip that both matches the video content and follows a user-specified theme. The authors propose TV-Dialogue, a multi-agent framework built on LLMs and a VLM: a central agent creates a theme-consistent plot and roles, sub-agents generate dialogue turn-by-turn using visual cues about emotion and behavior, and a self-correction module revises outputs that fail local and global coherence checks. The paper also introduces the MVD dataset (351 clips, 10 themes) and a six-metric evaluation benchmark scored by GPT-4o-mini. Experiments compare TV-Dialogue against text, image, and video LLM baselines, report ablations, a last-K sentence prediction study, a downstream video-text retrieval experiment, and a 20-participant user study showing 72.5% preference over GPT-4o. The paper claims zero-shot operation and applicability to videos of any length and any theme.

Significance. If the evaluation evidence is sound, this is a useful contribution: the task is new, the multi-agent design is well motivated, the MVD dataset is a resource, and the downstream retrieval improvement provides an external validation signal. The user study, although small, favors the method. However, the headline comparisons rest almost entirely on a single LLM judge whose agreement with human judgment is only weakly established, and several secondary claims go beyond the tested regime. The practical value of the framework is therefore conditional on stronger evaluation evidence.

major comments (4)
  1. [§4.2, Table 3] The central claim that TV-Dialogue 'consistently outperformed other approaches across all metrics' rests almost entirely on scores produced by the GPT-4o-mini evaluation pipeline. The human evidence in §4.7 is limited to (i) a pairwise preference test between TV-Dialogue (GPT-4o) and GPT-4o, which does not cover GPT-3.5, GPT-4V, PLLaVA, or the open LLMs, and (ii) three annotators rating TR and VC on 50 videos, yielding Pearson r=0.47 and 0.53 (Table 7). The other four metrics and the remaining baselines are unvalidated. The statement in §4.2 that 'changing the evaluation model will not affect the evaluation results' is unsupported, and the reported standard deviation of less than 0.01 only reflects run-to-run determinism at temperature 0, not inter-judge or judge-versus-human reliability. To support the headline, the authors should provide human scores for all six metrics across all methods or a representative subset, compute rank-order agreement between the LLM judge and humans, and report confidence intervals or significance tests. Without this, the margins in Table 3 (e.g., average 3.84 vs 3.33) cannot be interpreted as evidence of superiority.
  2. [§4.6, Table 4] The ablations are presented as evidence for each module's effectiveness, but the differences are very small and not evaluated statistically: the average score moves from 3.65 to 3.71 to 3.75, and individual metrics sometimes decrease (e.g., GQ 3.91 to 3.86 and SC 3.34 to 3.24 when adding the visual module). Since the scores are ordinal ratings from an LLM judge, the authors should report per-video standard deviations, paired significance tests (e.g., Wilcoxon signed-rank), and effect sizes before concluding that 'each component' contributes. Otherwise the observed modular gains are not distinguishable from noise.
  3. [§4.6, Table 6] The downstream retrieval experiment overstates the benefit. Table 6 shows that training on the 'New' dialogues alone degrades R@1 (16.0 to 12.0) and R@5 (32.0 to 28.0) relative to training on the original dialogues; only the combined 'Original+New' setting improves R@5 (32.0 to 38.0), while R@1 is unchanged (16.0) and the test set contains only 50 videos. No confidence intervals or significance tests are reported. The claim of 'more than 6%' improvement should be restricted to the combined training setting and qualified accordingly.
  4. [Abstract, §1, §4.1, Algorithm 1] The abstract and introduction claim that TV-Dialogue can handle videos of 'ANY length' and 'any theme' in a zero-shot manner, but the experiments only cover the MVD dataset, whose average video length is 16.49 seconds and which contains 10 hand-picked themes (Table 1, Figure 3). No long-video or out-of-distribution-theme evaluation is reported, and Algorithm 1's sequential per-round processing with growing memory provides no obvious guarantee of unbounded-length behavior. Please either restrict the claim to the tested regime or provide supporting experiments, such as length scaling and evaluation on novel themes.
minor comments (6)
  1. [§4.2] The evaluation benchmark is described only at a high level; the exact prompts, scoring rubrics, and aggregation rule for the six metrics are missing, so the results are not reproducible. Also, the claim that temperature-0 decoding gives a standard deviation below 0.01 conflates run-to-run variance with evaluation reliability.
  2. [Table 4] The checkmark encoding is ambiguous; please label each row explicitly (e.g., 'Role only', '+Visual', '+Visual+Correction') so readers can map the configurations without guessing.
  3. [§4.7] Please clarify how the 400 responses were distributed across the 20 participants and report inter-annotator agreement (e.g., Krippendorff's alpha) for the three annotators; Pearson r=0.47 and 0.53 are weak-to-moderate correlations and do not by themselves establish reliability of the benchmark.
  4. [§4.6] The phrase 'generate new dialogues for any video at no cost' should be rephrased as 'without manual annotation cost', since the framework incurs LLM and VLM inference costs.
  5. [Figure 5] The per-theme comparison is plotted without error bars or per-theme sample sizes; because the scores come from a single LLM judge, please include variance information or state the number of videos per theme.
  6. [References] The BLEU reference is misspelled as 'Papinesi' and should be 'Papineni'; a few other typographical errors remain in the references and main text.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the generation pipeline contains no fitted parameters or equation-level reductions, and the headline comparisons are against external baselines with additional human and downstream-task validation.

full rationale

We walked the paper's derivation chain. The method (role generation in Sec. 3.2, visual-driven prediction in Sec. 3.3, and self-correction in Sec. 3.4) is a prompting-based multi-agent loop; it estimates no parameters from the evaluation scores and then reports those scores as a prediction. The headline claim in Sec. 4.5 (Table 3) compares TV-Dialogue against external baselines (GPT-3.5, GPT-4o, GPT-4V, PLLaVA, LLaMA-3.1, QWen-2.5, GLM) under a fixed protocol, not against the method's own fitted outputs. The user study in Sec. 4.7 and the video-text retrieval transfer experiment in Sec. 4.6 (Table 6) provide additional external checks. The nearest thing to circularity is that the GPT-4o-mini evaluation benchmark in Sec. 4.2 is authored by the same team and is only partially human-validated (TR and VC on 50 videos, Pearson r = 0.47 and 0.53), and the paper asserts without evidence that changing the evaluation model will not affect results. These are legitimate reliability and validity concerns about the benchmark, but they are not instances of Eq. X reducing to Eq. Y by construction, of a fitted parameter being renamed as a prediction, or of a load-bearing self-citation chain. Under the hard rules requiring a specific exhibited reduction, the circularity score is 0.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

No free parameters are fitted to data; the method is zero-shot. The main assumptions are the reliability of the LLM-based evaluation, the fidelity of VLM visual perception, and the representativeness of the self-collected dataset. No new physical or formal entities are introduced.

assumptions (3)
  • domain assumption LLM judgments (GPT-4o-mini) are a valid proxy for human dialogue quality.
    The entire evaluation benchmark in Section 4.2 relies on GPT-4o-mini scoring of six dialogue qualities; the paper only provides partial human correlation evidence (Table 7) for two of the six metrics.
  • domain assumption The PLLaVA VLM accurately extracts characters' emotions and behaviors from video frames.
    Stage 2 (Section 3.3) assumes that the VLM's perception of emotions and behaviors is accurate enough to guide dialogue generation and to serve as a grounding signal for visual consistency.
  • domain assumption The 10 handpicked themes and self-collected MVD videos are representative of real-world video dialogue scenarios.
    Section 4.1 describes the MVD dataset as self-collected with handpicked themes; the generality of the 'any theme' claim depends on this representativeness.

how reviews work

0 comments
Cite this review

Pith. "Pith review of TV-Dialogue: Crafting Theme-Aware Video Dialogues with Immersive Interaction." pith.science (2026). https://pith.science/paper/JL5W5VX4

@misc{pith2026250118940,
  author       = {Pith},
  title        = {Pith review of: TV-Dialogue: Crafting Theme-Aware Video Dialogues with Immersive Interaction},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/JL5W5VX4}},
  note         = {Machine review of arXiv:2501.18940}
}
read the original abstract

Recent advancements in LLMs have accelerated the development of dialogue generation across text and images, yet video-based dialogue generation remains underexplored and presents unique challenges. In this paper, we introduce Theme-aware Video Dialogue Crafting (TVDC), a novel task aimed at generating new dialogues that align with video content and adhere to user-specified themes. We propose TV-Dialogue, a novel multi-modal agent framework that ensures both theme alignment (i.e., the dialogue revolves around the theme) and visual consistency (i.e., the dialogue matches the emotions and behaviors of characters in the video) by enabling real-time immersive interactions among video characters, thereby accurately understanding the video content and generating new dialogue that aligns with the given themes. To assess the generated dialogues, we present a multi-granularity evaluation benchmark with high accuracy, interpretability and reliability, demonstrating the effectiveness of TV-Dialogue on self-collected dataset over directly using existing LLMs. Extensive experiments reveal that TV-Dialogue can generate dialogues for videos of any length and any theme in a zero-shot manner without training. Our findings underscore the potential of TV-Dialogue for various applications, such as video re-creation, film dubbing and its use in downstream multimodal tasks.

Figures

Figures reproduced from arXiv: 2501.18940 by the authors.

Figure 1
Figure 1. Given an arbitrary user-specified theme, the Theme￾aware Video Dialogue Crafting (TVDC) task seeks to generate novel dialogues aligned with video content and theme. The solid box represents the original dialogue, while the dashed box repre￾sents the new dialogue about the “presidential election”. 2016; Huang et al., 2018) and image-based dialogue gen￾eration (Yang et al., 2021; Sun et al., 2022; Shen et al., 2021). … view at source ↗
Figure 2
Figure 2. Overview of TV-Dialogue. The TV-Dialogue initially assigns a relevant role to each sub-agent based on the given theme and video, enabling immersive interaction among the sub-agents in the dialogue process (Stage 1). Sub-agents maintain visual consistency by perceiving video content, querying historical memory, and receiving messages from other agents, thereby generating high-quality dialogues (Stage 2). The generate… view at source ↗
Figure 3
Figure 3. [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Comparison of dialogues generated by different methods in the last-1 sentence prediction. The top-left corner represents the first frame of the dialogue in the video. Although the values of traditional metrics are very low, the generated dialogues are consistent with t…
Figure 5
Figure 5. Figure 5: Comparison of different themes in terms of Theme Rele￾vance (TR) and Scenario Consistency (SC). Relevance, where the variance of TV-Dialogue (GPT-4o) across themes is only 0.019. 4.7. User Study To comprehensively and reliably evaluate the generated dialogue, we perfor…

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. ShoulderShot: Generating Over-the-Shoulder Dialogue Videos

    cs.CV 2025-08 unverdicted novelty 5.0 of 10

    ShoulderShot generates over-the-shoulder dialogue videos by pairing two linked camera shots and looping them, so characters stay consistent through long multi-turn conversations.

Reference graph

Works this paper leans on

59 extracted references · 55 canonical work pages · cited by 1 Pith paper

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  2. [2]

    L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al

    Achiam, J., Adler, S., Agarwal, S., Ahmad, L., Akkaya, I., Aleman, F. L., Almeida, D., Altenschmidt, J., Altman, S., Anadkat, S., et al. Gpt-4 technical report. arXiv, 2023

    2023

  3. [3]

    Qwen technical report

    Bai, J., Bai, S., Chu, Y., Cui, Z., Dang, K., Deng, X., Fan, Y., Ge, W., Han, Y., Huang, F., et al. Qwen technical report. arXiv, 2023

    work page 2023

  4. [4]

    and Lavie, A

    Banerjee, S. and Lavie, A. Meteor: An automatic metric for mt evaluation with improved correlation with human judgments. In Proceedings of the acl workshop on intrinsic and extrinsic evaluation measures for machine translation and/or summarization, pp.\ 65--72, 2005

    work page 2005

  5. [5]

    Diaggpt: An llm-based chatbot with automatic topic management for task-oriented dialogue

    Cao, L. Diaggpt: An llm-based chatbot with automatic topic management for task-oriented dialogue. arXiv, 2023

    work page 2023

  6. [6]

    A survey on evaluation of large language models

    Chang, Y., Wang, X., Wang, J., Wu, Y., Yang, L., Zhu, K., Chen, H., Yi, X., Wang, C., Wang, Y., et al. A survey on evaluation of large language models. ACM Trans. Intell. Syst. Technol., 15 0 (3): 0 1--45, 2024

    work page 2024

  7. [7]

    A survey on dialogue systems: Recent advances and new frontiers

    Chen, H., Liu, X., Yin, D., and Tang, J. A survey on dialogue systems: Recent advances and new frontiers. SIGKDD Explor. Newsl., 19 0 (2): 0 25--35, 2017

    work page 2017

  8. [8]

    Multimodal fusion of visual dialog: A survey

    Chen, X., Lao, S., and Duan, T. Multimodal fusion of visual dialog: A survey. In RICAI '20, pp.\ 302--308, 2020

    work page 2020

Show all 59 references
  1. [9]

    Llm as a scorer: The impact of output order on dialogue evaluation

    Chen, Y.-P., Chu, K., and Nakayama, H. Llm as a scorer: The impact of output order on dialogue evaluation. arXiv, 2024

    2024

  2. [10]

    and Rei, M

    Cummins, R. and Rei, M. Neural multi-task learning in automated assessment. arXiv, 2018

    2018

  3. [11]

    Videoagent: A memory-augmented multimodal agent for video understanding

    Fan, Y., Ma, X., Wu, R., Du, Y., Li, J., Gao, Z., and Li, Q. Videoagent: A memory-augmented multimodal agent for video understanding. In ECCV, 2024

    2024

  4. [12]

    MMD ialog: A large-scale multi-turn dialogue dataset towards multi-modal open-domain conversation

    Feng, J., Sun, Q., Xu, C., Zhao, P., Yang, Y., Tao, C., Zhao, D., and Lin, Q. MMD ialog: A large-scale multi-turn dialogue dataset towards multi-modal open-domain conversation. In ACL, pp.\ 7348--7363, 2023

    2023

  5. [13]

    Chatglm: A family of large language models from glm-130b to glm-4 all tools

    GLM, T., Zeng, A., Xu, B., Wang, B., Zhang, C., Yin, D., Zhang, D., Rojas, D., Feng, G., Zhao, H., et al. Chatglm: A family of large language models from glm-130b to glm-4 all tools. arXiv, 2024

    2024

  6. [14]

    Champagne: Learning real-world conversation from large-scale web videos

    Han, S., Hessel, J., Dziri, N., Choi, Y., and Yu, Y. Champagne: Learning real-world conversation from large-scale web videos. In ICCV, pp.\ 15498--15509, 2023

    2023

  7. [15]

    Metrics and evaluation of spoken dialogue systems

    Hastie, H. Metrics and evaluation of spoken dialogue systems. In Data-Driven Methods for Adaptive Spoken Dialogue Systems: Computational Learning for Conversational Interfaces, pp.\ 131--150. Springer, 2012

    2012

  8. [16]

    M., Belz, A., Clinciu, M., Gkatzia, D., Hasan, S

    Howcroft, D. M., Belz, A., Clinciu, M., Gkatzia, D., Hasan, S. A., Mahamood, S., Mille, S., Van Miltenburg, E., Santhanam, S., and Rieser, V. Twenty years of confusion in human evaluation: Nlg needs evaluation sheets and standardised definitions. In INLG, pp.\ 169--182, 2020

    2020

  9. [17]

    R., Trabelsi, A., and Dziri, N

    Huang, C., Zaiane, O. R., Trabelsi, A., and Dziri, N. Automatic dialogue generation with expressed emotions. In NAACL, pp.\ 49--54, 2018

    2018

  10. [18]

    Crafting papers on machine learning

    Langley, P. Crafting papers on machine learning. In Langley, P. (ed.), Proceedings of the 17th International Conference on Machine Learning (ICML 2000), pp.\ 1207--1216, Stanford, CA, 2000. Morgan Kaufmann

    2000

  11. [19]

    Deep reinforcement learning for dialogue generation

    Li, J., Monroe, W., Ritter, A., Jurafsky, D., Galley, M., and Gao, J. Deep reinforcement learning for dialogue generation. In EMNLP, pp.\ 1192--1202, 2016

    2016

  12. [20]

    Knowledge bridging for empathetic dialogue generation

    Li, Q., Li, P., Ren, Z., Ren, P., and Chen, Z. Knowledge bridging for empathetic dialogue generation. In AAAI, volume 36, pp.\ 10993--11001, 2022

    2022

  13. [21]

    How to evaluate your dialogue models: a review of approaches

    Li, X., Wu, W., Qin, L., and Yin, Q. How to evaluate your dialogue models: a review of approaches. arXiv, 2021

    2021

  14. [22]

    Knowledge-aware multimodal dialogue systems

    Liao, L., Ma, Y., He, X., Hong, R., and Chua, T.-s. Knowledge-aware multimodal dialogue systems. In ACM MM, pp.\ 801--809, 2018

    2018

  15. [23]

    Rouge: A package for automatic evaluation of summaries

    Lin, C.-Y. Rouge: A package for automatic evaluation of summaries. In Text summarization branches out, pp.\ 74--81, 2004

    2004

  16. [24]

    X., Jin, Q., et al

    Lin, H., Ruan, L., Xia, W., Liu, P., Wen, J., Xu, Y., Hu, D., Song, R., Zhao, W. X., Jin, Q., et al. Tiktalk: A video-based dialogue dataset for multi-modal chitchat in real world. In ACM MM, pp.\ 1303--1313, 2023

    2023

  17. [25]

    and Chen, Y.-N

    Lin, Y.-T. and Chen, Y.-N. LLM -eval: Unified multi-dimensional automatic evaluation for open-domain conversations with large language models. In NLP4ConvAI, pp.\ 47--58, July 2023

    2023

  18. [26]

    A survey on multimodal dialogue systems: recent advances and new frontiers

    Liu, G., Wang, S., Yu, J., and Yin, J. A survey on multimodal dialogue systems: recent advances and new frontiers. In AEMCSE, pp.\ 845--853, 2022

    2022

  19. [27]

    Llava-plus: Learning to use tools for creating multimodal agents

    Liu, S., Cheng, H., Liu, H., Zhang, H., Li, F., Ren, T., Zou, X., Yang, J., Su, H., Zhu, J., et al. Llava-plus: Learning to use tools for creating multimodal agents. In ECCV, 2024

    2024

  20. [28]

    Clip4clip: An empirical study of clip for end to end video clip retrieval and captioning

    Luo, H., Ji, L., Zhong, M., Chen, Y., Lei, W., Duan, N., and Li, T. Clip4clip: An empirical study of clip for end to end video clip retrieval and captioning. Neurocomput., 508: 0 293--304, 2022

    2022

  21. [29]

    Video- C hat GPT : Towards detailed video understanding via large vision and language models

    Maaz, M., Rasheed, H., Khan, S., and Khan, F. Video- C hat GPT : Towards detailed video understanding via large vision and language models. In ACL, pp.\ 12585--12602, 2024

    2024

  22. [30]

    Simple LLM prompting is state-of-the-art for robust and multilingual dialogue evaluation

    Mendon c a, J., Pereira, P., Moniz, H., Paulo Carvalho, J., Lavie, A., and Trancoso, I. Simple LLM prompting is state-of-the-art for robust and multilingual dialogue evaluation. In DSTC, pp.\ 133--143, September 2023

    2023

  23. [31]

    Recent advances in deep learning based dialogue systems: A systematic survey

    Ni, J., Young, T., Pandelea, V., Xue, F., and Cambria, E. Recent advances in deep learning based dialogue systems: A systematic survey. Artif Intell Rev, 56 0 (4): 0 3055--3155, 2023

    2023

  24. [32]

    Hello gpt-4o, May 2024

    OpenAI. Hello gpt-4o, May 2024. https://openai.com/index/hello-gpt-4o/

    2024

  25. [33]

    D ialog B ench: Evaluating LLM s as human-like dialogue systems

    Ou, J., Lu, J., Liu, C., Tang, Y., Zhang, F., Zhang, D., and Gai, K. D ialog B ench: Evaluating LLM s as human-like dialogue systems. In NAACL, pp.\ 6137--6170, 2024

    2024

  26. [34]

    Training language models to follow instructions with human feedback

    Ouyang, L., Wu, J., Jiang, X., Almeida, D., Wainwright, C., Mishkin, P., Zhang, C., Agarwal, S., Slama, K., Ray, A., et al. Training language models to follow instructions with human feedback. In NeurIPS, volume 35, pp.\ 27730--27744, 2022

    2022

  27. [35]

    Bleu: A method for automatic evaluation of machine translation

    Papinesi, K. Bleu: A method for automatic evaluation of machine translation. In ACL, pp.\ 311--318, 2002

    2002

  28. [36]

    W., Xu, T., Brockman, G., McLeavey, C., and Sutskever, I

    Radford, A., Kim, J. W., Xu, T., Brockman, G., McLeavey, C., and Sutskever, I. Robust speech recognition via large-scale weak supervision. In ICML, pp.\ 28492--28518, 2023

    2023

  29. [37]

    What makes a good conversation? how controllable attributes affect human judgments

    See, A., Roller, S., Kiela, D., and Weston, J. What makes a good conversation? how controllable attributes affect human judgments. In NAACL, pp.\ 1702--1723, 2019

    2019

  30. [38]

    BLEURT : Learning robust metrics for text generation

    Sellam, T., Das, D., and Parikh, A. BLEURT : Learning robust metrics for text generation. In ACL, pp.\ 7881--7892, 2020

    2020

  31. [39]

    Text is not enough: Integrating visual impressions into open-domain dialogue generation

    Shen, L., Zhan, H., Shen, X., Song, Y., and Zhao, X. Text is not enough: Integrating visual impressions into open-domain dialogue generation. In ACM MM, pp.\ 4287--4296, 2021

    2021

  32. [40]

    Image-chat: Engaging grounded conversations

    Shuster, K., Humeau, S., Bordes, A., and Weston, J. Image-chat: Engaging grounded conversations. In ACL, pp.\ 2414--2429, 2020

    2020

  33. [41]

    Generating responses with a specific emotion in dialog

    Song, Z., Zheng, X., Liu, L., Xu, M., and Huang, X.-J. Generating responses with a specific emotion in dialog. In ACL, pp.\ 3685--3695, 2019

    2019

  34. [42]

    Multimodal dialogue response generation

    Sun, Q., Wang, Y., Xu, C., Zheng, K., Yang, Y., Hu, H., Xu, F., Zhang, J., Geng, X., and Jiang, D. Multimodal dialogue response generation. In ACL, pp.\ 2854--2866, 2022

    2022

  35. [43]

    and Heck, L

    Sundar, A. and Heck, L. Multimodal conversational AI : A survey of datasets and approaches. In NLP4ConvAI, pp.\ 131--147, 2022

    2022

  36. [44]

    Ruber: An unsupervised method for automatic evaluation of open-domain dialog systems

    Tao, C., Mou, L., Zhao, D., and Yan, R. Ruber: An unsupervised method for automatic evaluation of open-domain dialog systems. In AAAI, volume 32, 2018

    2018

  37. [45]

    Llama: Open and efficient foundation language models

    Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozi \`e re, B., Goyal, N., Hambro, E., Azhar, F., et al. Llama: Open and efficient foundation language models. arXiv, 2023

    2023

  38. [46]

    On evaluating and comparing open domain dialog systems

    Venkatesh, A., Khatri, C., Ram, A., et al. On evaluating and comparing open domain dialog systems. arXiv, 2018

    2018

  39. [47]

    and Le, Q

    Vinyals, O. and Le, Q. A neural conversational model. arXiv, 2015

    2015

  40. [48]

    K.-W., and Lim, E.-P

    Wang, L., Xu, W., Lan, Y., Hu, Z., Lan, Y., Lee, R. K.-W., and Lim, E.-P. Plan-and-solve prompting: Improving zero-shot chain-of-thought reasoning by large language models. In ACL, pp.\ 2609--2634, July 2023

    2023

  41. [49]

    Videoagent: Long-form video understanding with large language model as agent

    Wang, X., Zhang, Y., Zohar, O., and Yeung-Levy, S. Videoagent: Long-form video understanding with large language model as agent. In ECCV, 2024

    2024

  42. [50]

    Autogen: Enabling next-gen llm applications via multi-agent conversation framework

    Wu, Q., Bansal, G., Zhang, J., Wu, Y., Zhang, S., Zhu, E., Li, B., Jiang, L., Zhang, X., and Wang, C. Autogen: Enabling next-gen llm applications via multi-agent conversation framework. In ICLRW, 2024

    2024

  43. [51]

    K., and Feng, J

    Xu, L., Zhao, Y., Zhou, D., Lin, Z., Ng, S. K., and Feng, J. Pllava: Parameter-free llava extension from images to videos for video dense captioning. arXiv, 2024

    2024

  44. [52]

    Open domain dialogue generation with latent images

    Yang, Z., Wu, W., Hu, H., Xu, C., Wang, W., and Li, Z. Open domain dialogue generation with latent images. In AAAI, volume 35, pp.\ 14239--14247, 2021

    2021

  45. [53]

    and Lewis, M

    Yarats, D. and Lewis, M. Hierarchical text generation and planning for strategic dialogue. In ICML, pp.\ 5591--5599, 2018

    2018

  46. [54]

    F., Chen, Y., Zhang, M., and Li, H

    Zhang, C., D'Haro, L. F., Chen, Y., Zhang, M., and Li, H. A comprehensive analysis of the effectiveness of large language models as automatic dialogue evaluators. In AAAI, volume 38, pp.\ 19515--19524, 2024

    2024

  47. [55]

    Q., and Artzi, Y

    Zhang, T., Kishore, V., Wu, F., Weinberger, K. Q., and Artzi, Y. Bertscore: Evaluating text generation with bert. In ICLR, 2019

    2019

  48. [56]

    M 3 ED : Multi-modal multi-scene multi-label emotional dialogue database

    Zhao, J., Zhang, T., Hu, J., Liu, Y., Jin, Q., Wang, X., and Li, H. M 3 ED : Multi-modal multi-scene multi-label emotional dialogue database. In ACL, pp.\ 5699--5710, May 2022

    2022

  49. [57]

    Designing precise and robust dialogue response evaluators

    Zhao, T., Lala, D., and Kawahara, T. Designing precise and robust dialogue response evaluators. In ACL, pp.\ 26--33, July 2020

    2020

  50. [58]

    X., Zhou, K., Li, J., Tang, T., Wang, X., Hou, Y., Min, Y., Zhang, B., Zhang, J., Dong, Z., et al

    Zhao, W. X., Zhou, K., Li, J., Tang, T., Wang, X., Hou, Y., Min, Y., Zhang, B., Zhang, J., Dong, Z., et al. A survey of large language models. arXiv, 2023

    2023

  51. [59]

    MMC hat: Multi-modal chat dataset on social media

    Zheng, Y., Chen, G., Liu, X., and Sun, J. MMC hat: Multi-modal chat dataset on social media. In LREC, pp.\ 5778--5786, June 2022

    2022

Pith tools

Reviewed August 9, 2026 · model on record in the stance chip above.