Pith. sign in

REVIEW 6 major objections 7 minor 55 references

LGSRR claims that a small classifier trained on LLM-generated, rank-ordered semantic descriptions of actions, expressions, and interactions outperforms both classical multimodal fusion and fine-tuned multimodal LLMs on intent and dialogue-a

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

LGSRR uses LLM-generated semantic descriptions and rankings to improve multimodal intent recognition, reporting SOTA results on MIntRec2.0 and IEMOCAP-DA with gains around 0.5-1.3%.

T0 review reviewed 2026-08-05 challenge →

load-bearing objection A solid, well-engineered LLM-to-small-model supervision pipeline that is genuinely new, but the SOTA claim rests on thin margins and missing significance tests. the 6 major comments →

arxiv 2509.01337 v1 pith:RMMIPMG5 submitted 2025-09-01 cs.MM cs.AIcs.CL

LLM-Guided Semantic Relational Reasoning for Multimodal Intent Recognition

Qianrui Zhou , Hua Xu , Yifan Wang , Xinzhi Dong , Hanlei Zhang This is my paper
classification cs.MM cs.AIcs.CL
keywords multimodal intent recognitiondialogue act recognitionlarge language modelschain-of-thoughtsemantic relational reasoningfine-grained semanticslogic-inspired relationsranking loss
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper proposes LGSRR, a method for multimodal intent and dialogue-act recognition that replaces raw audio/video processing with fine-grained semantic descriptions produced by a large language model. A three-step chain-of-thought prompts a video-language model to describe speakers' actions, facial expressions, and interactions, then a second LLM ranks these cues by contribution to the true intent. A small BERT-based classifier reads these descriptions and models three logic-inspired relations among them—relative importance (or), complementarity (and), and inconsistency (not)—guided by a ranking loss. On the MIntRec2.0 and IEMOCAP-DA benchmarks, the method reports the highest scores on five of six and all six metrics respectively, outperforming both classical fusion baselines and fine-tuned multimodal LLMs. If correct, it shows that a small, trainable model can beat much larger multimodal models by exploiting LLM-generated semantic foundations.

Core claim

The central claim is that fine-grained, intent-relevant semantics extracted and prioritized by LLMs give a small classifier enough information to outperform systems that fuse raw modality features. The paper's innovation is a shallow-to-deep chain-of-thought: first an LLM discovers candidate semantic aspects, then a video-language model generates structured descriptions of actions, expressions, and interactions from text plus 16 video frames, and finally an LLM ranks those aspects by their contribution to the ground-truth intent. The classifier encodes text and descriptions with BERT and combines them through three relations mapped from logic: weighted importance ('or'), cosine-similarity co

What carries the argument

The load-bearing mechanism is the LLM-guided semantic extraction pipeline feeding a small relational reasoning network. The semantic descriptions D = {DA, DE, DI} generated by a video-language model from text and video transform the multimodal input into text-form semantics; the LLM-generated importance ranking R supervises learned weights α through a differentiable ranking loss. Those weights implement relative importance, cosine similarity implements complementarity, and mean squared error between text and description features implements inconsistency; the final prediction is computed as weighted complementarity-enhanced features minus the inconsistency penalty.

Load-bearing premise

The method assumes that the text descriptions generated from the transcript and 16 video frames capture every intent-relevant cue, including audio and prosody, because raw audio and full video are never used.

What would settle it

On a held-out set of clips where the words are neutral but the tone conveys sarcasm or a joke, compare LGSRR against the same model given only the transcript; if accuracy does not fall, the semantic-description channel is not carrying the audio-based intent cues the method claims to harness.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • If correct, intent recognition can be improved without fine-tuning giant multimodal models; a BERT-scale classifier with LLM-written descriptions suffices.
  • The approach transfers across tasks: the same pipeline works for both intent classification (MIntRec2.0) and dialogue-act classification (IEMOCAP-DA), suggesting a general recipe.
  • The LLM guidance is reusable: the ranked semantics and descriptions act as training-only supervision, so at inference only the small model and precomputed descriptions are needed.
  • Replacing the description generator with different MLLMs causes little drop, implying the benefits come from the semantic-description paradigm, not one model.
  • Training cost is stated to be roughly two hours per dataset, well below the 8–10 hours reported for fine-tuning MLLMs.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Because audio and full video are never directly processed, the method's ceiling is set by what the video-language model can verbalize from text and 16 frames; intents carried chiefly by prosody, tone, or long-range visual dynamics may be invisible to it.
  • The three relations (importance, complementarity, inconsistency) are simple and symmetric; richer relations such as asymmetric entailment, temporal sequence, or speaker-specific cues could extend the same LLM-guided pipeline.
  • An interesting test would be to apply the same framework on datasets where the text is deliberately misleading relative to the video (sarcasm or jokes), to see whether the inconsistency term measurably improves accuracy on those classes.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

6 major / 7 minor

Summary. The paper proposes LGSRR, a framework for multimodal intent recognition that uses LLMs to extract fine-grained semantic descriptions (speakers' actions, facial expressions, interactions with others) and importance rankings, and then trains a compact BERT-based model with three relation-aware modules: relative importance (via a NeuralNDCG ranking loss), complementarity (cosine similarity), and inconsistency (MSE). Experiments on MIntRec2.0 and IEMOCAP-DA compare LGSRR with six recent baselines, ablations, and several MLLMs, reporting improvements on most metrics. The authors claim consistent state-of-the-art performance and release code and data.

Significance. If the claimed gains are reliable, LGSRR would be a valuable lightweight alternative to fine-tuned MLLMs, with interpretable semantic descriptions and an explicit relational-reasoning structure. The paper has notable strengths: it provides public code and data, uses five-run averaged results, evaluates against multiple baselines and MLLM settings, and includes detailed appendices on prompts, ranking statistics, and training cost. However, the central 'consistent SOTA' claim is currently undermined by (i) a missing statistical analysis with small metric differences, (ii) a confounded key ablation, and (iii) an overstatement of autonomy given the manual and label-dependent priors. The contribution is promising but needs substantial verification and recalibration before it can be accepted.

major comments (6)
  1. [Abstract; §4.2, Table 1] The abstract claims 'consistent performance gains' and 'superiority over state-of-the-art methods', but Table 1 shows LGSRR's accuracy on MIntRec2.0 (60.46) is lower than MulT (60.66). The §4.2 text acknowledges this only as 'a comparable ACC score'. Since ACC is the primary metric reported first, the SOTA claim must be qualified. Please either adjust the abstract and central claim to specify 'five of six metrics' or provide evidence that the ACC difference is not meaningful (e.g., via significance testing).
  2. [§4.1, §4.2, Table 1] All results are averages over five seeds, but no standard deviations, confidence intervals, or significance tests are reported. Many of the claimed gains are small in absolute terms — e.g., F1 +0.61 over MAG-BERT on MIntRec2.0, WF1 +0.11, and on IEMOCAP-DA ACC +0.58 over MIntOOD. Without variance estimates or paired tests, these differences could easily be within run-to-run noise. The phrase 'consistent performance gains' and the 'substantial gains' language in §4.2 are therefore not statistically supported. Please report error bars and run significance tests on the central comparisons, or temper the claims accordingly.
  3. [§4.3, Table 2] The 'w/o LGSE' ablation is confounded. Removing LGSE replaces the entire semantic-description pipeline (VideoLLaMA2 descriptions and ranking) with raw modality features extracted by BERT, Swin Transformer, and WavLM from text, video, and audio. This changes both the semantic granularity and the input modalities/encoders, so the observed drop cannot be attributed specifically to the LLM-guided fine-grained semantic extraction. A cleaner ablation would keep the same raw-modality backbones or the same description format without LLM ranking, or replace LLM descriptions with generic/non-guided captions. As written, the ablation does not isolate the contribution of LGSE.
  4. [§3.1 (Eq. 3), §3.2, Appendix B, Abstract] The paper repeatedly states that the method 'autonomously uncovers' semantics 'without relying on manually defined priors'. However, several important priors are manual or label-dependent: K=3 is selected by validation (Appendix B); the semantic set S={A,E,I} is chosen by frequency and then filtered; text is manually assigned the top rank in R (§3.2); and CoT-Step 3 uses the ground-truth label y to generate ranking supervision (Eq. 3). The ranking loss is therefore an auxiliary label-dependent supervisory signal, not an autonomous discovery mechanism. Please revise the contribution claims to acknowledge these design choices and the role of y during training.
  5. [§3.1, CoT-Step 2, Eq. (2), Appendix C Template2] The semantic descriptions are generated from text and 16 video frames only; the prompts ask about actions, facial expressions, and interactions, and never solicit audio/prosody information (tone, pitch, acoustic cues). Both datasets provide audio, and all baselines use it. The method is thus not using the full multimodal signal, and the 'multimodal' claim is weakened. Please either incorporate audio-derived descriptions (e.g., ask VideoLLaMA2 to describe tone and vocal delivery) or explicitly scope the method to vision+text and discuss how the comparison to audio-aware baselines is affected.
  6. [§3.2, Eqs. (9)–(15); Contribution (2)] The paper says it 'formally establishes' three logic-driven relations and maps 'or', 'and', 'not' to weighted importance, cosine similarity, and MSE. These mappings are heuristic analogies, not a formal logical semantics. The operations are standard feature interaction mechanisms (weighted sum, cosine similarity, difference). If the claim of a 'formal' logical grounding is to be retained, a more rigorous derivation is needed; otherwise the text should describe these as 'logic-inspired' design choices, not formal models.
minor comments (7)
  1. [References] Xu et al. 2024a and 2024b are the same paper (same title and venue) listed twice. Please merge or correct.
  2. [Appendix B] The number of background samples B used in CoT-Step 1 is not specified. Please state n and the sampling procedure for reproducibility.
  3. [§4.2] The paper reports improvements in percentage points (e.g., F1 '0.61%') but calls them 'substantial gains'. Please specify that these are absolute percentage-point differences, not relative improvements, and avoid 'substantial' without statistical support.
  4. [Figure 2] The confusion matrices are extremely small in the manuscript and hard to read. Please enlarge them or provide per-class accuracies in a table.
  5. [Appendix D] Typo: 'algin' should be 'align'.
  6. [§4.1] The paper does not state whether baseline numbers in Table 1 are re-run under the same protocol or taken from prior papers. This matters for fairness. Please clarify in the implementation details.
  7. [Section 6] The Limitations section is generic and does not mention the absence of audio information, the label-dependent ranking supervision, or the lack of significance testing. These are the most important limitations and should be acknowledged.

Circularity Check

0 steps flagged

No significant circularity: LGSRR is an empirical pipeline with training-time auxiliary supervision, but no prediction reduces to an input by construction.

full rationale

The paper's central claim is an empirical one: a model architecture with LLM-generated semantic descriptions and a relational reasoning module beats baselines on two benchmarks. There is no formal derivation chain whose conclusion is equivalent to its premises. The nearest potential concern is CoT-Step 3 (Eq. 3), where GPT-3.5 generates semantic rankings using the ground-truth label y during training. This is not circular: the ranking is not a test-time prediction that is evaluated against the label; it is an auxiliary training target used by the NeuralNDCG ranking loss (Eq. 9), while the final classification still uses cross-entropy on the label (Eq. 17). Using labels to create auxiliary supervision is a methodological choice, not a self-definitional reduction. The fixed text top-rank in the ranking loss and the selection of K=3 semantic aspects (Appendix B) are design choices; even if K were tuned on the test sets, that is benchmark overfitting, not circularity, because the reported metrics are not equal to the choice by construction. The 'or/and/not' relations are implemented with standard operations (weighted sum, cosine similarity, MSE), not derived from the target labels. Self-citations to MIntRec, MIntRec2.0, MIntOOD, and TCL-MAP are dataset/baseline references and do not carry the load of a uniqueness theorem or forced ansatz. The abstract's 'consistent performance gains' is overstated given the lower ACC on MIntRec2.0, and the lack of significance testing is a robustness concern, but neither constitutes circularity. Overall, the method is self-contained as an empirical contribution, so the circularity score is 0.

Axiom & Free-Parameter Ledger

4 free parameters · 5 axioms · 0 invented entities

The paper introduces no new physical or ontological entities. The 'semantic relations' are names for existing vector operations, and the 'fine-grained semantics' are natural-language descriptions, not new theoretical constructs.

free parameters (4)
  • number of semantic aspects K = 3
    Top-K frequent cues selected from GPT-3.5 output; K=3 is chosen and validated in Appendix B on the same benchmarks, making it a fitted design choice.
  • semantic aspect set S = {A,E,I} = Speakers' Actions, Facial Expressions, Interactions with Others
    The specific set of aspects is determined by frequency in a background subset and then confirmed by performance comparisons; the choice is data-dependent.
  • ranking loss weight lambda = not reported
    Appears in Eq. 18 as overall objective L = L_cls + lambda*L_rank; no value or tuning procedure is given, leaving a free hyperparameter that affects results.
  • text top-rank prior = text always rank 1
    Section 3.2 explicitly sets FT as top rank in R; this is a hand-chosen prior, not learned.
axioms (5)
  • domain assumption BERT representations of text and concatenated semantic descriptions are sufficient input features for intent classification.
    Used in Eq. 4-6; the classifier is a single linear layer on BERT pooled features.
  • domain assumption GPT-3.5 and VideoLLaMA2 produce accurate, unbiased semantic descriptions and importance rankings.
    Used in CoT steps 1-3, Eqs. 1-3; degraded outputs would propagate noise into training and inference.
  • domain assumption Ground-truth-label-conditioned LLM rankings provide a valid supervision signal for the weight network.
    CoT-Step 3, Eq. 3 uses y during training; this presumes the label-based ranking generalizes beyond the training set.
  • ad hoc to paper The logical operators 'or', 'and', 'not' are correctly mapped to weighted importance, cosine similarity, and MSE respectively.
    Section 3.2 states this mapping without derivation; it is an analogy rather than a formal result.
  • standard math NeuralNDCG (Pobrotyn & Bialobrzeski, 2021) loss is a valid differentiable ranking loss.
    Used in Eq. 9, the loss is imported from prior work and assumed correct.

reviewed 2026-08-05 · how reviews work

0 comments
Cite this review

Pith. "Pith review of LLM-Guided Semantic Relational Reasoning for Multimodal Intent Recognition." pith.science (2026). https://pith.science/paper/RMMIPMG5

@misc{pith2026250901337,
  author       = {Pith},
  title        = {Pith review of: LLM-Guided Semantic Relational Reasoning for Multimodal Intent Recognition},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/RMMIPMG5}},
  note         = {Machine review of arXiv:2509.01337}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Understanding human intents from multimodal signals is critical for analyzing human behaviors and enhancing human-machine interactions in real-world scenarios. However, existing methods exhibit limitations in their modality-level reliance, constraining relational reasoning over fine-grained semantics for complex intent understanding. This paper proposes a novel LLM-Guided Semantic Relational Reasoning (LGSRR) method, which harnesses the expansive knowledge of large language models (LLMs) to establish semantic foundations that boost smaller models' relational reasoning performance. Specifically, an LLM-based strategy is proposed to extract fine-grained semantics as guidance for subsequent reasoning, driven by a shallow-to-deep Chain-of-Thought (CoT) that autonomously uncovers, describes, and ranks semantic cues by their importance without relying on manually defined priors. Besides, we formally model three fundamental types of semantic relations grounded in logical principles and analyze their nuanced interplay to enable more effective relational reasoning. Extensive experiments on multimodal intent and dialogue act recognition tasks demonstrate LGSRR's superiority over state-of-the-art methods, with consistent performance gains across diverse semantic understanding scenarios. The complete data and code are available at https://github.com/thuiar/LGSRR.

Figures

Figures reproduced from arXiv: 2509.01337 by Hanlei Zhang, Hua Xu, Qianrui Zhou, Xinzhi Dong, Yifan Wang.

Figure 1
Figure 1. Figure 1: Overall architecture of our proposed LLM-guided semantic relational reasoning (LGSRR) method. [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Confusion matrices on both datasets. Moreover, the performance gains of LGSRR on P and R metrics, ranging from 2.37% to 17.36% and 3.83% to 5.99%, underscore the superiority of LGSRR in capturing deep semantic associations. Meanwhile, the stable performance on WF1 and WP metrics confirms the robustness of LGSRR when faced with imbalanced datasets. Overall, our method not only delivers more competitive perf… view at source ↗
Figure 3
Figure 3. Figure 3: Examples from the MIntRec2.0 dataset, showcasing descriptions and rankings of fine-grained semantics. [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Samples from the MIntRec2.0, showcasing descriptions and ranking results of fine-grained semantics. [PITH_FULL_IMAGE:figures/full_fig_p015_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Samples from the IEMOCAP-DA, showcasing descriptions and ranking results of fine-grained semantics. [PITH_FULL_IMAGE:figures/full_fig_p016_5.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

55 extracted references · 21 canonical work pages

  1. [1]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, Roman Ring, Eliza Rutherford, Serkan Cabi, Tengda Han, Zhitao Gong, Sina Samangooei, Marianne Monteiro, Jacob L Menick, Sebastian Borgeaud, and 8 others. 2022. https://proceedings.neurips.cc/paper_files/...

    work page 2022

  4. [4]

    Kirolos Ataallah, Xiaoqian Shen, Eslam Abdelrahman, Essam Sleiman, Deyao Zhu, Jian Ding, and Mohamed Elhoseiny. 2024. https://arxiv.org/abs/2404.03413 Minigpt4-video: Advancing multimodal llms for video understanding with interleaved visual-textual tokens . Preprint, arXiv:2404.03413

    Pith/arXiv arXiv 2024

  5. [5]

    Remi Cadene, Hedi Ben-younes, Matthieu Cord, and Nicolas Thome. 2019. Murel: Multimodal relational reasoning for visual question answering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

    work page 2019

  6. [6]

    Sanyuan Chen, Chengyi Wang, Zhengyang Chen, Yu Wu, Shujie Liu, Zhuo Chen, Jinyu Li, Naoyuki Kanda, Takuya Yoshioka, Xiong Xiao, Jian Wu, Long Zhou, Shuo Ren, Yanmin Qian, Yao Qian, Jian Wu, Michael Zeng, Xiangzhan Yu, and Furu Wei. 2022. https://doi.org/10.1109/JSTSP.2022.3188113 Wavlm: Large-scale self-supervised pre-training for full stack speech proces...

    arXiv 2022

  7. [7]

    Yunpeng Chen, Marcus Rohrbach, Zhicheng Yan, Yan Shuicheng, Jiashi Feng, and Yannis Kalantidis. 2019. Graph-based global reasoning networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)

    work page 2019

  8. [8]

    Zesen Cheng, Sicong Leng, Hang Zhang, Yifei Xin, Xin Li, Guanzheng Chen, Yongxin Zhu, Wenqi Zhang, Ziyang Luo, Deli Zhao, and Lidong Bing. 2024. https://arxiv.org/abs/2406.07476 Videollama 2: Advancing spatial-temporal modeling and audio understanding in video-llms . Preprint, arXiv:2406.07476

    Pith/arXiv arXiv 2024

  9. [9]

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. https://doi.org/10.18653/v1/N19-1423 BERT : Pre-training of deep bidirectional transformers for language understanding . In Proceedings of the 2019 Conference of the North A merican Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long a...

    doi:10.18653/v1/n19-1423 2019

  10. [10]

    Yan Fan, Chengyu Wang, Peng He, and Yunhua Hu. 2022. https://doi.org/10.1145/3488560.3502189 Building multi-turn query interpreters for e-commercial chatbots with sparse-to-dense attentive modeling . In Proceedings of the Fifteenth ACM International Conference on Web Search and Data Mining, WSDM '22, page 1577–1580, New York, NY, USA. Association for Comp...

    work page arXiv 2022

  11. [11]

    Roy Ganz, Yair Kittenplon, Aviad Aberdam, Elad Ben Avraham, Oren Nuriel, Shai Mazor, and Ron Litman. 2024. Question aware vision transformer for multimodal reasoning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 13861--13871

    work page 2024

  12. [12]

    Aditya Grover, Eric Wang, Aaron Zweig, and Stefano Ermon. 2019. https://openreview.net/forum?id=H1eSS3CcKX Stochastic optimization of sorting networks via continuous relaxations . In International Conference on Learning Representations

    work page 2019

  13. [13]

    Devamanyu Hazarika, Roger Zimmermann, and Soujanya Poria. 2020. https://doi.org/10.1145/3394171.3413678 Misa: Modality-invariant and -specific representations for multimodal sentiment analysis . In Proceedings of the 28th ACM International Conference on Multimedia, MM '20, page 1122–1131, New York, NY, USA. Association for Computing Machinery

    work page arXiv 2020

  14. [14]

    Shijue Huang, Libo Qin, Bingbing Wang, Geng Tu, and Ruifeng Xu. 2024. https://doi.org/10.1109/ICASSP48485.2024.10446922 Sdif-da: A shallow-to-deep interaction framework with data augmentation for multi-modal intent detection . In ICASSP 2024 - 2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 10206--10210

    arXiv 2024

  15. [15]

    a rvelin and Jaana Kek \

    Kalervo J \"a rvelin and Jaana Kek \"a l \"a inen. 2002. Cumulated gain-based evaluation of ir techniques. ACM Transactions on Information Systems (TOIS), 20(4):422--446

    2002

  16. [16]

    Ding Jiang and Mang Ye. 2023. Cross-modal implicit relation reasoning and aligning for text-to-image person retrieval. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 2787--2797

    work page 2023

  17. [17]

    Sepideh Kaffash, An Truong Nguyen, and Joe Zhu. 2021. https://doi.org/10.1016/j.ijpe.2020.107868 Big data algorithms and applications in intelligent transportation system: A review and bibliometric analysis . International Journal of Production Economics, 231:107868

    arXiv 2021

  18. [18]

    Julia Kruk, Jonah Lubin, Karan Sikka, Xiao Lin, Dan Jurafsky, and Ajay Divakaran. 2019. https://doi.org/10.18653/v1/D19-1469 Integrating text and image: Determining multimodal document intent in I nstagram posts . In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural...

    work page doi:10.18653/v1/d19-1469 2019

  19. [19]

    Xin Lai, Zhuotao Tian, Yukang Chen, Yanwei Li, Yuhui Yuan, Shu Liu, and Jiaya Jia. 2024. Lisa: Reasoning segmentation via large language model. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 9579--9589

    work page 2024

  20. [20]

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. 2023. https://proceedings.mlr.press/v202/li23q.html BLIP -2: Bootstrapping language-image pre-training with frozen image encoders and large language models . In Proceedings of the 40th International Conference on Machine Learning, volume 202 of Proceedings of Machine Learning Research, pages 19730--19...

    2023

  21. [21]

    KunChang Li, Yinan He, Yi Wang, Yizhuo Li, Wenhai Wang, Ping Luo, Yali Wang, Limin Wang, and Yu Qiao. 2024. https://arxiv.org/abs/2305.06355 Videochat: Chat-centric video understanding . Preprint, arXiv:2305.06355

    Pith/arXiv arXiv 2024

  22. [22]

    Yabo Liu, Jinghua Wang, Chao Huang, Yaowei Wang, and Yong Xu. 2023. Cigar: Cross-modality graph reasoning for domain adaptive object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 23776--23786

    work page 2023

  23. [23]

    Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. 2021. https://doi.org/10.1109/ICCV48922.2021.00986 Swin transformer: Hierarchical vision transformer using shifted windows . In 2021 IEEE/CVF International Conference on Computer Vision (ICCV), pages 9992--10002

    arXiv 2021

  24. [24]

    Zhun Liu, Ying Shen, Varun Bharadhwaj Lakshminarasimhan, Paul Pu Liang, Amir Zadeh, and Louis-Philippe Morency. 2018. Efficient low-rank multimodal fusion with modality-specific factors. arXiv preprint arXiv:1806.00064

    Pith/arXiv arXiv 2018

  25. [25]

    Ilya Loshchilov and Frank Hutter. 2019. https://openreview.net/forum?id=Bkg6RiCqY7 Decoupled weight decay regularization . In International Conference on Learning Representations

    2019

  26. [26]

    Jinpeng Mi, Song Tang, Zhen Deng, Michael Görner, and Jianwei Zhang. 2019. https://doi.org/10.1016/j.cogsys.2018.12.010 Object affordance based multimodal fusion for natural human-robot interaction . Cogn. Syst. Res., 54:128--137

    work page doi:10.1016/j.cogsys.2018.12.010 2019

  27. [27]

    Jong Hak Moon, Hyungyung Lee, Woncheol Shin, Young-Hak Kim, and Edward Choi. 2022. https://doi.org/10.1109/JBHI.2022.3207502 Multi-modal understanding and generation for medical images and text via vision-language pre-training . IEEE Journal of Biomedical and Health Informatics, 26(12):6070--6080

    work page arXiv 2022

  28. [28]

    Hyeonseob Nam, Jung-Woo Ha, and Jeonghee Kim. 2017. Dual attention networks for multimodal reasoning and matching. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)

    work page 2017

  29. [29]

    Sheuli Paul, Michael Sintek, Veton Këpuska, Marius Silaghi, and Liam Robertson. 2022. https://doi.org/10.1109/ICMLA55696.2022.00127 Intent based multimodal speech and gesture fusion for human-robot communication in assembly situation . In 2022 21st IEEE International Conference on Machine Learning and Applications (ICMLA), pages 760--763

    arXiv 2022

  30. [30]

    Przemysław Pobrotyn and Radosław Białobrzeski. 2021. https://arxiv.org/abs/2102.07831 Neuralndcg: Direct optimisation of a ranking metric via differentiable relaxation of sorting . Preprint, arXiv:2102.07831

    Pith/arXiv arXiv 2021

  31. [31]

    Wasifur Rahman, Md Kamrul Hasan, Sangwu Lee, AmirAli Bagher Zadeh, Chengfeng Mao, Louis-Philippe Morency, and Ehsan Hoque. 2020. https://doi.org/10.18653/v1/2020.acl-main.214 Integrating multimodal information in large pretrained transformers . In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2359--2369, On...

    doi:10.18653/v1/2020.acl-main.214 2020

  32. [32]

    Tulika Saha, Aditya Patra, Sriparna Saha, and Pushpak Bhattacharyya. 2020. https://doi.org/10.18653/v1/2020.acl-main.402 Towards emotion-aided multi-modal dialogue act classification . In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4361--4372, Online. Association for Computational Linguistics

    doi:10.18653/v1/2020.acl-main.402 2020

  33. [33]

    Yongliang Shen, Kaitao Song, Xu Tan, Dongsheng Li, Weiming Lu, and Yueting Zhuang. 2023. https://proceedings.neurips.cc/paper_files/paper/2023/file/77c33e6a367922d003ff102ffb92b658-Paper-Conference.pdf Hugginggpt: Solving ai tasks with chatgpt and its friends in hugging face . In Advances in Neural Information Processing Systems, volume 36, pages 38154--3...

    work page 2023

  34. [34]

    Kaili Sun, Zhiwen Xie, Mang Ye, and Huyin Zhang. 2024. Contextual augmented global contrast for multimodal intent recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 26963--26973

    work page 2024

  35. [35]

    Elarabawy, Mohammed Abd-Elnaby, Noor Mohd, Gaurav Dhiman, and Subhash Sharma

    Pallavi Tiwari, Bhaskar Pant, Mahmoud M. Elarabawy, Mohammed Abd-Elnaby, Noor Mohd, Gaurav Dhiman, and Subhash Sharma. 2022. https://doi.org/10.1155/2022/1830010 Cnn based multiclass brain tumor detection using medical imaging . Computational Intelligence and Neuroscience, 2022(1):1830010

    doi:10.1155/2022/1830010 2022

  36. [36]

    Zico Kolter, Louis-Philippe Morency, and Ruslan Salakhutdinov

    Yao-Hung Hubert Tsai, Shaojie Bai, Paul Pu Liang, J. Zico Kolter, Louis-Philippe Morency, and Ruslan Salakhutdinov. 2019. https://doi.org/10.18653/v1/P19-1656 Multimodal transformer for unaligned multimodal language sequences . In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 6558--6569, Florence, Italy. As...

    doi:10.18653/v1/p19-1656 2019

  37. [37]

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, ukasz Kaiser, and Illia Polosukhin. 2017. https://proceedings.neurips.cc/paper_files/paper/2017/file/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf Attention is all you need . In Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc

    2017

  38. [38]

    Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Yang Fan, Kai Dang, Mengfei Du, Xuancheng Ren, Rui Men, Dayiheng Liu, Chang Zhou, Jingren Zhou, and Junyang Lin. 2024 a . Qwen2-vl: Enhancing vision-language model's perception of the world at any resolution. arXiv preprint arXiv:2409.12191

    Pith/arXiv arXiv 2024

  39. [39]

    Xiaohan Wang, Yuhui Zhang, Orr Zohar, and Serena Yeung-Levy. 2024 b . Videoagent: Long-form video understanding with large language model as agent. European Conference on Computer Vision (ECCV)

    work page 2024

  40. [40]

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed Chi, Quoc V Le, and Denny Zhou. 2022. https://proceedings.neurips.cc/paper_files/paper/2022/file/9d5609613524ecf4f15af0f7b31abca4-Paper-Conference.pdf Chain-of-thought prompting elicits reasoning in large language models . In Advances in Neural Information Processing Systems...

    2022

  41. [41]

    Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, and 3 others. 2020. https://doi.org/10.18653/v1/2020.emnlp-demos.6 Transformers: Sta...

    doi:10.18653/v1/2020.emnlp-demos.6 2020

  42. [42]

    Wei Xu. 2019. https://doi.org/10.1145/3328485 Toward human-centered ai: a perspective from human-computer interaction . Interactions, 26(4):42–46

    doi:10.1145/3328485 2019

  43. [44]

    Yanzhi Xu, Yueying Hua, Shichen Li, and Zhongqing Wang. 2024 b . https://doi.org/10.18653/v1/2024.acl-long.6 Exploring chain-of-thought for multi-modal metaphor detection . In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 91--101, Bangkok, Thailand. Association for Computational Linguistics

    doi:10.18653/v1/2024.acl-long.6 2024

  44. [45]

    Zhengyuan Yang, Linjie Li, Jianfeng Wang, Kevin Lin, Ehsan Azarnasab, Faisal Ahmed, Zicheng Liu, Ce Liu, Michael Zeng, and Lijuan Wang. 2023. https://arxiv.org/abs/2303.11381 Mm-react: Prompting chatgpt for multimodal reasoning and action . Preprint, arXiv:2303.11381

    Pith/arXiv arXiv 2023

  45. [46]

    Yuan Yao, Tianyu Yu, Ao Zhang, Chongyi Wang, Junbo Cui, Hongji Zhu, Tianchi Cai, Haoyu Li, Weilin Zhao, Zhihui He, and 1 others. 2024. Minicpm-v: A gpt-4v level mllm on your phone. arXiv preprint arXiv:2408.01800

    Pith/arXiv arXiv 2024

  46. [47]

    Amir Zadeh, Minghai Chen, Soujanya Poria, Erik Cambria, and Louis-Philippe Morency. 2017. Tensor fusion network for multimodal sentiment analysis. arXiv preprint arXiv:1707.07250

    Pith/arXiv arXiv 2017

  47. [48]

    Amir Zadeh, Paul Pu Liang, Navonil Mazumder, Soujanya Poria, Erik Cambria, and Louis-Philippe Morency. 2018. Memory fusion network for multi-view sequential learning. In Proc. AAAI Conf. Artif. Intell

    work page 2018

  48. [49]

    Hang Zhang, Xin Li, and Lidong Bing. 2023. https://doi.org/10.18653/v1/2023.emnlp-demo.49 Video- LL a MA : An instruction-tuned audio-visual language model for video understanding . In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 543--553, Singapore. Association for Computational Linguistics

    doi:10.18653/v1/2023.emnlp-demo.49 2023

  49. [50]

    Hanlei Zhang, Xin Wang, Hua Xu, Qianrui Zhou, Kai Gao, Jianhua Su, jinyue Zhao, Wenrui Li, and Yanting Chen. 2024 a . https://openreview.net/forum?id=nY9nITZQjc MI ntrec2.0: A large-scale benchmark dataset for multimodal intent recognition and out-of-scope detection in conversations . In The Twelfth International Conference on Learning Representations

    work page 2024

  50. [51]

    Hanlei Zhang, Hua Xu, Xin Wang, Qianrui Zhou, Shaojie Zhao, and Jiayan Teng. 2022. https://doi.org/10.1145/3503161.3547906 Mintrec: A new dataset for multimodal intent recognition . In Proceedings of the 30th ACM International Conference on Multimedia, MM ’22, page 1688–1697. ACM

    arXiv 2022

  51. [52]

    Hanlei Zhang, Qianrui Zhou, Hua Xu, Jianhua Su, Roberto Evans, and Kai Gao. 2024 b . Multimodal classification and out-of-distribution detection for multimodal intent understanding. arXiv preprint arXiv:2412.12453

    Pith/arXiv arXiv 2024

  52. [53]

    Yuanhan Zhang, Bo Li, haotian Liu, Yong jae Lee, Liangke Gui, Di Fu, Jiashi Feng, Ziwei Liu, and Chunyuan Li. 2024 c . https://llava-vl.github.io/blog/2024-04-30-llava-next-video/ Llava-next: A strong zero-shot video understanding model

    work page 2024

  53. [54]

    Yu, and Lichao Sun

    Ce Zhou, Qian Li, Chen Li, Jun Yu, Yixin Liu, Guangjing Wang, Kai Zhang, Cheng Ji, Qiben Yan, Lifang He, Hao Peng, Jianxin Li, Jia Wu, Ziwei Liu, Pengtao Xie, Caiming Xiong, Jian Pei, Philip S. Yu, and Lichao Sun. 2023. https://arxiv.org/abs/2302.09419 A comprehensive survey on pretrained foundation models: A history from bert to chatgpt . Preprint, arXiv...

    Pith/arXiv arXiv 2023

  54. [55]

    Qianrui Zhou, Hua Xu, Hao Li, Hanlei Zhang, Xiaohan Zhang, Yifan Wang, and Kai Gao. 2024. https://doi.org/10.1609/aaai.v38i15.29656 Token-level contrastive learning with modality-aware prompting for multimodal intent recognition . Proceedings of the AAAI Conference on Artificial Intelligence, 38(15):17114--17122

    work page doi:10.1609/aaai.v38i15.29656 2024

  55. [56]

    Zhihong Zhu, Xuxin Cheng, Zhaorun Chen, Yuyan Chen, Yunyan Zhang, Xian Wu, Yefeng Zheng, and Bowen Xing. 2024. https://doi.org/10.1145/3664647.3681623 Inmu-net: Advancing multi-modal intent detection via information bottleneck and multi-sensory processing . In Proceedings of the 32nd ACM International Conference on Multimedia, MM '24, page 515–524, New Yo...

    arXiv 2024

This paper was first reviewed by deepseek-v4-flash on August 5, 2026.