Pith. sign in

REVIEW 4 major objections 5 minor 28 references

This paper claims pre-trained vision-language large models can recognize video actions when the video is first compressed into discrete semantic tokens—an 'action narrative'—and that this token-based pipeline beats prior methods on NTU RGB+

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-05 05:09 UTC pith:TF7M26FL

load-bearing objection Interesting idea, but the core VST module has no specified training objective and the token counts contradict each other; as is, the SOTA claims don't stand. the 4 major comments →

arxiv 2509.05695 v1 pith:TF7M26FL submitted 2025-09-06 cs.CV

Leveraging Vision-Language Large Models for Interpretable Video Action Recognition with Semantic Tokenization

Jingwei Peng , Zhixuan Qiu , Boyu Jin , Surasakdi Siripong This is my paper
classification cs.CV
keywords video action recognitionvision-language large modelssemantic tokenizationLoRA fine-tuninginterpretable AIaction narrativeNTU RGB+DVST module
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper introduces LVLM-VAR, a framework that claims to be the first to apply pre-trained vision-language large models (LVLMs) to video action recognition. Instead of feeding raw frames or hand-crafted features to the classifier, a Video-to-Semantic-Tokens (VST) module compresses a video into a short sequence of discrete 'semantic action tokens'; these are concatenated with a text instruction and passed to a LoRA-tuned LVLM such as LLaVA-13B. The authors report top accuracy on NTU RGB+D (94.1% X-Sub, 96.8% X-View) and NTU RGB+D 120 (90.0% X-Set), with competitive results on Toyota Smarthome and UAV-Human, while the model also generates natural-language explanations. A sympathetic reader would care because the paper proposes a concrete bridge between continuous video and discrete language-model reasoning, replacing pixel-level recognition with an interpretable action script.

Core claim

The central claim is that a pre-trained vision-language large model can perform competitive action recognition if the video is first rewritten as a compact sequence of discrete tokens that carry action semantics and temporal order. The VST module extracts spatio-temporal features with a Swin Transformer V2 pretrained on Kinetics-400, applies temporal self-attention, then projects and quantizes the attended features into K semantic tokens whose dimension matches the LVLM embedding. The tokens, preceded by a natural-language instruction, are processed by an LVLM fine-tuned with LoRA; the output is both an action class and an explanation. The paper reports that removing VST drops accuracy by ab

What carries the argument

The Video-to-Semantic-Tokens (VST) module is the load-bearing mechanism. It maps raw frames to K discrete 'semantic action tokens' through a pretrained visual encoder, a temporal self-attention layer, a projection, and a quantization step. These tokens are designed to behave like a written action narrative—capturing subject, predicate, object, and temporal progression—so a language model can 'read' the video. LoRA (low-rank adaptation) then adapts the frozen LVLM with a small number of trainable parameters, preserving general knowledge while steering it toward action classification and explanation generation.

Load-bearing premise

The whole result rests on the VST module being trainable to compress a video into a small set of discrete tokens that keep action-relevant meaning and temporal order, yet the paper specifies no loss or training procedure for VST and reports inconsistent token counts (512 vs 15.3 per video).

What would settle it

Reproduce the VST training as described: if the quantization step is non-differentiable and no auxiliary loss is defined, gradients cannot train the visual encoder, so the claimed 94.1% cannot arise from the stated pipeline—inspect the checkpoint's actual tokens per video to see whether it is 512 or ~15.3, since both numbers are asserted.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

Share X Bluesky LinkedIn Reddit HN

If this is right

  • Video action recognition can be reframed as a language-modeling task over a learned token vocabulary, not a visual classification task.
  • Accuracy and interpretability do not have to trade off: the same forward pass yields a class label and a rationale.
  • Adapting a 13B-parameter LVLM costs only about 0.1% trainable parameters and roughly 1.5 hours per epoch on one A100, making large-model video understanding practical.
  • The learned tokens form a compact action lexicon (~487 of 512 possible tokens used), suggesting actions can be decomposed into reusable semantic primitives.
  • The method generalizes beyond RGB to skeleton-derived inputs and drone footage, implying the token narrative is modality-agnostic.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The paper never states a training objective for VST; a natural extension is to test whether the module needs an explicit reconstruction, contrastive, or classification loss, or whether downstream LoRA fine-tuning alone shapes the tokens.
  • If the action-narrative idea is correct, token sequences could be probed as a compositional grammar—e.g., whether swapping or reordering tokens changes predicted actions the way reordering words changes a sentence.
  • The reported token counts are inconsistent (512 per video in Section 4.1.1, 15.3 average in Figure 1, K varied 256–1024 in Section 4.11); until resolved, efficiency and compression claims should be read cautiously.
  • A control experiment replacing the LVLM with a small non-language classifier on the same tokens would isolate whether the LVLM's reasoning or the VST tokenizer carries the accuracy gain.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes LVLM-VAR, a framework that converts raw video into discrete 'semantic action tokens' via a Video-to-Semantic-Tokens (VST) module, then feeds these tokens together with natural-language instructions into a LoRA-fine-tuned LVLM (e.g., LLaVA-13B) for action classification and explanatory text generation. The authors report state-of-the-art or competitive accuracy on NTU RGB+D, NTU RGB+D 120, Toyota Smarthome, and UAV-Human, and claim interpretability benefits supported by a human evaluation. The VST module is the only novel component, and the paper's central claim is that semantic tokenization enables both accuracy and interpretability.

Significance. If fully substantiated, this would be a meaningful step: using pre-trained LVLMs for video action recognition through learned semantic tokens is a plausible and timely direction, and the reported efficiency gains from LoRA are practically relevant. The paper also promises interpretability via generated explanations, which is valuable. However, the central novel mechanism—the VST module—is not specified at the level required to reproduce or even understand the method. The training objective for VST is never given, and the reported token counts are internally inconsistent. Because the ablations attribute a large accuracy gain to VST, the claimed SOTA results cannot currently be attributed to semantic tokenization. The human evaluation of interpretability is also circular and does not independently validate the semantic quality of the tokens.

major comments (4)
  1. [§3.1, Eq. (2)] The VST module's training objective is never specified. The text describes a projection P and quantization Q, but no loss function, codebook-learning rule, or alignment objective is given. Without an explicit objective, gradient descent cannot shape the quantizer or the token vocabulary, and the claim that VST is 'trained to encode' videos into semantic tokens is operationally undefined. Since Table 2 attributes a 5.6-point gain to the VST module, this missing objective is load-bearing and makes the reported results unreproducible.
  2. [§4.1.1, Figure 1, §4.11] There is a direct internal contradiction in the meaning of 'semantic tokens'. §4.1.1 says the module is 'designed to generate 512 semantic tokens' per video. Figure 1 reports 'Average Tokens per Video = 15.30' and 'Unique Tokens Utilized (out of 512) = 487', implying 512 is a vocabulary size and the per-video sequence length is about 15 tokens. Table 6 then treats K as the number of semantic tokens and varies it from 256 to 1024. These are incompatible definitions of K, and the paper never clarifies whether K is a sequence length, a vocabulary size, or both. The reported accuracy numbers cannot be interpreted or reproduced under this ambiguity.
  3. [§3.2.1, §4.3] The interface between learned token vectors and the LVLM is unspecified. The paper states each token's vector dimension matches the LVLM embedding dimension, but it does not explain how real-valued token vectors are injected into LLaVA's tokenizer/embedding space, whether they replace text embeddings, how positional information is handled, or how the LVLM distinguishes these tokens from language tokens. Without this mechanism, the architecture cannot be instantiated, and the claimed end-to-end fine-tuning via LoRA is not defined.
  4. [§4.6, §4.7] The interpretability evaluation is circular. In §4.6, human raters evaluate explanations generated by the same model that produced the predicted labels, with no blind comparison to explanation from an independent system or to human-written rationales. The 'Semantic Consistency Score' in §4.7 is also human-rated on tokens generated by the trained VST module. These ratings are descriptive statistics of the model's own outputs, not evidence that the tokens carry the semantic roles claimed (subject, predicate, object, temporal attributes). An independent benchmark or controlled comparison with a non-tokenized baseline is needed to support the interpretability claim.
minor comments (5)
  1. [Table 1] The table appears truncated in the text: the row for 'Ours (LVLM-VAR)' shows only X-Sub and X-View values, while the following sentence is cut off. Please report all four protocol columns consistently.
  2. [References] Several citations are imprecise: reference [10] is cited for LLaVA-13B and MiniGPT-4, but the listed paper is about LLaVA-generated prompts for image generation, not the LLaVA or MiniGPT-4 foundation papers. Reference [12] is cited for Swin Transformer V2 but points to a medical segmentation paper. Please correct these citations.
  3. [§4.2, Toyota Smarthome] The text says for Toyota Smarthome, skeleton data are 'processed to extract video features', but no specifics are given. Since the dataset is skeleton-based and the method is described as video-centric, clarify how skeleton sequences are converted to visual features and whether this matches the training protocol of NTU RGB+D.
  4. [Figure 1] The bar plot's x-axis labels are ambiguous: 'Value' is used as a category label for all metrics, but the metrics have different units (counts, entropy, Likert score). Consider using separate panels or a table with explicit units.
  5. [Table 4] The 'Full LVLM Fine-tuning' column is explicitly hypothetical, so the efficiency comparison is not a measured baseline. Please state this limitation in the text or provide measured numbers for a real full-fine-tuning run.

Circularity Check

0 steps flagged

No significant circularity; the central accuracy claim is empirical, though the VST module's training objective is underspecified.

full rationale

We examined the claimed derivation chain: raw video -> VST semantic tokens -> LoRA-fine-tuned LVLM -> action class and explanation. The accuracy results in Tables 1 and 5 are empirical benchmark measurements, not quantities derived from equations that already contain the answer. The VST output is defined by Eq. (2) as a quantization of a learned projection, but the paper omits the training objective; this is a completeness/reproducibility gap, not a circular reduction. The token-statistics and human-evaluation sections report descriptive measurements of the trained system rather than fitted parameters renamed as predictions. No self-citation is load-bearing: the cited prior work (Swin Transformer V2, LLaVA, LoRA, etc.) is external and not authored by the present authors. The internal inconsistency between the stated 512 tokens, Figure 1's 15.3 average tokens per video, and the K=256-1024 sensitivity sweep undermines clarity and reproducibility but does not constitute equation-level circularity. Accordingly, no specific circular step can be quoted and exhibited under the required standard.

Axiom & Free-Parameter Ledger

7 free parameters · 4 axioms · 1 invented entities

The framework rests on several unverified assumptions about tokenization and LVLM reasoning, and the key hyperparameters are chosen by hand without a principled basis. The internal inconsistency in token counts further undermines trust in the reported configuration.

free parameters (7)
  • Number of semantic tokens K = 512
    Chosen by hand; Section 4.11 varies K from 256 to 1024, and Figure 1 reports an inconsistent average of 15.3 tokens per video.
  • LoRA rank r = 8
    Set by hand; sensitivity tested with r=4,16,32.
  • LoRA scaling alpha = 16
    Set by hand.
  • VST learning rate = 2e-4
    AdamW initial learning rate, chosen by hand.
  • LoRA learning rate = 3e-3
    AdamW initial learning rate, chosen by hand.
  • Visual backbone = Swin Transformer V2 (Kinetics-400 pretrained)
    Choice of backbone; ablation with ResNet-50 shows a 2.1% drop, so the choice directly affects the reported accuracy.
  • LVLM choice = LLaVA-13B
    Choice of LVLM; comparison with MiniGPT-4 shows small differences, indicating sensitivity to this hyperparameter.
axioms (4)
  • domain assumption A pre-trained LVLM (LLaVA-13B) can classify actions from a short sequence of discrete tokens with LoRA fine-tuning.
    This is the fundamental premise; the paper does not prove it, only demonstrates empirically (if results are trusted).
  • domain assumption Swin Transformer V2 pre-trained on Kinetics-400 provides spatio-temporal features sufficient for action recognition.
    Used as visual encoder; no evidence is given for this specific choice beyond an ablation against ResNet-50.
  • ad hoc to paper Semantic tokens can be learned to align with the LVLM embedding space and carry semantic roles.
    The VST module's semantic embedding layer is asserted to produce tokens with subject/predicate/object structure, but no mechanism is specified.
  • ad hoc to paper A differentiable training signal flows from the LVLM classification loss back through the VST module.
    The paper never specifies the loss for VST training; it is implied by the pipeline but not demonstrated.
invented entities (1)
  • Semantic action tokens no independent evidence
    purpose: Discrete representations of video action segments intended to be interpretable and consumable by an LVLM.
    The paper provides no external evidence that these tokens carry semantic roles; only a small human rating of consistency on the same system.

pith-pipeline@v1.3.0-alltime-deepseek · 11597 in / 10316 out tokens · 97873 ms · 2026-08-05T05:09:15.097554+00:00 · methodology

0 comments
Cite this review

Pith. "Pith review of Leveraging Vision-Language Large Models for Interpretable Video Action Recognition with Semantic Tokenization." pith.science (2026). https://pith.science/paper/TF7M26FL

@misc{pith2026250905695,
  author       = {Pith},
  title        = {Pith review of: Leveraging Vision-Language Large Models for Interpretable Video Action Recognition with Semantic Tokenization},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/TF7M26FL}},
  note         = {Machine review of arXiv:2509.05695}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Human action recognition often struggles with deep semantic understanding, complex contextual information, and fine-grained distinction, limitations that traditional methods frequently encounter when dealing with diverse video data. Inspired by the remarkable capabilities of large language models, this paper introduces LVLM-VAR, a novel framework that pioneers the application of pre-trained Vision-Language Large Models (LVLMs) to video action recognition, emphasizing enhanced accuracy and interpretability. Our method features a Video-to-Semantic-Tokens (VST) Module, which innovatively transforms raw video sequences into discrete, semantically and temporally consistent "semantic action tokens," effectively crafting an "action narrative" that is comprehensible to an LVLM. These tokens, combined with natural language instructions, are then processed by a LoRA-fine-tuned LVLM (e.g., LLaVA-13B) for robust action classification and semantic reasoning. LVLM-VAR not only achieves state-of-the-art or highly competitive performance on challenging benchmarks such as NTU RGB+D and NTU RGB+D 120, demonstrating significant improvements (e.g., 94.1% on NTU RGB+D X-Sub and 90.0% on NTU RGB+D 120 X-Set), but also substantially boosts model interpretability by generating natural language explanations for its predictions.

Figures

Figures reproduced from arXiv: 2509.05695 by Boyu Jin, Jingwei Peng, Surasakdi Siripong, Zhixuan Qiu.

Figure 1
Figure 1. Figure 1: Analysis of Semantic Token Properties on NTU RGB+D. [PITH_FULL_IMAGE:figures/full_fig_p009_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: As shown in [PITH_FULL_IMAGE:figures/full_fig_p009_2.png] view at source ↗
Figure 2
Figure 2. Figure 2: Performance comparison of different LVLM backbones on NTU RGB+D (Accuracy %). [PITH_FULL_IMAGE:figures/full_fig_p010_2.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

28 extracted references · 16 canonical work pages · 13 internal anchors

  1. [1]

    Improving Human Action Recognition by Non-action Classification

    Yang Wang and Minh Hoai. Improving human action recognition by non-action classification. arXiv preprint arXiv:1604.06397v2, 2016

    work page internal anchor Pith review Pith/arXiv arXiv 2016

  2. [2]

    Bio-inspired hybrid path planning for efficient and smooth robotic navigation: F

    Fujiang Yuan, Zhihao Lin, Zhen Tian, Boyu Chen, Qifan Zhou, Chunhong Yuan, Hongyu Sun, and Zixuan Huang. Bio-inspired hybrid path planning for efficient and smooth robotic navigation: F. yuan et al.International Journal of Intelligent Robotics and Applications, pages 1–31, 2025

    work page 2025

  3. [3]

    Adaptive field effect planner for safe interactive autonomous driving on curved roads.arXiv preprint arXiv:2504.14747, 2025

    Qinghao Li, Zhen Tian, Xiaodan Wang, Jinming Yang, and Zhihao Lin. Adaptive field effect planner for safe interactive autonomous driving on curved roads.arXiv preprint arXiv:2504.14747, 2025

    Pith/arXiv arXiv 2025

  4. [4]

    Data-driven evolutionary game-based model predictive control for hybrid renewable energy dispatch in autonomous ships

    Yaoze Liu, Zhen Tian, Jinming Yang, and Zhihao Lin. Data-driven evolutionary game-based model predictive control for hybrid renewable energy dispatch in autonomous ships. In2025 4th Interna- 12 tional Conference on New Energy System and Power Engineering (NESP), pages 482–490. IEEE, 2025

    work page 2025

  5. [5]

    Cedille: A large autoregressive french language model.arXiv preprint arXiv:2202.03371v1, 2022

    Martin M ¨uller and Florian Laurent. Cedille: A large autoregressive french language model.arXiv preprint arXiv:2202.03371v1, 2022

    Pith/arXiv arXiv 2022

  6. [6]

    Thread of thought: Unraveling chaotic contexts.arXiv preprint arXiv:2311.08734, 2023

    Yucheng Zhou, Xiubo Geng, Tao Shen, Chongyang Tao, Guodong Long, Jian-Guang Lou, and Jianbing Shen. Thread of thought: Unraveling chaotic contexts.arXiv preprint arXiv:2311.08734, 2023

    Pith/arXiv arXiv 2023

  7. [7]

    Enhancing code llms with reinforcement learning in code generation.arXiv preprint arXiv:2412.20367, 2024

    Junqiao Wang, Zeng Zhang, Yangfan He, Yuyang Song, Tianyu Shi, Yuchen Li, Hengyuan Xu, Kunyu Wu, Guangwu Qian, Qiuwu Chen, et al. Enhancing code llms with reinforcement learning in code generation.arXiv preprint arXiv:2412.20367, 2024

    Pith/arXiv arXiv 2024

  8. [8]

    Score: Story coherence and retrieval enhancement for ai narra- tives.arXiv preprint arXiv:2503.23512, 2025

    Qiang Yi, Yangfan He, Jianhui Wang, Xinyuan Song, Shiyao Qian, Xinhang Yuan, Miao Zhang, Li Sun, Keqin Li, Kuan Lu, et al. Score: Story coherence and retrieval enhancement for ai narra- tives.arXiv preprint arXiv:2503.23512, 2025

    arXiv 2025

  9. [9]

    Visual in-context learning for large vision-language models

    Yucheng Zhou, Xiang Li, Qianning Wang, and Jianbing Shen. Visual in-context learning for large vision-language models. InFindings of the Association for Computational Linguistics, ACL 2024, Bangkok, Thailand and virtual meeting, August 11-16, 2024, pages 15890–15902. Association for Computational Linguistics, 2024

    2024

  10. [10]

    Enhance Image-to-Image Generation with LLaVA-generated Prompts

    Zhicheng Ding, Panfeng Li, Qikai Yang, and Siyang Li. Enhance image-to-image generation with llava-generated prompts.arXiv preprint arXiv:2406.01956v3, 2024

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  11. [11]

    KD-LoRA: A Hybrid Approach to Efficient Fine-Tuning with LoRA and Knowledge Distillation

    Rambod Azimi, Rishav Rishav, Marek Teichmann, and Samira Ebrahimi Kahou. Kd-lora: A hybrid approach to efficient fine-tuning with lora and knowledge distillation.arXiv preprint arXiv:2410.20777v1, 2024

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  12. [12]

    Swin SMT: Global Sequential Modeling in 3D Medical Image Segmentation

    Szymon Płotka, Maciej Chrabaszcz, and Przemyslaw Biecek. Swin smt: Global sequential model- ing in 3d medical image segmentation.arXiv preprint arXiv:2407.07514v1, 2024

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  13. [13]

    Analyzing Zero-Shot Abilities of Vision-Language Models on Video Understanding Tasks

    Avinash Madasu, Anahita Bhiwandiwalla, and Vasudev Lal. Analyzing zero-shot abilities of vision- language models on video understanding tasks.arXiv preprint arXiv:2310.04914v2, 2023

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  14. [14]

    Few-shot Action Recognition via Intra- and Inter-Video Information Maximization

    Huabin Liu, Weiyao Lin, Tieyuan Chen, Yuxi Li, Shuyuan Li, and John See. Few-shot action recog- nition via intra- and inter-video information maximization.arXiv preprint arXiv:2305.06114v1, 2023

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  15. [15]

    Manmatha, and Mu Li

    Yi Zhu, Xinyu Li, Chunhui Liu, Mohammadreza Zolfaghari, Yuanjun Xiong, Chongruo Wu, Zhi Zhang, Joseph Tighe, R. Manmatha, and Mu Li. A comprehensive study of deep video action recognition.arXiv preprint arXiv:2012.06567v1, 2020

    Pith/arXiv arXiv 2012

  16. [16]

    F4D: Factorized 4D Convolutional Neural Network for Efficient Video-level Representation Learning

    Mohammad Al-Saad, Lakshmish Ramaswamy, and Suchendra Bhandarkar. F4d: Factorized 4d convolutional neural network for efficient video-level representation learning.arXiv preprint arXiv:2401.08609v1, 2023

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  17. [17]

    Ziyuan Huang, Zhiwu Qing, Xiang Wang, Yutong Feng, Shiwei Zhang, Jianwen Jiang, Zhurong Xia, Mingqian Tang, Nong Sang, and Marcelo H. Ang Jr. Towards training stronger video vision transformers for epic-kitchens-100 action recognition.arXiv preprint arXiv:2106.05058v1, 2021

    work page internal anchor Pith review Pith/arXiv arXiv 2021

  18. [18]

    Video BagNet: short temporal receptive fields increase robustness in long-term action recognition

    Ombretta Strafforello, Xin Liu, Klamer Schutte, and Jan van Gemert. Video bagnet: Short temporal receptive fields increase robustness in long-term action recognition.arXiv preprint arXiv:2308.11249v1, 2023

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  19. [19]

    Bridge-Prompt: Towards Ordinal Action Understanding in Instructional Videos

    Muheng Li, Lei Chen, Yueqi Duan, Zhilan Hu, Jianjiang Feng, Jie Zhou, and Jiwen Lu. Bridge-prompt: Towards ordinal action understanding in instructional videos.arXiv preprint arXiv:2203.14104v1, 2022

    work page internal anchor Pith review Pith/arXiv arXiv 2022

  20. [20]

    Disttrain: Addressing model and data heterogeneity with disaggregated training for multi- modal large language models.arXiv preprint arXiv:2408.04275v2, 2024

    Zili Zhang, Yinmin Zhong, Ranchen Ming, Hanpeng Hu, Jianjian Sun, Zheng Ge, Yibo Zhu, and Xin Jin. Disttrain: Addressing model and data heterogeneity with disaggregated training for multi- modal large language models.arXiv preprint arXiv:2408.04275v2, 2024. 13

    arXiv 2024

  21. [21]

    Demonstrating and reducing shortcuts in vision-language representation learning.arXiv preprint arXiv:2402.17510v2, 2024

    Maurits Bleeker, Mariya Hendriksen, Andrew Yates, and Maarten de Rijke. Demonstrating and reducing shortcuts in vision-language representation learning.arXiv preprint arXiv:2402.17510v2, 2024

    Pith/arXiv arXiv 2024

  22. [22]

    Multimodal foundation models are better simula- tors of the human brain.arXiv preprint arXiv:2208.08263v1, 2022

    Haoyu Lu, Qiongyi Zhou, Nanyi Fei, Zhiwu Lu, Mingyu Ding, Jingyuan Wen, Changde Du, Xin Zhao, Hao Sun, Huiguang He, and Ji-Rong Wen. Multimodal foundation models are better simula- tors of the human brain.arXiv preprint arXiv:2208.08263v1, 2022

    Pith/arXiv arXiv 2022

  23. [23]

    Yiqi Wang, Wentao Chen, Xiaotian Han, Xudong Lin, Haiteng Zhao, Yongfei Liu, Bohan Zhai, Jianbo Yuan, Quanzeng You, and Hongxia Yang. Exploring the reasoning abilities of multimodal large language models (mllms): A comprehensive survey on emerging trends in multimodal rea- soning.arXiv preprint arXiv:2401.06805v2, 2024

    Pith/arXiv arXiv 2024

  24. [24]

    Accuracy vs. Complexity: A Trade-off in Visual Question Answering Models

    Moshiur R. Farazi, Salman H. Khan, and Nick Barnes. Accuracy vs. complexity: A trade-off in visual question answering models.arXiv preprint arXiv:2001.07059v1, 2020

    work page internal anchor Pith review Pith/arXiv arXiv 2001

  25. [25]

    MMSummary: Multimodal Summary Generation for Fetal Ultrasound Video

    Xiaoqing Guo, Qianhui Men, and J. Alison Noble. Mmsummary: Multimodal summary generation for fetal ultrasound video.arXiv preprint arXiv:2408.03761v2, 2024

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  26. [26]

    Improving medical large vision-language models with abnormal-aware feedback.arXiv preprint arXiv:2501.01377, 2025

    Yucheng Zhou, Lingran Song, and Jianbing Shen. Improving medical large vision-language models with abnormal-aware feedback.arXiv preprint arXiv:2501.01377, 2025

    Pith/arXiv arXiv 2025

  27. [27]

    Grounding the ungrounded: A spectral-graph framework for quantifying hallucinations in multimodal llms.arXiv preprint arXiv:2508.19366v1, 2025

    Supratik Sarkar and Swagatam Das. Grounding the ungrounded: A spectral-graph framework for quantifying hallucinations in multimodal llms.arXiv preprint arXiv:2508.19366v1, 2025

    work page arXiv 2025

  28. [28]

    Anwesha Mohanty, Venkatesh Balavadhani Parthasarathy, and Arsalan Shahid. The future of mllm prompting is adaptive: A comprehensive experimental evaluation of prompt engineering methods for robust multimodal performance.arXiv preprint arXiv:2504.10179v1, 2025. 14

    work page internal anchor Pith review Pith/arXiv arXiv 2025