Pith. sign in

REVIEW 4 major objections 6 minor 1 cited by

MLLM-SUL: Multimodal Large Language Model for Semantic Scene Understanding and Localization in Traffic Scenarios

T0 review · 4 major / 6 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that an image-only multimodal language model can jointly write structured traffic-scene descriptions and locate the riskiest object, matching or beating video-based models by routing coordinates through a regression token…

desk verdict A sensible image-based driving MLLM with a real efficiency win, but the headline SOTA claim leans on an unvalidated manual dataset extension and an underdocumented baseline comparison. read the letter →

arxiv 2412.19406 v1 pith:PGLRDNR5 submitted 2024-12-27 cs.CV

Jiaqi Fan , Jianhua Wu , Jincheng Gao , Jianhao Yu , Yafei Wang , Hongqing Chu , Bingzhao Gao This is my paper · ORCID
classification cs.CV
keywords multimodallargelanguagemodelautonomousdrivingtrafficsceneunderstandingriskobjectlocalizationimagecaptioningdual-branchvisualencodergate-attentionfusionDRAMAdataset
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to show that a single multimodal language model fed only a front-view image can do two driving tasks at once: write a structured description of the scene (scenario type, what the risky object is doing, and the ego vehicle's intention and suggested action) and draw a bounding box around the riskiest object. The proposed architecture, MLLM-SUL, fuses low- and high-resolution grid features with object-detector region features through a gate-attention module, then feeds them to a fine-tuned language model for text; a separate transformer branch with a learned regression token predicts coordinates, deliberately keeping numbers out of the language model. On the DRAMA-ROLISP dataset and an extended DRAMA-SRIS dataset with manually added scenario labels, the paper reports captioning scores (80.1% BLEU-1, 298.5% CIDEr) and localization accuracy (59.6% mIoU) that beat several image- and video-based baselines, including a video model, while using fewer FLOPs. A careful reader should care because the result suggests that temporal video input is not necessary for joint scene understanding and risk localization, and that offloading numeric prediction to a regression head may fix a known weakness of language models.

What carries the argument

The key machinery is a dual-branch visual encoder fused by a gate-attention module, paired with a query-based regression network. The low-resolution branch (ResNet-101) and high-resolution branch (Swin-L) each produce grid features that are concatenated; region features from a frozen Faster R-CNN detector are enhanced by self-attention, then attend to the concatenated grid features in a cross-attention module whose output is weighted by a zero-initialized gate and added to the region features by a residual connection. That fused representation is projected into the fine-tuned language model for description generation. Separately, a learned regression token is concatenated with the low-resolution grid features and the predicted text tokens, passed through six transformer layers, and mapped by an MLP and Sigmoid to a normalized bounding box $(x,y,w,h)$; the box head is trained with Smooth L1 and generalized IoU losses. This design is what lets the model describe scenes semantically without forcing the language model to produce unreliable numbers.

What would settle it

Take a random subset of, say, 500 DRAMA-SRIS images, have three independent annotators relabel the scenario type from the same definition, and measure agreement; then retrain or re-evaluate the captioning model on labels produced by a second annotator. If agreement is poor or the CIDEr gap over baselines shrinks materially, the claimed state-of-the-art captioning result depends on annotation idiosyncrasy rather than on the model.

Watch

Extended reading notes

Core claim

The paper's central claim is that an image-only multimodal model can achieve state-of-the-art results on joint semantic scene understanding and risk-object localization by separating the two skills: a fine-tuned language model composes natural-language descriptions, while a query-based transformer regresses bounding-box coordinates from a learned token rather than forcing the language model to produce numbers. The argument is carried by the dual-branch visual encoder: low-resolution grid features from a ResNet backbone and high-resolution grid features from a Swin backbone are concatenated, fused with Faster R-CNN region features in a gate-attention module with a zero-initialized gate, and then projected into the language model. The same low-resolution grid features, together with the generated text tokens and a regression query, enter a six-layer transformer whose first output position is mapped by an MLP and Sigmoid to a normalized (x,y,w,h) box; training uses Smooth L1 and GIoU losses. The paper reports that this architecture achieves 80.1% BLEU-1, 298.5% CIDEr, and 59.6% mIoU on DRAMA-SRIS, and that it improves BLEU-4 by 1.6%, METEOR by 2.5%, and average BLEU-4/mIoU by 0.8% over the video-based HiLM-D baseline on DRAMA-ROLISP while cutting FLOPs by 57.9% and memory by 6.4%.

Load-bearing premise

The paper assumes the manually added scenario-type labels in the extended DRAMA-SRIS dataset are consistent and unbiased, yet reports no annotation protocol or inter-annotator agreement; if those labels are noisy, the reported captioning scores and the comparison against baselines lose their footing.

Editorial extensions

If this is right

  • A single front-view image is enough to jointly generate a structured scenario description and a risk-object bounding box, so multi-frame video input may be unnecessary for these tasks.
  • Numeric localization should be delegated to a regression head rather than generated by the language model, because the method's localization accuracy (59.6% mIoU) exceeds models that let the LLM produce box coordinates.
  • Multi-scale visual features (two resolutions plus object regions) improve both captioning and localization; ablations show removing either branch lowers CIDEr by roughly 49 to 59 points and mIoU by up to 12.9 points.
  • On the same task, the image-only model can beat a video-based multimodal baseline while using 57.9% fewer FLOPs and 6.4% less memory.
  • The extended DRAMA-SRIS dataset introduces scenario-type annotations that make captions more informative, and the paper reports state-of-the-art results on that dataset.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the central claim holds, the division of labor between the language model and the regression head may transfer to other grounded-language tasks, such as referring-expression localization or visual question answering with spatial answers, where the language model comments but does not count.
  • The paper's own numbers suggest a cheap test: swap the frozen object detector for a lighter proposal module and measure whether the captioning gain survives; if region features are carrying the gain, detector choice should matter more than language-model scale.
  • Because the DRAMA-SRIS scenario labels were added by the authors with no agreement protocol, the 298.5% CIDEr figure should be read as conditional on those labels; an independent relabeling study could either confirm or shrink the gap to baselines.
  • A natural next experiment is to feed the predicted box back into the high-resolution branch as a crop and ask the language model to refine its description of the risky object, since the architecture already has the two-resolution machinery to support that.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The manuscript proposes MLLM-SUL, an image-based multimodal large language model for joint semantic scene description and risk-object localization in driving scenarios. The architecture combines a dual-branch visual encoder (a low-resolution ResNet-101 grid branch with Faster R-CNN region features and a high-resolution Swin-L grid branch) fused by a gate-attention module, a fine-tuned LLaMA-2-7B captioning module that generates descriptions of scenario type, risk-object actions, and ego-vehicle intentions and suggestions, and a transformer regression head with a learned [Reg] token that predicts bounding-box coordinates. The authors evaluate on the public DRAMA-ROLISP benchmark and on an extended DRAMA-SRIS dataset that adds manually labeled scenario types, reporting improvements over several image- and video-based MLLMs, with particular gains in BLEU-4 and mIoU on DRAMA-SRIS and reduced FLOPs relative to HiLM-D.

Significance. If the results hold, the paper would demonstrate that an image-only model with multi-resolution visual features can match or exceed video-based MLLMs on driving-scene captioning and risk localization, at substantially lower computational cost. The architecture is clearly motivated, the ablations in Tables I-III isolate the contributions of the two visual branches and the gate-attention module, and the FLOPs comparison in Table V shows a meaningful efficiency advantage. The release of code and the extended dataset is a useful community resource, and the regression-token design offers a clean alternative to letting the language model emit coordinates directly. However, the headline state-of-the-art claim currently rests on an author-created dataset whose annotation quality and baseline comparability are not established, and the discrepancy on the public DRAMA-ROLISP benchmark (Table IV) is not addressed. The significance is therefore conditional on the authors resolving these evaluation issues.

major comments (4)
  1. [IV-A1 and Table VI] The extended DRAMA-SRIS dataset is created by the authors through manual labeling of each scene into three scenario types, but the paper provides no annotation protocol, number of annotators, inter-annotator agreement, or label-quality statistics. Because the scenario phrase appears as a fixed template in every reference caption, the large CIDEr gains on this dataset (e.g., 298.5 vs. 275.0 for LLaMA-2-7B+Adapter in Table VI) could partly reflect a model's ability to exploit the template rather than better semantic understanding. The authors should report annotation statistics (e.g., class distribution, agreement scores) and validate the labels, for instance by showing a sample of images with the assigned scenario labels.
  2. [Table IV] On the public DRAMA-ROLISP benchmark, MLLM-SUL's CIDEr score (276.1) is lower than HiLM-D's (279.2), even though BLEU-4 and METEOR are higher. The abstract and conclusion claim state-of-the-art performance without qualifying this discrepancy. The authors should either temper the claim, provide a significance test for the differences, or explain why CIDEr is lower while other metrics improve; the current presentation overstates the consistency of the results.
  3. [All tables] All experimental tables report single-run results without error bars, multiple seeds, or statistical significance tests. This is particularly important for the DRAMA-ROLISP comparison, where the margins over HiLM-D are small (B4 +1.6, mIoU +0.1 in Table IV), and for the ablation differences in Table I (e.g., B4 65.2 vs. 63.5 for the gate-attention ablation). The authors should run at least three seeds and report means and standard deviations, or provide bootstrap confidence intervals, before claiming superiority.
  4. [IV-C] The baseline fine-tuning procedure is not described. It is unclear whether the LLaMA-Adapter baselines, BLIP-2, InstructBLIP, LLaVA, Video-LLaMA, and Shikra were fine-tuned on the same training split of DRAMA-ROLISP/DRAMA-SRIS, with the same prompt template and the same training budget. For the LLaMA+Adapter baselines, the paper does not state the adapter configuration, learning rate, or whether the regression module was trained for them. Without this information, the head-to-head comparison is not reproducible; the authors should document the baseline training protocol in the main text or in an appendix.
minor comments (6)
  1. [Abstract and Section IV-A3] The abstract describes 59.6% as 'accuracy' in the localization task, but Section IV-A3 defines the detection metric as mean intersection over union (mIoU). Please use the correct metric name throughout, including the abstract.
  2. [Fig. 4 caption] The caption cites 'DRAMA-ROLISP [14]', but reference [14] is BLIP-2; the dataset reference should be [31] (HiLM-D) or the original ROLISP source.
  3. [Reference [5]] Reference [5] is 'The Llama 3 Herd of Models,' but the manuscript fine-tunes LLaMA-2-7B; the LLaMA-2 reference is missing and should be cited where LLaMA is introduced in Section I.
  4. [Section III-A3] The gating factor w is described as zero-initialized, but it is not specified whether w is a scalar, a per-channel vector, or a per-token tensor; this detail affects the interpretation of the residual mechanism and should be clarified.
  5. [Fig. 5(c)] The qualitative example in Fig. 5(c) shows MLLM-SUL predicting 'urban road' while the ground truth is 'intersection,' yet the text does not discuss this scenario-type error; the authors should address how such errors are penalized by the captioning metrics, given the fixed template.
  6. [Section II-B] There is a typo in Section II-B: 'the driving intention and control action of ego-vehicle are are described' should be 'are described'.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the paper's claims are empirical benchmark results, not derivations that reduce to their inputs.

full rationale

The paper makes no load-bearing circular step. Its central claims are measured performances on DRAMA-ROLISP and on the author-extended DRAMA-SRIS dataset, and they are obtained by supervised training followed by evaluation against ground-truth captions and bounding boxes. The manual scenario-type labels added in Section IV-A1 are extra annotations, not parameters fitted to the model: the captioning metrics (BLEU, METEOR, CIDEr) measure agreement with held-out reference captions, and the localization metrics (mIoU, Acc(IoU>0.5)) measure IoU against ground-truth boxes. No equation in the paper defines a predicted quantity in terms of the same ground-truth quantity, and no fitted parameter is renamed as a prediction. The comparisons on the public DRAMA-ROLISP dataset provide an external benchmark, and the cited HiLM-D baseline [31] is not authored by the present paper's authors, so no self-citation chain is used to force a choice. The concern that the manually created DRAMA-SRIS labels may be noisy or that baseline training protocols on that extension are undocumented is a data-quality and evaluation-fairness issue, not circularity: even if the labels were unreliable, the reported numbers would still be empirical measurements rather than constructions that are equivalent to their inputs by definition.

Assumptions & free parameters 6 free parameters · 4 assumptions · 0 invented entities

The central claim rests on pretrained feature extractors, a frozen language model, and standard metrics. The only new data contribution is the manual DRAMA-SRIS scenario labels, which are not validated. No new physical or conceptual entities are introduced.

free parameters (6)
  • initial learning rate (captioning) = 5e-4
    Chosen by hand; decaying by half every 3 epochs. Affects convergence and final captioning quality.
  • regression learning rate = 1e-4
    Fixed learning rate for the regression module; hand-chosen.
  • input resolutions = 224x224 and 384x384
    Resolutions for low- and high-resolution branches; chosen by hand, affect performance as shown in Table III.
  • query length Q = not specified
    The number of learnable queries in the query former is fixed but not reported. This affects the number of visual tokens and model capacity.
  • gating factor w = zero-initialized
    Zero-initialized gating factor in the gate-attention module; its initialization is a design choice that impacts training.
  • number of regression transformer layers = 6
    Six cascaded transformer layers in the regression module; hand-chosen.
assumptions (4)
  • domain assumption Pretrained backbones (ResNet-101, Swin-L, Faster R-CNN) provide useful visual features.
    The model depends on features from these pretrained networks; their quality is taken for granted and not evaluated.
  • domain assumption Frozen LLaMA-2-7B can generate accurate descriptions when guided by visual features.
    The captioning module fine-tunes LLaMA with adapters; the base model's language priors are assumed beneficial.
  • standard math BLEU, METEOR, CIDEr, and mIoU are valid evaluation metrics for this task.
    Standard metrics are used without questioning their alignment with human judgment.
  • domain assumption The DRAMA-ROLISP ground truth is accurate and the manual DRAMA-SRIS scenario labels are reliable.
    The paper extends DRAMA with manual labels but reports no annotation quality measures.

how reviews work

0 comments
Cite this review

Pith. "Pith review of MLLM-SUL: Multimodal Large Language Model for Semantic Scene Understanding and Localization in Traffic Scenarios." pith.science (2026). https://pith.science/paper/PGLRDNR5

@misc{pith2026241219406,
  author       = {Pith},
  title        = {Pith review of: MLLM-SUL: Multimodal Large Language Model for Semantic Scene Understanding and Localization in Traffic Scenarios},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/PGLRDNR5}},
  note         = {Machine review of arXiv:2412.19406}
}
read the original abstract

Multimodal large language models (MLLMs) have shown satisfactory effects in many autonomous driving tasks. In this paper, MLLMs are utilized to solve joint semantic scene understanding and risk localization tasks, while only relying on front-view images. In the proposed MLLM-SUL framework, a dual-branch visual encoder is first designed to extract features from two resolutions, and rich visual information is conducive to the language model describing risk objects of different sizes accurately. Then for the language generation, LLaMA model is fine-tuned to predict scene descriptions, containing the type of driving scenario, actions of risk objects, and driving intentions and suggestions of ego-vehicle. Ultimately, a transformer-based network incorporating a regression token is trained to locate the risk objects. Extensive experiments on the existing DRAMA-ROLISP dataset and the extended DRAMA-SRIS dataset demonstrate that our method is efficient, surpassing many state-of-the-art image-based and video-based methods. Specifically, our method achieves 80.1% BLEU-1 score and 298.5% CIDEr score in the scene understanding task, and 59.6% accuracy in the localization task. Codes and datasets are available at https://github.com/fjq-tongji/MLLM-SUL.

Figures

Figures reproduced from arXiv: 2412.19406 by the authors.

Figure 1
Figure 1. An example of errors in MLLM. Wrong semantic descriptions and [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Overall structure of the proposed MLLM-SUL model. The prompts are: [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 4
Figure 4. Comparison of text descriptions for three traffic scenario dataset. [PITH_FULL_IMAGE:figures/full_fig_p004_4.png] view at source ↗
Figures from the paper (3 more)
Figure 3
Figure 3. Figure 3: The structure of the transformer-based regression module. (Purple [PITH_FULL_IMAGE:figures/full_fig_p004_3.png]
Figure 5
Figure 5. Figure 5: In Fig. 5(a), a white van in front is very close to the [PITH_FULL_IMAGE:figures/full_fig_p006_5.png]
Figure 5
Figure 5. Figure 5: Qualitative results in the DRAMA-SRIS dataset. The language generator in BLIP-2 and InstructBLIP is OPT [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. MCAM: Multimodal Causal Analysis Model for Ego-Vehicle-Level Driving Video Understanding

    cs.CV 2025-07 reject novelty 4.0 of 10

    MCAM is a video captioning model combining 3DResNet and VidSwin features with a graph-inspired fusion module, reporting mixed gains on BDD-X and CoVLA but failing to implement the promised causal reasoning.

Reference graph

Works this paper leans on

42 extracted references · 26 canonical work pages · cited by 1 Pith paper

  1. [1]

    Traffic sign interpretation via natural language description,

    C. Yang, K. Zhuang, M. Chen, H. Ma, X. Han, T. Han, C. Guo, H. Han, B. Zhao, and Q. Wang, “Traffic sign interpretation via natural language description,” IEEE Trans. Intell. Transp. Syst. , 2024

    work page 2024

  2. [2]

    Transcrib3D: 3D Referring Expression Resolution through Large Language Models

    J. Fang, X. Tan, S. Lin, I. Vasiljevic, V . Guizilini, H. Mei, R. Ambrus, G. Shakhnarovich, and M. R. Walter, “Transcrib3d: 3d referring expression resolution through large language models,” arXiv preprint arXiv:2404.19221, 2024

    work page Pith review arXiv 2024

  3. [3]

    Large language models powered context-aware motion prediction,

    X. Zheng, L. Wu, Z. Yan, Y . Tang, H. Zhao, C. Zhong, B. Chen, and J. Gong, “Large language models powered context-aware motion prediction,” arXiv preprint arXiv:2403.11057 , 2024

    arXiv 2024

  4. [4]

    Rlingua: Improving reinforcement learning sample efficiency in robotic manipulations with large language models,

    L. Chen, Y . Lei, S. Jin, Y . Zhang, and L. Zhang, “Rlingua: Improving reinforcement learning sample efficiency in robotic manipulations with large language models,” IEEE Robot Autom Lett. , 2024

    work page 2024

  5. [5]

    The llama 3 herd of models,

    A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Letman, A. Mathur, A. Schelten, A. Yang, A. Fan et al., “The llama 3 herd of models,” arXiv preprint arXiv:2407.21783 , 2024

    arXiv 2024

  6. [6]

    Gpt-4 technical report,

    J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkat et al. , “Gpt-4 technical report,” arXiv preprint arXiv:2303.08774 , 2023

    arXiv 2023

  7. [7]

    Drive like a human: Rethinking autonomous driving with large language models,

    D. Fu, X. Li, L. Wen, M. Dou, P. Cai, B. Shi, and Y . Qiao, “Drive like a human: Rethinking autonomous driving with large language models,” in Proc. IEEE Winter Conf. Appl. Comput. Vis.(WACV) , 2024, pp. 910–919

    work page 2024

  8. [8]

    Trafficgpt: Viewing, processing and interacting with traffic foundation models,

    S. Zhang, D. Fu, W. Liang, Z. Zhang, B. Yu, P. Cai, and B. Yao, “Trafficgpt: Viewing, processing and interacting with traffic foundation models,” Transport Policy, vol. 150, pp. 95–105, 2024

    work page 2024

Show all 42 references
  1. [9]

    Emma: End-to-end multimodal model for autonomous driving,

    J.-J. Hwang, R. Xu, H. Lin, W.-C. Hung, J. Ji, K. Choi, D. Huang, T. He, P. Covington, B. Sapp et al., “Emma: End-to-end multimodal model for autonomous driving,” arXiv preprint arXiv:2410.23262 , 2024

    2024 arXiv

  2. [10]

    Drivevlm: The convergence of au- tonomous driving and large vision-language models,

    X. Tian, J. Gu, B. Li, Y . Liu, Y . Wang, Z. Zhao, K. Zhan, P. Jia, X. Lang, and H. Zhao, “Drivevlm: The convergence of au- tonomous driving and large vision-language models,” arXiv preprint arXiv:2402.12289, 2024

    2024 arXiv

  3. [11]

    Idd- x: A multi-view dataset for ego-relative important object localization and explanation in dense and unstructured traffic,

    C. Parikh, R. Saluja, C. Jawahar, and R. K. Sarvadevabhatla, “Idd- x: A multi-view dataset for ego-relative important object localization and explanation in dense and unstructured traffic,” arXiv preprint arXiv:2404.08561, 2024

    2024 arXiv

  4. [12]

    Rank2tell: A multimodal driving dataset for joint importance ranking and reasoning,

    E. Sachdeva, N. Agarwal, S. Chundi, S. Roelofs, J. Li, M. Kochen- derfer, C. Choi, and B. Dariush, “Rank2tell: A multimodal driving dataset for joint importance ranking and reasoning,” in Proc. IEEE Winter Conf. Appl. Comput. Vis.(WACV) , 2024, pp. 7513–7522

    2024

  5. [13]

    Drama: Joint risk localization and captioning in driving,

    S. Malla, C. Choi, I. Dwivedi, J. H. Choi, and J. Li, “Drama: Joint risk localization and captioning in driving,” inProc. IEEE Winter Conf. Appl. Comput. Vis.(WACV), 2023, pp. 1043–1052

    2023

  6. [14]

    Blip-2: Bootstrapping language- image pre-training with frozen image encoders and large language models,

    J. Li, D. Li, S. Savarese, and S. Hoi, “Blip-2: Bootstrapping language- image pre-training with frozen image encoders and large language models,” arXiv preprint arXiv:2301.12597 , 2023

    2023 arXiv

  7. [15]

    Instructblip: Towards general-purpose vision- language models with instruction tuning,

    W. Dai, J. Li, D. Li, A. M. H. Tiong, J. Zhao, W. Wang, B. Li, P. Fung, and S. Hoi, “Instructblip: Towards general-purpose vision- language models with instruction tuning,” 2023

    2023

  8. [16]

    Llama-adapter: Efficient fine-tuning of language models with zero-init attention,

    R. Zhang, J. Han, A. Zhou, X. Hu, S. Yan, P. Lu, H. Li, P. Gao, and Y . Qiao, “Llama-adapter: Efficient fine-tuning of language models with zero-init attention,” arXiv preprint arXiv:2303.16199 , 2023

    2023 arXiv

  9. [17]

    A hybrid cnn-lstm approach for image caption generation,

    M. Srikanth, J. Javeed, M. G. Babu et al., “A hybrid cnn-lstm approach for image caption generation,” Musik In Bayern , vol. 89, no. 4, 2024

    2024

  10. [18]

    Improving pre-trained cnn-lstm models for image captioning with hyper-parameter optimization,

    N. M. Khassaf and N. H. M. Ali, “Improving pre-trained cnn-lstm models for image captioning with hyper-parameter optimization,” Engineering, Technology & Applied Science Research , vol. 14, no. 5, pp. 17 337–17 343, 2024

    2024

  11. [19]

    Benet: bi-directional enhanced network for image captioning,

    P. Yan, Z. Li, R. Hu, and X. Cao, “Benet: bi-directional enhanced network for image captioning,” Multimedia Syst., vol. 30, no. 1, p. 48, 2024

    2024

  12. [20]

    Regular constrained multi- modal fusion for image captioning,

    L. Wang, H. Chen, Y . Liu, and Y . Lyu, “Regular constrained multi- modal fusion for image captioning,” IEEE Trans. Circuits Syst. Video Technol., 2024

    2024

  13. [21]

    A dual-feature-based adaptive shared transformer network for image captioning,

    Y . Shi, J. Xia, M. Zhou, and Z. Cao, “A dual-feature-based adaptive shared transformer network for image captioning,”IEEE Trans Instrum Meas., 2024

    2024

  14. [22]

    Dual-adaptive interactive transformer with textual and visual context for image captioning,

    L. Chen and K. Li, “Dual-adaptive interactive transformer with textual and visual context for image captioning,” Expert Syst Appl. , vol. 243, p. 122955, 2024

    2024

  15. [23]

    Visual instruction tuning,

    H. Liu, C. Li, Q. Wu, and Y . J. Lee, “Visual instruction tuning,” vol. 36, 2024

    2024

  16. [24]

    Video-llama: An instruction-tuned audio-visual language model for video understanding,

    H. Zhang, X. Li, and L. Bing, “Video-llama: An instruction-tuned audio-visual language model for video understanding,” arXiv preprint arXiv:2306.02858, 2023

    2023 arXiv

  17. [25]

    ep-alm: Efficient perceptual augmentation of language models,

    M. Shukor, C. Dancette, and M. Cord, “ep-alm: Efficient perceptual augmentation of language models,” arXiv preprint arXiv:2303.11403 , 2023

    2023 arXiv

  18. [26]

    Vlaad: Vision and language assistant for autonomous driving,

    S. Park, M. Lee, J. Kang, H. Choi, Y . Park, J. Cho, A. Lee, and D. Kim, “Vlaad: Vision and language assistant for autonomous driving,” in Proc. IEEE Winter Conf. Appl. Comput. Vis.(WACV) , 2024, pp. 980– 987

    2024

  19. [27]

    Dolphins: Multimodal language model for driving,

    Y . Ma, Y . Cao, J. Sun, M. Pavone, and C. Xiao, “Dolphins: Multimodal language model for driving,” in Proc. Eur. Conf. Comput. Vis. (ECCV), 2025, pp. 403–420

    2025

  20. [28]

    Drivegpt4: Interpretable end-to-end autonomous driving via large language model,

    Z. Xu, Y . Zhang, E. Xie, Z. Zhao, Y . Guo, K.-Y . K. Wong, Z. Li, and H. Zhao, “Drivegpt4: Interpretable end-to-end autonomous driving via large language model,” IEEE Robot Autom Lett. , 2024

    2024

  21. [29]

    Holistic autonomous driving understanding by bird’s-eye-view injected multi- modal large models,

    X. Ding, J. Han, H. Xu, X. Liang, W. Zhang, and X. Li, “Holistic autonomous driving understanding by bird’s-eye-view injected multi- modal large models,” arXiv preprint arXiv:2401.00988 , 2024

    2024 arXiv

  22. [30]

    Rag-driver: Generalisable driving explanations with retrieval-augmented in-context learning in multi-modal large language model,

    J. Yuan, S. Sun, D. Omeiza, B. Zhao, P. Newman, L. Kunze, and M. Gadd, “Rag-driver: Generalisable driving explanations with retrieval-augmented in-context learning in multi-modal large language model,” arXiv preprint arXiv:2402.10828 , 2024

    2024

  23. [31]

    Hilm-d: Towards high-resolution understanding in multimodal large language models for autonomous driving,

    X. Ding, J. Han, H. Xu, W. Zhang, and X. Li, “Hilm-d: Towards high-resolution understanding in multimodal large language models for autonomous driving,” arXiv preprint arXiv:2309.05186 , 2023

    2023 arXiv

  24. [32]

    Deep residual learning for image recognition,

    K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR), 2016, pp. 770–778

    2016

  25. [33]

    Faster r-cnn: Towards real- time object detection with region proposal networks,

    S. Ren, K. He, R. Girshick, and J. Sun, “Faster r-cnn: Towards real- time object detection with region proposal networks,” vol. 28, 2015

    2015

  26. [34]

    An image is worth 16x16 words: Transformers for image recognition at scale,

    A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly et al., “An image is worth 16x16 words: Transformers for image recognition at scale,” arXiv preprint arXiv:2010.11929 , 2020

    2010 arXiv

  27. [35]

    Swin transformer: Hierarchical vision transformer using shifted windows,

    Z. Liu, Y . Lin, Y . Cao, H. Hu, Y . Wei, Z. Zhang, S. Lin, and B. Guo, “Swin transformer: Hierarchical vision transformer using shifted windows,” in Proc. IEEE Int. Conf. Comput. Vis. (ICCV), 2021, pp. 10 012–10 022

    2021

  28. [36]

    Sentencepiece: A simple and language independent subword tokenizer and detokenizer for neural text pro- cessing,

    T. Kudo and J. Richardson, “Sentencepiece: A simple and language independent subword tokenizer and detokenizer for neural text pro- cessing,” arXiv preprint arXiv:1808.06226 , 2018

    2018 arXiv

  29. [37]

    Generalized intersection over union: A metric and a loss for bounding box regression,

    H. Rezatofighi, N. Tsoi, J. Gwak, A. Sadeghian, I. Reid, and S. Savarese, “Generalized intersection over union: A metric and a loss for bounding box regression,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR), 2019, pp. 658–666

    2019

  30. [38]

    Bleu: a method for automatic evaluation of machine translation,

    K. Papineni, S. Roukos, T. Ward, and W.-J. Zhu, “Bleu: a method for automatic evaluation of machine translation,” in Proc. Annu. Meeting Assoc. Comput. Linguistics (ACL) , 2002, pp. 311–318

    2002

  31. [39]

    Meteor: An automatic metric for mt evaluation with improved correlation with human judgments,

    S. Banerjee and A. Lavie, “Meteor: An automatic metric for mt evaluation with improved correlation with human judgments,” in Proc. Acl Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization , 2005, pp. 65–72

    2005

  32. [40]

    Cider: Consensus- based image description evaluation,

    R. Vedantam, C. Lawrence Zitnick, and D. Parikh, “Cider: Consensus- based image description evaluation,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR), 2015, pp. 4566–4575

    2015

  33. [41]

    Swin transformer v2: Scaling up capacity and resolution,

    Z. Liu, H. Hu, Y . Lin, Z. Yao, Z. Xie, Y . Wei, J. Ning, Y . Cao, Z. Zhang, L. Dong, F. Wei, and B. Guo, “Swin transformer v2: Scaling up capacity and resolution,” in Proc. IEEE Conf. Comput. Vis. Pattern Recognit. (CVPR), 2022

    2022

  34. [42]

    Shikra: Unleashing multimodal llm’s referential dialogue magic,

    K. Chen, Z. Zhang, W. Zeng, R. Zhang, F. Zhu, and R. Zhao, “Shikra: Unleashing multimodal llm’s referential dialogue magic,” arXiv preprint arXiv:2306.15195 , 2023

    2023 arXiv

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.