Pith. sign in

REVIEW 4 major objections 6 minor 1 cited by

Task-Oriented Semantic Communication in Large Multimodal Models-based Vehicle Networks

T0 review · 4 major / 6 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read This paper claims that task-oriented image cropping and attention-weighted power allocation let a cloud-run LLaVA answer driver questions accurately over low-SNR channels, raising VQA accuracy by 13.4% at 12 dB and 33.1% at 10 dB.

desk verdict The SM-based image slicing for LLaVA is a genuinely useful efficiency idea; the attention-based power allocation is the weak joint and needs much more validation before the low-SNR gains can be trusted. read the letter →

arxiv 2505.02413 v1 pith:3AT2ISG6 submitted 2025-05-05 cs.AI

Baoxia Du , Hongyang Du , Dusit Niyato , Ruidong Li This is my paper · ORCID
classification cs.AI
keywords semanticcommunicationlargemultimodalmodelsLLaVAvisualquestionansweringresourceallocationuserattentionvehiclenetworkstask-oriented
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that a vehicle can offload everything except a small image encoder to a cloud server and still get accurate answers from a Large Language and Vision Assistant (LLaVA) over a noisy wireless link, as long as the communication is task-oriented. The first intervention is a Semantic Matching module that extracts keywords from the driver's question, detects candidate objects, matches them with word-vector similarity, and crops the image to the region the question is about; this drops the number of image tokens the language model must process from 2880 to 1152 with only 0.6% accuracy loss on Vicuna-7B. The second is a fusion-attention power allocation scheme that combines generic visual saliency with the user's subjective focus, then gives more transmission power to the image patches that carry the answer. On a custom traffic visual-question-answering dataset, the paper reports accuracy gains of 13.4% at 12 dB SNR and 33.1% at 10 dB SNR over equal-power transmission. If true, this would make LMM-based driving assistants practical in exactly the channel conditions where current systems fail.

What carries the argument

The load-bearing mechanism is the fused attention heatmap and the power-allocation law derived from it. Objective attention $H_{\mathrm{obj}}$ comes from a lightweight saliency network; subjective attention $H_{\mathrm{sub}}$ places high weight on the bounding box found by matching the user's keywords to detected objects; and the two are blended as $H_a = \alpha H_{\mathrm{obj}} + (1-\alpha)H_{\mathrm{sub}}$ with $\alpha = 0.5$. Each image patch then receives importance $W_p(p_i) = H_a(p_i)$, is quantized into $L=5$ levels via $W_L(p_i)$, and is assigned transmit power $P(p_i) = (W_L(p_i))^{\beta}/\sum_j (W_L(p_j))^{\beta} \cdot P_{\mathrm{total}}$. This law is what converts which parts of the image this user cares about into a physical resource allocation, sending the patches that answer the question at higher power and lower error rate.

What would settle it

Run the same low-SNR VQA comparison but scramble the patch importance weights before computing the power allocation: if accuracy stays at the FA-SemCom level, the attention-to-feature mapping is not doing the work, and if it drops to the equal-power level, the mapping is essential.

Watch

Extended reading notes

Core claim

The central claim is that both parts of the framework, task-oriented cropping and attention-weighted power allocation, are needed, and together they let a cloud-run LLaVA answer visual questions about traffic scenes despite poor channels. The Semantic Matching module uses YAKE to extract keyword phrases from the question, YOLO to detect objects, GloVe cosine similarity to pick the best matching object, and a flexible cropping function to produce a slice that the CLIP-ViT encoder converts into features alongside the base image. The FA-SemCom module then computes a fused attention heatmap $H_a = \alpha H_{\mathrm{obj}} + (1-\alpha) H_{\mathrm{sub}}$ with $\alpha = 0.5$, where $H_{\mathrm{obj}}$ is a saliency prediction and $H_{\mathrm{sub}}$ highlights the matched bounding box, quantizes each patch's importance to five levels, and allocates power $P(p_i) = (W_L(p_i))^{\beta}/\sum_{j}(W_L(p_j))^{\beta} \cdot P_{\mathrm{total}}$ with $\beta$ tunable. In simulations over a Fisher-Snedecor F fading channel this raises answer accuracy by 13.4% at 12 dB SNR and 33.1% at 10 dB SNR relative to equal power, with per-patch bit-error plots showing the target region, such as a license plate, transmitted with fewer errors.

Load-bearing premise

The gain rests on the assumption that boosting transmit power for a user-relevant image patch actually protects the part of the encoded feature vector the LLM will read, even though the visual encoder's global self-attention mixes information across all patches; the paper does not test that correspondence.

Editorial extensions

If this is right

  • Vehicles can run only the visual encoder locally and offload the LLM to a server, because the transmitted semantic features, not the raw image, carry the task-relevant content.
  • Cropping to the matched object cuts visual tokens from 2880 to 1152 and FLOPs from 43.58 T to 17.43 T, cutting average response time by roughly 27% with at most a 0.6% accuracy hit on Vicuna-7B.
  • At low SNR, attention-weighted power allocation beats equal-power transmission by 13.4% at 12 dB and 33.1% at 10 dB, so accurate LMM-based driving assistants become feasible in harsh channels that break naive transmission.
  • Because the method only assumes patch-based image encoding, it transfers to other LMMs with CLIP-ViT or SigLIP encoders, and the paper argues it can be retrained for other domains such as agricultural inspection.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper's accuracy curves show that the optimal $\beta$ shifts with channel conditions, yet $\beta$ is treated as a constant; an adaptive rule that raises $\beta$ when SNR drops is a natural next step that the paper does not work out.
  • If the per-patch attention weights are scrambled before power allocation and the low-SNR accuracy does not fall, then the claimed link between patch importance and feature-dimension protection is not causal; this experiment would settle the mechanism.
  • The reported numbers probably underestimate what a fine-tuned LMM would achieve, since the paper itself notes that LLaVA was not trained on a specialized traffic dataset, and a domain-adapted model would correct more of the noisy inputs.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. This paper proposes a task-oriented semantic communication framework for LMM-based vehicle networks. The system splits LLaVA's computation: the CLIP-ViT visual encoder runs on the vehicle, and the LLM runs on a cloud server, with encoded image features transmitted over a wireless channel. To reduce computational cost, the authors introduce a Semantic Matching (SM) module that extracts keywords from the user question, detects objects with YOLO, and crops the most relevant image slice for transmission, yielding a model called LLaVA-SM. To improve robustness under poor channel conditions, they further propose FA-SemCom, which fuses an objective saliency heatmap with a subjective attention heatmap derived from the user's question, quantizes the fused attention into weights, and allocates transmission power to image patch features according to Eq. (15). Experiments on a self-constructed traffic VQA dataset (41 images, 172 questions) show that LLaVA-SM reduces visual tokens from 2880 to 1152 with at most 0.6% accuracy loss, and that FA-SemCom improves answer accuracy by 13.4% at 12 dB and 33.1% at 10 dB over average-power transmission.

Significance. If the claims hold, the paper addresses a practically important problem: deploying large multimodal assistants in vehicles with limited onboard compute and unreliable wireless links. The idea of using user attention to guide both image cropping and transmission power is intuitively appealing and goes beyond prior CNN-based semantic communication schemes by targeting LMMs. The efficiency result for LLaVA-SM is plausible and well presented with token and FLOP counts. The paper is clearly written and situates itself well in the related literature on semantic communication, visual saliency, and efficient LMMs. However, the experimental evidence is thin (41 images, 172 hand-written questions, no released code or dataset, no confidence intervals), and the core power-allocation mechanism for ViT-based encoders is not validated. The reported low-SNR gains rest on an assumption about the correspondence between pixel-space attention and ViT output tokens that is not established in the paper.

major comments (4)
  1. [Section 5.3, Eq. (15) and Algorithm 2] The power-allocation scheme assumes a direct, monotone mapping from the fused pixel-space attention weight W_p(p_i) of patch p_i to the importance of the i-th output token of the CLIP-ViT encoder, and then to the individual dimensions of Z_v. This assumption is not justified for LLaVA: the ViT applies global self-attention across all patches, so each output token is a mixture of all input patches, and the projection W in Eq. (1) linearly mixes the token dimensions before the LLM. The surviving noise therefore acts on the LLM embedding space, not directly on the token space. The paper provides no diagnostic demonstrating that the allocated power protects the semantically relevant regions, for example by comparing against an oracle per-patch sensitivity map or by ablating the mapping. Since the central claim of 13.4% and 33.1% accuracy improvements at low SNR rests on this mechanism, this issue is load-bearing.
  2. [Algorithm 2, steps 9-12] The fused attention heatmap H_a is computed on the full image X_v, but the transmitted features come from both the resized base image X_b and the rescaled slice X_s. The patch grids of X_b and X_s do not coincide with the patch grid of X_v. The paper does not specify how W_p(p_i) is defined for the patches of X_s, nor how the importance weights computed on X_v are mapped to the slice. Without this coordinate transformation, Algorithm 2 as written is ambiguous and the implementation cannot be reproduced from the description.
  3. [Section 6.1 and Section 6.2] The evaluation uses a self-constructed dataset of 41 images and 172 questions, with manual accuracy scoring, and no confidence intervals or statistical tests are reported. The dataset and code are not released. Given the small sample size, the headline improvements of 13.4% (12 dB) and 33.1% (10 dB) may not be stable, and the per-question or per-image variance is not reported. The paper should report variance across images or bootstrapped confidence intervals, and ideally release the dataset and code to allow independent verification.
  4. [Section 6.2, Fig. 8 and Eq. (15)] The parameter beta is tuned per SNR (beta=4 at 10 dB in Fig. 7, beta=1 at 12 dB in Fig. 6), but the paper does not provide a mechanism for selecting beta online based on channel conditions. Without such a procedure, the reported gains are for a tuned parameter, not for an adaptive policy. Additionally, the abstract and conclusion report improvements of 13.4% and 33.1%, but the underlying numbers in Section 6.2 (e.g., 0.436 to 0.767) suggest these are percentage-point increases rather than relative percentage increases; the terminology should be corrected for precision.
minor comments (6)
  1. [Section 6.2, Fig. 6] The text states that Fig. 6(a) uses Vicuna-7B and Fig. 6(b) uses Mistral-7B, but the figure caption says the opposite. Please correct this inconsistency.
  2. [Abstract and Conclusion] The phrases 'improved by 13.4%' and '33.1%' are ambiguous; please specify whether these are relative improvements or absolute percentage-point improvements.
  3. [Algorithm 1, line 6] Step 6 says 'Calculate the cosine similarity between Kq and ci', but the cosine similarity is computed between the vector representations v_Kq and v_ci in Eqs. (4)-(5); the wording should be updated accordingly.
  4. [Section 4.1, Table 3] The text mentions that a 672x1008 image produces 3456 visual tokens for LLaVA-1.6, while Table 3 lists 2880 (576x5) tokens for the configuration used in the experiments; please clarify the resolution and slicing configuration used in the evaluation.
  5. [Section 6.1] The statement that YOLOv8-n is 'pre-trained on a dataset of 166 training images and 40 validation images' is surprising for an object detector; please clarify whether this refers to fine-tuning on a custom dataset rather than the pre-training corpus.
  6. [Section 4.2] The SM module is described as selecting a 'target area' for image slicing, but Algorithm 1 returns only one slice X_s. If multiple target boxes are detected, the paper should specify how they are combined or whether multiple slices are generated.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the accuracy improvements and token-reduction results are empirical measurements against hand-labeled ground truth, not quantities forced by the attention or power-allocation construction.

full rationale

The paper's central claims are empirical, not derivational. The 13.4% and 33.1% accuracy improvements are measured end-to-end answer accuracies on a hand-labeled traffic VQA dataset (Section 6.1: 'We use answer accuracy to evaluate whether the model can correctly recognize the information it receives'), so they are not computed from the attention weights or power-allocation formula. Eq. (13) sets W_p(p_i)=H_a(p_i) and Eq. (15) maps those weights to transmit power; this is a design heuristic, not a derivation that answer accuracy equals some function of H_a. The power-allocation and Fisher-Snedecor channel modeling are attributed to the authors' prior work [30], but the paper presents the formulas and tests them by simulation; no uniqueness theorem or exclusivity argument is imported from [30] to forbid alternatives. The beta sweep in Fig. 8 is selected on the test set, which is a methodological concern about optimistic reporting, but it is not a fitted parameter that makes the reported accuracy true by construction, since accuracy is still measured against ground-truth answers. The SM slicing and subjective-attention modules use the user's question to locate the queried object, and the dataset questions do concern such objects; that is the intended task-oriented evaluation, not a circular redefinition of the accuracy metric. The ViT token-to-region mapping worry (output tokens mix all input patches) is a generalization or validity risk, not a circularity. No load-bearing step reduces to its own input.

Assumptions & free parameters 4 free parameters · 6 assumptions · 0 invented entities

The framework rests on several unverified domain assumptions: the patch-to-feature power mapping, the channel model, the SM matching pipeline, the transferability of the saliency model, the manual accuracy metric, and the zero-shot use of LLaVA. The free parameters alpha, beta, and L are chosen by hand or tuned empirically, and the channel parameters are simulation settings.

free parameters (4)
  • alpha (attention fusion coefficient) = 0.5
    Set to 0.5 as an equitable blend of objective and subjective attention without optimization (Section 5.3, Eq. 12).
  • beta (power allocation weight adjustment) = 1 for headline results; swept up to 4 in Fig. 8
    Tunable parameter in Eq. (15) controlling how strongly importance weights affect power allocation; chosen empirically per experiment, with no closed-form rule.
  • L (number of importance quantization levels) = 5
    Motivated by a user-object-attention level dataset [66], not derived from the current problem (Eq. 14).
  • Channel fading and shadowing parameters = m_f=5, m_s=4
    Simulation settings for the Fisher-Snedecor F channel (Section 6.1), chosen by the authors, not fitted to the accuracy results.
assumptions (6)
  • domain assumption Patch-level importance can be mapped to per-dimension importance of the encoded feature vector Z_v for power allocation.
    Algorithm 2 line 12 assigns power to feature dimensions according to spatial patch weights W_p(p_i), citing [30]; no verification is provided for LLaVA, whose ViT mixes patch information via self-attention.
  • domain assumption The Fisher-Snedecor F model with Nakagami-m fading and inverse Nakagami-m shadowing describes the vehicle-to-server channel.
    Section 6.1 sets m_f=5 and m_s=4 based on reference [30]; the appropriateness for the scenario is not validated with measurements.
  • domain assumption YAKE keyword extraction, YOLOv8-n object detection, and GloVe cosine similarity correctly locate the image region that answers the user's question.
    The SM module (Section 4.2) relies on this pipeline; failure cases such as questions about weather (Table 2) are not analyzed.
  • domain assumption The saliency model pretrained on MIT1003 transfers to traffic scenes captured from a vehicle.
    Section 6.1 uses the model from [61], trained on MIT1003, as the objective attention predictor; no traffic-specific fine-tuning or evaluation is reported.
  • domain assumption The answer accuracy judgment, defined as generally matching the ground truth, is a reliable evaluation metric.
    Section 6.1 defines accuracy by human matching; no automated metric, annotation protocol, or inter-annotator agreement is described.
  • domain assumption LLaVA, without traffic-specific fine-tuning, can answer the constructed VQA questions well enough for evaluation.
    The paper notes in the footnote to Section 6.2 that LLaVA was not trained on a specialized traffic dataset; the evaluation assumes zero-shot performance is meaningful for the comparison.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Task-Oriented Semantic Communication in Large Multimodal Models-based Vehicle Networks." pith.science (2026). https://pith.science/paper/3AT2ISG6

@misc{pith2026250502413,
  author       = {Pith},
  title        = {Pith review of: Task-Oriented Semantic Communication in Large Multimodal Models-based Vehicle Networks},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/3AT2ISG6}},
  note         = {Machine review of arXiv:2505.02413}
}
read the original abstract

Task-oriented semantic communication has emerged as a fundamental approach for enhancing performance in various communication scenarios. While recent advances in Generative Artificial Intelligence (GenAI), such as Large Language Models (LLMs), have been applied to semantic communication designs, the potential of Large Multimodal Models (LMMs) remains largely unexplored. In this paper, we investigate an LMM-based vehicle AI assistant using a Large Language and Vision Assistant (LLaVA) and propose a task-oriented semantic communication framework to facilitate efficient interaction between users and cloud servers. To reduce computational demands and shorten response time, we optimize LLaVA's image slicing to selectively focus on areas of utmost interest to users. Additionally, we assess the importance of image patches by combining objective and subjective user attention, adjusting energy usage for transmitting semantic information. This strategy optimizes resource utilization, ensuring precise transmission of critical information. We construct a Visual Question Answering (VQA) dataset for traffic scenarios to evaluate effectiveness. Experimental results show that our semantic communication framework significantly increases accuracy in answering questions under the same channel conditions, performing particularly well in environments with poor Signal-to-Noise Ratios (SNR). Accuracy can be improved by 13.4% at an SNR of 12dB and 33.1% at 10dB, respectively.

Figures

Figures reproduced from arXiv: 2505.02413 by the authors.

Figure 1
Figure 1. The proposed task-oriented semantic communication system framework. Section 4.1 introduces the architecture of LLaVA, Section 4.2 [PITH_FULL_IMAGE:figures/full_fig_p005_1.png] view at source ↗
Figure 2
Figure 2. A simple processing flow of the system framework [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. Three different image encoding mechanisms (i.e., LLaVA-1.5, LLaVA-1.6, and the proposed LLaVA-SM). The visualization displays the clarity [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (6 more)
Figure 4
Figure 4. Figure 4: The proposed fusion attention-based resource allocation method. By evaluating the attention weights of each image patch, resources are [PITH_FULL_IMAGE:figures/full_fig_p009_4.png]
Figure 5
Figure 5. Figure 5: Comparison of average response times for LLaVA-1.6 and [PITH_FULL_IMAGE:figures/full_fig_p011_5.png]
Figure 8
Figure 8. Figure 8: The answer accuracy under different weight adjustment variables [PITH_FULL_IMAGE:figures/full_fig_p012_8.png]
Figure 6
Figure 6. Figure 6: The curves of model answer accuracy for different transmission [PITH_FULL_IMAGE:figures/full_fig_p012_6.png]
Figure 7
Figure 7. Figure 7: Comparison of different attention mechanisms. The LLM back [PITH_FULL_IMAGE:figures/full_fig_p012_7.png]
Figure 9
Figure 9. Figure 9: Visualization of Bit Error Rates (BER) across different image patches with different weight adjustment variables [PITH_FULL_IMAGE:figures/full_fig_p013_9.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. LLM-Enhanced Space-Air-Ground-Sea Integrated Networks

    eess.SP 2025-09 reject novelty 4.0 of 10

    The paper claims a single LLM can jointly predict fast-fading channels and enable semantic image transmission across SAGSIN links, but the experiments only demonstrate each piece separately on radio and underwater aco...

Reference graph

Works this paper leans on

71 extracted references · 44 canonical work pages · cited by 1 Pith paper

  1. [1]

    On the Hidden Mystery of OCR in Large Multimodal Models,

    Y. Liu, Z. Li, H. Li, W. Yu, M. Huang, D. Peng, M. Liu, M. Chen, C. Li, L. Jin, and X. Bai, “On the Hidden Mystery of OCR in Large Multimodal Models,” ArXiv, vol. abs/2305.07895, 2023

    arXiv 2023

  2. [2]

    MathVista: Evaluating Math Reasoning in Visual Contexts with GPT-4V , Bard, and Other Large Multimodal Models,

    P . Lu, H. Bansal, T. Xia, J. Liu, C. yue Li, H. Hajishirzi, H. Cheng, K.-W. Chang, M. Galley, and J. Gao, “MathVista: Evaluating Math Reasoning in Visual Contexts with GPT-4V , Bard, and Other Large Multimodal Models,” ArXiv, vol. abs/2310.02255, 2023

    arXiv

  3. [3]

    Improved Baselines with Visual Instruction Tuning,

    H. Liu, C. Li, Y. Li, and Y. J. Lee, “Improved Baselines with Visual Instruction Tuning,” ArXiv, vol. abs/2310.03744, 2023. [Online]. Available: https://api.semanticscholar.org/CorpusID:263672058

    arXiv 2023

  4. [4]

    Multimodal Foundation Models: From Specialists to General- Purpose Assistants,

    C. Li, Z. Gan, Z. Yang, J. Yang, L. Li, L. Wang, and J. Gao, “Multimodal Foundation Models: From Specialists to General- Purpose Assistants,” Found. T rends Comput. Graph. Vis., vol. 16, pp. 1–214, 2023

    work page 2023

  5. [5]

    GPT-4V in Wonderland: Large Multimodal Models for Zero-Shot Smartphone GUI Navigation,

    A. Yan, Z. Yang, W. Zhu, K. Q. Lin, L. Li, J. Wang, J. Yang, Y. Zhong, J. J. McAuley, J. Gao, Z. Liu, and L. Wang, “GPT-4V in Wonderland: Large Multimodal Models for Zero-Shot Smartphone GUI Navigation,” ArXiv, vol. abs/2311.07562, 2023

    arXiv 2023

  6. [6]

    LMaaS: Exploring Pricing Strategy of Large Model as a Service for Communication,

    P . Wu, Q. Liu, Y. Dong, Z. Wang, and F. Wang, “LMaaS: Exploring Pricing Strategy of Large Model as a Service for Communication,” IEEE T ransactions on Mobile Computing, pp. 1–13, 2024

    work page 2024

  7. [7]

    GPT-4 Technical Report,

    OpenAI, “GPT-4 Technical Report,” OpenAI, Tech. Rep., 2023

    work page 2023

  8. [8]

    ChatGPT,

    ——, “ChatGPT,” 2024. [Online]. Available: https://openai.com/ blog/chatgpt/ © 2025 IEEE. All rights reserved, including rights for text and data mining and training of artificial intelligence and similar technologies. Personal use is permitted, but republication/redistribution requires IEEE permission. See https://www.ieee.org/publications/rights/index.ht...

    work page arXiv 2024

Show all 71 references
  1. [9]

    A Survey on Multimodal Large Language Models for Autonomous Driving,

    C. Cui, Y. Ma, X. Cao, W. Ye, Y. Zhou, K. Liang, J. Chen, J. Lu, Z. Yang, K.-D. Liao, T. Gao, E. Li, K. Tang, Z. Cao, T. Zhou, A. Liu, X. Yan, S. Mei, J. Cao, Z. Wang, and C. Zheng, “A Survey on Multimodal Large Language Models for Autonomous Driving,” 2024 IEEE/CVF Winter Con...

    2024

  2. [10]

    Vision Language Models in Autonomous Driving: A Survey and Outlook,

    X. Zhou, M. Liu, E. Yurtsever, B. L. ˇZagar, W. Zimmer, H. Cao, and A. C. Knoll, “Vision Language Models in Autonomous Driving: A Survey and Outlook,” IEEE T ransactions on Intelligent Vehicles , 2023

    2023

  3. [11]

    HiLM- D: Towards High-Resolution Understanding in Multimodal Large Language Models for Autonomous Driving,

    X. Ding, J. Han, H. Xu, W. Zhang, and X. Li, “HiLM- D: Towards High-Resolution Understanding in Multimodal Large Language Models for Autonomous Driving,” ArXiv, vol. abs/2309.05186, 2023. [Online]. Available: https://api. semanticscholar.org/CorpusID:261682424

    2023 arXiv

  4. [12]

    Drive as You Speak: Enabling Human-Like Interaction with Large Language Models in Autonomous Vehicles,

    C. Cui, Y. Ma, X. Cao, W. Ye, and Z. Wang, “Drive as You Speak: Enabling Human-Like Interaction with Large Language Models in Autonomous Vehicles,” 2024 IEEE/CVF Winter Conference on Applications of Computer Vision Workshops (WACVW) , pp. 902–909, 2023

    2024

  5. [13]

    SurrealDriver: Designing Generative Driver Agent Simulation Framework in Urban Contexts based on Large Language Model,

    Y. Jin, X. Shen, H. Peng, X. Liu, J. Qin, J. Li, J. Xie, P . Gao, G. Zhou, and J. Gong, “SurrealDriver: Designing Generative Driver Agent Simulation Framework in Urban Contexts based on Large Language Model,” ArXiv, vol. abs/2309.13193, 2023. [Online]. Available: https://api.s...

    2023 arXiv

  6. [14]

    DriveLLaVA: Human-Level Behavior Decisions via Vision Language Model,

    R. Zhao, Q. Yuan, J. Li, Y. Fan, Y. Li, and F. Gao, “DriveLLaVA: Human-Level Behavior Decisions via Vision Language Model,” Sensors (Basel, Switzerland) , vol. 24, 2024

    2024

  7. [15]

    Efficient Multimodal Large Language Models: A Survey,

    Y. Jin, J. Li, Y. Liu, T. Gu, K. Wu, Z. Jiang, M. He, B. Zhao, X. Tan, Z. Gan, Y. Wang, C. Wang, and L. Ma, “Efficient Multimodal Large Language Models: A Survey,” ArXiv, vol. abs/2405.10739, 2024. [Online]. Available: https://api.semanticscholar.org/CorpusID: 269899856

    2024

  8. [16]

    Resource Allocation in Large Language Model Integrated 6G Vehicular Networks,

    C. Liu and J. Zhao, “Resource Allocation in Large Language Model Integrated 6G Vehicular Networks,” ArXiv, vol. abs/2403.19016, 2024. [Online]. Available: https://api.semanticscholar.org/CorpusID:268732601

    2024 arXiv

  9. [17]

    A Superalignment Framework in Autonomous Driving with Large Language Mod- els,

    X. Kong, T. Braunl, M. Fahmi, and Y. Wang, “A Superalignment Framework in Autonomous Driving with Large Language Mod- els,” in 2024 IEEE Intelligent Vehicles Symposium (IV) , 2024, pp. 1715–1720

    2024

  10. [18]

    A survey on multimodal large language models,

    S. Yin, C. Fu, S. Zhao, K. Li, X. Sun, T. Xu, and E. Chen, “A survey on multimodal large language models,” National Science Review , vol. 11, 2023

    2023

  11. [19]

    Imp: Highly Capable Large Multimodal Models for Mobile Devices,

    Z. Shao, Z. Yu, J. Yu, X. Ouyang, L. Zheng, Z. Gai, M. Wang, and J. Ding, “Imp: Highly Capable Large Multimodal Models for Mobile Devices,” ArXiv, vol. abs/2405.12107, 2024

    2024 arXiv

  12. [20]

    TinyGPT-V: Efficient Multimodal Large Language Model via Small Backbones,

    Z. Yuan, Z. Li, and L. Sun, “TinyGPT-V: Efficient Multimodal Large Language Model via Small Backbones,” ArXiv, vol. abs/2312.16862, 2023. [Online]. Available: https://api.semanticscholar.org/CorpusID:266572996

    2023 arXiv

  13. [21]

    TinyLLaVA: A Framework of Small-scale Large Multimodal Models,

    B. Zhou, Y. Hu, X. Weng, J. Jia, J. Luo, X. Liu, J. Wu, and L. Huang, “TinyLLaVA: A Framework of Small-scale Large Multimodal Models,” ArXiv, vol. abs/2402.14289, 2024. [Online]. Available: https://api.semanticscholar.org/CorpusID:267782659

    2024 arXiv

  14. [22]

    A Survey on Model Compression for Large Language Models,

    X. Zhu, J. Li, Y. Liu, C. Ma, and W. Wang, “A Survey on Model Compression for Large Language Models,” T ransactions of the Association for Computational Linguistics , vol. 12, pp. 1556–1577, 11

  15. [23]

    Large Language Models (LLMs) Inference Offloading and Resource Allocation in Cloud-Edge Computing: An Active Inference Approach,

    Y. He, J. Fang, F. R. Yu, and V . C. Leung, “Large Language Models (LLMs) Inference Offloading and Resource Allocation in Cloud-Edge Computing: An Active Inference Approach,” IEEE T ransactions on Mobile Computing, pp. 1–12, 2024

    2024

  16. [24]

    PerLLM: Personalized Inference Scheduling with Edge-Cloud Collaboration for Diverse LLM Services,

    Z. Yang, Y. Yang, C. Zhao, Q. Guo, W. He, and W. Ji, “PerLLM: Personalized Inference Scheduling with Edge-Cloud Collaboration for Diverse LLM Services,” ArXiv, vol. abs/2405.14636, 2024. [Online]. Available: https://api.semanticscholar.org/CorpusID:269982311

    2024 arXiv

  17. [25]

    On the Road to 6G: Visions, Requirements, Key Technologies, and Testbeds,

    C. Wang, X. You, X. Gao, X. Zhu, Z. sheng Li, C. Zhang, H. Wang, Y. Huang, Y. Chen, H. Haas, J. S. Thompson, E. G. Larsson, M. D. Renzo, W. Tong, P . Zhu, X. S. Shen, H. V . Poor, and L. H. Hanzo, “On the Road to 6G: Visions, Requirements, Key Technologies, and Testbeds,” IEEE...

    2023

  18. [26]

    Sixth Generation (6G) Wireless Networks: Vision, Research Activities, Challenges and Potential Solutions,

    M. H. Alsharif, A. H. Kelechi, M. A. M. Albreem, S. A. Chaudhry, M. S. Zia, and S. Kim, “Sixth Generation (6G) Wireless Networks: Vision, Research Activities, Challenges and Potential Solutions,” Symmetry, vol. 12, p. 676, 2020

    2020

  19. [27]

    A Uni- fied Multi-Task Semantic Communication System for Multimodal Data,

    G. Zhang, Q. Hu, Z. Qin, Y. Cai, G. Yu, and X. Tao, “A Uni- fied Multi-Task Semantic Communication System for Multimodal Data,” IEEE T ransactions on Communications, vol. 72, pp. 4101–4116, 2022

    2022

  20. [28]

    Deep Learning Enabled Semantic Communication Systems,

    H. Xie, Z. Qin, G. Y. Li, and B.-H. Juang, “Deep Learning Enabled Semantic Communication Systems,” IEEE T ransactions on Signal Processing, vol. 69, pp. 2663–2675, 2020

    2020

  21. [29]

    Joint Task and Data-Oriented Semantic Communications: A Deep Separate Source-Channel Coding Scheme,

    J. Huang, D. Li, C. X. Huang, X. Qin, and W. Zhang, “Joint Task and Data-Oriented Semantic Communications: A Deep Separate Source-Channel Coding Scheme,” IEEE Internet of Things Journal , vol. 11, pp. 2255–2272, 2023

    2023

  22. [30]

    Personalized Saliency in Task-Oriented Semantic Commu- nications: Image Transmission and Performance Analysis,

    J. Kang, H. Du, Z. Li, Z. Xiong, S. Ma, D. T. Niyato, and Y.-F. Li, “Personalized Saliency in Task-Oriented Semantic Commu- nications: Image Transmission and Performance Analysis,” IEEE Journal on Selected Areas in Communications , vol. 41, pp. 186–201, 2022

    2022

  23. [31]

    AI-Generated Incentive Mechanism and Full-Duplex Semantic Communications for Information Sharing,

    H. Du, J. Wang, D. T. Niyato, J. Kang, Z. Xiong, and D. I. Kim, “AI-Generated Incentive Mechanism and Full-Duplex Semantic Communications for Information Sharing,” IEEE Journal on Se- lected Areas in Communications , vol. 41, pp. 2981–2997, 2023

    2023

  24. [32]

    Semantic Communication Meets Edge Intelligence,

    W. Yang, Z. Q. Liew, W. Y. B. Lim, Z. Xiong, D. T. Niyato, X. Chi, X. Cao, and K. B. Letaief, “Semantic Communication Meets Edge Intelligence,” IEEE Wireless Communications , vol. 29, pp. 28–35, 2022

    2022

  25. [33]

    A Theory of Semantic Commu- nication,

    Y. Shao, Q. Cao, and D. G ¨und ¨uz, “A Theory of Semantic Commu- nication,” IEEE T ransactions on Mobile Computing, pp. 1–18, 2024

    2024

  26. [34]

    Edge- Cloud Collaborative Motion Planning for Autonomous Driving with Large Language Models,

    J. Chen, S. Dai, F. Chen, Z. Lv, and J. Tang, “Edge- Cloud Collaborative Motion Planning for Autonomous Driving with Large Language Models,” 2024. [Online]. Available: https://arxiv.org/abs/2408.09972

    2024 arXiv

  27. [35]

    Task-Oriented Multi-User Semantic Communications for VQA,

    H. Xie, Z. Qin, and G. Y. Li, “Task-Oriented Multi-User Semantic Communications for VQA,” IEEE Wireless Communications Letters , vol. 11, pp. 553–557, 2021

    2021

  28. [36]

    Learning Transferable Visual Models From Natu- ral Language Supervision,

    A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agar- wal, G. Sastry, A. Askell, P . Mishkin, J. Clark, G. Krueger, and I. Sutskever, “Learning Transferable Visual Models From Natu- ral Language Supervision,” in International Conference on Machine Learning, 2021

    2021

  29. [37]

    SQ- LLaVA: Self-Questioning for Large Vision-Language Assistant,

    G. Sun, C. Qin, J. Wang, Z. Chen, R. Xu, and Z. Tao, “SQ- LLaVA: Self-Questioning for Large Vision-Language Assistant,” ArXiv, vol. abs/2403.11299, 2024. [Online]. Available: https: //api.semanticscholar.org/CorpusID:268513172

    2024 arXiv

  30. [38]

    Vicuna: An Open-Source Chatbot Impressing GPT-4 with 90%* ChatGPT Quality,

    W.-L. Chiang, Z. Li, Z. Lin, Y. Sheng, Z. Wu, H. Zhang, L. Zheng, S. Zhuang, Y. Zhuang, J. E. Gonzalez, I. Stoica, and E. P . Xing, “Vicuna: An Open-Source Chatbot Impressing GPT-4 with 90%* ChatGPT Quality,” March 2023. [Online]. Available: https://lmsys.org/blog/2023-03-30-vicuna

    2023

  31. [39]

    Mistral 7B,

    A. Q. Jiang, A. Sablayrolles, A. Mensch, C. Bamford, D. S. Chaplot, D. de Las Casas, F. Bressand, G. Lengyel, G. Lample, L. Saulnier, L. R. Lavaud, M.-A. Lachaux, P . Stock, T. L. Scao, T. Lavril, T. Wang, T. Lacroix, and W. E. Sayed, “Mistral 7B,” ArXiv, vol. abs/2310.06825, ...

    arXiv

  32. [40]

    GloVe: Global Vec- tors for Word Representation,

    J. Pennington, R. Socher, and C. D. Manning, “GloVe: Global Vec- tors for Word Representation,” in Conference on Empirical Methods in Natural Language Processing , 2014

    2014

  33. [41]

    LLaMA: Open and Efficient Foundation Language Models,

    H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozi `ere, N. Goyal, E. Hambro, F. Azhar, A. Rodriguez, A. Joulin, E. Grave, and G. Lample, “LLaMA: Open and Efficient Foundation Language Models,” ArXiv, vol. abs/2302.13971, 2023. [Online]. Availab...

    2023 arXiv

  34. [42]

    LLaVA-UHD: an LMM Perceiving Any Aspect Ratio and High-Resolution Images,

    R. Xu, Y. Yao, Z. Guo, J. Cui, Z. Ni, C. Ge, T.-S. Chua, Z. Liu, M. Sun, and G. Huang, “LLaVA-UHD: an LMM Perceiving Any Aspect Ratio and High-Resolution Images,” ArXiv, vol. abs/2403.11703, 2024. [Online]. Available: https://api.semanticscholar.org/CorpusID:268531413

    2024 arXiv

  35. [43]

    Visual Instruction Tuning,

    H. Liu, C. Li, Q. Wu, and Y. J. Lee, “Visual Instruction Tuning,” ArXiv, vol. abs/2304.08485, 2023. [Online]. Available: https://api.semanticscholar.org/CorpusID:258179774

    2023 arXiv

  36. [44]

    MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models,

    D. Zhu, J. Chen, X. Shen, X. Li, and M. Elhoseiny, “MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models,” ArXiv, vol. abs/2304.10592, 2023. [Online]. Available: https://api.semanticscholar.org/CorpusID:258291930 © 2025 IEEE. All rights reserv...

    2023 arXiv

  37. [45]

    MobileVLM : A Fast, Strong and Open Vision Language Assistant for Mobile Devices,

    X. Chu, L. Qiao, X. Lin, S. Xu, Y. Yang, Y. Hu, F. Wei, X. Zhang, B. Zhang, X. Wei, and C. Shen, “MobileVLM : A Fast, Strong and Open Vision Language Assistant for Mobile Devices,” ArXiv, vol. abs/2312.16886, 2023. [Online]. Available: https://api.semanticscholar.org/CorpusID:...

    2023 arXiv

  38. [46]

    LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale,

    T. Dettmers, M. Lewis, Y. Belkada, and L. Zettlemoyer, “LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale,” ArXiv, vol. abs/2208.07339, 2022. [Online]. Available: https://api.semanticscholar.org/CorpusID:251564521

    2022 arXiv

  39. [47]

    PB- LLM: Partially Binarized Large Language Models,

    Y. Shang, Z. Yuan, Q. Wu, and Z. Dong, “PB- LLM: Partially Binarized Large Language Models,” ArXiv, vol. abs/2310.00034, 2023. [Online]. Available: https://api.semanticscholar.org/CorpusID:263333921

    2023 arXiv

  40. [48]

    LLaVA-PruMerge: Adaptive Token Reduction for Efficient Large Multimodal Models,

    Y. Shang, M. Cai, B. Xu, Y. J. Lee, and Y. Yan, “LLaVA-PruMerge: Adaptive Token Reduction for Efficient Large Multimodal Models,” ArXiv, vol. abs/2403.15388, 2024. [Online]. Available: https://api.semanticscholar.org/CorpusID:268667281

    2024

  41. [49]

    ConvLLaVA: Hierarchical Backbones as Visual Encoder for Large Multimodal Models,

    C. Ge, S. Cheng, Z. Wang, J. Yuan, Y. Gao, J. Song, S. Song, G. Huang, and B. Zheng, “ConvLLaVA: Hierarchical Backbones as Visual Encoder for Large Multimodal Models,” ArXiv, vol. abs/2405.15738, 2024. [Online]. Available: https: //api.semanticscholar.org/CorpusID:270045537

    2024 arXiv

  42. [50]

    A Convnet for the 2020s,

    Z. Liu, H. Mao, C.-Y. Wu, C. Feichtenhofer, T. Darrell, and S. Xie, “A Convnet for the 2020s,” in 2022 IEEE/CVF Conference on Com- puter Vision and Pattern Recognition (CVPR), 2022, pp. 11 966–11 976

    2022

  43. [51]

    Semantic Communications for Future Internet: Fundamentals, Applications, and Challenges,

    W. Yang, H. Du, Z. Q. Liew, W. Y. B. Lim, Z. Xiong, D. T. Niyato, X. Chi, X. S. Shen, and C. Miao, “Semantic Communications for Future Internet: Fundamentals, Applications, and Challenges,” IEEE Communications Surveys & T utorials, vol. 25, pp. 213–250, 2022

    2022

  44. [52]

    Generative AI-driven Semantic Communication Networks: Architecture, Technologies and Applications,

    C. Liang, H. Du, Y. Sun, D. T. Niyato, J. Kang, D. Zhao, and M. A. Imran, “Generative AI-driven Semantic Communication Networks: Architecture, Technologies and Applications,” IEEE T ransactions on Cognitive Communications and Networking, 2023

    2023

  45. [53]

    Semantic Communication With Memory,

    H. Xie, Z. Qin, and G. Y. Li, “Semantic Communication With Memory,” IEEE Journal on Selected Areas in Communications , vol. 41, pp. 2658–2669, 2023

    2023

  46. [54]

    The Cost of Cortical Computation,

    P . Lennie, “The Cost of Cortical Computation,” Current Biology , vol. 13, pp. 493–497, 2003

    2003

  47. [55]

    Saliency Guided DNL-Yolo for Optical Remote Sensing Images for Off-Shore Ship Detection,

    J. Guo, S. Wang, and Q. Xu, “Saliency Guided DNL-Yolo for Optical Remote Sensing Images for Off-Shore Ship Detection,” Applied Sciences, 2022

    2022

  48. [56]

    Utilising Visual Attention Cues for Vehicle Detection and Tracking,

    F. Hu, M. VenkateshG, N. E. O’Connor, A. F. Smeaton, and S. Little, “Utilising Visual Attention Cues for Vehicle Detection and Tracking,” 2020 25th International Conference on Pattern Recognition (ICPR), pp. 5535–5542, 2020

    2020

  49. [57]

    DeepFix: A Fully Convolutional Neural Network for Predicting Human Eye Fixations,

    S. S. S. Kruthiventi, K. Ayush, and R. V . Babu, “DeepFix: A Fully Convolutional Neural Network for Predicting Human Eye Fixations,” IEEE T ransactions on Image Processing, vol. 26, pp. 4446– 4456, 2015

    2015

  50. [58]

    DHSNet: Deep Hierarchical Saliency Network for Salient Object Detection,

    N. Liu and J. Han, “DHSNet: Deep Hierarchical Saliency Network for Salient Object Detection,” 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , pp. 678–686, 2016

    2016

  51. [59]

    SalGAN: Visual Saliency Prediction with Generative Adversarial Networks,

    J. Pan, C. Canton-Ferrer, K. McGuinness, N. E. O’Connor, J. Torres, E. Sayrol, and X. G. i Nieto, “SalGAN: Visual Saliency Prediction with Generative Adversarial Networks,” ArXiv, vol. abs/1701.01081, 2017. [Online]. Available: https: //api.semanticscholar.org/CorpusID:16173261

    2017 arXiv

  52. [60]

    SATSal: A Multi-level Self-Attention Based Architecture for Visual Saliency Prediction,

    M. Tliba, M. A. Kerkouri, B. Ghariba, A. Chetouani, A. Coltekin, M. Shehata, and A. Bruno, “SATSal: A Multi-level Self-Attention Based Architecture for Visual Saliency Prediction,” IEEE Access , vol. PP , pp. 1–1, 2022

    2022

  53. [61]

    Personalized Saliency and Its Prediction,

    Y. Xu, S. Gao, J. Wu, N. Li, and J. Yu, “Personalized Saliency and Its Prediction,” IEEE T ransactions on Pattern Analysis and Machine Intelligence, vol. 41, no. 12, pp. 2975–2989, 2019

    2019

  54. [62]

    Sigmoid Loss for Language Image Pre-Training,

    X. Zhai, B. Mustafa, A. Kolesnikov, and L. Beyer, “Sigmoid Loss for Language Image Pre-Training,” 2023

    2023

  55. [63]

    Math-LLaVA: Bootstrapping Mathematical Reasoning for Multimodal Large Language Models,

    W. Shi, Z. Hu, Y. Bin, J. Liu, Y. Yang, S.-K. Ng, L. Bing, and R. K. wei Lee, “Math-LLaVA: Bootstrapping Mathematical Reasoning for Multimodal Large Language Models,” ArXiv, vol. abs/2406.17294, 2024

    2024 arXiv

  56. [64]

    A Text Feature Based Automatic Keyword Ex- traction Method for Single Documents,

    R. Campos, V . Mangaravite, A. Pasquali, A. M. Jorge, C. Nunes, and A. Jatowt, “A Text Feature Based Automatic Keyword Ex- traction Method for Single Documents,” in Advances in Information Retrieval. Cham: Springer International Publishing, 2018, pp. 684– 691

    2018

  57. [65]

    YOLO by Ultralytics,

    G. Jocher, A. Chaurasia, and J. Qiu, “YOLO by Ultralytics,” https: //github.com/ultralytics/ultralytics, 2023, accessed: 2023-05-30

    2023

  58. [66]

    Exploring Attention-Aware Network Resource Allocation for Customized Metaverse Services,

    H. Du, J. Wang, D. Niyato, J. Kang, Z. Xiong, X. Shen, and D. I. Kim, “Exploring Attention-Aware Network Resource Allocation for Customized Metaverse Services,” IEEE Network, vol. 37, no. 6, pp. 166–175, 2023

    2023

  59. [67]

    Intrinsic and Extrinsic Effects on Image Memorability,

    Z. Bylinskii, P . Isola, C. Bainbridge, A. Torralba, and A. Oliva, “Intrinsic and Extrinsic Effects on Image Memorability,” Vision research, vol. 116, pp. 165–178, 2015

    2015

  60. [68]

    Learning to pre- dict where humans look,

    T. Judd, K. Ehinger, F. Durand, and A. Torralba, “Learning to pre- dict where humans look,” in2009 IEEE 12th International Conference on Computer Vision, 2009, pp. 2106–2113

    2009

  61. [69]

    LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Day,

    C. Li, C. Wong, S. Zhang, N. Usuyama, H. Liu, J. Yang, T. Naumann, H. Poon, and J. Gao, “LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Day,” ArXiv, vol. abs/2306.00890, 2023. [Online]. Available: https://api.semanticscholar.org/CorpusID:258999820

    2023 arXiv

  62. [70]

    [CLS] Attention is All You Need for Training-Free Visual Token Pruning: Make VLM Inference Faster,

    Q. Zhang, A. Cheng, M. Lu, Z. Zhuo, M. Wang, J. Cao, S. Guo, Q. She, and S. Zhang, “[CLS] Attention is All You Need for Training-Free Visual Token Pruning: Make VLM Inference Faster,” arXiv preprint arXiv:2412.01818 , 2024. Baoxia Du (Student Member, IEEE) is a PhD student in ...

    2024 arXiv

  63. [2024]

    Available: https://doi.org/10.1162/tacl a 00704

    [Online]. Available: https://doi.org/10.1162/tacl a 00704

    doi

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.