{"id":"a486bbf7-b613-460d-aaf9-45a544fc3929","arxiv_id":"2505.02413","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A task-oriented semantic communication framework for LLaVA-based vehicle assistants that crops image slices matched to the user's question and allocates transmission power by fused objective and subjective attention, improving VQA accuracy at low SNR.","lead":"The paper proposes a vehicle AI assistant that sends only the most relevant image features to a cloud-based vision-language model, prioritizing transmission power on the parts of the image the driver asks about. It reports higher question-answering accuracy at low signal-to-noise ratios and roughly half the computation of the standard high-resolution LLaVA pipeline.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Patch-level power allocation inherits a CNN-era mapping that is unvalidated for ViT feature tokens; if the mapping is weak, the low-SNR gains are not explained and may not generalize.","rationale":"The paper's headline claim is the low-SNR accuracy improvement from FA-SemCom. That improvement is meaningful only if the power allocation protects the semantic content the LLM actually uses. The reader's weakest assumption identifies precisely the unvalidated link: pixel-space saliency weights are mapped to ViT output tokens that, after global self-attention and the projection layer, are not guaranteed to carry localized information. I agree with this assessment and sharpen it by noting that Algorithm 2 also leaves the slice-to-heatmap mapping unspecified, which makes the method under-defined. The proposed concrete test can settle the concern: an oracle sensitivity map directly measures which patches matter, and the reversed allocation controls for the possibility that any unequal allocation helps. I do not see a reason to change the reader's CONDITIONAL verdict; the SM efficiency results are plausible, the experimental evidence is consistent with the stated pipeline, and the missing code/data would matter more if the technical assumption were confirmed. The test above should be a condition for acceptance.","tokens_in":25398,"tokens_out":10334,"duration_ms":124603,"concrete_test":"Rerun the Fig. 6 experiment at SNR=10 dB for Vicuna-7B with three ablations: (1) an oracle power allocation that sets each patch's power proportional to the measured drop in answer accuracy when that patch's token is corrupted (leave-one-patch-out noise injection); (2) a reversed allocation that gives high power to low-attention patches; (3) the original FA-SemCom. Compute the Spearman correlation between W_p(p_i) and the oracle importance. If FA-SemCom is not at least as accurate as the oracle, or if the reversed allocation also improves over AVG, the attention-to-token mapping assumed by Algorithm 2 is not the source of the gains. Also document the exact procedure for assigning W_p to the patches of X_s, such as heatmap rescaling versus recomputation on the slice.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central low-SNR result rests on Eq. 15/Algorithm 2, which allocates more transmit power to the ViT output token at patch position i when the pixel-space fused attention W_p(p_i) is high. This assumes that token i's embedding is the right unit to protect for image region i. LLaVA's CLIP-ViT-L/14 runs 24 layers of global self-attention, so each output token is a mixture of all input patches; the subsequent projection W in Eq. 1 linearly mixes the token dimensions before the LLM, so the noise that affects the answer lives in the LLM embedding space, not in the token space. The scheme is carried over from [30], a CNN-based semantic communication system, without any validation for ViT/LMM. Additionally, Algorithm 2 does not specify how the fused heatmap computed on X_v is mapped to the patch grid of the rescaled slice X_s, so the per-patch weights for the slice are undefined. If the token-to-region mapping is weak, the power may protect the wrong semantic content; the reported gains at 10-12 dB could be dataset-specific and could vanish under different backbones, resolutions, or question distributions. No diagnostic, such as comparison against an oracle per-patch sensitivity map, is provided.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper proposes a task-oriented semantic communication framework for LMM-based vehicle networks. The system splits LLaVA's computation: the CLIP-ViT visual encoder runs on the vehicle, and the LLM runs on a cloud server, with encoded image features transmitted over a wireless channel. To reduce computational cost, the authors introduce a Semantic Matching (SM) module that extracts keywords from the user question, detects objects with YOLO, and crops the most relevant image slice for transmission, yielding a model called LLaVA-SM. To improve robustness under poor channel conditions, they further propose FA-SemCom, which fuses an objective saliency heatmap with a subjective attention heatmap derived from the user's question, quantizes the fused attention into weights, and allocates transmission power to image patch features according to Eq. (15). Experiments on a self-constructed traffic VQA dataset (41 images, 172 questions) show that LLaVA-SM reduces visual tokens from 2880 to 1152 with at most 0.6% accuracy loss, and that FA-SemCom improves answer accuracy by 13.4% at 12 dB and 33.1% at 10 dB over average-power transmission.","tokens_in":25551,"tokens_out":6524,"duration_ms":71449,"significance":"If the claims hold, the paper addresses a practically important problem: deploying large multimodal assistants in vehicles with limited onboard compute and unreliable wireless links. The idea of using user attention to guide both image cropping and transmission power is intuitively appealing and goes beyond prior CNN-based semantic communication schemes by targeting LMMs. The efficiency result for LLaVA-SM is plausible and well presented with token and FLOP counts. The paper is clearly written and situates itself well in the related literature on semantic communication, visual saliency, and efficient LMMs. However, the experimental evidence is thin (41 images, 172 hand-written questions, no released code or dataset, no confidence intervals), and the core power-allocation mechanism for ViT-based encoders is not validated. The reported low-SNR gains rest on an assumption about the correspondence between pixel-space attention and ViT output tokens that is not established in the paper.","major_comments":[{"comment":"The power-allocation scheme assumes a direct, monotone mapping from the fused pixel-space attention weight W_p(p_i) of patch p_i to the importance of the i-th output token of the CLIP-ViT encoder, and then to the individual dimensions of Z_v. This assumption is not justified for LLaVA: the ViT applies global self-attention across all patches, so each output token is a mixture of all input patches, and the projection W in Eq. (1) linearly mixes the token dimensions before the LLM. The surviving noise therefore acts on the LLM embedding space, not directly on the token space. The paper provides no diagnostic demonstrating that the allocated power protects the semantically relevant regions, for example by comparing against an oracle per-patch sensitivity map or by ablating the mapping. Since the central claim of 13.4% and 33.1% accuracy improvements at low SNR rests on this mechanism, this issue is load-bearing.","section":"Section 5.3, Eq. (15) and Algorithm 2"},{"comment":"The fused attention heatmap H_a is computed on the full image X_v, but the transmitted features come from both the resized base image X_b and the rescaled slice X_s. The patch grids of X_b and X_s do not coincide with the patch grid of X_v. The paper does not specify how W_p(p_i) is defined for the patches of X_s, nor how the importance weights computed on X_v are mapped to the slice. Without this coordinate transformation, Algorithm 2 as written is ambiguous and the implementation cannot be reproduced from the description.","section":"Algorithm 2, steps 9-12"},{"comment":"The evaluation uses a self-constructed dataset of 41 images and 172 questions, with manual accuracy scoring, and no confidence intervals or statistical tests are reported. The dataset and code are not released. Given the small sample size, the headline improvements of 13.4% (12 dB) and 33.1% (10 dB) may not be stable, and the per-question or per-image variance is not reported. The paper should report variance across images or bootstrapped confidence intervals, and ideally release the dataset and code to allow independent verification.","section":"Section 6.1 and Section 6.2"},{"comment":"The parameter beta is tuned per SNR (beta=4 at 10 dB in Fig. 7, beta=1 at 12 dB in Fig. 6), but the paper does not provide a mechanism for selecting beta online based on channel conditions. Without such a procedure, the reported gains are for a tuned parameter, not for an adaptive policy. Additionally, the abstract and conclusion report improvements of 13.4% and 33.1%, but the underlying numbers in Section 6.2 (e.g., 0.436 to 0.767) suggest these are percentage-point increases rather than relative percentage increases; the terminology should be corrected for precision.","section":"Section 6.2, Fig. 8 and Eq. (15)"}],"minor_comments":[{"comment":"The text states that Fig. 6(a) uses Vicuna-7B and Fig. 6(b) uses Mistral-7B, but the figure caption says the opposite. Please correct this inconsistency.","section":"Section 6.2, Fig. 6"},{"comment":"The phrases 'improved by 13.4%' and '33.1%' are ambiguous; please specify whether these are relative improvements or absolute percentage-point improvements.","section":"Abstract and Conclusion"},{"comment":"Step 6 says 'Calculate the cosine similarity between Kq and ci', but the cosine similarity is computed between the vector representations v_Kq and v_ci in Eqs. (4)-(5); the wording should be updated accordingly.","section":"Algorithm 1, line 6"},{"comment":"The text mentions that a 672x1008 image produces 3456 visual tokens for LLaVA-1.6, while Table 3 lists 2880 (576x5) tokens for the configuration used in the experiments; please clarify the resolution and slicing configuration used in the evaluation.","section":"Section 4.1, Table 3"},{"comment":"The statement that YOLOv8-n is 'pre-trained on a dataset of 166 training images and 40 validation images' is surprising for an object detector; please clarify whether this refers to fine-tuning on a custom dataset rather than the pre-training corpus.","section":"Section 6.1"},{"comment":"The SM module is described as selecting a 'target area' for image slicing, but Algorithm 1 returns only one slice X_s. If multiple target boxes are detected, the paper should specify how they are combined or whether multiple slices are generated.","section":"Section 4.2"}],"recommendation":"major_revision","confidential_remarks":"The manuscript header states that it has been accepted for publication in IEEE Transactions on Mobile Computing. If this version is being considered by another venue, the editorial office should verify the dual-submission or prior-publication status. Scientifically, the paper's core idea is interesting, but the experimental evidence and the ViT-specific power-allocation validation are not yet at the level of the claims; a major revision addressing the mapping issue and evaluation robustness would be appropriate."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe thing to know: this paper's real contribution is the Semantic Matching (SM) slice selector—YAKE keywords from the user's question, matched against YOLO object labels via GloVe, to pick which high-resolution slice to feed LLaVA. That's a clean, practical idea, and the efficiency numbers are believable: 2880→1152 visual tokens with 0.6% accuracy loss on Vicuna, 0% on Mistral. If I were deploying LLaVA on a vehicle, I'd look at this.\n\nWhat it does well: the framework is coherent, the related work is decent, and the SM module is genuinely new in the LMM-semantic-communication space. The authors also admit LLaVA wasn't fine-tuned on traffic data (footnote), which is honest.\n\nNow the soft spots. The low-SNR accuracy gains rest on FA-SemCom, which maps a pixel-space attention heatmap to ViT output patch tokens and allocates power per token (Eq. 15, Algorithm 2). The stress-test concern is real: CLIP-ViT-L/14 runs 24 layers of global self-attention, so token i is a mixture of all input patches, and the projection W then mixes dimensions again before the LLM. Protecting token i according to patch i's saliency may be protecting the wrong thing. There is no diagnostic—no oracle sensitivity map, no ablation against, say, random patch power—to show the mechanism actually does what the narrative claims. Also Algorithm 2 doesn't specify how the fused heatmap on the original image is mapped onto the rescaled slice X_s, so for the slice tokens the per-patch weights are undefined. That's a concrete technical gap, not a nitpick.\n\nThe evidence base is thin: 41 images, 172 hand-written questions, no released data or code, no confidence intervals, manual accuracy scoring. Beta is tuned empirically on the same data shown in Fig. 8, so the 13.4% and 33.1% numbers are probably optimistic. None of this refutes the efficiency claim, which stands on Table 3, but it does mean the power allocation claim is not yet established.\n\nWho should read it: people working on semantic communication for multimodal models and edge offloading of LMMs. They'll get a useful idea (SM slicing) and a cautionary example of how hard it is to carry a CNN-era power allocation scheme into a ViT/LMM setting.\n\nI'd send this to peer review without hesitation. It deserves a serious referee. In fact, a good referee should demand: release the dataset and code, show the mapping from heatmap to patch tokens (or fix Algorithm 2), and run an ablation that tests whether the gains come from protecting salient patches or just from the slice selection itself.\n\nBottom line: worthwhile, needs revision on the core mechanism.","headline":"The SM-based image slicing for LLaVA is a genuinely useful efficiency idea; the attention-based power allocation is the weak joint and needs much more validation before the low-SNR gains can be trusted.","tokens_in":26205,"tokens_out":3945,"would_cite":true,"duration_ms":46721,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that task-oriented image cropping and attention-weighted power allocation let a cloud-run LLaVA answer driver questions accurately over low-SNR channels, raising VQA accuracy by 13.4% at 12 dB and 33.1% at 10 dB.","keywords":["semantic communication","large multimodal models","LLaVA","visual question answering","resource allocation","user attention","vehicle networks","task-oriented communication"],"falsifier":"Run the same low-SNR VQA comparison but scramble the patch importance weights before computing the power allocation: if accuracy stays at the FA-SemCom level, the attention-to-feature mapping is not doing the work, and if it drops to the equal-power level, the mapping is essential.","tokens_in":25084,"feed_emoji":"🚗","tokens_out":9388,"duration_ms":101978,"temperature":0.7,"pith_summary":"This paper claims that a vehicle can offload everything except a small image encoder to a cloud server and still get accurate answers from a Large Language and Vision Assistant (LLaVA) over a noisy wireless link, as long as the communication is task-oriented. The first intervention is a Semantic Matching module that extracts keywords from the driver's question, detects candidate objects, matches them with word-vector similarity, and crops the image to the region the question is about; this drops the number of image tokens the language model must process from 2880 to 1152 with only 0.6% accuracy loss on Vicuna-7B. The second is a fusion-attention power allocation scheme that combines generic visual saliency with the user's subjective focus, then gives more transmission power to the image patches that carry the answer. On a custom traffic visual-question-answering dataset, the paper reports accuracy gains of 13.4% at 12 dB SNR and 33.1% at 10 dB SNR over equal-power transmission. If true, this would make LMM-based driving assistants practical in exactly the channel conditions where current systems fail.","feed_headline":"Attention-guided power lifts low-SNR VQA accuracy by 33%","feed_subtitle":"Cropping to the user's question and boosting power to key patches keep LLaVA assistants accurate on noisy channels.","key_machinery":"The load-bearing mechanism is the fused attention heatmap and the power-allocation law derived from it. Objective attention $H_{\\mathrm{obj}}$ comes from a lightweight saliency network; subjective attention $H_{\\mathrm{sub}}$ places high weight on the bounding box found by matching the user's keywords to detected objects; and the two are blended as $H_a = \\alpha H_{\\mathrm{obj}} + (1-\\alpha)H_{\\mathrm{sub}}$ with $\\alpha = 0.5$. Each image patch then receives importance $W_p(p_i) = H_a(p_i)$, is quantized into $L=5$ levels via $W_L(p_i)$, and is assigned transmit power $P(p_i) = (W_L(p_i))^{\\beta}/\\sum_j (W_L(p_j))^{\\beta} \\cdot P_{\\mathrm{total}}$. This law is what converts which parts of the image this user cares about into a physical resource allocation, sending the patches that answer the question at higher power and lower error rate.","core_discovery":"The central claim is that both parts of the framework, task-oriented cropping and attention-weighted power allocation, are needed, and together they let a cloud-run LLaVA answer visual questions about traffic scenes despite poor channels. The Semantic Matching module uses YAKE to extract keyword phrases from the question, YOLO to detect objects, GloVe cosine similarity to pick the best matching object, and a flexible cropping function to produce a slice that the CLIP-ViT encoder converts into features alongside the base image. The FA-SemCom module then computes a fused attention heatmap $H_a = \\alpha H_{\\mathrm{obj}} + (1-\\alpha) H_{\\mathrm{sub}}$ with $\\alpha = 0.5$, where $H_{\\mathrm{obj}}$ is a saliency prediction and $H_{\\mathrm{sub}}$ highlights the matched bounding box, quantizes each patch's importance to five levels, and allocates power $P(p_i) = (W_L(p_i))^{\\beta}/\\sum_{j}(W_L(p_j))^{\\beta} \\cdot P_{\\mathrm{total}}$ with $\\beta$ tunable. In simulations over a Fisher-Snedecor F fading channel this raises answer accuracy by 13.4% at 12 dB SNR and 33.1% at 10 dB SNR relative to equal power, with per-patch bit-error plots showing the target region, such as a license plate, transmitted with fewer errors.","pith_inferences":["The paper's accuracy curves show that the optimal $\\beta$ shifts with channel conditions, yet $\\beta$ is treated as a constant; an adaptive rule that raises $\\beta$ when SNR drops is a natural next step that the paper does not work out.","If the per-patch attention weights are scrambled before power allocation and the low-SNR accuracy does not fall, then the claimed link between patch importance and feature-dimension protection is not causal; this experiment would settle the mechanism.","The reported numbers probably underestimate what a fine-tuned LMM would achieve, since the paper itself notes that LLaVA was not trained on a specialized traffic dataset, and a domain-adapted model would correct more of the noisy inputs."],"forward_implications":["Vehicles can run only the visual encoder locally and offload the LLM to a server, because the transmitted semantic features, not the raw image, carry the task-relevant content.","Cropping to the matched object cuts visual tokens from 2880 to 1152 and FLOPs from 43.58 T to 17.43 T, cutting average response time by roughly 27% with at most a 0.6% accuracy hit on Vicuna-7B.","At low SNR, attention-weighted power allocation beats equal-power transmission by 13.4% at 12 dB and 33.1% at 10 dB, so accurate LMM-based driving assistants become feasible in harsh channels that break naive transmission.","Because the method only assumes patch-based image encoding, it transfers to other LMMs with CLIP-ViT or SigLIP encoders, and the paper argues it can be retrained for other domains such as agricultural inspection."],"supporting_citations":[{"why":"Supplies the LLaVA-1.5/1.6 high-resolution image-slicing baseline whose 2880-token output the proposed Semantic Matching method compresses.","marker":"[3]"},{"why":"Supplies the per-patch power allocation rule and the Fisher-Snedecor F channel model used in Algorithm 2 and the simulations.","marker":"[30]"},{"why":"Provides the CLIP-ViT visual encoder whose patch-based features are the semantic information transmitted from the vehicle to the server.","marker":"[36]"},{"why":"Supplies the GloVe word-vector embedding used to compute cosine similarity between user keywords and detected object categories.","marker":"[40]"},{"why":"Introduces the LLaVA architecture of a CLIP-ViT encoder, a projection matrix, and an LLM, on which the whole framework is built.","marker":"[43]"},{"why":"Supplies the objective saliency prediction network and the notion of personalized saliency that motivates the subjective attention component.","marker":"[61]"},{"why":"Supplies the YAKE unsupervised keyword extraction algorithm that turns the user's question into keyword phrases for semantic matching.","marker":"[64]"},{"why":"Supplies the lightweight YOLOv8-n object detector used to locate the objects whose bounding boxes define the subjective attention heatmap.","marker":"[65]"},{"why":"Contributes highway scenes used to construct the traffic VQA dataset on which all accuracy numbers are measured.","marker":"[67]"}],"fun_headline_variants":["Attention-aware power boosts LMM VQA accuracy by 33% on noisy channels","Task-driven cropping and power yield 33% better VQA at low SNR","Semantic slicing + power control lift LLaVA answers 33%","LLaVA + attention power allocation answers traffic VQA 33% better","Noisy channels no match: LLaVA VQA accuracy up 33%"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The gain rests on the assumption that boosting transmit power for a user-relevant image patch actually protects the part of the encoded feature vector the LLM will read, even though the visual encoder's global self-attention mixes information across all patches; the paper does not test that correspondence.","fun_headline_variants_meta":{"raw":{"variants":["Attention-aware power boosts LMM VQA accuracy by 33% on noisy channels","Task-driven cropping and power yield 33% better VQA at low SNR","Semantic slicing + power control lift LLaVA answers 33%","LLaVA + attention power allocation answers traffic VQA 33% better","Noisy channels no match: LLaVA VQA accuracy up 33%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000762,"raw_usage":{"total_tokens":3446,"prompt_tokens":1070,"completion_tokens":2376,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":686,"completion_tokens_details":{"reasoning_tokens":2284}},"tokens_in":686,"tokens_out":2376,"duration_ms":18967,"temperature":1.0,"reasoning_tokens":2284,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T00:53:00.191140+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same low-SNR VQA comparison but scramble the patch importance weights before computing the power allocation: if accuracy stays at the FA-SemCom level, the attention-to-feature mapping is not doing the work, and if it drops to the equal-power level, the mapping is essential.","supporting_citations":[{"cited_title":"Personalized Saliency in Task-Oriented Semantic Commu- nications: Image Transmission and Performance Analysis,","cited_arxiv_id":null,"evidence_quote":"Supplies the per-patch power allocation rule and the Fisher-Snedecor F channel model used in Algorithm 2 and the simulations."},{"cited_title":"Learning Transferable Visual Models From Natu- ral Language Supervision,","cited_arxiv_id":null,"evidence_quote":"Provides the CLIP-ViT visual encoder whose patch-based features are the semantic information transmitted from the vehicle to the server."},{"cited_title":"GloVe: Global Vec- tors for Word Representation,","cited_arxiv_id":null,"evidence_quote":"Supplies the GloVe word-vector embedding used to compute cosine similarity between user keywords and detected object categories."},{"cited_title":"Personalized Saliency and Its Prediction,","cited_arxiv_id":null,"evidence_quote":"Supplies the objective saliency prediction network and the notion of personalized saliency that motivates the subjective attention component."},{"cited_title":"A Text Feature Based Automatic Keyword Ex- traction Method for Single Documents,","cited_arxiv_id":null,"evidence_quote":"Supplies the YAKE unsupervised keyword extraction algorithm that turns the user's question into keyword phrases for semantic matching."},{"cited_title":"YOLO by Ultralytics,","cited_arxiv_id":null,"evidence_quote":"Supplies the lightweight YOLOv8-n object detector used to locate the objects whose bounding boxes define the subjective attention heatmap."},{"cited_title":"Intrinsic and Extrinsic Effects on Image Memorability,","cited_arxiv_id":null,"evidence_quote":"Contributes highway scenes used to construct the traffic VQA dataset on which all accuracy numbers are measured."}],"review_version":1}