{"id":"59b9a7de-d37a-4135-900c-dcdb23bf7a73","arxiv_id":"2507.21454","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"high","formal_verification":"none","parameter_count":4,"one_line_summary":"Transmitting five fine-tuned token embeddings from a multimodal LLM over a noisy wireless channel is claimed to achieve higher visual-question-answering accuracy than raw-image transmission while cutting overhead to under 1% of the original image size on CLEVR and GQA.","lead":"This paper builds a wireless communication system where two AI agents exchange 'machine tokens', compressed internal signals from a large language model, instead of human language or images. The authors report that a few tokens can carry enough information to answer visual questions, with higher accuracy than transmitting the original image and far less data.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The accuracy advantage over Benchmark 1 is not established because the baseline is not fine-tuned; the overhead metric is also incomparable, but the baseline confound alone undermines the central claim.","rationale":"The reader's weakest assumption about the compression-ratio comparison is valid and independently important: the paper compares real-valued analog dimensions to JPEG bytes without specifying a channel-capacity or modulation mapping, and the inconsistent JTCC output dimensions (256 vs. 512) make the 0.81% figure impossible to verify. However, I regard the baseline fine-tuning confound as even more load-bearing, because it directly undermines the central empirical claim that machine-language tokens achieve higher downstream accuracy than raw-image transmission. The reader did mention this confound in the rationale, but selected the overhead commensurability issue as the weakest assumption. I therefore partially agree with the reader, and my stress-test reinforces the REJECT verdict rather than changing it. The paper's underlying idea may be plausible, but the experiments as presented cannot separate the effect of the proposed communication scheme from the effect of task-specific fine-tuning; this is a correctness risk, not merely a presentation issue.","tokens_in":9187,"tokens_out":4696,"duration_ms":60942,"concrete_test":"Reproduce Fig. 6 with Benchmark 1 replaced by a LoRA-fine-tuned LLaVA-V1.5-7B trained on the same CLEVR/GQA training split and answering directly from raw images. If this fine-tuned raw-image baseline reaches or exceeds the proposed method at 15 dB SNR, then the claimed accuracy advantage of machine-language tokens over raw-image transmission is not supported. As a secondary check, report the raw numbers behind Fig. 5, including the JTCC output dimension (256 vs. 512) and the explicit conversion from analog channel uses to bits.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Most load-bearing flaw: the claimed accuracy gain over raw-image transmission is confounded by task-specific fine-tuning. In Sec. IV.B, Benchmark 1 is 'transmitting raw images and employing the original LLaVA-V1.5-7B model to answer the questions' — i.e., zero-shot, pre-trained LLaVA. The proposed method, by contrast, is fine-tuned end-to-end on the CLEVR/GQA training sets with LoRA adapters in the transmitter and receiver (Secs. III.C, IV.A). Fig. 6 therefore compares a task-tuned machine-token system against an untuned raw-image system; the high-SNR accuracy gap can be explained entirely by parameter adaptation rather than by the machine-language-token representation or over-the-air scheme. No ablation supplies a raw-image baseline with identical fine-tuning. Additionally, the overhead claim is itself unverifiable: Fig. 5 compares analog real-valued dimensions to JPEG bytes without a capacity/bits mapping, and the JTCC output dimension is 256 in Sec. IV.A but 512 in Sec. IV.C. These issues are independent; the baseline confound alone is sufficient to reject the headline accuracy claim.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a task-oriented agent communication system in which a sensor agent uses a fine-tuned multimodal LLM to encode a vague task description plus raw image into K 'machine language tokens' taken from the final transformer block, compresses them with a learned JTCC module, and transmits them over an analog MIMO-OFDM channel. The receiver concatenates the received tokens with a detailed task description and feeds them to a second LLM that answers a visual question. On CLEVR and GQA, the authors report compression ratios as low as 0.81% of the average JPEG size and accuracy above all four benchmarks at high SNR. The paper claims that this learned machine-token representation is more efficient and robust than raw-image, DeepJSCC, or natural-language transmission.","tokens_in":9450,"tokens_out":8758,"duration_ms":101555,"significance":"The direction is timely: moving from human-oriented representations to learned, task-specific latent tokens for agent-agent communication is a natural extension of semantic communication. The combination of a fixed pretrained backbone with LoRA adapters and a learned joint source-channel coder is plausible and could be useful if properly validated. Credit is due for using standard public VQA datasets and for stating the straight-through gradient issue explicitly. However, the evidence presented does not yet establish the headline claims: the main accuracy comparison is confounded by unequal fine-tuning, the overhead comparison uses incomparable units, and the JTCC dimension is inconsistent across sections. The paper's contribution remains a proposal with promising but unverified results.","major_comments":[{"comment":"Benchmark 1 is defined in Sec. IV.B as transmitting raw images and using the original LLaVA-V1.5-7B model to answer questions, i.e., a zero-shot pretrained model. The proposed method, by contrast, is fine-tuned end-to-end on the training split with LoRA on the transmitter and receiver (Secs. III.C and IV.A). Consequently, the high-SNR accuracy advantage shown in Fig. 6 conflates the effect of the machine-token representation with the effect of task-specific fine-tuning; Benchmark 2, which also feeds reconstructed images to the original LLaVA, has the same limitation. Please add a raw-image baseline with identical LoRA fine-tuning (at least on the receiver side, and ideally with the same end-to-end training procedure but without the token bottleneck) and compare it with the proposed method across the same SNR range.","section":"IV.B, Fig. 6"},{"comment":"The compression-ratio comparison in Fig. 5 is not unit-consistent. The proposed method transmits K x L_t real-valued analog channel dimensions, while the denominator is the byte size of a JPEG image. A real-valued symbol transmitted over a noisy channel does not carry a fixed number of bits; the amount of information it can convey depends on SNR, bandwidth, modulation, and the rate-distortion tradeoff. Without a bits-per-channel-use mapping or an equivalent-rate calculation, the headline overhead percentage (0.81%) is not a meaningful comparison to JPEG bytes. Please report the number of channel uses, the SNR operating point, and an achievable-rate or rate-distortion-based comparison, or compare at a fixed bit rate with matched image compression.","section":"Fig. 5, Sec. IV.C"},{"comment":"Sec. IV.A states that JTCC compresses token embeddings from 4096 to 256 dimensions, whereas Sec. IV.C states that 'we reduce the dimensionality to 512' and uses this value to compute the proposed method's overhead in Fig. 5. Since the compression ratio scales linearly with the JTCC output dimension, this inconsistency changes the headline overhead by a factor of two. Please correct the discrepancy and recompute all overhead numbers with the actual deployed dimension.","section":"IV.A vs IV.C"},{"comment":"The end-to-end training relies on a straight-through gradient approximation in which transmitter-side gradients are replaced by receiver-side gradients computed on the received token embeddings. This is a nontrivial training-signal assumption, and the manuscript provides no validation that the approximation yields an unbiased or even useful gradient direction. Please provide an ablation or an analysis showing that this approximation converges and that the learned transmitter is genuinely optimized for the noisy channel, rather than relying on the receiver doing most of the task adaptation.","section":"III.C"}],"minor_comments":[{"comment":"Equation (2) defines e_d = LLM_v(I_v), but the surrounding text says e_d is the detailed task embedding obtained by tokenizing and embedding I_d; please make the notation consistent and clarify what LLM_v outputs.","section":"II.B, Eq. (2)"},{"comment":"Training hyperparameters (LoRA rank and target modules beyond K/V, learning rate, number of steps, batch size, channel and OFDM parameters) are not reported, which prevents reproduction.","section":"IV.A"},{"comment":"The threshold claim that five tokens are sufficient is inferred from the same experiment in which K=5 was already fixed as the operating point in Sec. IV.A; if K was selected by inspecting Fig. 7, describe the selection procedure or validate on a held-out split to avoid circularity.","section":"Fig. 7"},{"comment":"The JPEG size used as the denominator is described only as 'average storage size of JPG images'; the JPEG quality setting and the number of images used for the average are not stated, and no confidence intervals or error bars are provided for any of the compression-ratio bars.","section":"Fig. 5"},{"comment":"The overhead estimate for Benchmark 3 is based on the number of characters in a textual description with UTF-8 encoding, but the text does not specify how the description is generated or how variable-length UTF-8 encoding is counted; please specify.","section":"IV.C, Benchmark 3"},{"comment":"Accuracy curves are reported as point estimates with no error bars or multiple-seed trials, so it is unclear whether the differences between the proposed method, Benchmark 3, and Benchmark 4 are statistically significant.","section":"Fig. 6"},{"comment":"In the version provided, the internal labels of Fig. 2 appear as illegible glyphs; if this is not a rendering artifact, the figure needs to be redrawn so that the architecture is readable.","section":"Fig. 2"},{"comment":"The claim that GQA is used with 'the same sizes' as CLEVR needs clarification, since the standard GQA split is fixed; please state which images and questions were used for training and testing.","section":"IV.A"}],"recommendation":"major_revision","confidential_remarks":"The paper is within the journal's scope, but the experimental validation is currently too weak for acceptance. The two decisive issues are the un-matched baseline in Fig. 6 and the unit-inconsistent overhead metric in Fig. 5; both can be addressed with additional experiments and a corrected rate calculation. The JTCC dimension inconsistency (256 vs 512) must be resolved before any quantitative overhead claim is taken seriously. No code or data release is mentioned, which further hinders verification. If the authors cannot provide a fine-tuned raw-image baseline, I would move to rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague—here's my read of arXiv:2507.21454.\n\nThe core idea is genuinely new and worth taking seriously. Instead of transmitting natural-language output from an LLM, the authors fine-tune a multimodal LLM to emit a small number of token embeddings (K=5) that serve as a task-oriented 'machine language,' compress those with a learned JTCC module, and send them over a noisy analog OFDM-like channel. That framing—treating the last-layer token embeddings as a communication code, rather than as a precursor to text—is a real departure from the DeepJSCC and LLM-semantic-communication papers they cite. The system design is coherent, the training scheme (straight-through gradient approximation through the noisy channel) is sensible, and the writing is clear.\n\nThe problems are in the evidence, and they are load-bearing. The accuracy claim is confounded. Benchmark 1 is 'transmit raw image and use the original LLaVA-V1.5-7B'—i.e., a zero-shot, not-fine-tuned baseline. The proposed method is fine-tuned end-to-end on the same task with LoRA. At high SNR, the gap between the two can be explained entirely by task-specific fine-tuning. There is no ablation that fine-tunes a raw-image baseline. So the headline 'machine language tokens beat raw image' is not established.\n\nThe overhead metric is also incomparable. Fig. 5 compares the number of real-valued analog channel dimensions (K×Lt) to the byte size of a JPEG image, with no mapping through channel capacity, modulation, or SNR. The claim '0.81% of JPG size' is not a meaningful bits-over-the-air number as stated. On top of that, the JTCC output dimension is 256 in Sec. IV.A and 512 in Sec. IV.C; that internal inconsistency needs fixing.\n\nThe K=5 threshold result (Fig. 7) is post hoc: they pick K after seeing the same experiment. That's minor relative to the baseline confound, but worth noting.\n\nSo the central quantitative claims don't hold as written. That said, this is not a paper with a broken idea or incoherent reasoning. The idea is plausible, the system is well specified, and the flaws are fixable with a properly fine-tuned baseline and a bits-per-channel-use overhead analysis. I'd send it to peer review, not desk reject, because the novelty is real and the empirical claims are worth testing properly.\n\nFor the reading group: maybe worth a look as an example of a promising paradigm with weak evaluation, but not urgent. I would not cite the quantitative results; I might cite the formulation if I were working on LLM-based task-oriented communication.\n\nNet: deserves a serious referee, but needs major revision.","headline":"Genuinely new idea—transmitting fine-tuned LLM token embeddings as a communication code—but the accuracy and overhead claims are not established because the baseline is not fine-tuned and the overhead metric is unit-inconsistent.","tokens_in":9961,"tokens_out":2497,"would_cite":false,"duration_ms":27218,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that five learned machine-language tokens, sent over a noisy analog channel, outperform raw-image transmission on visual question answering while using 0.81% of the JPEG overhead.","keywords":["semantic communication","task-oriented communication","machine language tokens","large language models","joint source-channel coding","analog transmission","visual question answering","multi-modal LLM"],"falsifier":"Compare the proposed scheme to a raw-JPEG transmission under an equal end-to-end bit budget by converting each analog channel use to bits at the reported SNR (e.g., using a capacity formula for the OFAT/MIMO channel); if the token scheme needs a comparable or larger number of bits to reach the same accuracy, the overhead reduction is falsified.","tokens_in":9006,"feed_emoji":"🤖","tokens_out":8990,"duration_ms":93075,"temperature":0.7,"pith_summary":"This paper proposes that two agents can communicate for a downstream task by transmitting a small set of 'machine language tokens'—the last K embedding vectors produced by a fine-tuned multimodal LLM—instead of sending the raw image or a natural-language description. The central claim is that these learned tokens, passed over a noisy analog MIMO channel and compressed by a joint token-and-channel coding (JTCC) module, carry enough task-relevant information to match or beat a raw-image baseline in visual question answering on CLEVR and GQA, while using about 0.81% of the original JPG's transmission budget. The authors argue that natural language is redundant and ambiguous for machines, and that task-oriented token embeddings can be learned end-to-end with simulated channel noise so the representation is simultaneously compact and noise-resistant. If true, this points to a communication paradigm where agents exchange abstract learned semantics rather than human-interpretable data.","feed_headline":"Five tokens at 0.81% image size beat raw-image accuracy","feed_subtitle":"Five learned tokens beat sending the raw image, at 0.81% of its transmission cost.","key_machinery":"The load-bearing mechanism is end-to-end training of the whole semantic link: the transmitter LLM (LLaVA with LoRA adapters on Key/Value matrices) maps a vague task description plus multimodal input to machine language tokens; a JTCC autoencoder compresses the token embeddings; the compressed vectors are sent as analog amplitudes over an OFAT MIMO channel; the receiver reconstructs the tokens and prepends them to the detailed task description as a trainable prefix for a second LLM (LLaMA2-7B) that produces the answer. Channel noise is simulated during training, and gradients at the receiver are used to update the transmitter despite the non-differentiable channel. The prefix-injection design means the receiver LLM does not need fine-tuning, and the sparsity of domain-specific token embeddings justifies the dimensionality reduction.","core_discovery":"On its own terms, the paper's discovery is that a multimodal LLM can be steered, by a vague task description plus fine-tuning, to compress the information needed for that task into a very small number of token embeddings, and that those embeddings survive analog transmission well enough to outperform both raw-image question answering and DeepJSCC-based reconstruction at high signal-to-noise ratios. With K=5 tokens, each compressed to a lower dimension by JTCC, the method reaches higher accuracy than the raw-image LLaVA baseline on CLEVR and GQA when the channel is good, and remains competitive when SNR is low; natural-language summaries (Benchmark 3) perform substantially worse. The paper further reports a threshold effect: five tokens are enough to encode the task-relevant content, while fewer tokens cause an abrupt accuracy collapse and more tokens add little. The authors interpret this as evidence that agent communication should use a learned machine vocabulary rather than human language and that token sparsity can be exploited for transmission.","pith_inferences":["The headline overhead figure assumes that real-valued analog symbols can be compared to JPEG bytes without a bit-level channel model; a Shannon-capacity accounting at the operating SNR could significantly change the ratio.","The paper states the JTCC compressed dimension as 256 in the experimental setup but 512 in the compression-ratio discussion, so the exact 0.81% value is not uniquely determined by the text.","The receiver LLM is not fine-tuned and receives the learned tokens as a prefix, which the paper motivates as a generalizable design, but no cross-task or cross-modality transfer experiment is run; testing that would show whether the machine language is task-specific or reusable.","The sparsity argument predicts that task-relevant token embeddings have small intrinsic dimension; measuring that dimension per task could yield a principled way to choose K instead of the empirical threshold of five observed here."],"forward_implications":["If the token-based pipeline is correct, an agent can answer a visual reasoning question without the raw image ever being reconstructed at the receiver; only five learned embeddings need to arrive.","At high SNR the learned tokens beat both raw-image LLaVA and DeepJSCC-reconstructed images on CLEVR and GQA, so task-oriented token compression can be strictly better than source reconstruction.","The token-count experiments predict a threshold: below roughly five tokens accuracy collapses, while above five it saturates, so systems should adapt K to task complexity rather than use a fixed vocabulary.","JTCC contributes noise tolerance at low SNR while barely sacrificing high-SNR accuracy relative to uncompressed tokens (Benchmark 4).","Natural-language summaries (Benchmark 3) are a poor representation for image reasoning, supporting the paper's premise that human language is suboptimal for agent communication."],"supporting_citations":[{"why":"DeepJSCC supplies the joint source-channel coding baseline (Benchmark 2) that reconstructs the image before question answering.","marker":"[1]"},{"why":"Task-oriented semantic communication is the prior paradigm the paper extends by replacing natural-language output with machine tokens.","marker":"[3]"},{"why":"Provides the 'compression as directional collapse' argument used to justify task-aware compression direction.","marker":"[6]"},{"why":"Supports the claim that LLMs can express knowledge in newly learned languages without losing performance, motivating machine language tokens.","marker":"[7]"},{"why":"Supplies the geometric observation about contextual embeddings clustering by domain, used to justify token embedding sparsity.","marker":"[8]"},{"why":"CLEVR is one of the two visual reasoning datasets used to evaluate downstream accuracy and token-count thresholds.","marker":"[9]"},{"why":"GQA is the second evaluation dataset, for real-world visual reasoning and compositional question answering.","marker":"[10]"}],"fun_headline_variants":["Five learned tokens beat raw images at 0.81% of the transmission cost","Agent-to-agent chat in 5 machine tokens: beats raw-image QA","Task-focused token comms: 5 embeddings outperform full images","Five machine-language tokens beat raw pixels for task agents"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The claim that the method needs only 0.81% of the image's transmission overhead assumes that the number of real-valued analog symbols sent over the channel can be compared directly to the byte size of a JPEG file, without accounting for the bits or channel uses needed to deliver those symbols reliably.","fun_headline_variants_meta":{"raw":{"variants":["Five learned tokens beat raw images at 0.81% of the transmission cost","Agent-to-agent chat in 5 machine tokens: beats raw-image QA","Task-focused token comms: 5 embeddings outperform full images","Five machine-language tokens beat raw pixels for task agents"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000461,"raw_usage":{"total_tokens":2315,"prompt_tokens":960,"completion_tokens":1355,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":576,"completion_tokens_details":{"reasoning_tokens":1280}},"tokens_in":576,"tokens_out":1355,"duration_ms":15648,"temperature":1.0,"reasoning_tokens":1280,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T12:45:15.169685+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compare the proposed scheme to a raw-JPEG transmission under an equal end-to-end bit budget by converting each analog channel use to bits at the reported SNR (e.g., using a capacity formula for the OFAT/MIMO channel); if the token scheme needs a comparable or larger number of bits to reach the same accuracy, the overhead reduction is falsified.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"DeepJSCC supplies the joint source-channel coding baseline (Benchmark 2) that reconstructs the image before question answering."},{"cited_title":"Deletang, A","cited_arxiv_id":null,"evidence_quote":"Supports the claim that LLMs can express knowledge in newly learned languages without losing performance, motivating machine language tokens."},{"cited_title":"Johnson, B","cited_arxiv_id":null,"evidence_quote":"GQA is the second evaluation dataset, for real-world visual reasoning and compositional question answering."}],"review_version":1}