Pith. sign in

REVIEW 4 major objections 8 minor 1 cited by

Transmission With Machine Language Tokens: A Paradigm for Task-Oriented Agent Communication

T0 review · 4 major / 8 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims that five learned machine-language tokens, sent over a noisy analog channel, outperform raw-image transmission on visual question answering while using 0.81% of the JPEG overhead.

desk verdict Genuinely new idea—transmitting fine-tuned LLM token embeddings as a communication code—but the accuracy and overhead claims are not established because the baseline is not fine-tuned and the overhead metric is unit-inconsistent. read the letter →

arxiv 2507.21454 v1 pith:VHHAXOHG submitted 2025-07-29 eess.SP

Zhuoran Xiao , Chenhui Ye , Yijia Feng , Yunbo Hu , Tianyu Jiao , Liyu Cai , Guangyi Liu This is my paper · ORCID
classification eess.SP
keywords semanticcommunicationtask-orientedmachinelanguagetokenslargemodelsjointsource-channelcodinganalogtransmissionvisualquestionansweringmulti-modalLLM
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper proposes that two agents can communicate for a downstream task by transmitting a small set of 'machine language tokens'—the last K embedding vectors produced by a fine-tuned multimodal LLM—instead of sending the raw image or a natural-language description. The central claim is that these learned tokens, passed over a noisy analog MIMO channel and compressed by a joint token-and-channel coding (JTCC) module, carry enough task-relevant information to match or beat a raw-image baseline in visual question answering on CLEVR and GQA, while using about 0.81% of the original JPG's transmission budget. The authors argue that natural language is redundant and ambiguous for machines, and that task-oriented token embeddings can be learned end-to-end with simulated channel noise so the representation is simultaneously compact and noise-resistant. If true, this points to a communication paradigm where agents exchange abstract learned semantics rather than human-interpretable data.

What carries the argument

The load-bearing mechanism is end-to-end training of the whole semantic link: the transmitter LLM (LLaVA with LoRA adapters on Key/Value matrices) maps a vague task description plus multimodal input to machine language tokens; a JTCC autoencoder compresses the token embeddings; the compressed vectors are sent as analog amplitudes over an OFAT MIMO channel; the receiver reconstructs the tokens and prepends them to the detailed task description as a trainable prefix for a second LLM (LLaMA2-7B) that produces the answer. Channel noise is simulated during training, and gradients at the receiver are used to update the transmitter despite the non-differentiable channel. The prefix-injection design means the receiver LLM does not need fine-tuning, and the sparsity of domain-specific token embeddings justifies the dimensionality reduction.

What would settle it

Compare the proposed scheme to a raw-JPEG transmission under an equal end-to-end bit budget by converting each analog channel use to bits at the reported SNR (e.g., using a capacity formula for the OFAT/MIMO channel); if the token scheme needs a comparable or larger number of bits to reach the same accuracy, the overhead reduction is falsified.

Watch

Extended reading notes

Core claim

On its own terms, the paper's discovery is that a multimodal LLM can be steered, by a vague task description plus fine-tuning, to compress the information needed for that task into a very small number of token embeddings, and that those embeddings survive analog transmission well enough to outperform both raw-image question answering and DeepJSCC-based reconstruction at high signal-to-noise ratios. With K=5 tokens, each compressed to a lower dimension by JTCC, the method reaches higher accuracy than the raw-image LLaVA baseline on CLEVR and GQA when the channel is good, and remains competitive when SNR is low; natural-language summaries (Benchmark 3) perform substantially worse. The paper further reports a threshold effect: five tokens are enough to encode the task-relevant content, while fewer tokens cause an abrupt accuracy collapse and more tokens add little. The authors interpret this as evidence that agent communication should use a learned machine vocabulary rather than human language and that token sparsity can be exploited for transmission.

Load-bearing premise

The claim that the method needs only 0.81% of the image's transmission overhead assumes that the number of real-valued analog symbols sent over the channel can be compared directly to the byte size of a JPEG file, without accounting for the bits or channel uses needed to deliver those symbols reliably.

Editorial extensions

If this is right

  • If the token-based pipeline is correct, an agent can answer a visual reasoning question without the raw image ever being reconstructed at the receiver; only five learned embeddings need to arrive.
  • At high SNR the learned tokens beat both raw-image LLaVA and DeepJSCC-reconstructed images on CLEVR and GQA, so task-oriented token compression can be strictly better than source reconstruction.
  • The token-count experiments predict a threshold: below roughly five tokens accuracy collapses, while above five it saturates, so systems should adapt K to task complexity rather than use a fixed vocabulary.
  • JTCC contributes noise tolerance at low SNR while barely sacrificing high-SNR accuracy relative to uncompressed tokens (Benchmark 4).
  • Natural-language summaries (Benchmark 3) are a poor representation for image reasoning, supporting the paper's premise that human language is suboptimal for agent communication.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The headline overhead figure assumes that real-valued analog symbols can be compared to JPEG bytes without a bit-level channel model; a Shannon-capacity accounting at the operating SNR could significantly change the ratio.
  • The paper states the JTCC compressed dimension as 256 in the experimental setup but 512 in the compression-ratio discussion, so the exact 0.81% value is not uniquely determined by the text.
  • The receiver LLM is not fine-tuned and receives the learned tokens as a prefix, which the paper motivates as a generalizable design, but no cross-task or cross-modality transfer experiment is run; testing that would show whether the machine language is task-specific or reusable.
  • The sparsity argument predicts that task-relevant token embeddings have small intrinsic dimension; measuring that dimension per task could yield a principled way to choose K instead of the empirical threshold of five observed here.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 8 minor

Summary. The paper proposes a task-oriented agent communication system in which a sensor agent uses a fine-tuned multimodal LLM to encode a vague task description plus raw image into K 'machine language tokens' taken from the final transformer block, compresses them with a learned JTCC module, and transmits them over an analog MIMO-OFDM channel. The receiver concatenates the received tokens with a detailed task description and feeds them to a second LLM that answers a visual question. On CLEVR and GQA, the authors report compression ratios as low as 0.81% of the average JPEG size and accuracy above all four benchmarks at high SNR. The paper claims that this learned machine-token representation is more efficient and robust than raw-image, DeepJSCC, or natural-language transmission.

Significance. The direction is timely: moving from human-oriented representations to learned, task-specific latent tokens for agent-agent communication is a natural extension of semantic communication. The combination of a fixed pretrained backbone with LoRA adapters and a learned joint source-channel coder is plausible and could be useful if properly validated. Credit is due for using standard public VQA datasets and for stating the straight-through gradient issue explicitly. However, the evidence presented does not yet establish the headline claims: the main accuracy comparison is confounded by unequal fine-tuning, the overhead comparison uses incomparable units, and the JTCC dimension is inconsistent across sections. The paper's contribution remains a proposal with promising but unverified results.

major comments (4)
  1. [IV.B, Fig. 6] Benchmark 1 is defined in Sec. IV.B as transmitting raw images and using the original LLaVA-V1.5-7B model to answer questions, i.e., a zero-shot pretrained model. The proposed method, by contrast, is fine-tuned end-to-end on the training split with LoRA on the transmitter and receiver (Secs. III.C and IV.A). Consequently, the high-SNR accuracy advantage shown in Fig. 6 conflates the effect of the machine-token representation with the effect of task-specific fine-tuning; Benchmark 2, which also feeds reconstructed images to the original LLaVA, has the same limitation. Please add a raw-image baseline with identical LoRA fine-tuning (at least on the receiver side, and ideally with the same end-to-end training procedure but without the token bottleneck) and compare it with the proposed method across the same SNR range.
  2. [Fig. 5, Sec. IV.C] The compression-ratio comparison in Fig. 5 is not unit-consistent. The proposed method transmits K x L_t real-valued analog channel dimensions, while the denominator is the byte size of a JPEG image. A real-valued symbol transmitted over a noisy channel does not carry a fixed number of bits; the amount of information it can convey depends on SNR, bandwidth, modulation, and the rate-distortion tradeoff. Without a bits-per-channel-use mapping or an equivalent-rate calculation, the headline overhead percentage (0.81%) is not a meaningful comparison to JPEG bytes. Please report the number of channel uses, the SNR operating point, and an achievable-rate or rate-distortion-based comparison, or compare at a fixed bit rate with matched image compression.
  3. [IV.A vs IV.C] Sec. IV.A states that JTCC compresses token embeddings from 4096 to 256 dimensions, whereas Sec. IV.C states that 'we reduce the dimensionality to 512' and uses this value to compute the proposed method's overhead in Fig. 5. Since the compression ratio scales linearly with the JTCC output dimension, this inconsistency changes the headline overhead by a factor of two. Please correct the discrepancy and recompute all overhead numbers with the actual deployed dimension.
  4. [III.C] The end-to-end training relies on a straight-through gradient approximation in which transmitter-side gradients are replaced by receiver-side gradients computed on the received token embeddings. This is a nontrivial training-signal assumption, and the manuscript provides no validation that the approximation yields an unbiased or even useful gradient direction. Please provide an ablation or an analysis showing that this approximation converges and that the learned transmitter is genuinely optimized for the noisy channel, rather than relying on the receiver doing most of the task adaptation.
minor comments (8)
  1. [II.B, Eq. (2)] Equation (2) defines e_d = LLM_v(I_v), but the surrounding text says e_d is the detailed task embedding obtained by tokenizing and embedding I_d; please make the notation consistent and clarify what LLM_v outputs.
  2. [IV.A] Training hyperparameters (LoRA rank and target modules beyond K/V, learning rate, number of steps, batch size, channel and OFDM parameters) are not reported, which prevents reproduction.
  3. [Fig. 7] The threshold claim that five tokens are sufficient is inferred from the same experiment in which K=5 was already fixed as the operating point in Sec. IV.A; if K was selected by inspecting Fig. 7, describe the selection procedure or validate on a held-out split to avoid circularity.
  4. [Fig. 5] The JPEG size used as the denominator is described only as 'average storage size of JPG images'; the JPEG quality setting and the number of images used for the average are not stated, and no confidence intervals or error bars are provided for any of the compression-ratio bars.
  5. [IV.C, Benchmark 3] The overhead estimate for Benchmark 3 is based on the number of characters in a textual description with UTF-8 encoding, but the text does not specify how the description is generated or how variable-length UTF-8 encoding is counted; please specify.
  6. [Fig. 6] Accuracy curves are reported as point estimates with no error bars or multiple-seed trials, so it is unclear whether the differences between the proposed method, Benchmark 3, and Benchmark 4 are statistically significant.
  7. [Fig. 2] In the version provided, the internal labels of Fig. 2 appear as illegible glyphs; if this is not a rendering artifact, the figure needs to be redrawn so that the architecture is readable.
  8. [IV.A] The claim that GQA is used with 'the same sizes' as CLEVR needs clarification, since the standard GQA split is fixed; please state which images and questions were used for training and testing.

Circularity Check

1 steps flagged · score 1.0 of 10

No significant derivational circularity: the accuracy and overhead results are measured, not derived from fitted constants or self-citations; at most a mild post-hoc token-count selection is presented as a discovered threshold.

  1. fitted input called prediction [Section IV.C, Fig. 7 (token-count sweep), with K=5 fixed earlier in Section IV.A]
    "The results reveal a critical threshold phenomenon: for the two image reasoning datasets considered, five tokens are sufficient to capture all the information required for successful reasoning. Increasing the number of tokens beyond this point yields no further improvement in performance due to internal redundancy, although a marginal gain in robustness can be observed."

    Section IV.A fixes 'the transmitter encodes information into 5 tokens' before the experiments, and Fig. 7 then sweeps token counts 3-20. The 'critical threshold phenomenon' claim that five tokens suffice is a restatement of the same curve that motivated the choice of K, and the headline overhead ratio (Fig. 5, 0.81%) depends on K=5 being adequate. That adequacy is demonstrated only on the same in-distribution test sets used to justify the selection, so the threshold narrative is post-hoc parameter selection presented as a discovered law rather than an independent prediction. The effect is mild because the accuracy values in Fig. 6 are genuine measurements, not outputs of a fit.

full rationale

The paper's central claims are empirical: downstream-task accuracy (Fig. 6) is measured over CLEVR/GQA across SNR, and compression ratios (Fig. 5) are arithmetic consequences of the defined transmission dimensions and measured JPG sizes. No result is derived from a theorem whose truth is presupposed, and the reference list contains no self-citations at all — all ten citations (DeepJSCC, LLaVA, CLEVR, GQA, etc.) are external, so the self-citation load-bearing and uniqueness-imported-from-authors patterns do not apply. The only mild circularity is the token-count narrative: K=5 is fixed in the setup and the same token-count sweep is then reported as a discovered 'critical threshold phenomenon', with the overhead claim resting on K=5 being sufficient on the same in-distribution data used to justify the choice. This is post-hoc selection rather than a forced prediction and does not contaminate the measured accuracy numbers. The more serious threats to the paper's validity — the untuned Benchmark 1 baseline that confounds task-specific fine-tuning with the proposed representation, the byte-versus-analog-dimension overhead comparison, and the internal inconsistency between 256 (Sec. IV.A) and 512 (Fig. 5) JTCC dimensions — are correctness and fairness concerns, not circularity, and are outside the scope of this pass.

Assumptions & free parameters 4 free parameters · 4 assumptions · 1 invented entities

The central results depend on several unmeasured modeling choices: a post-hoc token count, an inconsistently specified compression dimension, and an asserted sparsity property that is never measured. The only invented concept, machine language tokens, is operationalized within the model but has no external falsifiable handle.

free parameters (4)
  • number of machine tokens K = 5
    Chosen based on the threshold observed in Fig. 7; not derived from theory. Used for all reported results.
  • JTCC output dimension = 256 (Section IV-A) or 512 (Section IV-C)
    Inconsistent across the text; selected by the authors, not justified theoretically or by a performance trade-off analysis.
  • LoRA rank and target modules = not reported
    LoRA applied to Key and Value matrices, rank not stated; a hyperparameter affecting capacity.
  • Training channel noise level = not reported
    The paper says simulated noise is injected during training but does not specify the SNR distribution or schedule used.
assumptions (4)
  • domain assumption Pre-trained LLMs can be adapted via LoRA to generate task-specific machine tokens without acquiring new knowledge
    Section III.C: the authors assume the large model can transform language to machine-interpretable forms without new knowledge, hence only K/V matrices are adapted.
  • domain assumption Token embeddings of a specialized agent cluster in a small region of the embedding space, enabling dimension reduction
    Section III.A.4: this sparsity is asserted based on [8] and is not measured for the learned machine tokens.
  • ad hoc to paper The straight-through gradient approximation (using receiver-side gradients as transmitter-side gradients) yields a valid training signal
    Section III.C: the gradient cannot propagate through the noisy channel, so they approximate; no analysis of bias or convergence is given.
  • domain assumption The OFAT analog transmission can be modeled by a channel matrix H with Ar >= At, enabling direct reconstruction
    Section III.B: assumes receiver has more antennas; no justification or simulation of the MIMO channel beyond SNR curves.
invented entities (1)
  • Machine language tokens
    purpose: Compact, task-specific semantic representation transmitted between agents
    Defined as the last K token embeddings of a fine-tuned LLM; no predictions about their properties outside the trained system are given, so they lack an external falsifiable handle.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Transmission With Machine Language Tokens: A Paradigm for Task-Oriented Agent Communication." pith.science (2026). https://pith.science/paper/VHHAXOHG

@misc{pith2026250721454,
  author       = {Pith},
  title        = {Pith review of: Transmission With Machine Language Tokens: A Paradigm for Task-Oriented Agent Communication},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/VHHAXOHG}},
  note         = {Machine review of arXiv:2507.21454}
}
read the original abstract

The rapid advancement in large foundation models is propelling the paradigm shifts across various industries. One significant change is that agents, instead of traditional machines or humans, will be the primary participants in the future production process, which consequently requires a novel AI-native communication system tailored for agent communications. Integrating the ability of large language models (LLMs) with task-oriented semantic communication is a potential approach. However, the output of existing LLM is human language, which is highly constrained and sub-optimal for agent-type communication. In this paper, we innovatively propose a task-oriented agent communication system. Specifically, we leverage the original LLM to learn a specialized machine language represented by token embeddings. Simultaneously, a multi-modal LLM is trained to comprehend the application task and to extract essential implicit information from multi-modal inputs, subsequently expressing it using machine language tokens. This representation is significantly more efficient for transmission over the air interface. Furthermore, to reduce transmission overhead, we introduce a joint token and channel coding (JTCC) scheme that compresses the token sequence by exploiting its sparsity while enhancing robustness against channel noise. Extensive experiments demonstrate that our approach reduces transmission overhead for downstream tasks while enhancing accuracy relative to the SOTA methods.

Figures

Figures reproduced from arXiv: 2507.21454 by the authors.

Figure 1
Figure 1. The system model of the proposed task-oriented agent [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. The network structure of the proposed task-oriented [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. The propose orthogonal frequency division anolog tr [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: Illustration of the datasets used in the experiment p [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]
Figure 5
Figure 5. Figure 5: Compression ratio relative to the original JPG image [PITH_FULL_IMAGE:figures/full_fig_p006_5.png]
Figure 6
Figure 6. Figure 6: The accuracy of downstream tasks across different me [PITH_FULL_IMAGE:figures/full_fig_p006_6.png]
Figure 7
Figure 7. Figure 7: The accuracy of downstream tasks for the proposed met [PITH_FULL_IMAGE:figures/full_fig_p006_7.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Token Communication for Multimodal Large Language Model

    eess.SP 2026-08 conditional novelty 6.0 of 10

    A token communication framework for MLLMs sends compressed image latents and reconstructs them as both an image prior and adapter-injected visual tokens, improving task performance at low bitrates.

Reference graph

Works this paper leans on

11 extracted references · 4 canonical work pages · cited by 1 Pith paper

  1. [1]

    11em plus .33em minus .07em 4000 4000 100 4000 4000 500 `\.=1000 = #1 \@IEEEnotcompsoconly \@IEEEcompsoconly #1 * [1] 0pt [0pt][0pt] #1 * [1] 0pt [0pt][0pt] #1 * \| ** #1 \@IEEEauthorblockNstyle \@IEEEcompsocnotconfonly \@IEEEauthorblockAstyle \@IEEEcompsocnotconfonly \@IEEEcompsocconfonly \@IEEEauthordefaulttextstyle \@IEEEcompsocnotconfonly \@IEEEauthor...

    work page

  2. [2]

    Xu, T.-Y

    J. Xu, T.-Y. Tung, B. Ai, W. Chen, Y. Sun, and D. Gündüz, `` Deep Joint Source-Channel Coding for Semantic Communications ,'' IEEE Communications Magazine, vol. 61, no. 11, pp. 42--48, 2023

    2023

  3. [3]

    S. Wang, J. Dai, Z. Liang, K. Niu, Z. Si, C. Dong, X. Qin, and P. Zhang, `` Wireless Deep Video Semantic Transmission ,'' IEEE Journal on Selected Areas in Communications, vol. 41, no. 1, pp. 214--229, 2023

    2023

  4. [4]

    Z. Weng, Z. Qin, and X. Tao, `` Task-Oriented Semantic Communications for Speech Transmission ,'' in 2023 IEEE 98th Vehicular Technology Conference (VTC2023-Fall), 2023, pp. 1--5

    2023

  5. [5]

    Y. Wang, Z. Sun, J. Fan, and H. Ma, `` On the Uses of Large Language Models to Design End-to-End Learning Semantic Communication ,'' in 2024 IEEE Wireless Communications and Networking Conference (WCNC), 2024, pp. 1--6

    2024

  6. [6]

    D. Cao, J. Wu, and A. K. Bashir, `` Multimodal Large Language Models Driven Privacy-Preserving Wireless Semantic Communication in 6G ,'' in 2024 IEEE International Conference on Communications Workshops (ICC Workshops), 2024, pp. 171--176

    2024

  7. [7]

    Deletang, A

    G. Deletang, A. Ruoss, P.-A. Duquenne, E. Catt, T. Genewein, C. Mattern, J. Grau-Moya, L. K. Wenliang, M. Aitchison, L. Orseau et al., `` Language Modeling Is Compression ,'' in The Twelfth International Conference on Learning Representations

    work page

  8. [8]

    Brown, B

    T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell et al., ``Language models are few-shot learners,'' Advances in neural information processing systems, vol. 33, pp. 1877--1901, 2020

    1901

Show all 11 references
  1. [9]

    Ethayarajh, ``How contextual are contextualized word representations? comparing the geometry of bert, elmo, and gpt-2 embeddings,'' arXiv preprint arXiv:1909.00512, 2019

    K. Ethayarajh, ``How contextual are contextualized word representations? comparing the geometry of bert, elmo, and gpt-2 embeddings,'' arXiv preprint arXiv:1909.00512, 2019

    1909 arXiv

  2. [10]

    Johnson, B

    J. Johnson, B. Hariharan, L. van der Maaten, L. Fei-Fei, C. L. Zitnick, and R. Girshick, `` CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoning ,'' in 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017, pp. 1988--1997

    2017

  3. [11]

    D. A. Hudson and C. D. Manning, `` GQA: A New Dataset for Real-World Visual Reasoning and Compositional Question Answering ,'' Conference on Computer Vision and Pattern Recognition (CVPR), 2019

    2019

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.