Pith. sign in

REVIEW 3 major objections 3 minor 1 cited by

Fine-tuning Multimodal Transformers on Edge: A Parallel Split Learning Approach

T0 review · 3 major / 3 minor · reviewed 2026-08-08 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A parallel split-learning design fine-tunes multimodal transformers on edge devices with 250x less client-side compute than federated learning.

desk verdict Careful empirical mapping of split learning to multimodal transformers, but the core server update as described cannot be computed from scalar client losses; the paper needs major revision before the results can be trusted. read the letter →

arxiv 2502.06355 v3 pith:CYRWOPCJ submitted 2025-02-10 cs.DC cs.LG

Timo Fudala , Vasileios Tsouvalas , Nirvana Meratnia This is my paper · ORCID
classification cs.DCcs.LG
keywords multimodaltransformerssplitlearningparallelfederatededgecomputingparameter-efficientfine-tuningimage-textretrievalmodalityfusion
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Multimodal transformers are too heavy to fine-tune directly on phones and sensors. This paper proposes MPSL, a parallel split-learning setup that cuts the model at the tokenizer/encoder boundary: edge clients run only lightweight tokenizers, the server runs the transformer encoder and task head, and training proceeds through a single aggregated backward pass. The central claim is that this setup matches or beats standard federated learning on seven multimodal datasets while cutting client-side computation by roughly 250x and keeping labels on the client. If right, it makes distributed fine-tuning of large vision-language-audio models practical on resource-constrained edge devices and gives a concrete scaling story for communication cost as encoders grow.

What carries the argument

The load-bearing object is the split point and the aggregated-loss backward pass. MPSL partitions a multimodal transformer into client-side modality-specific tokenizers and a server-side unified encoder plus task head; the tokenizers convert each modality into a sequence of embeddings that are concatenated early or late, global-average-pooled, and classified on the server. The server then computes one weighted loss $L_S = \sum_n \frac{|B_n|}{|B|} L_{C_n}$ over all clients' scalar losses and backpropagates once, which is what removes per-client sub-models, client synchronization, and label sharing from the loop while keeping client-side work at a few tokenizer forward passes.

What would settle it

Run the protocol exactly as written—clients send only their scalar loss, never labels or per-sample gradient vectors—and check whether the server-side encoder weights change after a training step. For the usual classification loss, knowing only the single loss number does not tell you how each logit contributed, so the server cannot compute its own gradient unless extra information is sent; this can be settled by instrumenting the message contents and the resulting gradients.

Watch

Extended reading notes

Core claim

The paper's central discovery is that multimodal transformers can be fine-tuned in a split-learning topology without requiring clients to hold a copy of the encoder or synchronize their local models. Clients tokenize their own image, audio, or text inputs, send only the resulting activations to the server, and receive the server's predictions; the client computes its own loss against its local label and sends that scalar loss to the server. The server weights the client losses by batch fraction, performs one backward pass on the whole encoder-head stack, and sends the cut-layer gradients back to clients, so coordination emerges from a shared global gradient signal rather than from FedAvg-style parameter averaging. Across seven datasets covering vision-text, vision-audio, and audio-text pairs, MPSL matches or exceeds the federated baselines on classification tasks while using about 1.0 million trainable client parameters and 0.1 GFLOPs per sample, and it improves communication-cost scaling relative to FedAvg as encoder size grows.

Load-bearing premise

The entire training loop rests on the assumption that the server can correctly update its encoder and head from only the scalar loss values sent by clients, with labels never leaving the devices.

Editorial extensions

If this is right

  • Client-side computation becomes nearly independent of transformer encoder size, since clients only execute tokenizers and receive gradients.
  • Communication cost per client is tied to activation (smashed-data) size rather than model parameter count, so MPSL overtakes FedAvg's communication cost as encoders grow toward ViT-L and ViT-H scale.
  • The shared server-side backward pass acts as a global alignment signal, which the paper reports as faster convergence and more stable training under high client counts and non-IID data.
  • Label privacy is preserved in principle because the ground-truth label never leaves the client, though the server still sees all intermediate representations.
  • Retrieval tasks remain the weakest point: distributed training underperforms centralized fine-tuning, and MPSL's gains there depend on large effective batch sizes.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The protocol as written sends only scalar losses to the server; if the server is to update its own layers, it needs gradient information that a single scalar cross-entropy value does not determine, so the practical implementation likely must transmit per-sample output gradients (or equivalent) alongside the loss—an implicit re-scoping of the label-privacy claim.
  • The same single-backward-pass mechanism might transfer to other split points, such as parameter-efficient adapters on frozen backbones, and to heterogeneous edge hardware, though client dropouts and missing modalities are expressly left to future work.
  • A direct test of the batch-size hypothesis would be to train MPSL with replay buffers or gradient accumulation to simulate large batches on clients that hold very few samples; the paper predicts this should close much of the retrieval gap with centralized training.
  • If the scalar-loss update can be made to work, MPSL's communication and compute profile makes it a candidate for on-device personalization of large multimodal models, since the post-training model can be reassembled per client or via FedAvg-style aggregation.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 3 minor

Summary. The paper proposes MPSL, a parallel split learning framework for fine-tuning multimodal transformers on edge devices. Clients run lightweight modality-specific tokenizers, send activations to a server that processes them through a shared encoder and task head, and the server supposedly performs a single backward pass using an aggregated loss computed from scalar client losses. The authors report evaluations on seven multimodal datasets, claiming that MPSL matches or outperforms federated learning baselines while reducing client-side computation by roughly 250x and improving communication scalability with model size. The paper also includes ablations on batch size, number of fine-tuned blocks, encoder depth, and fusion type.

Significance. The problem addressed is relevant: distributed fine-tuning of large multimodal transformers on resource-constrained clients is an important practical challenge, and the reported efficiency numbers (about 1.0M client-side trainable parameters and 0.1 GFLOPs per sample) would be valuable if the training method were sound. The manuscript is also commendable for its breadth of evaluation across seven datasets, multiple modality pairs, and several ablations, and for releasing code. However, the central training mechanism is not well-defined: the server cannot compute gradients of the aggregated loss with respect to its own model from scalar client losses alone. Because this under-specification affects the core learning loop, the empirical results cannot be attributed to the described algorithm, and the significance of the claimed contributions cannot be assessed without a corrected and re-evaluated training procedure.

major comments (3)
  1. [Section 3.2, Eq. (4)] The server-side update is not a well-defined computation. The text states that 'client n only sends its loss to the server' and Eq. (4) defines LS as a weighted sum of scalar client losses. To update FS by gradient descent, the server would need the gradient of LCn with respect to the server's output y_hat (or equivalently with respect to the activations an) for each sample. A scalar loss value LCn(y_hat, y) does not determine this gradient: for cross-entropy, the gradient is softmax(z) - one_hot(y), which depends on the unknown label y; for the contrastive loss used in image-text retrieval, it depends on the full set of positive and negative similarities in the batch. Therefore the claim that the server 'performs a single backward pass on FS' using only aggregated scalar losses is not implementable as written, and the results in Tables 1-4 cannot follow from the stated algorithm.
  2. [Section 3.2, client update paragraph] The same issue breaks the client-side update as well. The paper says that after backpropagation the server sends 'cut-layer gradients' to the clients, which the clients use to update FCn. These gradients are dLS/dan, and computing them requires dLS/dy_hat, which in turn requires labels or per-sample output gradients. Since neither is communicated under the described protocol, the client-side training step is also undefined. The training loop as a whole therefore cannot be executed as described.
  3. [Section 3.2, label privacy claim] The claim that MPSL 'eliminates label sharing' while still updating the server model is internally inconsistent. The only way for the server to compute the needed gradients is for clients to send either their labels or per-sample gradients of the loss with respect to the server's outputs. Sending the latter would preserve label privacy but would change the communication protocol and the stated communication-cost analysis; sending the former would contradict the privacy claim. The manuscript should specify which quantity is actually transmitted and should re-derive the communication and privacy statements accordingly.
minor comments (3)
  1. [Abstract and Section 3.3] Please fix the wording: 'computational efficient' should be 'computationally efficient' in the abstract, and 'splitted' should be 'split' in Section 3.3.
  2. [Table 3 caption] The caption says entries are 'normalized accuracy and top-1 recall vs. centralized fine-tuning', but the surrounding text describes raw percentages (e.g., accuracy improving from 50 to 500). Please clarify whether the reported numbers are raw or normalized, and if normalized, specify the normalization formula.
  3. [Figure 6] Unlike Tables 1 and 2, the encoder-depth experiments in Figure 6 do not report error bars or the number of runs. Adding this information would make the scalability claim easier to assess.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the server-side scalar-loss gradient issue is an algorithmic correctness concern, not a case where a prediction reduces by construction to its inputs.

full rationale

I walked the derivation chain in Section 3 and the evaluation in Section 4. The central training rule is Eq. 4, LS = sum |Bn|/|B| LCn, paired with the claim that the server 'performs a single backward pass on FS' after receiving only scalar losses from clients. This is not a circular reduction: the paper does not define the gradient of LS from the communicated scalar losses alone, and the labels staying on-device makes the update under-specified. That is a correctness or feasibility flaw, not a case where a derived quantity is identical by definition to an input. The headline efficiency claims (250x client computation reduction, communication scaling) are direct arithmetic consequences of the split architecture and are measured against external baselines (centralized fine-tuning, FedAvg, FedCLIP), not fitted parameters renamed as predictions. The method builds on external prior work (Lyu et al. 2023 for loss aggregation, Zhang et al. 2023 for Meta-Transformer tokenizers); these are not self-citations, and no uniqueness theorem is imported from the authors' own prior work. No equation is shown to equal another by construction, and no fitted value is relabeled as a prediction. Therefore, despite a serious internal inconsistency in the described server update, the paper does not exhibit circular reasoning in the sense defined here.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

No new physical entities or model components beyond a composition of existing tokenizers, encoders, and split-learning machinery. The main assumptions are the feasibility of scalar-loss backward pass and the stability of implicit client coordination; both are asserted rather than demonstrated.

free parameters (3)
  • Batch size per task = 50 to 500 depending on dataset (e.g., COCO-QA 500, T4SA 400, UCF101 300, Flickr30K 200, MS-COCO 200)
    Chosen based on Table 3 to maximize normalized accuracy or recall; large batch improves retrieval alignment and classification, so the central performance claim depends on these choices.
  • Fusion type per task = Early for COCO-QA, T4SA, MELD; late for Kinetics-Sounds, UCF101, and retrieval tasks
    Selected based on observed performance (Table 5) rather than a task-agnostic rule; this post-hoc choice affects final accuracy.
  • Number of fine-tuned ViT blocks = Last 6 for ViT-B classification tasks; last 6, 12, or 16 for ViT-Ti/S/B/L/H in Figure 6
    Fine-tuning only 1 block hurts performance; the paper chooses a threshold based on Figure 5, which affects comparability across models and tasks.
assumptions (4)
  • standard math Standard backpropagation and transformer architecture as implemented in PyTorch and Meta-Transformer
    The method assumes gradients can be computed through the unified encoder and tokenizers using standard deep learning frameworks.
  • ad hoc to paper Server can compute a single backward pass from aggregated scalar client losses without labels or per-sample gradients
    Section 3.2 defines LS as a weighted sum of client scalar losses and claims a single backward pass on FS; this assumes scalar losses carry enough information for gradient computation, which is not generally true.
  • domain assumption Client-side tokenizers can be updated using cut-layer gradients from an aggregated global loss, and implicit coordination without synchronization prevents client drift
    Section 3.2 states 'coordination emerges implicitly through shared cut-layer gradients'; the paper provides no convergence analysis supporting this.
  • domain assumption Pretrained Meta-Transformer (ViT-B/16) and CLIP tokenizers provide a suitable backbone for all evaluated tasks
    The method relies on these pretrained components being appropriate for diverse modalities and tasks; the paper does not ablate this choice.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Fine-tuning Multimodal Transformers on Edge: A Parallel Split Learning Approach." pith.science (2026). https://pith.science/paper/CYRWOPCJ

@misc{pith2026250206355,
  author       = {Pith},
  title        = {Pith review of: Fine-tuning Multimodal Transformers on Edge: A Parallel Split Learning Approach},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/CYRWOPCJ}},
  note         = {Machine review of arXiv:2502.06355}
}
read the original abstract

Multimodal transformers integrate diverse data types like images, audio, and text, advancing tasks such as audio-visual understanding and image-text retrieval; yet their high parameterization limits deployment on resource-constrained edge devices. Split Learning (SL), which partitions models at a designated cut-layer to offload compute-intensive operations to the server, offers a promising approach for distributed training of multimodal transformers, though its application remains underexplored. We present MPSL, a parallel SL approach for computational efficient fine-tuning of multimodal transformers in a distributed manner, while eliminating label sharing, client synchronization, and per-client sub-model management. MPSL employs lightweight client-side tokenizers and a unified modality-agnostic encoder, allowing flexible adaptation to task-specific needs. Our evaluation across 7 multimodal datasets demonstrates that MPSL matches or outperforms Federated Learning, reduces client-side computations by 250x, and achieves superior scalability in communication cost with model growth. Through extensive analysis, we highlight task suitability, trade-offs, and scenarios where MPSL excels, inspiring further exploration.

Figures

Figures reproduced from arXiv: 2502.06355 by the authors.

Figure 1
Figure 1. An illustration of MPSL’s training pipeline. Clients tokenize inputs of multiple modalities and offload intensive computations to the server, which processes the received activations through a unified encoder and then performs a single backward pass using an aggregated loss function. V: Vision, A: Audio, T: Text, BP: Back Propagation, FP: Forward Pass sub-model management on the server. • MPSL reduces client-side co… view at source ↗
Figure 2
Figure 2. Average number of rounds to reach 95% of peak perfor [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 4
Figure 4. Effect of batch size on embedding spaces after PCA for [PITH_FULL_IMAGE:figures/full_fig_p006_4.png] view at source ↗
Figures from the paper (2 more)
Figure 5
Figure 5. Figure 5: Impact of fine-tuned ViT blocks on model performance [PITH_FULL_IMAGE:figures/full_fig_p006_5.png]
Figure 6
Figure 6. Figure 6: Impact of encoder depth on performance for COCO-QA. # ViT blocks N = 25 N = 100 1 6 12 1 6 12 COCO-QA [Ren et al., 2015] 74.8 85.4 88.8 66.7 74.5 70.4 T4SA [Vadicamo et al., 2017] 80.7 88.5 90.9 75.7 89.0 89.6 Kinetics-Sounds [Arandjelovic and Zisserman, 2017] 80.1 90.…

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. AutoEncoder-Compressed Parallel Split Learning for Pre-trained Model Fine-Tuning

    cs.DC 2026-07 conditional novelty 6.0 of 10

    An autoencoder-based split-learning compressor with a two-stage alignment protocol achieves about 10x communication reduction during pre-trained vision-model fine-tuning with near-zero accuracy loss, outperforming heu...

Reference graph

Works this paper leans on

29 extracted references · 21 canonical work pages · cited by 1 Pith paper

  1. [1]

    Vatt: Transformers for multimodal self-supervised learning from raw video, audio and text,

    [Akbari et al., 2021] Hassan Akbari, Liangzhe Yuan, Rui Qian, Wei-Hong Chuang, Shih-Fu Chang, Yin Cui, and Boqing Gong. Vatt: Transformers for multimodal self-supervised learning from raw video, audio and text,

    work page 2021

  2. [7]

    Imagebind: One embedding space to bind them all,

    [Girdhar et al., 2023] Rohit Girdhar, Alaaeldin El-Nouby, Zhuang Liu, Mannat Singh, Kalyan Vasudev Alwala, Armand Joulin, and Ishan Misra. Imagebind: One embedding space to bind them all,

    work page 2023

  3. [9]

    Dis- tributed learning of deep neural network over multiple agents,

    [Gupta and Raskar, 2018] Otkrist Gupta and Ramesh Raskar. Dis- tributed learning of deep neural network over multiple agents,

    work page 2018

  4. [11]

    Parameter- efficient transfer learning for nlp

    [Houlsby et al., 2019] Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter- efficient transfer learning for nlp. In International conference on machine learning, pages 2790–2799. PMLR,

    work page 2019

  5. [13]

    ViT-Lens: Initiating Omni-Modal Exploration through 3D Insights

    [Lei et al., 2023] Weixian Lei, Yixiao Ge, Jianfeng Zhang, Dy- lan Sun, Kun Yi, Ying Shan, and Mike Zheng Shou. Vit- lens: Towards omni-modal representations. arXiv preprint arXiv:2308.10185,

    work page Pith review arXiv 2023

  6. [14]

    Federated learning on non-iid data silos: An experimental study,

    [Li et al., 2021] Qinbin Li, Yiqun Diao, Quan Chen, and Bingsheng He. Federated learning on non-iid data silos: An experimental study,

    2021

  7. [15]

    Microsoft coco: Common objects in con- text

    [Lin et al., 2014] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Doll ´ar, and C Lawrence Zitnick. Microsoft coco: Common objects in con- text. In Computer Vision–ECCV 2014: 13th European Confer- ence, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pages 740–755. Springer,

    work page 2014

  8. [17]

    Fedclip: Fast generalization and personalization for clip in feder- ated learning

    [Lu et al., 2023] Wang Lu, Xixu Hu, Jindong Wang, and Xing Xie. Fedclip: Fast generalization and personalization for clip in feder- ated learning. arXiv preprint arXiv:2302.13485,

    arXiv 2023

Show all 29 references
  1. [18]

    Scalable aggregated split learning for data-driven edge intelligence on internet-of-things

    [Lyu et al., 2023] Xinchen Lyu, Shuhan Liu, Junlin Liu, and Chen- shan Ren. Scalable aggregated split learning for data-driven edge intelligence on internet-of-things. IEEE Internet of Things Mag- azine, 6(4):124–129,

    2023

  2. [19]

    Communication-Efficient Learning of Deep Networks from De- centralized Data

    [McMahan et al., 2017] Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Aguera y Arcas. Communication-Efficient Learning of Deep Networks from De- centralized Data. In Aarti Singh and Jerry Zhu, editors, Pro- ceedings of the 20th International Conference o...

    2017

  3. [20]

    Mix2sfl: Two-way mixup for scalable, accurate, and communication-efficient split federated learning

    [Oh et al., 2023] Seungeun Oh, Hyelin Nam, Jihong Park, Praneeth Vepakomma, Ramesh Raskar, Mehdi Bennis, and Seong-Lyun Kim. Mix2sfl: Two-way mixup for scalable, accurate, and communication-efficient split federated learning. IEEE Trans- actions on Big Data,

    2023

  4. [21]

    Server-side local gradient averaging and learning rate acceleration for scalable split learning,

    [Pal et al., 2021] Shraman Pal, Mansi Uniyal, Jihong Park, Pra- neeth Vepakomma, Ramesh Raskar, Mehdi Bennis, Moongu Jeon, and Jinho Choi. Server-side local gradient averaging and learning rate acceleration for scalable split learning,

    2021

  5. [22]

    Flickr30k entities: Collecting region-to-phrase corre- spondences for richer image-to-sentence models

    [Plummer et al., 2015] Bryan A Plummer, Liwei Wang, Chris M Cervantes, Juan C Caicedo, Julia Hockenmaier, and Svetlana Lazebnik. Flickr30k entities: Collecting region-to-phrase corre- spondences for richer image-to-sentence models. In Proceedings of the IEEE international conf...

    2015

  6. [24]

    Learning transferable visual models from natural language supervision,

    [Radford et al., 2021] Alec Radford, Jong Wook Kim, Chris Hal- lacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision,

    2021

  7. [25]

    Exploring models and data for image question answering

    [Ren et al., 2015] Mengye Ren, Ryan Kiros, and Richard Zemel. Exploring models and data for image question answering. Ad- vances in neural information processing systems, 28,

    2015

  8. [26]

    Ucf101: A dataset of 101 human actions classes from videos in the wild

    [Soomro et al., 2012] Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah. Ucf101: A dataset of 101 human actions classes from videos in the wild. arXiv preprint arXiv:1212.0402,

    2012 arXiv

  9. [28]

    Shuffled trans- former for privacy-preserving split learning

    [Xu et al., 2023] Hengyuan Xu, Liyao Xiang, Hangyu Ye, Dixi Yao, Pengzhi Chu, and Baochun Li. Shuffled trans- former for privacy-preserving split learning. arXiv preprint arXiv:2304.07735,

    2023 arXiv

  10. [29]

    Meta-transformer: A unified framework for multimodal learning

    [Zhang et al., 2023] Yiyuan Zhang, Kaixiong Gong, Kaipeng Zhang, Hongsheng Li, Yu Qiao, Wanli Ouyang, and Xiangyu Yue. Meta-transformer: A unified framework for multimodal learning. arXiv preprint arXiv:2307.10802, 2023

    2023 arXiv

  11. [2012]

    Cross-media learning for image sentiment analysis in the wild

    [Vadicamo et al., 2017] Lucia Vadicamo, Fabio Carrara, Andrea Cimino, Stefano Cresci, Felice Dell’Orletta, Fabrizio Falchi, and Maurizio Tesconi. Cross-media learning for image sentiment analysis in the wild. In 2017 IEEE International Conference on Computer Vision Workshops (...

    2017

  12. [2014]

    Efficient par- allel split learning over resource-constrained wireless edge net- works

    [Lin et al., 2024] Zheng Lin, Guangyu Zhu, Yiqin Deng, Xianhao Chen, Yue Gao, Kaibin Huang, and Yuguang Fang. Efficient par- allel split learning over resource-constrained wireless edge net- works. IEEE Transactions on Mobile Computing,

    2024

  13. [2015]

    Meld: A multimodal multi-party dataset for emotion recognition in conversations

    [Poria et al., 2018] Soujanya Poria, Devamanyu Hazarika, Navonil Majumder, Gautam Naik, Erik Cambria, and Rada Mihalcea. Meld: A multimodal multi-party dataset for emotion recognition in conversations. arXiv preprint arXiv:1810.02508,

    2018 arXiv

  14. [2017]

    The iot breaches your household again

    [Bonaventura et al., 2024] Davide Bonaventura, Sergio Esposito, and Giampaolo Bella. The iot breaches your household again. In Proceedings of the 21st International Conference on Security and Cryptography, page 475–482. SCITEPRESS - Science and Technology Publications,

    2024

  15. [2018]

    Accelerating federated learning with split learning on locally generated losses

    [Han et al., 2021] Dong-Jun Han, Hasnain Irshad Bhatti, Jung- moon Lee, and Jaekyun Moon. Accelerating federated learning with split learning on locally generated losses. In ICML 2021 workshop on federated learning for user privacy and data confi- dentiality. ICML Board,

    2021

  16. [2019]

    Privacy-sensitive parallel split learning

    [Jeon and Kim, 2020] Joohyung Jeon and Joongheon Kim. Privacy-sensitive parallel split learning. In 2020 International Conference on Information Networking (ICOIN) , pages 7–9. IEEE,

    2020

  17. [2020]

    Multi-modal align- ment using representation codebook,

    [Duan et al., 2022] Jiali Duan, Liqun Chen, Son Tran, Jinyu Yang, Yi Xu, Belinda Zeng, and Trishul Chilimbi. Multi-modal align- ment using representation codebook,

    2022

  18. [2021]

    Look, listen and learn

    [Arandjelovic and Zisserman, 2017] Relja Arandjelovic and An- drew Zisserman. Look, listen and learn. In Proceedings of the IEEE international conference on computer vision , pages 609– 617,

    2017

  19. [2022]

    Split learn- ing of multi-modal medical image classification

    [Ghosh et al., 2024] Bishwamittra Ghosh, Yuan Wang, Huazhu Fu, Qingsong Wei, Yong Liu, and Rick Siow Mong Goh. Split learn- ing of multi-modal medical image classification. In 2024 IEEE Conference on Artificial Intelligence (CAI) , pages 1326–1331,

    2024

  20. [2023]

    Ast: Audio spectrogram transformer

    [Gong et al., 2021] Yuan Gong, Yu-An Chung, and James Glass. Ast: Audio spectrogram transformer. arXiv preprint arXiv:2104.01778,

    2021 arXiv

  21. [2024]

    An image is worth 16x16 words: Transformers for image recognition at scale

    [Dosovitskiy et al., 2020] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at ...

    2020 arXiv

Pith tools

Reviewed August 8, 2026 · model on record in the stance chip above.