Pith. sign in

REVIEW 3 major objections 5 minor 42 references

FedVLMBench: Benchmarking Federated Fine-Tuning of Vision-Language Models

T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read FedVLMBench establishes that a 2-layer MLP connector with joint connector and LLM tuning is the best federated fine-tuning configuration for encoder-based vision-language models, and that vision-centric tasks are far more sensitive to…

desk verdict Useful new benchmark with two genuinely new multi-task federated datasets, but the vision-vs-text heterogeneity claim is confounded by unmatched partition severity and the paper needs variance reporting. read the letter →

arxiv 2506.09638 v1 pith:6ZJTMBBX submitted 2025-06-11 cs.LG cs.CV

Weiying Zheng , Ziyue Lin , Pengxin Guo , Yuyin Zhou , Feifei Wang , Liangqiong Qu This is my paper · ORCID
classification cs.LGcs.CV
keywords federatedlearningvision-languagemodelsfine-tuningbenchmarknon-IIDdataheterogeneitymultimodalmultitaskparameter-efficientconnectorLoRA
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper builds FedVLMBench, a systematic benchmark for federated fine-tuning of vision-language models that spans two architectures, four fine-tuning strategies, five federated algorithms, and six datasets. Using it, the authors try to establish practical rules: for encoder-based VLMs, a 2-layer MLP connector outperforms both a linear layer and a 6-layer MLP, and tuning the connector and LLM together beats tuning the connector first. They also claim that text-centric tasks such as VQA and captioning remain largely robust to non-IID client data, while vision-centric tasks such as classification and detection degrade sharply, and that existing FL optimizers do not fix this. In multi-task federated settings, however, both architectures reach near-centralized performance. If right, these results give practitioners concrete defaults and expose where new FL algorithms are needed.

What carries the argument

The load-bearing object is the vision-language connector, the trainable projection (linear, 2-layer MLP, or 6-layer MLP) that maps visual encoder features into the token space consumed by the LLM. The benchmark systematically varies connector depth, tuning order (connector-only, LLM-only, joint, two-stage), and data partition, making the connector the variable that carries the main takeaways. A second mechanism is the comparison between encoder-based and encoder-free architectures, which lets the paper attribute heterogeneity sensitivity to the presence or absence of a trainable alignment module.

What would settle it

Run the same four fine-tuning strategies and three connector depths on a held-out encoder-based VLM with a different pretrained visual encoder and a larger LLM, and on a real-world non-IID partition with more than three clients; the central claim would be falsified if the 2-layer MLP joint-tuning advantage disappears or if text-centric tasks show non-IID drops comparable to vision-centric ones.

Watch

Extended reading notes

Core claim

The paper's central discovery is that the connector between visual encoder and LLM is the decisive design point for federated fine-tuning of encoder-based VLMs. Among the tested configurations, a 2-layer MLP connector combined with simultaneous connector and LoRA-based LLM tuning yields the best balance of accuracy, stability, and computational cost. The paper further finds that vision-centric tasks suffer significantly larger performance drops under non-IID data than text-centric tasks, across both encoder-based and encoder-free VLM architectures, and that standard FL optimizers such as FedProx and FedYogi fail to recover this loss. In contrast, when each client is assigned a distinct task in a multitask setup, federated training approaches centralized-training quality, regardless of architecture.

Load-bearing premise

The results depend on LLaVA-1.5 with CLIP ViT-B/32 plus LLaMA3.2-3B and Show-O being representative of encoder-based and encoder-free VLMs, and on the paper's synthetic and simulated non-IID partitions standing in for real federated deployments.

Editorial extensions

If this is right

  • Federated deployments of encoder-based VLMs should default to a 2-layer MLP connector and simultaneous connector plus LoRA LLM tuning, since this combination balances accuracy, stability, and cost in the paper's experiments.
  • When a federated task is text-centric, most of the tuning budget should go to the LLM; when it is vision-centric, connector tuning matters at least as much as LLM tuning.
  • Existing FL optimizers such as FedProx and FedYogi will not recover the vision-centric accuracy lost under non-IID partitions; the paper's results call for new FL methods designed around the connector and visual features.
  • In multi-task deployments where each client owns one task, federated fine-tuning can approach centralized-training quality, so privacy-preserving multimodal systems are viable despite non-IID data.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A testable extension the paper does not run: varying the visual encoder and LLM scale would show whether the 2-layer MLP recommendation is tied to the specific LLaVA-1.5/LLaMA3.2-3B pairing or is a general property of encoder-based VLMs.
  • The pronounced vision-centric sensitivity could be caused by the connector being trained against inconsistent per-client visual distributions; a follow-up could test whether regularizing connector updates or aggregating connector parameters separately from LLM LoRA parameters reduces the drop.
  • The multi-task recovery result suggests that task-dedicated clients act like natural personalization; a follow-up could compare multi-task FL against personalized FL baselines on the same datasets to see whether the near-ceiling performance comes from task separation rather than from the FL algorithm.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper introduces FedVLMBench, a benchmark for federated fine-tuning of vision-language models (VLMs) that integrates two VLM architectures (encoder-based LLaVA-1.5 with CLIP ViT-B/32 and LLaMA3.2-3B; encoder-free Show-O), four fine-tuning strategies, five FL algorithms, and six datasets covering four single-task and two multitask scenarios. The main empirical claims are: (1) a 2-layer MLP connector with concurrent connector and LLM tuning is the optimal configuration for encoder-based VLMs in FL; (2) text-centric tasks benefit from LLM tuning while vision-centric tasks benefit from connector tuning; (3) vision-centric tasks are significantly more sensitive to data heterogeneity than text-centric tasks across both architectures; and (4) federated multitask training achieves near-ceiling performance comparable to centralized training. The authors also state that conventional FL optimizers are largely ineffective for vision-centric heterogeneity and call for new solutions.

Significance. If the findings hold, FedVLMBench is a useful resource: it provides public datasets and code, covers task types that existing FL-VLM benchmarks omit (report generation, visual grounding, detection), and includes both encoder-based and encoder-free architectures, which is a real gap in the literature. The paper also gives actionable guidance for practitioners, e.g., concurrent connector-LLM tuning and task-dependent tuning priorities. However, the strength of the contributions is currently diminished by the lack of statistical support and by a controlled-heterogeneity design that does not yet match the strength of the headline claims. The dataset and benchmark infrastructure are valuable, but the stated conclusions—especially the vision-versus-text sensitivity contrast and the optimal-connector claim—require additional experiments to be considered established.

major comments (3)
  1. [Section 5.3 / Takeaway 5, with Tables 4 and 5] The claim that vision-centric tasks are significantly more sensitive to data heterogeneity than text-centric tasks is confounded by unmatched partition rules and client counts across the compared datasets. Fed-SLAKE uses 3 clients partitioned by modality; Fed-ScienceCap uses up to 27 category-based clients; Fed-FGVC uses up to 30 manufacturer-based clients plus additional Dirichlet partitions; Fed-RadGenome uses 3 organ-based clients. Client number and partition rule directly control non-IID severity, so the larger relative drops on Fed-FGVC and Fed-RadGenome (e.g., encoder-free FedAvg accuracy 0.721 to 0.493 on FGVC and 0.604 to 0.485 on RadGenome) could reflect stronger partition-induced heterogeneity rather than a property of vision-centric tasks. To support Takeaway 5, the authors should compare tasks under matched heterogeneity, e.g., a common Dirichlet alpha with equal client counts, or at least report a quantitative heterogeneity metric (such as label-distribution divergence) for each split and demonstrate that the vision-centric drop is larger after controlling for that metric.
  2. [Section 5.2 / Table 3 and Takeaway 1] The comparison that selects the linear connector as suboptimal is biased by the authors' own admission that the linear connector results were 'derived from optimal hyperparameter tuning, including the selection of the most favorable random seeds,' while the MLP results are not described as receiving the same favorable-seed treatment. The text also states that the linear connector is 'highly susceptible to parameter initialization' and exhibits 'significant fluctuations,' but no variance or seed distribution is reported. Consequently, Table 3 does not provide a symmetric comparison and cannot support the conclusion that the 2-layer MLP is more effective and more stable. Please report mean and standard deviation across multiple seeds for all connector types, or apply the same seed-selection rule to every configuration, so that the comparison is fair.
  3. [Section 5.4 / Takeaway 6 and Table 6] The 'near-ceiling' conclusion for federated multitask learning is based on point estimates without error bars or significance tests. Several FedAvg results actually exceed the MT-Central baseline (e.g., encoder-free Fed-Nature VQA accuracy 0.781 vs. 0.752 and CIDER 0.930 vs. 0.912), which is possible but should be addressed explicitly. Without repeated runs and variance reporting, the reader cannot judge whether 'near-ceiling' is a real phenomenon or an artifact of a single seed. Please provide mean and standard deviation over at least three seeds, or a paired significance test, for the multitask results in Table 6.
minor comments (5)
  1. [Throughout] There are several typos and inconsistencies: 'LLAV A 1.5' in Section 5.1, 'as denoted as denoted as F-CL' in Section 5.2, 'Fed-RadGnome' in the Table 5 header, and inconsistent capitalization of 'LLaMA'/'LLAMA'.
  2. [References] References [34] and [35] appear to refer to the same paper (FedMLLM) with different years; please disambiguate or merge them.
  3. [Section 3 and Section 5.2] The text states that Fed-FGVC includes 'four heterogeneous partitions are generated using varying Dirichlet coefficients,' but the main tables report only a single IID/non-IID column, and the Dirichlet results are referenced as a figure in the supplementary material. Since the supplementary file is not part of this manuscript, please make the Dirichlet results (or a summary) available in the main text or an accessible appendix.
  4. [Tables 4 and 5] The 'Central' baseline is not fully defined in the main text; it is described as training on aggregated client data, but it is unclear whether it uses the same local-update schedule, LoRA rank, and number of communication rounds as the FL methods, and whether it is evaluated under IID or non-IID conditions. Please specify this in the experimental setup.
  5. [Table 3] The notation 'Mlp2x' and 'Mlp6x' is not defined in the main text; please define the number of layers and hidden sizes in the caption or in Section 5.1.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity found; the paper's claims are direct empirical observations from a benchmark, with no fitted-to-predicted loop, load-bearing self-citation, or definitional reduction.

full rationale

The paper's central claims are inductive generalizations drawn from controlled experiments over fixed model architectures, datasets, and FL algorithms. The optimal connector selection (Takeaway 1) is supported by direct comparisons in Table 3, and adopting the 2-layer MLP for later experiments is model selection, not a prediction of performance from a fitted input. The fine-tuning-strategy conclusions (Takeaways 2-3) and the heterogeneity-sensitivity findings (Takeaways 4-5) are direct comparisons of Table 4 and Table 5 entries under IID versus non-IID splits; the reported performance drops are measured from the benchmark runs. No parameter is fitted to a subset and then 'predicted' on a closely related quantity, and no connector or FL-method conclusion is hard-coded into the benchmark construction. The self-citations (e.g., refs [6-8,38,41]) appear in related-work and FL-method context and do not carry the benchmark's load; none is invoked as a uniqueness theorem or as the sole justification for an ansatz. The vision-vs-text sensitivity contrast may be vulnerable to a heterogeneity-severity confound because datasets use different client counts and partition rules, but that is an experimental-validity concern rather than circularity: the drops are empirical observations, not artifacts of a definition or an equation. Therefore no specific circular step can be exhibited, and the circularity score is 0.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

No new theoretical entities are introduced. The free parameters reflect experimental choices (seed selection and heterogeneity levels) that could affect the headline findings. The axioms are domain assumptions about representativeness of models, realism of multitask partitions, and standardness of metrics.

free parameters (2)
  • Random seeds for linear connector evaluation = not reported
    The paper states the linear connector's FL performance 'is derived from optimal hyperparameter tuning, including the selection of the most favorable random seeds,' so the comparison uses the best seed rather than average, a confounding factor for the connector recommendation.
  • Dirichlet heterogeneity coefficient alpha = not reported
    Non-IID partitions in Fed-FGVC and other datasets are generated with varying Dirichlet coefficients, but exact values are only in the supplementary figure; the sensitivity conclusions depend on these chosen heterogeneity levels.
assumptions (4)
  • domain assumption LLaVA-1.5 with CLIP ViT-B/32 and LLaMA3.2-3B is representative of encoder-based VLMs
    Section 5.1 selects this single architecture for all encoder-based experiments; conclusions about optimal connector may not transfer to other encoder-based VLMs.
  • domain assumption Show-O is representative of encoder-free VLMs
    Section 5.1 uses Show-O as the only encoder-free model; generality across encoder-free family is assumed.
  • domain assumption The client-task mapping in Fed-Nature and Fed-Med (one task per client) is a realistic model of federated multitask deployment
    Section 3 designs the two multitask datasets by assigning each dataset/task to a dedicated client; real deployments may have mixed tasks per client.
  • domain assumption Public dataset metrics (accuracy, CIDEr, ROUGE_L, IoU) are valid measures of task performance
    Evaluation uses standard metrics; no sensitivity analysis around metric choice.

how reviews work

0 comments
Cite this review

Pith. "Pith review of FedVLMBench: Benchmarking Federated Fine-Tuning of Vision-Language Models." pith.science (2026). https://pith.science/paper/6ZJTMBBX

@misc{pith2026250609638,
  author       = {Pith},
  title        = {Pith review of: FedVLMBench: Benchmarking Federated Fine-Tuning of Vision-Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/6ZJTMBBX}},
  note         = {Machine review of arXiv:2506.09638}
}
read the original abstract

Vision-Language Models (VLMs) have demonstrated remarkable capabilities in cross-modal understanding and generation by integrating visual and textual information. While instruction tuning and parameter-efficient fine-tuning methods have substantially improved the generalization of VLMs, most existing approaches rely on centralized training, posing challenges for deployment in domains with strict privacy requirements like healthcare. Recent efforts have introduced Federated Learning (FL) into VLM fine-tuning to address these privacy concerns, yet comprehensive benchmarks for evaluating federated fine-tuning strategies, model architectures, and task generalization remain lacking. In this work, we present \textbf{FedVLMBench}, the first systematic benchmark for federated fine-tuning of VLMs. FedVLMBench integrates two mainstream VLM architectures (encoder-based and encoder-free), four fine-tuning strategies, five FL algorithms, six multimodal datasets spanning four cross-domain single-task scenarios and two cross-domain multitask settings, covering four distinct downstream task categories. Through extensive experiments, we uncover key insights into the interplay between VLM architectures, fine-tuning strategies, data heterogeneity, and multi-task federated optimization. Notably, we find that a 2-layer multilayer perceptron (MLP) connector with concurrent connector and LLM tuning emerges as the optimal configuration for encoder-based VLMs in FL. Furthermore, current FL methods exhibit significantly higher sensitivity to data heterogeneity in vision-centric tasks than text-centric ones, across both encoder-free and encoder-based VLM architectures. Our benchmark provides essential tools, datasets, and empirical guidance for the research community, offering a standardized platform to advance privacy-preserving, federated training of multimodal foundation models.

Figures

Figures reproduced from arXiv: 2506.09638 by the authors.

Figure 1
Figure 1. Overview of our proposed FedVLMBench, which integrates two types of mainstream VLM [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

42 extracted references · 27 canonical work pages

  1. [1]

    Gpt-4 technical report.arXiv preprint arXiv:2303.08774, 2023

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report.arXiv preprint arXiv:2303.08774, 2023

    arXiv 2023

  2. [2]

    FedMBridge: Bridgeable multimodal federated learning

    Jiayi Chen and Aidong Zhang. FedMBridge: Bridgeable multimodal federated learning. InICML, 2024

    work page 2024

  3. [3]

    An image is worth 16x16 words: Transformers for image recognition at scale, 2021

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale, 2021

    2021

  4. [4]

    Personalized federated learning with theoretical guarantees: A model-agnostic meta-learning approach

    Alireza Fallah, Aryan Mokhtari, and Asuman Ozdaglar. Personalized federated learning with theoretical guarantees: A model-agnostic meta-learning approach. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors,Advances in Neural Information Processing Systems, volume 33, pages 3557–3568. Curran Associates, Inc., 2020

    work page 2020

  5. [5]

    Llama-adapter v2: Parameter-efficient visual instruction model.arXiv preprint arXiv:2304.15010, 2023

    Peng Gao, Jiaming Han, Renrui Zhang, Ziyi Lin, Shijie Geng, Aojun Zhou, Wei Zhang, Pan Lu, Conghui He, Xiangyu Yue, et al. Llama-adapter v2: Parameter-efficient visual instruction model.arXiv preprint arXiv:2304.15010, 2023

    arXiv 2023

  6. [6]

    Exploring the vulnerabilities of federated learning: A deep dive into gradient inversion attacks.arXiv preprint arXiv:2503.11514, 2025

    Pengxin Guo, Runxi Wang, Shuang Zeng, Jinjing Zhu, Haoning Jiang, Yanran Wang, Yuyin Zhou, Feifei Wang, Hui Xiong, and Liangqiong Qu. Exploring the vulnerabilities of federated learning: A deep dive into gradient inversion attacks.arXiv preprint arXiv:2503.11514, 2025

    work page arXiv 2025

  7. [7]

    A new federated learning framework against gradient inversion attacks

    Pengxin Guo, Shuang Zeng, Wenhao Chen, Xiaodan Zhang, Weihong Ren, Yuyin Zhou, and Liangqiong Qu. A new federated learning framework against gradient inversion attacks. InAAAI, volume 39, pages 16969–16977, 2025

    work page 2025

  8. [8]

    Selective aggregation for low-rank adaptation in federated learning

    Pengxin Guo, Shuang Zeng, Yanran Wang, Huijie Fan, Feifei Wang, and Liangqiong Qu. Selective aggregation for low-rank adaptation in federated learning. InICLR, 2025

    work page 2025

Show all 42 references
  1. [9]

    Measuring the effects of non-identical data distribu- tion for federated visual classification, 2019

    Tzu-Ming Harry Hsu, Hang Qi, and Matthew Brown. Measuring the effects of non-identical data distribu- tion for federated visual classification, 2019

    2019

  2. [10]

    Lora: Low-rank adaptation of large language models.ICLR, 1(2):3, 2022

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models.ICLR, 1(2):3, 2022

    2022

  3. [11]

    Fedlps: Heterogeneous federated learning for multiple tasks with local parameter sharing, 2024

    Yongzhe Jia, Xuyun Zhang, Amin Beheshti, and Wanchun Dou. Fedlps: Heterogeneous federated learning for multiple tasks with local parameter sharing, 2024

    2024

  4. [12]

    Alistair E. W. Johnson, Tom J. Pollard, Nathaniel R. Greenbaum, Matthew P. Lungren, Chih ying Deng, Yifan Peng, Zhiyong Lu, Roger G. Mark, Seth J. Berkowitz, and Steven Horng. Mimic-cxr-jpg, a large publicly available database of labeled chest radiographs, 2019

    2019

  5. [13]

    ReferItGame: Referring to objects in photographs of natural scenes

    Sahar Kazemzadeh, Vicente Ordonez, Mark Matten, and Tamara Berg. ReferItGame: Referring to objects in photographs of natural scenes. In Alessandro Moschitti, Bo Pang, and Walter Daelemans, editors, EMNLP, pages 787–798, Doha, Qatar, October 2014. Association for Computational ...

    2014

  6. [14]

    Federatedscope-llm: A comprehensive package for fine-tuning large language models in federated learning

    Weirui Kuang, Bingchen Qian, Zitao Li, Daoyuan Chen, Dawei Gao, Xuchen Pan, Yuexiang Xie, Yaliang Li, Bolin Ding, and Jingren Zhou. Federatedscope-llm: A comprehensive package for fine-tuning large language models in federated learning. InKDD, pages 5260–5271, 2024

    2024

  7. [15]

    A dataset of clinically generated visual questions and answers about radiology images.Scientific data, 5(1):1–10, 2018

    Jason J Lau, Soumya Gayen, Asma Ben Abacha, and Dina Demner-Fushman. A dataset of clinically generated visual questions and answers about radiology images.Scientific data, 5(1):1–10, 2018

    2018

  8. [16]

    Federated optimization in heterogeneous networks, 2020

    Tian Li, Anit Kumar Sahu, Manzil Zaheer, Maziar Sanjabi, Ameet Talwalkar, and Virginia Smith. Federated optimization in heterogeneous networks, 2020

    2020

  9. [17]

    Unifiedmllm: Enabling unified representation for multi-modal multi-tasks with large language model.arXiv preprint arXiv:2408.02503, 2024

    Zhaowei Li, Wei Wang, YiQing Cai, Xu Qi, Pengyu Wang, Dong Zhang, Hang Song, Botian Jiang, Zhida Huang, and Tao Wang. Unifiedmllm: Enabling unified representation for multi-modal multi-tasks with large language model.arXiv preprint arXiv:2408.02503, 2024

    2024 arXiv

  10. [18]

    Lawrence Zitnick, and Piotr Dollár

    Tsung-Yi Lin, Michael Maire, Serge Belongie, Lubomir Bourdev, Ross Girshick, James Hays, Pietro Perona, Deva Ramanan, C. Lawrence Zitnick, and Piotr Dollár. Microsoft coco: Common objects in context, 2015

    2015

  11. [19]

    Slake: A semantically-labeled knowledge-enhanced dataset for medical visual question answering, 2021

    Bo Liu, Li-Ming Zhan, Li Xu, Lin Ma, Yan Yang, and Xiao-Ming Wu. Slake: A semantically-labeled knowledge-enhanced dataset for medical visual question answering, 2021. 10

    2021

  12. [20]

    Visual instruction tuning.NIPS, 36:34892– 34916, 2023

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning.NIPS, 36:34892– 34916, 2023

    2023

  13. [21]

    Learn to explain: Multimodal reasoning via thought chains for science question answering, 2022

    Pan Lu, Swaroop Mishra, Tony Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering, 2022

    2022

  14. [22]

    Fine-grained visual classification of aircraft, 2013

    Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew Blaschko, and Andrea Vedaldi. Fine-grained visual classification of aircraft, 2013

    2013

  15. [23]

    Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Agüera y Arcas

    H. Brendan McMahan, Eider Moore, Daniel Ramage, Seth Hampson, and Blaise Agüera y Arcas. Communication-efficient learning of deep networks from decentralized data, 2023

    2023

  16. [24]

    Introducing meta llama 3: The most capable openly available llm to date.Meta AI, 2024

    AI Meta. Introducing meta llama 3: The most capable openly available llm to date.Meta AI, 2024

    2024

  17. [25]

    Learning transferable visual models from natural language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. InICML, pages 8748–8763. PmLR, 2021

    2021

  18. [26]

    Learning transferable visual models from natural language supervision, 2021

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision, 2021

    2021

  19. [27]

    Adaptive federated optimization.arXiv preprint arXiv:2003.00295, 2020

    Sashank Reddi, Zachary Charles, Manzil Zaheer, Zachary Garrett, Keith Rush, Jakub Kone ˇcn`y, Sanjiv Kumar, and H Brendan McMahan. Adaptive federated optimization.arXiv preprint arXiv:2003.00295, 2020

    2003 arXiv

  20. [28]

    Brendan McMahan

    Sashank Reddi, Zachary Charles, Manzil Zaheer, Zachary Garrett, Keith Rush, Jakub Kone ˇcný, Sanjiv Kumar, and H. Brendan McMahan. Adaptive federated optimization, 2021

    2021

  21. [29]

    Exploring models and data for image question answering, 2015

    Mengye Ren, Ryan Kiros, and Richard Zemel. Exploring models and data for image question answering, 2015

    2015

  22. [30]

    Chameleon: Mixed-modal early-fusion foundation models, 2025

    Chameleon Team. Chameleon: Mixed-modal early-fusion foundation models, 2025

    2025

  23. [31]

    Gemini: a family of highly capable multimodal models.arXiv preprint arXiv:2312.11805, 2023

    Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, Katie Millican, et al. Gemini: a family of highly capable multimodal models.arXiv preprint arXiv:2312.11805, 2023

    2023 arXiv

  24. [32]

    Show-o: One single transformer to unify multimodal understanding and generation.arXiv preprint arXiv:2408.12528, 2024

    Jinheng Xie, Weijia Mao, Zechen Bai, David Junhao Zhang, Weihao Wang, Kevin Qinghong Lin, Yuchao Gu, Zhijie Chen, Zhenheng Yang, and Mike Zheng Shou. Show-o: One single transformer to unify multimodal understanding and generation.arXiv preprint arXiv:2408.12528, 2024

    2024 arXiv

  25. [33]

    Pilot: Building the federated multimodal instruction tuning framework, 2025

    Baochen Xiong, Xiaoshan Yang, Yaguang Song, Yaowei Wang, and Changsheng Xu. Pilot: Building the federated multimodal instruction tuning framework, 2025

    2025

  26. [34]

    Fedmllm: Federated fine-tuning mllm on multimodal heterogeneity data.arXiv preprint arXiv:2411.14717, 2024

    Binqian Xu, Xiangbo Shu, Haiyang Mei, Guosen Xie, Basura Fernando, and Jinhui Tang. Fedmllm: Federated fine-tuning mllm on multimodal heterogeneity data.arXiv preprint arXiv:2411.14717, 2024

    2024 arXiv

  27. [35]

    Fedmllm: Federated fine-tuning mllm on multimodal heterogeneity data, 2025

    Binqian Xu, Xiangbo Shu, Haiyang Mei, Guosen Xie, Basura Fernando, and Jinhui Tang. Fedmllm: Federated fine-tuning mllm on multimodal heterogeneity data, 2025

    2025

  28. [36]

    Fedllm- bench: Realistic benchmarks for federated learning of large language models.NIPS, 37:111106–111130, 2024

    Rui Ye, Rui Ge, Xinyu Zhu, Jingyi Chai, Du Yaxin, Yang Liu, Yanfeng Wang, and Siheng Chen. Fedllm- bench: Realistic benchmarks for federated learning of large language models.NIPS, 37:111106–111130, 2024

    2024

  29. [37]

    Openfedllm: Training large language models on decentralized private data via federated learning

    Rui Ye, Wenhao Wang, Jingyi Chai, Dihan Li, Zexi Li, Yinda Xu, Yaxin Du, Yanfeng Wang, and Siheng Chen. Openfedllm: Training large language models on decentralized private data via federated learning. In KDD, pages 6137–6147, 2024

    2024

  30. [38]

    Tackling data heterogeneity in federated learning via loss decomposition

    Shuang Zeng, Pengxin Guo, Shuai Wang, Jianbo Wang, Yuyin Zhou, and Liangqiong Qu. Tackling data heterogeneity in federated learning via loss decomposition. InMICAI, pages 707–717. Springer, 2024

    2024

  31. [39]

    Fedtgp: Trainable global prototypes with adaptive- margin-enhanced contrastive learning for data and model heterogeneity in federated learning, 2024

    Jianqing Zhang, Yang Liu, Yang Hua, and Jian Cao. Fedtgp: Trainable global prototypes with adaptive- margin-enhanced contrastive learning for data and model heterogeneity in federated learning, 2024

    2024

  32. [40]

    Mllm- llava-fl: Multimodal large language model assisted federated learning

    Jianyi Zhang, Hao Yang, Ang Li, Xin Guo, Pu Wang, Haiming Wang, Yiran Chen, and Hai Li. Mllm- llava-fl: Multimodal large language model assisted federated learning. InWACV, pages 4066–4076. IEEE, 2025. 11

    2025

  33. [41]

    Flhetbench: Benchmarking device and state heterogeneity in federated learning

    Junyuan Zhang, Shuang Zeng, Miao Zhang, Runxi Wang, Feifei Wang, Yuyin Zhou, Paul Pu Liang, and Liangqiong Qu. Flhetbench: Benchmarking device and state heterogeneity in federated learning. InCVPR, pages 12098–12108, 2024

    2024

  34. [42]

    Radgenome-chest ct: A grounded vision-language dataset for chest ct analysis, 2024

    Xiaoman Zhang, Chaoyi Wu, Ziheng Zhao, Jiayu Lei, Ya Zhang, Yanfeng Wang, and Weidi Xie. Radgenome-chest ct: A grounded vision-language dataset for chest ct analysis, 2024. 12

    2024

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.