Pith. sign in

REVIEW 4 major objections 6 minor 1 cited by

Dual-Priv Pruning : Efficient Differential Private Fine-Tuning in Multimodal Large Language Models

T0 review · 4 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Dual-Priv Pruning claims to make differentially private fine-tuning practical for 7B multimodal LLMs by pruning visual tokens and selectively applying noisy gradient updates, achieving a formal (epsilon, delta)-DP guarantee with about…

desk verdict The DP argument is sound, but the utility claims are not yet credible: the pruning ratios are tuned on the test set and no results have error bars. read the letter →

arxiv 2506.07077 v1 pith:GICGZKEK submitted 2025-06-08 cs.CR cs.AI

Qianshan Wei , Jiaqi Li , Zihan You , Yi Zhan , Kecen Li , Jialin Wu , Xinfeng Li Hengjun Liu , Yi Yu
show 4 more authors
Bin Cao Yiwen Xu Yang Liu Guilin Qi
This is my paper
classification cs.CRcs.AI
keywords differentialprivacyDP-SGDmultimodallargelanguagemodelsvisualtokenpruninggradient-updateparameter-efficientfine-tuningLoRAmembershipinferenceattack
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

DP-SGD is notoriously harsh for multimodal LLMs because the noise needed for privacy scales with parameter count, and long visual token sequences inflate computation. This paper proposes Dual-Priv Pruning, which first prunes and fuses visual tokens to shrink the input, then, after DP noise is added, updates only the parameter blocks whose noisy-gradient norms rank in the top 80%, treating that mask as post-processing so the formal (epsilon, delta)-DP guarantee is untouched. The authors report competitive or better accuracy than DP-SGD across ScienceQA, TextVQA, GQA, MME-RealWorld, PathVQA, and VQA-RAD, especially at epsilon=1, while reducing peak GPU memory by about 14.34%. If correct, the paper would be the first demonstration that DP fine-tuning is practical for a 7B multimodal model under strict privacy budgets.

What carries the argument

The load-bearing object is the binary update mask M computed after noise injection: the noisy gradient is partitioned into logical parameter blocks, each block's L2 norm is compared, and only blocks in the top K% receive updates via M ⊙ g̃. Because M depends only on the already-private noisy gradient, the mask is post-processing and the (epsilon, delta)-DP guarantee is preserved. The complementary mechanism is attention-based visual token selection plus fusion of non-dominant tokens into noisy context tokens, which shortens the visual sequence before DP-SGD runs.

What would settle it

Compare the top-80% block selection against selecting the same number of blocks at random, or the bottom 80%, under identical DP noise and budget: if random or bottom selection matches its accuracy, the norm-based signal assumption is not doing the work. A direct check is to correlate each block's pre-noise gradient norm with its post-noise norm across training steps; low correlation means noise, not signal, drives the mask.

Watch

Extended reading notes

Core claim

Dual-Priv Pruning is claimed to make differentially private fine-tuning of a 7B MLLM workable: it keeps a formal (epsilon, delta)-DP guarantee while improving utility and memory. The privacy guarantee comes entirely from a DP-SGD step applied to LoRA parameters, and the adaptive update mask is derived from the noisy gradient alone, so by the post-processing property it cannot weaken the guarantee. The paper's empirical claim is that this post-noise gating, updating the top 80% of parameter blocks by noisy-gradient L2 norm, improves accuracy relative to updating all blocks, and that the visual-token pruning and fusion stage lowers compute with minimal information loss. On its benchmarks the method matches or beats DP-SGD in most settings, beats DPZO by a wide margin, and lowers peak GPU memory by about 14.34%.

Load-bearing premise

The claim that updating only the blocks whose noisy gradients have the largest L2 norms improves utility rests on the assumption that a large post-noise norm means a strong pre-noise signal; if noise itself produces large norms, the method could systematically discard the most informative updates.

Editorial extensions

If this is right

  • Formal (epsilon, delta)-DP fine-tuning of a 7B multimodal LLM is achievable at epsilon as low as 1 with accuracy close to the non-private baseline on several benchmarks.
  • Gradient-update pruning can be layered onto any DP-SGD or LoRA pipeline without consuming extra privacy budget, since the mask is post-processing on the noisy gradient.
  • Input-side visual token pruning reduces the cost of per-sample gradient computation, which is the dominant expense in DP-SGD for multimodal inputs.
  • The membership-inference results suggest that the fused, heuristically noised visual context adds practical hardening on top of the formal DP guarantee.
  • If the top-80% update rule generalizes, sparse noisy updates are not just a memory saving but a utility win under high noise.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper's claims, the same post-processing argument would allow any data-independent or noisy-gradient-dependent mask, so the method could be combined with structured sparsity, top-k by block variance, or masks derived from public reference data.
  • The heuristic that large noisy-gradient norms mark reliable signal is testable across model scales; if it holds mainly because LoRA blocks have heterogeneous sizes, different block granularities could change the ranking and the optimal K%.
  • The authors only test 7B models, as they note in Appendix J, so the utility and memory gains remain unverified for larger MLLMs and for full fine-tuning rather than LoRA.
  • Matching the heuristic input noise variance to the DP noise variance is a tunable regularizer; varying sigma_fuse relative to sigma could be studied for its effect on both utility and membership-inference resistance.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes Dual-Priv Pruning, a framework for differentially private fine-tuning of multimodal LLMs. Mechanism 1 prunes and fuses visual tokens via CLS attention and adds heuristic Gaussian noise; Mechanism 2 applies DP-SGD with a mask that keeps only the top-K% of logical blocks by L2 norm of the noisy gradient. The authors argue that the masking step is post-processing and therefore the pipeline inherits (epsilon, delta)-DP from DP-SGD. Experiments on ScienceQA, TextVQA, GQA, MME-RealWorld, PathVQA, and VQA-RAD compare against DP-SGD and DPZO, and the paper reports memory reductions and membership-inference-attack results.

Significance. The paper addresses a timely problem, and the observation that gradient masking on the noisy gradient is post-processing is correct in isolation. The two-pruning view is sensible, and the memory/utility claims would be useful if substantiated. However, as written the formal privacy guarantee is invalid because Mechanism 1 is an arbitrary data-dependent preprocessing step on the private images before the DP mechanism runs, and the empirical evaluation does not support the reported gains because of test-set tuning, lack of error bars, and a collapsed DPZO baseline.

major comments (4)
  1. [Section 4.3 and Appendix D] The overall (epsilon, delta)-DP claim rests on a misapplication of post-processing. The post-processing property applies to a function f applied to the output A(D) of a DP mechanism, but here Mechanism 1 computes attention maps (Eq. (5)) and selects/fuses tokens (Eqs. (6)-(8)) from the private image I before DP-SGD is run. Since V' = M1(I) is data-dependent, the input to DP-SGD is a function of the private dataset, and the DP guarantee of DP-SGD with respect to V' does not imply a guarantee with respect to the original image-text pair (I, T). Section 4.3's statement that Mechanism 1 'does not consume privacy budget' is therefore incorrect; the entire pipeline A(D) = M2(M1(D)) needs a privacy analysis over the original neighboring datasets, not only over the pruned representation.
  2. [Section 5.1, Figure 3, Tables 1 and 4] The main empirical claim is compromised by selection on the test set and lack of statistical significance. The two pruning ratios used in all experiments (80% gradient blocks, 40% visual tokens) are chosen from Figure 3, which is computed on ScienceQA at epsilon = 1, and the same ScienceQA test set is then used for the headline results in Tables 1 and 4. The checklist explicitly answers 'No' to whether error bars or statistical significance are reported. Without multiple seeds or confidence intervals, the reported differences (e.g., 84.20 vs. 82.80 in Table 1) cannot be distinguished from noise.
  3. [Appendix F] The core utility premise of Mechanism 2 is asserted but not verified. The paper assumes that a large noisy-gradient block norm N_j means the true gradient signal survived the noise, but for a high-dimensional block the noise term dominates: E||g + z||^2 = ||g||^2 + sigma^2 d, so the ranking of noisy norms is mostly a ranking of noise magnitudes unless ||g|| is large relative to sigma sqrt(d). No experiment or analysis shows a correlation between the top noisy-norm blocks and the top true-gradient-norm blocks; if the ranking is noise-driven, the mask discards informative updates and the reported utility gains are not a reliable consequence of the method.
  4. [Tables 1-3 and Section 5.5] The DPZO baseline is not an informative comparison: it achieves 0.00 on GQA and VQA-RAD (Tables 1, 2) and near-zero performance on MME-RealWorld (Table 3), so the claim that Dual-Priv uses 'only 1.74% more memory' than DPZO is a comparison against a method that is not functioning as a useful baseline in this setting. The memory results also need a clear statement of whether peak or average memory is reported and how it is measured.
minor comments (6)
  1. [Section 5.5 and Abstract] The abstract and introduction claim a reduction in peak GPU memory, while Section 5.5 and Figure 2 refer to average GPU memory usage; please clarify which quantity is measured and report the variance.
  2. [Checklist item 5 and Abstract] The checklist says code is included in the supplemental material, but the abstract says 'Our code is coming soon'; this inconsistency should be resolved.
  3. [Table 2] The header 'BLUE' should be 'BLEU' in Table 2 and Table 7.
  4. [Section 5.1] The configuration of Mechanism 1 says K = 191 attention-selected tokens plus [CLS] and 30 fused tokens, but the attention layer used to compute the CLS attention maps is not specified; please provide this detail for reproducibility.
  5. [Section 2 and Abstract] The claim of being 'the first to explore DP fine-tuning in MLLMs' should be qualified, since prior works on differentially private CLIP training and low-rank private learning already address multimodal settings.
  6. [Figure 2] The x-axis label 'SciencQA' is a typo for 'ScienceQA', and the axis values in the figure are hard to read at the current resolution.

Circularity Check

1 steps flagged · score 4.0 of 10

The formal DP guarantee is self-contained and sound (the mask is post-processing of the noisy gradient), but the reported ScienceQA utility advantage is partially circular because the pruning ratios are selected on the ScienceQA test set and then evaluated on the same set.

  1. fitted input called prediction [Section 5.7 / Figure 3; Section 5.1 'Dual-Priv Pruning Configuration'; Table 1 (ScienceQA, epsilon=1)]
    "Section 5.1: 'We evaluate performance by fine-tuning on the training sets and evaluating on the test sets of several vision-language benchmarks. ... Mechanism 2 (Section 4.2) employs gradient-update pruning by selecting parameter blocks for update if their noisy gradient norms are among the top 80% of all block norms (Eq. (10)).' Section 5.7: 'We examine the impact of different pruning ratios ... on the ScienceQA dataset ( ϵ = 1 ). Figure 3 (a) ... The ACC peaks at 84.20 when the top 80% of blocks are updated.'"

    The gradient-update ratio (80%) and visual-token retention ratio (40%) are chosen by maximizing ACC/IMG on the ScienceQA test set, the same split used in Table 1 to report 'our method maintains a clear advantage on ScienceQA (84.20 vs 81.54)' at epsilon=1. The reported ScienceQA superiority is therefore the value of the tuned objective evaluated on the tuning set, not an independent prediction. This is a data-fitting loop in the utility evaluation. The privacy claim is not affected: the mask is computed solely from the noisy gradient, so the post-processing argument in Section 4.3 and Appendix D is a valid application of the DP post-processing lemma.

full rationale

I walked the derivation chain for the central privacy claim. The analysis follows DP-SGD: per-sample clipping bounds sensitivity, Gaussian noise is added to the averaged gradient, and the binary mask (Eq. 10) is a deterministic function of the noisy gradient, so the final update (Eq. 11) is post-processing of an already private quantity. This is a correct application of the post-processing property, and the RDP accounting is standard. Mechanism 1 is a per-sample input transform outside the formal budget, and it is explicitly not claimed to provide DP. No self-citation chain or imported uniqueness theorem is load-bearing. I find no formal circularity in the privacy proof. The only concrete reduction I can exhibit is in the empirical utility evaluation: the two pruning ratios are shown to be chosen on the ScienceQA test set (Section 5.7) and the same test set is used to demonstrate the method's advantage in Table 1. That makes the ScienceQA utility claim partially fitted rather than predicted; other benchmarks (TextVQA, GQA, MME-RealWorld, PathVQA, VQA-RAD) were not used for ratio selection, so the method retains independent evidence. Given the formal result is self-contained and only the ScienceQA utility comparison is compromised, a moderate score is appropriate.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

The paper introduces no new physical or mathematical entities. The free parameters are the two pruning ratios (80% and 40%) and the heuristic noise variance. The key assumptions are standard DP-SGD clipping, the per-record fixed nature of the visual preprocessing, and the post-processing property for gradient gating.

free parameters (3)
  • Top-K% gradient blocks updated = 80%
    Selected based on ScienceQA (epsilon=1) accuracy in Figure 3a; peak at 80%. This ratio is tuned on the evaluation set and directly controls the gradient-update pruning mechanism.
  • Visual token retention ratio = 40% (K=191 plus 30 fused tokens)
    Selected based on ScienceQA (epsilon=1) IMG in Figure 3b; peak at 40%. This ratio controls Mechanism 1 and is tuned on the evaluation set.
  • sigma_fuse = set equal to per-step DP noise variance
    Variance of the heuristic Gaussian noise added to fused visual tokens. It is not tuned, but it is a design choice that could affect utility and is not derived from first principles.
assumptions (3)
  • standard math Per-sample gradient clipping bounds sensitivity, ||g_i||_2 <= C
    Standard DP-SGD assumption, stated in Section 3.1.1 and Appendix B, used to calibrate the Gaussian mechanism noise.
  • domain assumption Mechanism 1 is a fixed per-record transformation independent across samples
    Section 4.3 claims Mechanism 1 'does not consume privacy budget', but the paper does not explicitly prove that the attention-based token selection and fusion is a per-record fixed function, which is needed for neighboring datasets to map to neighboring transformed datasets under DP-SGD.
  • standard math Masking the noisy gradient is post-processing
    Correctly relies on the DP post-processing property (Appendix D) to claim the gradient-update mask does not affect the privacy guarantee, since the mask is a function of the private noisy gradient only.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Dual-Priv Pruning : Efficient Differential Private Fine-Tuning in Multimodal Large Language Models." pith.science (2026). https://pith.science/paper/GICGZKEK

@misc{pith2026250607077,
  author       = {Pith},
  title        = {Pith review of: Dual-Priv Pruning : Efficient Differential Private Fine-Tuning in Multimodal Large Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/GICGZKEK}},
  note         = {Machine review of arXiv:2506.07077}
}
read the original abstract

Differential Privacy (DP) is a widely adopted technique, valued for its effectiveness in protecting the privacy of task-specific datasets, making it a critical tool for large language models. However, its effectiveness in Multimodal Large Language Models (MLLMs) remains uncertain. Applying Differential Privacy (DP) inherently introduces substantial computation overhead, a concern particularly relevant for MLLMs which process extensive textual and visual data. Furthermore, a critical challenge of DP is that the injected noise, necessary for privacy, scales with parameter dimensionality, leading to pronounced model degradation; This trade-off between privacy and utility complicates the application of Differential Privacy (DP) to complex architectures like MLLMs. To address these, we propose Dual-Priv Pruning, a framework that employs two complementary pruning mechanisms for DP fine-tuning in MLLMs: (i) visual token pruning to reduce input dimensionality by removing redundant visual information, and (ii) gradient-update pruning during the DP optimization process. This second mechanism selectively prunes parameter updates based on the magnitude of noisy gradients, aiming to mitigate noise impact and improve utility. Experiments demonstrate that our approach achieves competitive results with minimal performance degradation. In terms of computational efficiency, our approach consistently utilizes less memory than standard DP-SGD. While requiring only 1.74% more memory than zeroth-order methods which suffer from severe performance issues on A100 GPUs, our method demonstrates leading memory efficiency on H20 GPUs. To the best of our knowledge, we are the first to explore DP fine-tuning in MLLMs. Our code is coming soon.

Figures

Figures reproduced from arXiv: 2506.07077 by the authors.

Figure 1
Figure 1. Overview of our Dual-Priv Pruning. (Left): Visual Token Pruning and Fusion. Using [CLS] attention, dominant tokens are selected; less important ones are averaged with heuristic noise. (Right): DP Fine-tuning with gradient pruning. Noise is added to gradients in LLM blocks, and updates are selectively applied based on noisy gradient magnitude. Frozen parameters remain unchanged. To identify these dominant tokens, we … view at source ↗
Figure 2
Figure 2. Average GPU memory consumption (in GB) during fine-tuning for DPZO, DP-SGD, and [PITH_FULL_IMAGE:figures/full_fig_p009_2.png] view at source ↗
Figure 3
Figure 3. Pruning ratios impacts on ScienceQA (ϵ = 1). (a) percentage of top K% gradient blocks updated (Mechanism 2). (b) percentage of visual tokens retained (Mechanism 1). We examine the impact of different pruning ra￾tios within the Dual-Priv Pruning framework on the ScienceQA dataset (ϵ = 1) [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗
Figures from the paper (1 more)
Figure 4
Figure 4. Figure 4: Radar chart of AUC under varying Rényi entropy orders and top en￾tropy percentages. Metrics use strict pri￾vacy budget (ϵ= 1). Distribution places smaller values near edges. To further test the privacy protection capability of our ap￾proach, we validate the performance…

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. PANOPTICON: A PII-Based Assemblage of Naturalistic Output Tokens for Investigating Privacy Leakage Within LLM Context Window

    cs.AI 2026-07 conditional novelty 6.0 of 10

    PANOPTICON is a synthetic benchmark of 67,718 PII-laden prompts for measuring inference-time privacy leakage in LLMs, but its realism and label accuracy are not externally validated.

Reference graph

Works this paper leans on

71 extracted references · 32 canonical work pages · cited by 1 Pith paper

  1. [1]

    Deep learning with differential privacy

    Martin Abadi, Andy Chu, Ian Goodfellow, H Brendan McMahan, Ilya Mironov, Kunal Talwar, and Li Zhang. Deep learning with differential privacy. InProceedings of the 2016 ACM SIGSAC conference on computer and communications security, pages 308–318, 2016

    2016

  2. [2]

    Phi-3 technical report: A highly capable language model locally on your phone

    Marah Abdin, Jyoti Aneja, Hany Awadalla, Ahmed Awadallah, Ammar Ahmad Awan, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Jianmin Bao, Harkirat Behl, et al. Phi-3 technical report: A highly capable language model locally on your phone. arXiv preprint arXiv:2404.14219, 2024

    arXiv 2024

  3. [3]

    Flamingo: a visual language model for few-shot learning

    Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. Advances in neural information processing systems, 35: 23716–23736, 2022

    2022

  4. [4]

    Qwen-vl: A frontier large vision-language model with versatile abilities

    Jinze Bai, Shuai Bai, Shusheng Yang, Shijie Wang, Sinan Tan, Peng Wang, Junyang Lin, Chang Zhou, and Jingren Zhou. Qwen-vl: A frontier large vision-language model with versatile abilities. arXiv preprint arXiv:2308.12966, 1(2):3, 2023

    arXiv 2023

  5. [5]

    Training with noise is equivalent to tikhonov regularization

    Chris M Bishop. Training with noise is equivalent to tikhonov regularization. Neural computa- tion, 7(1):108–116, 1995

    1995

  6. [6]

    An image is worth 1/2 tokens after layer 2: Plug-and-play inference acceleration for large vision-language models

    Liang Chen, Haozhe Zhao, Tianyu Liu, Shuai Bai, Junyang Lin, Chang Zhou, and Baobao Chang. An image is worth 1/2 tokens after layer 2: Plug-and-play inference acceleration for large vision-language models. In Ales Leonardis, Elisa Ricci, Stefan Roth, Olga Russakovsky, Torsten Sattler, and Gül Varol, editors, Computer Vision - ECCV 2024 - 18th European Con...

    2024

  7. [7]

    Instructblip: Towards general-purpose vision-language models with instruction tuning, 2023

    Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. Instructblip: Towards general-purpose vision-language models with instruction tuning, 2023. URL https://arxiv.org/abs/2305. 06500

    2023

  8. [8]

    Security and privacy challenges of large language models: A survey

    Badhan Chandra Das, M Hadi Amini, and Yanzhao Wu. Security and privacy challenges of large language models: A survey. ACM Computing Surveys, 57(6):1–39, 2025

    work page 2025

Show all 71 references
  1. [9]

    Differential privacy

    Cynthia Dwork. Differential privacy. In International colloquium on automata, languages, and programming, pages 1–12. Springer, 2006

    2006

  2. [10]

    The algorithmic foundations of differential privacy

    Cynthia Dwork, Aaron Roth, et al. The algorithmic foundations of differential privacy. Founda- tions and Trends® in Theoretical Computer Science, 9(3–4):211–407, 2014

    2014

  3. [11]

    Differentially private steering for large language model alignment

    Anmol Goel, Yaxi Hu, Iryna Gurevych, and Amartya Sanyal. Differentially private steering for large language model alignment. arXiv preprint arXiv:2501.18532, 2025

    2025 arXiv

  4. [12]

    Which tokens to use? investigating token reduction in vision transformers

    Joakim Bruslund Haurum, Sergio Escalera, Graham W Taylor, and Thomas B Moeslund. Which tokens to use? investigating token reduction in vision transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 773–783, 2023

    2023

  5. [13]

    Pathvqa: 30000+ questions for medical visual question answering

    Xuehai He, Yichen Zhang, Luntian Mou, Eric Xing, and Pengtao Xie. Pathvqa: 30000+ questions for medical visual question answering. arXiv preprint arXiv:2003.10286, 2020

    2003 arXiv

  6. [14]

    Lora: Low-rank adaptation of large language models

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. ICLR, 1 (2):3, 2022

    2022

  7. [15]

    Safeguard- ing data in multimodal ai: A differentially private approach to clip training

    Alyssa Huang, Peihan Liu, Ryumei Nakada, Linjun Zhang, and Wanrong Zhang. Safeguard- ing data in multimodal ai: A differentially private approach to clip training. arXiv preprint arXiv:2306.08173, 2023. 11

    2023 arXiv

  8. [16]

    Gqa: A new dataset for real-world visual reasoning and compositional question answering

    Drew A Hudson and Christopher D Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 6700–6709, 2019

    2019

  9. [17]

    End-to-end privacy preserving deep learning on multi-institutional medical imaging

    Georgios Kaissis, Alexander Ziller, Jonathan Passerat-Palmbach, Théo Ryffel, Dmitrii Usynin, Andrew Trask, Ionésio Lima Jr, Jason Mancuso, Friederike Jungmann, Marc-Matthias Steinborn, et al. End-to-end privacy preserving deep learning on multi-institutional medical imaging. N...

    2021

  10. [18]

    Differentially private language models benefit from public pre-training

    Gavin Kerrigan, Dylan Slack, and Jens Tuyls. Differentially private language models benefit from public pre-training. arXiv preprint arXiv:2009.05886, 2020

    2009 arXiv

  11. [19]

    Adam: A method for stochastic optimization

    Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014

    2014 arXiv

  12. [20]

    Spvit: Enabling faster vision transformers via latency- aware soft token pruning

    Zhenglun Kong, Peiyan Dong, Xiaolong Ma, Xin Meng, Wei Niu, Mengshu Sun, Xuan Shen, Geng Yuan, Bin Ren, Hao Tang, et al. Spvit: Enabling faster vision transformers via latency- aware soft token pruning. In European conference on computer vision, pages 620–640. Springer, 2022

    2022

  13. [21]

    A dataset of clinically generated visual questions and answers about radiology images

    Jason J Lau, Soumya Gayen, Asma Ben Abacha, and Dina Demner-Fushman. A dataset of clinically generated visual questions and answers about radiology images. Scientific data, 5(1): 1–10, 2018

    2018

  14. [22]

    Llava-med: Training a large language-and-vision assistant for biomedicine in one day

    Chunyuan Li, Cliff Wong, Sheng Zhang, Naoto Usuyama, Haotian Liu, Jianwei Yang, Tristan Naumann, Hoifung Poon, and Jianfeng Gao. Llava-med: Training a large language-and-vision assistant for biomedicine in one day. Advances in Neural Information Processing Systems, 36: 28541–2...

    2023

  15. [23]

    Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models

    Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In International conference on machine learning, pages 19730–19742. PMLR, 2023

    2023

  16. [24]

    Fine-tuning lan- guage models with differential privacy through adaptive noise allocation

    Xianzhi Li, Ran Zmigrod, Zhiqiang Ma, Xiaomo Liu, and Xiaodan Zhu. Fine-tuning lan- guage models with differential privacy through adaptive noise allocation. arXiv preprint arXiv:2410.02912, 2024

    2024 arXiv

  17. [25]

    Large language models can be strong differentially private learners

    Xuechen Li, Florian Tramer, Percy Liang, and Tatsunori Hashimoto. Large language models can be strong differentially private learners. arXiv preprint arXiv:2110.05679, 2021

    2021 arXiv

  18. [26]

    Membership inference attacks against large vision-language models

    Zhan Li, Yongtao Wu, Yihang Chen, Francesco Tonin, Elias Abad Rocamora, and V olkan Cevher. Membership inference attacks against large vision-language models. Advances in Neural Information Processing Systems, 37:98645–98674, 2024

    2024

  19. [27]

    Visual instruction tuning.Advances in neural information processing systems, 36:34892–34916, 2023

    Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning.Advances in neural information processing systems, 36:34892–34916, 2023

    2023

  20. [28]

    Differentially private zeroth-order methods for scalable large language model finetuning

    Zhihao Liu, Jian Lou, Wenjie Bao, Yuke Hu, Bo Li, Zhan Qin, and Kui Ren. Differentially private zeroth-order methods for scalable large language model finetuning. arXiv preprint arXiv:2402.07818, 2024

    2024 arXiv

  21. [29]

    Learn to explain: Multimodal reasoning via thought chains for science question answering

    Pan Lu, Swaroop Mishra, Tanglin Xia, Liang Qiu, Kai-Wei Chang, Song-Chun Zhu, Oyvind Tafjord, Peter Clark, and Ashwin Kalyan. Learn to explain: Multimodal reasoning via thought chains for science question answering. Advances in Neural Information Processing Systems, 35: 2507–2...

    2022

  22. [30]

    Differentially private video activity recognition

    Zelun Luo, Yuliang Zou, Yijin Yang, Zane Durante, De-An Huang, Zhiding Yu, Chaowei Xiao, Li Fei-Fei, and Animashree Anandkumar. Differentially private video activity recognition. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 6657–66...

    2024 arXiv

  23. [31]

    Learning differentially private recurrent language models

    H Brendan McMahan, Daniel Ramage, Kunal Talwar, and Li Zhang. Learning differentially private recurrent language models. arXiv preprint arXiv:1710.06963, 2017. 12

    2017 arXiv

  24. [32]

    The impact of multimodal large language models on health care’s future

    Bertalan Meskó. The impact of multimodal large language models on health care’s future. Journal of medical Internet research, 25:e52865, 2023

    2023

  25. [33]

    Rényi differential privacy

    Ilya Mironov. Rényi differential privacy. In 30th IEEE Computer Security Foundations Symposium, CSF 2017, Santa Barbara, CA, USA, August 21-25, 2017, pages 263–275. IEEE Computer Society, 2017. doi: 10.1109/CSF.2017.11. URL https://doi.org/10.1109/CSF. 2017.11

    2017 doi

  26. [34]

    Regularizing deep neu- ral networks by noise: Its interpretation and optimization

    Hyeonwoo Noh, Tackgeun You, Jonghwan Mun, and Bohyung Han. Regularizing deep neu- ral networks by noise: Its interpretation and optimization. Advances in neural information processing systems, 30, 2017

    2017

  27. [35]

    Language models are unsupervised multitask learners

    Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. Language models are unsupervised multitask learners. 2019

    2019

  28. [36]

    Learning transferable visual models from natural language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pa...

    2021

  29. [37]

    Dynamicvit: Efficient vision transformers with dynamic token sparsification

    Yongming Rao, Wenliang Zhao, Benlin Liu, Jiwen Lu, Jie Zhou, and Cho-Jui Hsieh. Dynamicvit: Efficient vision transformers with dynamic token sparsification. Advances in neural information processing systems, 34:13937–13949, 2021

    2021

  30. [38]

    Seco de Herrera, et al

    Johannes Rückert, Louise Bloch, Raphael Brüngel, Ahmad Idrissi-Yaghir, Henning Schäfer, Cynthia S Schmidt, Sven Koitka, Obioma Pelka, Asma Ben Abacha, Alba G. Seco de Herrera, et al. Rocov2: Radiology objects in context version 2, an updated multimodal image dataset. Scientifi...

    2024

  31. [39]

    Membership inference attacks against machine learning models

    Reza Shokri, Marco Stronati, Congzheng Song, and Vitaly Shmatikov. Membership inference attacks against machine learning models. In 2017 IEEE symposium on security and privacy (SP), pages 3–18. IEEE, 2017

    2017

  32. [40]

    Towards vqa models that can read

    Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach. Towards vqa models that can read. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 8317–8326, 2019

    2019

  33. [41]

    Differentially private image classification by learning priors from random processes

    Xinyu Tang, Ashwinee Panda, Vikash Sehwag, and Prateek Mittal. Differentially private image classification by learning priors from random processes. Advances in Neural Information Processing Systems, 36:35855–35877, 2023. URL https://arxiv.org/abs/2301.12707

    2023

  34. [42]

    Pri- vate fine-tuning of large language models with zeroth-order optimization

    Xinyu Tang, Ashwinee Panda, Milad Nasr, Saeed Mahloujifar, and Prateek Mittal. Pri- vate fine-tuning of large language models with zeroth-order optimization. arXiv preprint arXiv:2401.04343, 2024

    2024 arXiv

  35. [43]

    Llama: Open and efficient foundation language models, 2023

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timo- thée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. Llama: Open and efficient foundation langua...

    2023 arXiv

  36. [44]

    Fastvlm: Efficient vision encoding for vision language models

    Pavan Kumar Anasosalu Vasu, Fartash Faghri, Chun-Liang Li, Cem Koc, Nate True, Albert Antony, Gokul Santhanam, James Gabriel, Peter Grasch, Oncel Tuzel, et al. Fastvlm: Efficient vision encoding for vision language models. arXiv preprint arXiv:2412.13303, 2024

    2024 arXiv

  37. [45]

    Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution

    Peng Wang, Shuai Bai, Sinan Tan, Shijie Wang, Zhihao Fan, Jinze Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, et al. Qwen2-vl: Enhancing vision-language model’s perception of the world at any resolution. arXiv preprint arXiv:2409.12191, 2024

    2024 arXiv

  38. [46]

    Cogvlm: Visual expert for pretrained language models

    Weihan Wang, Qingsong Lv, Wenmeng Yu, Wenyi Hong, Ji Qi, Yan Wang, Junhui Ji, Zhuoyi Yang, Lei Zhao, Song XiXuan, et al. Cogvlm: Visual expert for pretrained language models. Advances in Neural Information Processing Systems, 37:121475–121499, 2024. 13

    2024

  39. [47]

    Joint token pruning and squeezing towards more aggressive compression of vision transformers

    Siyuan Wei, Tianzhu Ye, Shen Zhang, Yao Tang, and Jiajun Liang. Joint token pruning and squeezing towards more aggressive compression of vision transformers. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 2092–2101, 2023

    2023

  40. [48]

    Improving differentially-private deep learning with gradients index pruning,

    Jungang Yang, Liyao Xiang, Size Peng, Yifan Bao, Hui Xu, Pengzhi Chu, Xinbing Wang, and Chenghu Zhou. Improving differentially-private deep learning with gradients index pruning,

  41. [49]

    Visionzip: Longer is better but not necessary in vision language models

    Senqiao Yang, Yukang Chen, Zhuotao Tian, Chengyao Wang, Jingyao Li, Bei Yu, and Jiaya Jia. Visionzip: Longer is better but not necessary in vision language models. arXiv preprint arXiv:2412.04467, 2024

    2024

  42. [50]

    Differentially private fine-tuning of language models

    Da Yu, Saurabh Naik, Arturs Backurs, Sivakanth Gopi, Huseyin A Inan, Gautam Kamath, Janardhan Kulkarni, Yin Tat Lee, Andre Manoel, Lukas Wutschitz, et al. Differentially private fine-tuning of language models. arXiv preprint arXiv:2110.06500, 2021

    arXiv

  43. [51]

    Large scale private learning via low-rank reparametrization

    Da Yu, Huishuai Zhang, Wei Chen, Jian Yin, and Tie-Yan Liu. Large scale private learning via low-rank reparametrization. In International Conference on Machine Learning , pages 12208–12218. PMLR, 2021

    2021

  44. [52]

    Opt: Open pre-trained transformer language models, 2022

    Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, Todor Mihaylov, Myle Ott, Sam Shleifer, Kurt Shuster, Daniel Simig, Punit Singh Koura, Anjali Sridhar, Tianlu Wang, and Luke Zettlemoyer. O...

    2022 arXiv

  45. [53]

    Mme-realworld: Could your multimodal llm challenge high-resolution real-world scenarios that are difficult for humans? arXiv preprint arXiv:2408.13257, 2024

    Yi-Fan Zhang, Huanyu Zhang, Haochen Tian, Chaoyou Fu, Shuangqing Zhang, Junfei Wu, Feng Li, Kun Wang, Qingsong Wen, Zhang Zhang, et al. Mme-realworld: Could your multimodal llm challenge high-resolution real-world scenarios that are difficult for humans? arXiv preprint arXiv:2...

    2024 arXiv

  46. [54]

    one record

    Daniel M Ziegler, Nisan Stiennon, Jeffrey Wu, Tom B Brown, Alec Radford, Dario Amodei, Paul Christiano, and Geoffrey Irving. Fine-tuning language models from human preferences. arXiv preprint arXiv:1909.08593, 2019. 14 A Key Differential Privacy Facts The following facts eluci...

    1909 arXiv

  47. [56]

    Guidelines: • The answer NA means that the abstract and introduction do not include the claims made in the paper

    Claims Question: Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope? Answer: [Yes] Justification: The abstract and introduction accurately reflect the paper’s contributions. Guidelines: • The answer NA means that the...

  48. [57]

    Limitations

    Limitations Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes] Justification: See Appendix J Guidelines: • The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those...

  49. [58]

    • All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced

    Theory assumptions and proofs Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof? Answer: [Yes] 23 Justification: See Section 3, Appendix A, Appendix D and Appendix C Guidelines: • The answer NA means th...

  50. [59]

    Guidelines: • The answer NA means that the paper does not include experiments

    Experimental result reproducibility Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and da...

  51. [60]

    Guidelines: • The answer NA means that paper does not include experiments requiring code

    Open access to data and code Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material? 24 Answer: [Yes] Justification: We include the code in o...

  52. [61]

    • The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them

    Experimental setting/details Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results? Answer: [Yes] Justification:: The experimental settings ar...

  53. [62]

    Guidelines: • The answer NA means that the paper does not include experiments

    Experiment statistical significance Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments? Answer: [No] Justification: We believe the pattern is clear. Guidelines: • The ...

  54. [63]

    • The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage

    Experiments compute resources Question: For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments? Answer: [Yes] Justification:See Section 5 Guidelines...

  55. [64]

    Guidelines: • The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics

    Code of ethics Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines? Answer: [Yes] Justification: It does. Guidelines: • The answer NA means that the authors have not reviewed t...

  56. [65]

    • If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact

    Broader impacts Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed? Answer: [Yes] Justification: See Appendix K Guidelines: • The answer NA means that there is no societal impact of the work performed. ...

  57. [66]

    Guidelines: • The answer NA means that the paper poses no such risks

    Safeguards Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? Answer: [NA] Justification: The paper poses...

  58. [67]

    Guidelines: • The answer NA means that the paper does not use existing assets

    Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected? Answer: [Yes] Justification: We cite all work...

  59. [68]

    New assets Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets? 27 Answer: [Yes] Justification: We offer documentation alongside our code. The anonymized repository in- cluding code and documentation can be fo...

  60. [69]

    Guidelines: • The answer NA means that the paper does not involve crowdsourcing nor research with human subjects

    Crowdsourcing and research with human subjects Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)? A...

  61. [70]

    Guidelines: • The answer NA means that the paper does not involve crowdsourcing nor research with human subjects

    Institutional review board (IRB) approvals or equivalent for research with human subjects Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals...

  62. [71]

    28 Answer: [Yes] Justification: The core methodology of this research is centered on the differential private fine-tuning of Multimodal Large Language Models (MLLMs)

    Declaration of LLM usage Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the ...

    2025

  63. [2023]

    net/forum

    In URL https://openreview. net/forum

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.