Pith. sign in

REVIEW 5 major objections 5 minor 53 references

FlashDP: Private Training Large Language Models with Efficient DP-SGD

T0 review · 5 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read FlashDP claims private 13B LLM pretraining at 90% of non-DP speed.

desk verdict A solid DP-SGD kernel optimization with code, undermined by unmeasured efficiency percentages and a single-kernel claim that contradicts its own implementation. read the letter →

arxiv 2507.01154 v1 pith:UWWS3KEY submitted 2025-07-01 cs.LG cs.CR

Liangyu Wang , Junxiao Wang , Jie Ren , Zihang Xiang , David E. Keyes , Di Wang This is my paper
classification cs.LGcs.CR
keywords differentialprivacyDP-SGDper-layerclippinglargelanguagemodelsGPUkernelfusionper-samplegradientblock-wiseall-reducememoryefficiency
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

FlashDP aims to make differentially private training of large language models almost as cheap as ordinary training. It claims to fuse per-sample gradient computation, norm calculation, clipping, and noise addition for per-layer DP-SGD into a single kernel pass, so gradients are computed once and never stored or recalculated. On a four-A100 system pretraining Llama-13B, it reports 90% of non-DP throughput with no added memory, and validation loss identical to standard per-layer clipped DP-SGD. If true, the main obstacle to private pretraining of billion-parameter models, namely the memory and speed overhead of per-sample clipping, largely disappears.

What carries the argument

The machinery is a fused per-layer DP-SGD kernel built on a Block-wise All-Reduce algorithm inside a Hierarchical Reduction Architecture. Per-sample gradient blocks are computed with batched GEMM in SRAM, norms are reduced within each block and then across blocks with atomics in HBM, and after a synchronization point the same blocks are clipped and noised before aggregation. Because CUDA cannot synchronize blocks inside one kernel, the implementation uses an adaptive kernel design that launches kernels serially over the batch dimension, using kernel boundaries as synchronization points.

What would settle it

Measure FlashDP's throughput at batch sizes much larger than 8 while holding the model fixed: if per-batch-element kernel launches add overhead linearly, throughput will fall visibly below the 90% non-DP ratio as batch size grows. A cleaner test is to compare FlashDP against a cooperative-groups single-kernel implementation with identical math; if the single kernel is faster, the adaptive launch strategy, not the fusion, is responsible for the headline number.

Watch

Extended reading notes

Core claim

The paper's central claim is that the expensive parts of per-layer DP-SGD can be reorganized so that per-sample gradients are produced once on-chip, their norms are aggregated with a block-wise all-reduce, clipping and noise are applied in SRAM, and only the final parameter gradient is written to HBM. FlashDP reports that this reduces memory movement relative to prior implicit methods and cuts redundant computation, so it matches non-DP memory usage and reaches 90% of non-DP throughput while preserving the same accuracy and privacy guarantees as standard per-layer clipped DP-SGD and DP-Adam.

Load-bearing premise

FlashDP assumes that launching many small kernels serially, one per batch element, costs little enough that the fused computation still reaches near-non-DP throughput; the paper does not report kernel-launch overhead or large-batch scaling.

Editorial extensions

If this is right

  • Pretraining 13B-parameter models under DP can avoid extra memory beyond the non-private baseline.
  • Private pretraining at near-non-DP speed becomes feasible on a four-GPU server rather than a large cluster.
  • The same fused pattern should transfer to DP-Adam, since the clipping and noise-addition stage is identical.
  • Memory use stays flat as micro-batch size grows, unlike explicit methods that exhaust memory at larger batches.
  • Per-layer clipping becomes a practical default for LLM DP training because its main cost has been removed.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper's 90% figure is measured at batch sizes up to 8; an untested implication is that the serial per-batch-element kernel launches could erode the advantage at much larger batch sizes, where launch overhead begins to dominate.
  • The same fusion strategy could apply to other per-sample operations beyond DP-SGD, such as per-sample loss penalties or per-sample evaluation metrics, which currently pay the same storage or recomputation tax.
  • A direct testable extension is to benchmark FlashDP against a cooperative-groups single-kernel implementation on newer GPUs; if launch overhead is negligible, a single-kernel version would be both simpler and faster.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper proposes FlashDP, an implementation of per-layer clipped DP-SGD for training large language models. FlashDP uses block-wise all-reduce and a hierarchical reduction architecture to compute per-sample gradient norms and perform clipping in a cache-friendly way, avoiding both the explicit per-sample gradient storage of Opacus and the gradient recomputation of GhostClip/BK. The authors report memory usage close to non-private training, throughput up to 90% of non-DP training for Llama-13B on four A100s, and validation loss identical to standard per-layer DP-SGD, alongside qualitative claims of 50% lower memory movement and 20% lower redundant computation.

Significance. If fully substantiated, FlashDP would be a valuable systems contribution: it targets a real bottleneck in differentially private LLM pre-training, and the open-source release is a concrete reproducibility asset. The underlying DP-SGD arithmetic is standard, and the reported numerical parity with per-layer DP-SGD is a reasonable correctness check. However, the paper's headline quantitative claims—50% memory-movement reduction, 20% redundant-computation reduction, and 90% of non-DP throughput on Llama-13B—are not directly measured with profiling or FLOP counts, and the implementation narrative contains a direct contradiction between the claimed single-kernel design and the actual adaptive multi-kernel implementation. These gaps make the central efficiency claims currently under-supported.

major comments (5)
  1. [§4.2 vs §1 and contribution bullets] Section 4.2 states that FlashDP does not rely on a monolithic kernel and instead splits the process across different kernels executed iteratively over the batch dimension, using kernel termination as the synchronization point. This directly contradicts Section 1 and the contribution list, which claim that FlashDP consolidates the entire DP-SGD process into a single GPU kernel. Because the adaptive multi-kernel design introduces kernel-launch and inter-kernel synchronization overhead that the paper does not measure, please either report this overhead as a function of batch size or revise the single-kernel claims.
  2. [Abstract, §1, Tables 1–4 and Figures 4–7] The abstract and Section 1 claim reductions of up to 50% in memory movement and 20% in redundant computation, but no experiment in the paper measures memory traffic or computational redundancy. Tables 1 and 2 report memory capacity and throughput, and Figures 4–7 report memory and throughput; no CUPTI/NCU profiling, memory-transaction counts, or FLOP counts are provided. These two numbers are headline contributions and need direct measurement, or the claims should be removed.
  3. [Table 1] In Table 1, FlashDP's memory usage is identical to NonDP in every reported row (e.g., GPT2-large B=4: 6.60 vs 6.60; GPT2-medium B=8: 6.60 vs 6.60). Exact equality across all configurations is difficult to reconcile with the extra per-layer clipping thresholds, noise states, and synchronization buffers required by DP-SGD. Please state precisely what is included in the memory measurement and explain why FlashDP adds zero memory overhead; if the numbers are rounded or dominated by a shared buffer, say so explicitly.
  4. [Algorithm 1 vs Algorithm 2/3] Algorithm 1 presents a single clipped-gradient update with a single threshold C and noise scale σ, with no layer index, while the paper's stated method is per-layer clipping (Algorithm 3). Moreover, line 13 accumulates the per-block clipped gradients and adds noise without dividing by batch size, whereas Algorithm 2 and Appendix A.1 define the DP-SGD update as (1/B)∑ g'_i + N(0, σ²C²I). Please clarify the relationship between the block-level sums and the final averaged update, and show how per-layer thresholds enter the kernel; this is needed to verify the privacy semantics of the implementation.
  5. [§5.3 and Figure 6] The abstract's headline '90% throughput compared to the Non-DP method' for Llama-13B is not stated in the experimental section; Section 5.3 only says FlashDP shows 'minimal throughput degradation.' Please report the exact Llama-13B throughput values, the pipeline-parallel configuration, batch and micro-batch sizes, and how the 90% figure is computed. Without these details, the central scaling claim is not verifiable.
minor comments (5)
  1. [Appendix B.1] The inserted 'Clarification on Gradient Formulations and Reviewer Feedback' is a reviewer-response artifact that should not appear in a submitted manuscript; it also states that Figure 2(a)'s per-sample nodes are 'purely illustrative' while the actual NonDP baseline uses batched GEMM. Please remove the self-referential text and revise the Figure 2 caption so the figure accurately represents the implemented baseline.
  2. [§4.1 vs Algorithm 1] Section 4.1 says the block-wise all-reduce is 'executed efficiently within the SRAM,' but Algorithm 1 line 9 says to offload the norm-square blocks from SRAM to HBM and perform the all-reduce there, with line 11 uploading the result back. Please resolve this internal inconsistency and describe where the all-reduce actually takes place.
  3. [Table 3 and §5.1] The utility experiment in Table 3 reports identical validation loss for DP-SGD and FlashDP at three epsilon values, but no non-private baseline, no error bars or seeds, and no training curves are shown. Also, Section 5.1 says the dataset is Wikitext, while Appendix C.1 says the utility experiment uses Fineweb-edu; please clarify which dataset is used where.
  4. [Tables 1–2 and Figure 7] The unit notation 'x1e4' in the table headers is confusing and should be replaced with standard units (e.g., MB and tokens/sec). There are also typos such as 'implicate methods' in the Section 3 heading and 'GPT-samll' in the Figure 7 caption.
  5. [Figures 4 and 6] Figures 4 and 6 appear to be identical plots of the pipeline-parallel Llama experiments; please avoid duplicating the same figure in the main text and appendix without cross-referencing.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: FlashDP is a systems optimization whose accuracy parity is arithmetic equivalence and whose throughput/memory claims are external benchmark measurements, not fitted predictions.

full rationale

FlashDP is an implementation-level optimization of standard per-layer DP-SGD: it fuses per-sample gradient computation, norm computation, clipping, noise addition, and aggregation without changing the mathematical operations. The claimed accuracy parity with per-layer DP-SGD is therefore an arithmetic equivalence, and the paper explicitly verifies it in Table 3 by reporting identical validation losses for DP-SGD and FlashDP across privacy levels, which is an equivalence check rather than a fitted prediction. The throughput and memory claims are measured against NonDP, Opacus, GhostClip, and BK baselines, not derived from any fitted parameter or from the definition of the target quantity. There is no load-bearing self-citation: the cited prior work on per-layer clipping (Bu et al., 2023a; He et al., 2022) is external to the authors and is used only to justify the choice of per-layer clipping rather than to establish FlashDP's efficiency or accuracy. The paper's use of per-sample gradient arithmetic and its reporting of identical validation loss confirm that FlashDP performs the same computation as the baseline. The open-sourced code and external benchmarks make the results independently checkable. The weakest point noted by the reader—absence of a measured kernel-launch overhead scaling study—is an empirical validation gap, not a circular derivation. Therefore no circular step can be exhibited, and the appropriate score is 0.

Assumptions & free parameters 1 free parameters · 3 assumptions · 0 invented entities

The central claim is an engineering efficiency result, not a new mathematical derivation. The method reuses standard per-layer DP-SGD and GPU blocking concepts; the only implementation-specific choices are tiling parameters. No new entities are postulated. The method relies on prior results for per-layer clipping privacy/accuracy and on standard assumptions about GPU memory hierarchies.

free parameters (1)
  • Block dimensions (b, t, d, p) = Chosen based on GPU SRAM capacity M
    These tiling parameters determine how tensors are split across SRAM and affect the achieved throughput. They are chosen by hand for the hardware, not fitted to hit a particular result, but they are implementation-specific tuning constants.
assumptions (3)
  • domain assumption Per-layer clipping in DP-SGD provides DP guarantees and maintains accuracy comparable to global clipping (from Bu et al. 2023a; He et al. 2022).
    The paper builds its method on per-layer clipping without re-deriving its privacy analysis.
  • domain assumption GPU memory hierarchy (SRAM/HBM) behaves as described, and blocking improves data locality (standard CUDA/GPU architecture).
    The efficiency claims assume the standard GPU memory hierarchy model described in Appendix A.3.
  • domain assumption CUDA cooperative groups grid synchronization is impractical for DP applications due to requirement to launch all blocks simultaneously (statement in Section 4.2).
    This motivates the adaptive kernel design; if false, a simpler implementation might exist.

how reviews work

0 comments
Cite this review

Pith. "Pith review of FlashDP: Private Training Large Language Models with Efficient DP-SGD." pith.science (2026). https://pith.science/paper/UWWS3KEY

@misc{pith2026250701154,
  author       = {Pith},
  title        = {Pith review of: FlashDP: Private Training Large Language Models with Efficient DP-SGD},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/UWWS3KEY}},
  note         = {Machine review of arXiv:2507.01154}
}
read the original abstract

As large language models (LLMs) increasingly underpin technological advancements, the privacy of their training data emerges as a critical concern. Differential Privacy (DP) serves as a rigorous mechanism to protect this data, yet its integration via Differentially Private Stochastic Gradient Descent (DP-SGD) introduces substantial challenges, primarily due to the complexities of per-sample gradient clipping. Current explicit methods, such as Opacus, necessitate extensive storage for per-sample gradients, significantly inflating memory requirements. Conversely, implicit methods like GhostClip reduce storage needs by recalculating gradients multiple times, which leads to inefficiencies due to redundant computations. This paper introduces FlashDP, an innovative cache-friendly per-layer DP-SGD that consolidates necessary operations into a single task, calculating gradients only once in a fused manner. This approach not only diminishes memory movement by up to \textbf{50\%} but also cuts down redundant computations by \textbf{20\%}, compared to previous methods. Consequently, FlashDP does not increase memory demands and achieves a \textbf{90\%} throughput compared to the Non-DP method on a four-A100 system during the pre-training of the Llama-13B model, while maintaining parity with standard per-layer clipped DP-SGD in terms of accuracy. These advancements establish FlashDP as a pivotal development for efficient and privacy-preserving training of LLMs. FlashDP's code has been open-sourced in https://github.com/kaustpradalab/flashdp.

Figures

Figures reproduced from arXiv: 2507.01154 by the authors.

Figure 1
Figure 1. Comparison of different training methods. (a) Stan [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Comparison of different training methods. (a) Non-DP: Basic training without DP. (b) [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Illustration of FlashDP. It depicts the core algorithm design of FlashDP. Its features are integrated with on-chip per-sample gradient norm calculations. The workflow incorporates block-wise all-reduce and synchronization to facilitate efficient norm aggregation. SRAM (orange) and HBM (green) are optimally utilized to manage memory efficiently, addressing the kernel fusion challenges and reducing computational redun… view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: Memory and Throughput for Llama Models Using Pipeline Parallel Training. (a) Memory usage for Llama-3B, Llama-7B, and Llama-13B models. (b) Throughput in tokens per second across these model sizes. A value of 0 indicates out of memory. Distributed Data Parallel (DDP) …
Figure 5
Figure 5. Figure 5: Memory and Throughput Analysis of GPT-2 Models Using Automatic Mixed Precision (AMP) Training Across Float16 and BFloat16 Precision.: (a) Demonstrates the memory usage for GPT-2 small, medium, and large models with Float16 precision. (b) shows throughput using Float16 …
Figure 6
Figure 6. Figure 6: Memory and Throughput for Llama Models Using Pipeline Parallel Training. (a) Memory usage for Llama-3B, Llama-7B, and Llama-13B models. (b) Throughput in tokens per second across these model sizes. A value of 0 indicates out of memory. Distributed Data Parallel (DDP) …
Figure 7
Figure 7. Figure 7: Memory and Throughput for GPT Models Using Distributed Data Parallel Training. (a) Memory usage for GPT-samll, GPT-medium, and GPT-large models. (b) Throughput in tokens per second across these model sizes. A value of 0 indicates out of memory. 23 [PITH_FULL_IMAGE:fig…

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

53 extracted references · 22 canonical work pages

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  2. [2]

    Mart\' n Abadi, Ashish Agarwal, Paul Barham, Eugene Brevdo, Zhifeng Chen, Craig Citro, Greg S. Corrado, Andy Davis, Jeffrey Dean, Matthieu Devin, Sanjay Ghemawat, Ian Goodfellow, Andrew Harp, Geoffrey Irving, Michael Isard, Yangqing Jia, Rafal Jozefowicz, Lukasz Kaiser, Manjunath Kudlur, Josh Levenberg, Dan Man\' e , Rajat Monga, Sherry Moore, Derek Murra...

    work page 2015

  3. [3]

    Deep learning with differential privacy

    Martin Abadi, Andy Chu, Ian Goodfellow, H Brendan McMahan, Ilya Mironov, Kunal Talwar, and Li Zhang. Deep learning with differential privacy. In Proceedings of the 2016 ACM SIGSAC conference on computer and communications security, pp.\ 308--318, 2016

    2016

  4. [4]

    Gpt-4 technical report

    Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023

    arXiv 2023

  5. [5]

    Using chatgpt to write patient clinic letters

    Stephen R Ali, Thomas D Dobbs, Hayley A Hutchings, and Iain S Whitaker. Using chatgpt to write patient clinic letters. The Lancet Digital Health, 5 0 (4): 0 e179--e181, 2023

    work page 2023

  6. [6]

    Large-scale differentially private bert

    Rohan Anil, Badih Ghazi, Vineet Gupta, Ravi Kumar, and Pasin Manurangsi. Large-scale differentially private bert. arXiv preprint arXiv:2108.01624, 2021

    arXiv 2021

  7. [7]

    Large-scale differentially private bert

    Rohan Anil, Badih Ghazi, Vineet Gupta, Ravi Kumar, and Pasin Manurangsi. Large-scale differentially private bert. In Findings of the Association for Computational Linguistics: EMNLP 2022, pp.\ 6481--6491, 2022

    work page 2022

  8. [8]

    Towards end-to-end 4-bit inference on generative large language models

    Saleh Ashkboos, Ilia Markov, Elias Frantar, Tingxuan Zhong, Xincheng Wang, Jie Ren, Torsten Hoefler, and Dan Alistarh. Towards end-to-end 4-bit inference on generative large language models. arXiv preprint arXiv:2310.09259, 2023

    arXiv

Show all 53 references
  1. [9]

    Role of chat gpt in public health

    Som S Biswas. Role of chat gpt in public health. Annals of biomedical engineering, 51 0 (5): 0 868--869, 2023

    2023

  2. [10]

    On the opportunities and risks of foundation models

    Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. On the opportunities and risks of foundation models. arXiv preprint arXiv:2108.07258, 2021

    2021 arXiv

  3. [11]

    Scalable and efficient training of large convolutional neural networks with differential privacy

    Zhiqi Bu, Jialin Mao, and Shiyun Xu. Scalable and efficient training of large convolutional neural networks with differential privacy. Advances in Neural Information Processing Systems, 35: 0 38305--38318, 2022

    2022

  4. [12]

    On the accuracy and efficiency of group-wise clipping in differentially private optimization

    Zhiqi Bu, Ruixuan Liu, Yu-Xiang Wang, Sheng Zha, and George Karypis. On the accuracy and efficiency of group-wise clipping in differentially private optimization. arXiv preprint arXiv:2310.19215, 2023 a

    2023 arXiv

  5. [13]

    Differentially private optimization on large model at small cost

    Zhiqi Bu, Yu-Xiang Wang, Sheng Zha, and George Karypis. Differentially private optimization on large model at small cost. In International Conference on Machine Learning, pp.\ 3192--3218. PMLR, 2023 b

    2023

  6. [14]

    A comprehensive survey of ai-generated content (aigc): A history of generative ai from gan to chatgpt

    Yihan Cao, Siyu Li, Yixin Liu, Zhiling Yan, Yutong Dai, Philip S Yu, and Lichao Sun. A comprehensive survey of ai-generated content (aigc): A history of generative ai from gan to chatgpt. arXiv preprint arXiv:2303.04226, 2023

    2023 arXiv

  7. [15]

    Quantifying memorization across neural language models

    Nicholas Carlini, Daphne Ippolito, Matthew Jagielski, Katherine Lee, Florian Tramer, and Chiyuan Zhang. Quantifying memorization across neural language models. In International Conference on Learning Representations, 2023

    2023

  8. [16]

    A survey of embodied ai: From simulators to research tasks

    Jiafei Duan, Samson Yu, Hui Li Tan, Hongyuan Zhu, and Cheston Tan. A survey of embodied ai: From simulators to research tasks. IEEE Transactions on Emerging Topics in Computational Intelligence, 6 0 (2): 0 230--244, 2022

    2022

  9. [17]

    An efficient dp-sgd mechanism for large scale nlu models

    Christophe Dupuy, Radhika Arava, Rahul Gupta, and Anna Rumshisky. An efficient dp-sgd mechanism for large scale nlu models. In ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp.\ 4118--4122. IEEE, 2022

    2022

  10. [18]

    Differential privacy

    Cynthia Dwork. Differential privacy. In International colloquium on automata, languages, and programming, pp.\ 1--12. Springer, 2006

    2006

  11. [19]

    Calibrating noise to sensitivity in private data analysis

    Cynthia Dwork, Frank McSherry, Kobbi Nissim, and Adam Smith. Calibrating noise to sensitivity in private data analysis. In Theory of Cryptography: Third Theory of Cryptography Conference, TCC 2006, New York, NY, USA, March 4-7, 2006. Proceedings 3, pp.\ 265--284. Springer, 2006

    2006

  12. [20]

    Llm-based nlg evaluation: Current status and challenges

    Mingqi Gao, Xinyu Hu, Jie Ruan, Xiao Pu, and Xiaojun Wan. Llm-based nlg evaluation: Current status and challenges. arXiv preprint arXiv:2402.01383, 2024

    2024 arXiv

  13. [21]

    Efficient per-example gradient computations

    Ian Goodfellow. Efficient per-example gradient computations. arXiv preprint arXiv:1510.01799, 2015

    2015 arXiv

  14. [22]

    Exploring the limits of differentially private deep learning with group-wise clipping

    Jiyan He, Xuechen Li, Da Yu, Huishuai Zhang, Janardhan Kulkarni, Yin Tat Lee, Arturs Backurs, Nenghai Yu, and Jiang Bian. Exploring the limits of differentially private deep learning with group-wise clipping. arXiv preprint arXiv:2212.01539, 2022

    2022 arXiv

  15. [23]

    Learning and evaluating a differentially private pre-trained language model

    Shlomo Hoory, Amir Feder, Avichai Tendler, Sofia Erell, Alon Peled-Cohen, Itay Laish, Hootan Nakhost, Uri Stemmer, Ayelet Benjamini, Avinatan Hassidim, et al. Learning and evaluating a differentially private pre-trained language model. In Findings of the Association for Comput...

    2021

  16. [24]

    Preventing verbatim memorization in language models gives a false sense of privacy

    Daphne Ippolito, Florian Tram \`e r, Milad Nasr, Chiyuan Zhang, Matthew Jagielski, Katherine Lee, Christopher A Choquette-Choo, and Nicholas Carlini. Preventing verbatim memorization in language models gives a false sense of privacy. arXiv preprint arXiv:2210.17546, 2022

    arXiv

  17. [25]

    Differentially private language models benefit from public pre-training

    Gavin Kerrigan, Dylan Slack, and Jens Tuyls. Differentially private language models benefit from public pre-training. arXiv preprint arXiv:2009.05886, 2020

    2009 arXiv

  18. [26]

    torchgpipe: On-the-fly pipeline parallelism for training giant models

    Chiheon Kim, Heungsub Lee, Myungryong Jeong, Woonhyuk Baek, Boogeon Yoon, Ildoo Kim, Sungbin Lim, and Sungwoong Kim. torchgpipe: On-the-fly pipeline parallelism for training giant models. arXiv preprint arXiv:2004.09910, 2020

    2004 arXiv

  19. [27]

    Scaling up differentially private deep learning with fast per-example gradient clipping

    Jaewoo Lee and Daniel Kifer. Scaling up differentially private deep learning with fast per-example gradient clipping. arXiv preprint arXiv:2009.03106, 2020

    2009 arXiv

  20. [28]

    Scaling up differentially private deep learning with fast per-example gradient clipping

    Jaewoo Lee and Daniel Kifer. Scaling up differentially private deep learning with fast per-example gradient clipping. Proceedings on Privacy Enhancing Technologies, 2021

    2021

  21. [29]

    Pytorch distributed: Experiences on accelerating data parallel training

    Shen Li, Yanli Zhao, Rohan Varma, Omkar Salpekar, Pieter Noordhuis, Teng Li, Adam Paszke, Jeff Smith, Brian Vaughan, Pritam Damania, et al. Pytorch distributed: Experiences on accelerating data parallel training. arXiv preprint arXiv:2006.15704, 2020

    2006 arXiv

  22. [30]

    Large language models can be strong differentially private learners

    Xuechen Li, Florian Tramer, Percy Liang, and Tatsunori Hashimoto. Large language models can be strong differentially private learners. arXiv preprint arXiv:2110.05679, 2021

    2021 arXiv

  23. [31]

    Large language models can be strong differentially private learners

    Xuechen Li, Florian Tramer, Percy Liang, and Tatsunori Hashimoto. Large language models can be strong differentially private learners. In International Conference on Learning Representations, 2022

    2022

  24. [32]

    Fineweb-edu: the finest collection of educational content, 2024

    Anton Lozhkov, Loubna Ben Allal, Leandro von Werra, and Thomas Wolf. Fineweb-edu: the finest collection of educational content, 2024. URL https://huggingface.co/datasets/HuggingFaceFW/fineweb-edu

    2024

  25. [33]

    How much do language models copy from their training data? evaluating linguistic novelty in text generation using raven

    R Thomas McCoy, Paul Smolensky, Tal Linzen, Jianfeng Gao, and Asli Celikyilmaz. How much do language models copy from their training data? evaluating linguistic novelty in text generation using raven. Transactions of the Association for Computational Linguistics, 11: 0 652--670, 2023

    2023

  26. [34]

    The wikitext long term dependency language modeling dataset

    Stephen Merity. The wikitext long term dependency language modeling dataset. Salesforce Metamind, 9, 2016

    2016

  27. [35]

    Mixed precision training

    Paulius Micikevicius, Sharan Narang, Jonah Alben, Gregory Diamos, Erich Elsen, David Garcia, Boris Ginsburg, Michael Houston, Oleksii Kuchaiev, Ganesh Venkatesh, et al. Mixed precision training. arXiv preprint arXiv:1710.03740, 2017

    arXiv

  28. [36]

    Scalable extraction of training data from (production) language models

    Milad Nasr, Nicholas Carlini, Jonathan Hayase, Matthew Jagielski, A Feder Cooper, Daphne Ippolito, Christopher A Choquette-Choo, Eric Wallace, Florian Tram \`e r, and Katherine Lee. Scalable extraction of training data from (production) language models. arXiv preprint arXiv:23...

    2023 arXiv

  29. [37]

    Bolt: Privacy-preserving, accurate and efficient inference for transformers

    Qi Pang, Jinhao Zhu, Helen M \"o llering, Wenting Zheng, and Thomas Schneider. Bolt: Privacy-preserving, accurate and efficient inference for transformers. In 2024 IEEE Symposium on Security and Privacy (SP), pp.\ 130--130. IEEE Computer Society, 2024

    2024

  30. [38]

    Pytorch: An imperative style, high-performance deep learning library

    Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems, 32, 2019

    2019

  31. [39]

    Privacy-adaptive bert for natural language understanding

    Chen Qu, Weize Kong, Liu Yang, Mingyang Zhang, Michael Bendersky, and Marc Najork. Privacy-adaptive bert for natural language understanding. arXiv preprint arXiv:2104.07504, 190, 2021

    2021 arXiv

  32. [40]

    Language models are unsupervised multitask learners

    Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language models are unsupervised multitask learners. OpenAI blog, 1 0 (8): 0 9, 2019

    2019

  33. [41]

    Efficient per-example gradient computations in convolutional neural networks

    Gaspar Rochette, Andre Manoel, and Eric W Tramel. Efficient per-example gradient computations in convolutional neural networks. arXiv preprint arXiv:1912.06015, 2019

    1912 arXiv

  34. [42]

    Chatgpt utility in healthcare education, research, and practice: systematic review on the promising perspectives and valid concerns

    Malik Sallam. Chatgpt utility in healthcare education, research, and practice: systematic review on the promising perspectives and valid concerns. In Healthcare, volume 11, pp.\ 887. MDPI, 2023

    2023

  35. [43]

    Natural language processing of clinical notes on chronic diseases: systematic review

    Seyedmostafa Sheikhalishahi, Riccardo Miotto, Joel T Dudley, Alberto Lavelli, Fabio Rinaldi, Venet Osmani, et al. Natural language processing of clinical notes on chronic diseases: systematic review. JMIR medical informatics, 7 0 (2): 0 e12239, 2019

    2019

  36. [44]

    Llm-planner: Few-shot grounded planning for embodied agents with large language models

    Chan Hee Song, Jiaman Wu, Clayton Washington, Brian M Sadler, Wei-Lun Chao, and Yu Su. Llm-planner: Few-shot grounded planning for embodied agents with large language models. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp.\ 2998--3009, 2023

    2023

  37. [45]

    Memorization without overfitting: Analyzing the training dynamics of large language models

    Kushal Tirumala, Aram Markosyan, Luke Zettlemoyer, and Armen Aghajanyan. Memorization without overfitting: Analyzing the training dynamics of large language models. Advances in Neural Information Processing Systems, 35: 0 38274--38290, 2022

    2022

  38. [46]

    Clinical camel: An open-source expert-level medical language model with dialogue-based knowledge encoding

    Augustin Toma, Patrick R Lawler, Jimmy Ba, Rahul G Krishnan, Barry B Rubin, and Bo Wang. Clinical camel: An open-source expert-level medical language model with dialogue-based knowledge encoding. arXiv preprint arXiv:2305.12031, 2023

    2023 arXiv

  39. [47]

    Llama: Open and efficient foundation language models

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timoth \'e e Lacroix, Baptiste Rozi \`e re, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971, 2023

    2023 arXiv

  40. [48]

    Attention is all you need

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, ukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017

    2017

  41. [49]

    Ai-generated content (aigc): A survey

    Jiayang Wu, Wensheng Gan, Zefeng Chen, Shicheng Wan, and Hong Lin. Ai-generated content (aigc): A survey. arXiv preprint arXiv:2304.06632, 2023

    2023 arXiv

  42. [50]

    Translating natural language to planning goals with large-language models

    Yaqi Xie, Chen Yu, Tongyao Zhu, Jinbin Bai, Ze Gong, and Harold Soh. Translating natural language to planning goals with large-language models. arXiv preprint arXiv:2302.05128, 2023

    2023 arXiv

  43. [51]

    A survey on robotics with foundation models: toward embodied ai

    Zhiyuan Xu, Kun Wu, Junjie Wen, Jinming Li, Ning Liu, Zhengping Che, and Jian Tang. A survey on robotics with foundation models: toward embodied ai. arXiv preprint arXiv:2402.02385, 2024

    2024 arXiv

  44. [52]

    Opacus: User-friendly differential privacy library in pytorch

    Ashkan Yousefpour, Igor Shilov, Alexandre Sablayrolles, Davide Testuggine, Karthik Prasad, Mani Malek, John Nguyen, Sayan Ghosh, Akash Bharadwaj, Jessica Zhao, et al. Opacus: User-friendly differential privacy library in pytorch. arXiv preprint arXiv:2109.12298, 2021

    2021 arXiv

  45. [53]

    Counterfactual memorization in neural language models

    Chiyuan Zhang, Daphne Ippolito, Katherine Lee, Matthew Jagielski, Florian Tram \`e r, and Nicholas Carlini. Counterfactual memorization in neural language models. Advances in Neural Information Processing Systems, 36: 0 39321--39362, 2023

    2023

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.