Pith. sign in

REVIEW 3 major objections 4 minor 1 cited by

Scaling up FHE-based Privacy-Preserving ML: Higher Throughput, Longer Inputs for LLama-3-8B

T0 review · 3 major / 4 minor · reviewed 2026-08-03 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that a CKKS-based private Llama-2-7B inference can process a 4096-token prompt, with only the last 128 tokens encrypted, in 85 seconds for prefill and 33 seconds per generated token on eight consumer GPUs.

desk verdict Clean algorithmic ideas, unsupported headline numbers — the abstract describes a different run than the body. read the letter →

arxiv 2601.18511 v2 pith:4T4P2Z3K submitted 2026-01-26 cs.CR

Jaiyoung Park , Sejin Park , Jai Hyun Park , Jung Ho Ahn , Jung Hee Cheon , Guillaume Hanrot , Jung Woo Kim , Minje Park
show 1 more author
Damien Stehlé
This is my paper · ORCID
classification cs.CR
keywords fullyhomomorphicencryptionCKKSprivateLLMinferenceplaintext-ciphertextmatrixmultiplicationSoftmaxoutliermitigationchunkedprefillLlama
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to show that fully homomorphic encryption can serve practical private LLM inference for long prompts when most of the prompt is public and only the tail is confidential. It builds a CKKS implementation of Llama-2-7B that accepts 4096 input tokens, of which the last 128 are encrypted, and reports 85 seconds for prefill summarization and 33 seconds per generated token on eight RTX 4090 GPUs. To get there, it splits prefill into a public clear stage and a private encrypted stage, tames activation outliers with attention-sink prefixes and orthogonal rotations, and introduces a shallow plaintext-ciphertext matrix multiplication plus a slim Softmax evaluation. If the reported numbers hold, this would move FHE LLM inference from tens of encrypted tokens to thousands of partially encrypted tokens at speeds below a minute per generated token. The abstract quotes even faster Llama-3-8B numbers on different hardware than the full text reports, so the exact model and configuration behind the headline timings need careful reading.

What carries the argument

The central mechanism is a heterogeneous chunked prefill pipeline: a public prefill stage processes the long plaintext prefix in the clear to build a key-value cache, then a private prefill stage evaluates QKV projection, attention, output projection, and the feed-forward network on the encrypted suffix. Within that pipeline, the load-bearing new identities are a tau-shifted rotation-based matrix product formula that performs plaintext-ciphertext multiplication in one multiplicative level without encoding conversion, and a recursive sum-of-two-squares decomposition of polynomials that lets Softmax's auxiliary track be evaluated on sparsely packed ciphertexts in O(log d) homomorphic operation

What would settle it

Take the claimed 4096-token split (3968 plaintext, 128 encrypted), run the actual CKKS implementation on the same eight GPUs, and compare end-to-end output perplexity against the FP16 baseline; if perplexity degrades beyond the 12-bit target or timings exceed 85 seconds/33 seconds per token, the central claim fails. A cheaper test: compare simulator-injected noise values against real CKKS slot values for one block; divergence beyond the assumed noise budget would show the precision analysis is unsound.

Watch

Extended reading notes

Core claim

The central claim is that private inference for a Llama-scale model becomes practical when only the final part of the input is confidential. The paper reports a CKKS implementation of Llama-2-7B that ingests 4096 tokens—3968 plaintext, 128 encrypted—in 85 seconds for summarization and generates each following token in 33 seconds, on eight RTX 4090 GPUs. The abstract, but not the body, claims Llama-3-8B timings of 20s/18s per token for 128 encrypted tokens and 64s/22s per token for the heterogeneous 4096-token input on eight RTX PRO 6000 GPUs. The author's claim is that the enabling ingredients—heterogeneous chunked prefill, attention-sink prefixing, orthogonal rotations, and the new linear-a

Load-bearing premise

The load-bearing premise is that the custom fixed-point simulator accurately reproduces CKKS noise growth and the effect of the attention-sink prefix, so that the selected 12-bit precision target carries over to real encrypted inference; the paper reports no end-to-end accuracy measurement of the actual encrypted pipeline, and an optimistic simulator would invalidate the whole cost/quality trade-off.

Editorial extensions

If this is right

  • FHE LLM inference need not treat every input token as a ciphertext: when only the prompt tail is sensitive, the public prefix can be handled in the clear, turning a wide encrypted attention computation into a manageable plaintext-ciphertext problem.
  • Outlier-suppression techniques—attention-sink prefixing and orthogonal rotation—cut the input range of RMSNorm from roughly 2244 to about 7.7, making low-degree polynomial approximations and standard-precision bootstrapping feasible.
  • A one-level plaintext-ciphertext matrix multiplication that updates its packing structure on the fly avoids both bootstrapping and large plaintext-matrix memory footprints in wide attention, keeping the attention phase almost bootstrap-free.
  • Softmax can be evaluated on slim ciphertexts with about log(depth) homomorphic levels through sum-of-squares decomposition, at a cost of roughly one bit of precision per recursion level.
  • If the body's timings reproduce, the same eight-GPU cluster completes a 4096-token private-prefill inference in 85 seconds, compared with 295 seconds in prior work for only 128 encrypted tokens on more expensive hardware.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Inference: the abstract's Llama-3-8B numbers should not be read as measured results of this manuscript; the body's tables and per-block breakdowns target Llama-2-7B on RTX 4090 GPUs, so the two sets of timings likely come from different builds, different hardware, or projections.
  • Inference: if the simulator's noise model is faithful, the sum-of-squares evaluation trick could be lifted to other slim-slot nonlinearities in FHE, potentially reducing bootstraps in decoding or in fully encrypted prefill more broadly.
  • Inference: the public-prefix assumption is the real boundary of the method; the speedups come from processing most tokens in the clear, and extending to fully encrypted long prompts would require ciphertext-ciphertext attention over the whole context, which the paper does not claim to scale.
  • Inference: a natural testable extension is to vary the public/private split (e.g., 1k/128 vs 4k/128) and measure whether cost scales roughly linearly in the encrypted suffix, which would confirm the framework rather than the specific implementation.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper proposes new techniques for CKKS-based private LLM inference when the prompt has a large public prefix and a short encrypted suffix. The main algorithmic contributions are: (i) an unbalanced chunked prefill framework with public/private stages; (ii) a depth-1 plaintext-ciphertext matrix multiplication (PCMM) algorithm for PC-attention; (iii) a slim polynomial evaluation method for sparsely packed ciphertexts; and (iv) outlier mitigation via token prepending and orthogonal rotations. The body reports detailed per-block timings on 8 RTX-4090 GPUs for Llama-2-7B and Llama-3-8B, and extrapolates to 85 s prefill / 33 s generation for the full model. The submitted abstract, however, claims different results (Llama-3-8B on RTX PRO 6000 GPUs, with 20 s / 18 s and 64 s / 22 s numbers) that do not appear in the body.

Significance. If validated, the work would be a meaningful advance in FHE-based LLM inference: it explicitly targets long-context private inference, addresses outlier-induced approximation costs with ML techniques, and introduces an almost bootstrap-free attention design. The algorithmic descriptions are detailed and the choice of building blocks from recent work (e.g., BCH+24, CHK+24, HYT+24) is reasonable. The strength of the paper is the concrete system design and the per-block cost analysis. However, the central quantitative claim is an extrapolation, and no end-to-end encrypted inference accuracy is reported, so the practical significance of the numbers is currently not established.

major comments (3)
  1. [Abstract vs §1.1 and §5] The submitted abstract reports Llama-3-8B results on 8 NVIDIA RTX PRO 6000 GPUs: 20 s / 18 s for 128 encrypted tokens and 64 s / 22 s for the 4096-token heterogeneous setting. The body title, body abstract, §1.1, and §5 describe Llama-2-7B on 8 RTX-4090 GPUs with 85 s / 33 s. These two sets of claims are inconsistent, and the abstract numbers do not appear anywhere in the body. As it stands, the headline claim is not supported by the manuscript. This must be resolved before review can proceed.
  2. [§5.4, Table 5] The 85 s / 33 s end-to-end numbers are not measured. Table 5 reports per-block timings for the second transformer block, and the text explicitly says 'for an estimate of end-by-end computation time these figures should be multiplied by 32'. This assumes per-layer cost uniformity, but the paper itself documents layer-wise and dimension-wise variation in activation ranges (§3.3, §5.1), layer-wise polynomial approximations, and layer-dependent bootstrap placements. NCCL reduction/broadcast and pipeline overheads also need not scale linearly. The §1.1 claim that the system is 'capable of processing 4096 input tokens ... in 85s ... and 33s' is therefore an extrapolation, not a measurement. Please report at least one full-model execution (or a tightly justified scaling model with per-layer data).
  3. [§3.2, Table 3] The 12-bit precision target, and consequently the polynomial degrees, Softmax depth, and bootstrapping parameters, are derived entirely from a custom PyTorch fake-quantization simulator that is asserted to 'precisely mirror' CKKS noise but is never compared to a real encrypted execution. No end-to-end encrypted-inference perplexity or output correctness is reported. If the simulator underestimates noise growth or inaccurately models the interaction of the attention-sink prefix with CKKS rescaling, the claimed cost/quality trade-off collapses. Please validate the simulator against an actual CKKS circuit (e.g., one block or one layer) and report encrypted-inference accuracy/perplexity.
minor comments (4)
  1. [Throughout] Typos and spacing issues: 'Jayashankaret al', 'Moonet al', 'Departement' in the affiliation, 'end-by-end' for 'end-to-end', and 'Paterson-Stockmayer' should be 'Paterson-Stockmeyer'.
  2. [§3.3, §3.4] The notation β_{i,(s,L)} is defined in §3.3 but used with inconsistent subscript ordering in §3.4. Please define the ordering once and use it consistently.
  3. [§4.5] Algorithm 1 states a cost dominated by O(2^{(k-j)/2}) + j ciphertext-ciphertext multiplications and j rotations, but the abstract-level claim is 'O(log d) homomorphic operations'; please clarify the exact dependency on degree d and the role of Paterson-Stockmeyer evaluation.
  4. [§5.2 / §5.3] The paper does not give concrete CKKS parameter sets (ring degree, scaling factor, modulus bit-length, bootstrap precision) for the reported timings. Adding a parameter table would substantially improve reproducibility.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: the core FHE algorithms are derived in-paper, while the headline timings rest on a per-block extrapolation and the precision target on an unvalidated simulator — both are evidence gaps, not circular reasoning.

full rationale

The paper's derivation chain does not exhibit definitional circularity. The main algorithmic claims are proved in-paper: Theorem 4.2 is an algebraic transformation of the JKLS identity, Lemma 4.3 and Algorithm 1 are self-contained polynomial-evaluation constructions, and the Softmax design is an adaptation of CHK+24 with explicit level counts. The 12-bit precision target (Table 3) is chosen by simulating CKKS noise in PyTorch and comparing perplexity to FP16; this is a parameter-selection input, not a predicted output. The paper does not report an end-to-end encrypted perplexity measurement, so the simulator is unvalidated evidence, which is a correctness/evidence risk rather than a circular step. The 85s/33s headline times come from Table 5 per-block times 'for an estimate of end-by-end computation time these figures should be multiplied by 32' (Sec. 5.4), i.e., an extrapolation, and the abstract's Llama-3-8B RTX PRO 6000 numbers do not appear in the measurement section; these are reporting inconsistencies, not circularity. Several cited building blocks ([BCH+24], [CHK+24], [Par25]) share authors with this paper, but they are external publications used as components, and the load-bearing algorithmic identities are derived here rather than assumed from those citations. No uniqueness theorem or ansatz is imported solely from a self-citation. Score 1 reflects minor self-citations without load-bearing circularity.

Assumptions & free parameters 6 free parameters · 6 assumptions · 0 invented entities

No new physical or algebraic entities are introduced; the 'faulty tokens' prefix and tau-packing are input manipulations and encodings, not postulated entities. The load-bearing free parameters are the empirically measured ranges and precision budgets.

free parameters (6)
  • CKKS precision target = 12 bits
    Chosen from Table 3 so simulated perplexity matches FP16; drives all polynomial and bootstrapping parameters.
  • SiLU approximation intervals = [−16,16] and [−24,24]
    Layer-wise/dimension-wise ranges from activation measurements (Table 4).
  • RMSNorm approximation interval = [1/sqrt(30), sqrt(30)]
    Range chosen from measured data after layerwise scaling (Table 4).
  • Softmax parameters = k=2, degree-15 exp, degree-128 inv-sqrt
    Chosen from range statistics to minimize depth (§4.4).
  • Per-coordinate bounds beta_{i,(s,L)} = not listed numerically
    Measured from simulation; used to scale bootstrapping inputs and set polynomial domains (§3.3).
  • CKKS scaling factor and bootstrap parameters = not listed numerically
    Optimized on the basis of measured ranges and precision (§1.1).
assumptions (6)
  • standard math RLWE and CKKS security/correctness
    The scheme's correctness and 128-bit security are assumed from prior literature (§2.1).
  • standard math Lemma 4.3: positive polynomial = sum of two squares
    Used to build the slim polynomial evaluation; cited to Benoist 2017.
  • standard math Orthogonal rotations preserve model outputs when fused into weights
    True over exact arithmetic; in practice relies on floating-point fusion (§3.1).
  • domain assumption Attention-sink prefix tokens do not degrade task performance
    The tailored model is not the original Llama; no perplexity comparison of the tailored vs base model is reported.
  • domain assumption Simulator faithfully reproduces CKKS noise
    Precision/range targets and the 12-bit budget are derived from a custom simulator (§3.2); no real-encrypted quality validation.
  • domain assumption Only the tail of the prompt is sensitive
    The heterogeneous split assumes the public prefix leaks nothing; privacy of the encrypted tail w.r.t. the public context is not formally analyzed.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Scaling up FHE-based Privacy-Preserving ML: Higher Throughput, Longer Inputs for LLama-3-8B." pith.science (2026). https://pith.science/paper/4T4P2Z3K

@misc{pith2026260118511,
  author       = {Pith},
  title        = {Pith review of: Scaling up FHE-based Privacy-Preserving ML: Higher Throughput, Longer Inputs for LLama-3-8B},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/4T4P2Z3K}},
  note         = {Machine review of arXiv:2601.18511}
}
read the original abstract

As large language models (LLMs) become ubiquitous, privacy concerns pertaining to inference keep growing. Fully homomorphic encryption (FHE) has emerged as a primary cryptographic solution for non-interactive confidential LLM inference. However, existing solutions scale poorly with input token length, focusing on small models or input sizes. They also suffer from large outlier values, which strongly impact the evaluation of non-linear layers, leading to heavy polynomial approximation costs. We scale up FHE-based LLM inference in two directions. First, we accelerate FHE-based inference for 128 encrypted tokens. We adopt ML techniques (token prepending and orthogonal rotations) to mitigate outlier impacts on the FHE evaluation of non-linear layers. Separately, we devise a novel polynomial evaluation method for sparsely-packed ciphertexts to speed up our homomorphic SoftMax implementation. We combine these with recent fast homomorphic linear algebra techniques, achieving significantly improved efficiency. Second, we expand the prompt size up to thousands of tokens for contexts where only the final part of the input is sensitive and encrypted. Processing this requires handling standard plaintext-plaintext and ciphertext-ciphertext components, alongside a wide homomorphic computation for a novel plaintext-ciphertext component. To address this, we devise a dedicated homomorphic linear algebra algorithm, building a shallow homomorphic attention circuit that minimizes bootstrapping costs. Based on these ingredients, we present a CKKS-based end-to-end implementation of Llama-3-8B private inference. On 8 NVIDIA RTX PRO 6000 GPUs, 128 encrypted tokens take 20s for summarization and 18s/token for generation (vastly outperforming the SOTA 295s on costlier H100 GPUs). For a heterogeneous 4096-token input (last 128 encrypted), it takes 64s for summarization and 22s/token for generation.

Figures

Figures reproduced from arXiv: 2601.18511 by the authors.

Figure 1
Figure 1. Llama model architecture cost of deep homomorphic circuits. As such, organizing the compu￾tation so as to minimize the number of bootstraps is often a key to the design of efficient homomorphic algorithms. The bootstrapping process uses a significant part of the modulus that it restores, with a total amount usage directly connected to its precision. CKKS bootstrapping, when called on a ciphertext ct, assumes that it… view at source ↗
Figure 2
Figure 2. Visualization of the proposed plaintext algorithm [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 3
Figure 3. Overview of encrypted computation. The red and orange symbols indicate the bootstrap placements. The bootstrapping [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. WHET: Welding Homomorphic Encryption to Accelerator Architectures

    cs.CR 2026-06 unverdicted novelty 6.0 of 10

    WHET applies fine-grained coefficient-to-slot transforms, plaintext compression, and modulus raising plus lightweight hardware tweaks to FHE accelerators, delivering 1.38-8.74x per-area gains and sub-millisecond CKKS ...

Reference graph

Works this paper leans on

3 extracted references · 2 linked inside Pith · cited by 1 Pith paper

  1. [3]

    [ZWS+25] L

    Available at https://arxiv.org/abs/2312.00025. [ZWS+25] L. Zhang, X. Wang, J. J. Sim, Z. Huang, J. Zhong, H. Wang, P. Duan, and K. Y. Lam. MOAI: Module-optimizing architecture for non-interactive secure transformer inference, 2025. Available at https://eprint.iacr.org/2025/991. [ZYH+25] J. Zhang, X. Yang, L. He, K. Chen, W.-j. Lu, Y. Wang, X. Hou, J. Liu,...

    arXiv 2025

  2. [2023]

    [TMS+23] H

    Available at https://arxiv.org/abs/2302.13971. [TMS+23] H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y. Babaei, N. Bash- lykov, S. Batra, P. Bhargava, S. Bhosale, D. Bikel, L. Blecher, C. Canton- Ferrer, M. Chen, G. Cucurull, D. Esiobu, J. Fernandes, J. Fu, W. Fu, B. Fuller, C. Gao, V. Goswami, N. Goyal, A. Hartshorn, S. Hosseini, R. Hou, H. ...

    arXiv 2023

  3. [2025]

    [HLL+23] X

    Available at https://eprint.iacr.org/2025/1935. [HLL+23] X. Hou, J. Liu, J. Li, Y. Li, W.-j. Lu, C. Hong, and K. Ren. CipherGPT: Secure two-party GPT inference. 2023. Available at https://eprint.iacr.org/2023/ 1147. [HS14] S. Halevi and V. Shoup. Algorithms in HElib. InCRYPTO, 2014. [HYT+24] J. He, K. Yang, G. Tang, Z. Huang, L. Lin, C. Wei, Y. Yan, and W...

    arXiv 2025

Pith tools

Reviewed August 3, 2026 · model on record in the stance chip above.