REVIEW 3 major objections 6 minor 44 references
SecPE: Secure Prompt Ensembling for Private and Robust Large Language Models
T0 review · 3 major / 6 minor · reviewed 2026-08-09 · deepseek-v4-flash
Pith's one-line read SecPE shows that prompt ensembling, a defense that aggregates several prompted predictions, can run on encrypted LLM inputs by replacing the costly encrypted argmax with a logarithmic-depth comparison tree, cutting that step 35.4x and…
desk verdict Useful private argmax optimization, but a missing normalization step leaves the core algorithm incomplete as written. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is Algorithm 1, the private argmax built from a QuickMax reduction tree on an RNS-CKKS ciphertext. The sign function is approximated by two composed 9-degree polynomials with $\alpha=12$, $d_f=2$, $d_g=2$, giving a stated error bound below $10^{-4}$, and the polynomials are evaluated with the Baby-Step-Giant-Step algorithm to keep multiplicative depth low. Because all operations are SIMD, one ciphertext can carry many independent argmax problems at once, batching $N/(2n)$ inputs in parallel when the ciphertext has $N$ slots.
What would settle it
Take encrypted logit vectors whose pairwise differences exceed $[-1,1]$ (for example, raw logits in $[0,10]$), run Algorithm 1 exactly as written, and compare the decrypted one-hot vector with the plaintext argmax: any mismatch would show that correctness depends on a normalization step the paper does not specify.
Extended reading notes
Core claim
The central claim is that the encrypted argmax needed for prompt ensembling can be computed with a tree reduction rather than pairwise comparison. After duplicating the logit vector, a QuickMax loop rotates by $2^i$ slots at level $i$ and takes pairwise maxima using $M(a,b) = (a+b)/2 + ((a-b)/2)\cdot\text{Sign}(a-b)$, so after $\log n$ levels every slot holds the maximum logit. Subtracting that maximum and applying a polynomial approximation of the sign function followed by $+1$ turns the winning slot into 1 and all others into 0, producing an encrypted one-hot selection vector. This reduces the number of sign evaluations and ciphertext rotations from $n+1$ to $\log n + 1$. On ImageNet with 247 prompts, SecPE reaches 68.33% zero-shot accuracy versus 68.60% for plaintext and 66.71% for the prior encrypted argmax, and in the end-to-end setting the argmax runtime drops from 2060s to 891s, with the paper reporting 2.5–3.8% overall overhead over private inference without ensembling.
Load-bearing premise
The polynomial sign approximation in Algorithm 1 only returns a correct argmax if every pairwise logit difference lies in $[-1,1]$, and the paper's stated normalization via Eq. (5) requires bounds $D_{\min}$ and $D_{\max}$ that are never explained or shown in the pseudocode.
Editorial extensions
If this is right
- Private inference with prompt ensembling becomes practical: SecPE reports only a few percent extra runtime over private single-prompt inference while adding adversarial robustness.
- The log-depth comparison tree applies to any associative reduction on ciphertexts, so encrypted sums, maxima, and minima become cheaper whenever the inputs can be normalized in advance.
- The encrypted argmax alone is a reusable primitive: any FHE pipeline that needs a one-hot selection from encrypted scores can use it independently of the LLM setting.
- Robustness transfers through encryption: on adversarial GLUE tasks, SecPE's ensemble accuracy is close to the plaintext ensemble baseline and clearly above the private single-prompt baseline.
- Larger prompt sets become relatively cheaper because the argmax cost grows as $\log n$, not $n$, so the gap between SecPE and the previous method widens with more prompts.
Reading between the lines
- The same tree reduction may port to other encrypted selection tasks, such as top-$k$ voting or secure aggregation in analytics, but the paper does not evaluate those settings.
- If per-task bounds $D_{\min}$ and $D_{\max}$ can be fixed from public statistics of the logit distribution, the Eq. (5) normalization could be folded into the plaintext preprocessing before encryption; the paper leaves this step unspecified, which is the main gap a user must fill.
- Because SecPE uses fewer sign operations than the previous method, it also accumulates less homomorphic noise, so the accuracy advantage over the baseline should grow with the number of prompts rather than shrink.
- The speedup claims are for CPU with 128-bit-security parameters; using hardware acceleration for FHE would likely multiply both methods' throughput, but the relative $O(\log n)$ versus $O(n)$ advantage should remain.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This paper proposes SecPE, a framework for private and robust LLM inference by combining fully homomorphic encryption (RNS-CKKS) with prompt ensembling. The main technical contribution is an efficient encrypted argmax algorithm (Algorithm 1) that uses a polynomial approximation of the sign function inside a QuickMax procedure to compute the maximum of n encrypted logits with O(log n) sign evaluations and rotations, claimed to be 35.4x faster than the Phoenix baseline. The paper evaluates SecPE on GLUE, AdvGLUE, GSM8K/MultiArith, and CLIP-based image classification, reporting near-plaintext accuracy and improved adversarial robustness at small efficiency overhead.
Significance. If the technical gaps are resolved, SecPE would be a useful contribution: the log-depth argmax idea is simple and could benefit FHE-based inference beyond prompt ensembling. The paper also demonstrates empirically that prompt ensembling can be made private with modest overhead, and it provides a broad evaluation across NLP and vision tasks. The claimed accuracy results on CLIP with 247 prompts (68.33% vs. 68.60% plaintext) are particularly encouraging. However, the missing normalization procedure for the sign approximation and the unclear efficiency accounting currently prevent a full endorsement.
major comments (3)
- [Section 3.2, Eq. (5) and Algorithm 1] The pseudocode of Algorithm 1 does not perform the normalization that Eq. (5) states is required for the Sign approximation, and the paper never explains how D_min and D_max are obtained in the ciphertext domain. Since the server cannot compute the min or max of encrypted logits without comparison operations, and the user cannot know the logits, the proposed Argmax is incomplete as written. If the logits are not (or cannot be) normalized to the interval [-1,1] for pairwise differences, the polynomial Sign from Eq. (3)-(4) is evaluated out of its approximation range and can yield incorrect Max decisions inside QuickMax and an incorrect final one-hot vector. The authors should either specify a concrete homomorphic procedure for the normalization (e.g., public model-specific output bounds with clipping) or demonstrate that the logits in their experiments already satisfy the required range.
- [Section 3.2, Eq. (3) and Algorithm 1] The behavior of the approximate Sign for near-tie logit differences is not analyzed. For any input x with |x| < 2^{-alpha} (and for the polynomial approximation error in Eq. (4)), Sign(x) can return 0 or an incorrect value, so QuickMax may select the wrong element when two logits are very close. The authors provide no bound on the probability of such near-ties for their benchmarks, nor any experimental evidence that the reported accuracies are insensitive to this effect. This matters because the final one-hot vector is built from the same Sign evaluations.
- [Abstract and Section 4.5, Figures 6-7] The speedup claims are internally inconsistent. The abstract states that SecPE is 35.4x faster than state-of-the-art for encrypted Argmax, but Section 4.5 reports 20.8x speedup for input length 256, and Figure 7 shows Argmax time being reduced from 2060s to 891s (about 2.3x). The authors should reconcile these numbers and specify the exact input dimension and configuration for each reported speedup, since the efficiency gain is a central claimed contribution.
minor comments (6)
- [Section 4.1 and 4.2] The task counts are inconsistent: the abstract says "8 tasks", the introduction says "15 tasks from 4 popular benchmarks", while the experimental section actually evaluates 6 GLUE, 6 AdvGLUE, 2 math, and 7 image-classification tasks. Please clarify the counts.
- [Section 4.2] "GELU and AdvGELU" should be "GLUE and AdvGLUE".
- [Section 3.2] "Phoneix" should be "Phoenix".
- [Section 4.2] The text says "BERT is used as the large pre-trained language model" after earlier stating ALBERT-XXLarge-v2; please align these descriptions.
- [Equation (2)] Equation (2) is easy to misread: as written, Sign(yi - ymax) + 1 yields 2 for the maximum element; the correctness follows from the fact that in Algorithm 1 the role of y is played by y - ymax so the maximum becomes zero. Consider clarifying this to avoid confusion.
- [Section 2.2] The symbol L is used both for the multiplicative depth and for the language model; please disambiguate.
Circularity Check
No significant circularity: SecPE's private argmax is a direct construction using an externally cited sign-polynomial approximation; the missing logit normalization is a correctness gap, not a circularity.
full rationale
SecPE's claimed derivation chain is not circular. Algorithm 1's argmax construction is explicit: Eq. (2) defines z_i = Sign(y_i - y_max) + 1, Eq. (6) defines Max(a,b) from Sign, and QuickMax repeatedly applies Max. The claimed complexity of (log n + 1) sign evaluations and rotations follows by counting loop iterations in Algorithm 1. The sign polynomial in Eqs. (3)-(4) is cited to standard approximation literature (Han and Ki [10]) with fixed parameters alpha = 12, d_f = 2, d_g = 2 and a stated error bound; it is not fitted to the benchmark results. The speedup figures (20.8x for input length 256, 35.4x in the abstract) are timing measurements against Phoenix on the same RNS-CKKS setting, not predictions forced by a fit. The accuracy tables compare ciphertext and plaintext outputs, and no parameter is tuned on those tables to construct the central claim. The self-citation [42] appears only in background statements about privacy demands and is not load-bearing for the SecPE argmax design. The paper's real weakness is the normalization gap: Eq. (5) says inputs must be mapped to [0,1] using D_min and D_max before Sign is applied, but Algorithm 1's pseudocode never shows this normalization, and the paper does not explain how a server obtains these bounds homomorphically or how the benchmark logits satisfy the requirement. That is a correctness and reproducibility gap, not a circular derivation, because the missing normalization does not make the method's output equivalent to its input or to a fitted quantity.
Assumptions & free parameters
free parameters (3)
- Sign approximation parameters (alpha, df, dg) =
alpha=12, df=2, dg=2
- Polynomial degrees for f and g =
9, 9
- Normalization bounds D_min, D_max =
Not specified
assumptions (4)
- domain assumption All logits can be normalized so that pairwise differences lie in [-1,1] without leaking information.
- domain assumption The polynomial approximation of Sign has exactly zero value at x=0 and error less than 10^-4 throughout [-1,1].
- domain assumption Underlying private LLM inference (Step 2) is correct and can be implemented in FHE as described in prior works [11,21,42].
- standard math RNS-CKKS homomorphic operations have the stated SIMD and rotation semantics.
Cite this review
Pith. "Pith review of SecPE: Secure Prompt Ensembling for Private and Robust Large Language Models." pith.science (2026). https://pith.science/paper/HEXSZIRT
@misc{pith2026250200847,
author = {Pith},
title = {Pith review of: SecPE: Secure Prompt Ensembling for Private and Robust Large Language Models},
year = {2026},
howpublished = {\url{https://pith.science/paper/HEXSZIRT}},
note = {Machine review of arXiv:2502.00847}
}
abstract
With the growing popularity of LLMs among the general public users, privacy-preserving and adversarial robustness have become two pressing demands for LLM-based services, which have largely been pursued separately but rarely jointly. In this paper, to the best of our knowledge, we are among the first attempts towards robust and private LLM inference by tightly integrating two disconnected fields: private inference and prompt ensembling. The former protects users' privacy by encrypting inference data transmitted and processed by LLMs, while the latter enhances adversarial robustness by yielding an aggregated output from multiple prompted LLM responses. Although widely recognized as effective individually, private inference for prompt ensembling together entails new challenges that render the naive combination of existing techniques inefficient. To overcome the hurdles, we propose SecPE, which designs efficient fully homomorphic encryption (FHE) counterparts for the core algorithmic building blocks of prompt ensembling. We conduct extensive experiments on 8 tasks to evaluate the accuracy, robustness, and efficiency of SecPE. The results show that SecPE maintains high clean accuracy and offers better robustness at the expense of merely $2.5\%$ efficiency overhead compared to baseline private inference methods, indicating a satisfactory ``accuracy-robustness-efficiency'' tradeoff. For the efficiency of the encrypted Argmax operation that incurs major slowdown for prompt ensembling, SecPE is 35.4x faster than the state-of-the-art peers, which can be of independent interest beyond this work.
Figures
Figures from the paper (3 more)
Reference graph
Works this paper leans on
-
[1]
J. U. Allingham, J. Ren, M. W. Dusenberry, X. Gu, Y . Cui, D. Tran, J. Z. Liu, and B. Lakshminarayanan. A simple zero-shot prompt weighting technique to improve prompt ensembling in text-image models. In In- ternational Conference on Machine Learning, pages 547–568. PMLR, 2023
work page 2023
-
[2]
J.-P. Bossuat, C. Mouchet, J. Troncoso-Pastoriza, and J.-P. Hubaux. Efficient bootstrapping for approximate homomorphic encryption with non-sparse keys. In Annual International Conference on the Theory and Applications of Cryptographic Techniques , pages 587–617. Springer, 2021
work page 2021
-
[3]
Brown, B
T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al. Language mod- els are few-shot learners. Advances in neural information processing systems, 33:1877–1901, 2020
1901
-
[4]
M. Chen, J. Tworek, H. Jun, Q. Yuan, H. P. d. O. Pinto, J. Kaplan, H. Ed- wards, Y . Burda, N. Joseph, G. Brockman, et al. Evaluating large lan- guage models trained on code. arXiv preprint arXiv:2107.03374, 2021
arXiv 2021
-
[5]
J. H. Cheon, K. Han, A. Kim, M. Kim, and Y . Song. A full rns vari- ant of approximate homomorphic encryption. In Selected Areas in Cryptography–SAC 2018: 25th International Conference, Calgary, AB, Canada, August 15–17, 2018, Revised Selected Papers 25 , pages 347–
work page 2018
- [6]
-
[7]
N. Dvornik, C. Schmid, and J. Mairal. Diversity with cooperation: Ensemble methods for few-shot classification. In Proceedings of the IEEE/CVF international conference on computer vision , pages 3723– 3731, 2019
work page 2019
-
[8]
T. Gao, A. Fisch, and D. Chen. Making pre-trained language mod- els better few-shot learners. In C. Zong, F. Xia, W. Li, and R. Nav- igli, editors, Proceedings of the 59th Annual Meeting of the Associ- ation for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Pa- pers), pages 3816–3830, ...
Show all 44 references
-
[9]
C. Gentry. A fully homomorphic encryption scheme . Stanford univer- sity, 2009
2009
-
[10]
Han and D
K. Han and D. Ki. Better bootstrapping for approximate homomorphic encryption. In Cryptographers’ Track at the RSA Conference , pages 364–390. Springer, 2020
2020
-
[11]
M. Hao, H. Li, H. Chen, P. Xing, G. Xu, and T. Zhang. Iron: Private inference on transformers. Advances in Neural Information Processing Systems, 35:15718–15731, 2022
2022
-
[12]
Jovanovic, M
N. Jovanovic, M. Fischer, S. Steffen, and M. Vechev. Private and re- liable neural network inference. In Proceedings of the 2022 ACM SIGSAC Conference on Computer and Communications Security, pages 1663–1677, 2022
2022
-
[13]
Juvekar, V
C. Juvekar, V . Vaikuntanathan, and A. Chandrakasan. {GAZELLE}: A low latency framework for secure neural network inference. In 27th USENIX Security Symposium (USENIX Security 18), pages 1651–1669, 2018
2018
-
[14]
J. D. M.-W. C. Kenton and L. K. Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of naacL-HLT, volume 1, page 2, 2019
2019
-
[15]
Z. Lan, M. Chen, S. Goodman, K. Gimpel, P. Sharma, and R. Soricut. Albert: A lite bert for self-supervised learning of language representa- tions. arXiv preprint arXiv:1909.11942, 2019
1909 arXiv
-
[16]
Lee, J.-W
E. Lee, J.-W. Lee, J. Lee, Y .-S. Kim, Y . Kim, J.-S. No, and W. Choi. Low-complexity deep convolutional neural networks on fully homo- morphic encryption using multiplexed parallel convolutions. InInterna- tional Conference on Machine Learning , pages 12403–12422. PMLR, 2022
2022
-
[17]
Y . Li, Z. Lin, S. Zhang, Q. Fu, B. Chen, J.-G. Lou, and W. Chen. Making language models better reasoners with step-aware verifier. In A. Rogers, J. Boyd-Graber, and N. Okazaki, editors, Proceedings of the 61st An- nual Meeting of the Association for Computational Linguistics ...
2023 doi
-
[18]
Li, Y .-L
Y . Li, Y .-L. Tsai, C.-M. Yu, P.-Y . Chen, and X. Ren. Exploring the ben- efits of visual prompting in differential privacy. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5158– 5167, 2023
2023
-
[19]
C. Mauran. Whoops, samsung workers accidentally leaked trade se- crets via chatgpt. Mashable [online]. Dostupné z: https://mashable. com/article/samsungchatgpt-leak-details, 2023
2023
-
[20]
Italy orders chatgpt blocked citing data protection concerns
Natasha Lomas. Italy orders chatgpt blocked citing data protection concerns. https://techcrunch.com/2023/03/31/chatgpt-blocked-italy/,
2023
-
[21]
Q. Pang, J. Zhu, H. Möllering, W. Zheng, and T. Schneider. Bolt: Privacy-preserving, accurate and efficient inference for transformers. IEEE Symposium on Security and Privacy (SP), 2024
2024
-
[22]
Radford, J
A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, et al. Learning transferable visual models from natural language supervision. In International con- ference on machine learning, pages 8748–8763. PMLR, 2021
2021
-
[23]
Raffel, N
C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y . Zhou, W. Li, and P. J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. The Journal of Machine Learn- ing Research, 21(1):5485–5551, 2020
2020
-
[24]
Roy and D
S. Roy and D. Roth. Solving general arithmetic word problems. arXiv preprint arXiv:1608.01413, 2016
2016 arXiv
-
[25]
Schick and H
T. Schick and H. Schütze. Exploiting cloze questions for few shot text classification and natural language inference. arXiv preprint arXiv:2001.07676, 2020
2001 arXiv
-
[26]
Schick and H
T. Schick and H. Schütze. Exploiting cloze-questions for few-shot text classification and natural language inference. In P. Merlo, J. Tiede- mann, and R. Tsarfaty, editors, Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguis-...
2021 doi
-
[27]
Schick, H
T. Schick, H. Schmid, and H. Schütze. Automatically identifying words that can serve as labels for few-shot text classification. arXiv preprint arXiv:2010.13641, 2020
2010 arXiv
-
[28]
X. Shen, B. Tan, and C. Zhai. Privacy protection in personalized search. In ACM SIGIR Forum, volume 41, pages 4–17. ACM New York, NY , USA, 2007
2007
-
[29]
Shokri, M
R. Shokri, M. Stronati, C. Song, and V . Shmatikov. Membership infer- ence attacks against machine learning models. In2017 IEEE symposium on security and privacy (SP), pages 3–18. IEEE, 2017
2017
-
[30]
W. Z. Srinivasan, P. Akshayaram, and P. R. Ada. Delphi: A crypto- graphic inference service for neural networks. In Proc. 29th USENIX Secur. Symp, pages 2505–2522, 2019
2019
-
[31]
Viand, P
A. Viand, P. Jattke, M. Haller, and A. Hithnawi. {HECO}: Fully ho- momorphic encryption compiler. In 32nd USENIX Security Symposium (USENIX Security 23), pages 4715–4732, 2023
2023
-
[32]
B. Wang, C. Xu, S. Wang, Z. Gan, Y . Cheng, J. Gao, A. H. Awadallah, and B. Li. Adversarial glue: A multi-task benchmark for robustness evaluation of language models.arXiv preprint arXiv:2111.02840, 2021
2021 arXiv
-
[33]
Y . Wang, Y . Chen, Z. Li, Z. Tang, R. Guo, X. Wang, Q. Wang, A. C. Zhou, and X. Chu. Towards efficient and reliable llm serving: A real- world workload study. arXiv preprint arXiv:2401.17644, 2024
2024 arXiv
-
[34]
Z. Wang, P. Li, R. Hou, Z. Li, J. Cao, X. Wang, and D. Meng. He- booster: An efficient polynomial arithmetic acceleration on gpus for fully homomorphic encryption. IEEE Transactions on Parallel and Dis- tributed Systems, 34(4):1067–1081, 2023
2023
-
[35]
J. Wei, X. Wang, D. Schuurmans, M. Bosma, F. Xia, E. Chi, Q. V . Le, D. Zhou, et al. Chain-of-thought prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems, 35:24824–24837, 2022
2022
-
[36]
H. Xu, B. Liu, L. Shu, and P. S. Yu. Bert post-training for review read- ing comprehension and aspect-based sentiment analysis. arXiv preprint arXiv:1904.02232, 2019
1904 arXiv
-
[37]
H. Yan, S. Li, Y . Wang, Y . Zhang, K. Sharif, H. Hu, and Y . Li. Mem- bership inference attacks against deep learning models via logits dis- tribution. IEEE Transactions on Dependable and Secure Computing , 2022
2022
-
[38]
W. Yang, Y . Xie, A. Lin, X. Li, L. Tan, K. Xiong, M. Li, and J. Lin. End- to-end open-domain question answering with bertserini. arXiv preprint arXiv:1902.01718, 2019
1902 arXiv
-
[39]
W. Yang, H. Zhang, and J. Lin. Simple applications of bert for ad hoc document retrieval. arXiv preprint arXiv:1903.10972, 2019
1903 arXiv
-
[40]
J. Ye, A. Maddi, S. K. Murakonda, V . Bindschaedler, and R. Shokri. En- hanced membership inference attacks against machine learning models. In Proceedings of the 2022 ACM SIGSAC Conference on Computer and Communications Security, pages 3093–3106, 2022
2022
-
[41]
W. Zeng, M. Li, W. Xiong, T. Tong, W.-j. Lu, J. Tan, R. Wang, and R. Huang. Mpcvit: Searching for accurate and efficient mpc-friendly vision transformer with heterogeneous attention. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5052– 5063, 2023
2023
-
[42]
Zhang, X
J. Zhang, X. Yang, L. He, K. Chen, W.-j. Lu, Y . Wang, X. Hou, J. Liu, K. Ren, and X. Yang. Secure transformer inference made non- interactive. Cryptology ePrint Archive, 2024
2024
-
[291]
URL https://aclanthology.org/2023.acl-long.291
2023
-
[2023]
Accessed: 2023-05-28
2023
Reviewed August 9, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.