Pith. sign in

REVIEW 3 major objections 4 minor 1 cited by

TruncFormer: Private LLM Inference Using Only Truncations

T0 review · 3 major / 4 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read TruncFormer shows that private LLM inference can run on additions, multiplications, and statically scheduled truncations alone, cutting latency by up to 1.92x while matching plaintext accuracy.

desk verdict The core idea of statically scheduling truncations is promising, but Algorithm 1's width update is wrong for multiplications and the latency claims rest on unit tests—worth a serious referee, not a desk reject. read the letter →

arxiv 2412.01042 v1 pith:HFB634ID submitted 2024-12-02 cs.CR cs.LG

Patrick Yubeaton , Jianqiao Cambridge Mo , Karthik Garimella , Nandan Kumar Jha , Brandon Reagen , Chinmay Hegde , Siddharth Garg This is my paper · ORCID
classification cs.CRcs.LG
keywords privateinferencesecuremultipartycomputationlargelanguagemodelstruncationschedulingfixed-pointarithmeticgarbledcircuitsnonlinearityapproximationtransformer
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that the main latency cost in private LLM inference is not the exotic nonlinear functions that receive the most attention, but the routine truncation that follows every multiplication in a fixed-size cryptographic field. TruncFormer therefore turns private inference into a scheduling problem: examine the model's arithmetic graph once, compute worst-case bit growth, and insert a truncation only where the result could overflow the field. On Llama-7B and Gemma-2B, the method matches plaintext accuracy on nearly all tested benchmarks while cutting the dominant truncation cost by up to 1.92x relative to the standard truncate-after-every-multiplication recipe. A sympathetic reader would care because this points at a single operation that consumes 90-99% of private-inference time, giving the field one lever to pull instead of a moving target of new activation functions.

What carries the argument

The load-bearing object is a directed acyclic graph whose vertices are the model's additions and multiplications and whose edges carry worst-case bit widths. Starting from an input representation of $k$ bits, the scheduler sets each addition's output to $\max(m,n)+1$ bits and each multiplication's output to $m+n$ bits; if that exceeds the field size $F$, it inserts a truncation that resets the larger input to $k$ bits and repeats. Because the graph is known before any private computation, the resulting truncation list is static and data-independent. The same pass is paired with fixed-point encoding at scale $M=2^n$ and with Newton-Raphson and limit approximations for softmax, GeLU, SiLU, and layer norm, modified with lower initial guesses and more iterations so they converge at 64- and 128-bit field sizes.

What would settle it

Run a true end-to-end private inference of Llama-7B and Gemma-2B at 64- and 128-bit fields using the TruncFormer schedule, with actual secret-shared multiplications and garbled-circuit truncations, and compare total wall-clock time against the summed unit-test estimate; if protocol-level overheads materially change the ratio to the truncate-every-multiply baseline, the claimed speedups will not hold.

Watch

Extended reading notes

Core claim

The paper's central discovery is that the standard private-inference practice of truncating after every multiplication is wasteful, and that a static pass over the model's arithmetic graph can identify the only multiplications whose results can overflow the fixed field. For each addition the worst-case bit width is $\max(m,n)+1$; for each multiplication it is $m+n$; when that bound exceeds the field size, the scheduler resets the larger input to the input bit-width and records a truncation. Iterating this over the whole transformer produces a data-independent truncation schedule, and every out-of-field nonlinearity is replaced by iterative approximations built from additions, multiplications, and truncations. On Llama-7B and Gemma-2B this preserves accuracy on the vast majority of downstream benchmarks, with one acknowledged perplexity outlier, while cutting private-inference truncation latency by up to roughly 1.9x compared with a reimplementation of the previous state of the art.

Load-bearing premise

The load-bearing premise is that timing a single garbled-circuit truncation and a single secret-shared multiplication in isolation, then adding the costs over the whole model, faithfully predicts how long true end-to-end private inference will take.

Editorial extensions

If this is right

  • Because truncations consume 90-99% of private-inference time in the measured models, reducing the number of truncations is a larger lever than speeding any single nonlinearity approximation.
  • New differentiable activation functions no longer require bespoke private-inference protocols: they can be approximated by additions, multiplications, and truncations and then scheduled by the same static pass.
  • The truncation schedule is computed once per model and is data-independent, so it adds no per-query cryptographic cost.
  • Field size interacts with scheduling: going from 64 to 128 bits removes many truncations but roughly doubles each truncation's cost, yielding only 0.98x-1.04x net speedups, so the choice of field size is part of the optimization.
  • The method preserves plaintext accuracy on nearly all tested benchmarks, so the latency savings are not bought with degraded model quality.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper leaves open the natural next lever: choosing field size per layer or per operation instead of one global 64- or 128-bit field, since each truncation costs more in larger fields and a mixed schedule could beat both uniform choices.
  • The same static analysis should transfer to other differentiable models beyond transformers; the paper only demonstrates it on two LLM families, but the DAG scheduler is architecture-agnostic.
  • The paper's own operation breakdown predicts where speedups will show up in a real system: long-prompt, single-token settings gain most from softmax truncation savings, while activation-heavy autoregressive generation gains less, and this is a testable prediction for any end-to-end implementation.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper proposes TruncFormer, a framework for private LLM inference that statically schedules truncation operations in fixed-point arithmetic, avoiding the common practice of truncating after every multiplication. The framework replaces LLM nonlinearities with differentiable approximations built from additions, multiplications, and truncations, and determines truncation locations from a bit-width analysis of the model graph. The authors evaluate accuracy on Llama-7B and Gemma-2B across coding, math, commonsense, and perplexity benchmarks, and report truncation-latency speedups of 0.98x--1.92x over a self-implemented PUMA baseline. The core claim is that truncation scheduling, rather than bespoke nonlinearity approximations, is the key lever for faster private LLM inference.

Significance. If the static truncation scheduling is correct, the paper identifies a valuable and relatively unexplored optimization axis for private inference: deciding at compile time where truncations are actually needed, instead of after every multiplication. The accuracy results are broad and mostly close to plaintext, the operation-level breakdown is informative, and the authors have open-sourced their code. However, the central algorithm as written has a correctness bug in multiplication bit-width accounting, and the quantitative speedup claims rest on unit-test timings summed over the model graph rather than measured end-to-end private inference. These issues are fixable, but they currently prevent the paper's central claims from being fully supported.

major comments (3)
  1. [Algorithm 1 (Section 3)] The inner while loop recomputes `worst case <- max(E1,E2)+1` after truncating an operand, which is the addition rule and is incorrect for multiplication. After truncating one operand of a multiplication to k bits, the output width is k plus the other operand's width, not max(E1,E2)+1. For example, with F=64 and k=13, a product of two 52-bit operands has worst-case width 104 bits; truncating one operand to 13 bits leaves a 65-bit product, which still exceeds the field, but the pseudocode computes max(13,52)+1=53, stops, and records E3=53. This under-estimates downstream bit widths and contradicts the abstract's claim that truncations are inserted wherever overflows can occur. Please correct the multiplication rule and, if the released code mirrors the pseudocode, re-run the bit-width analysis and experiments; if the code uses the correct k+other rule, state that explicitly.
  2. [Section 4.1, Tables 2 and 4] All reported speedups are ratios of unit-test timings summed over the model graph, not measured end-to-end private inference. The PUMA baseline is the authors' reimplementation inside their own framework rather than the published PUMA system. Truncation time may dominate (Table 3), but the headline 0.98x--1.92x figures should be labeled as truncation-time speedups under a unit-test cost model, and the authors should report a sensitivity analysis over non-truncation overheads (communication, preprocessing, protocol orchestration) before claiming end-to-end latency improvements.
  3. [Abstract and Contribution 3] The title and contribution 3 claim private inference with "adds/multiplies and truncations alone," but Section 3 states that differentiable functions can be represented "with only truncations and comparisons as the out-of-field operations." If comparisons remain in the approximated graphs, their cost must be included and the claim should be revised; if not, the text should explain how comparisons are avoided.
minor comments (4)
  1. [Algorithm 2 (Section 3)] The variable `iterations` in `while iterations <= Niter` is never initialized; please specify the starting value and the loop increment.
  2. [Table 3] Table 3 should state explicitly that the percentages are for the TruncFormer model under the unit-test cost model, and should indicate the field size and input bit-width used.
  3. [Abstract and Contributions] The paper says "takes any transformer model," but experiments cover only Llama-7B and Gemma-2B; please soften the claim or add evidence for architectural generality.
  4. [Algorithm 1] The algorithm title contains the typo "Trunctions," and the notation `V <- (operation, E1, E2, E3)` and `H <- (V,H)` should be clarified so that the graph update is unambiguous.

Circularity Check

0 steps flagged · score 1.0 of 10

No circular derivation found: static truncation scheduling and accuracy results are independently meaningful; the main quantitative concerns are benchmarking fairness and an Algorithm 1 correctness bug, not circularity.

full rationale

The paper's central claim is that truncations can be scheduled statically rather than after every multiplication. This claim is derived from a straightforward bit-width analysis: for each add/multiply vertex, the algorithm computes a worst-case output width and inserts a truncation only when that width exceeds the field size. The resulting truncation schedule is then evaluated by actually running plaintext emulations on Llama-7B and Gemma-2B, with accuracy compared against the original models and PUMA. No parameter is fitted to a subset of the reported results and then renamed as a prediction; the input bit width of 13 bits is selected empirically, but that is ordinary hyperparameter choice, not a fitted parameter masquerading as an output. The speedups over PUMA are transparently computed as ratios of unit-test truncation times, and because TruncFormer deliberately uses fewer truncations, the direction of the speedup follows by construction; this is a property of the method, not a circular inference. The quantitative comparison is weakened by two factors: the PUMA baseline is implemented by the authors inside their own framework rather than using the official SecretFlow system, and the truncation latency model follows the EMP-Toolkit benchmarking methodology of (Mo et al., 2023), which is a self-citation by a co-author. These are validity and fairness concerns, but they do not reduce the central claim to its own inputs. The manuscript's stated limitation that only the Delphi protocol is tested is likewise a scope limitation rather than circularity. Separately, Algorithm 1 appears to contain a genuine correctness bug: after truncating one operand of a multiplication, it recomputes the output width as max(E1,E2)+1, which is the addition rule, instead of using the truncated width plus the other operand's width; this could undercount product widths and miss required truncations. That is an important correctness risk in the central method, but it is not a circular-derivation pattern, so it does not raise the circularity score.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

The central claim rests on fixed-point arithmetic choices, approximation iteration counts, and a latency model built from unit tests. The most fragile element is the unit-test-to-end-to-end extrapolation: every reported speedup in Table 4 is computed under this assumption. No invented entities are introduced.

free parameters (3)
  • Input fixed-point bit width k = 13 bits
    Chosen empirically as the smallest representation that preserves accuracy on Hellaswag and WikiText-2-raw (Table 8); this choice determines the number and location of truncations and hence the reported speedups.
  • Approximation iteration counts = exponentiation 8, reciprocal 20, inverse square-root 12
    Set in Section 4.2; they trade accuracy against latency for softmax, layer norm, and activation approximations and directly affect both Table 4 accuracy and truncation cost.
  • Initial guess scaling for Newton-Raphson = modified from CrypTen (not fully specified)
    Section 3 states that CrypTen's initial guesses were lowered to ensure convergence and iteration counts increased; the exact choices are not enumerated and affect approximation accuracy and speed.
assumptions (5)
  • domain assumption All LLM nonlinearities (softmax, layer norm, GeLU, SiLU, inverse square-root) are differentiable and can be approximated by Newton-Raphson/Taylor/limit chains using only add/multiply/truncate.
    Invoked in Section 1 and used throughout Section 3; it underlies the claim that TruncFormer generalizes to any LLM. The paper validates this only for Llama-7B and Gemma-2B.
  • domain assumption The modified CrypTen approximations converge to acceptable accuracy at 64/128-bit fields with the chosen iteration counts.
    Section 3 says the original CrypTen approximations are stable only at 16-bit precision and are modified; accuracy is checked indirectly through downstream benchmarks, not through a formal error bound.
  • domain assumption Per-operation unit latencies (benchmarked scalar truncation and beaver-triple multiplication) can be summed to estimate end-to-end PI latency.
    Section 4.1 explicitly avoids end-to-end benchmarking and uses unit tests and a plaintext emulation; this assumption is load-bearing for all speedup numbers.
  • domain assumption Fixed-point scaling M=2^n preserves model behavior to the accuracy reported.
    Section 3 describes multiplying by M and rounding; Table 8 shows sharp degradation below 13 bits, so the assumption holds only with a sufficiently large n.
  • domain assumption Worst-case bit-width propagation in Algorithm 1 correctly identifies all overflow points.
    The algorithm tracks only bit counts, not actual numeric ranges or distributions, so a value could overflow even when worst-case bits are below the field size, or truncation may be inserted unnecessarily. No overflow analysis is provided.

how reviews work

0 comments
Cite this review

Pith. "Pith review of TruncFormer: Private LLM Inference Using Only Truncations." pith.science (2026). https://pith.science/paper/HFB634ID

@misc{pith2026241201042,
  author       = {Pith},
  title        = {Pith review of: TruncFormer: Private LLM Inference Using Only Truncations},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/HFB634ID}},
  note         = {Machine review of arXiv:2412.01042}
}
read the original abstract

Private inference (PI) serves an important role in guaranteeing the privacy of user data when interfacing with proprietary machine learning models such as LLMs. However, PI remains practically intractable due to the massive latency costs associated with nonlinear functions present in LLMs. Existing works have focused on improving latency of specific LLM nonlinearities (such as the Softmax, or the GeLU) via approximations. However, new types of nonlinearities are regularly introduced with new LLM architectures, and this has led to a constant game of catch-up where PI researchers attempt to optimize the newest nonlinear function. We introduce TruncFormer, a framework for taking any LLM and transforming it into a plaintext emulation of PI. Our framework leverages the fact that nonlinearities in LLMs are differentiable and can be accurately approximated with a sequence of additions, multiplications, and truncations. Further, we decouple the add/multiply and truncation operations, and statically determine where truncations should be inserted based on a given field size and input representation size. This leads to latency improvements over existing cryptographic protocols that enforce truncation after every multiplication operation. We open source our code for community use.

Figures

Figures reproduced from arXiv: 2412.01042 by the authors.

Figure 1
Figure 1. Newton Raphson flowchart for the approximation of the inverse squareroot. Standard private inference protocols make use of a truncation after every operation which we show is excessive. This protocol assumes that both parties are honest-but￾curious which means that both parties will follow the pro￾tocol, but will attempt to learn information about the other party’s input. Delphi uses secret sharing and homomorphic e… view at source ↗
Figure 2
Figure 2. We compare the impact of input length and token gener￾ation count on private inference latency. to 0.175ms). Therefore, despite the number of truncations significantly decreasing, we spend a longer time on each truncation. This leads to minimal speedup despite a larger field size. Benchmark Field Size (bits) Accuracy (%) Speed-up (×) ARC-Easy 128 73.02 1.04 64 72.94 MMLU 128 29.98 0.98 64 29.92 [PITH_FULL_IMAGE:fig… view at source ↗
Figure 3
Figure 3. We approximate the inverse-sqrt/reciprocal of the range of numbers (0, 10] with intervals of 0.1. The plots showcase the error between these approximations and the true function as defined by PyTorch. We vary the number of Newton-Raphson iterations for different plotted lines. REFERENCES Austin, J., Odena, A., Nye, M., Bosma, M., Michalewski, H., Dohan, D., Jiang, E., Cai, C., Terry, M., Le, Q., et al. Program synth… view at source ↗

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. CrypTorch: PyTorch-based Auto-tuning Compiler for Machine Learning with Multi-party Computation

    cs.CR 2025-11 conditional novelty 6.0 of 10

    An MPC-ML compiler that modularizes and auto-tunes operator approximations, delivering 1.2–1.8x speedups over an optimized baseline under user-set accuracy bounds.

Reference graph

Works this paper leans on

22 extracted references · 5 canonical work pages · cited by 1 Pith paper

  1. [1]

    Program synthesis with large language models

    Austin, J., Odena, A., Nye, M., Bosma, M., Michalewski, H., Dohan, D., Jiang, E., Cai, C., Terry, M., Le, Q., et al. Program synthesis with large language models. arXiv preprint arXiv:2108.07732,

    arXiv

  2. [3]

    Privit: Vision transformers for fast private inference

    Dhyani, N., Mo, J., Cho, M., Joshi, A., Garg, S., Reagen, B., and Hegde, C. Privit: Vision transformers for fast private inference. arXiv preprint arXiv:2310.04604,

    arXiv

  3. [7]

    Kundu, S., Lu, S., Zhang, Y ., Liu, J., and Beerel, P. A. Learn- ing to linearize deep neural networks for secure and effi- cient private inference. arXiv preprint arXiv:2301.09254,

    arXiv

  4. [8]

    P., and Zhang, H

    Li, D., Shao, R., Wang, H., Guo, H., Xing, E. P., and Zhang, H. Mpcformer: fast, performant and private transformer inference with mpc. arXiv preprint arXiv:2211.01452,

    arXiv

  5. [9]

    Oblivious neu- ral network predictions via minionn transformations

    Liu, J., Juuti, M., Lu, Y ., and Asokan, N. Oblivious neu- ral network predictions via minionn transformations. In Proceedings of the 2017 ACM SIGSAC conference on computer and communications security , pp. 619–631,

    work page 2017

  6. [10]

    Secformer: Towards fast and accurate privacy- preserving inference for large language models

    Luo, J., Zhang, Y ., Zhang, J., Mu, X., Wang, H., Yu, Y ., and Xu, Z. Secformer: Towards fast and accurate privacy- preserving inference for large language models. arXiv preprint arXiv:2401.00793,

    arXiv

  7. [12]

    Pointer sentinel mixture models

    Merity, S., Xiong, C., Bradbury, J., and Socher, R. Pointer sentinel mixture models. arXiv preprint arXiv:1609.07843,

    arXiv

  8. [14]

    Glu variants improve transformer

    Shazeer, N. Glu variants improve transformer. arXiv preprint arXiv:2002.05202,

    arXiv 2002

Show all 22 references
  1. [15]

    S., Love, J., et al

    Team, G., Mesnard, T., Hardin, C., Dadashi, R., Bhupatiraju, S., Pathak, S., Sifre, L., Rivi`ere, M., Kale, M. S., Love, J., et al. Gemma: Open models based on gemini research and technology. arXiv preprint arXiv:2403.08295,

    arXiv

  2. [16]

    Llama: Open and efficient foundation lan- guage models

    Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.-A., Lacroix, T., Rozi`ere, B., Goyal, N., Hambro, E., Azhar, F., et al. Llama: Open and efficient foundation lan- guage models. arXiv preprint arXiv:2302.13971, 2023a. Touvron, H., Martin, L., Stone, K., Albert, P...

    arXiv

  3. [17]

    Quick and (not so) dirty: Unsupervised selection of justification sen- tences for multi-hop question answering

    Yadav, V ., Bethard, S., and Surdeanu, M. Quick and (not so) dirty: Unsupervised selection of justification sen- tences for multi-hop question answering. arXiv preprint arXiv:1911.07176,

    1911 arXiv

  4. [20]

    Mpcvit: Searching for accurate and efficient mpc-friendly vision transformer with het- erogeneous attention

    Zeng, W., Li, M., Xiong, W., Tong, T., Lu, W.-j., Tan, J., Wang, R., and Huang, R. Mpcvit: Searching for accurate and efficient mpc-friendly vision transformer with het- erogeneous attention. arXiv preprint arXiv:2211.13955,

    arXiv

  5. [21]

    Primer: Fast private transformer inference on encrypted data

    Zheng, M., Lou, Q., and Jiang, L. Primer: Fast private transformer inference on encrypted data. In 2023 60th ACM/IEEE Design Automation Conference (DAC) , pp. 1–6. IEEE,

    2023

  6. [22]

    Y ., Vu, M

    Zhuo, T. Y ., Vu, M. C., Chim, J., Hu, H., Yu, W., Widyasari, R., Yusuf, I. N. B., Zhan, H., He, J., Paul, I., et al. Big- codebench: Benchmarking code generation with diverse function calls and complex instructions. arXiv preprint arXiv:2406.15877, 2024

    2024 arXiv

  7. [2014]

    Hellaswag: Can a machine really finish your sentence? arXiv preprint arXiv:1905.07830,

    Zellers, R., Holtzman, A., Bisk, Y ., Farhadi, A., and Choi, Y . Hellaswag: Can a machine really finish your sentence? arXiv preprint arXiv:1905.07830,

    1905 arXiv

  8. [2016]

    Mishra, P., Lehmkuhl, R., Srinivasan, A., Zheng, W., and Popa, R. A. Delphi: A cryptographic inference system for neural networks. In Proceedings of the 2020 Workshop on Privacy-Preserving Machine Learning in Practice, pp. 27–30,

    2020

  9. [2019]

    TruncFormer Yao, A. C.-C. How to generate and exchange secrets. In27th annual symposium on foundations of computer science (Sfcs 1986), pp. 162–167. IEEE,

    1986

  10. [2020]

    Measuring math- ematical problem solving with the math dataset

    Hendrycks, D., Burns, C., Kadavath, S., Arora, A., Basart, S., Tang, E., Song, D., and Steinhardt, J. Measuring math- ematical problem solving with the math dataset. arXiv preprint arXiv:2103.03874,

    arXiv

  11. [2021]

    Sphynx: A deep neural network design for private inference

    Cho, M., Ghodsi, Z., Reagen, B., Garg, S., and Hegde, C. Sphynx: A deep neural network design for private inference. IEEE Security & Privacy, 20(5):22–34, 2022a. Cho, M., Joshi, A., Reagen, B., Garg, S., and Hegde, C. Selective network linearization for efficient private infer...

    arXiv

  12. [2022]

    Measuring mas- sive multitask language understanding

    Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., and Steinhardt, J. Measuring mas- sive multitask language understanding. arXiv preprint arXiv:2009.03300,

    2009 arXiv

  13. [2023]

    Puma: Se- cure inference of llama-7b in five minutes

    Dong, Y ., Lu, W.-j., Zheng, Y ., Wu, H., Zhao, D., Tan, J., Huang, Z., Hong, C., Wei, T., and Cheng, W. Puma: Se- cure inference of llama-7b in five minutes. arXiv preprint arXiv:2307.12533,

  14. [2024]

    {SecretFlow- SPU}: A performant and {User-Friendly} framework for {Privacy-Preserving} machine learning

    Ma, J., Zheng, Y ., Feng, J., Zhao, D., Wu, H., Fang, W., Tan, J., Yu, C., Zhang, B., and Wang, L. {SecretFlow- SPU}: A performant and {User-Friendly} framework for {Privacy-Preserving} machine learning. In 2023 USENIX Annual Technical Conference (USENIX ATC 23), pp. 17– 33,

    2023

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.