Pith. sign in

REVIEW 3 major objections 4 minor 9 references

Benchmarking Confidential GPU Inference on NVIDIA H100 under Intel TDX

T0 review · 3 major / 4 minor · reviewed 2026-08-02 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Confidential mode on NVIDIA H100 GPUs costs 21–30% extra latency and 18–21% token throughput.

desk verdict Useful operational data on a narrow H100/TDX configuration, but the fixed-rate throughput table has a definitional inconsistency that the authors need to resolve before the headline numbers can be trusted. read the letter →

arxiv 2607.19353 v1 pith:FGTEOQJV submitted 2026-05-20 cs.AI cs.LG

Wei Wang , Abdul Hyee Waqas , Burns Smith This is my paper · ORCID
classification cs.AIcs.LG
keywords confidentialcomputingGPUinferenceNVIDIAH100IntelTDXlargelanguagemodelsperformancebenchmarkingtimetofirsttokensaturationknee
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to quantify, for a single NVIDIA H100 80GB GPU in an Intel TDX confidential VM, how much performance a serving team gives up when they enable confidential computing. Using two models (Mistral-7B and Qwen3-30B-A3B) and two load patterns, it finds a consistent latency penalty and throughput gap in confidential mode: TTFT and request latency rise by roughly 21% for the smaller model and 27–30% for the larger one, while global token throughput falls by 17.7% and 21.1%. In closed-loop concurrency tests, throughput gaps stay within 11.5–20.2%, and the larger model hits its saturation knee at a lower concurrency under confidential mode. A sympathetic reader would care because this gives deployment teams a concrete starting heuristic: reserve 15–25% extra capacity and measure the saturation knee per model rather than assuming a fixed overhead.

What carries the argument

The load-bearing apparatus is a paired benchmark design: the same models, same machine type, same GPU, run in Non-CC and CC modes under two load generators. The fixed request-rate generator (10 req/s for Mistral, 5 req/s for Qwen) isolates per-request latency and token throughput; the closed-loop generator holds N in-flight requests to expose throughput saturation curves. The analytic concept that carries the argument is the saturation knee — the concurrency where throughput stops scaling — because the paper uses its left/right position to explain TTFT anomalies (e.g., Qwen CC TTFT of 20.8 s at c=32 vs 2.9 s Non-CC) and to argue for model-specific capacity planning.

What would settle it

Re-run the closed-loop Qwen3-30B-A3B experiment at c=32 with, say, ten trials, explicit warmup, and fresh instance placement; if CC p50 TTFT drops from the reported 20.8 s to near the Non-CC value, the 'earlier saturation knee under CC' conclusion would still be supported by the throughput plateau at 0.84 req/s, but the extreme TTFT anomaly would be an artifact rather than a steady-state property.

Watch

Extended reading notes

Core claim

The central claim is that confidential mode on this H100/TDX configuration has a measurable, user-visible cost and that the cost is not a single multiplier. Fixed request-rate measurements show a latency overhead of about 21% for Mistral-7B and 27–30% for Qwen3-30B-A3B, plus global token throughput gaps of 17.7% and 21.1%. Closed-loop concurrency tests show a throughput gap between 11.5% and 20.2%, but also show that Qwen3-30B-A3B saturates around c=32 in CC mode while Non-CC keeps improving to c=40. The paper concludes that confidential GPU inference is operationally feasible but must be capacity-planned model-by-model, and that TTFT comparisons are only meaningful when both modes are at th

Load-bearing premise

The results assume that three averaged trials per concurrency level give stable p50 latency and throughput estimates, and that the TDX instance differs from the Non-CC instance only by confidential mode; a single noisy trial — like the c=32 Qwen3 TTFT — could shift the saturation-knee conclusion.

Editorial extensions

If this is right

  • A serving team can start capacity planning for single-H100 confidential inference by reserving 15–25% throughput headroom, then validating with workload-specific traces.
  • Larger models may hit their saturation knee at lower concurrency in CC mode, so sizing from low-load measurements can under-provision production traffic.
  • TTFT comparisons between CC and Non-CC are only interpretable when both modes are in the same operating regime relative to their saturation knee.
  • Confidential mode does not make the GPU unusable: at moderate concurrency both models still serve requests with a throughput gap mostly in the 12–20% range.
  • The stable across-model gaps support treating confidential mode as a performance profile to be tuned rather than a binary overhead.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the overhead source is CPU-GPU transfer or memory encryption (as prior work suggests), then workloads with high prompt-token volume or frequent KV-cache transfers could see larger gaps than these two models show; the paper's own Section 7 lists this as an unmeasured factor.
  • A natural extension is to repeat the closed-loop tests with more trials and warmup, and to report p90/p99 tail latencies; if the c=32 Qwen3 TTFT outlier is a scheduling artifact rather than steady-state, the saturation-knee story still holds but the magnitude of the CC penalty at the knee would need revision.
  • The 15–25% reserve heuristic is likely configuration-specific; a testable extension is to run the same protocol on multi-GPU or NVLink-connected systems, where peer-memory encryption may change both the gap and the knee location.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. This paper compares confidential (CC) and non-confidential (Non-CC) LLM inference on a single NVIDIA H100 80GB GPU hosted in an Intel TDX confidential VM. Using Mistral-7B v0.1 and Qwen3-30B-A3B, it reports fixed-rate TTFT/request-latency overheads of 21–30%, global token-throughput gaps of 17.7% and 21.1%, and closed-loop throughput gaps of 11.5–20.2%, concluding that CC mode is operationally feasible but requires 15–25% extra capacity headroom and model-specific saturation planning. The paper is candid about its limited scope, missing tail-latency data, and disabled Secure Boot. However, the central fixed-rate results in Table 2 are not internally coherent under any consistent unit/metric interpretation, which undermines the headline quantitative claims until the table is reconciled.

Significance. If the measurements hold, the paper would provide a useful, concrete data point for capacity planners evaluating confidential GPU inference on H100/TDX, and its separation of fixed-rate and closed-loop load models is methodologically sensible. The paper also explicitly lists reproducibility and tail-latency limitations, and it does not fit any free parameters to its claims. That said, the significance is conditional: the main overhead percentages and the 15–25% heuristic are direct arithmetic products of Table 2, so the table's metric incoherence must be resolved before those conclusions can be accepted.

major comments (3)
  1. [§3.3, Table 2] The fixed-rate table cannot support the headline overheads as reported. The paper states that the artifact labels fixed-rate latency metrics in seconds and that units are preserved. If TTFT/latency are seconds, a 279.56 s TTFT at 10 req/s is physically implausible and implies over three thousand in-flight requests. If the values are milliseconds, then per-request throughput of 28.50 tok/s over a 314.72 ms request gives roughly 9 generated tokens per request, while global throughput of 1458.65 tok/s at 10 req/s implies about 146 tokens per request, a ~16-fold discrepancy. The same issue appears for Qwen3: 5 req/s × 50.20 tok/s × 0.41451 s ≈ 104 tok/s vs. reported 993.8 tok/s. The only way to reconcile these numbers is if 'global throughput' includes prompt tokens or uses a different time window, but the paper never defines this metric or reports prompt/output length distributions. Please
  2. [§3.4, §5.2, Table 4] All closed-loop conclusions rest on three-trial averages per concurrency level, with no standard deviations, confidence intervals, or per-trial data. The Qwen3 c=32 TTFT anomaly (CC 20,823 ms vs. Non-CC 2,874 ms) is excluded as a saturation artifact, but without variance information a reader cannot tell whether this is a stable regime difference or a single noisy trial. The 'earlier saturation knee' claim for Qwen3 also depends on the small throughput differences at c=32 (0.84 vs. 0.97 req/s). Please provide per-trial values or at least min/max and standard deviation for all closed-loop points, and justify the exclusion of the c=32 TTFT point with data rather than assertion.
  3. [§4.2, §6.4, §9] The 17.7% and 21.1% global-throughput gaps and the 15–25% capacity-planning rule are directly computed from Table 2 values whose definitions are unresolved. Consequently, even if the closed-loop results are sound, the paper's central quantitative takeaways are not verifiable from the manuscript as written. Please either correct Table 2 with coherent metric definitions and recompute all derived percentages, or restrict the conclusions to the closed-loop measurements whose units and Little's-law consistency can be checked.
minor comments (4)
  1. [§3.3] The phrase 'positive overhead is interpreted as the relative increase of CC over Non-CC' followed by the throughput-gap formula (Non-CC − CC)/CC is unconventional. Calling a 17.7% value a 'drop' in the abstract and conclusion is misleading; it is a 17.7% gap relative to CC, not a 17.7% reduction relative to Non-CC. Please use consistent wording, e.g., 'CC global throughput is 17.7% lower than Non-CC.'
  2. [Table 2] The column header 'A vg overhead/gap' appears to contain a typo; the p50 latency and TTFT values should also specify whether these are trial-averaged p50s, as in the closed-loop table.
  3. [Appendix B] The reproducibility notes describe the artifact but do not provide a URL, checksum, or data-release identifier. If the artifact can be shared, please include a link or DOI.
  4. [§4.1] The text says 'average TTFT from 279.56 to 340.64' without units. Given the confusion in §3.3, please state units in every table and every mention of these values.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the paper is a direct empirical benchmark whose overhead claims are arithmetic restatements of its own measurements, with no fitted parameters or self-referential derivation chain.

full rationale

The paper is a straightforward measurement study. Its central claims (TTFT/latency overheads of 21–30%, global-throughput gaps of 17.7%/21.1%, closed-loop saturation behavior) are computed directly from the reported benchmark tables via the explicit definitions in Section 3.3: overhead is (CC−Non-CC)/Non-CC for latency and (Non-CC−CC)/CC for throughput. No model parameter is fitted to a subset of data and then used to predict a closely related quantity; no uniqueness theorem or prior result by the same authors is invoked as the justification for the conclusions. The 'saturation knee' is a descriptive characterization of the measured closed-loop curves, and identifying it from the same data is inherent to empirical benchmarking rather than circular. Related-work citations to Zhu et al. and Martinez Ibarra et al. are contextual and not load-bearing for the paper's own measurements. The limitations section candidly notes missing tail-latency data, disabled Secure Boot, and lack of low-level profiling; these are validity threats, not circularity. The internal-consistency concerns about Table 2 units (e.g., whether global token throughput includes prompt tokens) are correctness/definitional issues that would affect the validity of the conclusions, but they are not instances of the paper deriving X from Y where Y was defined in terms of X. Accordingly, the circularity score is 0.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

The benchmark is purely observational; no model parameters are fitted. The main input choices are the request rates, the concurrency grid, and the three-trial averaging, which shape the reported numbers and any saturation-knee interpretation.

free parameters (3)
  • Mistral-7B fixed request rate = 10.0 req/s
    Chosen by hand as a load level; measured overheads depend on this operating point, though the comparison is internal to that rate.
  • Qwen3-30B-A3B fixed request rate = 5.0 req/s
    Chosen by hand; the different rate for the larger model limits direct cross-model comparability of throughput numbers.
  • Concurrency levels tested = Mistral: 32,48,64,80,100; Qwen3: 24,32,40,50
    Selected to expose scaling behavior; the saturation-knee claim depends on this particular grid, and a different grid could shift the inferred knee.
assumptions (3)
  • domain assumption The cc_benchmarks tool accurately records TTFT, latency, and throughput
    All results depend on this; no independent validation of the tool is presented (§3.1, Appendix B).
  • domain assumption The TDX instance differs from Non-CC only in confidential mode
    Assumes cloud placement, software versions, GPU behavior, and background noise are otherwise identical; no repeated-zone or repeated-instance checks are reported (§3.1).
  • domain assumption Three trials per concurrency level yield stable p50 and average estimates
    No variance, confidence intervals, or warmup analysis are provided; a single outlier trial (such as the c=32 Qwen3 TTFT) could change saturation conclusions (§3.4, Appendix B).

how reviews work

0 comments
Cite this review

Pith. "Pith review of Benchmarking Confidential GPU Inference on NVIDIA H100 under Intel TDX." pith.science (2026). https://pith.science/paper/FGTEOQJV

@misc{pith2026260719353,
  author       = {Pith},
  title        = {Pith review of: Benchmarking Confidential GPU Inference on NVIDIA H100 under Intel TDX},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/FGTEOQJV}},
  note         = {Machine review of arXiv:2607.19353}
}
read the original abstract

Confidential computing is becoming a practical deployment requirement for AI inference workloads that process sensitive inputs or protect proprietary model assets. However, the performance cost of enabling confidential execution for GPU-accelerated large language model serving remains workload dependent and operationally important. This paper presents a benchmark study comparing standard non-confidential execution with confidential computing mode on a single NVIDIA H100 80GB GPU hosted in an Intel TDX confidential instance. The evaluation uses two representative language models, Mistral-7B v0.1 and Qwen3-30B-A3B, and measures time to first token, end-to-end request latency, per-request token generation throughput, global token throughput, and closed-loop request throughput under increasing concurrency. In fixed request-rate experiments, confidential mode increases average TTFT by 21.8% for Mistral-7B and 27.8% for Qwen3-30B-A3B, while global token throughput drops by 17.7% and 21.1%, respectively. In closed-loop concurrency experiments, throughput gaps remain in the 11.5-20.2% range, but the larger model reaches its saturation knee earlier under confidential mode. The results suggest that confidential GPU inference can retain usable throughput under load, but capacity planning must account for both the steady throughput penalty and the earlier saturation behavior observed for larger models.

Figures

Figures reproduced from arXiv: 2607.19353 by the authors.

Figure 1
Figure 1. Fixed request-rate CC versus Non-CC comparison using average values from [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Mistral-7B request throughput under closed-loop load. Points show trial-averaged request throughput for CC and Non-CC. 5 [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. Qwen3-30B-A3B request throughput under closed-loop load. Points show trial-averaged request throughput for CC and Non-CC. 5.1 Throughput Scaling Mistral-7B throughput grows with concurrency for both modes and begins to plateau in the 80–100 concurrency range. At c = 100, CC reaches 1.32 req/s while Non-CC reaches 1.54 req/s, corresponding to a 16.7% throughput gap. Across the tested Mistral range, the throughput gap… view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Mistral-7B p50 end-to-end latency under closed-loop load. Points show trial-averaged p50 latency for CC and Non-CC [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: Qwen3-30B-A3B p50 end-to-end latency under closed-loop load. Points show trial-averaged p50 latency for CC and Non-CC. Mistral-7B p50 request latency is consistently higher in CC mode across all tested concurrencies. The overhead ranges from 12.3% to 15.6%, with no sin…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

9 extracted references · 4 linked inside Pith

  1. [1]

    Confidential Computing on NVIDIA Hopper GPUs: A Performance Benchmark Study,

    J. Zhu, H. Yin, P. Deng, A. Almeida, and S. Zhou, “Confidential Computing on NVIDIA Hopper GPUs: A Performance Benchmark Study,” arXiv:2409.03992, 2024. doi:https://doi.org/10. 48550/arXiv.2409.03992

    arXiv doi:10.48550/arxiv.2409.03992 2024

  2. [2]

    Performance of Confidential Computing GPUs,

    A. Martínez Ibarra, J. J. Stephen, A. González Vidal, K. R. Jayaram, and A. F. Skarmeta Gómez, “Performance of Confidential Computing GPUs,” arXiv:2505.16501, 2025. doi:https: //doi.org/10.48550/arXiv.2505.16501

    arXiv doi:10.48550/arxiv.2505.16501 2025

  3. [3]

    Confidential Compute on NVIDIA Hopper H100,

    NVIDIA, “Confidential Compute on NVIDIA Hopper H100,” White Paper WP-11459-001, 2023. https://images.nvidia.com/aem-dam/en-zz/Solutions/data-center/HCC-Whitepaper-v1. 0.pdf

    2023

  4. [4]

    Confidential Computing on H100 GPUs for Se- cure and Trustworthy AI,

    NVIDIA Developer Blog, “Confidential Computing on H100 GPUs for Se- cure and Trustworthy AI,” 2023. https://developer.nvidia.com/blog/ confidential-computing-on-h100-gpus-for-secure-and-trustworthy-ai/

    2023

  5. [5]

    Intel Trust Domain Extensions (Intel TDX),

    Intel, “Intel Trust Domain Extensions (Intel TDX),” developer documentation.https://www.intel. com/content/www/us/en/developer/tools/trust-domain-extensions/overview.html

  6. [6]

    Create a Confidential VM instance with GPU,

    Google Cloud, “Create a Confidential VM instance with GPU,” documentation. https://docs.cloud.google.com/confidential-computing/confidential-vm/docs/ create-a-confidential-vm-instance-with-gpu

  7. [7]

    Mistral 7B,

    A. Q. Jiang, A. Sablayrolles, A. Mensch, C. Bamford, D. S. Chaplot, D. de las Casas, F. Bressand, G. Lengyel, G. Lample, L. Saulnier, L. R. Lavaud, M.-A. Lachaux, P. Stock, T. Le Scao, T. Lavril, T. Wang, T. Lacroix, and W. El Sayed, “Mistral 7B,” arXiv:2310.06825, 2023. doi:https: //doi.org/10.48550/arXiv.2310.06825

    arXiv doi:10.48550/arxiv.2310.06825 2023

  8. [8]

    Qwen3 Technical Report,

    A. Yang et al., “Qwen3 Technical Report,” arXiv:2505.09388, 2025. doi:https://doi.org/10. 48550/arXiv.2505.09388

    arXiv doi:10.48550/arxiv.2505.09388 2025

Show all 9 references
  1. [9]

    Efficient Memory Management for Large Language Model Serving with PagedAttention,

    W. Kwon, Z. Li, S. Zhuang, Y. Sheng, L. Zheng, C. H. Yu, J. E. Gonzalez, H. Zhang, and I. Stoica, “Efficient Memory Management for Large Language Model Serving with PagedAttention,” inProceedings of the ACM Symposium on Operating Systems Principles (SOSP), 2023. doi: https: //...

    2023

Pith tools

Reviewed August 2, 2026 · model on record in the stance chip above.