arxiv: 2605.10352 · v1 · submitted 2026-05-11 · 📡 eess.SP

Recognition: no theorem link

Quantifying System Level KPI Deviations of Sionna RT: Material and Near-Field Error Analysis Using a 5G OAI Testbed

Aydin Sezgin, Faizan Rauf, Markus Heinrichs, Srijita Sanyal

Pith reviewed 2026-05-12 05:10 UTC · model grok-4.3

classification 📡 eess.SP

keywords ray tracing5G NRdigital twinchannel modelingKPInear-fieldSionnaOAI testbed

0 comments

The pith

Sionna ray tracing produces KPI deviations in 5G systems mainly from near-field antenna effects and material mismatches.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper compares Sionna RT simulated channels to vector network analyzer measurements at 20 indoor positions. These are injected into a 5G OAI testbed via hardware-in-the-loop emulation to assess real hardware KPIs including RSRP, PUCCH SNR, and SINR. It establishes that near-field transition effects and material property errors are key sources of deviation, offering a benchmark for digital twin applications in 5G network planning. A reader cares because this quantifies how modeling inaccuracies affect actual system performance rather than just channel statistics.

Core claim

By contrasting ray-tracing channels from Sionna with measured channels from a VNA and evaluating them on an OpenAirInterface 5G NR testbed with a channel emulator, the authors find that antenna near-field transition effects act as a critical position-dependent error source alongside material property mismatch, leading to deviations in system level KPIs.

What carries the argument

The hardware-in-the-loop channel emulator that interfaces the OAIBOX MAX base station and Quectel UE, allowing direct comparison of RT-simulated and VNA-measured channels on real 5G hardware.

If this is right

Material property databases in ray tracing tools require higher fidelity to reduce KPI prediction errors in digital twins.
Position-dependent near-field corrections should be integrated into RT models for indoor 5G scenarios.
Quantitative benchmarks like these can guide validation of other RT simulators for beyond-5G network planning.
System-level KPI evaluation reveals error propagation not visible in channel-level comparisons alone.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

Applying similar tests in larger or outdoor environments could reveal whether near-field effects remain dominant or if other factors like diffraction take precedence.
Digital twin frameworks might benefit from adaptive modeling that switches between RT and empirical corrections based on distance to antennas.
This analysis could extend to other frequencies or 6G scenarios where near-field regions are larger due to massive MIMO arrays.

Load-bearing premise

The specific indoor laboratory environment and the 20 chosen positions, combined with the testbed setup, isolate the RT modeling inaccuracies without significant interference from hardware imperfections or atypical propagation conditions.

What would settle it

Conducting the same comparison in an anechoic chamber or with accurately measured material properties and finding no significant KPI deviation would falsify the identification of these as primary error sources.

Figures

Figures reproduced from arXiv: 2605.10352 by Aydin Sezgin, Faizan Rauf, Markus Heinrichs, Srijita Sanyal.

**Figure 2.** Figure 2: The instrument provides two measurement ports, de [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗

**Figure 4.** Figure 4: Physical hardware testbed used for end to end 5G NR [PITH_FULL_IMAGE:figures/full_fig_p004_4.png] view at source ↗

**Figure 5.** Figure 5: E-field magnitude of the Schwarzbeck BBHA 9120L [PITH_FULL_IMAGE:figures/full_fig_p004_5.png] view at source ↗

**Figure 6.** Figure 6: Spatial distribution of absolute prediction errors [PITH_FULL_IMAGE:figures/full_fig_p005_6.png] view at source ↗

read the original abstract

Ray tracing (RT) has recently gained renewed interest in wireless communications, driven by its integration into digital twin (DT) frameworks for site specific channel modeling. Several previous studies have validated RT at the channel level, yet how these errors propagate into real 5G system level key performance indicators (KPIs) on actual hardware remains unquantified. This paper addresses this gap by comparing Sionna RT simulated channels against vector network analyzer (VNA) measured channels using an OpenAirInterface (OAI) 5G NR testbed. Channel measurements are conducted at 20 receiver positions in an indoor laboratory, with both channel types injected into a hardware in the loop channel emulator interfacing an OAIBOX MAX base station and a Quectel UE. RSRP, PUCCH SNR, and SINR are evaluated under both conditions. The results identify antenna near-field transition effects as a critical position-dependent error source, alongside material property mismatch, providing a quantitative benchmark for digital twin-based 5G and beyond network planning.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Desk Editor's Note private letter to a colleague

The paper moves RT validation to real 5G hardware KPIs with concrete indoor numbers, but the lack of emulator calibration leaves the error sources only partly isolated.

read the letter

The paper gives the first hardware-based numbers on how Sionna RT channel errors affect 5G KPIs like RSRP, PUCCH SNR, and SINR. It compares RT-generated channels against VNA measurements at 20 indoor lab positions, then feeds both through an OAI testbed with a hardware-in-the-loop emulator to an OAIBOX base station and Quectel UE. This directly addresses the gap the abstract flags between channel-level checks and system-level impact on actual equipment. The results flag antenna near-field transitions as a position-dependent issue and material permittivity mismatches as another, which supplies usable benchmarks for digital-twin planning work. The experimental approach with real hardware and standard 5G stack is a clear step forward from pure simulation studies. It shows straightforward engagement with prior RT validation literature without overclaiming generality. The soft spot is the one the stress-test note flags. The setup does not report a calibration loop that quantifies what the emulator itself adds—such as direct-cable versus emulator comparisons or synthetic-channel injection tests. Without that, position-dependent distortions from quantization, delay truncation, or hardware non-linearities could appear in both arms and get misread as RT modeling error. This is a moderate rather than fatal issue, but it means the quantitative split between near-field effects, material errors, and testbed artifacts is not yet clean. The work is aimed at researchers doing site-specific modeling and digital-twin network planning for 5G and beyond. Readers who need practical numbers on simulation-to-hardware gaps will find the experimental design and KPI results worth their time. It deserves a serious referee because it tackles an explicitly unquantified area with hardware data, even if revisions will be needed to tighten the error isolation. I would send it for peer review after asking for the missing calibration results.

Referee Report

1 major / 2 minor

Summary. The paper compares Sionna RT simulated channels against VNA-measured channels at 20 indoor laboratory positions using an OpenAirInterface 5G NR testbed. Both channel realizations are injected via a hardware-in-the-loop emulator into an OAIBOX MAX base station and Quectel UE; the resulting RSRP, PUCCH SNR, and SINR values are contrasted to quantify deviations attributable to RT modeling inaccuracies, specifically antenna near-field transition effects and material permittivity mismatches.

Significance. If the error attribution holds after isolating testbed contributions, the work supplies a rare quantitative bridge from ray-tracing channel errors to real 5G hardware KPIs. Its direct use of VNA measurements, OAI hardware, and emulator injection (rather than end-to-end simulation) constitutes a concrete strength for validating digital-twin channel models in site-specific deployments.

major comments (1)

[Experimental setup and KPI evaluation (methods and results sections describing the emulator injection)] The central claim that KPI differences (RSRP, PUCCH SNR, SINR) are caused by Sionna RT near-field and material errors requires that the hardware-in-the-loop emulator adds no position-dependent or channel-type-dependent artifacts. The manuscript reports no calibration loop (direct cable versus emulator, or known synthetic-channel injection) that quantifies any such added distortion; without it, emulator effects cannot be cleanly partitioned from RT modeling errors in the 20-position comparison.

minor comments (2)

[Measurement campaign description] The selection criteria and spatial distribution of the 20 receiver positions (near-field versus far-field) should be stated explicitly so readers can judge how representative the near-field error observations are.
[Ray-tracing configuration] The carrier frequency, bandwidth, and exact material permittivity values assigned in Sionna RT should be tabulated for reproducibility.

Simulated Author's Rebuttal

1 responses · 0 unresolved

We thank the referee for the constructive review and for recognizing the value of bridging ray-tracing errors to hardware-measured 5G KPIs. We address the single major comment below and will revise the manuscript to strengthen the experimental validation.

read point-by-point responses

Referee: [Experimental setup and KPI evaluation (methods and results sections describing the emulator injection)] The central claim that KPI differences (RSRP, PUCCH SNR, SINR) are caused by Sionna RT near-field and material errors requires that the hardware-in-the-loop emulator adds no position-dependent or channel-type-dependent artifacts. The manuscript reports no calibration loop (direct cable versus emulator, or known synthetic-channel injection) that quantifies any such added distortion; without it, emulator effects cannot be cleanly partitioned from RT modeling errors in the 20-position comparison.

Authors: We agree that an explicit calibration is required to rigorously isolate emulator contributions from RT modeling errors. The original manuscript did not report a dedicated calibration loop, instead relying on the use of an identical emulation chain for both VNA-measured and Sionna RT channels at each of the 20 positions. While this differential approach minimizes common-mode effects, it does not fully exclude potential channel-type-dependent artifacts arising from differences in delay spread or power delay profile. In the revised manuscript we will add a new methods subsection describing emulator calibration measurements: (i) direct cable bypass versus emulated injection of the same VNA traces, and (ii) injection of known synthetic channels (e.g., single-tap and exponential PDP) at representative positions. These data will quantify any residual position-dependent or channel-dependent distortion and confirm that it remains below the observed KPI deviations. We will also report the emulator's measured linearity and frequency response over the 3.5 GHz band used in the experiments. revision: yes

Circularity Check

0 steps flagged

No circularity: direct empirical comparison of RT simulation vs. measurement

full rationale

The manuscript conducts an experimental benchmark by capturing real channels with a VNA at 20 indoor positions, simulating the same geometry with Sionna RT, and injecting both channel realizations into an OAI hardware-in-the-loop emulator to obtain system-level KPIs (RSRP, PUCCH SNR, SINR). No derivation, prediction, or uniqueness claim is advanced; the central result is simply the observed difference between the two arms. No equations reduce a fitted parameter to a renamed output, no self-citation supplies a load-bearing premise, and no ansatz is smuggled in. The work is therefore self-contained against external benchmarks and receives the default non-circularity finding.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 0 invented entities

The central claim rests on the testbed faithfully reproducing both simulated and measured channels and on the lab conditions being representative enough to generalize the identified error sources.

axioms (1)

domain assumption The channel emulator and hardware setup introduce negligible additional distortions when injecting simulated versus measured channels.
Required to attribute observed KPI differences solely to Sionna RT modeling inaccuracies.

pith-pipeline@v0.9.0 · 5501 in / 1060 out tokens · 59168 ms · 2026-05-12T05:10:59.746257+00:00 · methodology

discussion (0)

Reference graph

Works this paper leans on

16 extracted references · 16 canonical work pages · 1 internal anchor

[1]

Ray tracing for radio propagation modeling: Principles and applications,

Z. Yun and M. F. Iskander, “Ray tracing for radio propagation modeling: Principles and applications,”IEEE Access, vol. 3, pp. 1089–1100, 2015

work page 2015
[2]

Sionna rt: Differentiable ray tracing for radio propagation modeling,

J. Hoydis, F. Aït Aoudia, S. Cammerer, M. Nimier-David, N. Binder, G. Marcus, and A. Keller, “Sionna rt: Differentiable ray tracing for radio propagation modeling,” in2023 IEEE Globecom Workshops (GC Wkshps). IEEE, 2023, pp. 317–321

work page 2023
[3]

Comparison of ray tracing simulations and channel measurements at mmwave bands for indoor scenarios,

A. Karstensen, W. Fan, I. Carton, and G. F. Pedersen, “Comparison of ray tracing simulations and channel measurements at mmwave bands for indoor scenarios,” in2016 10th European Conference on Antennas and Propagation (EuCAP). IEEE, 2016, pp. 1–5

work page 2016
[4]

Validation of ray- tracing simulated channels for massive mimo systems at millimeter- wave bands,

J. Di, Z. Yuan, Y . Lyu, F. Zhang, and W. Fan, “Validation of ray- tracing simulated channels for massive mimo systems at millimeter- wave bands,” in2024 18th European Conference on Antennas and Propagation (EuCAP). IEEE, 2024, pp. 1–5

work page 2024
[5]

Ray tracing simulation and experimental validation for millimeter-wave massive mimo systems,

J. Di, Z. Yuan, Y . Lyv, R. Tan, and W. Fan, “Ray tracing simulation and experimental validation for millimeter-wave massive mimo systems,” International Journal of Microwave and Wireless Technologies, vol. 17, no. 5, pp. 794–803, 2025

work page 2025
[6]

Measurement- based validation of ray-tracing model at sub-thz for isac applications of blockage in industrial scenario,

D. Dupleich, D. Sitdikov, A. Ebert, and M. Boban, “Measurement- based validation of ray-tracing model at sub-thz for isac applications of blockage in industrial scenario,” in2024 4th URSI Atlantic Radio Science Meeting (AT-RASC). IEEE, 2024, pp. 1–4

work page 2024
[7]

Digital twin empowered in-vehicular channel modeling and wireless planning in the terahertz band,

M. Zhu, Y . Lyu, and C. Han, “Digital twin empowered in-vehicular channel modeling and wireless planning in the terahertz band,”arXiv preprint arXiv:2511.07789, 2025

work page arXiv 2025
[8]

On the limitations of ray-tracing for learning-based rf tasks in urban environments,

A. Manukyan, H. Khachatrian, E. Ghukasyan, and T. P. Raptis, “On the limitations of ray-tracing for learning-based rf tasks in urban environments,”arXiv preprint arXiv:2507.19653, 2025

work page arXiv 2025
[9]

Calibrating wireless ray tracing for digital twinning using local phase error esti- mates,

C. Ruah, O. Simeone, J. Hoydis, and B. Al-Hashimi, “Calibrating wireless ray tracing for digital twinning using local phase error esti- mates,”IEEE Transactions on Machine Learning in Communications and Networking, vol. 2, pp. 1193–1215, 2024

work page 2024
[10]

Wireless Digital Twin Calibration: Refining DFT-Domain Channel Information

H. Luo, S. R. Khosravirad, and A. Alkhateeb, “Wireless digital twin calibration: Refining DFT-domain channel information,”arXiv preprint arXiv:2603.16126, 2026

work page internal anchor Pith review Pith/arXiv arXiv 2026
[11]

Learnable wireless digital twins: Reconstructing electromagnetic field with neural representations,

S. Jiang, Q. Qu, X. Pan, A. K. Agrawal, R. Newcombe, and A. Alkha- teeb, “Learnable wireless digital twins: Reconstructing electromagnetic field with neural representations,”IEEE Open Journal of the Communi- cations Society, vol. 6, pp. 1568–1590, 2025

work page 2025
[12]

Vision-language-model-guided differentiable ray tracing for fast and accurate multi-material rf parameter estimation,

Z. Kang, Y . Lim, Z. Gu, S.-W. Ko, T. Q. Quek, and J. Park, “Vision-language-model-guided differentiable ray tracing for fast and accurate multi-material rf parameter estimation,”arXiv preprint arXiv:2601.18242, 2026

work page arXiv 2026
[13]

Open wireless digital twin: End- to-end 5g mobility emulation in o-ran framework,

T. Iye, M. Sakamoto, S. Takaya, E. Sato, Y . Susukida, Y . Na- gaoka, K. Maruta, and J. Nakazato, “Open wireless digital twin: End- to-end 5g mobility emulation in o-ran framework,”arXiv preprint arXiv:2503.12177, 2025

work page arXiv 2025
[14]

Study on channel model for frequencies from 0.5 to 100 ghz,

3GPP, “Study on channel model for frequencies from 0.5 to 100 ghz,” 3rd Generation Partnership Project, Tech. Rep. TR 38.901 V18.0.0, 2024, accessed: May 2025. [Online]. Available: https://www.etsi.org/deliver/etsi_tr/138900_138999/138901/ 18.00.00_60/tr_138901v180000p.pdf

work page 2024
[15]

R&S ZNA Vector Network Analyzer,

Rohde & Schwarz, “R&S ZNA Vector Network Analyzer,” https://www.rohde-schwarz.com/de/produkte/messtechnik/ vektornetzwerkanalysatoren/rs-zna-vektornetzwerkanalysatoren_ 63493-551810.html, accessed: May 2025

work page 2025
[16]

BBHA 9120 L Double Ridged Broadband Horn Antenna,

Schwarzbeck Mess-Elektronik, “BBHA 9120 L Double Ridged Broadband Horn Antenna,” https://www.schwarzbeck.com/antennas/ broadband-horn-antennas/double-ridged-horn-antennas/bbha-9120-l/, accessed: May 2025

work page 2025