Pith. sign in

REVIEW 4 major objections 4 minor 21 references

Watching only a trigger's output hides deadlocks and overstates its time-to-failure by up to 78.7%.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 08:51 UTC pith:7K477MYS

load-bearing objection First RTL fault-injection study of a deployed HEP L1 NN trigger; the MTTF-inflation asymmetry is credible, but the per-stage attribution and hardening-priority wording outrun what the measurements actually support. the 4 major comments →

arxiv 2607.03153 v2 pith:7K477MYS submitted 2026-07-03 hep-ex cs.AR

RTL Fault Injection of a Deployed Graph Neural Network Trigger for Belle II

Georgios Sotiropoulos , Marc Neu , Tanja Harbaum , Torben Ferber , J\"urgen Becker This is my paper
classification hep-ex cs.AR
keywords fault injectionsingle-event upsetRTL simulationgraph neural networkFPGA triggerBelle IIdeadlock detectionmean time to failure
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper reports the first register-transfer-level fault-injection study of a deployed Level-1 hardware neural-network trigger: Belle II's GNN-ETM, a graph-neural-network dataflow accelerator for calorimeter clustering. The authors inject 1,442,840 single-event upsets across all 211,245 RTL signals and classify failures into deadlocks, timeouts, and packet-integrity violations. Their central claim is that the existing verification setup, which watches an internal GNN watchdog and the design output, suffers from a monitoring asymmetry: a stalled pipeline is counted only when the output eventually misbehaves or times out. Re-analyzing the same simulations with four inter-stage liveness monitors on the AXI-Stream interfaces reclassifies most timeouts as stage-level deadlocks and lowers mean-time-to-failure estimates by up to 78.7%. The result identifies the GNN dataflow accelerator and preprocessing stage as the highest-priority hardening targets for a radiation-exposed trigger.

Core claim

The central discovery is a monitoring asymmetry in how failures of a deployed trigger are observed. The baseline infrastructure can see deadlocks only inside the HLS-generated GNN dataflow accelerator, via an embedded watchdog, and packet-integrity violations only at the Belle2Link output; failures originating in the Chisel-based preprocessing, postprocessing, and MAC blocks therefore surface only as late packet violations or simulation timeouts. By re-analyzing the same simulation logs with four liveness monitors placed at the inter-stage AXI-Stream interfaces, the paper reclassifies nearly all of those timeouts as deadlocks localized to a particular stage, and shows that output-only observ

What carries the argument

The key mechanism is the inter-stage liveness monitor: four monitors placed on the AXI-Stream handshake interfaces between the pipeline's stages (preprocessing-to-header, GNN output, postprocessing-to-CPS, postprocessing-to-GDL). In normal operation each stream emits a transaction roughly every 125 ns; a stream is declared deadlocked if it stays silent for more than 2 µs after the design's 1,846 ns end-to-end latency, a threshold chosen to avoid false positives from natural backpressure. The paper re-analyzes the same simulation logs with these monitors and compares that view with the baseline view (GNN watchdog plus output-level protocol checker). Two injection campaigns support the compari

Load-bearing premise

The paper's conclusions rest on two assumptions: that every RTL signal contributes equally to the FPGA's radiation cross-section, and that offline log re-analysis matches what a live monitor with a 2 µs silence window would catch; if either fails, the MTTF gap and the hardening priorities would shift.

What would settle it

A neutron-beam test of the target FPGA running GNN-ETM with the four inter-stage liveness monitors implemented in fabric would settle the core claim: if hardware stall-detection latency differs from the simulated 2 µs silence window, the reported MTTF gap of up to 78.7% would need revision.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • For hangs and deadlocks, output-only observation is systematically optimistic: the same fault-injection runs give mean-time-to-failure estimates up to 78.7% higher than inter-stage liveness monitoring, so output-level reliability numbers should be treated as upper bounds.
  • Most simulation timeouts in the baseline setup are actually deadlocks at a specific pipeline stage; inter-stage liveness monitoring reclassifies them and drops the timeout class below 0.5% for the preprocessing and GNN targets.
  • Under proportional-rate injection, the GNN dataflow accelerator and preprocessing stage have the shortest mean times to failure, making them the highest-priority hardening targets; the fixed-rate campaign reaches the same conclusion even when small stages are injected at up to 35 times higher rates.
  • The liveness monitors are not part of the accelerator pipeline, so they could be implemented as parallel hardware IP without affecting the 1,846 ns latency budget, and a reset on detected deadlock would restore normal operation most of the time.
  • The Belle2Link MAC is not covered by the four monitors because all of them sit upstream of it; a fifth monitor at the Belle2Link output would close that gap, though with longer detection latency since Belle2Link packets arrive less often.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If per-signal cross-sections were measured from synthesis or beam tests instead of assumed uniform, the proportional-rate MTTF ordering could shift; the hardening priorities are only as robust as that assumption.
  • The asserted equivalence between offline log re-analysis and a live hardware monitor is testable: implementing one inter-stage liveness monitor in the target FPGA fabric and measuring real stall-detection latency would validate the 2 µs silence-window assumption, and a longer latency would compress the 78.7% gap.
  • The monitoring approach should transfer to any pipelined FPGA trigger with AXI-Stream or similar interfaces; inter-stage liveness monitoring is a low-cost observability upgrade that separates deadlocks from timeouts and could become a standard verification step before radiation testing.
  • The 2 µs silence threshold is a free parameter: the paper notes a tighter threshold would lower MTTF further at the cost of false positives from backpressure, so reporting MTTF as a function of threshold would give designers a fuller picture of the trade-off.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper presents an RTL fault-injection study of GNN-ETM, the Belle II L1 trigger module for the electromagnetic calorimeter, implemented on an FPGA. Using two campaigns (proportional-rate and fixed-rate injection) and approximately 1.44 million single-event upsets across 211,245 signals, the authors characterize deadlocks, timeouts, and packet-integrity violations. They compare the existing baseline detection setup (an HLS watchdog plus a Belle2Link output checker) with a proposed inter-stage AXI-Stream liveness monitoring approach, reporting that MTTF estimates differ by up to 78.7% and using per-stage data to identify the GNN Dataflow Accelerator and Preprocessing Stage as the highest-priority hardening targets.

Significance. If the central asymmetry claim holds, this is a valuable contribution: it is, to my knowledge, the first RTL fault-injection study of a deployed L1 hardware neural-network trigger, and the scale of the campaign is substantial. The two-campaign design is thoughtful, and re-analyzing the same simulation logs under two detection views is a clean way to isolate detection latency effects. The observation that output-only DUE detection systematically inflates MTTF is well motivated and internally consistent. However, the per-stage hardening-priority conclusion is undermined by the backpressure misattribution problem identified in the text itself, and the priority ordering also depends on the uniform-cross-section assumption. The paper's primary methodological point about monitoring placement is defensible, but the stronger per-stage claims need additional work.

major comments (4)
  1. [§V-B, Table IV] The paper states that 4.13% (proportional-rate) and 33.14% (fixed-rate) of Postprocessing-injected runs are classified as GNN DF output deadlocks because a Postprocessing stall back-pressures the upstream interface. This is exactly the problem for the hardening-priority claim: every monitored interface that is upstream of a fault origin can be declared deadlocked when the fault actually injected downstream. Consequently, the per-stage MTTF columns in Table IV conflate the observation point with the fault origin, and the conclusion that the GNN Dataflow Accelerator and Preprocessing Stage are the highest-priority targets is not supported by these measurements. The paper should either report a joint distribution of injected target versus observed deadlock interface, or explicitly limit the claim to interface-level observability rather than component-level hardening priority.
  2. [§IV-B, Eq. (2)] The proportional-rate campaign assumes that sensitive area is uniformly distributed across signals, i.e., every signal has equal per-signal cross-section. The paper acknowledges this is necessary for a platform-agnostic RTL study, but the resulting MTTF ordering and the hardening priorities derived from it are sensitive to this assumption. If, for example, memory elements or particular control signals have materially higher per-signal cross-sections after FPGA mapping, the component-level rankings in Table IV could shift. A sensitivity analysis varying per-signal weights, or at least a qualitative statement of how robust the ordering is, is needed before the proportional-rate priority conclusion can be accepted.
  3. [§IV-C] The asserted equivalence between offline log re-analysis and an implemented live monitor ('An equivalent live monitor would catch the same events with the same detection latency') is an assumption, not a demonstrated fact. A hardware monitor would itself consume resources, may be affected by faults, and could interact with the pipeline's timing or backpressure behavior. The paper proposes these monitors as a practical addition, but it does not validate that the 2-µs silence threshold or the event classification observed in simulation would reproduce in hardware. At minimum, this should be framed as a design premise and an implementation/validation plan should be indicated.
  4. [§V, Table IV] The headline quantitative result is the MTTF deviation of up to 78.7%, but the reported 95% margin of error applies to percentage cells only. MTTF and MTTF-deviation values are presented as point estimates without confidence intervals. Since the magnitude of the deviation is a central claim, the authors should report uncertainty on the MTTF estimates (e.g., bootstrap or binomial-derived intervals), even if the qualitative direction of the asymmetry is clear.
minor comments (4)
  1. [Table IV caption] The MTTF deviation formula is not defined. Please state explicitly that it is (MTTF_baseline - MTTF_interstage)/MTTF_baseline × 100%, or give the definition in the text.
  2. [Table III/IV] The Full Design row is identical in the two campaigns and the total faults are counted once, but Table IV repeats the row and the caption does not clarify this relationship. A note would prevent confusion.
  3. [§IV-A] The phrase 'a single bit of the targeted signal is flipped and remains flipped until the next normal write to that signal' is clear, but it may help to state explicitly whether the injected bit is in a flip-flop or could also be a wire/combinational node, given that the RTL signal set is heterogeneous.
  4. [§V-A] The failure classes under inter-stage monitoring are described as mutually exclusive by priority, but the baseline classes (HLS watchdog, packet integrity, timeout) also need explicit priority ordering. The current text says only that inter-stage signatures take priority; the reader should know how a run with both a watchdog and a packet-integrity event is classified.

Circularity Check

0 steps flagged

No significant circularity: the MTTF asymmetry is an empirical comparison of two detection views, not a fitted prediction.

full rationale

The paper's central result—that output-only observation inflates MTTF estimates for DUEs and that inter-stage liveness monitoring reduces them by up to 78.7%—is an empirical comparison of two detection views applied to the same simulation logs, not a prediction derived by fitting a parameter. The direction of the effect follows from the definition of detection latency (a stall at t_d is registered at t_d+Δ), but the paper does not use this definition to construct the numerical MTTF values; those values are measured from the reclassified runs in Table IV. The injection-rate equations (1)–(2) set the fault exposure and incorporate a clearly stated uniform cross-section assumption; they do not determine the failure outcomes or MTTF ratios. The per-stage hardening priorities are an interpretation of the measured per-stage MTTFs; the paper itself acknowledges in §V-B that backpressure from Postprocessing can cause upstream interfaces to be tagged as deadlocked (e.g., 33.14% of Postprocessing fixed-rate runs classified as GNN DF output deadlocks), which is a validity threat to the attribution, not a circular reduction. No load-bearing self-citation appears: reference [3] supplies the system architecture, and [9] is an external comparison point; neither is invoked to prove the MTTF asymmetry. The 'sooner is more accurate' statement is a property of the measurement definition, but the 38.5–78.7% deviations are data, not consequences of an assumed model. Accordingly, no circular step can be exhibited by quote and reduction; the score is 0.

Axiom & Free-Parameter Ledger

3 free parameters · 5 axioms · 0 invented entities

The central comparison is a measurement, so the ledger is mostly about simulation choices rather than fitted physics constants. The injection rate and deadlock threshold set the scale of all reported MTTF values; the uniform per-signal cross-section assumption converts between campaigns and underpins the hardening priorities; the Poisson single-bit SEU model and the asserted live-monitor equivalence are domain assumptions. None of these are fitted to the 78.7% result, but they do control its magnitude.

free parameters (3)
  • Full-design fault injection rate λ_full = 1 fault / 1846 ns
    Chosen to make exhaustive simulation feasible (Section IV-A); anchors all MTTF magnitudes and is orders of magnitude above the real radiation rate, so absolute MTTFs are simulation metrics.
  • Inter-stage deadlock silence threshold = 2 µs after 1,846 ns end-to-end latency
    Chosen to avoid false positives from natural backpressure (Section IV-C); affects which runs are classified as deadlocked and hence MTTF values and the reported deviation.
  • Uniform per-signal cross-section weight
    Eq. (2) assumes each of the 211,245 signals contributes an equal share of the total cross-section; unmeasurable at RTL and explicitly acknowledged as necessary.
axioms (5)
  • domain assumption SEU arrivals follow a Poisson process
    Used to schedule injections (one fault per 1846 ns nominal) and cited to [14,15]; a standard model for SEU ground testing and FPGA operation, but an idealization for real detector environments.
  • ad hoc to paper Uniform sensitive area per signal
    Eq. (2) converts full-design injection rate to per-component rates by assuming every signal contributes equal cross-section; explicitly acknowledged as necessary because per-signal cross-sections require beam measurements.
  • domain assumption Single-bit SEU model, no multi-bit upsets
    Section IV-A injects one flipped bit per fault; multi-bit upsets are noted as relevant at advanced nodes [19] but not modeled.
  • ad hoc to paper Offline re-analysis equals an implemented live monitor
    Section IV-C asserts that an equivalent live monitor would observe the same AXI-Stream handshake events with the same detection latency; no hardware implementation or timing verification is provided.
  • domain assumption Random TC input stimulus is representative
    Inputs are randomly generated TC data with fresh seeds, since targeted failures are control-signal driven; no coverage study demonstrates input-independence.

pith-pipeline@v1.3.0-alltime-deepseek · 9265 in / 13543 out tokens · 137963 ms · 2026-08-02T08:51:32.802299+00:00 · methodology

0 comments
read the original abstract

As particle physics detectors grow in scale, High Energy Physics experiments must process ever-increasing data volumes. Level-1 trigger systems, implemented on Field-Programmable Gate Arrays and increasingly using neural-network algorithms, filter this data in real time. However, their proximity to the interaction point exposes them to radiation, which can corrupt outputs, stall processing pipelines, or damage hardware, with significant financial and scientific consequences. In this work, we present the first Register Transfer Level fault-injection study of a deployed Level-1 hardware neural-network trigger, GNN-ETM in the Belle II trigger system. We target three failure modes most consequential to a real-time trigger pipeline: deadlocks, timeouts, and packet-integrity violations. Through two complementary campaigns, we inject 1 442 840 Single-Event Upsets across 211 245 signals. We find a monitoring asymmetry in the existing verification infrastructure and propose inter-stage liveness monitoring as a more accurate alternative to output-only observation, showing that Mean Time To Failure estimates from the two approaches differ by up to 78.7%. The resulting per-stage data identifies the highest-priority hardening targets.

Figures

Figures reproduced from arXiv: 2607.03153 by Georgios Sotiropoulos, J\"urgen Becker, Marc Neu, Tanja Harbaum, Torben Ferber.

Figure 1
Figure 1. Figure 1: Overview of the RTL fault injection testbench used to evaluate the [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Architecture overview of GNN-ETM, comprising the Preprocessing [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗

Review history (2 revisions) →

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

21 extracted references · 3 linked inside Pith

  1. [1]

    The neural network first-level hardware track trigger of the Belle II experiment,

    S. B ¨ahret al., “The neural network first-level hardware track trigger of the Belle II experiment,”Nucl. Instrum. Meth. A, vol. 1073, 2025

    2025

  2. [2]

    Development of deep neural network first-level hardware track trigger for the Belle II experiment,

    Y .-X. Liuet al., “Development of deep neural network first-level hardware track trigger for the Belle II experiment,”Nucl. Instrum. Meth. A, vol. 1084, 4 2026

    2026

  3. [3]

    Real-time graph neural networks on fpgas for the belle ii electromagnetic calorimeter,

    I. Haideet al., “Real-time graph neural networks on fpgas for the belle ii electromagnetic calorimeter,” 2026. [Online]. Available: https://arxiv.org/abs/2602.15118

    Pith/arXiv arXiv 2026

  4. [4]

    Testing a Neural Network for Anomaly Detection in the CMS GlobalTrigger Test Crate during Run 3,

    N. Zipperet al., “Testing a Neural Network for Anomaly Detection in the CMS GlobalTrigger Test Crate during Run 3,”JINST, vol. 19, no. 03, 3 2024

    2024

  5. [5]

    Real-time anomaly detection at the l1 trigger of cms experiment,

    A. Gandrakota, “Real-time anomaly detection at the l1 trigger of cms experiment,” no. arXiv:2411.19506, Nov. 2024, arXiv:2411.19506 [hep-ex]. [Online]. Available: http://arxiv.org/abs/2411.19506

    Pith/arXiv arXiv 2024

  6. [6]

    High-reliability fpga-based systems: Space, high-energy physics, and beyond,

    M. Wirthlin, “High-reliability fpga-based systems: Space, high-energy physics, and beyond,”Proceedings of the IEEE, vol. 103, no. 3, p. 379–389, Mar. 2015

    2015

  7. [7]

    Current and future challenges in radiation effects on cmos electronics,

    P. E. Dodd, M. R. Shaneyfelt, J. R. Schwank, and J. A. Felix, “Current and future challenges in radiation effects on cmos electronics,”IEEE Transactions on Nuclear Science, vol. 57, no. 4, p. 1747–1763, Aug. 2010

    2010

  8. [8]

    A systematic literature review on hardware reliability assessment methods for deep neural networks,

    M. H. Ahmadilivani, M. Taheri, J. Raik, M. Daneshtalab, and M. Jeni- hhin, “A systematic literature review on hardware reliability assessment methods for deep neural networks,”ACM Comput. Surv., vol. 56, no. 6, pp. 141:1–141:39, Jan. 2024

    2024

  9. [9]

    Reliability evaluation and analysis of fpga-based neural network acceleration system,

    D. Xu, Z. Zhu, C. Liu, Y . Wang, S. Zhao, L. Zhang, H. Liang, H. Li, and K.-T. Cheng, “Reliability evaluation and analysis of fpga-based neural network acceleration system,”IEEE Transactions on Very Large Scale Integration (VLSI) Systems, vol. 29, no. 3, p. 472–484, Mar. 2021

    2021

  10. [10]

    Soft error susceptibility analysis methodology of hls designs in sram-based fpgas,

    J. Tonfat, L. Tambara, A. Santos, and F. L. Kastensmidt, “Soft error susceptibility analysis methodology of hls designs in sram-based fpgas,” Microprocessors and Microsystems, vol. 51, p. 209–219, 2017

    2017

  11. [11]

    Single-event upset analysis of a systolic array based deep neural net- work accelerator,

    N. Jonckers, T. Vinck, G. Dekkers, P. Karsmakers, and J. Prinzie, “Single-event upset analysis of a systolic array based deep neural net- work accelerator,” no. arXiv:2405.15381, May 2024, arXiv:2405.15381 [cs]. [Online]. Available: http://arxiv.org/abs/2405.15381

    Pith/arXiv arXiv 2024

  12. [12]

    Analysis of single event induced bit faults in a deep neural network accelerator pipeline,

    N. Jonckers, T. Vinck, P. Karsmakers, and J. Prinzie, “Analysis of single event induced bit faults in a deep neural network accelerator pipeline,” Journal of Instrumentation, vol. 21, no. 02, p. C02021, Feb. 2026

    2026

  13. [13]

    Artificial neural networks for space and safety-critical appli- cations: Reliability issues and potential solutions,

    P. Rech, “Artificial neural networks for space and safety-critical appli- cations: Reliability issues and potential solutions,”IEEE Transactions on Nuclear Science, vol. 71, no. 4, p. 377–404, Apr. 2024

    2024

  14. [14]

    Single-event-upset-like fault injection: a comprehensive framework,

    F. Faure, R. Velazco, and P. Peronnard, “Single-event-upset-like fault injection: a comprehensive framework,”IEEE Transactions on Nuclear Science, vol. 52, no. 6, p. 2205–2209, Dec. 2005

    2005

  15. [15]

    Fpga qualification and failure rate estimation methodology for lhc environments using benchmarks test circuits,

    A. Scialdone, R. Ferraro, R. G. Al ´ıa, L. Sterpone, S. Danzeca, and A. Masi, “Fpga qualification and failure rate estimation methodology for lhc environments using benchmarks test circuits,”IEEE Transactions on Nuclear Science, vol. 69, no. 7, p. 1633–1641, 2022

    2022

  16. [16]

    ModelSim HDL simulator,

    Siemens EDA, “ModelSim HDL simulator,” https://eda.sw.siemens.com/ en-US/ic/modelsim/, 2025, version 2023.4, accessed 2025-05-13

    2025

  17. [17]

    cocotb: Python-based chip (RTL) verification,

    S. Hodgsonet al., “cocotb: Python-based chip (RTL) verification,” https: //github.com/cocotb/cocotb, 2025, accessed: 2025-05-13

    2025

  18. [18]

    cocotb fault injection,

    S. Biereigel, “cocotb fault injection,” GitLab, CERN. https://gitlab.cern. ch/tmrg/cocotb fault injection, Feb. 2024, commitaa8d3760. Ac- cessed: Apr. 18, 2026

    2024

  19. [19]

    Radiation- induced multi-bit upsets in sram-based fpgas,

    H. Quinn, P. Graham, J. Krone, M. Caffrey, and S. Rezgui, “Radiation- induced multi-bit upsets in sram-based fpgas,”IEEE Transactions on Nuclear Science, vol. 52, no. 6, p. 2455–2461, Dec. 2005

    2005

  20. [20]

    Statistical fault injection: Quantified error and confidence,

    R. Leveugle, A. Calvez, P. Maistri, and P. Vanhauwaert, “Statistical fault injection: Quantified error and confidence,” inAutomation & Test in Europe Conference & Exhibition 2009 Design, 2009, p. 502–506, iSSN: 1558-1101. [Online]. Available: https://ieeexplore.ieee. org/document/5090716

    arXiv 2009

  21. [21]

    Trikarenos: Design and experimental characterization of a fault-tolerant 28nm risc-v-based soc,

    M. Rogenmoser, P. Wiese, B. E. Forlin, F. K. G ¨urkaynak, P. Rech, A. Menicucci, M. Ottavi, and L. Benini, “Trikarenos: Design and experimental characterization of a fault-tolerant 28nm risc-v-based soc,” IEEE Transactions on Nuclear Science, 2025

    2025