REVIEW 4 major objections 4 minor 29 references
Evaluating Different Fault Injection Abstractions on the Assessment of DNN SW Hardening Strategies
T0 review · 4 major / 4 minor · reviewed 2026-08-11 · deepseek-v4-flash
Pith's one-line read Fault-injection abstraction level changes DNN hardening rankings.
desk verdict Useful head-to-head comparison of APP vs ISA fault injection for ranking DNN hardening techniques, but the strong claim that APP-level FI is misleading and ISA-level FI is accurate goes beyond what the confounded setup can support. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The key mechanism is a BER bridge between the two abstraction levels. After each instruction-level stuck-at injection, the authors count how many times the fault is excited — the target bit assumes the opposite polarity during kernel execution — and divide by the total uses of that register to obtain a Bit Error Rate. They then use the observed BER range to configure application-level multiple bit-flips at the same error rate, holding the error-activation rate roughly constant so differences can be attributed to the abstraction level. The evaluation is organized around a four-way fault classification (Masked, Safe-SDC, Critical-SDC, DUE) and the Relative Accuracy Degradation metric.
What would settle it
Run the same application-level multiple-bit-flip campaigns on the same GPU that was used for instruction-level injection, with bit-flip locations and timing matched to the instruction sites where stuck-at faults were injected; if the hardening ranking then matches the instruction-level ranking, the original difference came from platform or fault-model mismatches rather than from abstraction level, whereas if the ranking still flips, the abstraction-level interpretation is supported.
Extended reading notes
Core claim
The central discovery is that the abstraction level used for fault injection is not a neutral experimental detail: it changes the measured reliability of the same DNN and the same hardening method. Application-level injection underestimates Critical SDCs (no more than 10% of faults, versus up to 31% at instruction level) and misses detectable unrecoverable errors entirely, although register faults can produce DUEs in up to 47.5% of cases. As a direct consequence, the ranking of hardening techniques flips: in the paper's example, application-level injection picks Adaptive Clipper as the best hardening for MobileNetV2 while instruction-level injection picks Swap ReLU6. The authors conclude that instruction-level results are closer to what happens on real hardware and therefore APP-based conclusions about hardening effectiveness are misleading.
Load-bearing premise
The load-bearing assumption is that matching the Bit Error Rate makes the application-level and instruction-level campaigns equivalent; in reality the two campaigns also differ in what they corrupt, in the fault model (stuck-at vs bit-flip), and in the GPU used, so those differences could cause the ranking change instead.
Editorial extensions
If this is right
- If the central claim holds, reliability rankings produced by application-level fault injection alone should not be used to choose a hardening technique for deployment.
- Hardening methods that look neutral or harmful at instruction level, such as Adaptive Clipper on ResNet-18, would be missed or mis-ranked by APP-only studies.
- Because instruction-level injection reveals DUEs that application-level injection cannot produce, safety-critical systems need hardware-aware fault injection to budget for crashes and hangs, not just accuracy loss.
- The same hardening technique can be the best for one DNN and ineffective or harmful for another, and this interaction only becomes visible at instruction level.
Reading between the lines
- A testable extension is to repeat the comparison with a transient (single-event upset) fault model; the ranking flip may be smaller or larger, which would delimit how much of the effect is specific to permanent faults.
- The paper's BER-matching step could be strengthened by matching fault excitation timing and data-dependent activation between levels, not just average rates; without that, part of the observed difference may be an artifact of the calibration.
- If the effect generalizes, published comparisons of DNN hardening that rely solely on application-level fault injection are not portable across GPUs, and reliability evaluation standards should mandate the injection abstraction and hardware target.
- The implication for practice is that cheap application-level fault injection may still be useful for screening many candidates, with instruction-level injection reserved for the top-ranked few — a two-stage evaluation worth testing.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This paper compares two fault injection (FI) abstraction levels—application-level (APP) and instruction-level (ISA)—for evaluating software-based hardening strategies on DNNs. The authors adapt PyTorchFI for APP injections (bit-flips on weights and neuron outputs) and an NVBitFI-based tool for ISA injections (permanent stuck-at faults on register files and functional units) on GPUs. They apply both to LeNet5, MobileNetV2, and ResNet18, with and without three hardening techniques (Ranger, Swap ReLU6, Adaptive Clipper), and compare fault-class distributions and accuracy. The key claim is that APP-level FI overestimates the beneficial impact of hardening techniques and yields a different—and, in the authors' view, misleading—ranking of those techniques compared with ISA-level FI.
Significance. If the central claim were rigorously supported, the paper would be a useful methodological warning for the DNN reliability community, where APP-level FI is widely used to evaluate hardening techniques. The raw comparative data, including exhaustive ISA fault counts and the fault-class distributions in Fig. 2, are valuable and appear internally consistent. Strong points are the concrete fault counts (10,496 register and 1,536 FU injections per DNN) and the attempt to equalize BER across abstraction levels. However, as detailed below, the comparison varies multiple factors simultaneously, and the claim that ISA results are 'closer to the real ones' lacks any ground-truth reference, so the paper's strongest conclusions are not currently supported.
major comments (4)
- [Section III.C, Table I] The claimed 'fair comparison' between ISA and APP is undermined by confounds. ISA injection (Section III.A) places permanent stuck-at faults on register-file destinations and FU inputs/outputs on an RTX 3060 Ti, while APP injection (Section III.B) applies single or multiple random bit-flips to weight tensors and neuron output feature maps on V100 GPUs. A stuck-at fault produces correlated, repeated errors whenever the target register or FU is exercised, whereas an APP bit-flip is an independent one-shot perturbation. The BER equalization described in Section III.C matches only the average activation rate; it does not control for error polarity, spatial or temporal error structure, affected data type (addresses and control vs. weights and activations), or GPU microarchitecture. Therefore the rank reversal in Fig. 2 and Table I cannot be attributed uniquely to the abstraction level, which is the paper's central conclusion.
- [Section IV.A and Section V] The statement that ISA FI results are 'closer to the real ones' is an unsupported assertion. The paper provides no ground-truth measurement—such as hardware-level FI, radiation testing, or RTL/microarchitectural simulation—against which either abstraction can be validated. Disagreement between two proxies does not privilege one as more realistic. The authors should either remove this claim or substantiate it with an external reference, since it is load-bearing for the conclusion that APP FI is 'misleading.'
- [Section V vs. Section IV.A] The key example illustrating the ranking reversal is internally contradictory. Section IV.A states that for MobileNetV2, under Regs and FUs corruptions, 'Adaptive Clipper outperforms the other hardening strategies,' while 'APP FI highlights that Swap ReLU6 is the best-performing technique.' The Conclusion (Section V) inverts this: 'APP FI identifies Adaptive Clipper as the best solution ... while for ISA FI the best solution is Swap ReLU6.' Table I does not straightforwardly support either statement: under ISA FUs for MobileNetV2, Ranger attains the highest accuracy (68.47%), whereas under APP Weights at BER=10^-6, Swap ReLU6 is best (84.12%). Because the ranking reversal is the main result, this contradiction must be resolved.
- [Section IV.A (Fig. 2)] The cross-abstraction fault-distribution comparison is not controlled. APP SBFs corrupt only weight tensors, while ISA stuck-at faults target registers and FUs. The ISA campaign exhaustively injects faults on one SM and a single thread, whereas the APP campaign uses statistical sampling at 95% confidence over the whole execution. The large differences in Critical-SDC and DUE rates (e.g., DUEs up to 47.5% in Regs) may therefore reflect the choice of fault location and campaign scope rather than the abstraction level. The conclusion that 'APP FIs report a limited percentage of Critical SDCs' and thus overestimate hardening effectiveness depends on comparing experiments that differ in coverage and target, not only in abstraction.
minor comments (4)
- [Section III.C] The fault-class definitions are unclear: Masked is defined as 'if they match' and Safe-SDC as 'when the fault does not change the prediction,' which appear to describe the same outcome. Clarify the distinction, for example whether Safe-SDC refers to a change in confidence while the top-1 label remains correct.
- [Section V and Section III.A] There are minor typos: 'understimate' should be 'underestimate' (Section V), 'crush' should be 'crash' (Section III.A), and 'As illustrated in 1' should read 'in Figure 1' (Section III.A).
- [Section I] The phrase 'for the first time' is used without a systematic comparison to prior work beyond the cited references; consider softening to 'to our knowledge, no prior study has directly compared...'.
- [Table I] The color scale used to rank results may be ambiguous when printed in grayscale; adding numerical ranks or arrows would improve readability.
Circularity Check
No significant circularity: the ISA-vs-APP comparison is an empirical campaign with a calibrated input, not a derivation that assumes its conclusion.
full rationale
The paper's central claim is empirical: two fault-injection abstractions produce different fault distributions, accuracies, and hardening rankings. The BER calibration in Section III.C ('we have computed the error induced by single stuck-at faults at ISA and we induced the same error at APP to make a fair comparison') is a measurement-to-input mapping: ISA excitation counts are used to choose APP bit-flip rates, but this does not force any particular ranking outcome. The observed rank reversals in Fig. 2 and Table I are measured results, not identities. The statement that ISA results are 'closer to the real ones' (Section IV.A) is an unsupported assertion in the absence of hardware ground truth, but unsupported assertion is a validity concern rather than circularity. Several cited tools and studies come from the same research group ([12], [22], [23], [24], [29]), but the main injection platform NVBitFI [20] is external, and the self-citations provide methodology and baseline techniques rather than a uniqueness theorem or a result equivalent to the conclusion. No equation or definition in the paper reduces the conclusion to its own inputs, so no circular step can be exhibited.
Assumptions & free parameters
free parameters (2)
- APP BER range and bins =
BER range [0, 1e-5]; bins 10^-6 and 10^-5
- ISA injection location =
First SM, one Thread ID
assumptions (3)
- domain assumption Permanent stuck-at faults injected at ISA in registers/FUs of one thread on one SM adequately represent the effect of real permanent faults on the GPU during DNN inference.
- ad hoc to paper A BER derived from the number of fault excitations at ISA is equivalent, for fairness purposes, to injecting the same BER as multiple bit flips at APP.
- domain assumption Differences in outcome between the two campaigns are attributable to FI abstraction level rather than to the different injection sites, fault models, timing, or GPU platforms.
Cite this review
Pith. "Pith review of Evaluating Different Fault Injection Abstractions on the Assessment of DNN SW Hardening Strategies." pith.science (2026). https://pith.science/paper/SGR4Q6UD
@misc{pith2026241208466,
author = {Pith},
title = {Pith review of: Evaluating Different Fault Injection Abstractions on the Assessment of DNN SW Hardening Strategies},
year = {2026},
howpublished = {\url{https://pith.science/paper/SGR4Q6UD}},
note = {Machine review of arXiv:2412.08466}
}
read the original abstract
The reliability of Neural Networks has gained significant attention, prompting efforts to develop SW-based hardening techniques for safety-critical scenarios. However, evaluating hardening techniques using application-level fault injection (FI) strategies, which are commonly hardware-agnostic, may yield misleading results. This study for the first time compares two FI approaches (at the application level (APP) and instruction level (ISA)) to evaluate deep neural network SW hardening strategies. Results show that injecting permanent faults at ISA (a more detailed abstraction level than APP) changes completely the ranking of SW hardening techniques, in terms of both reliability and accuracy. These results highlight the relevance of using an adequate analysis abstraction for evaluating such techniques.
Figures
Reference graph
Works this paper leans on
-
[1]
paella: Edge ai-based real-time malware detection in data centers,
A. Libri et al., “paella: Edge ai-based real-time malware detection in data centers,” IEEE Internet Things J. , vol. 7, no. 10, 2020
work page 2020
-
[2]
Nvidia Corporation, “Nvidia drive for automotive,” https://developer. nvidia.com/drive, 2024, accessed: 2024-08-09
work page 2024
-
[3]
Advance Micro Devices, “Amd instinct,” https://www.amd.com/en/ products/accelerators/instinct/mi300.html, 2024, accessed: 2024-08-09
work page 2024
-
[4]
Silent data corruptions at scale,
H. D. Dixit et al. , “Silent data corruptions at scale,” arXiv preprint arXiv:2102.11245, 2021
arXiv 2021
-
[5]
P. H. Hochschild et al., “Cores that don’t count,” in Proc. of the Workshop on Hot Topics in Operating Systems , 2021
work page 2021
-
[6]
Silent data errors: Sources, detection, and modeling,
A. Singh et al., “Silent data errors: Sources, detection, and modeling,” in IEEE 41st VLSI Test Symp. (VTS’23) , 2023
work page 2023
-
[7]
Transient-fault-aware design and training to enhance dnns reliability with zero-overhead,
N. Cavagnero et al., “Transient-fault-aware design and training to enhance dnns reliability with zero-overhead,” in IEEE 28th Int. Symp. on On-Line Testing and Robust System Design (IOLTS) , 2022
work page 2022
-
[8]
Practical hardening of crash-tolerant systems,
M. Correia et al. , “Practical hardening of crash-tolerant systems,” in USENIX Ann. Tech. Conf. (USENIX ATC’12) , 2012
work page 2012
Show all 29 references
-
[9]
W. W. Peterson et al., Error-correcting codes. MIT press, 1972
1972
-
[10]
Algorithm based fault tolerance: Review and experimental study,
V . Stefanidis et al. , “Algorithm based fault tolerance: Review and experimental study,” in International Conference of Numerical Analysis and Applied Mathematics . Citeseer, 2004
2004
-
[11]
A low-cost fault corrector for deep neural networks through range restriction,
Z. Chen et al. , “A low-cost fault corrector for deep neural networks through range restriction,” in 51st Ann. IEEE/IFIP Int. Conf. on Dependable Systems and Networks (DSN) , 2021
2021
-
[12]
Enhancing the reliability of split computing deep neural networks,
G. Esposito et al., “Enhancing the reliability of split computing deep neural networks,” in 2024 IEEE 30th International Symposium on On- Line Testing and Robust System Design (IOLTS) . IEEE, 2024
2024
-
[13]
Boosting bit-error resilience of dnn accelerators through median feature selection,
E. Ozen et al., “Boosting bit-error resilience of dnn accelerators through median feature selection,” IEEE Trans. Comput.-Aided Des. Integr. Circuits Syst., vol. 39, no. 11, 2020
2020
-
[14]
Salvagednn: salvaging deep neural network accelerators with permanent faults through saliency-driven fault-aware mapping,
M. Abdullah Hanif et al., “Salvagednn: salvaging deep neural network accelerators with permanent faults through saliency-driven fault-aware mapping,” Philos. Trans. R. Soc. A , vol. 378, no. 2164, 2020
2020
-
[15]
Reliability and security of ai hardware,
D. Gnad et al., “Reliability and security of ai hardware,” in 2024 IEEE European Test Symp. (ETS) , 2024
2024
-
[16]
Pros and cons of fault injection approaches for the reliability assessment of deep neural networks,
A. Ruospo et al., “Pros and cons of fault injection approaches for the reliability assessment of deep neural networks,” in IEEE 22nd Latin American Test Symp. (LATS) , 2021
2021
-
[17]
Pytorchfi: A runtime perturbation tool for dnns,
A. Mahmoud et al., “Pytorchfi: A runtime perturbation tool for dnns,” in 50th Ann. IEEE/IFIP Int. Conf. on Dependable Systems and Networks Workshops (DSN-W), 2020
2020
-
[18]
Evaluating convolutional neural networks reliability depending on their data representation,
A. Ruospo et al., “Evaluating convolutional neural networks reliability depending on their data representation,” in 23rd Euromicro Conf. on Digital System Design (DSD) , 2020
2020
-
[19]
The impact of faults on dnns: A case study,
E. Malekzadeh et al. , “The impact of faults on dnns: A case study,” in IEEE Int. Symp. on Defect and Fault Tolerance in VLSI and Nanotechnology Systems (DFT) , 2021
2021
-
[20]
Nvbitfi: Dynamic fault injection for gpus,
T. Tsai et al., “Nvbitfi: Dynamic fault injection for gpus,” in 51st Ann. IEEE/IFIP Int. Conf. on Dependable Systems and Networks (DSN) , 2021
2021
-
[21]
Sassifi: An architecture-level fault injection tool for gpu application resilience evaluation,
S. K. S. Hari et al., “Sassifi: An architecture-level fault injection tool for gpu application resilience evaluation,” in Int. Symp. on Performance Analysis of Systems and Software (ISPASS) , 2017
2017
-
[22]
Evaluating the impact of permanent faults in a gpu running a deep neural network,
J.-D. Guerrero-Balaguera et al., “Evaluating the impact of permanent faults in a gpu running a deep neural network,” in IEEE Int. Test Conf. in Asia (ITC-Asia) , 2022
2022
-
[23]
Reliability assessment of neural networks in gpus: A framework for permanent faults injections,
J. D. Guerrero Balaguera et al., “Reliability assessment of neural networks in gpus: A framework for permanent faults injections,” in IEEE 31st Int. Symp. on Industrial Electronics , 2022
2022
-
[24]
A multi-level approach to evaluate the impact of gpu permanent faults on cnn’s reliability,
J. E. R. Condia et al., “A multi-level approach to evaluate the impact of gpu permanent faults on cnn’s reliability,” in IEEE International Test Conference (ITC), 2022
2022
-
[25]
Demystifying the system vulnerability stack: Transient fault effects across the layers,
G. Papadimitriou et al., “Demystifying the system vulnerability stack: Transient fault effects across the layers,” in 2021 ACM/IEEE 48th Annual International Symposium on Computer Architecture (ISCA) . IEEE, 2021
2021
-
[26]
Gufi: A framework for gpus reliability assessment,
S. Tselonis et al., “Gufi: A framework for gpus reliability assessment,” in 2016 IEEE International Symposium on Performance Analysis of Systems and Software (ISPASS). IEEE, 2016
2016
-
[27]
An effective method to identify microarchitectural vulnerabilities in gpus,
J. E. R. Condia et al., “An effective method to identify microarchitectural vulnerabilities in gpus,” IEEE Transactions on Device and Materials Reliability, vol. 22, no. 2, pp. 129–141, 2022
2022
-
[28]
Assessing convolutional neural networks reliability through statistical fault injections,
A. Ruospo et al., “Assessing convolutional neural networks reliability through statistical fault injections,” in Design, Automation & Test in Europe Conf. & Exhibit. (DATE) , 2023
2023
-
[29]
Evaluating the reliability of supervised compression for split computing,
J.-D. Guerrero-Balaguera et al., “Evaluating the reliability of supervised compression for split computing,” in IEEE 42nd VLSI Test Symp. (VTS) , 2024
2024
Reviewed August 11, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.