Pith. sign in

REVIEW 4 major objections 5 minor 1 cited by

Statistical Runtime Verification for LLMs via Robustness Estimation

T0 review · 4 major / 5 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper claims that adapting RoMA to black-box LLM monitoring estimates adversarial robustness within about 1% of formal verification while cutting runtime from hours to minutes.

desk verdict A real but modest case study: the first RoMA-style statistical robustness monitor for black-box LLMs, with an honest external check against an exact verifier, but the headline 'statistically validated bounds' claim outruns the reported experiments. read the letter →

arxiv 2504.17723 v2 pith:M5Z7SRPJ submitted 2025-04-24 cs.LG

Natan Levy , Adiel Ashrov , Guy Katz This is my paper · ORCID
classification cs.LG
keywords LLMsafetyneuralnetworkverificationrobustnessruntimemonitoringstatisticalsemanticperturbationBERT
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper is a case study asking whether a statistical, black-box verification method can fill in where formal verification of LLMs hits its exponential wall. It adapts RoMA, a framework that estimates robustness from the distribution of a network's runner-up confidence scores under perturbations, to language inputs via Word2Vec-based word substitution. Against the Exact Count formal algorithm on six small networks, RoMA's robustness estimates land within about 1 percentage point while reducing runtime from hours to minutes, and on ACAS Xu it finishes in under 16 minutes where exact counting times out after 24 hours. On BERT sentiment classifiers over SST-2, the same method produces quantitative robustness scores under semantic, categorial, and orthographic perturbations, pointing toward online monitoring of deployed LLMs without white-box access. The paper's upshot is that statistical robustness estimation may be a scalable complement to formal verification in runtime-critical LLM deployments.

What carries the argument

The load-bearing object is the probabilistic-local-robustness (plr) score, the probability that the predicted label is preserved when the input is sampled uniformly from an $\epsilon$-ball around the original input. RoMA never enumerates that ball; instead it records the runner-up confidence score (the highest class probability other than the predicted label) across thousands of perturbations, checks the collected scores for normality with the Anderson-Darling test, applies the Box-Cox transformation when normality is rejected, and uses the resulting distribution to estimate the probability of misclassification. In the LLM adaptation, the perturbation step replaces words with Word2Vec neighbors whose cosine similarity to the original word exceeds $1-\epsilon$, making the finite set of synonym substitutions the practical stand-in for the continuous epsilon-ball. This runner-up-score-to-distribution pipeline is what converts a black-box confidence histogram into a quantitative robustness estimate.

What would settle it

Enumerate, for a small enough vocabulary or a tiny BERT-like model, every single-word substitution that passes the Word2Vec cosine threshold on a set of sentences, compute the exact misclassification rate over that full substitution set, run RoMA on the same sentences, and compare: if the two differ by more than the claimed 1% on a non-trivial share of inputs, the transfer claim is refuted.

Watch

Extended reading notes

Core claim

The paper's central claim is that RoMA, a black-box statistical robustness estimator, can be adapted to serve as a runtime monitor for LLMs when formal verification is infeasible. For text input it replaces pixel noise with Word2Vec-based synonym substitution, collecting up to 1,000 perturbed variants per sentence and analyzing the distribution of the runner-up confidence score. On the Exact Count formal baseline, RoMA matches probabilistic-local-robustness values to within about 1 percentage point on the six small synthetic networks where exact counting terminates, and estimates ACAS Xu instances in under 16 minutes where Exact Count times out after 24 hours. On BERT-base sentiment classifiers over SST-2, the method reports 97.18% semantic robustness for the best checkpoint, per-class robustness differences up to 1.5%, and orthographic robustness of 94.44% that agrees with an exhaustive ground-truth evaluation to within 0.17 percentage points. These results are offered as evidence that statistical estimation can narrow the gap between theoretical verification guarantees and operational deployment constraints for LLMs.

Load-bearing premise

The whole method assumes that a few hundred to a thousand synonym-substituted sentence variants, with the model's second-highest confidence scores treated as a normal (or normalizable) distribution, estimate the same probability of misclassification that formal verification defines over all small input changes—an equivalence the paper never checks on the language models themselves.

Editorial extensions

If this is right

  • On the six synthetic models with exact counts, RoMA's estimates land within about 1 percentage point of exact PLR while running in 444-487 seconds versus 211-3,696 seconds for Exact Count.
  • RoMA processes half of the SST-2 test set in under 15 minutes and the full set in under 36 minutes, which the paper takes as evidence that the framework is fast enough for online monitoring of BERT-scale classifiers.
  • The best checkpoint scores 97.18% semantic robustness versus 96.60% for the final checkpoint, indicating that optimizing for classification performance may also improve resilience to semantic perturbations.
  • Per-class robustness varies by up to 1.5%, so a runtime monitor that reports only an aggregate score can miss class-specific weaknesses.
  • Orthographic robustness estimates of 94.44% and 93.94% remain within 0.17% of exhaustive ground truth even when the confidence distributions fail normality tests, suggesting the estimator is not critically dependent on its normality assumption.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Editorial extension: the 1% validation is carried out on small feedforward networks and ACAS Xu, not on transformers; a true transfer claim would require an exact-count comparison on a small transformer, where exhaustive counting is still feasible.
  • Editorial extension: the LLM experiments define robustness as the empirical fraction of perturbed inputs that keep confidence above 0.50, while the formal baseline measures plr over a continuous epsilon-ball; checking whether these two quantities converge on the same sentences would directly test the measure being used.
  • Editorial extension: the orthographic experiment already performs exhaustive character-level enumeration, so the same exhaustive logic could be applied to the semantic substitution set on a small vocabulary, giving a cheap falsification test of RoMA's estimate on the exact distribution it samples from.
  • Editorial extension: if non-normal confidence distributions still yield accurate estimates, a targeted stress test using deliberately bimodal or heavy-tailed runner-up score distributions would map where the estimator breaks.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper adapts the RoMA statistical robustness-verification framework to black-box LLM classification. It replaces pixel noise with semantic perturbations obtained by Word2Vec-based synonym substitution, and evaluates BERT-base sentiment classifiers on SST-2 across embedding, categorial, and orthographic perturbation domains. The authors claim that RoMA estimates robustness within 1% of the Exact Count formal verification baseline on small networks and ACAS Xu, while reducing verification time from hours to minutes, and they report LLM robustness scores of 97.18%/96.60% for the embedding domain and 94.44%/93.94% for the orthographic domain. The paper concludes that RoMA is a scalable candidate for runtime robustness monitoring of LLMs in black-box deployment settings.

Significance. If the central claims held, the paper would offer a useful black-box runtime monitoring approach for LLMs, with a concrete external validation against the Exact Count formal baseline and a public implementation. The orthographic experiment includes an exhaustive ground-truth comparison that is a genuine strength, and the Exact Count reproduction provides an independent check for small feedforward networks. However, the LLM experiments do not actually implement the distribution-derived plr estimator described in Section 4, and the empirical 1% accuracy claim is established only on six small synthetic networks, not on transformers. As presented, the abstract's claim of 'statistically validated bounds' for LLM robustness is not supported by the evidence, though it is plausibly fixable by re-scoping the claims and adding the missing statistical analysis.

major comments (4)
  1. [§5.2 Results] The LLM robustness metric is defined as 'the percentage of semantically perturbed inputs maintaining classification confidence scores above the 0.50 threshold' (Section 5.2, Results). This is an empirical accuracy rate, not the plr estimate described in Section 4, which fits runner-up confidence distributions with the Anderson-Darling test and Box-Cox transformation and then derives a probability from the fitted distribution. No confidence intervals are reported for any LLM score, and the paper never shows that the empirical percentage equals or bounds the distribution-derived plr. Consequently, the abstract's 'statistically validated bounds' claim is unsupported for the LLM experiments.
  2. [§4 vs. Definition 2] Definition 2 defines plr with respect to a uniform measure on a continuous ℓ∞ ball around x0, while the perturbation generator in Section 4 selects discrete synonym substitutions using a Word2Vec cosine-similarity threshold of 1−ϵ. The sampling distribution is induced by the embedding vocabulary, sentence structure, and the random choice of word positions, and no argument is given that this distribution approximates the uniform measure on the ℓ∞ ball. Thus the reported scores cannot be directly interpreted as estimates of plr as defined, and the paper should either state explicitly that the target quantity is robustness under the specific semantic perturbation distribution or provide a uniformity argument.
  3. [§5.1, Table 1] The 'within 1% deviation' claim is established only on the six synthetic models for which Exact Count terminates (Model_2_20 through Model_5_95). For Model_10_76 and the five ACAS Xu properties, Exact Count times out after 24 hours, so no ground-truth comparison exists. These networks are also small feedforward ReLU networks, not transformer-based LLMs. The authors acknowledge the architectural gap in the text accompanying Table 1, but the abstract and Section 5.1's 'Implications for Runtime Verification' still present the sub-1% accuracy as applying to the LLM setting. This extrapolation needs to be explicitly qualified or supported by LLM-specific validation.
  4. [§5.4] In the orthographic experiment, the paper reports that runner-up confidence scores fail Anderson-Darling normality tests even after Box-Cox transformation, yet the framework still produces robustness estimates that match exhaustive ground truth within 0.17%. This agreement is a useful empirical sanity check, but it directly undercuts the premise that distributional normality is required for 'statistically validated bounds'. The paper should either present the orthographic results as an empirical validation of a non-parametric estimator, or restrict the 'statistically validated' wording to the settings where the distributional assumptions are actually satisfied.
minor comments (5)
  1. [Abstract and §1] The phrase 'with 50% of SST-2 sentences were processed within 15 minutes' is grammatically awkward and the percentage refers to instances, so please revise for clarity.
  2. [References] Several reference entries contain encoding artifacts, for example reference [1] ('Anderson, T.: Anderson-Darling Tests...') and reference [3] ('B¯ erzin,š, J.'); these should be cleaned up.
  3. [Table 1] The text refers to 'T able 1' with an extra space, and the table column alignment could be improved for readability.
  4. [Figure 3] The vertical axis label reads 'Frequency (percentage)' while the caption describes a cumulative distribution function; please correct the axis label to reflect the CDF.
  5. [§5.2] Please clarify whether the reported processing times include the time for generating up to 1,000 perturbations per sentence or only the model inference time.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the central 1% accuracy check uses an external Exact Count baseline, and the LLM robustness scores are direct empirical rates rather than fitted outputs.

full rationale

The paper's headline numerical claim—RoMA within 1% of Exact Count—is validated against Marzari et al.'s external formal verification algorithm (Section 5.1, Table 1); the six reported comparisons are computed from independent algorithms and are not derived from RoMA's own assumptions. The LLM robustness percentages in Sections 5.2–5.4 are defined directly as the percentage of perturbed inputs whose confidence stays above 0.5, i.e., an empirical frequency under the specified perturbation distribution, not a quantity forced to equal the fitted distribution's output. The orthographic subsection even provides an exhaustive ground-truth check (94.61% vs 94.44%), an external consistency test. Self-citations to the authors' RoMA/gRoMA papers motivate the framework, but the framework's statistical assumptions are re-tested here on BERT confidence distributions (81.60%/75.07% normality), so the citation is not load-bearing. The skeptical concerns about the Word2Vec sampler not being uniform over the continuous l-infinity ball and about unstated confidence intervals are real validity/reporting gaps, but they are not circularity: no equation in the paper reduces to its own input by construction, and no fitted parameter is renamed as a prediction. Per the hard rules, concerns about unsupported generalization or missing bounds belong to correctness risk, not circularity.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

The paper's central numbers depend on two hand-picked thresholds (epsilon and the 0.50 confidence cutoff), a chosen sampling budget, and several domain assumptions about how discrete text perturbations relate to the continuous plr definition. No new theoretical entities are introduced.

free parameters (4)
  • epsilon (perturbation threshold 1-epsilon cosine similarity) = 0.35
    Hand-picked threshold controlling semantic drift in Section 4; it determines which word replacements are allowed and therefore directly shapes the robustness percentages.
  • confidence threshold for correct classification = 0.50
    Used in Section 5.2 to define whether a perturbed input is counted as robust; changes in this threshold change the reported scores.
  • number of perturbations per sentence = up to 1,000 (semantic), 500 (orthographic)
    Sampling budget chosen for the experiments; affects the variance and reliability of the estimates.
  • sentence subset size = 1,821 test sentences; 500 for orthographic estimate
    The semantic robustness is evaluated on the whole SST-2 test set, while the orthographic estimate uses 500 sentences and is validated exhaustively on all 1,821.
assumptions (5)
  • domain assumption Uniformly sampled semantic word substitutions represent a uniform sample over the epsilon-ball in which plr is defined.
    Section 4 generates up to 1,000 variants per sentence by sampling word replacements, then treats the runner-up score distribution as evidence about local robustness; the paper does not prove this sampling distribution matches the input-space measure in Definition 2.
  • domain assumption Runner-up confidence scores after Box-Cox transformation are approximately normal.
    Section 4 relies on Anderson-Darling tests and Box-Cox to approximate normality; the paper reports normality in 81.60% and 75.07% of cases, so the assumption is empirically tested but not guaranteed. The orthographic experiments show it fails and the method still proceeds.
  • ad hoc to paper Empirical fraction of perturbations keeping correct-class confidence above 0.50 equals the plr robustness score.
    Section 5.2 defines robustness as this percentage, but the text does not show it equals the runner-up distribution-based estimate described in Section 4; this is the key unexplained bridge.
  • domain assumption Statistical accuracy measured on small feedforward networks transfers to transformer-based LLMs.
    Section 5.1 acknowledges this limitation and says LLM experiments provide complementary evidence, but the 1% accuracy claim from the small networks is used to motivate the LLM application.
  • domain assumption Exact Count's violation rate is ground truth for plr.
    The paper treats Exact Count as an authoritative formal baseline; this is reasonable for small networks but exact only up to its own implementation and partitioning assumptions.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Statistical Runtime Verification for LLMs via Robustness Estimation." pith.science (2026). https://pith.science/paper/M5Z7SRPJ

@misc{pith2026250417723,
  author       = {Pith},
  title        = {Pith review of: Statistical Runtime Verification for LLMs via Robustness Estimation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/M5Z7SRPJ}},
  note         = {Machine review of arXiv:2504.17723}
}
read the original abstract

Adversarial robustness verification is essential for ensuring the safe deployment of Large Language Models (LLMs) in runtime-critical applications. However, formal verification techniques remain computationally infeasible for modern LLMs due to their exponential runtime and white-box access requirements. This paper presents a case study adapting and extending the RoMA statistical verification framework to assess its feasibility as an online runtime robustness monitor for LLMs in black-box deployment settings. Our adaptation of RoMA analyzes confidence score distributions under semantic perturbations to provide quantitative robustness assessments with statistically validated bounds. Our empirical validation against formal verification baselines demonstrates that RoMA achieves comparable accuracy (within 1\% deviation), and reduces verification times from hours to minutes. We evaluate this framework across semantic, categorial, and orthographic perturbation domains. Our results demonstrate RoMA's effectiveness for robustness monitoring in operational LLM deployments. These findings point to RoMA as a potentially scalable alternative when formal methods are infeasible, with promising implications for runtime verification in LLM-based systems.

Figures

Figures reproduced from arXiv: 2504.17723 by the authors.

Figure 1
Figure 1. PLR scores of RoMA and the Exact Count algorithm across bench￾mark models. 100 1000 10000 2_20 2_56 2_68 5_09 5_50 5_95 10_76 Runtime (Seconds) - log scale RoMA vs. Exact Count - Run-time RoMA - Runtime Exact Count - Runtime [PITH_FULL_IMAGE:figures/full_fig_p012_1.png] view at source ↗
Figure 3
Figure 3. Cumulative distribution function (CDF) showing the percentage of SST-2 dataset instances processed over time by RoMA in the LLM embedding case study for the two models. Results. We quantified embedding robustness as the percentage of semantically perturbed inputs maintaining classification confidence scores above the 0.50 threshold for correct sentiment classification. Lower confidence scores indicate reduced classi… view at source ↗
Figure 4
Figure 4. A comparison of categorial robustness between Mbest and Mfinal. 5.4 Orthographic Perturbation Analysis for Runtime Input Validation Experimental Design. To evaluate LLM resilience to typographical errors in operational environments, we implemented systematic character-level perturba￾tions simulating common human typing mistakes. Specifically, each character in every word was systematically replaced with all possible… view at source ↗

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. T5-CSBoost: Adversarial Perturbation Resistant LLM Fingerprinting

    cs.CL 2026-05 conditional novelty 6.0 of 10

    Adding a margin-based triplet loss to T5-Sentinel's decoder embeddings improves LLM source attribution robustness to word/character edits, paraphrasing, and unseen models/domains.

Reference graph

Works this paper leans on

57 extracted references · 38 canonical work pages · cited by 1 Pith paper

  1. [1]

    Anderson, T.: Anderson-Darling Tests of Goodness-of-Fit. Int. Encyclopedia of Statistical Science 1, 52–54 (2011)

    work page 2011

  2. [2]

    BERT-Base-Uncased (2023),https://huggingface.co/google-bert/bert-base- uncased

    work page 2023

  3. [3]

    MZ Computing Journal5(2) (2024)

    B¯ erzin,š, J., Kalnin,a, E.: Robustness of Pre-Trained Language Models Against Adversarial Attacks. MZ Computing Journal5(2) (2024)

    work page 2024

  4. [4]

    Journal of the American Statistical Association77(377), 209–210 (1982)

    Box, G., Cox, D.: An Analysis of Transformations Revisited, Rebutted. Journal of the American Statistical Association77(377), 209–210 (1982)

    work page 1982

  5. [5]

    Verification of Neural Networks Competition (VNN-COMP): Summary and Results (2023), Technical Report

    Brix, C., Bak, S., Liu, C., Johnson, T.: The Fourth Int. Verification of Neural Networks Competition (VNN-COMP): Summary and Results (2023), Technical Report. https://arxiv.org/abs/2312.16760 18 N. Levy et al

    arXiv 2023

  6. [6]

    Carlini, N., Katz, G., Barrett, C., Dill, D.: Provably Minimally-Distorted Adversarial Examples (2017), Technical Report.https://arxiv.org/abs/1709.10207

    arXiv 2017

  7. [7]

    In: Proc

    Cheong, I., Xia, K., Feng, K.K., Chen, Q.Z., Zhang, A.X.: (A) I am Not A Lawyer, But...: Engaging Legal Experts Towards Responsible LLM Policies for Legal Advice. In: Proc. ACM Conf. on Fairness, Accountability, and Transparency (FACCT). pp. 2454–2469 (2024)

    work page 2024

  8. [8]

    In: Proc

    Cohen, J., Rosenfeld, E., Kolter, Z.: Certified Adversarial Robustness Via Ran- domized Smoothing. In: Proc. 36th Int. Conf. on Machine Learning (ICML). pp. 1310–1320 (2019)

    work page 2019

Show all 57 references
  1. [9]

    http://arxiv.org/abs/1810.04805

    Devlin, J., Chang, M., Lee, K., Toutanova, K.: BERT: Pre-Training of Deep Bidirectional Transformers for Language Understanding (2018), Technical Report. http://arxiv.org/abs/1810.04805

    2018 arXiv

  2. [10]

    Dong, X., Luu, A.T., Ji, R., Liu, H.: Towards Robustness Against Natural Language Word Substitutions (2021), Technical Report.https://arxiv.org/abs/2107.13541

    2021 arXiv

  3. [11]

    Elbadawi, M., Li, H., Basit, A.W., Gaisford, S.: The Role of Artificial Intelligence in Generating Original Scientific Research. Int. Journal of Pharmaceutics652, 123741 (2024)

    2024

  4. [12]

    Goodfellow, I., Shlens, J., Szegedy, C.: Explaining and Harnessing Adversarial Examples (2014), Technical Report.http://arxiv.org/abs/1412.6572

    2014 arXiv

  5. [13]

    In: Proc

    Guo, C., Pleiss, G., Sun, Y., Weinberger, Q.: On Calibration of Modern Neural Networks. In: Proc. 34th Int. Conf. on Machine Learning. pp. 1321–1330 (2017)

    2017

  6. [14]

    Hadar, A., Levy, N., Winokur, M.: Management and Detection System for Medical Surgical Equipment (2022), Technical Report.http://arxiv.org/abs/2211.02351

    2022 arXiv

  7. [15]

    In: Proc

    Hashemi, V., Křetínsk` y, J., Rieder, S., Schön, T., Vorhoff, J.: Gaussian-Based and Outside-the-Box Runtime Monitoring Join Forces. In: Proc. 24th Int. Conf. on Runtime Verification. pp. 218–228 (2024)

    2024

  8. [16]

    In: Proc

    He, W., Wu, C., Bensalem, S.: Box-Based Monitor Approach for Out-of-Distribution Detection in YOLO: An Exploratory Study. In: Proc. 24th Int. Conf. on Runtime Verification (RV). pp. 229–239 (2024)

    2024

  9. [17]

    In: Proc

    Huang, C., Hu, Z., Huang, X., Pei, K.: Statistical Certification of Acceptable Robustness for Neural Networks. In: Proc. 30th Int. Conf. on Artificial Neural Networks (ICANN). pp. 79–90 (2021)

    2021

  10. [18]

    In: IEEE/RSJ Int

    Huang, Y., Sansom, J., Ma, Z., Gervits, F., Chai, J.: Drivlme: Enhancing LLM- based Autonomous Driving Agents with Embodied and Social Experiences. In: IEEE/RSJ Int. Conf. on Intelligent Robots and Systems (IROS). pp. 3153–3160. IEEE (2024)

    2024

  11. [19]

    In: Proc

    Jin, D., Jin, Z., Zhou, J.T., Szolovits, P.: Is Bert Really Robust? A Strong Baseline for Natural Language Attack on Text Classification and Entailment. In: Proc. 34th of the AAAI Conf. on Artificial Intelligence. pp. 8018–8025 (2020)

    2020

  12. [20]

    Jones, E., Jia, R., Raghunathan, A., Liang, P.: Robust Encodings: A Framework for Combating Adversarial Typos (2020), Technical Report.https://arxiv.org/ abs/2005.01229

    2020 arXiv

  13. [21]

    Journal of Guidance, Control, and Dynamics 42(3), 598–608 (2019)

    Julian, D., Kochenderfer, J., Owen, P.: Deep Neural Network Compression for Aircraft Collision Avoidance Systems. Journal of Guidance, Control, and Dynamics 42(3), 598–608 (2019)

    2019

  14. [22]

    In: Proc

    Katz, G., Barrett, C., Dill, D., Julian, K., Kochenderfer, M.: Reluplex: An Efficient SMT Solver for Verifying Deep Neural Networks. In: Proc. 29th Int. Conf. on Computer Aided Verification (CAV). pp. 97–117 (2017)

    2017

  15. [23]

    Formal Methods in System Design (FMSD) (2021) Statistical Runtime Verification for LLMs via Robustness Estimation 19

    Katz, G., Barrett, C., Dill, D., Julian, K., Kochenderfer, M.: Reluplex: a Calculus for Reasoning about Deep Neural Networks. Formal Methods in System Design (FMSD) (2021) Statistical Runtime Verification for LLMs via Robustness Estimation 19

    2021

  16. [24]

    In: Proc

    Katz, G., Huang, D., Ibeling, D., Julian, K., Lazarus, C., Lim, R., Shah, P., Thakoor, S., Wu, H., Zeljić, A., Dill, D., Kochenderfer, M., Barrett, C.: The Marabou Framework for Verification and Analysis of Deep Neural Networks. In: Proc. 31st Int. Conf. on Computer Aided Veri...

    2019

  17. [25]

    In: Proc

    Katz, G., Levy, N., Refaeli, I., Yerushalmi, R.: DEM: A Method for Certifying Deep Neural Network Classifier Outputs in Aerospace. In: Proc. 43rd Digital Avionics Systems Conf. (DASC) (2024)

    2024

  18. [26]

    In: Proc

    Kim, H., Papamakarios, G., Mnih, A.: The Lipschitz Constant of Self-Attention. In: Proc. 38th Int. Conf. on Machine Learning (ICML). pp. 5562–5571 (2021)

    2021

  19. [27]

    Landi, A., Nicholson, M.: ARP4754A/ED-79A-Guidelines for Development of Civil Aircraft and Systems-Enhancements, Novelties and Key Topics. SAE Int. Journal of Aerospace 4, 871–879 (2011)

    2011

  20. [28]

    Levy, N., Ashrov, A., Katz, G.: Towards Robust LLMs: an Adversarial Robust- ness Measurement Framework — Code (2024),https://github.com/adielashrov/ trust-ai-roma-for-llm

    2024

  21. [29]

    In: Proc

    Levy, N., Katz, G.: RoMA: a Method for Neural Network Robustness Measure- ment and Assessment. In: Proc. 29th Int. Conf. on Neural Information Processing (ICONIP) (2021)

    2021

  22. [30]

    In: Proc

    Levy, N., Yerushalmi, R., Katz, G.: gRoMA: a Tool For Measuring the Global Robustness of Deep Neural Networks. In: Proc. 12th Int. Symposium on Leveraging Applications of Formal Methods, Verification and Validation (ISoLA). pp. 160–170 (2023)

    2023

  23. [31]

    In: Proc

    Levy, O., Dikman, I., Levy, N., Winokur, M.: Work in Progress: AI-Powered Engineering-Bridging Theory and Practice. In: Proc. 9th IEEE World Engineering Education Conf. (EDUNINE) (2025)

    2025

  24. [32]

    Marzari, L., Corsi, D., Cicalese, F., Farinelli, A.: The #Dnn-Verification Problem: Counting Unsafe Inputs for Deep Neural Networks (2023), Technical Report.https: //arxiv.org/abs/2301.07068

    2023 arXiv

  25. [33]

    Mikolov, T., Chen, K., Corrado, G., Dean, J.: Efficient Estimation of Word Rep- resentations in Vector Space (2013), Technical Report.https://arxiv.org/abs/ 1301.3781

    2013 arXiv

  26. [34]

    Morris, J.X., Lifland, E., Yoo, J.Y., Grigsby, J., Jin, D., Qi, Y.: Textattack: A Framework for Adversarial Attacks, Data Augmentation, and Adversarial Training in NLP (2020), Technical Report.https://arxiv.org/abs/2005.05909

    2020 arXiv

  27. [35]

    OpenAI: GPT-4 Technical Report (2024), Technical Report.https://arxiv.org/ abs/2303.08774

    2024 arXiv

  28. [36]

    OpenAI: Chatgpt (july 2025 version).https://chat.openai.com (2025), accessed: July 2025

    2025

  29. [37]

    In: Proc

    Owen, M., Panken, A., Moss, R., Alvarez, L., Leeper, C.: ACAS Xu: Integrated Collision Avoidance and Detect and Avoid Capability for UAS. In: Proc. 38th IEEE/AIAA Digital Avionics Systems Conf. (DASC). pp. 1–10 (2019)

    2019

  30. [38]

    In: Proc

    Pennington, J., Socher, R., Manning, C.: Glove: Global Vectors for Word Represen- tation. In: Proc. Int. Conf. on Empirical Methods in Natural Language Processing (EMNLP). pp. 1532–1543 (2014)

    2014

  31. [39]

    Scientific Reports 15(1), 9378 (2025)

    Qiu, S., Liu, Q., Zhou, S., Gou, M., Zeng, Y., Zhang, Z., Wu, Z.: Hard Label Adversarial Attack with High Query Efficiency Against Nlp Models. Scientific Reports 15(1), 9378 (2025)

    2025

  32. [40]

    Levy et al

    Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., Liu, P.J.: Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer (2023), Technical Report.https://arxiv.org/abs/1910.10683 20 N. Levy et al

    2023 arXiv

  33. [41]

    Rohan, A., et al.: PaLM 2 Technical Report (2023), Technical Report.https: //arxiv.org/abs/2305.10403

    2023 arXiv

  34. [42]

    25th Int

    Romero-Alvarado, D., Hernández-Orallo, J., Martínez-Plumed, F.: How Resilient are Language Models to Text Perturbations? In: Proc. 25th Int. Conf. on Intelligent Data Engineering and Automated Learning (IDEAL). pp. 85–96 (2024)

    2024

  35. [43]

    Future Internet15(6), 192 (2023)

    Roumeliotis, K., Tselikas, N.: ChatGPT and Open-AI Models: A Preliminary Review. Future Internet15(6), 192 (2023)

    2023

  36. [44]

    Sato, M., Suzuki, J., Shindo, H., Matsumoto, Y.: Interpretable Adversarial Per- turbation in Input Embedding Space for Text (2018), Technical Report.https: //arxiv.org/abs/1805.02917

    2018 arXiv

  37. [45]

    Singh, A., Singh, N., Vatsal, S.: Robustness of LLMs to Perturbations in Text (2024), Technical Report.https://arxiv.org/abs/2407.08989

    2024

  38. [46]

    and Ng, Andrew and Potts, Christopher: Recursive Deep Models for Semantic Compositionality Over a Sentiment Treebank

    Socher, Richard and Perelygin, Alex and Wu, Jean and Chuang, Jason and Manning, Christopher D. and Ng, Andrew and Potts, Christopher: Recursive Deep Models for Semantic Compositionality Over a Sentiment Treebank. In: Proc. of the 2013 Conf. on Empirical Methods in Natural Lang...

    2013

  39. [47]

    Subramanian, V., Benetos, E., Xu, N., McDonald, S., Sandler, M.: Adversarial At- tacks in Sound Event Classification (2019), Technical Report.https://arxiv.org/ abs/1907.02477

    2019 arXiv

  40. [48]

    Temple, B., Buescher, K., Armstrong, J.: PyRAT-Python Radiography Analysis Tool (u) (2011), Los Alamos National Laboratory (LANL)

    2011

  41. [49]

    Touvron, H., et al.: LLaMA: Open and Efficient Foundation Language Models (2023), Technical Report.https://arxiv.org/abs/2302.13971

    2023 arXiv

  42. [50]

    In: Proc

    Tsuzuku, Y., Sato, I., Sugiyama, M.: Lipschitz-Margin Training: Scalable Certifica- tion of Perturbation Invariance for Deep Neural Networks. In: Proc. 32nd Advances in Neural Information Processing Systems (NeurIPS). pp. 6541–6550 (2018)

    2018

  43. [51]

    Wang, B., Xu, C., Wang, S., Gan, Z., Cheng, Y., Gao, J., Awadallah, A.H., Li, B.: Adversarial GLUE: A Multi-Task Benchmark for Robustness Evaluation of Language Models (2021), Technical Report.https://arxiv.org/abs/2111.02840

    2021 arXiv

  44. [52]

    In: Proc

    Wang, S., Zhang, H., Xu, K., Lin, X., Jana, S., Hsieh, C.J., Kolter, Z.: Beta-CROWN: Efficient Bound Propagation with Per-Neuron Split Constraints for Complete and Incomplete Neural Network Verification. In: Proc. 35th Conf. on Neural Information Processing Systems (NeurIPS) (2021)

    2021

  45. [53]

    https://arxiv.org/abs/2406.11020

    Wang, Y., Zhao, Y.: Rupbench: Benchmarking Reasoning under Perturbations for Robustness Evaluation in Large Language Models (2024), Technical Report. https://arxiv.org/abs/2406.11020

    2024 arXiv

  46. [54]

    In: Proc

    Webb, S., Rainforth, T., Teh, Y.W., Kumar, P.: A Statistical Approach to Assessing Neural Network Robustness. In: Proc. 7th Int. Conf. on Learning Representations (ICLR) (2019)

    2019

  47. [55]

    In: Proc

    Yang, F., Zhan, S.S., Wang, Y., Huang, C., Zhu, Q.: Case Study: Runtime Safety Verification of Neural Network Controlled System. In: Proc. 24th Int. Conf. on Runtime Verification (RV). pp. 205–217 (2024)

    2024

  48. [56]

    Yoshida, Y., Miyato, T.: Spectral Norm Regularization for Improving the Gener- alizability of Deep Learning (2017), Technical Report.https://arxiv.org/abs/ 1705.10941

    2017 arXiv

  49. [57]

    https://arxiv.org/abs/2303.18223

    Zhao, W.X., et al.: A Survey of Large Language Models (2025), Technical Report. https://arxiv.org/abs/2303.18223

    2025 arXiv

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.