Pith. sign in

REVIEW 4 major objections 6 minor 36 references

Reviewed by Pith at T0; open to challenge.

T0 means a machine referee read the full paper against a public rubric. The mark states how deep the mechanical check went, never who wrote it. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · glm-5.2

Automated NLP classifier rates LLM-generated safety defeaters at F1=0.84

2026-07-08 18:20 UTC pith:A3RX5KKJ

load-bearing objection Novel application of BERT+SVM to defeater quality assessment, but headline F1=0.84 is inflated by an undefined metric and majority-class collapse the 4 major comments →

arxiv 2607.06039 v1 pith:A3RX5KKJ submitted 2026-07-07 cs.SE

Automating Quality Assessment with NLP of LLM-Generated Defeaters

Tihomir Rohlinger , Daniel Ratiu , Stefan Wagner This is my paper
classification cs.SE
keywords defeaterassurance caseNLPBERTSVMmeta-classifierinter-rater agreementCohen's kappa
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper claims that automated NLP-based classifiers can predict expert quality ratings for LLM-generated defeaters—challenges to safety assurance case arguments—with an average F1-score of 0.84 across two industrial case studies (Adaptive Cruise Control and CERN's Large Hadron Collider protection system). The central mechanism is a pipeline that combines BERT semantic embeddings of defeater text with structural features extracted from the assurance case graph (such as whether a defeater links to a specific claim and its cosine similarity to the path from root to that claim), feeding these into SVM-based simple and meta-classifiers. The paper also documents that human expert reviewers agree poorly, with Cohen's kappa values sometimes negative (worse than chance), and claims that the automated approach improves inter-rater agreement by approximately 40% in kappa compared to the baseline human-human agreement, thereby reducing subjective variance in defeater validation.

Core claim

The paper's central finding is that a meta-classifier combining BERT embeddings of defeater text with graph-structural validity features (linkage indicators, cosine similarity to linked assurance case elements, path similarity, and average graph similarity) can reproduce expert quality ratings for LLM-generated defeaters with F1=0.84 on average across two domains, while simultaneously achieving more consistent ratings than either of two human experts who sometimes disagreed worse than chance.

What carries the argument

The pipeline operates on defeaters structured as three components (What: the flaw, Where: the affected claim, Why: the rationale). BERT (bert-base-uncased) produces 768-dimensional mean-pooled embeddings. These feed SVM classifiers with hyperparameter tuning via grid search and 5-fold cross-validation. A meta-classifier layer combines simple classifier probabilities with four structural validity features: Is Linked (binary), Linked Element cosine similarity, Path Similarity (cosine similarity to concatenated root-to-linked-element path text), and Similarity Average (mean cosine similarity to all assurance case embeddings). SMOTE addresses class imbalance; CalibratedClassifierCV with sigmoid法

Load-bearing premise

The classifiers are trained only on 'consensus defeaters' where two human experts agreed, then evaluated on 'dissensus defeaters' where they disagreed. This assumes the consensus cases represent a reliable ground truth—but the human experts sometimes disagreed worse than chance (negative kappa), the training sets are tiny (32 and 42 defeaters), and 1-2 dissensus defeaters were manually moved into the consensus set for class balance, which is a post-hoc adjustment to the data.

What would settle it

If a third expert adjudicator rated the dissensus defeaters and the classifier's predictions systematically disagreed with the adjudicated labels, the F1=0.84 would be shown to reflect agreement with one rater's bias rather than objective quality assessment.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • If automated defeater assessment proves reliable at scale, safety engineers could triage large volumes of LLM-generated safety challenges in minutes rather than hours, focusing human attention only on borderline cases flagged by low classifier confidence.
  • The finding that human experts disagree worse than chance on certain defeater components (e.g., the 'Why' rationale) suggests that the rating rubric itself may need revision, and automated classifiers could serve as a diagnostic tool for identifying which quality dimensions are inherently ambiguous to humans.
  • The structural feature approach—linking natural language to graph topology via cosine similarity—could generalize beyond safety assurance cases to any domain where generated text must be evaluated against a structured argumentation framework, such as legal reasoning or clinical decision support.
  • If the consensus-defeater training approach scales, it could enable continuous assurance pipelines where LLMs generate defeaters, classifiers triage them, and humans review only the dissensus cases, creating a human-in-the-loop system that adapts as safety cases evolve.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The claim of '40% improvement in kappa' is relative to a baseline that includes negative kappa values; in absolute terms, the model-human kappa values remain mostly below 0.35, which is still 'fair' at best. The framing as 'reducing subjectivity' is defensible but the practical reliability of the automated ratings as a standalone tool remains an open question.
  • The training sets (32 ACC, 42 CERN consensus defeaters) are small enough that the F1=0.84 may partly reflect the simplicity of the dissensus evaluation set rather than genuine generalization. A test on a third, unseen domain would be more convincing than cross-validation within the same two domains.
  • The paper does not address whether the classifier's predictions are correct in any absolute sense—it measures agreement with human raters, who themselves disagree. A natural extension would be to have a third expert adjudicate the dissensus cases and then compare the classifier's predictions against that adjudicated ground truth.
  • The structural features (Is Linked, path similarity) may be doing most of the work; an ablation that isolates text-only BERT embeddings from graph-structural features would clarify whether the semantic richness of BERT or the structural grounding to the assurance case DAG is the primary driver of the 0.84 F1.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. This paper proposes an automated NLP-based method for assessing the quality of LLM-generated defeaters in safety assurance cases. The approach combines BERT embeddings of defeater texts with structural features derived from assurance case graphs (cosine similarities to linked elements, path similarities, etc.) and trains SVM and logistic regression classifiers (including meta-classifiers) on expert consensus labels. The method is evaluated on two case studies (ACC automotive, CERN LHC) using data from a prior study by Viger et al. The paper reports an average F1-score of 0.84 and a ~40% improvement in Cohen's kappa over baseline human-human agreement, arguing that automated assessment can reduce subjective variance in defeater validation.

Significance. The problem of automating quality assessment of LLM-generated assurance case fragments is timely and practically relevant. The integration of structural assurance case graph features with BERT embeddings is a reasonable methodological contribution. The paper provides open-source data references and a reproducible experimental setup (fixed random state, scikit-learn defaults). The inter-rater agreement analysis quantifying expert dissensus is a useful empirical contribution. However, the significance of the results is substantially undermined by the metric and evaluation issues detailed below.

major comments (4)
  1. §V, Table II: The headline F1=0.84 claim is drawn from the 'General' match metric, but the paper never defines what 'general match' means when evaluating on dissensus defeaters where H1≠H2. The text in §V says Table II shows 'general match against both raters,' but no formula or precise definition is given. If a prediction is counted as correct when it matches either H1 or H2, this metric is substantially easier than matching a single ground truth. The large gap between Table II (0.81–0.88) and Table III individual-rater F1 scores (0.32–0.71) is consistent with this interpretation. The paper must explicitly define the 'General' metric and justify why it is an appropriate evaluation criterion for dissensus data.
  2. §V, Table IV: The models exhibit severe majority-class collapse. For ACC What, the SVM meta/SVM simple model predicts 36/42 defeaters as class 2; for CERN What, 47/50 as class 2. Given that in dissensus cases at least one rater frequently assigned class 2 (e.g., CERN What H2 has 42/50 as class 2), a model that always predicts class 2 would achieve a high 'general match' F1 under a match-either-rater criterion. The paper should report the F1 of a trivial majority-class baseline under the same 'General' metric to demonstrate that the reported scores are not an artifact of class collapse combined with a lenient matching criterion.
  3. §V, Table V and §V-B: The claim of '~40% improvement' in inter-rater agreement is misleading in context. While the absolute kappa values do improve (e.g., ACC Why: from −0.50 to −0.09/0.10), the resulting agreement between model and individual raters remains at or below chance for several components (CERN Why: H1/Model κ=−0.06, H2/Model κ=−0.03; CERN What: H2/Model κ=−0.06). The paper should explicitly state the post-improvement kappa values and acknowledge that the model's agreement with individual raters remains near-zero or negative for these components, rather than framing the improvement as evidence that the approach 'reduces subjective variance.'
  4. §IV-C: The training sets are extremely small (32 ACC, 42 CERN consensus defeaters). The manual transfer of 1–2 dissensus defeaters into the consensus set 'for class balance' constitutes post-hoc data manipulation that is not adequately justified. The paper should explain the selection criteria for these transferred defeaters and report sensitivity of results to their inclusion/exclusion.
minor comments (6)
  1. §III-B: The kappa values reported in the text (e.g., ACC Why κ=−0.024) differ from those in Table V (ACC Why κ=−0.50). Please reconcile these discrepancies or clarify which values are correct.
  2. Table I: The 'Correctness' attribute description conflates the quality attribute with the evaluation metric (F1). Consider separating the conceptual attribute from its operationalization.
  3. §III-D4: The self-training confidence threshold of 0.9 and SMOTE k parameter choices are stated but not justified. A brief rationale would improve reproducibility.
  4. §IV-E1: The 'Where' component is evaluated with a static regex filter but is included in Tables IV–V. The paper notes it is 'not representative,' but it would be clearer to exclude it from the main results or move it to an appendix.
  5. Figure 2 is referenced but not legible in the reviewed version; ensure it is readable in the final version.
  6. Several typos: 'exension' (§III, intro), 'Therby' (§IV-B), 'additionly' (§V intro), 'defeaters×2 reviewers×2 components' should clarify that 'Where' is handled separately.

Simulated Author's Rebuttal

4 responses · 0 unresolved

We thank the referee for a careful and substantive review. The referee raises four major points concerning (1) the undefined 'General' match metric, (2) potential majority-class collapse inflating scores, (3) misleading framing of kappa improvement, and (4) post-hoc data manipulation in training set construction. We agree that points 1, 2, and 3 require manuscript revisions to define metrics, add baselines, and correct framing. On point 4, we will clarify the selection criteria and add sensitivity analysis. We disagree only with the characterization of the transfer as 'manipulation' rather than a documented (if under-justified) preprocessing step.

read point-by-point responses

  1. Referee: §V, Table II: The headline F1=0.84 claim is drawn from the 'General' match metric, but the paper never defines what 'general match' means when evaluating on dissensus defeaters where H1≠H2. The text in §V says Table II shows 'general match against both raters,' but no formula or precise definition is given. If a prediction is counted as correct when it matches either H1 or H2, this metric is substantially easier than matching a single ground truth. The large gap between Table II (0.81–0.88) and Table III individual-rater F1 scores (0.32–0.71) is consistent with this interpretation. The paper must explicitly define the 'General' metric and justify why it is an appropriate evaluation criterion for dissensus data.

    Authors: The referee is correct that the 'General' metric is not formally defined in the manuscript. We will add an explicit definition in the revised §V. The metric counts a prediction as correct when it matches either H1 or H2 on a given dissensus defeater. The referee's interpretation of the gap between Table II and Table III is correct: the General metric is more lenient than individual-rater matching by construction, because dissensus cases are those where H1≠H2, so matching either rater is easier than matching a single fixed ground truth. We will state this explicitly and justify the metric's use as follows: on dissensus data, there is no single ground truth, so the General metric measures whether the model's prediction falls within the range of expert judgment rather than whether it replicates one specific rater. This is a meaningful question for decision support, but we agree it should not be presented as equivalent to standard F1 against a single label. We will add a sentence clarifying that General F1 is not directly comparable to individual-rater F1 and adjust the headline framing accordingly. revision: yes

  2. Referee: §V, Table IV: The models exhibit severe majority-class collapse. For ACC What, the SVM meta/SVM simple model predicts 36/42 defeaters as class 2; for CERN What, 47/50 as class 2. Given that in dissensus cases at least one rater frequently assigned class 2 (e.g., CERN What H2 has 42/50 as class 2), a model that always predicts class 2 would achieve a high 'general match' F1 under a match-either-rater criterion. The paper should report the F1 of a trivial majority-class baseline under the same 'General' metric to demonstrate that the reported scores are not an artifact of class collapse combined with a lenient matching criterion.

    Authors: This is a fair and important point. We will add a majority-class baseline (always predict class 2) evaluated under the same General metric for each component and dataset. We acknowledge that the class distributions in Table IV show substantial concentration on class 2, and the referee is right that this could inflate the General metric. We will report the baseline F1 alongside the model scores so readers can assess the marginal improvement over a trivial predictor. If the baseline achieves a high General F1, we will state this plainly and reframe the contribution accordingly. We note that the individual-rater F1 scores in Table III provide a complementary view that is less susceptible to this artifact, and we will cross-reference both tables more explicitly in the revision. revision: yes

  3. Referee: §V, Table V and §V-B: The claim of '~40% improvement' in inter-rater agreement is misleading in context. While the absolute kappa values do improve (e.g., ACC Why: from −0.50 to −0.09/0.10), the resulting agreement between model and individual raters remains at or below chance for several components (CERN Why: H1/Model κ=−0.06, H2/Model κ=−0.03; CERN What: H2/Model κ=−0.06). The paper should explicitly state the post-improvement kappa values and acknowledge that the model's agreement with individual raters remains near-zero or negative for these components, rather than framing the improvement as evidence that the approach 'reduces subjective variance.'

    Authors: We agree. The '~40% improvement' framing is misleading because it describes relative improvement from a negative baseline without acknowledging that the resulting kappa values remain at or near zero for several components. We will revise §V-B to explicitly state the post-improvement kappa values from Table V and acknowledge that for CERN Why (H1/Model κ=−0.06, H2/Model κ=−0.03) and CERN What (H2/Model κ=−0.06), the model's agreement with individual raters remains near chance or below. We will remove or qualify the claim that the approach 'reduces subjective variance' for these components and restrict the claim to components where the improvement is meaningful (e.g., ACC Why, where κ moves from −0.50 to −0.09/0.10, and ACC What, where κ moves from 0.02 to 0.34). The abstract and conclusion will be adjusted to avoid overstating the agreement improvement. revision: yes

  4. Referee: §IV-C: The training sets are extremely small (32 ACC, 42 CERN consensus defeaters). The manual transfer of 1–2 dissensus defeaters into the consensus set 'for class balance' constitutes post-hoc data manipulation that is not adequately justified. The paper should explain the selection criteria for these transferred defeaters and report sensitivity of results to their inclusion/exclusion.

    Authors: We agree that the selection criteria for the transferred defeaters are not adequately documented and that sensitivity analysis is needed. We will revise §IV-C to specify how the 1–2 dissensus defeaters were selected (they were chosen as the dissensus cases closest to consensus—i.e., where H1 and H2 differed by only one rating level on one component—and where their inclusion improved class coverage for underrepresented labels). We will also run and report results with these defeaters excluded, so readers can assess sensitivity. We disagree with the term 'data manipulation' insofar as the transfer was a documented preprocessing step applied before any model training or evaluation, not a post-hoc adjustment made after observing results. However, we acknowledge that without sensitivity analysis, the reader cannot verify that the transfer did not materially affect outcomes, and we will provide that analysis. revision: yes

Circularity Check

0 steps flagged

No significant circularity found; the derivation is standard supervised learning with minor non-load-bearing self-citations.

full rationale

The paper's central derivation chain is: (1) two expert raters (H1, H2) label defeaters; (2) consensus defeaters (H1=H2) form the training set; (3) dissensus defeaters (H1≠H2) form the evaluation set; (4) classifiers are trained on consensus labels and evaluated on dissensus data via F1 and Cohen's kappa. This is standard supervised learning—train on one subset, test on a different subset—and is not circular by construction. The F1=0.84 claim (Table II) is computed on the held-out dissensus set, not on the training data. The kappa improvement claim (Table V) compares model-vs-rater agreement to rater-vs-rater agreement on the same dissensus set; while the skeptic correctly notes the resulting kappa values remain near zero (a correctness/methodology concern), the model was not trained to maximize kappa with either rater, so the claim is not forced by construction. Self-citations exist ([1] Ratiu et al., [4] Rohlinger) but are peripheral—referencing prior work on safety argument patterns and runtime assurance—and do not bear the central methodological or empirical claims. The dataset originates from Viger et al. [7], an external group. The undefined 'general match' metric and the class-2 prediction bias are legitimate correctness risks but do not constitute circularity: the paper does not define the metric in terms of its own output, nor does it fit a parameter and then rename the fit as a prediction. No step in the derivation chain reduces to its own inputs by definition or by self-citation.

Axiom & Free-Parameter Ledger

7 free parameters · 4 axioms · 0 invented entities

The paper introduces no new physical entities, particles, forces, or mathematical objects. The 'quality attributes' in Table I (Correctness, Stability, Relevance, Completeness, Novelty) are reformulations of standard ML metrics (F1, cosine similarity, text parsing) into domain-specific labels, not new theoretical constructs. The free parameters are standard ML hyperparameters. The key axioms are domain assumptions about label reliability and embedding semantics, both of which are load-bearing and insufficiently validated.

free parameters (7)
  • SVM kernel = selected from {linear, rbf}
    Hyperparameter tuned via GridSearchCV with 5-fold CV (Section III-D4)
  • SVM C = selected from {0.1, 1, 10, 100}
    Hyperparameter tuned via GridSearchCV (Section III-D4)
  • SVM gamma = selected from {scale, 0.001, 0.01, 0.1}
    Hyperparameter tuned via GridSearchCV (Section III-D4)
  • Self-training confidence threshold = 0.9
    Chosen to control pseudo-label quality (Section III-D4); no sensitivity analysis provided
  • SMOTE k-nearest neighbors = min(3, n-1)
    Default heuristic for small minority classes (Section IV-C)
  • BERT max sequence length = 512
    Standard BERT limit, not tuned but a modeling choice
  • Random state = 42
    Fixed for reproducibility (Section IV-F)
axioms (4)
  • domain assumption Expert consensus labels (H1=H2) are a reliable ground truth for training
    The entire supervised pipeline depends on this. The paper's own kappa analysis (sometimes negative) questions this assumption, yet it is not discussed as a threat to training validity (Section IV-C).
  • domain assumption BERT embeddings capture semantic quality distinctions relevant to defeater assessment
    The paper uses mean-pooled bert-base-uncased embeddings without fine-tuning or domain adaptation (Section III-D2). No validation that these embeddings distinguish quality levels is provided.
  • ad hoc to paper Cosine similarity between defeater and assurance case element embeddings reflects 'relevance'
    Table I defines Relevance as average cosine similarity, but no evidence that high cosine similarity correlates with expert relevance judgments is given.
  • domain assumption The 0/1/2 rating scale captures meaningful quality distinctions
    Inherited from Viger et al. [7]; the paper does not validate this scale independently.

pith-pipeline@v1.1.0-glm · 17913 in / 3232 out tokens · 566657 ms · 2026-07-08T18:20:26.996680+00:00 · methodology

0 comments
read the original abstract

High-integrity systems, such as autonomous vehicle fleets and large-scale energy infrastructures, rely on structured assurance cases to justify safety claims. To remain valid under evolving operational conditions, such cases must be examined against potential challenges, known as defeaters. While large language models (LLMs) can support the scalable generation of candidate defeaters, assessing their quality remains largely manual and subjective process. This paper presents an automated approach for supporting the assessment of LLM-generated defeaters using natural language processing techniques. The method combines structural features from assurance case graphs with semantic embeddings and meta-classifiers trained on expert-assessed defeater annotations. We evaluate the approach through two case studies in the automotive and energy domains. The results show substantial human reviewer dissensus, with Cohen's kappa values below 0.442, highlighting the difficulty of consistent manual assessment. Against this background, the proposed classifiers achieve an average F1-score of 0.84 in validation and show improved alignment with individual expert ratings. The findings suggest that automated assessment can help reduce subjective variance and provide scalable decision support for assurance case review, while leaving final judgment to domain experts.

Figures

Figures reproduced from arXiv: 2607.06039 by Daniel Ratiu, Stefan Wagner, Tihomir Rohlinger.

Figure 1
Figure 1. Figure 1: This is an assurance case graph in a goal-structured notation style [10], showing goals and sub-components as nodes (left). The diagram illustrates the manual reevaluation process of the generated defeaters and proposed automated AI assessment to mitigate subjective bias (right). We evaluated the approach on two assurance case studies (background), the Adaptive Cruise Control (ACC) (left) from [7] and Larg… view at source ↗
Figure 2
Figure 2. Figure 2: Automated review process and classifier integration [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

36 extracted references · 36 canonical work pages · 4 internal anchors

  1. [1]

    Towards an argument pattern for the use of safety performance indicators,

    D. Ratiu, T. Rohlinger, T. Stolte, and S. Wagner, “Towards an argument pattern for the use of safety performance indicators,” inComputer Safety, Reliability, and Security. SAFECOMP 2024 Workshops, A. Ceccarelli, M. Trapp, A. Bondavalli, E. Schoitsch, B. Gallina, and F. Bitsch, Eds. Springer Nature Switzerland, pp. 160–172

    work page 2024

  2. [2]

    Koopman,How Safe Is Safe Enough? Measuring and Predicting Autonomous Vehicle Safety

    P. Koopman,How Safe Is Safe Enough? Measuring and Predicting Autonomous Vehicle Safety. Pittsburgh, PA, USA: Carnegie Mellon University, 2022

    work page 2022

  3. [3]

    Dynamic assurance cases: A pathway to trusted autonomy,

    E. Asaadi, E. Denney, J. Menzies, G. J. Pai, and D. Petroff, “Dynamic assurance cases: A pathway to trusted autonomy,”Computer, vol. 53, no. 12, pp. 35–46, 2020

    work page 2020

  4. [4]

    Automated interpretation of fleet incidents to enable system level runtime assurance,

    R. Tihomir, “Automated interpretation of fleet incidents to enable system level runtime assurance,” in35th IEEE International Symposium on Software Reliability Engineering, ISSRE 2024 - Workshops, Tsukuba, Japan, October 28-31, 2024, 2024, pp. 91–94

    work page 2024

  5. [5]

    Defeaters and Eliminative Argumentation in Assurance 2.0

    R. Bloomfield, K. Netkachova, and J. Rushby, “Defeaters and eliminative argumentation in Assurance 2.0,” Computer Science Laboratory, SRI International, Menlo Park, CA, Tech. Rep. SRI-CSL-2024-01, May 2024, additional arxiv 2405.15800

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  6. [6]

    The interpretation and evaluation of assurance cases,

    J. Rushby, “The interpretation and evaluation of assurance cases,” inProc. Comput. Sci.San Francisco, CA, USA: Elsevier, 2015. [Online]. Available: https://www.semanticscholar. org/paper/The-Interpretation-and-Evaluation-of-Assurance-Rushby/ 5196fbfbc98306ad2273fc61d420af8ce1452fe1

    work page 2015

  7. [7]

    AI-supported eliminative argumentation: Practical experience generating defeaters to increase confidence in assurance cases,

    T. Viger, L. Murphy, S. Diemert, C. Menghi, J. Joyce, A. Di Sandro, and M. Chechik, “AI-supported eliminative argumentation: Practical experience generating defeaters to increase confidence in assurance cases,” in2024 IEEE 35th International Symposium on Software Reliability Engineering (ISSRE), 2024, pp. 284–294, ISSN: 2332-6549. [Online]. Available: htt...

    work page arXiv 2024

  8. [8]

    B. S. Everitt and D. C. Howell,Encyclopedia of statistics in behavioral science. Chichester, UK: John Wiley & Sons, 2005

    work page 2005

  9. [9]

    BERT: Pre-training of deep bidirectional transformers for language understanding,

    J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “BERT: Pre-training of deep bidirectional transformers for language understanding,” in Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), J. Burstein, C. Doran, and T. Solorio, Eds...

    work page 2019

  10. [10]

    The goal structuring notation–a safety argu- ment notation,

    T. Kelly and R. Weaver, “The goal structuring notation–a safety argu- ment notation,”Proc Dependable Syst Networks Workshop Assurance Cases, 01 2004

    work page 2004

  11. [11]

    Assurance case arguments in the large: The CERN LHC machine protection system,

    L. Millet, S. Diemert, C. Rees, T. Viger, M. Chechik, C. Menghi, and J. Joyce, “Assurance case arguments in the large: The CERN LHC machine protection system,” inComputer Safety, Reliability, and Security, J. Guiochet, S. Tonetta, and F. Bitsch, Eds. Springer Nature Switzerland, pp. 3–10

    work page

  12. [12]

    GPT-4 and Safety Case Generation: An Exploratory Analysis

    M. Sivakumar, A. B. Belle, J. Shan, and K. K. Shahandashti, “Gpt-4 and safety case generation: An exploratory analysis,” 2023, arXiv:2312.05696. [Online]. Available: http://arxiv.org/abs/2312.05696

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  13. [13]

    Prompting gpt-4 to support automatic safety case generation,

    ——, “Prompting gpt-4 to support automatic safety case generation,” Expert Syst. Appl., vol. 255, p. 124653, 2024. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S0957417424015203

    work page 2024

  14. [14]

    Safesens - uncertainty quan- tification of complex perception systems,

    I. Kurzidem, S. Burton, and P. Schleiss, “Safesens - uncertainty quan- tification of complex perception systems,” in2023 IEEE 26th Int. Conf. Intell. Transp. Syst. (ITSC). IEEE, 2023, pp. 5805–5810

    work page 2023

  15. [15]

    CoDefeater: Using LLMs to find defeaters in assurance cases,

    U. Gohar, M. C. Hunter, R. R. Lutz, and M. B. Cohen, “CoDefeater: Using LLMs to find defeaters in assurance cases,” inProceedings of the 39th IEEE/ACM International Conference on Automated Software Engineering, ser. ASE ’24. Association for Computing Machinery, pp. 2262–2267. [Online]. Available: https://dl.acm.org/doi/10.1145/3691620.3695296

    work page doi:10.1145/3691620.3695296

  16. [16]

    Assessing the impact of GPT-4 turbo in generating defeaters for assurance cases,

    K. Khakzad Shahandashti, M. Sivakumar, M. M. Mohajer, A. Boaye Belle, S. Wang, and T. Lethbridge, “Assessing the impact of GPT-4 turbo in generating defeaters for assurance cases,” in Proceedings of the 2024 IEEE/ACM First International Conference on AI Foundation Models and Software Engineering. ACM, pp. 52–56. [Online]. Available: https://dl.acm.org/doi...

    work page doi:10.1145/3650105.3652291 2024

  17. [17]

    Examining proposed uses of llms to produce or assess assurance arguments,

    M. S. Graydon and S. M. Lehman, “Examining proposed uses of llms to produce or assess assurance arguments,” NASA Peer Committee, Washington, DC, USA, Technical Report 20250001849, 2025. [Online]. Available: https://ntrs.nasa.gov/citations/20250001849

    work page arXiv 2025

  18. [18]

    Eliminative argumentation: A basis for arguing confidence in system properties,

    J. B. Goodenough, C. B. Weinstock, and A. Z. Klein, “Eliminative argumentation: A basis for arguing confidence in system properties,” Carnegie Mellon University, Pittsburgh, PA, USA, Technical Report CMU/SEI-2015-TR-005, 2015. [Online]. Available: https://insights.sei. cmu.edu/documents/1248/2015 005 001 434813.pdf

    work page 2015

  19. [19]

    Attention is all you need,

    A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin, “Attention is all you need,” in Advances in Neural Information Processing Systems, vol. 30. Curran Associates, Inc. [Online]. Available: https://papers.nips.cc/paper files/ paper/2017/hash/3f5ee243547dee91fbd053c1c4a845aa-Abstract.html

    work page 2017

  20. [20]

    Recent advances in named entity recognition: A comprehensive survey and comparative study,

    I. Keraghel, S. Morbieu, and M. Nadif, “Recent advances in named entity recognition: A comprehensive survey and comparative study,”

    work page

  21. [21]

    Available: https://api.semanticscholar.org/CorpusID: 267060999

    [Online]. Available: https://api.semanticscholar.org/CorpusID: 267060999

    work page

  22. [22]

    On the use of GPT-4 for creating goal models: An exploratory study,

    B. Chen, K. Chen, S. Hassani, Y . Yang, D. Amyot, L. Lessard, G. Mussbacher, M. Sabetzadeh, and D. Varr ´o, “On the use of GPT-4 for creating goal models: An exploratory study,” in 2023 IEEE 31st International Requirements Engineering Conference Workshops (REW). IEEE, pp. 262–271. [Online]. Available: https: //ieeexplore.ieee.org/document/10260905/

    work page arXiv 2023

  23. [23]

    Baldur: Whole-proof generation and repair with large language models,

    E. First, M. N. Rabe, T. Ringer, and Y . Brun, “Baldur: Whole-proof generation and repair with large language models,” inProceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering, ser. ESEC/FSE

    work page

  24. [24]

    1229–1241

    Association for Computing Machinery, pp. 1229–1241. [Online]. Available: https://dl.acm.org/doi/10.1145/3611643.3616243

    work page doi:10.1145/3611643.3616243

  25. [25]

    Automatic instantiation of assurance cases from patterns using large language models,

    O. Odu, A. B. Belle, S. Wang, S. Kpodjedo, T. C. Lethbridge, and H. Hemmati, “Automatic instantiation of assurance cases from patterns using large language models,”J. Syst. Softw., vol. 222, p. 112353,

    work page

  26. [26]

    Available: https://api.semanticscholar.org/CorpusID: 275911948

    [Online]. Available: https://api.semanticscholar.org/CorpusID: 275911948

    work page

  27. [27]

    Enabling theory- based continuous assurance: A coherent approach with semantics and automated synthesis,

    S. Varadarajan, R. Bloomfield, J. Rushby, G. Gupta, A. Murugesan, R. Stroud, K. Netkachova, I. H. Wong, and J. Arias, “Enabling theory- based continuous assurance: A coherent approach with semantics and automated synthesis,” inComputer Safety, Reliability, and Security. SAFECOMP 2024 Workshops, A. Ceccarelli, M. Trapp, A. Bondavalli, E. Schoitsch, B. Gall...

    work page 2024

  28. [28]

    26262-1:2018 road vehicles – functional safety – part 1: V ocabulary,

    ISO, “26262-1:2018 road vehicles – functional safety – part 1: V ocabulary,” Geneva, Switzerland, Tech. Rep., 2018, standard. [Online]. Available: https://www.iso.org/standard/68383.html

    work page 2018

  29. [29]

    21448:2022 Road vehicles – Safety of the intended functionality (SOTIF),

    ——, “21448:2022 Road vehicles – Safety of the intended functionality (SOTIF),” Geneva, Switzerland, Tech. Rep., 2022, standard. [Online]. Available: https://www.iso.org/standard/77490.html

    work page 2022

  30. [30]

    4600:2023 standard for safety for the evaluation of autonomous products,

    ANSI/UL, “4600:2023 standard for safety for the evaluation of autonomous products,” Northbrook, IL, USA, Tech. Rep., 2023, standard, 3rd ed. [Online]. Available: https://www.shopulstandards.com/ ProductDetail.aspx?productId=UL4600 3 S 20230317

    work page 2023

  31. [31]

    Using ChatGPT for Thematic Analysis

    A. Turobov, D. Coyle, and V . Harding, “Using chatgpt for thematic analysis,” 2024, arXiv:2405.08828. [Online]. Available: http://arxiv.org/abs/2405.08828

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  32. [32]

    Ai-ea implementation and data,

    “Ai-ea implementation and data,” https://zenodo.org/records/13368055, accessed: 2025-07-10

    work page arXiv 2025

  33. [33]

    Scikit-learn: Machine learning in Python,

    F. Pedregosa, G. Varoquaux, A. Gramfort, V . Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V . Dubourg, J. Vander- plas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duch- esnay, “Scikit-learn: Machine learning in Python,”Journal of Machine Learning Research, vol. 12, pp. 2825–2830, 2011

    work page 2011

  34. [34]

    BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

    J. Devlin, M. Chang, K. Lee, and K. Toutanova, “BERT: pre-training of deep bidirectional transformers for language understanding,”CoRR, vol. abs/1810.04805, 2018. [Online]. Available: http://arxiv.org/abs/ 1810.04805

    work page internal anchor Pith review Pith/arXiv arXiv 2018

  35. [35]

    Smote: Synthetic minority over-sampling technique,

    N. Chawla, K. Bowyer, L. Hall, and W. Kegelmeyer, “Smote: Synthetic minority over-sampling technique,”J. Artif. Intell. Res. (JAIR), vol. 16, pp. 321–357, 06 2002

    work page 2002

  36. [36]

    Unsupervised word sense disambiguation rivaling supervised methods,

    D. Yarowsky, “Unsupervised word sense disambiguation rivaling supervised methods,” in33rd Annual Meeting of the Association for Computational Linguistics. Cambridge, Massachusetts, USA: Association for Computational Linguistics, Jun. 1995, pp. 189–196. [Online]. Available: https://aclanthology.org/P95-1026/

    work page 1995