Pith. sign in

REVIEW 3 major objections 4 minor 43 references

Automating AI Failure Tracking: Semantic Association of Reports in AI Incident Database

T0 review · 3 major / 4 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper claims that associating a new AI failure report with the correct previously documented incident can be automated by ranking incidents with sentence-embedding cosine similarity, and that combining the report's title and…

desk verdict Useful task and honest limitations, but the headline numbers rely on a random report-level split that likely measures near-duplicate retrieval, so the central claims need an incident-disjoint split before they hold. read the letter →

arxiv 2507.23669 v1 pith:UQFO5Q52 submitted 2025-07-31 cs.CY cs.AIcs.IR

Diego Russo , Gian Marco Orlando , Valerio La Gatta , Vincenzo Moscato This is my paper
classification cs.CYcs.AIcs.IR
keywords AIIncidentDatabasesemanticretrievalsentenceembeddingsreportmatchingrankingsafetytransformermodelscuration
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The AI Incident Database (AIID) holds thousands of reports of real-world AI failures, but matching each new report to the incident it describes is done by human editors, which is slow and hard to scale. The paper tries to automate that match by treating it as a ranking problem: embed a new report and every known incident with a sentence transformer, score their cosine similarity, and return the top-ranked incidents. On a random 75/12.5/12.5 split of 3,805 reports across 815 incidents, the best model (multi-qa-MiniLM-L6-cos-v1) reaches Accuracy@3 of 0.982 when the report title and description are combined, up from 0.772 with title alone. The authors read this as evidence that manual AIID curation can be replaced by a scalable semantic-retrieval assistant.

What carries the argument

The load-bearing object is the sentence-transformer dual encoder: report texts and incident texts are mapped into the same dense vector space, and cosine similarity between a report vector and each incident vector produces the ranking. The pipeline concatenates title and description, cleans and stopword-filters the text, embeds both sides, then returns a ranked incident list. The authors fine-tune three sentence transformers and select multi-qa-MiniLM-L6-cos-v1 as the best configuration for answering their three research questions.

What would settle it

Run the same experiments with an incident-aware split that keeps all reports of an incident in one partition, or a purely temporal split by incident date, and compare Accuracy@3; a large drop from 0.982 would show the reported number depends on near-duplicate overlap between training and test.

Watch

Extended reading notes

Core claim

The central claim is that report-to-incident association in the AIID is a semantic similarity ranking task and that fine-tuned sentence-embedding models solve it almost perfectly. On the test split, multi-qa-MiniLM-L6-cos-v1 achieves Accuracy@3 of 0.982 ± 0.006, MRR@3 of 0.963 ± 0.010, and NDCG@3 of 0.968 ± 0.008 using title and description together, and it beats BM25, BM25+T5, and cross-encoders at every cutoff. The paper further claims performance is stable across description lengths and improves monotonically as the training set grows, which would make the retriever a practical tool for database maintenance.

Load-bearing premise

The reported accuracy depends on a random split that does not keep all reports of the same incident together; if the same event's text appears in both training and testing, the model may be rewarded for recognizing near-duplicates rather than generalizing to genuinely new incidents.

Editorial extensions

If this is right

  • Curators of the AIID could be given a short ranked candidate list per new report, shrinking a manual database-wide scan to a few confirmations.
  • Because title plus description beats title alone by 15–25 points, any deployed matcher should ingest full report text rather than headlines.
  • The monotone improvement with training-fold size means the retriever can be re-fine-tuned as the database grows without redesigning the pipeline.
  • The method is lightweight enough to run on a single workstation with one GPU, making it feasible as a real-time assistant during report submission.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If reports of one incident bleed across the random train/test split, the 0.982 Accuracy@3 likely measures near-duplicate detection; an incident-grouped split is the natural stress test.
  • A production tool would need a rejection threshold below which no incident is suggested, and the paper does not derive such a threshold from the reported scores.
  • The same ranking formulation transfers to other multi-report incident repositories, such as aviation safety or vulnerability databases, if their ground-truth incident groupings are exposed.
  • Ignored metadata like harm type or system function could be used as a hard pre-filter, which would likely raise precision further at no embedding cost.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper proposes a retrieval-based framework to automate the association of new reports in the AI Incident Database (AIID) with existing AI Incidents. The task is formalized as ranking candidate incidents by semantic similarity between report text (title and description) and incident text. The authors benchmark BM25, BM25+T5, cross-encoders, and sentence transformer models on a snapshot of 3,805 reports and 815 incidents, and report that sentence transformers, especially multi-qa-MiniLM-L6-cos-v1, outperform all baselines, with Accuracy@3 reaching 0.982 when titles and descriptions are combined. They also analyze robustness to description length and the effect of training-set size, concluding that retrieval performance improves consistently as training data grows.

Significance. If the reported results are valid, the paper offers a practical tool for AIID curators and a useful benchmark for report-to-incident retrieval. The task is well motivated, the dataset is public and clearly described, and the evaluation uses standard retrieval metrics with error bars over multiple runs. The manuscript also commits to releasing code upon acceptance. However, the central quantitative claims rest entirely on the soundness of the evaluation split, and the current split does not support the stated generalization claims; the practical significance is therefore conditional on a corrected experimental protocol.

major comments (3)
  1. [Section 4.1, with Section 3.1 and Table 1] The random 75/12.5/12.5 split is performed at the report level, but Section 3.1 explicitly states that a single AI Incident may be associated with multiple reports and the dataset contains 3,805 reports for 815 incidents. Because the retrieval target is the incident, test reports from an incident whose other reports appear in the training set are near-duplicates of training content. Consequently, the reported Accuracy@3 of 0.982 likely measures near-duplicate retrieval rather than generalization to previously unseen incidents. The evaluation should be rerun with an incident-disjoint split, or with a temporal split that ensures no incident spans both training and test sets.
  2. [Section 4.5, Table 5] The claim of consistent improvement with training data is contradicted by Table 5: Fold 3 is worse than Fold 2 for every metric (e.g., Accuracy@3 drops from 0.940 to 0.883, MRR@3 from 0.878 to 0.808, and NDCG@3 from 0.894 to 0.827), and Accuracy@10 for Fold 5 equals Fold 1. In addition, Figure 2 shows that only the training folds are chronologically ordered; the validation and test sets are fixed but inherit the same random report-level split, so the same-incident leakage remains and RQ3 does not isolate generalization to genuinely new incidents.
  3. [Abstract and Sections 4.2-4.4] The headline conclusions—that sentence transformers consistently outperform all other approaches and that combining title and description yields large gains—are based entirely on the leaked split. Since every test report may share its target incident with training reports, the quantitative claims in the abstract and in RQ1 cannot be interpreted as estimates of deployment performance for linking a new report to a previously unobserved incident. The authors should re-evaluate all three research questions under an incident-disjoint protocol and report whether the relative ordering of models and the reported gains persist.
minor comments (4)
  1. [Abstract and Introduction] The phrase 'mitigate these risk' should read 'mitigate these risks'; a similar grammar issue appears in the Introduction ('systematically study and mitigate these risk').
  2. [References] References [28] and [29] are duplicate entries for the same paper (Nogueira et al., 'Document ranking with a pretrained sequence-to-sequence model'); one should be removed and the citations consolidated.
  3. [Section 4.1] The sentence 'the test set contains 475 variants' should say '475 reports' (12.5% of 3,805), since 'variant' is a distinct AIID concept introduced in Section 2.3.
  4. [Figure 2 and Section 4.5] The text should clarify whether the validation and test sets in the progressive protocol are fixed from the original random split or are temporal holdouts; as written, the terms 'temporal' and 'chronologically ordered' apply only to the training folds.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation: this is an empirical retrieval benchmark whose reported results are measurements on a held-out test set, not consequences of its own definitions.

full rationale

The paper does not contain a derivation chain that reduces to its inputs. It formalizes report-to-incident association as a semantic ranking task (Sections 3.2 and 4.1), fine-tunes standard sentence encoders on a random 75/12.5/12.5 split, and reports held-out test metrics in Tables 2–5. The ground-truth links come from the AI Incident Database's editorial structure, not from the model itself, and no fitted parameter is renamed as a prediction. References [23,24] are to McGregor and co-authors, not to the present authors, so there is no load-bearing self-citation chain, no imported uniqueness theorem, and no ansatz smuggled in via citation. The closest concern is a validity issue: Section 3.1 states that a single incident may have multiple reports, while Section 4.1 splits reports without grouping by incident, so test reports can share their target incident with training reports. This is a potential generalization and test-construction limitation, not a circularity step, because the reported accuracy is an empirical measurement rather than an identity imposed by construction. The Conclusion also explicitly acknowledges the pairwise-only evaluation scope. The apparent monotonicity error in Table 5 (Fold 3 drops below Fold 2) is a factual/consistency issue, not a circularity issue. Overall, the empirical claims are self-contained against an external dataset and do not reduce to their own definitions.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

No free parameters or invented entities are introduced. The central claim rests on the quality of existing editorial labels and on the absence of leakage, neither of which is independently verified. The third axiom is the most fragile: if reports of the same incident appear in both training and test, the reported accuracy is inflated.

assumptions (3)
  • domain assumption The AIID ground-truth links between reports and incidents are correct and exhaustive.
    The paper uses these links as training labels and evaluation ground truth in Section 3.1, without verifying the quality or completeness of editorial assignments.
  • domain assumption Semantic similarity of title and description text is a sufficient signal for report-to-incident association.
    The entire pipeline in Section 3.2 is built on this assumption. The authors themselves acknowledge in the conclusions that causal and structural cues are ignored.
  • ad hoc to paper The random 75/12.5/12.5 train/validation/test split does not create leakage across reports of the same incident.
    Section 4.1 describes the split without incident-level grouping. The paper's high accuracy depends on this assumption, which is not demonstrated and is likely false.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Automating AI Failure Tracking: Semantic Association of Reports in AI Incident Database." pith.science (2026). https://pith.science/paper/UQFO5Q52

@misc{pith2026250723669,
  author       = {Pith},
  title        = {Pith review of: Automating AI Failure Tracking: Semantic Association of Reports in AI Incident Database},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/UQFO5Q52}},
  note         = {Machine review of arXiv:2507.23669}
}
read the original abstract

Artificial Intelligence (AI) systems are transforming critical sectors such as healthcare, finance, and transportation, enhancing operational efficiency and decision-making processes. However, their deployment in high-stakes domains has exposed vulnerabilities that can result in significant societal harm. To systematically study and mitigate these risk, initiatives like the AI Incident Database (AIID) have emerged, cataloging over 3,000 real-world AI failure reports. Currently, associating a new report with the appropriate AI Incident relies on manual expert intervention, limiting scalability and delaying the identification of emerging failure patterns. To address this limitation, we propose a retrieval-based framework that automates the association of new reports with existing AI Incidents through semantic similarity modeling. We formalize the task as a ranking problem, where each report-comprising a title and a full textual description-is compared to previously documented AI Incidents based on embedding cosine similarity. Benchmarking traditional lexical methods, cross-encoder architectures, and transformer-based sentence embedding models, we find that the latter consistently achieve superior performance. Our analysis further shows that combining titles and descriptions yields substantial improvements in ranking accuracy compared to using titles alone. Moreover, retrieval performance remains stable across variations in description length, highlighting the robustness of the framework. Finally, we find that retrieval performance consistently improves as the training set expands. Our approach provides a scalable and efficient solution for supporting the maintenance of the AIID.

Figures

Figures reproduced from arXiv: 2507.23669 by the authors.

Figure 1
Figure 1. Overview of the semantic retrieval pipeline for linking new reports to existing [PITH_FULL_IMAGE:figures/full_fig_p005_1.png] view at source ↗
Figure 2
Figure 2. Evaluation protocol for assessing the impact of training data scale. [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

43 extracted references · 35 canonical work pages

  1. [1]

    W. A. Addy, A. O. Ajayi-Nifise, B. G. Bello, S. T. Tula, O. Odeyemi, and T. Falaiye. Ai in credit scoring: A comprehensive review of models and predictive analytics.Global Journal of Engineering and Technology Advances, 18(02):118–129, 2024

    work page 2024

  2. [2]

    real attackers don’t compute gradients

    G. Apruzzese, H. S. Anderson, S. Dambra, D. Freeman, F. Pierazzi, and K. Roundy. “real attackers don’t compute gradients”: bridging the gap between adversarial ml research and practice. In 2023 IEEE conference on secure and trustworthy machine learning (SaTML), pages 339–364. IEEE, 2023

    work page 2023

  3. [3]

    O. A. Bello, A. Ogundipe, D. Mohammed, F. Adebola, and O. A. Alonge. Ai-driven approaches for real-time fraud detection in us fi- nancial transactions: challenges and opportunities. European Journal of Computer Science and Information Technology, 11(6):84–102, 2023

    work page 2023

  4. [4]

    D. M. Blei, A. Y . Ng, and M. I. Jordan. Latent dirichlet allocation. Journal of machine Learning research, 3(Jan):993–1022, 2003

    work page 2003

  5. [5]

    Bondi-Kelly, T

    E. Bondi-Kelly, T. Hartvigsen, L. M. Sanneman, S. Sankaranarayanan, Z. Harned, G. Wickerson, J. W. Gichoya, L. Oakden-Rayner, L. A. Celi, M. P. Lungren, et al. Taking off with ai: lessons from aviation for health- care. In Proceedings of the 3rd ACM Conference on Equity and Access in Algorithms, Mechanisms, and Optimization, pages 1–14, 2023

    work page 2023

  6. [6]

    A. K. Chaurasia, C. J. Greatbatch, and A. W. Hewitt. Diagnostic accu- racy of artificial intelligence in glaucoma screening and clinical prac- tice. Journal of Glaucoma, 31(5):285–299, 2022

    work page 2022

  7. [7]

    Chopra, D

    H. Chopra, D. K. Shin, K. Munjal, K. Dhama, T. B. Emran, et al. Rev- olutionizing clinical trials: the role of ai in accelerating medical break- throughs. International Journal of Surgery, 109(12):4211–4220, 2023

    work page 2023

  8. [8]

    G. Cohen. Algorithmic trading and financial forecasting using advanced artificial intelligence methodologies. Mathematics, 10(18):3302, 2022

    work page 2022

Show all 43 references
  1. [9]

    Dai and J

    Z. Dai and J. Callan. Context-aware term weighting for first stage passage retrieval. In Proceedings of the 43rd International ACM SI- GIR conference on research and development in Information Retrieval, pages 1533–1536, 2020

    2020

  2. [10]

    Dartmann, A

    G. Dartmann, A. Schmeink, V . Lücken, H. Song, M. Ziefle, and G. Pres- tiflippo. Smart transportation: AI enabled mobility and autonomous driving. CRC Press, 2021

    2021

  3. [11]

    Diaconescu and V .-E

    P. Diaconescu and V .-E. Neagoe. Credit scoring using deep learning driven by optimization algorithms. In 2020 12th International Con- ference on Electronics, Computers and Artificial Intelligence (ECAI) , pages 1–6. IEEE, 2020

    2020

  4. [12]

    Feffer, N

    M. Feffer, N. Martelaro, and H. Heidari. The ai incident database as an educational tool to raise awareness of ai harms: A classroom exploration of efficacy, limitations, & future improvements. In Proceedings of the 3rd ACM Conference on Equity and Access in Algorithms, Mechan...

    2023

  5. [13]

    G. W. Furnas, T. K. Landauer, L. M. Gomez, and S. T. Dumais. The vo- cabulary problem in human-system communication. Communications of the ACM, 30(11):964–971, 1987

    1987

  6. [14]

    Gillespie

    T. Gillespie. Content moderation, ai, and the question of scale. Big Data & Society, 7(2):2053951720943234, 2020

    2020

  7. [15]

    J. Guo, Y . Cai, Y . Fan, F. Sun, R. Zhang, and X. Cheng. Semantic mod- els for the first-stage retrieval: A comprehensive review.ACM Transac- tions on Information Systems (TOIS), 40(4):1–42, 2022

    2022

  8. [16]

    Hurley and J

    M. Hurley and J. Adebayo. Credit scoring in the era of big data. Yale JL & Tech., 18:148, 2016

    2016

  9. [17]

    Karpukhin, B

    V . Karpukhin, B. Oguz, S. Min, P. S. Lewis, L. Wu, S. Edunov, D. Chen, and W.-t. Yih. Dense passage retrieval for open-domain question an- swering. In EMNLP (1), pages 6769–6781, 2020

    2020

  10. [18]

    S. E. Kase, C. P. Hung, T. Krayzman, J. Z. Hare, B. C. Rinderspacher, and S. M. Su. The future of collaborative human-artificial intelligence decision-making for mission planning. Frontiers in Psychology , 13: 850628, 2022

    2022

  11. [19]

    Khattab and M

    O. Khattab and M. Zaharia. Colbert: Efficient and effective passage search via contextualized late interaction over bert. In Proceedings of the 43rd International ACM SIGIR conference on research and devel- opment in Information Retrieval, pages 39–48, 2020

    2020

  12. [20]

    Lee and H

    D. Lee and H. S. Seung. Algorithms for non-negative matrix factoriza- tion. Advances in neural information processing systems, 13, 2000

    2000

  13. [21]

    M. Lu, C. Chen, and C. Eickhoff. Cross-encoder rediscovers a semantic variant of BM25. CoRR, abs/2502.04645, 2025. doi: 10.48550/ARXIV . 2502.04645. URL https://doi.org/10.48550/arXiv.2502.04645

    2025 doi

  14. [22]

    G. Lupo. Risky artificial intelligence: The role of incidents in the path to ai regulation. Law, Technology and Humans, 5(1):133–152, 2023

    2023

  15. [23]

    McGregor

    S. McGregor. Preventing repeated real world ai failures by cataloging incidents: The ai incident database. Proceedings of the AAAI Confer- ence on Artificial Intelligence , 35(17):15458–15463, May 2021. doi: 10.1609/aaai.v35i17.17817. URL https://ojs.aaai.org/index.php/AAAI/ a...

    2021 doi

  16. [24]

    McGregor, K

    S. McGregor, K. Paeth, and K. Lam. Indexing ai risks with incidents, issues, and variants, 2022. URL https://arxiv.org/abs/2211.10384

    2022 arXiv

  17. [25]

    Metzler and W

    D. Metzler and W. B. Croft. A markov random field model for term dependencies. In Proceedings of the 28th annual international ACM SI- GIR conference on Research and development in information retrieval, pages 472–479, 2005

    2005

  18. [26]

    P. Nie, Y . Zhang, X. Geng, A. Ramamurthy, L. Song, and D. Jiang. Dc- bert: Decoupling question and document for efficient contextual encod- ing. In Proceedings of the 43rd international ACM SIGIR conference on research and development in information retrieval , pages 1829–1832, 2020

    2020

  19. [27]

    Nogueira, W

    R. Nogueira, W. Yang, J. Lin, and K. Cho. Document expansion by query prediction. arXiv preprint arXiv:1904.08375, 2019

    1904 arXiv

  20. [29]

    Nogueira, Z

    R. Nogueira, Z. Jiang, and J. Lin. Document ranking with a pretrained sequence-to-sequence model. arXiv preprint arXiv:2003.06713, 2020

    2003 arXiv

  21. [30]

    Paeth, D

    K. Paeth, D. Atherton, N. Pittaras, H. Frase, and S. McGregor. Lessons for editors of ai incidents from the ai incident database, 2024. URL https://arxiv.org/abs/2409.16425

    2024 arXiv

  22. [31]

    S. R. Pfohl, A. Foryciarz, and N. H. Shah. An empirical characteri- zation of fair machine learning for clinical risk prediction. Journal of biomedical informatics, 113:103621, 2021

    2021

  23. [32]

    Reimers and I

    N. Reimers and I. Gurevych. Sentence-bert: Sentence embeddings using siamese bert-networks. arXiv preprint arXiv:1908.10084, 2019

    1908 arXiv

  24. [33]

    Reimers and I

    N. Reimers and I. Gurevych. Making monolingual sentence embed- dings multilingual using knowledge distillation. In B. Webber, T. Cohn, Y . He, and Y . Liu, editors,Proceedings of the 2020 Conference on Em- pirical Methods in Natural Language Processing, EMNLP 2020, On- line, N...

    2020 doi

  25. [34]

    C. Rigano. Using artificial intelligence to address criminal justice needs. National Institute of Justice Journal, 280(1-10):17, 2019

    2019

  26. [35]

    S. E. Robertson and H. Zaragoza. The probabilistic relevance frame- work: BM25 and beyond. Found. Trends Inf. Retr., 3(4):333–389, 2009. doi: 10.1561/1500000019. URL https://doi.org/10.1561/1500000019

    2009 doi

  27. [36]

    Rodrigues, A

    R. Rodrigues, A. Resseguier, and N. Santiago. When artificial intelli- gence fails: The emerging role of incident databases. Pub. Governance, Admin. & Fin. L. Rev., 8:17, 2023

    2023

  28. [37]

    G. Rosa, L. H. Bonifacio, V . Jeronymo, H. Q. Abonizio, M. Fadaee, R. A. Lotufo, and R. Nogueira. In defense of cross-encoders for zero- shot retrieval. CoRR, abs/2212.06121, 2022. doi: 10.48550/ARXIV . 2212.06121. URL https://doi.org/10.48550/arXiv.2212.06121

    2022 doi arXiv

  29. [38]

    K. J. Ruskin, C. Corvin, S. Rice, G. Richards, S. R. Winter, and A. Cle- bone Ruskin. Alarms, alerts, and warnings in air traffic control: An anal- ysis of reports from the aviation safety reporting system. Transporta- tion Research Interdisciplinary Perspectives, 12:100502, 2...

    2021

  30. [39]

    J. D. Schloetzer and K. Yoshinaga. Algorithmic hiring systems: Impli- cations and recommendations for organisations and policymakers. In YSEC Yearbook of Socio-Economic Constitutions 2023: Law and the Governance of Artificial Intelligence, pages 213–246. Springer, 2023

    2023

  31. [40]

    K. Song, X. Tan, T. Qin, J. Lu, and T. Liu. Mpnet: Masked and permuted pre-training for language understanding. In H. Larochelle, M. Ran- zato, R. Hadsell, M. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems 33: Annual Conference on Neural In- for...

    2020

  32. [41]

    Talmor and J

    A. Talmor and J. Berant. Multiqa: An empirical investigation of gen- eralization and transfer in reading comprehension. arXiv preprint arXiv:1905.13453, 2019

    1905 arXiv

  33. [42]

    W. Wang, F. Wei, L. Dong, H. Bao, N. Yang, and M. Zhou. Minilm: Deep self-attention distillation for task-agnostic compression of pre-trained transformers. In H. Larochelle, M. Ranzato, R. Had- sell, M. Balcan, and H. Lin, editors, Advances in Neural Informa- tion Processing S...

    2020

  34. [43]

    X. Zhao, J. Liu, Y . Zhang, Z. Yu, and B. Guo. Haiformer: Human- ai collaboration framework for disease diagnosis via doctor-enhanced transformer. In ECAI 2024, pages 1495–1502. IOS Press, 2024

    2024

  35. [365]

    URL https://doi.org/10.18653/v1/2020.emnlp-main.365

    2020 doi

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.