Pith. sign in

REVIEW 4 major objections 6 minor 4 references

The Eticas AI Risk Taxonomy: Open Infrastructure for Operationalizing AI Audits

T0 review · 4 major / 6 minor · reviewed 2026-08-02 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that an AI risk taxonomy can be more than a catalog: a named risk can be carried to a tested, graded audit finding, and it demonstrates this end to end.

desk verdict Honest, useful infrastructure paper that overreaches slightly by calling a benchmark-recycling grading exercise a 'tested instrument'; the taxonomy itself is still worth refereeing. read the letter →

arxiv 2607.02201 v3 pith:I3IRXIUL submitted 2026-07-02 cs.CY cs.AI

Gemma Galdon Clavell , Pablo Accuosto , Usman Gohar This is my paper · ORCID
classification cs.CYcs.AI
keywords AIrisktaxonomyalgorithmicauditingoperationalizationmeasurement-to-gradechainPIIleakageseveritygradingopensemanticinfrastructureagentic
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that most AI risk taxonomies fail where it matters: they name risks but do not show how to test them. It attempts to prove this can be fixed by tracing one risk, PII leakage, from the regulatory frameworks that require it, through a test design, to measured disclosure rates of 0%, 51%, and 84% on a production model under increasing adversarial pressure, and finally to a grade of E with a SYSTEMIC pattern. The authors claim this makes their taxonomy, with 70 active subcategories across 10 categories, the first openly published audit taxonomy with a documented, benchmark-validated path from concept to graded finding. If true, auditors, regulators, and clients could share one vocabulary and still disagree productively about tests and thresholds. The load-bearing move is separating the risk concept from the mechanisms by which it surfaces, so that any operationalization attaches to stable identifiers.

What carries the argument

The key machinery is the mechanism field: each subcategory (an abstract risk) carries a list of concrete mechanisms by which it surfaces, each with a stable identifier. This separates the what (the risk) from the how (the manifestations), so a test design can attach to a mechanism without redefining the risk. The companion engine is the measurement-to-grade chain — probe, check, metric value, severity band, subcategory grade with pattern flag, dimension grade — codified once in the methodology and instantiated per audit.

What would settle it

Independently re-run the disclosed three probes on the same model and benchmark, apply alternative severity bands derived from inter-auditor agreement or regulatory harm data, and check whether the 0/51/84 disclosure rates and the E grade reproduce; alternatively, run a live deployed-system audit using the same mechanism identifiers and see whether the grades diverge from the benchmark proxy.

Watch

Extended reading notes

Core claim

The central claim is that an audit taxonomy becomes infrastructure only when a named risk can be carried end to end: from concept to test, measured value, calibrated severity, and defensible grade. The paper demonstrates this with PII leakage: the same risk mapped as a formal or near target of seven external frameworks is probed on a GPT-4 model through a zero-shot baseline, a single in-context demonstration, and three reinforced demonstrations, yielding disclosure rates of 0%, 51%, and 84%. Those rates pass through mechanism-specific severity bands to severities 1, 4, and 5, and aggregate to a subcategory grade of E with a SYSTEMIC pattern flag. Around that worked example, the paper present

Load-bearing premise

The demonstrated grading rests on severity-band thresholds the paper itself calls a first-pass calibration and on treating a public benchmark's published numbers as a proxy for a live audit; if those bands or that proxy are not defensible, the claim that the taxonomy is a tested instrument collapses.

Editorial extensions

If this is right

  • A second auditor who attaches a different test to the same mechanism identifier can produce a result comparable at the conceptual level, enabling cross-provider audit comparability.
  • Regulators and clients could compare audit outputs because findings are tagged to stable concept URIs and rendered from one canonical record into multiple report formats.
  • Coverage grows by repeating a template: each newly operationalized subcategory compounds the shared calibration, and unoperationalized mechanisms remain visible as declared empty slots.
  • The taxonomy's mapping to 18 frameworks, with agentic AI treated as a first-class category, makes a structural governance gap legible: binding compliance frameworks predate agentic deployment and must be supplemented by purpose-built standards.
  • Pluggable benchmark bindings mean the validated pipeline can consume other public evaluation suites without changing the methodology.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the open-core model works, the durable value shifts from owning a risk vocabulary to owning calibration quality; this is testable by asking independent auditors to grade the same system using the same open concept and mechanism identifiers and comparing grades.
  • The disclosed 0 to 84 percent escalation suggests that a single baseline probe can badly overstate an LLM's privacy protection; a natural extension would require an adversarial ladder of probes in any audit of disclosure risks, though the paper does not frame this as general guidance.
  • The severity bands are explicitly first-pass and mechanism-specific; a natural next step is to calibrate them against inter-auditor agreement or observed harm and publish the resulting bands for other mechanisms.
  • The paper's 'first' claim is a positional claim that later work could supersede; the durable contribution is the open pattern of concept-to-graded-finding, not the priority itself.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper presents the Eticas AI Risk Taxonomy v3.0.0, an open semantic scaffold (10 categories, 21 sub-groups, 70 active subcategories, 32 established) together with a four-layer operationalization methodology that is claimed to carry a named risk from taxonomy entry to test design, measurement, severity calibration, and graded audit finding. The central worked example is PII leakage: using published DecodingTrust Privacy Scenario 2 results for GPT-4-0314 (0%, 51%, and 84% disclosure under zero-/one-/three-shot adversarial conditioning), the authors map these rates through hand-set severity bands to severities S1/S4/S5, aggregate to a subcategory grade of E with a SYSTEMIC pattern flag, and roll this up to a Privacy dimension grade of E. The paper also describes the risk/mechanism separation, mappings to 18 external frameworks in three tiers, SKOS/JSON-LD distributions, and an analysis of the agentic AI governance gap.

Significance. If the claims were fully supported, the paper would make a useful contribution: it is one of the few practitioner taxonomies that attempt to show, with a concrete example, how a risk concept can lead to a measured, graded audit finding. The risk/mechanism separation is a sound design idea with clear practical payoff; the visible-gaps principle and the three-tier mapping approach are genuinely useful for the AI-auditing community; and the publication of the conceptual layer under CC BY 4.0 with stable URIs and SKOS/JSON-LD is real open infrastructure. The paper is also unusually transparent for a practitioner contribution, explicitly acknowledging that the numbers are a public-benchmark proxy and the severity bands are a first-pass calibration. However, the paper's central validation claim — that this constitutes a 'benchmark-validated path' and that the taxonomy is 'a tested instrument' — is stronger than what is actually demonstrated. The evidence shown is a mapping of external published numbers through author-defined thresholds, not an independent execution of the audit pipeline.

major comments (4)
  1. [§2.1, §2.6, §6.6; Data and Code Availability] The load-bearing claim of a 'benchmark-validated path' and 'tested instrument' is not supported by the reported evidence. The disclosure rates 0%, 51%, and 84% are quoted from DecodingTrust's published results; the paper did not run the probes. The honest-boundaries note in §2.1 confirms this: 'the numeric results here are the published DecodingTrust figures... should be read as a public-benchmark proxy.' What is exercised end to end is the step from metric values to severity to grade; the step from a named risk to a measured value is borrowed from another team's benchmark. The claim in §6.6 that this is the 'first openly published audit taxonomy accompanied by a documented, benchmark-validated path from its concepts to graded audit findings' therefore conflates reusing benchmark numbers with validating the Eticas methodology. The authors should either re-run the probes on the same or an
  2. [§2.1, severity bands] The final grade depends entirely on author-set thresholds described as a 'documented first-pass calibration.' No evidence is given for why 51% should be S4 and 84% S5 for the disclosure route, and no sensitivity analysis is reported. The grading is not robust in an obvious sense: if the disclosure-route S5 threshold were set above 84%, the three-shot probe would be S4 and the subcategory grade would drop from E to D. Because the entire 'defensible grade' claim rests on these bands, the paper should provide an external calibration basis, expert elicitation, or at least a sensitivity analysis showing how grade changes with threshold choice. Without this, the demonstrated chain is a formal mapping, not a calibrated measurement-to-grade instrument.
  3. [§2.6, validation coverage] The paper claims that 'the pipeline, from authored findings through grading, has been exercised end to end against DecodingTrust... in both operationalized dimensions' (Privacy and Bias and Fairness), and that the bias-and-fairness dimension 'exercises the dimension-level breadth-of-concern aggregation across multiple subcategories.' No bias-and-fairness measurements, severity bands, or grades are reported anywhere in the manuscript. As written, this part of the validation is unverifiable. The authors should either present the bias-and-fairness run (at least as a table of metrics and grades) or restrict the claim to the Privacy dimension.
  4. [§3.3, §7, Data and Code Availability] The 'open infrastructure' framing is qualified by a very narrow public surface: only 8 of 10 categories, 32 of 70 subcategories, 17 of 21 sub-groups, and one fully disclosed operationalized entry are public; the mechanism layer for the rest of the catalog, all severity calibrations, the methodology repository, and engagement data are proprietary. This is a legitimate open-core model, but it directly limits the reproducibility of the central validation and the extent to which third parties can build on the claimed stable mechanism identifiers for other risks. The paper should state this boundary more prominently in the abstract and in the validation section, and should temper claims that the infrastructure is 'demonstrably operable' by others beyond the one disclosed example.
minor comments (6)
  1. [Figure 1] Figure 1 is extremely dense and the small text in the mechanism and severity-band boxes is likely illegible in print. Consider splitting it into two figures (frameworks-and-mechanisms; measurement-and-grading) or providing a larger, simplified version.
  2. [§2.1] The terminology footnote appears in the middle of the worked example. Move it to the first use of 'dimension' in the text or to a footnote at the start of Section 2.
  3. [Table 3 caption] The caption says 'full framework names in the paper caption,' but the caption does not actually spell out the framework codes (EU, ISO, AIUC, etc.). Either spell them out in the caption or point readers explicitly to Section 5.1.
  4. [§2.1, measurement uncertainty] The DecodingTrust rates are reported as point values (0%, 51%, 84%) without trial counts or confidence intervals. Given that the severity bands are thresholds, reporting sample sizes or intervals would materially help the reader judge whether the band assignments are stable.
  5. [§3.3] The terms 'established' and 'emerging' are used both as maturity levels and as part of category names (e.g., 'Emerging Categories'). A consistent typographic convention (e.g., capitalizing maturity levels) would reduce ambiguity.
  6. [References] Reference formatting is inconsistent: some entries have full DOIs/arXiv identifiers and others (e.g., AIUC-1, OWASP) have only URLs. Consider standardizing.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the worked example applies the paper's own disclosed severity-band mapping to independent external benchmark numbers, and the main weakness is validation scope rather than circular derivation.

full rationale

The paper's central demonstration is a worked example in which three PII disclosure rates (0%, 51%, 84%) are taken from the external DecodingTrust benchmark, not derived from the paper's own assumptions; Section 2.1 describes them as 'the published DecodingTrust figures for GPT-4-0314.' The severity bands are explicitly a 'documented first-pass calibration,' and the aggregation from severity to grade is a stated rule ('the subcategory grade is the peak severity across the checks'). Applying a disclosed, non-fitted mapping to independent external measurements is a definitional exercise, not a circular derivation: the paper does not fit the bands to the benchmark values and then claim to predict those values. The only self-citation is the availability reference to the Eticas taxonomy itself, which is not load-bearing for any derivation. The limitations note at the end of Section 2.1 explicitly discloses that the numbers are a 'public-benchmark proxy' and that the bands are a first-pass calibration, so the paper's own transparency supports the non-circular reading. The legitimate weakness is that the probes themselves were not re-run by the authors, making the 'benchmark-validated path' claim thinner than its strongest wording suggests; however, that is an evidentiary/validation overstatement about what was tested, not circularity in the derivation chain.

Assumptions & free parameters 5 free parameters · 4 assumptions · 1 invented entities

The central claim rests on hand-set severity thresholds, a public-benchmark proxy, manual framework mappings, and the assumption that an open-core boundary with proprietary calibration still allows independent use. The external DecodingTrust data provides some independent grounding, but the grading outcome is sensitive to the authors' choices.

free parameters (5)
  • Severity band thresholds (disclosure route) = S5 ≥ 60%; S4 band containing 51%; S1 band at 0% (exact boundaries not fully disclosed)
    First-pass calibration chosen by the authors; changing these boundaries changes the severities and the final grade.
  • Severity band thresholds (memorization route) = S5 ≥ 30% extraction
    Different calibration for memorization versus disclosure, justified by a qualitative user-awareness distinction rather than an external standard.
  • Pattern flag rule = SYSTEMIC if ≥2 of ≥3 checks at severity ≥3
    Hand-set boundary that determines whether a subcategory is flagged SYSTEMIC instead of ISOLATED or FOCAL.
  • Subcategory and dimension grade aggregation = subcategory grade = peak severity; dimension grade = peak with breadth-of-concern adjustment
    Choice to use peak rather than mean; the dimension-level breadth adjustment is not fully specified.
  • Metric pooling choice = disclosure rate computed per PII type, not pooled across types
    Deliberate choice to avoid damping worst-case PII types; affects measured rates and subsequent grades.
assumptions (4)
  • domain assumption DecodingTrust Privacy Scenario 2 results for GPT-4-0314 are a valid proxy for a live PII-leakage audit.
    Used in Section 2.1 and 2.6 as the end-to-end empirical validation, but the authors did not run the probes themselves.
  • ad hoc to paper The first-pass severity bands encode a defensible severity judgment.
    Stated explicitly as a first-pass calibration in Section 2.1; no external benchmark, regulation, or consensus justifies the threshold values.
  • domain assumption Manually maintained framework mappings are accurate best-alignments.
    Section 3.6 states mappings are maintained by manual Eticas auditor review and treated as tentative; mapping choices could embed subjective alignment.
  • domain assumption The open-core boundary still permits independent reimplementation and comparability.
    Section 2.7 claims others can build on the open concepts, but the full mechanism layer and most calibrations are internal, limiting true comparability.
invented entities (1)
  • SYSTEMIC / FOCAL / ISOLATED pattern flags
    purpose: Preserve the distribution signal of severity across checks rather than relying only on the peak severity.
    Methodological constructs introduced by the paper; no external validation shows these flags improve audit defensibility or comparability.

how reviews work

0 comments
Cite this review

Pith. "Pith review of The Eticas AI Risk Taxonomy: Open Infrastructure for Operationalizing AI Audits." pith.science (2026). https://pith.science/paper/I3IRXIUL

@misc{pith2026260702201,
  author       = {Pith},
  title        = {Pith review of: The Eticas AI Risk Taxonomy: Open Infrastructure for Operationalizing AI Audits},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/I3IRXIUL}},
  note         = {Machine review of arXiv:2607.02201}
}
read the original abstract

The rapid deployment of AI systems across high-stakes domains has created urgent demand for standardized evaluation, yet the field remains fragmented across competing risk taxonomies that catalog risks without showing how an audit is executed. At least 74 AI risk taxonomies exist, and almost all stop at the catalog. The hard part of auditing is not naming a risk but operationalizing it: turning it into a test run against a real system, a measured value, a calibrated severity, and a defensible grade. This paper leads with that bridge. We present the operationalization layer Eticas has built and run, shown end to end on a single risk (PII leakage) against a public benchmark, and then the open taxonomy that makes the method scale. On GPT-4-0314, a disclosure risk that seven external frameworks require be controlled is measured at 0%, 51%, and 84% disclosure as adversarial conditioning increases, mapping through calibrated severity bands to a subcategory grade of E with a SYSTEMIC pattern. Around this example, the Eticas AI Risk Taxonomy v3.0.0 organizes 70 active subcategories across 10 categories and 21 sub-groups, with mappings to 18 external frameworks across compliance, reference, and academic tiers. Its established layer - categories, sub-groups, and the 32 established subcategories - is published under CC BY 4.0 as open semantic infrastructure with stable URIs and SKOS/JSON-LD distributions, and a worked subcategory example shows the operational layer down to its severity thresholds. The contribution is the demonstrated bridge from concept to graded finding, anchored by a clean separation of risks from the mechanisms by which they surface, and framed by an open-core model in which the conceptual scaffold is open and the methodology calibration is the practitioner layer. This is the infrastructure the AI auditing field needs: shared, open, and demonstrably operable.

Figures

Figures reproduced from arXiv: 2607.02201 by the authors.

Figure 1
Figure 1. Operationalizing one risk, end to end. PII leakage is formally mapped to seven external [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

4 extracted references · 1 linked inside Pith

  1. [1]

    Artificial Intelligence Underwriting Company (AIUC). (2025). AIUC-1: The AI agent standard. https://www.aiuc-1.com/ Autio, C., Schwartz, R., Dunietz, J., Jain, S., Stanley, M., Tabassi, E., Hall, P., & Roberts, K. (2024). Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile (NIST AI 600-1). National Institute of St...

    arXiv 2025

  2. [2024]

    Eticas. (2026). Eticas AI Risk Taxonomy, v3.0.0.https://taxonomy.eticas.ai/risk/ European Union. (2024). Regulation (EU) 2024/1689 of the European Parliament and of the Coun- cil of 13 June 2024 laying down harmonised rules on artificial intelligence (Artificial Intelligence Act). Official Journal of the European Union, L 2024/1689.https://eur-lex.europa....

    arXiv 2026

  3. [2025]

    Organisation for Economic Co-operation and Development (OECD)

    Cyberspace Administration of China. Organisation for Economic Co-operation and Development (OECD). (2024). Recommendation of the Council on Artificial Intelligence (updated May 2024). OECD Legal Instruments, OECD/LEGAL/0449.https://legalinstruments.oecd.org/en/instruments/OECD-LEGAL -0449 OWASP Foundation. (2025, December 9). OWASP Top 10 for Agentic Applications

    2024

  4. [2026]

    J., & Golpayegani, D

    OWASP GenAI Security Project.https://genai.owasp.org/resource/owasp-top-10-for-agentic -applications-for-2026/ Pandit, H. J., & Golpayegani, D. (Eds.). (2026). AI Technology Concepts (Data Privacy Vocabulary v2.3 AI Extension). W3C Data Privacy Vocabularies and Controls Community Group Final Report, February 2026.https://w3id.org/dpv/2.3/ai Raji, I. D., S...

    arXiv 2026

Pith tools

Reviewed August 2, 2026 · model on record in the stance chip above.