Pith. sign in

REVIEW 3 major objections 5 minor 28 references

Documenting Deployment with Fabric: A Repository of Real-World AI Governance

T0 review · 3 major / 5 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper introduces Fabric, a public repository of 20 deployed AI use cases, and claims that co-designed workflow diagrams reveal common patterns of human and institutional oversight.

desk verdict Fabric is a genuinely useful empirical artifact—the first workflow-level repository of deployed AI governance cases—but the headline pattern claims contain a checkable contradiction with the paper's own table and need rework before they can be cited. read the letter →

arxiv 2508.14119 v4 pith:OUZH66YI submitted 2025-08-18 cs.CY cs.AIcs.HC

Mackenzie Jorgensen , Kendall Brogle , Katherine M. Collins , Lujain Ibrahim , Arina Shah , Petra Ivanovic , Noah Broestl , Gabriel Piles
show 5 more authors
Paul Dongha Hatim Abdulhussein Adrian Weller Jillian Powers Umang Bhatt
This is my paper · ORCID
classification cs.CYcs.AIcs.HC
keywords AIgovernancehumanoversightinstitutionaldeployedsystemsworkflowdiagramssemi-structuredinterviewsco-designpublicrepository
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper is trying to show what AI governance actually looks like when systems are in operation. It releases Fabric, a public repository built from semi-structured interviews with practitioners about 20 real deployed AI systems, each accompanied by a co-designed diagram of the AI workflow and its oversight checkpoints. From these cases, the paper derives two independent axes of governance: four levels of human oversight (autonomous AI, conditionally autonomous AI, human-approved AI, and human-led with AI assistance) and five levels of institutional oversight (ad-hoc practice up to regulation). The authors argue that this workflow-level, empirical view fills a gap left by incident databases and documentation standards, which capture risks and model details but not how governance is enacted in practice. If the repository is right, researchers and practitioners gain a concrete, cross-sectoral map of governance patterns and gaps they can build on.

What carries the argument

The central object is the co-designed AI workflow diagram: a decision flowchart built together with the practitioner during the interview, showing inputs, AI components, decision points, and feedback loops. Working alongside it is a two-axis oversight taxonomy. The human oversight axis runs from autonomous AI (AI output is final) through conditionally autonomous AI, human-approved AI, to human-led with AI assistance; the institutional oversight axis runs from ad-hoc practice, organization best practice, organization policy, and industry standard to regulation. The diagrams anchor the taxonomy in concrete workflows, and the taxonomy gives the diagrams a comparative structure across sectors.

What would settle it

An independent audit of a sample of the 20 organizations that compared actual workflows, logs, or escalation behaviour against the co-designed diagrams and oversight labels would settle the central claim. If, for example, systems labelled autonomous turn out to route borderline cases to humans, or systems labelled human-approved turn out to be rubber-stamped without review, the governance patterns would not describe how oversight actually operates.

Watch

Extended reading notes

Core claim

The central claim is that a repository of 20 co-designed AI workflow diagrams, each annotated with human and institutional oversight labels, can surface common patterns in how deployed AI systems are governed. The paper reports that all four autonomous AI use cases in the sample fall under the strictest institutional oversight, regulation; that the most common human oversight pattern is human-led with AI assistance, where a person decides whether and how to use the AI output; and that conditionally autonomous systems tend to rely on the least formal institutional oversight, ad-hoc practice. These patterns are meant to be the beginning of an extendable resource, not a final verdict on governa

Load-bearing premise

The repository's patterns rest on the assumption that practitioners' descriptions of their deployed workflows and governance are truthful and complete, since the paper states it has no way to verify or audit the correctness of those claims.

Editorial extensions

If this is right

  • Researchers can compare governance structure against system risk and sector using the repository's consistent labels and diagrams.
  • The oversight levels can be added to model cards and other system documentation, making governance visible before deployment rather than after the fact.
  • The paper's questionnaire gives practitioners a practical way to self-assess their human oversight level and start conversations about institutional oversight.
  • The observed patterns become testable hypotheses: for instance, that fully autonomous deployments only appear under regulatory pressure, and that lightly governed conditional autonomy is common in lower-risk settings.
  • As more use cases are contributed, the repository could support empirical study of which governance mechanisms are associated with better deployment outcomes.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because the 20 cases were recruited through personal and professional networks with snowball sampling, the observed patterns may shift once the repository includes a more representative set of organizations; the autonomous-AI-under-regulation correlation in particular should be treated as sample-bound until then.
  • The authors state they cannot audit practitioners' claims, so the repository is best read as a record of what organizations say they do; a verification study comparing diagrams against logs, audits, or direct observation would test whether the governance patterns hold up.
  • The two-axis taxonomy could be turned into a coding scheme for existing incident databases: applying the same oversight labels to documented failures would allow a direct test of whether certain governance patterns are associated with fewer or less severe incidents.
  • Re-interviewing the same practitioners over time and diffing the workflow diagrams would reveal how governance evolves as systems are updated, procured, or scaled.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper introduces Fabric, a public repository of 20 deployed AI use cases gathered through semi-structured interviews and co-designed workflow diagrams. It defines four levels of human oversight (Autonomous AI, Conditionally Autonomous AI, Human-Approved AI, Human-Led with AI-Assistance) and five levels of institutional oversight (ad-hoc practice through regulation), assigns each use case to these categories, and reports cross-cutting 'patterns of oversight,' such as autonomous systems following regulation and conditionally autonomous systems following ad-hoc practice. The paper is explicitly qualitative and descriptive; it does not claim to measure governance effectiveness, and it discloses limits around selection bias, evolving interview questions, and reliance on practitioner self-reports.

Significance. If the classification and pattern claims hold, Fabric is a useful empirical resource: it provides workflow-level, cross-sectoral documentation of governance in real deployments, complements risk- and incident-focused repositories, and offers a taxonomy plus a practitioner questionnaire. The paper's strengths include the public repository, the co-design methodology with practitioner validation, and a candid limitations statement. The main weakness is internal consistency: a headline pattern in §5.1 is contradicted by the paper's own Table 2, and at least one autonomous-use-case classification is questionable. Because the pattern claims are the paper's central empirical contribution, these issues must be fixed before publication.

major comments (3)
  1. [§5.1, Table 2] The bullet 'All conditionally autonomous AI use cases, except for the more high-risk financial use case, follow the least strict level of institutional oversight: ad-hoc practice' is contradicted by Table 2. OriginTrail DKG (No. 17) is listed with ad-hoc practice and organization policy; Call Center Virtual Assistant (No. 19) is listed with ad-hoc practice, organization best practice, organization policy, industry standard, and regulation. Only Personalized Feedback Assessor (No. 4) has ad-hoc practice alone. The sentence is only salvageable by reading 'follow' as 'include at least,' which is not what it says and would trivialize the claim. Since this bullet is a headline 'pattern of oversight,' the pattern analysis is not currently supported by the published repository.
  2. [Appendix B, No. 13 and §4.3] The Mental Health Triage Tool No. 13 is classified as Autonomous AI, but its output description states 'The report is reviewed by the healthcare provider with the patient.' Under Table 1's definition, Autonomous AI means 'AI output is the final output' with no human able to change it. If a provider reviews the report with the patient, either the provider can influence the final output (making the classification at least Conditionally Autonomous), or the term 'reviewed' is being used in a non-standard way. This coding needs to be justified or corrected, because the 'all autonomous AI use cases follow regulation' pattern in §5.1 and §5.3 depends on the membership of this set.
  3. [§5.1–§5.3, overall pattern claims] The paper's central contribution is the set of 'common patterns of oversight.' Because the taxonomy was induced from and then applied to the same 20 cases, the pattern statements are descriptive, not predictive, and therefore they must be exactly consistent with Table 2. The two mismatches above mean that the repository, as currently coded, does not support the headline patterns without re-analysis or re-coding. This is fixable, but it is load-bearing rather than cosmetic.
minor comments (5)
  1. [Table 2] The checkmark placement is visually ambiguous; rows for No. 8, No. 16, and No. 20 have variable numbers of checkmarks without clear column alignment. Please use explicit column entries (e.g., '✓' or '—') for every cell.
  2. [§4.2, Regulation subsection] 'Credit Lending Classifier No. 6' appears to be a numbering error: No. 6 is Insurance Claims Classifier, while the credit lending use case is No. 7 (see §4.3 and Table 2).
  3. [§3.2/Appendix B cross-reference] The text refers to 'Figure 1 in Appendix B' for the diagram legend, but the legend in Appendix B is Figure 5; Figure 1 is the main-text example workflow. Update the cross-reference.
  4. [Appendix C, Table 4] In the first row, 'Is the AI output the final output?' appears to have a checkmark under Human-Approved AI, which conflicts with Table 1's definition that human approval is required before the output becomes final. Please clarify the questionnaire table or the checkmark alignment.
  5. [§6, miscellaneous] There are typos in the conclusion: 'part ofy Fabric' and 'thefabric of our everyday life.' Please proofread the final section.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the paper reports a qualitative repository and descriptive summaries; its data-derived taxonomy creates a mild coding limitation, not a circular derivation.

full rationale

The paper makes no predictive or first-principles derivation; it introduces a repository of 20 use cases and describes patterns observed in that repository. The only potentially circular element is that the oversight taxonomies were induced from the same 20 use cases they are then used to summarize (Section 4.1: 'on review of the AI use cases, our analysis resulted in four different levels'), so the 'common patterns' in Section 5.1 are counts of the authors' own post-hoc labels rather than external tests. This is a mild coding/taxonomic limitation, not a fitted-input prediction, because the paper explicitly disclaims novelty of the categories ('the specific governance types and levels we present are not entirely novel') and makes no out-of-sample claim. The Ethical Considerations Statement's admission that 'we have no way of verifying or auditing the correctness of a practitioner's claims' is a data-validity limitation, not circularity. The Section 5.1 pattern claim that conditionally autonomous AI use cases 'follow the least strict level of institutional oversight: ad-hoc practice' is contradicted by Table 2 (No. 17 and No. 19 have institutional oversight beyond ad-hoc practice), and the classification of Mental Health Triage Tool No. 13 as Autonomous AI is questionable given its workflow says the report 'is reviewed by the healthcare provider with the patient'; these are internal-validity/coding-consistency issues, not circular derivation. No equations, fitted parameters, or load-bearing self-citations appear. The repository's documentation value stands independently of the pattern claims.

Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

No new particles, forces, mediators, or formal entities are postulated. Fabric is a concrete repository, not an invented theoretical entity; the oversight taxonomy is a descriptive classification. The load-bearing assumptions are qualitative: self-report accuracy, sample adequacy, taxonomy validity, and diagram fidelity.

assumptions (4)
  • domain assumption Practitioners' self-reports of their AI workflows and governance are accurate and complete.
    The entire repository is built from semi-structured interviews; the paper states it has no way of verifying or auditing the claims (Ethical Considerations Statement).
  • domain assumption The convenience and snowball sample of 20 practitioners is adequate to support the reported governance patterns and gaps.
    The paper acknowledges selection bias and a small sample in Section 5.4; the patterns are descriptive rather than generalizable.
  • ad hoc to paper The four human oversight and five institutional oversight levels defined by the authors are the right categorical scheme for the workflows.
    The human oversight levels were revised during analysis after the authors expected three categories (Section 4.1); no external validation or inter-rater reliability is reported.
  • domain assumption Co-designed diagrams capture the essential workflow and governance structure of each system.
    Diagrams are produced from interviews and then used as primary data for categorization; if a diagram omits a step, the oversight level may be mislabeled (Sections 3.2, 3.3).

how reviews work

0 comments
Cite this review

Pith. "Pith review of Documenting Deployment with Fabric: A Repository of Real-World AI Governance." pith.science (2026). https://pith.science/paper/OUZH66YI

@misc{pith2026250814119,
  author       = {Pith},
  title        = {Pith review of: Documenting Deployment with Fabric: A Repository of Real-World AI Governance},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/OUZH66YI}},
  note         = {Machine review of arXiv:2508.14119}
}
read the original abstract

Artificial intelligence (AI) is increasingly integrated into society, from financial services and traffic management to creative writing. Academic literature on the deployment of AI has mostly focused on the risks and harms that result from the use of AI. We introduce Fabric, a publicly available repository of deployed AI use cases to outline their governance mechanisms. Through semi-structured interviews with practitioners, we collect an initial set of 20 AI use cases. In addition, we co-design diagrams of the AI workflow with the practitioners. We discuss the oversight mechanisms and guardrails used in practice to safeguard AI use. The Fabric repository includes visual diagrams of AI use cases and descriptions of the deployed systems. Using the repository, we surface gaps in governance and find common patterns in human oversight of deployed AI systems. We intend for Fabric to serve as an extendable, evolving tool for researchers to study the effectiveness of AI governance.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

28 extracted references · 2 linked inside Pith

  1. [1]

    System-Level Questions (a) Do you have a specific AI use case in mind for our conversation? (b) What task does the AI system complete or help complete? How do they support in this task? (c) How would you categorize the AI system (e.g., data driven, including subsymbolic AI or machine learning, or rules/logic- driven, including symbolic AI and)? (d) Does t...

  2. [2]

    Governance Questions (a) Who makes the final decision, the AI system or the user? (b) What external regulations or internal policies govern the AI system’s use? (c) How are risks and compliance assessed? (d) In deciding what mechanisms of governance to include in your system, did you consider any best practice guidance? If so, what frameworks or guidance ...

  3. [3]

    Use Case Name & Number Human Oversight Ad-Hoc Org

    Impact Questions (a) What were the intended impacts of the system? (b) What impacts has the AI system had? (c) How are the impacts measured? B Fabric Repository: Use Cases In this appendix, we (1) include a summary statistics section of all of the use cases, and (2) include the full repository. Use Case Name & Number Human Oversight Ad-Hoc Org. Best Org. ...

  4. [4]

    In Proceedings of the 2023 ACM Conference on Fairness, Accountability, and Transparency, 1369–1385

    Towards a Science of Human-AI Decision Making: An Overview of Design Space in Empirical Human-Subject Studies. In Proceedings of the 2023 ACM Conference on Fairness, Accountability, and Transparency, 1369–1385. Le, T.; Miller, T.; Singh, R.; and Sonenberg, L. 2023. Ex- plaining model confidence using counterfactuals. In Pro- ceedings of the AAAI Conferenc...

    2023

  5. [5]

    arXiv preprint arXiv:2410.09645

    AI Model Registries: A Foundational Tool for AI Governance. arXiv preprint arXiv:2410.09645. Mehrabi, N.; Morstatter, F.; Saxena, N.; Lerman, K.; and Galstyan, A. 2021. A survey on bias and fairness in machine learning. ACM computing surveys (CSUR), 54(6): 1–35. Meng, X.-L. 2025. Data Science and AI: Everything Every- where All at Once. Harvard Data Scien...

    arXiv 2021

  6. [6]

    Sector: Private Domain: Finance Task: Automotive insurance claims are summarized and reviewed by an AI system

    Insurance Claims Classifier Figure 11: Insurance Claims Classifier diagram. Sector: Private Domain: Finance Task: Automotive insurance claims are summarized and reviewed by an AI system. Intent: The AI system increases productivity and profit by reducing the number of simple claims that analysts must review. Risks: There is a risk of incorrectly denying l...

  7. [7]

    https://oecd.ai

    OECD AI Policy Observatory. https://oecd.ai. Ac- cessed: 2025-05-23. Organisation for Economic Co-operation and Development. 2025a. Tools for Trustworthy AI. Organisation for Economic Co-operation and Development. 2025b. Towards a Common Reporting Framework for AI Incidents. OECD Artificial Intelligence Papers, No. 18. Ac- cessed May 19, 2025. Parker, C.;...

    arXiv 2025

  8. [8]

    Physio-Risk Reporter Figure 13: Physio-Risk Reporter diagram. Sector: Private Domain: Health Task: An older adult with limited mobility uses the AI system to obtain a physio-related risk assessment; the report is for the user but can be shared with carers. Users may use the AI system multiple times over time. Intent: The AI system provides a private risk ...

Show all 28 references
  1. [9]

    Sector: Private Domain: Cross-Domain Task: Users receive support in writing, brainstorming, and editing their work via an AI system

    Writing Assistant Figure 14: Writing Assistant diagram. Sector: Private Domain: Cross-Domain Task: Users receive support in writing, brainstorming, and editing their work via an AI system. Intent: The AI system helps users communicate better and support creativity, efficiency,...

  2. [10]

    Hyperparameter Optimizer for LLM and RAG Systems Figure 15: Hyperparameter Optimizer for LLM and RAG Systems diagram. Sector: Private Domain: Cross-Domain Task: The AI system determines optimal configurations for retrieval-augmented generation (RAG) pipelines through multi- ob...

  3. [11]

    Sector: Civil Society Domain: Human Rights Task: A user can train an AI model to extract metadata from text or documents

    Metadata Extractor Figure 6: Metadata Extractor diagram. Sector: Civil Society Domain: Human Rights Task: A user can train an AI model to extract metadata from text or documents. If the user is satisfied with the performance of the outputted model, then they can use it. Intent...

  4. [12]

    Sector: Civil Society Domain: Human Rights Task: A user requires their PDF to be segmented, so an AI system segments it into an annotated PDF and JSON

    PDF Segmentator Figure 7: PDF Segmentator diagram. Sector: Civil Society Domain: Human Rights Task: A user requires their PDF to be segmented, so an AI system segments it into an annotated PDF and JSON. Intent: The AI system supports users in collecting information they need f...

  5. [13]

    Sector: Public Domain: Human Rights Task: This AI system generates responses to questions that users ask in a chat format

    Family Court Support Chatbot Figure 8: Family Court Support Chatbot diagram. Sector: Public Domain: Human Rights Task: This AI system generates responses to questions that users ask in a chat format. The AI pulls from information from the institution’s website and can route us...

  6. [14]

    Personalized Feedback Assessor Figure 9: Personalized Feedback Assessor diagram. Sector: Public Domain: Education Task: Students take practice quizzes to prepare for their exams and this AI system supports them by providing personalized feedback on their answers. Intent: The A...

  7. [15]

    Sector: Public Domain: Public Services Task: A user queries a chatbot and asks questions about information on a website

    Local Government Chatbot Figure 10: Local Government Chatbot diagram. Sector: Public Domain: Public Services Task: A user queries a chatbot and asks questions about information on a website. Intent: The AI system provides efficient 24/7 answers and reduces administrative burde...

  8. [16]

    Carer-AI Kit Risk Assessor Figure 21: Carer-AI Kit Risk Assessor diagram. Sector: Public Domain: Health Task: The AI system is apart of a platform used to validate a carer’s biomarker measurements (e.g., heart rate, pulse, blood pressure) and to provide a deterioration risk sc...

  9. [17]

    Sector: Private Domain: Finance Task: The AI system processes loan applications and generates approval recommendations; one or more credit officers review depending on loan size

    Credit Lending Classifier Figure 12: Credit Lending Classifier diagram. Sector: Private Domain: Finance Task: The AI system processes loan applications and generates approval recommendations; one or more credit officers review depending on loan size. Intent: The aims of the AI...

  10. [18]

    Sector: Public Domain: Health Task: The AI system automatically selects radiology exposure parameters based on patient information to support the radiog- rapher

    Automated Imaging Protocol Selector Figure 23: Automated Imaging Protocol Selector diagram. Sector: Public Domain: Health Task: The AI system automatically selects radiology exposure parameters based on patient information to support the radiog- rapher. Intent: The AI system s...

  11. [21]

    Smart Clinical Triage Tool Figure 16: Smart Clinical Triage Tool diagram. Sector: Public Domain: Health Task: The AI system automates clinical triage for general practice, determines urgency, collects structured information, and enables direct booking with appropriate provider...

  12. [22]

    Sector: Public Domain: Health Task: The AI system reviews chest X-rays, and flags risks and locations for the radiologist

    Chest X-Ray Abnormality Detector Figure 17: Chest X-Ray Abnormality Detector diagram. Sector: Public Domain: Health Task: The AI system reviews chest X-rays, and flags risks and locations for the radiologist. Intent: The aims of the AI system are to increase speed and accuracy...

  13. [23]

    Sector: Public Domain: Health Task: The AI system generates a preliminary patient report for providers based on responses to intake forms

    Mental Health Triage Tool Figure 18: Mental Health Triage Tool diagram. Sector: Public Domain: Health Task: The AI system generates a preliminary patient report for providers based on responses to intake forms. Intent: The aims of the AI system are to improve intake efficiency...

  14. [24]

    Sector: Public Domain: Health Task: For a patient CT scan, the AI generates a secondary view with flagged concerns and a prioritization score to speed review by a radiologist

    CT Scan Risk Detector Figure 19: CT Scan Risk Detector diagram. Sector: Public Domain: Health Task: For a patient CT scan, the AI generates a secondary view with flagged concerns and a prioritization score to speed review by a radiologist. Intent: The AI system supports radiol...

  15. [25]

    Sector: Public Domain: Health Task: During a patient–doctor consultation, the system generates notes and a referral letter (if needed) for the doctor to review and use

    Consultation AI Note-Taker Figure 20: Consultation AI Note-Taker diagram. Sector: Public Domain: Health Task: During a patient–doctor consultation, the system generates notes and a referral letter (if needed) for the doctor to review and use. Intent: The AI system improves not...

  16. [27]

    OriginTrail Decentralized Knowledge Graph (DKG) Figure 22: OriginTrail Decentralized Knowledge Graph (DKG) diagram. Sector: Private Domain: Cross-Domain Task: OriginTrail DKG enables people and agents to turn data into verifiable, structured, interconnected knowledge that re- ...

  17. [2016]

    Published in the Official Journal of the European Union, L 119, 27 April 2016

    Regulation (EU) 2016/679 of the European Parlia- ment and of the Council of 27 April 2016 on the protection of natural persons with regard to the processing of personal data and on the free movement of such data, and repealing Directive 95/46/EC (General Data Protection Regula...

    2016

  18. [2019]

    In Proceedings of the conference on fairness, accountability, and trans- parency, 220–229

    Model cards for model reporting. In Proceedings of the conference on fairness, accountability, and trans- parency, 220–229. Mökander, J.; and Floridi, L. 2023. Operationalising AI governance through ethics-based auditing: an industry case study. AI and Ethics, 3(2): 451–468. M...

    2023

  19. [2023]

    In Pro- ceedings of the 3rd ACM Conference on Equity and Access in Algorithms, Mechanisms, and Optimization, 1–15

    FeedbackLogs: Recording and incorporating stake- holder feedback into machine learning pipelines. In Pro- ceedings of the 3rd ACM Conference on Equity and Access in Algorithms, Mechanisms, and Optimization, 1–15. Biernacki, P.; and Waldorf, D. 1981. Snowball sampling: Problems...

    1981 arXiv

  20. [2024]

    Published in the Official Journal of the European Union, L 259, 12 July 2024

    Regulation (EU) 2024/1689 of the European Par- liament and of the Council of 13 June 2024 laying down harmonised rules on artificial intelligence (Artificial Intelli- gence Act). Published in the Official Journal of the European Union, L 259, 12 July 2024. Executive Office of ...

    2024

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.