{"id":"50e89f33-0ece-4143-bce4-35203917eea4","arxiv_id":"2505.23417","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A rapid tertiary review of nine AI governance reviews finds a focus on high-level frameworks and principles, with little concrete guidance on governance mechanisms.","lead":"This paper reviews nine existing literature reviews on AI governance, finding that the EU AI Act and the NIST risk framework are the most discussed, with transparency and accountability as the leading principles. It is a quick orientation to how the review literature frames AI governance, but its narrow sample and internal inconsistencies limit how much weight readers should give it.","discovery_kind":"review","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The abstract's 'transparency and accountability are the most common principles' contradicts the body's RQ2 finding that transparency and privacy are most cited, and no frequency counts are reported; the central claim is not verifiable.","rationale":"The paper's purpose is a tertiary synthesis; its scientific value lies in accurately ranking frameworks and principles in the secondary literature. The strongest claim is the abstract's summary. My stress-test focuses on the internal consistency of that claim rather than on the sample's external generalizability. The body's RQ2 result explicitly gives a different ranking than the abstract: 'transparency and privacy are the most frequently cited principles, followed by fairness, accountability...' versus 'transparency and accountability are the most common principles.' No counts accompany either statement. This is a direct, checkable failure of reporting. The absence of counts also affects the 'most cited frameworks' assertion. The concern is load-bearing because the abstract and conclusion are what readers will act on. The proposed test—re-extracting principle and framework frequencies from the nine included studies—would settle it. If counts match the abstract, my concern is refuted; if they match the body or neither, the central claim fails. I do not see a need to change the reader's REJECT verdict; the identified issue strengthens the rejection. I agree partially with the reader: the reader's weakest_assumption concerns sample representativeness, while my primary concern is internal reporting; but the reader's rationale also notes the abstract/body contradiction.","tokens_in":7611,"tokens_out":4214,"duration_ms":40196,"concrete_test":"Independently re-extract from the nine included studies (resolving the reference-list mismatches, e.g., [13] vs [2]) a frequency table of how many studies explicitly identify each principle (transparency, accountability, privacy, fairness, explainability) and how many explicitly cite the EU AI Act or NIST RMF. If the table does not show transparency and accountability as the top two principles, the abstract's central claim fails. The same table would also test whether 'most cited frameworks' is accurate.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—that the secondary literature most emphasizes transparency and accountability—is not supported by the paper's own evidence. Section III (RQ2) states that 'Across all studies, transparency and privacy are the most frequently cited principles, followed by fairness, accountability...', while both the abstract and the introduction's bullet list name transparency and accountability as the most common principles. The paper never reports the frequency counts that would justify either ranking, so the contradiction cannot be resolved from the reported data. The same applies to frameworks: 'most cited' is asserted for the EU AI Act and NIST RMF without counts or a list of which of the nine studies cite them. Because the reader-facing summary depends on these rankings, the strongest claim is unverifiable as written. This is a correctness risk independent of sample representativeness: even if the two-database sample were accepted, the internal reporting does not establish the stated result.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper presents a rapid tertiary review of nine secondary studies on AI governance, retrieved from IEEE Xplore and ACM Digital Library, with the aim of mapping frameworks, principles, organizational mechanisms, and stakeholder roles. It reports that the EU AI Act and NIST RMF are the most cited frameworks, that transparency and accountability are the most common principles, and that few reviews detail actionable governance mechanisms. It concludes with implications for industry, society, and research, and it acknowledges the streamlined and limited nature of the review.","tokens_in":7723,"tokens_out":5978,"duration_ms":58884,"significance":"A rigorous synthesis of secondary literature on AI governance would be valuable to both researchers and practitioners, and the paper's four-question structure is sensible. The paper is transparent about its rapid-review constraints, provides its search string and an open data link, and explicitly acknowledges limitations such as the two-database scope and single-author screening. However, as submitted, the headline rankings are not supported by reproducible counts, the selection process includes non-peer-reviewed preprints despite a stated peer-review criterion, and the evidence table contains broken references. The contribution would be useful after these issues are corrected and the reported findings are made internally consistent and verifiable.","major_comments":[{"comment":"The abstract and the introduction's bullet list state that transparency and accountability are the most common principles, but §III RQ2 states that 'Across all studies, transparency and privacy are the most frequently cited principles, followed by fairness, accountability, explainability...' No frequency counts are reported anywhere, and the only citations given for the RQ2 claim are [16] and [17], which do not support the abstract's phrasing. This internal contradiction makes the headline result unverifiable; the authors should report per-study counts and align the abstract, introduction, and results.","section":"Abstract and §III (RQ2)"},{"comment":"The inclusion criteria require that articles be peer-reviewed, and §III.A asserts that 'All selected articles were published between 2020 and 2024 in peer-reviewed venues.' However, refs [11], [16], and [17] are arXiv preprints (with [11] marked 'forthcoming' at the time), and Table I includes them. This violates the stated criterion and changes the composition of the reviewed sample. Either the criterion must be relaxed with a clear justification, or the preprints must be replaced or reclassified, and all reported rankings must be re-derived from the corrected sample.","section":"§II.C–D and §III.A"},{"comment":"The claim that the EU AI Act and NIST RMF are 'the most cited frameworks' is not supported by any quantitative extraction. RQ1 provides only qualitative examples and points to individual studies; no table or count shows how often each framework appears across the nine included studies. The phrase 'most cited' is therefore an assertion rather than a reported finding, and it needs to be backed by a frequency count or a clear coding table.","section":"§III (RQ1)"},{"comment":"The evidence table has broken reference numbering: reference [10] is assigned to both 'Towards a Privacy and Security-Aware Framework...' and 'IT Governance in the Artificial Intelligence Age...', and the row 'Responsible AI Systems: Who are the Stakeholders?' is keyed to [2], which in the reference list is Raji et al. 'Closing the AI accountability gap' (2020), not the Deshpande and Sharp study. This prevents readers from mapping the included studies to the reported synthesis, and the table and reference list must be corrected.","section":"Table I and References"}],"minor_comments":[{"comment":"In §II, the sentence beginning 'However, the growing number of such reviews, we believe it is time...' is grammatically incomplete, and the word 'belive' should be 'believe'.","section":"§II"},{"comment":"Table I contains a typographical error: 'Metrics Catalogue:cA Collection' should read 'Metrics Catalogue: A Collection'.","section":"Table I"},{"comment":"Reference [6] has a truncated author field ('B. , G. Pinto, and S. Soares') and should be completed.","section":"References"},{"comment":"The conclusion claims that the literature shows convergence on 'algorithmic auditing' as a best practice, but the results section does not report any included study specifically recommending algorithmic auditing; this claim should either be supported with extracted evidence or removed.","section":"§VI Conclusion"}],"recommendation":"major_revision","confidential_remarks":"The weaknesses are substantive but, in my assessment, fixable within a revision: the authors can add explicit frequency counts, resolve the transparency/accountability versus transparency/privacy contradiction, correct the inclusion of non-peer-reviewed preprints, and repair the reference numbering. I therefore recommend major revision rather than rejection. The central contribution of a tertiary review is only as reliable as its evidence table and selection transparency, so these corrections are prerequisites for publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper is a genuine tertiary review, not a fake, but its central rankings do not hold up. The abstract and the introduction's bullet list say transparency and accountability are the most common principles, while Section III (RQ2) says transparency and privacy are most frequently cited. No frequency counts are reported anywhere, so the reader cannot tell which claim is right. The same goes for frameworks: 'most cited' is asserted for the EU AI Act and NIST RMF without a list of which of the nine studies cite them. That is the load-bearing result, and it is unverifiable as written.\n\nWhat the paper does well: it compacts nine secondary studies into a readable four-RQ synthesis, and it uses a clearly described rapid-review method. The gap between high-level principles and concrete organizational mechanisms is a fair observation that emerges from the literature. The authors are transparent about the two-database scope and the single-author screening, and they acknowledge threats to validity. That buys them some goodwill.\n\nThe soft spots are bigger than surface typos. The inclusion criteria say peer-reviewed, but at least three included studies are arXiv preprints (refs 11, 16, 17). The reference list has duplicate numbers (ref [10] used twice) and a mismatched entry: Table I's 'Developing an Ethical Regulatory Framework' is cited as [13], but the reference list assigns [13] to a completely different paper. These are traceability failures in a review whose whole value is traceability.\n\nI don't think the authors are hiding anything; the open-data link suggests they intended to share extraction sheets. But as presented, the main claims about principle and framework prevalence are not supported by the reported evidence. For a practitioner wanting a quick narrative snapshot of common AI governance frameworks, the paper gives a reasonable overview. For a researcher who needs dependable rankings or gap claims, it is not usable until the authors supply the raw counts and fix the internal contradiction.\n\nRecommendation: send it to peer review, but with a clear request for major revision. The authors need to (a) report the frequency data behind every 'most cited' claim, (b) reconcile the abstract/body discrepancy, and (c) either justify the inclusion of preprints or remove them. If those changes are made, this could be a modest but legitimate contribution. If the data cannot be produced, it should be rejected. As it stands, I would not cite it.","headline":"A well-intentioned but under-supported tertiary review: the abstract and body disagree on the top principles, and no frequency data backs either ranking.","tokens_in":8252,"tokens_out":2506,"would_cite":false,"duration_ms":25981,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Nine secondary studies show AI governance literature converges on the EU AI Act and NIST RMF but offers few actionable implementation mechanisms.","keywords":["AI governance","Responsible AI","tertiary review","rapid review","transparency","accountability","EU AI Act","NIST RMF"],"falsifier":"Re-running the same search with dual screening and a third digital library would falsify the central claims if it returns a different dominant framework or principle set—if, say, human oversight or fairness outranks transparency, or if detailed implementation mechanisms turn out to be common in the wider literature.","tokens_in":7396,"feed_emoji":"🏛️","tokens_out":9544,"duration_ms":88353,"temperature":0.7,"pith_summary":"This paper is a rapid tertiary review: a synthesis of existing reviews rather than a new primary study. The authors ask what the secondary literature on AI governance emphasizes and synthesize nine secondary studies published between 2020 and 2024, drawn from two major computing-oriented digital libraries. Their central finding is that the literature converges on high-level frameworks—most often the EU AI Act and the NIST RMF—and on a small set of principles, with transparency and accountability most prominent, but that few studies give concrete, organization-level governance mechanisms or stakeholder engagement strategies. If the finding holds, it gives practitioners and researchers a consolidated map of what the field regards as important and a clear statement of where the evidence stops: principle-level agreement, not implementation guidance.","feed_headline":"AI governance reviews lean on EU AI Act and NIST, skip how to act","feed_subtitle":"A tertiary review of nine studies finds principles outpace concrete governance mechanisms and stakeholder strategies.","key_machinery":"The carrying mechanism is the rapid tertiary review design: a search of two computing-oriented digital libraries with a fixed query, manual screening of 55 candidate records down to nine secondary studies, structured data extraction, and thematic synthesis following the steps of extract, code, translate into themes, and build higher-order themes. The classification grid is a set of six governance pillars—fairness, transparency, privacy and security, sustainability, accountability, and explainability—used to code the reviewed studies. The main evidence carriers are the included studies themselves, especially the Responsible AI Pattern Catalogue, the Responsible AI Metrics Catalogue, a privacy-and-security-aware framework, and an explainability roadmap. This machinery matters because the reported rankings of frameworks and principles are produced by what those nine studies happen to cite and emphasize.","core_discovery":"On the paper's own terms, the discovery is a gap analysis of the AI governance review literature. After screening 55 candidate records down to nine secondary studies, the authors report that the most frequently cited governance frameworks are the EU AI Act and the NIST RMF, alongside a range of other instruments such as AI Verify, the EU's capAI project, the EU Trustworthy AI Assessment List, and the NSW AI Assurance Framework. The most emphasized principles are transparency and accountability, followed by fairness, privacy, explainability, and safety; accountability is often decomposed into responsibility, auditability, and redressability. Stakeholders tend to be categorized at industry, organizational, and team levels, but the authors conclude that concrete guidance is scarce: only a subset of the reviewed studies describe audit procedures, ethics committees, documentation protocols, or similar mechanisms, and even less attention is paid to empirical validation or the inclusion of underrepresented groups.","pith_inferences":["A broader search that included policy, law, and human-computer-interaction databases would probably surface more implementation-oriented governance mechanisms, meaning the reported operational gap may partly reflect the computing-only corpus rather than the whole field of AI governance.","Because at least one of the nine selected studies is a preprint rather than a peer-reviewed publication, the synthesis actually draws on non-peer-reviewed sources; if that is true, the peer-reviewed framing in the abstract is doing less work than claimed.","A conflict-oriented reading of the same nine studies, looking for tensions between transparency and privacy or between accountability and efficiency, might find that the principles form not a shared foundation but a set of unresolved trade-offs."],"forward_implications":["Organizations designing AI governance can expect consensus on what matters—transparency, accountability, fairness—but little tested guidance on how to implement it; the paper collects the few concrete mechanisms that appear, such as ethics committees, algorithmic audits, standardized reporting, maturity models, and certification.","Researchers mapping AI governance can treat the computing-oriented literature as converging on the EU AI Act and NIST RMF, so new studies should either build on these anchors or justify diverging from them.","Stakeholder strategy in the current literature is largely a three-level categorization (industry, organization, team) plus calls for inclusive engagement, not an operational procedure.","The review's conclusion implies that the most useful next studies are empirical evaluations of governance practices and methodologically consistent re-reviews, rather than additional surveys of principles."],"supporting_citations":[{"why":"Supplies the multi-level governance patterns, the list of regulatory frameworks such as AI Verify and the NIST RMF, and the concrete mechanisms (committees, audits, maturity models) that drive RQ1 and RQ3.","marker":"[16]"},{"why":"Supplies the accountability metrics catalogue, the responsibility/auditability/redressability decomposition, and one of the few concrete mechanism sets in the corpus.","marker":"[17]"},{"why":"Supplies the privacy-and-security-aware framework with data, technology, people, and process dimensions, and the finding that privacy is discussed more than security.","marker":"[12]"},{"why":"Supplies the transparency and explainability emphasis and the nine-stakeholder taxonomy used to answer RQ4.","marker":"[14]"},{"why":"Supplies governance trends and the international guideline context that supports the framework ranking in RQ1.","marker":"[10]"}],"fun_headline_variants":["AI governance reviews: principles outpace concrete mechanisms","Nine-study AI governance review finds principles, few actions","AI governance lit review: EU AI Act and NIST, but little how-to","Tertiary review of AI governance: transparency touted, tools lacking","AI governance review shows principles, but not practical steps"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that nine studies located in two computing-oriented digital libraries by a single screener—including some that were not peer-reviewed—fairly represent what the AI governance literature emphasizes; if that sample is biased, the rankings change.","fun_headline_variants_meta":{"raw":{"variants":["AI governance reviews: principles outpace concrete mechanisms","Nine-study AI governance review finds principles, few actions","AI governance lit review: EU AI Act and NIST, but little how-to","Tertiary review of AI governance: transparency touted, tools lacking","AI governance review shows principles, but not practical steps"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000582,"raw_usage":{"total_tokens":2701,"prompt_tokens":869,"completion_tokens":1832,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":485,"completion_tokens_details":{"reasoning_tokens":1747}},"tokens_in":485,"tokens_out":1832,"duration_ms":15641,"temperature":1.0,"reasoning_tokens":1747,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T12:45:49.981102+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-running the same search with dual screening and a third digital library would falsify the central claims if it returns a different dominant framework or principle set—if, say, human oversight or fairness outranks transparency, or if detailed implementation mechanisms turn out to be common in the wider literature.","supporting_citations":[{"cited_title":"Responsible AI Pattern Catalogue: A Collection of Best Practices for AI Governance and Engineering","cited_arxiv_id":"2209.04963","evidence_quote":"Supplies the multi-level governance patterns, the list of regulatory frameworks such as AI Verify and the NIST RMF, and the concrete mechanisms (committees, audits, maturity models) that drive RQ1 and RQ3."},{"cited_title":"Towards a Responsible AI Metrics Catalogue: A Collection of Metrics for AI Accountability","cited_arxiv_id":"2311.13158","evidence_quote":"Supplies the accountability metrics catalogue, the responsibility/auditability/redressability decomposition, and one of the few concrete mechanism sets in the corpus."},{"cited_title":"Towards a privacy and security-aware framework for ethical ai: Guiding the development and assessment of ai systems,","cited_arxiv_id":null,"evidence_quote":"Supplies the privacy-and-security-aware framework with data, technology, people, and process dimensions, and the finding that privacy is discussed more than security."},{"cited_title":"A roadmap of explainable artificial intelligence: Explain to whom, when, what and how?","cited_arxiv_id":null,"evidence_quote":"Supplies the transparency and explainability emphasis and the nine-stakeholder taxonomy used to answer RQ4."},{"cited_title":"It gov- ernance in the artificial intelligence age: Trends and practices,","cited_arxiv_id":null,"evidence_quote":"Supplies governance trends and the international guideline context that supports the framework ranking in RQ1."}],"review_version":1}