REVIEW 5 major objections 6 minor
AI Governance InternationaL Evaluation Index (AGILE Index) 2025
T0 review · 5 major / 6 minor · reviewed 2026-08-06 · deepseek-v4-flash
Pith's one-line read AGILE Index 2025 argues that national AI governance is measurable and comparable: a 4-pillar, 17-dimension, 43-indicator scorecard places China, the United States, and Germany in the top tier and correlates broadly with GDP per capita.
desk verdict A transparent, policy-useful index update whose exact rankings are hostage to a mean-reverting imputation rule and untested equal weights. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is the index architecture itself: a 4-pillar, 17-dimension, 43-indicator scoring system in which each indicator is meant to translate one facet of AI governance into a quantifiable measure. Raw values are turned into 0–100 scores through simple standardization and percentile-fit normalization, dimension scores are averaged (with the AI-risk-exposure dimension reversed so higher risk lowers the score) to form pillar scores, and pillar scores are equally averaged into the total index score. Missing data are imputed hierarchically: first from prior-year growth trends, then by using the score computed from countries with available data to fill the missing entry and recalculating. This machinery is what lets 40 heterogeneous countries be placed on one common scale for ranking and year-on-year comparison.
What would settle it
Recompute the AGILE Index from the paper's published indicator data under two variations: first, drop every country with any imputed indicator value and see whether China, the United States, and Germany still form the top tier; second, re-weight the four pillars (for example, giving Pillar 4 twice the weight of Pillar 1) and check whether the ranking and the four governance types materially change. If the top tier or the GDP correlation disappears under either variation, the core results are artifacts of the imputation or equal-weighting choices rather than robust features of the underlying data.
Extended reading notes
Core claim
On the paper's own terms, the discovery is that a harmonized indicator system can turn heterogeneous evidence about AI governance into a single comparable score. Governance capacity is defined as a four-part structure: how advanced a country's AI ecosystem is (Pillar 1), how risky and how ready its environment is (Pillar 2), what strategies, laws, standards, and international engagements it has put in place (Pillar 3), and what outcomes appear in public understanding, acceptance, inclusivity, openness, and research (Pillar 4). Applied to 40 countries, the framework reports that China, the United States, and Germany form the leading tier; that top rankings shifted between the 2024 and 2025 editions, including a U.S.–China position swap tied to changes in U.S. executive AI policy; and that higher GDP per capita generally accompanies higher index scores, with clear exceptions. The paper also groups countries into four governance types—All-round Leaders, Governance Overachievers, Governance Shortfallers, and Foundation Seekers—to show that AI development and governance instruments do not always advance together.
Load-bearing premise
The ranking depends on the assumption that filling in missing country data with scores derived from the countries that do report, and then averaging all indicators with equal weight, gives an unbiased picture of national AI governance rather than systematically flattening differences between well-measured and poorly measured countries.
Editorial extensions
If this is right
- Year-on-year movement is visible in the scores: between the 2024 and 2025 editions top-ranked countries shifted while lower-ranked countries stayed comparatively stable.
- Policy changes can move a country's position: the report attributes the U.S. drop from first to second place to Executive Order 14179 replacing the more comprehensive Executive Order 14110.
- The pillar structure localizes weaknesses, so a country like Ireland, Israel, or New Zealand can see that its AI development outruns its governance instruments.
- The positive GDP-per-capita correlation suggests economic capacity supports governance, but the exceptions imply that lower-income countries can still build strong governance capacity.
- The framework is built to grow: expanding from 14 to 40 countries and from 39 to 43 indicators is presented as continued refinement toward broader, more reliable coverage.
Reading between the lines
- Editorial inference: the hierarchical imputation that fills missing values with scores computed from reporting countries likely compresses data-sparse countries toward the average, so the true spread of governance capacity may be wider than the published tiers suggest.
- Editorial inference: equal weighting of all 43 indicators implies that countries with many formal instruments (strategies, laws, standards, memberships) can score well even if enforcement is weak; comparing the index against an outcome-based measure such as documented enforcement actions would test this.
- Editorial inference: the GDP correlation probably comes mostly from Pillar 1 and Pillar 3, which reward infrastructure and formal instruments; a partial-correlation analysis across the four pillars would clarify whether governance effectiveness truly tracks income.
- Editorial inference: a robustness exercise that recalculates rankings after excluding imputed entries or varying pillar weights would show whether the China–US–Germany tier and the four governance types are stable features or artifacts of the scoring choices.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The manuscript presents the AI Governance InternationaL Evaluation Index (AGILE Index) 2025, a composite index that scores 40 countries on AI governance using 4 pillars, 17 dimensions, and 43 indicators. Data sources include bibliometric databases (DBLP, WIPO), incident repositories (OECD AIM, AIID, AIAAIC), surveys (IPSOS AI Monitor 2024, IBM AI Adoption Index), platform statistics (Hugging Face, GitHub), and desk research on national strategies and legislation. The report's headline results are a three-tier ranking with China, the United States, and Germany in the top tier; a US-China position swap relative to the 2024 edition attributed to US executive-order changes; a positive association between index score and GDP per capita; a four-type classification of countries; and a series of pillar-level observations. The appendix specifies scoring rules for strategies (D6) and legislation (D11), equal-weight aggregation, normalization, and a hierarchical missing-data imputation procedure.
Significance. Should the index's measurement properties be established, the AGILE Index would be a useful comparative tool for AI governance, and the manuscript has genuine strengths: it expands coverage from 14 to 40 countries, links every dimension to UNESCO's Recommendation and Readiness Assessment Methodology, lists data sources indicator by indicator, publishes a website with supporting data, and gives unusually detailed coding rules for the strategy and legislation dimensions. Those strengths are, however, offset by the absence of validation evidence. The central ranking and its headline comparisons rest on an imputation rule that is circular by construction, equal weighting that is never tested, and a normalization formula that is not fully specified. Because the report contains no robustness checks, uncertainty intervals, or missingness disclosure, the paper currently supports descriptive observations but not the strong comparative claims made in the Executive Summary and Key Observations.
major comments (5)
- [Appendix §2.5 (Score Normalization and Data Imputation)] The hierarchical imputation rule is circular and mean-reverting. The text states that for missing data under an indicator, the team first calculates the indicator score from available data, uses that score to fill the missing item's score, and then recalculates the indicator score. A missing country is therefore assigned approximately the cross-country average for that indicator, and the recalculated score has artificially reduced dispersion. The manuscript does not provide a country-by-indicator missingness matrix, so the number and distribution of imputed values cannot be assessed. This matters for the headline results: sources such as IPSOS AI Monitor 2024 and the OECD Going Digital Toolkit do not cover all 40 countries, and if missingness is concentrated in particular countries or indicators, the imputation can change tier membership, the four-type classification in Key Observation 4, and the US-China swap in Key Observation 2. At minimum, the authors should report the missingness matrix, repeat the analysis with alternative imputation methods (e.g., no imputation, mean imputation, regression imputation), and show that the rankings are stable.
- [Main text §2.2 (Key Observation 2) and Appendix §2.3] The explanation of the US-China ranking swap is internally inconsistent with the D11 scoring rule. Appendix §2.3 says that a country such as the United States that has both implemented comprehensive AI laws and has AI laws in development scores 100 on D11.1, but Key Observation 2 says the United States dropped to second place because Executive Order 14179 revoked Executive Order 14110 and the new order is less comprehensive, decreasing the relevant indicator score. The manuscript does not say how the revocation was coded under the three-state rule (100 enacted / 50 in process / 0 neither), nor why the Section 2.3 example remains valid after the revocation. Since the swap is one of the most prominent findings, this coding decision must be documented precisely.
- [Appendix §2.4 (Score Calculation at Each Level)] Equal weighting is asserted rather than justified, and no sensitivity analysis is reported. Indicator scores are averaged into dimensions, dimension scores into pillars, and pillar scores into the index, all with equal weights; because dimensions contain different numbers of indicators (D4 has one indicator, D6 has four, D15 has five), the implicit weight of a single indicator varies substantially across constructs. The tier cutoffs in Figure 2, the four-type classification in Key Observation 4, and the GDP correlation in Figure 5 could all change under plausible alternative weighting schemes. The authors should report how the ranking behaves under alternative weighting (e.g., principal-component weights, expert weights, or dropping the most imputed indicators) and should provide bootstrap or other uncertainty intervals around the scores and ranks.
- [Appendix §2.4 and Appendix §2.1 (gender inference)] The gender inference step is an unexplained arbitrary allocation. The text says that 'allocating 22.9% of unidentified genders as female' was combined with average-level inference before percentile-fit normalization, but it does not justify the 22.9% figure, state how it was derived, or report the share of unidentified names by country. This assumption directly affects D15.1 (gender ratio of active AI researchers) and Observation 4.2, including the headline male-to-female ratio of 2.2:1. The authors should report the fraction of names that could not be classified, justify the allocation rule, and test sensitivity to alternative allocations (e.g., 0%, 50%, or country-specific rates).
- [Appendix §2.5 (percentile-fit normalization)] The percentile-fit normalization cannot be reproduced from the text. The equation after 'we use:' is missing in the manuscript, and the verbal description—'extracted and removed one percentile of scores, then repeated the standardization and extraction on the remaining data until four score quartiles were obtained'—does not specify the algorithm (e.g., whether one percentile is removed in each pass, whether the quartiles are defined on the original or transformed scale, or how truncation interacts with the iterative removal). Since this normalization is applied to many indicators before aggregation, the authors must provide the explicit formula and a worked example.
minor comments (6)
- [What's New in AGILE Index 2025, bullet 4] The claim that imputation 'combines historical performance trends and correlations between indicators' is not supported by Appendix §2.5, which describes growth-rate imputation and hierarchical score-based imputation but no correlation-based method; the two descriptions should be reconciled.
- [Appendix §2.1] The keyword list for AI governance, which includes the phrase 'for human', appears too broad to be selective; a paper whose title contains 'for human' is not necessarily about AI governance. The authors should provide the complete keyword list and report precision/recall checks or at least a sample validation.
- [Figure 5 and Observation 4.1] The correlations between GDP per capita and index scores or public awareness are described qualitatively; please report the correlation coefficients, sample sizes, and confidence intervals for Figure 5, Figure 17, and Figure 20.
- [Appendix §2.3 (D11.1)] The three-level scoring for D11.1 equates widely different legal statuses (e.g., EU AI Act member-state implementation versus a national draft under review), but the aggregation treats 50 and 100 as linear. Please justify the 0/50/100 spacing or provide a sensitivity check with alternative ordinal codings.
- [Data source notes throughout Appendix §1] Several data-source descriptions use 'assessed' where 'accessed' is intended (e.g., D4.1, D15.4), and D15.1 contains the typo 'duiring'; these should be corrected.
- [Table 4] The IPSOS AI Monitor 2024 covers a subset of the 40 countries, but Table 4 does not indicate which countries are missing or provide survey sample sizes; please add coverage notes to all survey-based tables and figures.
Circularity Check
No significant circularity; the disclosed mean-imputation step is self-referential for missing entries but not load-bearing, and the index rests on external indicator data.
full rationale
AGILE Index 2025 derives its scores from externally sourced indicator data (DBLP, WIPO, OECD AIM, IPSOS, Hugging Face, GitHub, WGI, HDI, etc., per Appendix 1). Each indicator is normalized and aggregated through dimensions and pillars to produce a total score (Section 2.4). The main findings—tier membership, the China/US ranking swap, the GDP-per-capita correlation, and the four governance types—are descriptive summaries of those aggregated scores. No indicator is defined in terms of the final score, no parameter is fitted to the target ranking, and no previous result by the same authors is invoked to justify a modeling choice. The one potentially self-referential step is the hierarchical imputation described in Section 2.5: for a missing indicator value, the indicator score computed from available countries is used as the imputed score, after which the indicator score is recalculated. This is a fixed-point mean imputation; because the imputed value equals the available-data score, the recalculated score is unchanged. The procedure is disclosed and does not create a new prediction from fitted values. Whether mean imputation is appropriate under nonrandom missingness is a robustness/validity concern, not a circularity of the derivation. The paper does not provide a missingness matrix or sensitivity analysis, which limits confidence in the rankings, but this does not make the derivation equivalent to its inputs. Therefore no significant circularity is found.
Assumptions & free parameters
free parameters (4)
- Equal component weights =
1/n for each indicator, dimension, pillar
- Unidentified gender allocation =
22.9% allocated as female
- Percentile-fit trimming =
One percentile removed per iteration
- Legislation scoring thresholds =
100/50/0 for implemented/in process/absent
assumptions (4)
- domain assumption The level of governance should match the level of development.
- domain assumption Existence of national AI strategies, laws, and governance bodies is a valid proxy for AI governance capacity.
- domain assumption Third-party data sources (DBLP, WIPO, OECD, Hugging Face, GitHub, Ipsos) are reliable, complete, and comparable across countries.
- ad hoc to paper Missing data can be imputed using the indicator score itself without introducing bias.
Cite this review
Pith. "Pith review of AI Governance InternationaL Evaluation Index (AGILE Index) 2025." pith.science (2026). https://pith.science/paper/OZGLBDYL
@misc{pith2026250711546,
author = {Pith},
title = {Pith review of: AI Governance InternationaL Evaluation Index (AGILE Index) 2025},
year = {2026},
howpublished = {\url{https://pith.science/paper/OZGLBDYL}},
note = {Machine review of arXiv:2507.11546}
}
read the original abstract
The year 2024 witnessed accelerated global AI governance advancements, marked by strengthened multilateral frameworks and proliferating national regulatory initiatives. This acceleration underscores an unprecedented need to systematically track governance progress--an imperative that drove the launch of the AI Governance InternationaL Evaluation Index (AGILE Index) project since 2023. The inaugural AGILE Index, released in February 2024 after assessing 14 countries, established an operational and comparable baseline framework. Building on pilot insights, AGILE Index 2025 incorporates systematic refinements to better balance scientific rigor with practical adaptability. The updated methodology expands data diversity while enhancing metric validity and cross-national comparability. Reflecting both research advancements and practical policy evolution, AGILE Index 2025 evaluates 40 countries across income levels, regions, and technological development stages, with 4 Pillars, 17 Dimensions and 43 Indicators. In compiling the data, the team integrates multi-source evidence including policy documents, governance practices, research outputs, and risk exposure to construct a unified comparison framework. This approach maps global disparities while enabling countries to identify governance strengths, gaps, and systemic constraints. Through ongoing refinement and iterations, we hope the AGILE Index will fundamentally advance transparency and measurability in global AI governance, delivering data-driven assessments that depict national AI governance capacity, assist governments in recognizing their maturation stages and critical governance issues, and ultimately provide actionable insights for enhancing AI governance systems nationally and globally.
Reviewed August 6, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.