{"id":"30e25216-e28c-444f-a8fd-73ae6f1925d5","arxiv_id":"2509.10104","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":3.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"AI Harmonics measures how unevenly AI harm severity ranks are distributed across stakeholder groups, producing rankings that reduce to the existing Criticality Index.","lead":"This paper introduces AI Harmonics, a framework that ranks AI harm types using only the relative ordering of how severely different stakeholder groups are harmed, not exact numeric scores. It applies the framework to expert-annotated AI incident data and reports that political and physical harms show the most concentrated severity.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"AIH is an affine transform of the mean severity rank, so it measures location rather than concentration; the Gini-convergence and CI-validation claims therefore fail.","rationale":"The paper's central contribution is AIH as a novel ordinal metric quantifying AI harm concentration. The load-bearing problem is mathematical: Eq. 3 evaluates to an affine function of the mean severity rank, so AIH is a location measure, not a dispersion measure. My uniform-versus-bimodal counterexample shows that distributions with the same mean rank but very different spread receive identical AIH, directly falsifying the paper's interpretation. The equivalence with CI follows from the same algebra; the Figure 5 validation is thus expected rather than confirmatory. The numerical input extension to Gini is also unjustified. However, the applied pipeline, dataset construction, sensitivity analyses, and the empirical ranking itself remain useful if re-interpreted as a mean-severity or CI ranking. Since the framework is coherent and correctable but the main claims overstate what the metric does, conditional acceptance is appropriate, with revision to reframe the metric, correct the CI formula, qualify the Gini claim, and reword the interpretation. I agree only partially with the reader's weakest assumption because the severity-ordering dependence is a secondary issue; the primary defect is the metric's mathematical characterization, which subsumes two of the reader's own flagged issues.","tokens_in":1307,"tokens_out":947,"duration_ms":502331,"concrete_test":"Independently recompute AIH from Eq. 3 for M=9 on two synthetic severity distributions with equal mean rank 5: uniform (f_k=1/9 for all k) and two-point (f_1=f_9=0.5, otherwise 0). If both integrals equal 0.5, AIH measures only mean rank and cannot capture unevenness. Also recompute CI for Political & economic from the Figure 1 frequencies using the ascending-cumulative definition in Section 4; if the result is about 0.10 rather than the reported 0.89, the definitional inconsistency that underpins the AIH-CI equivalence is confirmed.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"Section 4 defines AIH as the integral of the derivative Lorenz curve with points (sum_{j=0}^k f_ij, k/M). With linear interpolation, this integral equals (mean severity rank)/M - 1/(2M), a strictly increasing affine function of the mean severity rank, not a measure of dispersion or concentration. Construct M=9 with two distributions having equal mean rank 5: uniform f_k=1/9 and two-point f_1=f_9=0.5, others zero. Both give AIH=0.5, so AIH cannot detect that the first is perfectly even while the second is maximally uneven; this contradicts Section 8's claim that AIH measures how unevenly harms are distributed and Section 4's claim that low values mean severity is more evenly distributed. Under the descending-cumulative convention that reproduces Table 4, CI_i=(sum_k k f_ik - 1)/(M-1), making AIH and CI affine transforms of each other, exactly as Eq. 4 states. The Figure 5 correlation is therefore tautological, and Section 6.2's statement that AIH emphasizes inequality at the extremes while CI captures average severity rank is false. The claim that AIH converges to standard Gini under numerical inputs is also unsupported: numeric severities never enter the AIH formula, and the area under the classic Lorenz curve is (1-Gini)/2, not Gini. Additionally, the Section 4 definition of CI via ascending cumulative frequencies is inconsistent with Table 4: for Political & economic it gives about 0.10, not 0.89; only the descending convention fits. These issues together undermine the metric's novelty and its stated interpretation as an inequality measure.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes AI Harmonics, a pipeline for prioritizing AI harm categories from ordinal severity annotations. The central contribution is a new metric AIH, defined as the area under a \"derivative Lorenz curve\" whose x-coordinates are cumulative stakeholder frequencies and whose y-coordinates are normalized severity ranks. The authors apply AIH and the Criticality Index (CI) to 816 AIAAIC annotations across nine harm categories and nine stakeholder groups, report that Political & Economic and Physical harms have the highest concentration, and support the findings with permutation, boundary, and annotation-removal sensitivity analyses plus an open-source implementation.","tokens_in":28031,"tokens_out":13253,"duration_ms":116031,"significance":"The framework's strengths are its dataset-agnostic design, use of real expert-annotated incidents, extensive sensitivity analyses, and public code and dashboard. If the metric were valid, the paper would offer a practical ordinal alternative to numeric risk scoring. However, the central mathematical claim is not correct: AIH is an affine function of the mean severity rank, so it is a location statistic rather than a concentration/inequality measure. The CI validation is tautological given Eq. (4), the printed CI formula contradicts Table 4, and the claimed convergence to the classic Gini is unsupported. These issues bear directly on the paper's stated contribution and on the headline empirical ranking.","major_comments":[{"comment":"AIH is an affine function of the mean severity rank, not a measure of inequality or concentration. With M ordered ranks and frequencies f_i1,...,f_iM, the integral of the linear interpolation of the points (sum_{j=0}^k f_ij, k/M) equals (sum_j j f_ij - 1/2)/M, i.e., (mean rank - 1/2)/M. For M=9, both the perfectly even distribution f_j=1/9 and the highly uneven two-point distribution f_1=f_9=1/2 have mean rank 5 and therefore both give AIH=0.5. This directly contradicts the Section 8 claim that AIH \"measures how unevenly harms are distributed\" and the Section 4 claim that lower values mean severity is \"more evenly distributed.\" The metric is a location statistic and cannot identify unevenness in the harm distribution.","section":"Section 4, Eqs. (1) and (3); Section 8"},{"comment":"The printed formula F_i^k = sum_{j=1}^k f_ij is an ascending cumulative frequency, which yields CI = (M - mean rank)/(M-1). For the Political & Economic category, using the heatmap frequencies and the Table 3 ordering gives a mean rank of about 8.20, so this ascending CI would be about 0.10, not the 0.89 reported in Table 4. The Table 4 values are reproduced only by a descending-cumulative version, CI = (mean rank - 1)/(M-1). Thus Eq. (4) relies on an unstated convention and the paper's benchmark table is inconsistent with the defining formula.","section":"Section 4, CI definition and Eq. (4); Table 4"},{"comment":"The validation of AIH against CI is tautological. Eq. (4) states AIH = CI*(M-1)/M + 1/(2M), so the two metrics are deterministically affine-equivalent; any scatter plot is perfectly collinear up to rounding. The paper itself acknowledges this equivalence, so the claim that \"a strong empirical correlation ... confirms that our method faithfully captures ordinal concentration\" is circular. The accompanying assertion that \"AIH emphasizes inequality at the extremes while CI captures average severity rank\" is false: both are affine transforms of the mean severity rank.","section":"Section 2, Section 6.2, Figure 5"},{"comment":"The claim that AIH \"converges to standard Gini under numerical inputs\" is unsupported and, as stated, false. Numeric severities never enter the AIH formula, which uses only the ranks k/M. The area under the classic Lorenz curve is (1 - Gini)/2, not Gini, and AIH is not that area. For M=9, the uniform distribution has Gini=0 but AIH=0.5, while the two-point distribution f_1=f_9=1/2 has Gini=0.8 but AIH=0.5. The proposed metric therefore does not unify the ordinal and numerical cases in the claimed way, and no proof or formal statement of any limiting result is provided.","section":"Section 4 and Section 8"}],"minor_comments":[{"comment":"The text says \"its highest value, equal to 0.85, is reached for the Political & Economic harm category,\" but Table 4 lists CI=0.89 and AIH=0.85; please clarify which metric is being described.","section":"Section 6.2"},{"comment":"The caption refers to a 45-degree line, but the axes are restricted to the range 0.55-0.9; please state whether the line is y=x over the displayed range or a different reference line.","section":"Figure 5"},{"comment":"The notation for cumulative frequencies is inconsistent: F_i^k and F_ij are used interchangeably, and the relationship between them should be stated explicitly.","section":"Section 4"},{"comment":"The Pros/Cons table contains a duplicated row (\"Supports pure ordinal severity scales\") and the checkmark placement appears misaligned; please proofread the table.","section":"Table 9"},{"comment":"The sentence saying that the wide best/worst-case range in Table 5 is \"confirming the findings of our previous experiments\" is confusing, because a large range indicates sensitivity to the severity ordering rather than robustness; please clarify the intended interpretation.","section":"Section 6.3.1"},{"comment":"The claim that AIH \"seamlessly collapses to the classic numerical Gini\" needs a precise theorem or should be removed, since no numerical severity input appears in the AIH formula.","section":"Section 4 and Section 8"},{"comment":"The table attributes \"Supports pure ordinal severity scales\" to the Gini Index, but the classic Gini requires numeric values; it is the AIH variant that is ordinal, and the table should be corrected.","section":"Appendix A, Table 9"}],"recommendation":"reject","confidential_remarks":"The central mathematical claim of the paper fails: AIH is an affine transform of the mean severity rank, not a measure of concentration, and the validation against CI is tautological. The printed CI formula is inconsistent with the reported benchmark values, and the claimed Gini convergence is unsupported. The empirical and software infrastructure is useful, but the core contribution would need to be redefined or substantially reframed, which is beyond a minor revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know two things about this one: the pipeline is useful, but the headline result is over-sold. AIH is not a concentration or inequality measure; it's a normalized mean severity rank. The math is right there in the paper: after linear interpolation, Eq. (1) evaluates to (Σ f_ik k)/M − 1/(2M), an affine function of the average rank. Two distributions with the same mean rank but different shapes get the same AIH — uniform over all nine levels and a two-point distribution on levels 1 and 9 both give 0.5. So the claim in Section 8 that AIH measures how unevenly harms are distributed is false, and the Section 6.2 statement that AIH emphasizes inequality at the extremes while CI captures average severity rank is also false: AIH is just a linear rescaling of the average rank.\n\nWhat's actually new and worth having: the end-to-end framework, the application to AIAAIC incident data, the ordinal treatment of severity, the sensitivity analysis, and the fact that the authors ship code and data. Reproducibility counts for a lot, and the pipeline can be plugged into other incident repositories.\n\nThe soft spots are load-bearing, not cosmetic. First, the validation of AIH against CI is circular: their own Eq. (4) says AIH is a deterministic linear function of CI, so the strong correlation in Figure 5 is forced by construction, not an independent confirmation. Second, the claim that AIH 'converges to standard Gini' under numeric inputs has no support — numeric severities never enter the AIH formula, and the area under a classic Lorenz curve is (1 − Gini)/2, not Gini. Third, the CI formula in Section 4 uses ascending cumulative frequencies, which for Political & economic gives about 0.10, not the 0.89 in Table 4; only a descending convention matches, and that ordering isn't the one in the text. Finally, the paper's own boundary analysis (Section 6.3.1) shows AIH for Political & economic ranges from 0.13 to 0.86 under different severity orderings, so the empirical ranking is heavily dependent on the assumed stakeholder ordering — a modeling choice, not a property of the data.\n\nWho is this for? Policymakers and AI governance practitioners who want a turnkey, data-driven way to rank harm categories from ordinal annotations. The ranking is usable once you stop calling it an inequality measure. A methods-minded reader looking for a new ordinal concentration metric will be disappointed.\n\nI'd send it to peer review anyway, because the framework is reproducible and the empirical content is new, and the authors are transparent about Eq. (4), which suggests the interpretive errors are fixable rather than deceptive. But the revision must either correct the metric's interpretation or reframe the contribution as an application of ordinal severity ranking, not a novel inequality index.","headline":"The framework is usable and the empirical content is new, but AIH is a normalized mean severity rank, not a concentration measure, and the validation against CI is circular.","tokens_in":28578,"tokens_out":3649,"would_cite":false,"duration_ms":32066,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper proposes AI Harmonics, a human-centric framework whose AIH metric measures AI harm concentration from ordinal severity labels alone, and reports that Political & economic and Physical harms are the most concentrated and most…","keywords":["AI harms","risk prioritization","ordinal severity","harm concentration","Lorenz curve","Criticality Index","stakeholder impacts","AI governance"],"falsifier":"Elicit category-specific severity orderings from a stakeholder panel and recompute AIH; if the top-ranked harm categories change from the paper's headline ranking, the central prioritization result depends on the assumed severity ordering rather than on the data itself.","tokens_in":27528,"feed_emoji":"⚖️","tokens_out":6780,"duration_ms":56940,"temperature":0.7,"pith_summary":"The paper aims to turn AI risk assessment away from internal compliance checks and toward the people who actually experience AI harms, and it argues this can be done from rank-only severity judgments. Its central proposal is the AIH metric, defined as the area under a derivative Lorenz curve, which measures how concentrated a harm category's impact is across stakeholder groups without assuming numeric severity values. Applied to a corpus of expert-annotated incidents, the metric ranks Political & economic harms (AIH 0.85) and Physical harms (0.73) as the most concentrated and thus most urgent, with Financial & business (0.51) and Autonomy (0.53) at the bottom. The paper further claims the pipeline is dataset-agnostic and robust: harm-category rankings stay nearly unchanged under random permutations of the severity ordering and under removal of up to 80 percent of annotations.","feed_headline":"Political and physical AI harms rank highest in new metric","feed_subtitle":"AIH scores harm concentration from ordinal labels, letting regulators prioritize without numeric estimates.","key_machinery":"The load-bearing object is the derivative Lorenz curve $\\ell_i(x)$. For harm category $c_i$, order the $M$ stakeholder groups by severity rank and plot cumulative frequencies $\\sum_{j=0}^k f_{ij}$ against normalized severity ranks $k/M$; the metric is $AIH_i = \\int_0^1 \\ell_i(x)\\,dx$. Because the y-axis uses ranks rather than cumulative severities, the curve needs only a total ordering of severity and is invariant to the numeric values attached to ranks. This is what lets the framework operate where numerical severity scores are unavailable or unreliable. AIH collapses to the standard Gini when numeric severities exist and relates to the Criticality Index by $AIH = CI\\cdot(M-1)/M + 1/(2M)$, which is the formal link used to validate it.","core_discovery":"The discovery is that a Gini-style concentration measure can be rebuilt for ordinal severity data by swapping the Lorenz curve for its derivative. For each harm category, stakeholders are ranked from least to most severely affected, and the derivative Lorenz curve plots the cumulative share of stakeholders against the normalized severity rank; AIH is the integral of that curve. The paper shows that AIH is a linear transform of the Criticality Index, so the two ordinal metrics carry the same information, and reports that on its benchmark data Political & economic harms are by far the most concentrated (AIH 0.85), followed by Physical and Psychological harms at 0.73, while Financial & business and Autonomy are the most evenly spread. The conclusion is that harm concentration, measured this way, identifies where mitigation is most urgent.","pith_inferences":["Inference: the fixed severity ordering across stakeholder groups is an input assumption, not a data product; if severity is category-dependent, per-category orderings could produce different priorities. The paper's own boundary analysis shows Political & economic AIH moving from 0.13 to 0.86 under extreme reorderings, so the assumption is load-bearing.","Inference: a decisive extension would be to run the pipeline with severity orderings elicited separately for each harm category from stakeholder panels and compare the resulting rankings to the headline ones.","Inference: the framework could be evaluated predictively instead of descriptively, by testing whether the harm categories flagged as most concentrated are also those where interventions reduce the frequency or severity of reported incidents."],"forward_implications":["If AIH is correct, any structured incident dataset with ordinal severity labels can be converted into a prioritized list of harm categories without requiring numerical loss estimates.","On the benchmark data, mitigation resources should be directed first to Political & economic and Physical harms, since these show the sharpest concentration of severe harm.","The reported stability (Spearman correlation at least 0.97 under up to five random swaps of the severity ordering) means that reasonable disagreement about which stakeholder suffers more should not change the harm ranking.","Because the pipeline recomputes the metric from annotations, revised or newly collected severity judgments can be absorbed without redesigning the assessment.","The framework extends to datasets without stakeholder annotations by treating incidents or categories as the ranked units, so the same procedure applies across a range of incident repositories."],"supporting_citations":[{"why":"supplies the expert-annotated incident repository used for the empirical evaluation.","marker":"[17]"},{"why":"defines the harm taxonomy and stakeholder groups the framework operates on.","marker":"[52]"},{"why":"introduces the Criticality Index for ordinal cyber-risk measurement, the benchmark AIH is matched against.","marker":"[47]"},{"why":"gives the CI formula and ordinal-data treatment that AIH is shown to be a linear transform of.","marker":"[56]"},{"why":"provides the Gini inequality measure from which the pseudo-Gini AIH construction is adapted.","marker":"[55]"},{"why":"documents inconsistent severity labels in AI incident databases, motivating the rank-only ordinal approach.","marker":"[11]"}],"fun_headline_variants":["Ordinal harm metric ranks political AI risks top","New AIH index flags political harm concentration","Derivative Lorenz curve pinpoints worst AI harms","AI Harmonics: political harms dominate concentration"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The ranking assumes one fixed ordering of how severely different stakeholder groups are harmed applies to every harm category; if severity order varies by category, the AIH values and resulting priorities can change.","fun_headline_variants_meta":{"raw":{"variants":["Ordinal harm metric ranks political AI risks top","New AIH index flags political harm concentration","Derivative Lorenz curve pinpoints worst AI harms","AI Harmonics: political harms dominate concentration"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000491,"raw_usage":{"total_tokens":2371,"prompt_tokens":856,"completion_tokens":1515,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":472,"completion_tokens_details":{"reasoning_tokens":1458}},"tokens_in":472,"tokens_out":1515,"duration_ms":10183,"temperature":1.0,"reasoning_tokens":1458,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T15:58:25.495339+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Elicit category-specific severity orderings from a stakeholder panel and recompute AIH; if the top-ranked harm categories change from the paper's headline ranking, the central prioritization result depends on the assumed severity ordering rather than on the data itself.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"supplies the expert-annotated incident repository used for the empirical evaluation."},{"cited_title":"Facchinetti, P","cited_arxiv_id":null,"evidence_quote":"introduces the Criticality Index for ordinal cyber-risk measurement, the benchmark AIH is matched against."},{"cited_title":"Facchinetti, P","cited_arxiv_id":null,"evidence_quote":"gives the CI formula and ordinal-data treatment that AIH is shown to be a linear transform of."},{"cited_title":"Giudici, E","cited_arxiv_id":null,"evidence_quote":"provides the Gini inequality measure from which the pseudo-Gini AIH construction is adapted."},{"cited_title":"Paeth, D","cited_arxiv_id":null,"evidence_quote":"documents inconsistent severity labels in AI incident databases, motivating the rank-only ordinal approach."}],"review_version":1}