Pith. sign in

REVIEW 5 major objections 5 minor 41 references

Surfacing Semantic Orthogonality Across Model Safety Benchmarks: A Multi-Dimensional Analysis

T0 review · 5 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Five open-source safety benchmarks separate in embedding space into six harm clusters with uneven per-benchmark coverage, giving quantitative evidence of coverage gaps.

desk verdict Useful descriptive snapshot of five safety benchmarks, but the orthogonality and coverage-gap claims rest on unvalidated clusters and an internal five-vs-six category mismatch. read the letter →

arxiv 2505.17636 v1 pith:I6TEBTTI submitted 2025-05-23 cs.LG cs.AIcs.CL

Jonathan Bennion , Shaona Ghosh , Mantek Singh , Nouha Dziri This is my paper · ORCID
classification cs.LGcs.AIcs.CL
keywords AIbenchmarkmeta-analysissemanticorthogonalityLLMembeddingsdimensionalityreductionk-meansclusteringsafetybenchmarkscoveragegaps
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that recently released open-source AI safety benchmarks are not interchangeable: when their prompts are embedded, projected, and clustered, they split into six distinct harm regions, and the five benchmarks populate those regions very unevenly. The strongest evidence is a silhouette score of 0.470 for the optimal MiniLM-plus-UMAP configuration, with GretelAI concentrated on privacy and WildGuardMix concentrated on self-harm. If the claim is right, benchmark designers can see coverage gaps and redundancies directly instead of inferring them from topic lists, and teams can choose datasets that cover the harm categories they actually care about.

What carries the argument

The central mechanism is a semantic-orthogonality pipeline: encode each prompt with MiniLM sentence embeddings, project to two dimensions with UMAP, a neighbor-graph dimensionality reduction, cluster with k-means, and score cluster quality with the silhouette coefficient, a measure of whether points sit inside their assigned cluster rather than between clusters. The paper optimizes over embedding model, projection method, distance metric, and hyperparameters, then labels each centroid by feeding nearby prompts to an LLM with the Aegis 2.0 taxonomy. The output is a coverage map of how each benchmark's prompts distribute across emergent harm clusters.

What would settle it

Re-run clustering on the original high-dimensional MiniLM embeddings without UMAP and compare cluster assignments to each benchmark's human-labeled harm categories; if the six clusters do not reappear or show low agreement with human labels, the claimed semantic orthogonality is an artifact of the projection rather than a property of the prompts.

Watch

Extended reading notes

Core claim

In the paper's own terms, the discovery is that the semantic space of safety prompts organizes into six primary harm categories, and the five benchmarks are orthogonal in how much of each category they cover. The clusters are labeled, via a separate LLM using the Aegis 2.0 taxonomy, as harm types such as hate/identity hate, suicide and self-harm, guns/illegal weapons, PII/privacy, and criminal planning; the silhouette score of 0.470 and a corroborating run at 0.472 indicate clear separation. Prompt-length distributions differ sharply, with GretelAI and WildGuardMix using medians above 700 characters while Aegis, BeaverTails, and AILuminate use medians of 55-136 characters, which the paper treats as a confound and context for the coverage differences.

Load-bearing premise

The results assume that the two-dimensional UMAP projection of MiniLM embeddings preserves the semantic structure of safety prompts, so that the k-means clusters correspond to genuine harm categories rather than artifacts of dimensionality reduction.

Editorial extensions

If this is right

  • A high score on one benchmark no longer implies coverage of another: passing AILuminate-style hazards says little about WildGuardMix-style self-harm prompts.
  • Dataset creators can target underrepresented clusters explicitly, using the same pipeline to check whether a proposed dataset fills observed gaps.
  • The large prompt-length differences mean part of the apparent orthogonality is a collection-style artifact, so comparisons should control for prompt length or interpret it as a design choice.
  • The elbow and silhouette methods both settling near k=6 supports consolidating safety taxonomies into roughly 5-7 primary categories while retaining finer subcategories.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A cheap pre-release test follows from the method: embed a new benchmark's prompts into the same fitted space and report which of the six clusters it populates before human annotation; the paper does not propose this workflow, but it is a direct use of the coverage map.
  • The named categories are LLM-inferred, not human ground truth; an agreement study against human labels would show how much of the six-cluster structure is embedding geometry versus labeling-model priors.
  • A natural next experiment is correlating model safety scores with cluster coverage to test whether safetywashing occurs when models over-fit the dominant harm cluster of whichever benchmark is used for evaluation.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper analyzes five open-source safety benchmarks (Aegis 2.0, WildGuardMix, BeaverTails, GretelAI, AILuminate) by embedding prompts with MiniLM/MPNet, reducing dimensionality with UMAP/t-SNE, and clustering with k-means. It reports a silhouette score of 0.470, claims the semantic space organizes into six primary harm categories, and argues that benchmarks differ in coverage, with GretelAI emphasizing privacy and WildGuardMix emphasizing self-harm. The paper also examines prompt-length distributions as a potential confound and concludes with recommendations for cross-benchmark evaluation.

Significance. If the central claims were fully supported, this would be a useful contribution to AI safety benchmarking: quantifying semantic overlap and gaps across datasets has practical value for builders of safety evaluations, and the comparison of embedding models, distance metrics, and dimensionality-reduction choices is a reasonable exploratory design. The paper also deserves credit for explicitly discussing limitations, including dimensionality-reduction information loss, equal benchmark weighting, and cultural bias in harm definitions. However, the load-bearing conclusion that clusters correspond to meaningful harm categories rests on an unvalidated cluster-labeling procedure, and the manuscript contains unresolved quantitative inconsistencies. As presented, the evidence supports a qualitative observation of benchmark differences but not the specific six-category coverage-gap claims in the abstract.

major comments (5)
  1. [Sec. 3.8 and Sec. 5, Fig. 5] The paper reports inconsistent cluster counts and silhouette scores. Section 4 states that five categories were identified by the inference model ('Hate/Identity Hate', 'Suicide and Self Harm', 'Guns/Illegal Weapons', 'PII/Privacy', 'Criminal Planning'), while Section 5 and the abstract claim six primary harm categories. Figure 5 shows an elbow at k=5 and a silhouette peak at k=6 with a score of 0.455, whereas the abstract and Section 3.7 report 0.470 and Section 5 states 0.472. These discrepancies must be reconciled before the central claim of 'six primary harm categories' can be accepted.
  2. [Sec. 3.3, Sec. 3.8, Sec. 4] The mapping from k-means clusters to harm categories is not validated against any external ground truth. Each benchmark ships with its own taxonomy (BeaverTails 14 categories, AILuminate 12 hazard categories, Aegis categories, WildGuard risk labels, GretelAI privacy labels), but the paper only uses the internal silhouette score and GPT-4 labeling under the Aegis 2.0 taxonomy. The text itself concedes in Sec. 3.3 that UMAP can distort point distances; clusters in the 2D embedding could be driven by prompt length, dataset source, or style rather than harm semantics. The authors should at least report a contingency table of benchmark membership versus cluster labels and compare cluster assignments with the benchmarks' own human labels, reporting agreement statistics.
  3. [Sec. 3.1, Eq. (1), Eq. (4), Eq. (7), Eq. (8)] Several equations that are load-bearing for the methodology are missing or blank in the manuscript: Equation (1) for Cohen's sample-size formula, Equation (4) for the z-score outlier threshold, Equation (7) for the silhouette score, and Equation (8) for within-cluster sum of squares. Without these definitions, the sample-size calculation (8,175 prompts), the outlier-removal comparison, and the cluster-count selection cannot be checked. This is a reproducibility issue that must be fixed.
  4. [Sec. 5 and Sec. 6] The claims of uneven benchmark coverage, such as 'GretelAI focuses heavily on privacy' and 'WildGuardMix emphasizes self-harm scenarios', are made qualitatively without a supporting table or statistical test. There is no per-benchmark by per-cluster count table, no chi-square or Fisher exact test for benchmark-topic association, and no effect-size estimate for the imbalance. Adding this table and test is essential to support the paper's central coverage-gap conclusions.
  5. [Sec. 4] The centroid-labeling step is underspecified and potentially unstable. The paper says 'four prompt values at each centroid edge' were extracted and labeled by GPT-4 using the Aegis 2.0 taxonomy, with multiple inference runs, but no details are given about how the 'Other' category was handled despite the instruction that it should be used, no agreement metric between runs is reported, and no validation of the GPT-4 labels against human annotators is provided. The fact that the model did not use 'Other' is noted but not analyzed as a failure mode.
minor comments (5)
  1. [Sec. 3.8, Fig. 5 caption] The text says the elbow and silhouette methods converge near k=6, but the caption states the elbow plot indicates an optimal k=5. Please clarify whether the elbow was read as 5 or 6 and justify the choice of 6 over 5 beyond 'we expected a higher number of clusters'.
  2. [Sec. 3.1, Eq. (2)] The notation 'max(n_kmeans)' in Equation (2) is undefined; it should be explained how n_kmeans = 109 is derived from the assumptions in Table 1.
  3. [Sec. 1, Sec. 2] Two of the five benchmarks are co-authored by members of this paper's author team (Aegis 2.0 in Ref. [9] and WildGuardMix in Ref. [10]). This is not a methodological error, but it should be disclosed explicitly in the paper, especially because the Aegis taxonomy is used as the labeling scheme.
  4. [Sec. 7] The limitations section is thoughtful, but it would be stronger if it explicitly acknowledged that the silhouette score measures geometric cohesion, not semantic validity, and that the GPT-4 label inference is an unvalidated auxiliary model.
  5. [References] Several references are incomplete or non-standard (e.g., [12] is a blog post, [13] is a Wired article, [38] is marked 'unpublished'). For a reproducibility-oriented paper, the authors should provide DOIs or arXiv identifiers where available and flag non-archival sources.

Circularity Check

1 steps flagged · score 3.0 of 10

Cluster geometry is independently computed, but the semantic labels and the choice of k=6 are imported from the authors' own Aegis 2.0 taxonomy, making the harm-category interpretation partially circular.

  1. self citation load bearing [Section 4, 'Centroid Labeling and Inference', with Section 3.8 and Section 5]
    "We extracted four prompt values at each centroid edge based on past research [37] and created a prompt template using GPT-4 from OpenAI with the Aegis 2.0 taxonomy as labels [9]."

    The cluster names used in the abstract and Section 5 (privacy, self-harm, etc.) are obtained by prompting GPT-4 with the Aegis 2.0 taxonomy, a benchmark co-authored by one of the current authors (ref. [9] includes S. Ghosh). The same paper later reads conclusions about coverage 'gaps' off these labels, including coverage of AEGIS itself. Thus the semantic interpretation of the clusters is not an independent discovery; it is the authors' own taxonomy re-applied to embedding clusters. The cluster coordinates are computed independently of the taxonomy, so this is partial, not total, circularity.

full rationale

The paper's strongest claim—that the five safety benchmarks form distinct semantic clusters (silhouette 0.470/0.472)—is not circular by construction: the embeddings, UMAP projection, and k-means clustering are computed from prompt text without using any benchmark's harm labels, and the silhouette score is an internal measure of cluster cohesion. However, the semantic interpretation is less independent. Section 4 labels centroids using the Aegis 2.0 taxonomy, which is a co-authored prior benchmark, and Section 3.8 chooses k=6 partly from that same taxonomy; the abstract then reports 'six primary harm categories' as a discovery. This makes the harm-category naming and the category count partially inherited from the authors' own prior work, warranting a modest score. A separate internal inconsistency (Section 4 names five categories; abstract/Section 5 claim six) and the absence of any validation against human harm labels are correctness concerns, not circularity, and are not counted in the score here. Overall, the underlying cluster geometry has independent content, so the paper is only mildly-to-moderately circular.

Assumptions & free parameters 6 free parameters · 5 assumptions · 0 invented entities

The paper's central claims depend on a standard but unvalidated embedding-plus-clustering pipeline, plus several domain assumptions about the representativeness of the five benchmarks and the validity of GPT-4 labels. No new entities are introduced.

free parameters (6)
  • Number of clusters k = 6
    Selected via elbow and silhouette methods plus prior expectation of 5-7 harm categories from Aegis 2.0 (Section 3.8). The paper's claim of six primary harm categories depends directly on this choice.
  • UMAP n_neighbors = not reported (15 or 30)
    Optimized for silhouette score; final value not stated in text (Section 3.4).
  • t-SNE perplexity = 30 or 50 (not reported final)
    Grid-searched between 30 and 50; final value not stated (Section 3.4).
  • UMAP min_dist = 0.1
    Set to 0.1 for tighter clusters (Section 3.4).
  • Outlier removal threshold = z-score > 3
    Chose z-score over IQR because it retained more variance, affecting the final dataset and cluster composition (Section 3.1).
  • Significance level alpha = 0.15
    Chosen for sample size calculation to allow broader dataset inclusion; increases Type I error (Section 3.1).
assumptions (5)
  • domain assumption MiniLM/MPNet sentence embeddings capture semantic similarity among safety prompts accurately enough for clustering.
    The entire clustering pipeline relies on this, stated in Sections 3.2-3.3, with no validation against human harm labels.
  • domain assumption Two-dimensional UMAP projection preserves the global and local structure of the embedding space sufficiently for meaningful k-means clusters.
    Invoked in Section 3.3; UMAP is known to distort distances, but the paper assumes this distortion does not invalidate the cluster interpretation.
  • domain assumption GPT-4 labeling of cluster centroids using the Aegis 2.0 taxonomy produces accurate harm-category labels.
    Section 4; no human verification of the labels is reported, and the model is instructed to use an 'Other' category but does not use it.
  • domain assumption The five benchmarks are treated as equally important and equally representative of the safety benchmark landscape.
    Section 7 acknowledges equal weighting fails to reflect deployment likelihood; no weighting or representative sampling is used.
  • domain assumption The k-means algorithm on UMAP embeddings yields clusters that correspond to stable harm categories rather than artifacts of hyperparameter choice.
    Clusters are claimed as 'six primary harm categories' without testing stability across random seeds or embedding variants beyond one bootstrapped silhouette CI.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Surfacing Semantic Orthogonality Across Model Safety Benchmarks: A Multi-Dimensional Analysis." pith.science (2026). https://pith.science/paper/I6TEBTTI

@misc{pith2026250517636,
  author       = {Pith},
  title        = {Pith review of: Surfacing Semantic Orthogonality Across Model Safety Benchmarks: A Multi-Dimensional Analysis},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/I6TEBTTI}},
  note         = {Machine review of arXiv:2505.17636}
}
read the original abstract

Various AI safety datasets have been developed to measure LLMs against evolving interpretations of harm. Our evaluation of five recently published open-source safety benchmarks reveals distinct semantic clusters using UMAP dimensionality reduction and kmeans clustering (silhouette score: 0.470). We identify six primary harm categories with varying benchmark representation. GretelAI, for example, focuses heavily on privacy concerns, while WildGuardMix emphasizes self-harm scenarios. Significant differences in prompt length distribution suggests confounds to data collection and interpretations of harm as well as offer possible context. Our analysis quantifies benchmark orthogonality among AI benchmarks, allowing for transparency in coverage gaps despite topical similarities. Our quantitative framework for analyzing semantic orthogonality across safety benchmarks enables more targeted development of datasets that comprehensively address the evolving landscape of harms in AI use, however that is defined in the future.

Figures

Figures reproduced from arXiv: 2505.17636 by the authors.

Figure 1
Figure 1. Fig.1. Experimental methodology for clustering optimization [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Fig.2. Distribution of prompt lengths show more prompts removed when removing outliers by IQR rather [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Fig.3. Distribution of prompt lengths showing a right [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: Fig.4. Comparative analysis of model configurations showing silhouette scores and computational [PITH_FULL_IMAGE:figures/full_fig_p007_4.png]
Figure 5
Figure 5. Figure 5: Fig.5. Cluster optimization analysis using elbow and silhouette methods. The elbow plot (left) indicates an [PITH_FULL_IMAGE:figures/full_fig_p007_5.png]
Figure 6
Figure 6. Figure 6: Fig.6. UMAP visualization of safety benchmark prompts colored by dataset source, with labeled clusters [PITH_FULL_IMAGE:figures/full_fig_p009_6.png]
Figure 7
Figure 7. Figure 7: Fig.7. Kernel density distribution of prompt lengths across five AI safety benchmarks, revealing a bimodal [PITH_FULL_IMAGE:figures/full_fig_p009_7.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

41 extracted references · 31 canonical work pages

  1. [1]

    (2021) ”Ethical and social risks of harm fromlanguage models”, arXiv preprint arXiv:2112.04359

    Weidinger, L., Mellor, J., Rauh, M., Griffin, C., Uesato, J., Huang, P.S., et al. (2021) ”Ethical and social risks of harm fromlanguage models”, arXiv preprint arXiv:2112.04359

    arXiv 2021

  2. [2]

    (2021) ”An overview of catastrophic ai risks”, arXiv preprint arXiv:2306.12001

    Hendrycks, D., Mazeika, M., & W oodside, T. (2021) ”An overview of catastrophic ai risks”, arXiv preprint arXiv:2306.12001

    arXiv 2021

  3. [3]

    (2021) ”On the dangers of stochastic parrots: Can languagemodels be too big?”, Proceedings of the 2021 ACM conference on fairness, accountability, and transparency

    Bender, E.M., Gebru, T., McMillan-Major, A., & Mitchell, M. (2021) ”On the dangers of stochastic parrots: Can languagemodels be too big?”, Proceedings of the 2021 ACM conference on fairness, accountability, and transparency

    work page 2021

  4. [4]

    (2024) ”SafetyPrompts: a Systematic Review of Open Datasets for Evaluating and¨ Improving Large Language Model Safety”, arXiv preprint arXiv:2404.05399

    Rottger, P., Pierleoni, F., &Hovy, D. (2024) ”SafetyPrompts: a Systematic Review of Open Datasets for Evaluating and¨ Improving Large Language Model Safety”, arXiv preprint arXiv:2404.05399

    arXiv 2024

  5. [5]

    (2024) ”Safetywashing: Do AI Safety Benchmarks ActuallyMeasure Safety Progress?”, arXiv preprint arXiv:2405.19928

    Ren, R., Basart, S., Khoja, A., Gatti, A., Phan, L., Yin, X., et al. (2024) ”Safetywashing: Do AI Safety Benchmarks ActuallyMeasure Safety Progress?”, arXiv preprint arXiv:2405.19928

    arXiv 2024

  6. [6]

    (2025) ”Evaluating the Consistency of LLM Evaluators”, Proceedings of the 31st International Conference on Computational Linguistics, pp

    Lee, N., Hong, J., & Thorne, J. (2025) ”Evaluating the Consistency of LLM Evaluators”, Proceedings of the 31st International Conference on Computational Linguistics, pp. 10650-10659. Computer Science & Information Technology (CS & IT) 37

    work page 2025

  7. [7]

    &Gurevych, I

    Reimers, N. &Gurevych, I. (2019) ”Sentence -BERT: Sentence embeddings using Siamese BERT - networks”, Proceedingsof the 2019 Conference on EMNLP-IJCNLP, pp. 3982-3992

    work page 2019

  8. [8]

    (1967) ”So me methods for classification and analysis of multivariate observations”, Proceedings of the fifthBerkeley symposium on mathematical statistics and probability, V ol

    MacQueen, J. (1967) ”So me methods for classification and analysis of multivariate observations”, Proceedings of the fifthBerkeley symposium on mathematical statistics and probability, V ol. 1, pp. 281-297

    work page 1967

Show all 41 references
  1. [9]

    (2025) ”AEGIS2.0: A Diverse AI Safety Dataset and Risks Taxonomy for Alignment of LLM Guardrails”, arXiv preprint arXiv:2501.09004

    Ghosh, S., Varshney, P., Sreedhar, M.N., Padmakumar, A., Rebedea, T., Varghese, J.R., & Parisien, C. (2025) ”AEGIS2.0: A Diverse AI Safety Dataset and Risks Taxonomy for Alignment of LLM Guardrails”, arXiv preprint arXiv:2501.09004

    2025 arXiv

  2. [10]

    (2024) ”WildGuard: Open One-stop Moderation Tools for Safety Risks, Jailbreaks, andRefusals of LLMs”, arXiv preprint arXiv:2406.18495

    Han, X., Dziri, N., Zhang, J., et al. (2024) ”WildGuard: Open One-stop Moderation Tools for Safety Risks, Jailbreaks, andRefusals of LLMs”, arXiv preprint arXiv:2406.18495

    2024 arXiv

  3. [11]

    Ji,J.,Liu,M.,Dai,J.,Pan,X.,Zhang,C.,Bian,C.,Chen,B.,Sun,R.,Wang,Y .,&Yang,Y .(2023)”BEA VERTAI LS:TowardsImproved Safety Alignment of LLM via a Human -Preference Dataset”, NeurIPS 2023 Track on Datasets and Benchmarks

    2023

  4. [12]

    (2024) ”Building Datasets to Enable Safer AI Responses”, Gretel.aiBlog

    Ramaswamy, L., Van Segbroeck, M., &Nathawani, D. (2024) ”Building Datasets to Enable Safer AI Responses”, Gretel.aiBlog

    2024

  5. [13]

    MLCommons AI Safety Working Group (2024) ”AILuminate: A Benchmark for Evaluating AI Risks”, Wired

    2024

  6. [14]

    H., & Dang, S

    Ahmad, P. H., & Dang, S. (2015). Performance evaluation of clustering algorithm using different datasets. Journal of Information Engineering and Applications, 5(1), 39-45

    2015

  7. [15]

    (2021) ”Pitfalls in clustering”, Scientific reports, V ol

    Thrun, M.C. (2021) ”Pitfalls in clustering”, Scientific reports, V ol. 11, No. 1, pp. 18997

    2021

  8. [16]

    (1988) ”Statistical power analysis for the behavioral sciences”, Routledge

    Cohen, J. (1988) ”Statistical power analysis for the behavioral sciences”, Routledge

    1988

  9. [17]

    & Shukla, D

    Pandey, K.K. & Shukla, D. (2021) ”Euclidean distance stratified random sampling based clustering model for big data mining”, Computational and Mathematical Methods, V ol. 3, No. 6

    2021

  10. [18]

    Roscoe, J. T. (1975). Fundamental research statistics for the behavioral sciences (2nd ed.). Holt, Rinehart and Winston

    1975

  11. [19]

    (2013) ”Detecting outliers: Do not use standard deviation aroun d themean, use absolute deviation around the median”, Journal of Experimental Social Psychology, V ol

    Leys, C., Ley, C., Klein, O., Bernard, P., & Licata, L. (2013) ”Detecting outliers: Do not use standard deviation aroun d themean, use absolute deviation around the median”, Journal of Experimental Social Psychology, V ol. 49, No. 4, pp. 764-766. 20

    2013

  12. [20]

    (2012) ”Modern statistics for the social and behavioral sciences: A practical introduction”, CRC press

    Wilcox, R.R. (2012) ”Modern statistics for the social and behavioral sciences: A practical introduction”, CRC press

    2012

  13. [21]

    (2020) ”MiniLM: Deep self-attention distillation for task-agnosticcompression of pre-trained transformers”, arXiv preprint arXiv:2002.10957

    Wang, W ., Bao, H., Huang, S., Dong, L., & Wei, F. (2020) ”MiniLM: Deep self-attention distillation for task-agnosticcompression of pre-trained transformers”, arXiv preprint arXiv:2002.10957

    2020 arXiv

  14. [22]

    (2020) ”MPNet: Masked and Permuted Pre-training for Language Understanding”, arXiv preprint arXiv:2004.09297

    Song, K., Tan, X., Qin, T., Lu, J., & Liu, T.Y . (2020) ”MPNet: Masked and Permuted Pre-training for Language Understanding”, arXiv preprint arXiv:2004.09297

    2020 arXiv

  15. [23]

    & Hinton, G

    Van der Maaten, L. & Hinton, G. (2008) ”Visualizing data using t -SNE”, Journal of Machine Learning Research, V ol. 9,pp. 2579-2605

    2008

  16. [24]

    ( 2018) ”UMAP: Uniform Manifold Approximation and Projection for DimensionReduction”, arXiv preprint arXiv:1802.03426

    McInnes, L., Healy, J., & Melville, J. ( 2018) ”UMAP: Uniform Manifold Approximation and Projection for DimensionReduction”, arXiv preprint arXiv:1802.03426

    2018 arXiv

  17. [25]

    & Berens, P

    Kobak, D. & Berens, P. (2019) ”The art of using t -SNE for single -cell transcriptomics”, Nature Communications, V ol. 10,No. 1, pp. 1-14

    2019

  18. [26]

    Wang, Y ., Huang, H., Rudin, C., &Shaposhnik, Y . (2021). Understanding how dimension reduction tools work: An empirical approach to deciphering t -SNE, UMAP, TriMap, and PaCMAP for data visualization. Journal of Machine Learning Research, 22, 1-73

    2021

  19. [27]

    Bergstra, J., & Bengio, Y . (2012). ”Random search for hyper-parameter optimization.” Journal of Machine Learning Research, 13(Feb), 281-305

    2012

  20. [28]

    (2019) ”Automated optimizedparamet ers for t -distributed stochastic neighbor embedding improve visualization and analysis of large datasets”, Nature Communications, V ol

    Belkina, A.C., Ciccolella, C.O., Anno, R., Halpert, R., Spidlen, J., & Snyder -Cappione, J.E. (2019) ”Automated optimizedparamet ers for t -distributed stochastic neighbor embedding improve visualization and analysis of large datasets”, Nature Communications, V ol. 10, No. 1, pp. 1-12

    2019

  21. [29]

    (2001) ”On the surprising behavior of distance metrics in high-dimensionalspace”, Proceedings of the 8th International Conference on Database Theory (ICDT), pp

    Aggarwal, C.C., Hinneburg, A., & Keim, D.A. (2001) ”On the surprising behavior of distance metrics in high-dimensionalspace”, Proceedings of the 8th International Conference on Database Theory (ICDT), pp. 420-434

    2001

  22. [30]

    (2000) ”TheMahalanobis distance”, Chemometrics and Intelligent Laboratory Systems, V ol

    De Maesschalck, R., Jouan -Rimbaud, D., &Massart, D.L. (2000) ”TheMahalanobis distance”, Chemometrics and Intelligent Laboratory Systems, V ol. 50, No. 1, pp. 1-18

    2000

  23. [31]

    (1999) ”Mahalanobis distance”, Research Letters in Information and Mathematical Sciences, V ol

    McLachlan, G.J. (1999) ”Mahalanobis distance”, Research Letters in Information and Mathematical Sciences, V ol. 1, No.1, pp. 33-41

    1999

  24. [32]

    EleutherAI (2023) ”EleutherAI: Democratizing AI Research”, EleutherAI Blog

    2023

  25. [33]

    Efron, B., &Tibshirani, R. J. (1994). An Introduction to the Bootstrap. Chapman and Hall/CRC. 38 Computer Science & Information Technology (CS & IT)

    1994

  26. [34]

    A., Albrecht, J., Irwin, D., & Raghavan, B

    Satopa¨a, V . A., Albrecht, J., Irwin, D., & Raghavan, B. (2011). Finding a ”kneedle” in a haystack: Detecting knee points in¨ system behavior. In Proceedings of the 31st International Confe rence on Distributed Computing Systems Workshops (pp. 166-171). IEEE

    2011

  27. [35]

    (1987) ”Silhouettes: a graphical aid to the interpretation and validation of cluster analysis”, Journal ofComputational and Applied Mathematics, V ol

    Rousseeuw, P.J. (1987) ”Silhouettes: a graphical aid to the interpretation and validation of cluster analysis”, Journal ofComputational and Applied Mathematics, V ol. 20, pp. 53-65

    1987

  28. [36]

    (2001) ”Estimating the number of clusters in a data set via the gap statistic”,Journal of the Royal Statistical Society B, V ol

    Tibshirani, R., Walther, G., & Hastie, T. (2001) ”Estimating the number of clusters in a data set via the gap statistic”,Journal of the Royal Statistical Society B, V ol. 63, No. 2, pp. 411-423

    2001

  29. [37]

    (2024) ”Examinin g the robustness of LLM evaluation to the distributional assumptions of benchmarks”, Annual Meeting of the Association for Computational Linguistics

    Ailem, M., Marazopoulou, K., Siska, C., & Bono, J. (2024) ”Examinin g the robustness of LLM evaluation to the distributional assumptions of benchmarks”, Annual Meeting of the Association for Computational Linguistics

    2024

  30. [38]

    & Luccioni, A.S

    Lacroix, T. & Luccioni, A.S. (2024) ”Metaethical perspectives on ’benchmarking’ AI ethics”, unpublished

    2024

  31. [39]

    (2007) ”G* power 3: A flexible statistical power analysis program for the social, behavioral, and biomedical sciences”, Behavior research methods, V ol

    Faul, F., Erdfelder, E., Lang, A.G., & Buchner, A. (2007) ”G* power 3: A flexible statistical power analysis program for the social, behavioral, and biomedical sciences”, Behavior research methods, V ol. 39, No. 2, pp. 175-191

    2007

  32. [40]

    (2017) ”Semantics derived automatically from language corpora containhuman-like biases”, arXiv preprint arXiv:1608.07187

    Caliskan, A., Bryson, J.J., & Narayanan, A. (2017) ”Semantics derived automatically from language corpora containhuman-like biases”, arXiv preprint arXiv:1608.07187

    2017 arXiv

  33. [41]

    © 2025 By AIRCC Publishing Corporation

    Kenton,J.,Xu,A.,Dathathri,A.,Glaese,M.,Martens,S.,Lillicrap,T.,etal.(2021)”Alignment for honesty”,arXivpreprintarXiv:2112.00849. © 2025 By AIRCC Publishing Corporation . This article is published under the Creative Commons Attribution (CC BY) license

    2021 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.