REVIEW 7 cited by
Large Language Models are Geographically Biased
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
abstract
Large Language Models (LLMs) inherently carry the biases contained in their training corpora, which can lead to the perpetuation of societal harm. As the impact of these foundation models grows, understanding and evaluating their biases becomes crucial to achieving fairness and accuracy. We propose to study what LLMs know about the world we live in through the lens of geography. This approach is particularly powerful as there is ground truth for the numerous aspects of human life that are meaningfully projected onto geographic space such as culture, race, language, politics, and religion. We show various problematic geographic biases, which we define as systemic errors in geospatial predictions. Initially, we demonstrate that LLMs are capable of making accurate zero-shot geospatial predictions in the form of ratings that show strong monotonic correlation with ground truth (Spearman's $\rho$ of up to 0.89). We then show that LLMs exhibit common biases across a range of objective and subjective topics. In particular, LLMs are clearly biased against locations with lower socioeconomic conditions (e.g. most of Africa) on a variety of sensitive subjective topics such as attractiveness, morality, and intelligence (Spearman's $\rho$ of up to 0.70). Finally, we introduce a bias score to quantify this and find that there is significant variation in the magnitude of bias across existing LLMs. Code is available on the project website: https://rohinmanvi.github.io/GeoLLM
Forward citations
Cited by 7 Pith papers
-
Mapping the City Through the Lens of Language Models
Ten open-weight language models, rating anonymized 40-indicator city profiles, systematically favor larger, faster-growing, infrastructure-rich, less sparse urban forms.
-
Benchmarking Open-Weight Foundation Models for Global AI Technical Governance
Open-weight frontier models fabricate ~72% of AI-governance numeric answers, almost never refuse, and show inverted North/South accuracy driven largely by a proportional ±10% scoring rule and sparse high-value indicators.
-
Generating the Modal Worker: A Cross-Model Audit of Race and Gender in LLM-Generated Personas Across 41 Occupations
Four major LLMs generate occupational personas whose race and gender distributions deviate from U.S. workforce data in shared, patterned ways: White and Black workers are underrepresented while Hispanic and Asian work...
-
Around the World in 24 Hours: Probing LLM Knowledge of Time and Place
On GeoTemp, the best open model answers only 56% of two-city time questions and 33% when an hour shift is added, despite near-perfect scores on pure time arithmetic.
-
Mitigating Geospatial Knowledge Hallucination in Large Language Models: Benchmarking and Dynamic Factuality Aligning
A new benchmark called GEOHALUBENCH measures how often LLMs invent, omit, or confuse real-world places and relations, and a dynamic-beta KTO method reduces these errors on the benchmark.
-
On the missing benchmarks layer and a potential solution
An open regional EvalsHub, starting with LatamBoard, is proposed as the answer to Latin America's missing benchmark layer, but no benchmark or evaluation is presented.
-
The Rise of AI in Weather and Climate Information and its Impact on Global Inequality
AI weather and climate tools inherit Northern-controlled data and compute, risking worse forecasts and maladaptation for the Global South rather than democratizing climate information.
Discussion (0). Continue with ORCID to comment.