Pith. sign in

REVIEW 4 major objections 5 minor 1 cited by

The World As Large Language Models See It: Exploring the reliability of LLMs in representing geographical features

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Large language models approximate the geographic world but are systematically unreliable: in this paper, GPT-4o and Gemini 2.0 Flash geocode a known Innsbruck landmark with root-mean-square errors of 268 m and 317 m, underestimate…

desk verdict A useful empirical check that current LLMs can't be trusted for precise geospatial facts, but the elevation experiment's scale mismatch undermines one of the paper's three pillars. read the letter →

arxiv 2506.00203 v1 pith:LTI2NQJ5 submitted 2025-05-30 cs.CY cs.AIcs.IR

Omid Reza Abbasi , Franz Welscher , Georg Weinberger , Johannes Scholz This is my paper
classification cs.CYcs.AIcs.IR
keywords largelanguagemodelsgeocodingelevationestimationreverseGPT-4oGemini2.0FlashgeospatialreliabilityAustriafederalstates
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Large language models can recite fragments of geographic fact, but this paper asks whether they can be trusted to produce geographic data on their own. The authors put GPT-4o and Gemini 2.0 Flash through three zero-shot tasks: geocoding a known monument, estimating elevations across Austria on a 10 km grid, and reverse-geocoding coordinates to Austrian federal states. In all three tasks the models approximate the target but carry systematic errors: hundreds of meters of coordinate bias, a consistent underestimation of elevation, and persistent misclassification of administrative regions. If these results hold, LLMs are not yet independent geospatial data generators and should only be used as an aid inside GIS workflows.

What carries the argument

The method is a repeated zero-shot prompting protocol with statistical error analysis. For geocoding, a single point feature (St. Anne's Column) was queried 100 times so the authors could separate systematic bias (mean offset) from random scatter (standard deviations, RMSE, and eigenanalysis of the error covariance matrix to obtain 1-sigma and 95% error ellipses). For elevation and reverse geocoding, the same 840-point grid covering Austria at 10 km spacing was queried five times per point and the median response was taken to suppress outliers, with residuals relative to the official DEM and to Nominatim reverse geocoding used as ground truth. This design lets the paper turn free-form language-model answers into quantitative error distributions that can be compared across models and tasks.

What would settle it

Rerun the geocoding task with the same two models while logging sampling parameters, and check whether the 100 responses are distinct and normally scattered around the reported centroids. If the models return identical coordinates on every repeat, or if the scatter vanishes at temperature 0, the reported standard deviations and error ellipses would not reflect genuine uncertainty in internal geographic knowledge; if the centroids move by hundreds of meters when the query is rephrased, the systematic-bias claim would need to be restated as prompt-sensitivity rather than fixed bias.

Watch

Extended reading notes

Core claim

The paper's central claim is that LLMs' internal representation of the geographical world is approximate and systematically skewed rather than randomly noisy. Geocoding St. Anne's Column in Innsbruck over 100 repeated queries, GPT-4o produced a mean position 214 m north and 39 m east of the true location with RMSE 268 m, while Gemini 2.0 Flash was more precise but biased 316 m northward, giving RMSE 317 m; both models' estimates cluster near neighboring landmarks rather than at the target. Across 840 points in Austria, both models consistently underestimated true elevations, GPT-4o by a mean of 114.26 m and Gemini 2.0 Flash by 43.51 m, while still reproducing major topographic trends. In reverse geocoding, GPT-4o identified the correct federal state 81% of the time and Gemini 2.0 Flash 86% of the time, but neither model reconstructed Austria's regional layout faithfully, with the largest failures in small states and along boundaries. The conclusion is that fine-tuning with geographic information will be required before LLMs can serve as reliable geospatial knowledge sources.

Load-bearing premise

The study treats 100 repeated zero-shot API queries as independent random samples, but it does not report temperature, top-p, model snapshot, or API settings; if repeated identical prompts are correlated or near-deterministic, the reported RMSE values and error ellipses do not describe the models' true estimation variability.

Editorial extensions

If this is right

  • Neither model can yet be used as an independent geocoding engine: even a well-known point landmark is placed hundreds of meters away with a consistent directional pull.
  • LLM-generated elevations are usable for coarse topographic awareness, such as distinguishing lowland from alpine terrain, but not for measurement, since both models systematically under-report true elevation.
  • Automated administrative boundary assignment from coordinates is feasible at about 81 to 86 percent accuracy, but the errors cluster in small states and border regions, so downstream decisions need a verification step.
  • Retraining or fine-tuning on structured geographic data, rather than only scaling up the model, is the concrete path the results point to for reducing these systematic biases.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because the paper does not report sampling parameters, the geocoding precision numbers may measure API nondeterminism rather than the models' internal geographic estimates; a direct test would be to set temperature to 0 and see whether the 100-point scatter collapses.
  • The northward pull toward the Goldenes Dachl, Hofkirche, and Stadtturm suggests the models interpolate a landmark's position from nearby named places in their latent representation; this could be tested by geocoding landmarks in landmark-rich versus landmark-poor regions and comparing bias directions.
  • The same 840-point grid could be run on countries with different border geometries to see whether the reverse-geocoding errors are driven by Austria's specific shape or by a general difficulty with administrative boundaries.
  • A cheap extension would be to ask the models to state a confidence interval with each answer; if their stated uncertainty fails to cover the true point at the claimed 95 percent rate, then the error ellipses are not usable for propagation in GIS workflows.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper evaluates GPT-4o and Gemini 2.0 Flash on three geospatial tasks: geocoding a single landmark in Innsbruck (St. Anne's Column), estimating elevations on a 10 km grid across Austria, and reverse geocoding the same grid points into Austrian federal states. The authors report that both models show systematic geocoding biases, tend to underestimate elevations, and misclassify federal states with 81% and 86% accuracy, respectively. They conclude that LLMs can approximate geographic information but are not yet reliable enough to be treated as independent geospatial data generators. The study uses zero-shot prompting, external ground-truth sources (true coordinates, an official DEM, Nominatim), and repeated queries to quantify variability.

Significance. This is a straightforward empirical evaluation with external ground-truth sources and no fitted parameters, so there is no circularity concern. The tasks are clearly specified and the zero-shot protocol is appropriate for probing internal knowledge. If the elevation comparison is corrected, the paper would provide a useful benchmark for LLM geospatial reliability and would strengthen the growing evidence that LLMs are unreliable for precise geospatial facts. Its significance is moderate: the geocoding conclusion rests on a single landmark, and the elevation inference is presently undermined by a reference-scale mismatch, so the strength of the central claim is lower than the current text suggests.

major comments (4)
  1. [§3.2 (Elevation estimation)] The ground truth used in the elevation task is not aligned with the query scale. Models are asked for the elevation of a single coordinate (Prompt 2), but the official 10 m DEM is aggregated by median to 10 km cells and then compared with the point estimate. In Austria's alpine terrain, a 10 km cell can span more than 2,000 m of relief, so the cell median is not the elevation at the queried point. The reported mean underestimates of 114.26 m for GPT-4o and 43.51 m for Gemini 2.0 Flash, and the t-statistics in Table 1, are therefore computed against a reference that is not the quantity requested. To establish the elevation claim, the authors should extract the native-resolution DEM value at each coordinate and compare against that, or aggregate model responses to cell medians and compare cell-to-cell.
  2. [§3.1 (Geocoding)] The statistical treatment of the geocoding results assumes that 100 repeated zero-shot API queries are independent draws from the model's output distribution. The paper does not report temperature, top-p, model snapshot/version, or other decoding parameters. If the API was called with near-zero temperature, the responses may be near-identical, in which case the reported standard deviations (153 m in latitude and 32 m in longitude for GPT-4o; 18.5 m and 15 m for Gemini 2.0 Flash) and the error-ellipse geometry in Figure 1 do not estimate random variability. The systematic bias estimates remain relevant, but the precision component needs to be re-derived under a stated sampling protocol or explicitly labeled as conditional on deterministic decoding.
  3. [§3.2 (Table 1)] The one-sample t-test treats the 840 grid points as independent observations, but elevations on a 10 km grid are strongly spatially autocorrelated. The p-values in Table 1 (9.38e-24 for GPT-4o and 0.00063 for Gemini 2.0 Flash) therefore overstate the statistical evidence. Because the observed mean biases are large, a spatially aware analysis may still find significance, but the current inference is not valid as reported. The authors should use a test that accounts for spatial correlation, such as a block bootstrap or a spatial mixed model, and report the effective sample size.
  4. [§3.1 and Conclusions] The geocoding conclusion rests on a single point feature, St. Anne's Column in Innsbruck. The observed northward bias is interpreted as attraction toward nearby landmarks, but with only one landmark one cannot separate model-specific behavior from location-specific behavior. The abstract's claim that 'both models exhibited systematic and random errors' is only demonstrated for this one location. A stronger design would include several landmarks of different types and in different regions before making a general claim about geocoding reliability.
minor comments (5)
  1. [Throughout] The text often refers to '2.0 Flash' without the model family name; use 'Gemini 2.0 Flash' consistently after first mention.
  2. [Table 2] The class label 'V orarlberg' contains a stray space; it should be 'Vorarlberg'.
  3. [Figure 2] The color scale in Figures 2(c) and 2(e) is labeled '-250% to 250%', but the relative error is never defined in the text. Please define the denominator and clarify how values above 100% are intended to be read.
  4. [§3.1--3.3] The paper does not state how raw model outputs were parsed into numeric coordinates, elevations, or state names, nor how malformed responses were handled. A short paragraph on output cleaning and validation would improve reproducibility.
  5. [§3.3] Nominatim is treated as ground truth for reverse geocoding without quantifying its own positional or administrative-boundary error. A brief discussion of this limitation would be appropriate, especially for points near state boundaries.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: each task is benchmarked against independent external ground truth.

full rationale

The paper's claims are derived by comparing model outputs to external references, not to quantities fitted inside the paper. For geocoding, estimates are compared to the known coordinate of St. Anne's Column (47.265556 N, 11.394167 E). For elevation, responses are compared to an official 10 m DEM aggregated to 10 km via median. For reverse geocoding, labels are compared to Nominatim references. None of these references is constructed from the model outputs or from the authors' prior work, and no parameter is fitted and then renamed a prediction. The only concerns are methodological—the 10 km median aggregation may not represent point elevations in mountainous Austria, and the t-test treats spatially autocorrelated residuals as independent—but these are validity issues about the reference scale and statistical inference, not circular derivation. Because the central comparisons reduce to external benchmark data, the derivation chain is self-contained and no circular step is present.

Assumptions & free parameters 0 free parameters · 5 assumptions · 0 invented entities

This study fits no parameters; all quantitative claims are descriptive statistics against external references, namely the true coordinate, the official DEM, and Nominatim. The assumptions are domain-level about benchmark fidelity and statistical independence. The API-sampling independence assumption is the most fragile because the paper gives no temperature, top-p, or model snapshot information.

assumptions (5)
  • domain assumption Official DEM from data.gv.at aggregated to a 10 km grid is an accurate ground truth for elevation.
    Used as the reference for all elevation errors in Section 3.2; errors in the source DEM would propagate into the reported biases.
  • domain assumption Nominatim reverse geocoding provides correct Austrian federal state labels.
    Used as the benchmark in Section 3.3; incorrect Nominatim labels would change the confusion matrices and accuracy scores.
  • domain assumption Repeated API calls are independent samples of model output.
    Section 3.1 treats 100 repetitions as random draws for error ellipses and precision estimates; no temperature or sampling settings are described.
  • domain assumption A grid of 840 points at 10 km spacing is representative of Austria.
    Used for both elevation and reverse geocoding; resolution and boundary effects may bias regional metrics, especially for small states.
  • domain assumption One-sample t-test validity despite spatial autocorrelation.
    Table 1 reports p-values assuming independent observations, but neighboring grid cells are spatially correlated, which inflates the reported significance.

how reviews work

0 comments
Cite this review

Pith. "Pith review of The World As Large Language Models See It: Exploring the reliability of LLMs in representing geographical features." pith.science (2026). https://pith.science/paper/LTI2NQJ5

@misc{pith2026250600203,
  author       = {Pith},
  title        = {Pith review of: The World As Large Language Models See It: Exploring the reliability of LLMs in representing geographical features},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/LTI2NQJ5}},
  note         = {Machine review of arXiv:2506.00203}
}
read the original abstract

As large language models (LLMs) continue to evolve, questions about their trustworthiness in delivering factual information have become increasingly important. This concern also applies to their ability to accurately represent the geographic world. With recent advancements in this field, it is relevant to consider whether and to what extent LLMs' representations of the geographical world can be trusted. This study evaluates the performance of GPT-4o and Gemini 2.0 Flash in three key geospatial tasks: geocoding, elevation estimation, and reverse geocoding. In the geocoding task, both models exhibited systematic and random errors in estimating the coordinates of St. Anne's Column in Innsbruck, Austria, with GPT-4o showing greater deviations and Gemini 2.0 Flash demonstrating more precision but a significant systematic offset. For elevation estimation, both models tended to underestimate elevations across Austria, though they captured overall topographical trends, and Gemini 2.0 Flash performed better in eastern regions. The reverse geocoding task, which involved identifying Austrian federal states from coordinates, revealed that Gemini 2.0 Flash outperformed GPT-4o in overall accuracy and F1-scores, demonstrating better consistency across regions. Despite these findings, neither model achieved an accurate reconstruction of Austria's federal states, highlighting persistent misclassifications. The study concludes that while LLMs can approximate geographic information, their accuracy and reliability are inconsistent, underscoring the need for fine-tuning with geographical information to enhance their utility in GIScience and Geoinformatics.

Figures

Figures reproduced from arXiv: 2506.00203 by the authors.

Figure 2
Figure 2. (a) DEM of Austria; (b) Estimated DEM generated from GPT-4o responses; (c) Relative errors of the [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. The confusion matrices of the reverse geocoding task [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figure 4
Figure 4. The layout of Austria’s states as perceived by the models in the task of reverse geocoding [PITH_FULL_IMAGE:figures/full_fig_p007_4.png] view at source ↗

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. A Vision for Geo-Temporal Deep Research Systems: Towards Comprehensive, Transparent, and Reproducible Geo-Temporal Information Synthesis

    cs.CL 2025-06 conditional novelty 4.0 of 10

    A research agenda calling for geo-temporal reasoning in deep research systems, with no experiments or system implementation.

Reference graph

Works this paper leans on

30 extracted references · 19 canonical work pages · cited by 1 Pith paper

  1. [1]

    A., Wornow, M., Swaminathan, A., Lehmann, L

    Bedi, S., Liu, Y., Orr-Ewing, L., Dash, D., Koyejo, S., Callahan, A., Fries, J. A., Wornow, M., Swaminathan, A., Lehmann, L. S., et al. (2024). A systematic review of testing and evaluation of healthcare applications of large language models (llms). medRxiv , pages 2024--04

    work page 2024

  2. [2]

    M., Gebru, T., McMillan-Major, A., and Shmitchell, S

    Bender, E. M., Gebru, T., McMillan-Major, A., and Shmitchell, S. (2021). On the dangers of stochastic parrots: Can language models be too big? In Proceedings of the 2021 ACM conference on fairness, accountability, and transparency , pages 610--623

    2021

  3. [3]

    Bhandari, P., Anastasopoulos, A., and Pfoser, D. (2023). Are large language models geospatially knowledgeable? In Proceedings of the 31st ACM International Conference on Advances in Geographic Information Systems , pages 1--4

    work page 2023

  4. [4]

    D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al

    Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J. D., Dhariwal, P., Neelakantan, A., Shyam, P., Sastry, G., Askell, A., et al. (2020). Language models are few-shot learners. Advances in neural information processing systems , 33:1877--1901

    2020

  5. [5]

    A., Salah, K., Jayaraman, R., ur Rehman, M

    Gebreab, S. A., Salah, K., Jayaraman, R., ur Rehman, M. H., and Ellaham, S. (2024). Llm-based framework for administrative task automation in healthcare. In 2024 12th International Symposium on Digital Forensics and Security (ISDFS) , pages 1--7. IEEE

    work page 2024

  6. [6]

    Guha, N., Nyarko, J., Ho, D., R \'e , C., Chilton, A., Chohlas-Wood, A., Peters, A., Waldon, B., Rockmore, D., Zambrano, D., et al. (2024). Legalbench: A collaboratively built benchmark for measuring legal reasoning in large language models. Advances in Neural Information Processing Systems , 36

    work page 2024

  7. [7]

    and Weber, P

    Haklay, M. and Weber, P. (2008). Openstreetmap: User-generated street maps. IEEE Pervasive computing , 7(4):12--18

    work page 2008

  8. [8]

    Exploring Spatial Representations in the Historical Lake District Texts with LLM-based Relation Extraction

    Haris, E., Cohn, A. G., and Stell, J. G. (2024). Exploring spatial representations in the historical lake district texts with llm-based relation extraction. arXiv preprint arXiv:2406.14336

    work page Pith review arXiv 2024

Show all 30 references
  1. [9]

    Holm, K. O. M. (2024). Geogpt: An llm-based geographic information system. Master's thesis, NTNU

    2024

  2. [10]

    Huang, Y., Sun, L., Wang, H., Wu, S., Zhang, Q., Li, Y., Gao, C., Huang, Y., Lyu, W., Zhang, Y., et al. (2024). Trustllm: Trustworthiness in large language models. arXiv preprint arXiv:2401.05561

    2024 arXiv

  3. [11]

    Li, C., Yang, R., Li, T., Bafarassat, M., Sharifi, K., Bergemann, D., and Yang, Z. (2024a). Stride: A tool-assisted llm agent framework for strategic and interactive decision-making. arXiv preprint arXiv:2405.16376

    2024 arXiv

  4. [12]

    X., Nie, J.-Y., and Wen, J.-R

    Li, J., Tang, T., Zhao, W. X., Nie, J.-Y., and Wen, J.-R. (2024b). Pre-trained language models for text generation: A survey. ACM Computing Surveys , 56(9):1--39

    2024

  5. [13]

    and Ning, H

    Li, Z. and Ning, H. (2023). Autonomous gis: the next-generation ai-powered gis. International Journal of Digital Earth , 16(2):4668--4686

    2023

  6. [14]

    Liu, O., Fu, D., Yogatama, D., and Neiswanger, W. (2024a). Dellma: A framework for decision making under uncertainty with large language models. arXiv preprint arXiv:2402.02392

    2024 arXiv

  7. [15]

    Liu, Z., Janowicz, K., Currier, K., and Shi, M. (2024b). Measuring geographic diversity of foundation models with a natural language--based geo-guessing experiment on gpt-4. AGILE: GIScience Series , 5:38

    2024

  8. [16]

    A., Goodchild, M

    Longley, P. A., Goodchild, M. F., Maguire, D. J., and Rhind, D. W. (2015). Geographic information science and systems . John Wiley & Sons

    2015

  9. [17]

    Majic, I., Wang, Z., Janowicz, K., and Karimi, M. (2024). Spatial task-explicity matters in prompting large multimodal models for spatial planning. In Proceedings of the 7th ACM SIGSPATIAL International Workshop on AI for Geographic Knowledge Discovery , pages 99--105

    2024

  10. [18]

    and Oucheikh, R

    Mansourian, A. and Oucheikh, R. (2024). Chatgeoai: Enabling geospatial analysis for public through natural language, with large language models. ISPRS International Journal of Geo-Information , 13(10):348

    2024

  11. [19]

    Manvi, R., Khanna, S., Mai, G., Burke, M., Lobell, D., and Ermon, S. (2023). Geollm: Extracting geospatial knowledge from large language models. arXiv preprint arXiv:2310.06213

    2023 arXiv

  12. [20]

    Ning, H., Li, Z., Akinboyewa, T., and Lessani, M. N. (2024). An autonomous gis agent framework for geospatial data retrieval. arXiv preprint arXiv:2407.21024

    2024 arXiv

  13. [21]

    R., and Samet, H

    O'Sullivan, K., Schneider, N. R., and Samet, H. (2024). Metric reasoning in large language models. In Proceedings of the 32nd ACM International Conference on Advances in Geographic Information Systems , pages 501--504

    2024

  14. [22]

    H., and Riedel, S

    Petroni, F., Rockt \"a schel, T., Lewis, P., Bakhtin, A., Wu, Y., Miller, A. H., and Riedel, S. (2019). Language models as knowledge bases? arXiv preprint arXiv:1909.01066

    2019 arXiv

  15. [23]

    Roberts, J., L \"u ddecke, T., Das, S., Han, K., and Albanie, S. (2023). Gpt4geo: How a language model sees the world's geography. arXiv preprint arXiv:2306.00020

    2023 arXiv

  16. [24]

    Shi, X., Liu, J., Liu, Y., Cheng, Q., and Lu, W. (2025). Know where to go: Make llm a relevant, responsible, and trustworthy searchers. Decision Support Systems , 188:114354

    2025

  17. [25]

    Wang, S., Hu, T., Xiao, H., Li, Y., Zhang, C., Ning, H., Zhu, R., Li, Z., and Ye, X. (2024). Gpt, large language models (llms) and generative artificial intelligence (gai) models in geospatial science: a systematic review. International Journal of Digital Earth , 17(1):2353122

    2024

  18. [26]

    Wei, J., Tay, Y., Bommasani, R., Raffel, C., Zoph, B., Borgeaud, S., Yogatama, D., Bosma, M., Zhou, D., Metzler, D., et al. (2022). Emergent abilities of large language models. arXiv preprint arXiv:2206.07682

    2022 arXiv

  19. [27]

    J.-J., Jiang, S., Liu, Y., Zhang, Y., and Liu, Y

    Wen, H., Li, Y., Liu, G., Zhao, S., Yu, T., Li, T. J.-J., Jiang, S., Liu, Y., Zhang, Y., and Liu, Y. (2024). Autodroid: Llm-powered task automation in android. In Proceedings of the 30th Annual International Conference on Mobile Computing and Networking , pages 543--557

    2024

  20. [28]

    K., Kasai, J., and Yildirim, I

    Yamada, Y., Bao, Y., Lampinen, A. K., Kasai, J., and Yildirim, I. (2023). Evaluating spatial understanding of large language models. arXiv preprint arXiv:2310.14540

    2023 arXiv

  21. [29]

    Zhang, Y., He, Z., Li, J., Lin, J., Guan, Q., and Yu, W. (2024). Mapgpt: an autonomous framework for mapping by integrating large language model and cartographic tools. Cartography and Geographic Information Science , 51(6):717--743

    2024

  22. [30]

    Zhao, H., Liu, Z., Wu, Z., Li, Y., Yang, T., Shu, P., Xu, S., Dai, H., Zhao, L., Mai, G., et al. (2024). Revolutionizing finance with llms: An overview of applications and insights. arXiv preprint arXiv:2401.11641

    2024

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.