REVIEW 4 major objections 6 minor 42 references
Assessing the Geolocation Capabilities, Limitations and Societal Risks of Generative Vision-Language Models
T0 review · 4 major / 6 minor · reviewed 2026-08-05 · deepseek-v4-flash
Pith's one-line read A 25-model benchmark shows closed-source vision-language models already geolocate most social-media-style photos within a kilometer.
desk verdict Useful and timely benchmark of VLM geolocation, but the headline 61% privacy-risk claim needs a contamination check before it can carry the policy weight. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The instrument is a standardized black-box benchmark: 25 VLMs, categorized as closed-source, open-weight, or open-source, are prompted with a fixed JSON request for country, city, neighborhood, and coordinates; outputs are parsed and scored by haversine distance through Recall@1/25/200/750 km and by country and city accuracy. The decisive dataset is GPTGeoChat, 1,000 images curated to mimic social media content, because it separates the high-accuracy regime (61% Recall@1km) from the generic street-view regime (OSV5), and its 85% embedded-text rate points to a mechanism: models read visible text and recognize people-centric scenes.
What would settle it
Build a fresh set of 1,000 social-media-like photos uploaded after GPT-4.1's training cutoff, run the same prompt, and compare Recall@1km; if it falls from 61% to near the street-level range, the headline capability is mostly memorization. As a partial check, blur or remove the embedded text in GPTGeoChat and see whether recall collapses, which would show the model is reading signs rather than recognizing geography.
Extended reading notes
Core claim
The paper establishes that current generative VLMs exhibit a sharp capability gradient. On the GPTGeoChat dataset, the top closed-source model achieves 61% Recall@1km, meaning over half of social-media-like photos are placed within one kilometer of their true location; neighboring models from Anthropic and OpenAI reach 56–60%. On the OSV5 street-view dataset, by contrast, the best model manages only 11.6% Recall@100km, and most open-weight and open-source models fall far behind. The authors conclude that model geolocation strength is concentrated in people-centric scenes and text-bearing images—exactly the kind of content shared on social media—and therefore that stripping location metadata
Load-bearing premise
The privacy-risk conclusion assumes the 61 percent success rate on the social-media-style test set reflects genuine image geolocation, not the model having memorized those images during training; the authors never check for that overlap, even though they attribute performance patterns to training-corpus composition.
Editorial extensions
If this is right
- Removing GPS metadata will no longer protect photo privacy: a picture posted to social media can be reverse-geolocated to within a kilometer more often than not with the best closed-source model.
- Beneficial geolocation applications will be unevenly served: disaster-response and street-level tasks get weaker support, while person-centric images—precisely the ones associated with stalking—are where accuracy is highest.
- Because the top geolocators are closed-source APIs, regulators and researchers cannot audit their training data or filters; the highest-risk capability is also the least inspectable.
- Country-level accuracy reaches 65–77% for several closed models, so even coarse outputs can reliably reveal a person's country or city, enabling surveillance and targeting.
- High refusal and failure rates on some models (75.5% for the transparent open-source baseline, 61.4% for one closed model) mean reliability is a current bottleneck, but the best performers fail only 0.0–0.1% of the time.
Reading between the lines
- Beyond the paper's setup, the 61% figure should be read as an upper bound until a training-data contamination check is done; a fresh, never-published social-media-like test set could settle whether closed-source VLMs generalize or memorize.
- Beyond the paper's setup, because roughly 85% of GPTGeoChat images contain embedded text, masking that text would test whether the high recall comes from reading signs and captions rather than recognizing places—an ablation that also suggests a concrete privacy defense.
- Beyond the paper's setup, the training-corpus bias the authors note implies the capability gradient is not a fixed property of the architectures; as closed-source training mixes change, the balance between street-level and social-media geolocation may shift, arguing for continuous external auditing rather than one-time certification.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The manuscript presents a black-box benchmark of 25 generative vision-language models (VLMs) on four geolocation datasets: im2gps, im2gps3k, OSV5, and GPTGeoChat. Models are evaluated with haversine-based Recall@Nkm, country/city accuracy, and failure rates. The central claim is that current VLMs perform poorly on generic street-level imagery but achieve notably high accuracy (61% Recall@1km by GPT-4.1) on images resembling social media content, raising urgent privacy concerns. The paper also discusses societal risks and possible policy implications.
Significance. If the headline result holds, the paper provides a useful first large-scale comparison of generative VLMs for geolocation across transparency regimes, with a sensible emphasis on failure modes and privacy risk. Its strengths include the breadth of models (closed-source, open-weight, open-source), the use of multiple external benchmarks, and a standardized prompt and extraction pipeline. However, the supporting evidence for the central privacy-risk claim is incomplete: no contamination analysis, no uncertainty quantification, and several methodological details are ambiguous. These gaps currently limit the strength of the conclusions that can be drawn from the 61% figure.
major comments (4)
- [Results and Analysis; Discussion and Implications] The headline claim that GPT-4.1 attains 61% Recall@1km on GPTGeoChat and that this 'raises significant and urgent privacy concerns' assumes that the GPTGeoChat images are not part of the closed-source models' training data. The manuscript provides no leakage or memorization check. GPTGeoChat consists of public, web-scraped, social-media-like images, and closed-source training corpora are undisclosed; the paper itself concedes that the observed patterns 'likely arise from the composition and biases of the training corpus rather than inherent architectural properties.' A near-duplicate or memorization explanation would change the conclusion from 'can localize novel social media images' to 'retrieves seen instances.' Please add: (i) a near-duplicate retrieval analysis against large public image corpora or an explicit estimate of potential overlap; (ii) a fresh-image control set of contempor
- [Table 1; Experimental Setup, Evaluation Metrics] No error bars, confidence intervals, or repeated runs are reported, despite stochastic model outputs. For a 1,000-image benchmark, the 95% confidence interval for 61% is approximately ±3 percentage points; the reported ordering of GPT-4.1 (61%) over Claude-3.7-Sonnet (60%) and GPT-4-Turbo (60%) is therefore within sampling noise. Decoding parameters (e.g., temperature, top-p, number of samples) and API access dates/versions are not stated, hurting reproducibility and the claim that 'GPT-4.1 consistently emerges as the strongest performer.' Please report binomial confidence intervals or repeated-run means, specify decoding hyperparameters, and avoid claiming significant differences between models whose scores are within the reported uncertainty.
- [Experimental Setup, Datasets (OSV5)] The OSV5 sampling description is internally inconsistent: 'randomly sample 500 images, ensuring broad geographic representation, including non-Western and rural areas' — random sampling cannot ensure such representation. No seed, sampling frame, stratification criteria, or geographic coverage assessment is given. The OSV5 results (e.g., GPT-4.1 R@100km = 11.6%) are used to support the claim that models perform poorly on generic street-level images; without a reproducible sampling protocol and an analysis of selection bias, this part of the claim is not supported. Please specify the exact sampling procedure and, ideally, release sample indices or a stratified sampling plan.
- [Experimental Setup, Evaluation Metrics; Table 1] The denominator of Recall@Nkm when a model fails to produce numeric coordinates is ambiguous. Eq. (2) defines Recall as the proportion of 'predictions' within N km, but Table 1 separately reports failure rates. If failures are excluded from the denominator, then GPT-o4-Mini's conditional Recall@1km is roughly 24/(100-61.4) ≈ 62%, comparable to GPT-4.1's 61%, which would materially affect the privacy-risk comparison. If failures are counted as incorrect, that should be stated explicitly. Please clarify and report both conditional and unconditional recall where applicable.
minor comments (6)
- [Experimental Setup, Prompting and Output Extraction] The full prompt is omitted ('prompt omitted for brevity'). For a benchmark paper, the exact prompt is essential for reproducibility; please include it in an appendix.
- [Experimental Setup, Vision-Language Models] The text says Gemma-3 models include '4B, 12B', but Table 1 lists Gemma-3-12B and Gemma-3-27B-it. Please reconcile and confirm the correct model set.
- [Experimental Setup, Datasets] OSV5 is described as spanning '225 countries'; standard counts are around 195 countries. If territories are included, say so explicitly.
- [Table 1] The column header 'Avg. across datasets' followed by 'City Acc (%)', 'Country Acc (%)', and 'Failure Rate (%)' is confusing. Clarify whether these are averages over datasets or overall aggregates, and define the averaging procedure.
- [References] There are two identical references to the GPT-4 technical report (Achiam et al. 2023a and 2023b). Please remove the duplicate.
- [General] Minor stylistic issues: inconsistent capitalization of 'im2gps'/'IM2GPS', and the reference 'Vo, Jacobs, and Hays 2017' has a formatting artifact ('V o').
Circularity Check
No circularity: the paper reports measured benchmark results against external datasets with no fitted parameters or self-referential derivation.
full rationale
The paper reports measured Recall@Nkm and administrative accuracies of 25 VLMs on four pre-existing benchmarks (im2gps, im2gps3k, OSV5, GPTGeoChat). The headline 61% figure is an observed frequency obtained by prompting GPT-4.1 on GPTGeoChat images, not a quantity derived from assumptions or fitted to data. GPTGeoChat was created by Mendes et al. (2024), and the model responses are evaluated with the Haversine formula and indicator sums that do not incorporate the target result. The only self-citation (Waheed et al. 2025) appears in Related Work to motivate the study and is not load-bearing for any numeric claim. The acknowledged possibility that performance 'likely arise[s] from the composition and biases of the training corpus' is a caveat about external validity, such as training-data contamination, not an argument whose conclusion is assumed in its premises; contamination would be a measurement confound rather than circular reasoning. Hence no circular step is present.
Assumptions & free parameters
assumptions (5)
- standard math The haversine distance with Earth radius r=6371 km correctly measures prediction error between predicted and ground-truth coordinates.
- domain assumption GPTGeoChat's 1,000 images are representative of social media content and therefore a valid proxy for real-world privacy risk.
- domain assumption The randomly sampled 500 OSV5 images are representative of the full dataset spanning 225 countries and 70,000+ cities.
- domain assumption Regex-based extraction correctly recovers model-provided coordinates despite minor generation errors.
- domain assumption Recall@1km on the test sets is an appropriate measure of geolocation capability for privacy-risk assessment.
Cite this review
Pith. "Pith review of Assessing the Geolocation Capabilities, Limitations and Societal Risks of Generative Vision-Language Models." pith.science (2026). https://pith.science/paper/2POZI5OM
@misc{pith2026250819967,
author = {Pith},
title = {Pith review of: Assessing the Geolocation Capabilities, Limitations and Societal Risks of Generative Vision-Language Models},
year = {2026},
howpublished = {\url{https://pith.science/paper/2POZI5OM}},
note = {Machine review of arXiv:2508.19967}
}
read the original abstract
Geo-localization is the task of identifying the location of an image using visual cues alone. It has beneficial applications, such as improving disaster response, enhancing navigation, and geography education. Recently, Vision-Language Models (VLMs) are increasingly demonstrating capabilities as accurate image geo-locators. This brings significant privacy risks, including those related to stalking and surveillance, considering the widespread uses of AI models and sharing of photos on social media. The precision of these models is likely to improve in the future. Despite these risks, there is little work on systematically evaluating the geolocation precision of Generative VLMs, their limits and potential for unintended inferences. To bridge this gap, we conduct a comprehensive assessment of the geolocation capabilities of 25 state-of-the-art VLMs on four benchmark image datasets captured in diverse environments. Our results offer insight into the internal reasoning of VLMs and highlight their strengths, limitations, and potential societal risks. Our findings indicate that current VLMs perform poorly on generic street-level images yet achieve notably high accuracy (61\%) on images resembling social media content, raising significant and urgent privacy concerns.
Figures
Reference graph
Works this paper leans on
-
[1]
, " * write output.state after.block = add.period write newline
ENTRY address archivePrefix author booktitle chapter edition editor eid eprint howpublished institution isbn journal key month note number organization pages publisher school series title type volume year label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block FUNCTION init.state.consts #0 'before.a...
-
[2]
write newline
" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...
-
[3]
L.; Almeida, D.; Altenschmidt, J.; Altman, S.; Anadkat, S.; et al
Achiam, J.; Adler, S.; Agarwal, S.; Ahmad, L.; Akkaya, I.; Aleman, F. L.; Almeida, D.; Altenschmidt, J.; Altman, S.; Anadkat, S.; et al. 2023 a . Gpt-4 technical report. arXiv preprint arXiv:2303.08774
arXiv 2023
-
[4]
L.; Almeida, D.; Altenschmidt, J.; Altman, S.; Anadkat, S.; et al
Achiam, J.; Adler, S.; Agarwal, S.; Ahmad, L.; Akkaya, I.; Aleman, F. L.; Almeida, D.; Altenschmidt, J.; Altman, S.; Anadkat, S.; et al. 2023 b . Gpt-4 technical report
work page 2023
-
[5]
Arandjelovic, R.; Gronat, P.; Torii, A.; Pajdla, T.; and Sivic, J. 2016. NetVLAD: CNN architecture for weakly supervised place recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, 5297--5307
work page 2016
-
[6]
Ashbrook, D.; and Starner, T. 2003. Starner, T.: Using GPS to learn significant locations and predict movement across multiple users. Personal and Ubiquitous Computing 7(5), 275-286. Personal and Ubiquitous Computing, 7: 275--286
work page 2003
-
[7]
N.; Raude, C.; Vincent, E.; Xu, L.; Zhou, H.; and Landrieu, L
Astruc, G.; Dufour, N.; Siglidis, I.; Aronssohn, C.; Bouia, N.; Fu, S.; Loiseau, R.; Nguyen, V. N.; Raude, C.; Vincent, E.; Xu, L.; Zhou, H.; and Landrieu, L. 2024. OpenStreetView-5M : T he Many Roads to Global Visual Geolocation. CVPR
work page 2024
-
[8]
Bai, S.; Chen, K.; Liu, X.; Wang, J.; Ge, W.; Song, S.; Dang, K.; Wang, P.; Wang, S.; Tang, J.; Zhong, H.; Zhu, Y.; Yang, M.; Li, Z.; Wan, J.; Wang, P.; Ding, W.; Fu, Z.; Xu, Y.; Ye, J.; Zhang, X.; Xie, T.; Cheng, Z.; Zhang, H.; Yang, Z.; Xu, H.; and Lin, J. 2025. Qwen2.5-VL Technical Report. arXiv preprint arXiv:2502.13923
arXiv 2025
Show all 42 references
-
[9]
Carlini, N.; Ippolito, D.; Jagielski, M.; Lee, K.; Tram\`er, F.; and Zhang, C. 2022. Quantifying Memorization Across Neural Language Models. arXiv:2202.07646
2022 arXiv
-
[10]
B.; Song, D
Carlini, N.; Tram \`e r, F.; Wallace, E.; Jagielski, M.; Herbert-Voss, A.; Lee, K.; Roberts, A.; Brown, T. B.; Song, D. X.; Erlingsson, \'U .; Oprea, A.; and Raffel, C. 2020. Extracting Training Data from Large Language Models. In USENIX Security Symposium
2020
-
[11]
C.; Xie, W.; Welsch, R
Chen, N. C.; Xie, W.; Welsch, R. E.; Larson, K.; and Xie, J. 2017. Comprehensive Predictions of Tourists' Next Visit Location Based on Call Detail Records Using Machine Learning and Deep Learning Methods. In 2017 IEEE International Congress on Big Data (BigData Congress), 1--6
2017
-
[12]
Dubey, A.; Jauhri, A.; Pandey, A.; Kadian, A.; Al-Dahle, A.; Letman, A.; Mathur, A.; Schelten, A.; Yang, A.; Fan, A.; et al. 2024. The llama 3 herd of models. arXiv e-prints, arXiv--2407
2024
-
[13]
Gerber, N.; Gerber, P.; and Volkamer, M. 2018. Explaining the privacy paradox: A systematic review of literature investigating privacy attitude and behavior. Computers & Security, 77: 226--261
2018
-
[14]
Haas, L.; Alberti, S.; and Skreta, M. 2023. Learning Generalized Zero-Shot Learners for Open-Domain Image Geolocalization. arXiv:2302.00275
2023 arXiv
-
[15]
Haas, L.; Skreta, M.; Alberti, S.; and Finn, C. 2024. PIGEON: Predicting Image Geolocations. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 12893--12902
2024
-
[16]
What Can I Really Do?
Hargittai, E.; and Marwick, A. 2016. “What Can I Really Do?” Explaining the Privacy Paradox with Online Apathy. International Journal of Communication, 10(0): 21
2016
-
[17]
Hays, J.; and Efros, A. A. 2008. Im2gps: estimating geographic information from a single image. In 2008 ieee conference on computer vision and pattern recognition, 1--8. IEEE
2008
-
[18]
H.; Juh \'a sz, L.; and Kemp, T
Hochmair, H. H.; Juh \'a sz, L.; and Kemp, T. 2024. Correctness comparison of ChatGPT-4, Gemini, Claude-3, and Copilot for spatial tasks. Transactions in GIS, 28(7): 2219--2231
2024
-
[19]
Hongladarom, S. 2020. Shoshana Zuboff, The age of surveillance capitalism: the fight for a human future at the new frontier of power: New York: Public Affairs, 2019, 704 pp. ISBN 978-1-61039-569-4 (hardcover) 978-1-61039-270-0 (ebook). AI Soc., 38(6): 2359–2361
2020
-
[20]
E.; and Tsotras, V
Izbicki, M.; Papalexakis, E. E.; and Tsotras, V. J. 2020. Exploiting the earth’s spherical geometry to geolocate images. In Machine Learning and Knowledge Discovery in Databases: European Conference, ECML PKDD 2019, W \"u rzburg, Germany, September 16--20, 2019, Proceedings, P...
2020
-
[21]
Kendall, A.; Grimes, M.; and Cipolla, R. 2015. Posenet: A convolutional network for real-time 6-dof camera relocalization. In Proceedings of the IEEE international conference on computer vision, 2938--2946
2015
-
[22]
Kokolakis, S. 2017. Privacy attitudes and privacy behaviour: A review of current research on the privacy paradox phenomenon. Computers & Security, 64: 122--134
2017
-
[23]
Kordopatis-Zilos, G.; Galopoulos, P.; Papadopoulos, S.; and Kompatsiaris, I. 2021. Leveraging efficientnet and contrastive learning for accurate global-scale location estimation. In Proceedings of the 2021 International Conference on Multimedia Retrieval, 155--163
2021
-
[24]
Krumm, J. 2022. Sensitivity Analysis of Personal Location Disclosure. In 2022 23rd IEEE International Conference on Mobile Data Management (MDM), 73--82
2022
-
[25]
Lauren c on, H.; Tronchon, L.; Cord, M.; and Sanh, V. 2024. What matters when building vision-language models? Advances in Neural Information Processing Systems, 37: 87874--87907
2024
-
[26]
Lazar, S.; and Nelson, A. 2023. AI safety on whose terms? Science, 381(6654): 138--138
2023
-
[27]
Luo, G.; Biamby, G.; Darrell, T.; Fried, D.; and Rohrbach, A. 2022. G^3 : Geolocation via Guidebook Grounding. arXiv:2211.15521
2022 arXiv
-
[28]
Marmasse, N.; and Schmandt, C. 2002. A User-Centered Location Model. Personal Ubiquitous Comput., 6(5–6): 318–321
2002
-
[29]
Mendes, E.; Chen, Y.; Hays, J.; Das, S.; Xu, W.; and Ritter, A. 2024. Granular Privacy Control for Geolocation with Vision Language Models. In Al-Onaizan, Y.; Bansal, M.; and Chen, Y.-N., eds., Proceedings of the 2024 Conference on Empirical Methods in Natural Language Process...
2024
-
[30]
Muller-Budack, E.; Pustu-Iren, K.; and Ewerth, R. 2018. Geolocation estimation of photos using a hierarchical model and scene classification. In Proceedings of the European conference on computer vision (ECCV), 563--579
2018
-
[31]
W.; Hallacy, C.; Ramesh, A.; Goh, G.; Agarwal, S.; Sastry, G.; Askell, A.; Mishkin, P.; Clark, J.; Krueger, G.; and Sutskever, I
Radford, A.; Kim, J. W.; Hallacy, C.; Ramesh, A.; Goh, G.; Agarwal, S.; Sastry, G.; Askell, A.; Mishkin, P.; Clark, J.; Krueger, G.; and Sutskever, I. 2021. Learning Transferable Visual Models From Natural Language Supervision. In International Conference on Machine Learning
2021
-
[32]
Regmi, K.; and Shah, M. 2019. Bridging the domain gap for ground-to-aerial image matching. In Proceedings of the IEEE/CVF International Conference on Computer Vision, 470--479
2019
-
[33]
Solove, D. J. 2025. On Privacy and Technology (excerpt). https://ssrn.com/abstract=5159448. Daniel J. Solove, On Privacy and Technology (Oxford University Press, 2025)
2025
-
[34]
Team, G.; Kamath, A.; Ferret, J.; Pathak, S.; Vieillard, N.; Merhej, R.; Perrin, S.; Matejovicova, T.; Ram \'e , A.; Rivi \`e re, M.; et al. 2025. Gemma 3 technical report
2025
-
[35]
K.; and Shah, M
Vivanco, V.; Nayak, G. K.; and Shah, M. 2023. GeoCLIP: Clip-Inspired Alignment between Locations and Images for Effective Worldwide Geo-localization. In Advances in Neural Information Processing Systems
2023
-
[36]
Vo, N.; Jacobs, N.; and Hays, J. 2017. Revisiting IM2GPS in the Deep Learning Era. arXiv:1705.04838
2017 arXiv
-
[37]
D.; and Ehsan, S
Waheed, S.; Ferrarini, B.; Milford, M.; Ramchurn, S. D.; and Ehsan, S. 2025. Image-based Geo-localization for Robotics: Are Black-box Vision-Language Models there yet? arXiv:2501.16947
2025 arXiv
-
[38]
Wang, H.; Wang, C.; and Xie, L. 2020. Online visual place recognition via saliency re-identification. In 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 5030--5036. IEEE
2020
-
[39]
Weyand, T.; Araujo, A.; Cao, B.; and Sim, J. 2020. Google landmarks dataset v2-a large-scale benchmark for instance-level recognition and retrieval. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2575--2584
2020
-
[40]
Weyand, T.; Kostrikov, I.; and Philbin, J. 2016. Planet-photo geolocation with convolutional neural networks. In Computer Vision--ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part VIII 14, 37--55. Springer
2016
-
[41]
Wu, M.; and Huang, Q. 2022. IM2City: image geo-localization via multi-modal learning. In Proceedings of the 5th ACM SIGSPATIAL International Workshop on AI for Geographic Knowledge Discovery, GeoAI '22, 50–61. New York, NY, USA: Association for Computing Machinery. ISBN 9781450395328
2022
-
[42]
Zhou, Z.; Zhang, J.; Guan, Z.; Hu, M.; Lao, N.; Mu, L.; Li, S.; and Mai, G. 2024. Img2Loc: Revisiting Image Geolocalization using Multi-modality Foundation Models and Image-based Retrieval-Augmented Generation. In Proceedings of the 47th International ACM SIGIR Conference on R...
2024
Reviewed August 5, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.