Pith. sign in

REVIEW 3 major objections 4 minor 36 references

Towards LLM Agents for Earth Observation

T0 review · 3 major / 4 minor · reviewed 2026-08-16 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read On a new 140-question Earth observation benchmark, the best LLM agent scores 33% accuracy with Google Earth Engine access because generated code fails to run over 58% of the time.

desk verdict A useful new benchmark for LLM agents in Earth Observation, but the headline accuracy figure rests on a verification step the paper doesn't actually describe. read the letter →

arxiv 2504.12110 v2 pith:OIKLPDFW submitted 2025-04-16 cs.AI

Chia Hsiang Kao , Wenting Zhao , Shreelekha Revankar , Samuel Speas , Snehal Bhagat , Rajeev Datta , Cheng Perng Phoo , Utkarsh Mall
show 3 more authors
Carl Vondrick Kavita Bala Bharath Hariharan
This is my paper · ORCID
classification cs.AI
keywords earthobservationagentsaccuracyachievedatamodelsmuch
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The authors built a benchmark called UnivEARTH from 140 yes/no questions taken from NASA's Earth Observatory articles. The questions cover topics like snow cover, night lights, forest loss, and air pollution, and span 17 satellite sensors. For every question, a correct answer is known from the article. The benchmark then asks an AI agent to answer the question by writing a small program that queries Google Earth Engine, a massive public repository of satellite data, and to execute that program to obtain evidence.

The authors tested several large language models, including Claude, GPT, and DeepSeek. Even the best model, Claude-3.7-Sonnet, answered correctly only 33% of the time. The main problem was not reasoning but mechanics: the models could not produce Python code that runs without errors. Code failed to execute or access the right data in about 60% of attempts. When the code did run, the models were often accurate, which suggests the bottleneck is knowledge of Google Earth Engine's data catalog, not understanding the science.

The authors also showed that a smaller open model, Llama-3.1-8B, could reach 25% accuracy after fine-tuning on synthetic data designed to teach correct data catalog names. This is close to the larger commercial models at a fraction of the cost. The paper concludes that current AI agents are not ready to automate Earth observation, and that improving the models' knowledge of available satellite datasets is the main path forward.

Extended reading notes

Core claim

The central assertion, stated in the abstract and conclusion, is: "Using Google Earth Engine API as a tool, LLM agents can only achieve an accuracy of 33% because the code fails to run over 58% of the time," and "current AI systems fall short of reliably facilitating earth science applications." If correct, this means state-of-the-art LLM agents cannot yet produce reliable, executable code for Earth observation analysis, and improving data catalog knowledge and code generation are the key bottlenecks.

Load-bearing premise

The benchmark labels are correct and answerable from Google Earth Engine in the specific way the authors intend. Section 2.2 verifies only that relevant GEE datasets exist ("verified dataset availability"), and reviewer labels come from article text and images, not from GEE computations. If many article-derived answers cannot be recovered by any reasonable GEE query, then the 33% accuracy reflects benchmark ambiguity or methodology mismatch rather than LLM incapability.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper introduces UnivEarth, a benchmark of 140 yes/no questions derived from NASA Earth Observatory articles, spanning 13 topics and 17 satellite sensors. The authors evaluate LLM agents that must answer the questions by generating and executing Python code against the Google Earth Engine API. They report that the best agents achieve only 33% accuracy because, they claim, code fails to run over 58% of the time; they also show that fine-tuning a Llama-3.1-8B model on synthetic data improves its accuracy to about 25%, comparable to much larger models. The central claim is that current AI systems cannot yet reliably support Earth observation workflows when answers must be grounded in executable analysis.

Significance. If the central result holds, the paper makes a useful contribution: it is among the first benchmarks to require LLM agents to ground Earth-observation answers in executable Google Earth Engine code, rather than relying on parametric knowledge. The curation process is thoughtful in several respects: using NASA Earth Observatory articles as an authoritative source, combining LLM-assisted question generation with manual editing, and running an independent human review that reached full agreement on the yes/no labels. The evaluation covers a broad set of commercial and open models, and the synthetic fine-tuning experiment is a constructive step toward improving smaller models. The dataset could be a valuable resource for future work on tool-using agents for scientific data analysis, provided the validation issues described below are addressed.

major comments (3)
  1. [2.2, Appendix E] The benchmark's central claim that all 140 questions are 'reasonably answerable' via Google Earth Engine is not actually validated. Section 2.2 states that the authors 'wrote test implementations,' 'verified dataset availability,' and 'sometimes identified alternative data sources,' but it never states that they computed the article-derived ground-truth answer from GEE data for each question. The independent review described in Appendix E asks reviewers to answer from the article text and images, with Google Maps only for location checks; it does not validate that the ground-truth answer is recoverable by any competent GEE query. If many article conclusions cannot be reproduced from GEE data because of transient events, mixed pixels, threshold sensitivity, or missing products, then the reported 33% accuracy and 58% failure rate measure benchmark ambiguity or execution-environment strictness rather than LLM incapability. This is load-bearing because the headline claim is precisely about grounded, executable GEE analysis. The paper should provide, for each question, a reference GEE implementation that reproduces the ground-truth label, or otherwise demonstrate that the labels are computable from GEE data.
  2. [3.1, Table 1] The abstract states that accuracy is low 'because the code fails to run over 58% of the time,' but the failure rate in Table 1 is defined more broadly. Section 3.1 says failure is recorded when 'the code was incorrect, or the data requested by the code was not available,' which includes cases where code executed but accessed unavailable data. It also includes cases where the model produced no answer because it could not complete the task. The paper should decompose the failure rate into code execution errors, wrong asset or data errors, and other causes, and should rephrase the abstract and conclusion so that the causal claim is not stronger than what Table 1 actually measures.
  3. [Table 3] One of the showcased examples is internally inconsistent. The Atmosphere row asks 'Did nitrogen oxide concentrations in the Northern Hemisphere increase from 2019 to 2020?' but the supporting sentence is about methane measurements: 'The annual growth rate for 2020 was the highest scientists had recorded since systematic annual methane measurements began in 1983.' If this is a typo, it should be corrected; if not, it indicates that some question-answer pairs may not be grounded in the cited article text, which would undermine the integrity of the benchmark labels. Please clarify.
minor comments (4)
  1. [1. Contributions] The phrase 'an novel evaluation benchmark' contains a grammatical error; it should be 'a novel evaluation benchmark.'
  2. [Figure 2] The correlations (r = 0.87 and r = -0.88) are computed over only ten model points; the paper should report confidence intervals or a permutation test to show that the correlation is not driven by one or two outliers.
  3. [4. Conclusion] The sentence 'even the best models achieving only 49.0% accuracy without internet access' should specify that this is the three-option no-internet scenario; in the two-option scenario the best model achieves 83.8%, so the unsupported reading is misleading.
  4. [2.2] The claim that all questions are 'reasonably answered' via GEE would benefit from a precise definition of what counts as reasonable, including how thresholding ambiguities and transient phenomena are handled.
Assumptions & free parameters 0 free parameters · 4 assumptions · 0 invented entities

The central claim rests on the assumptions that NASA articles are correct, that GEE data can support the intended answers, and that the QA generation and review pipeline produces unambiguous labels. There are no fitted physical parameters or invented entities; OpenEARTH is a fine-tuned model, not a new theoretical entity.

assumptions (4)
  • domain assumption NASA Earth Observatory articles provide accurate and representative ground truth for Earth observation questions.
    Section 2.1 treats NASA EO as an authoritative source; if article conclusions are wrong or unrepresentative, the benchmark labels are wrong.
  • domain assumption Google Earth Engine data can support answers to all 140 questions in the intended way.
    Section 2.2 verifies only dataset availability, not that GEE computations recover the article-derived answer; this assumption is load-bearing for the accuracy metric.
  • domain assumption LLM-generated QA pairs with human review yield valid, unambiguous questions.
    Section 2.2 uses Claude-3.5-Sonnet to propose questions and reviewers to clean them; if the questions are ambiguous, the 33% accuracy is unreliable.
  • domain assumption Running model-generated code locally and parsing outputs with GPT-4o-mini is a valid way to determine model answers.
    Section 3.1 describes this pipeline; parser errors could inflate failure rates.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Towards LLM Agents for Earth Observation." pith.science (2026). https://pith.science/paper/OIKLPDFW

@misc{pith2026250412110,
  author       = {Pith},
  title        = {Pith review of: Towards LLM Agents for Earth Observation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/OIKLPDFW}},
  note         = {Machine review of arXiv:2504.12110}
}
read the original abstract

Earth Observation (EO) provides critical planetary data for environmental monitoring, disaster management, climate science, and other scientific domains. Here we ask: Are AI systems ready for reliable Earth Observation? We introduce \datasetnamenospace, a benchmark of 140 yes/no questions from NASA Earth Observatory articles across 13 topics and 17 satellite sensors. Using Google Earth Engine API as a tool, LLM agents can only achieve an accuracy of 33% because the code fails to run over 58% of the time. We improve the failure rate for open models by fine-tuning synthetic data, allowing much smaller models (Llama-3.1-8B) to achieve comparable accuracy to much larger ones (e.g., DeepSeek-R1). Taken together, our findings identify significant challenges to be solved before AI agents can automate earth observation, and suggest paths forward. The project page is available at https://iandrover.github.io/UnivEarth.

Figures

Figures reproduced from arXiv: 2504.12110 by the authors.

Figure 1
Figure 1. We propose UnivEARTH for benchmarking AI agents in Earth Observation. to evaluate LLMs for earth observation. There are two chal￾lenges in building such a benchmark: (1) we need to know the kind of questions that one might ask about earth obser￾vation data and the corresponding answers, and (2) we need to ensure that the evidence or data needed to support the answer exists and is available. Unlike existing benchmark… view at source ↗
Figure 2
Figure 2. Zero-shot accuracy is correlated with number of unique imagery collections used (left) and negatively with the fraction of times the “Wrong Asset Name” error is encountered (right). To test this hypothesis, we looked at the number of unique imagery collections accessed by different LLM agents and whether higher-performing models leverage a more diverse range of data sources. As shown in [PITH_FULL_IMAGE:figures/ful… view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

36 extracted references · 11 canonical work pages

  1. [1]

    S., Krishnan, N., and Jablonka, K

    Alampara, N., Schilling-Wilhelmi, M., R ´ıos-Garc´ıa, M., Mandal, I., Khetarpal, P., Grover, H. S., Krishnan, N., and Jablonka, K. M. Probing the limitations of multimodal language models for chemistry and materials research. arXiv preprint arXiv:2411.16955,

    arXiv

  2. [6]

    The table shows zero-shot performance with accuracy (%), failure rate (%), and selective accuracy (%)

    Performance of Our Trained Model Across Different Checkpoints. The table shows zero-shot performance with accuracy (%), failure rate (%), and selective accuracy (%). Checkpoint Accuracy Failure Selective Accuracy 1 17.63 72.63 57.74 2 19.24 68.04 54.30 3 23.26 64.78 58.87 4 25.04 61.07 57.61 11 Submission and Formatting Instructions for ICML 2025 E. Revie...

    work page 2025

  3. [7]

    Swift- dossier: Tailored automatic dossier for drug discovery with llms and agents.arXiv preprint arXiv:2409.15817,

    Fossi, G., Boulaimen, Y ., Outemzabet, L., Jeanray, N., Ger- art, S., Vachenc, S., Giemza, J., and Raieli, S. Swift- dossier: Tailored automatic dossier for drug discovery with llms and agents.arXiv preprint arXiv:2409.15817,

    arXiv

  4. [9]

    Deepseek-r1: In- centivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948,

    Guo, D., Yang, D., Zhang, H., Song, J., Zhang, R., Xu, R., Zhu, Q., Ma, S., Wang, P., Bi, X., et al. Deepseek-r1: In- centivizing reasoning capability in llms via reinforcement learning.arXiv preprint arXiv:2501.12948,

    arXiv

  5. [10]

    Text-to-sql in the wild: A naturally-occurring dataset based on stack ex- change data.arXiv preprint arXiv:2106.05006,

    Hazoom, M., Malik, V ., and Bogin, B. Text-to-sql in the wild: A naturally-occurring dataset based on stack ex- change data.arXiv preprint arXiv:2106.05006,

    arXiv

  6. [11]

    Pathvqa: 30000+ questions for medical visual question answering

    He, X., Zhang, Y ., Mou, L., Xing, E., and Xie, P. Pathvqa: 30000+ questions for medical visual question answering. arXiv preprint arXiv:2003.10286,

    arXiv 2003

  7. [13]

    Hui, B., Yang, J., Cui, Z., Yang, J., Liu, D., Zhang, L., Liu, T., Zhang, J., Yu, B., Lu, K., et al. Qwen2. 5-coder technical report.arXiv preprint arXiv:2409.12186,

    arXiv

  8. [14]

    P., Perelman, A., Ramesh, A., Clark, A., Ostrow, A., Welihinda, A., Hayes, A., Radford, A., et al

    Hurst, A., Lerer, A., Goucher, A. P., Perelman, A., Ramesh, A., Clark, A., Ostrow, A., Welihinda, A., Hayes, A., Radford, A., et al. Gpt-4o system card.arXiv preprint arXiv:2410.21276,

    arXiv

Show all 36 references
  1. [15]

    Livecodebench: Holistic and contamination free eval- uation of large language models for code.arXiv preprint arXiv:2403.07974,

    5 Submission and Formatting Instructions for ICML 2025 Jain, N., Han, K., Gu, A., Li, W.-D., Yan, F., Zhang, T., Wang, S., Solar-Lezama, A., Sen, K., and Stoica, I. Livecodebench: Holistic and contamination free eval- uation of large language models for code.arXiv preprint arX...

    2025 arXiv

  2. [16]

    E., Yang, J., Wettig, A., Yao, S., Pei, K., Press, O., and Narasimhan, K

    Jimenez, C. E., Yang, J., Wettig, A., Yao, S., Pei, K., Press, O., and Narasimhan, K. Swe-bench: Can language mod- els resolve real-world github issues?arXiv preprint arXiv:2310.06770,

    arXiv

  3. [17]

    of satellite sensors. MODIS (Moderate Resolution Imaging Spectrora- diometer) observations are most numerous due to its daily temporal resolution and complementary morning and afternoon observations from MODIS Terra and MODIS Aqua satellites. The second highest is Landsat (Lan...

    1972

  4. [19]

    Spider 2.0: Evaluat- ing language models on real-world enterprise text-to-sql workflows.arXiv preprint arXiv:2411.07763,

    Lei, F., Chen, J., Ye, Y ., Cao, R., Shin, D., Su, H., Suo, Z., Gao, H., Hu, W., Yin, P., et al. Spider 2.0: Evaluat- ing language models on real-world enterprise text-to-sql workflows.arXiv preprint arXiv:2411.07763,

    arXiv

  5. [20]

    Unprecedented lake expansion in 2017–2018 on the tibetan plateau: Processes and envi- ronmental impacts.Journal of Hydrology, 619:129333,

    Lei, Y ., Yao, T., Sheng, Y ., Yang, K., Yang, W., Li, S., Zhou, J., Jiang, Y ., and Yu, Y . Unprecedented lake expansion in 2017–2018 on the tibetan plateau: Processes and envi- ronmental impacts.Journal of Hydrology, 619:129333,

    2017

  6. [21]

    Show me what and where has changed? ques- tion answering and grounding for remote sensing change detection.arXiv preprint arXiv:2410.23828, 2024a

    Li, K., Dong, F., Wang, D., Li, S., Wang, Q., Gao, X., and Chua, T.-S. Show me what and where has changed? ques- tion answering and grounding for remote sensing change detection.arXiv preprint arXiv:2410.23828, 2024a. Li, M., Weng, B., Yan, D., Bi, W., and Wang, H. Variation t...

    arXiv

  7. [22]

    H., Ji, S., Lee, B., Yan, X., et al

    Li, Z., Yang, X., Choi, K., Zhu, W., Hsieh, R., Kim, H., Lim, J. H., Ji, S., Lee, B., Yan, X., et al. Mmsci: A dataset for graduate-level multi-discipline multimodal scientific understanding.arXiv preprint arXiv:2407.04903, 2024b. Liang, Z., Guo, K., Liu, G., Guo, T., Zhou, Y ...

    arXiv

  8. [23]

    Deepseek-v3 technical report.arXiv preprint arXiv:2412.19437,

    Liu, A., Feng, B., Xue, B., Wang, B., Wu, B., Lu, C., Zhao, C., Deng, C., Zhang, C., Ruan, C., et al. Deepseek-v3 technical report.arXiv preprint arXiv:2412.19437,

    arXiv

  9. [24]

    NASA earth observatory articles

    NASA. NASA earth observatory articles. https:// earthobservatory.nasa.gov. Nasiri, V ., Deljouei, A., Moradi, F., Sadeghi, S. M. M., and Borz, S. A. Land use and land cover mapping using sentinel-2, landsat-8 satellite images, and google earth en- gine: A comparison of two com...

    1977

  10. [25]

    Know what you don’t know: Unanswerable questions for squad.arXiv preprint arXiv:1806.03822,

    Rajpurkar, P., Jia, R., and Liang, P. Know what you don’t know: Unanswerable questions for squad.arXiv preprint arXiv:1806.03822,

    arXiv

  11. [26]

    Reflexion: Language agents with verbal rein- forcement learning.Advances in Neural Information Processing Systems, 36:8634–8652,

    6 Submission and Formatting Instructions for ICML 2025 Shinn, N., Cassano, F., Gopinath, A., Narasimhan, K., and Yao, S. Reflexion: Language agents with verbal rein- forcement learning.Advances in Neural Information Processing Systems, 36:8634–8652,

    2025

  12. [27]

    L., Pak, J

    Swanson, K., Wu, W., Bulaong, N. L., Pak, J. E., and Zou, J. The virtual lab: Ai agents design new sars-cov-2 nanobod- ies with experimental validation.bioRxiv, pp. 2024–11,

    2024

  13. [29]

    Watch, G. F. Global forest watch.World Resources Institute, Washington, DC Available from http://www. globalforest- watch. org (accessed March 2002),

    2002

  14. [31]

    Accelerating thermokarst lake changes on the qinghai–tibetan plateau.Scientific Reports, 14(1):2985, 2024a

    Zhou, G., Liu, W., Xie, C., Song, X., Zhang, Q., Li, Q., Liu, G., Li, Q., and Luo, B. Accelerating thermokarst lake changes on the qinghai–tibetan plateau.Scientific Reports, 14(1):2985, 2024a. Zhou, Y ., Liu, B., Cui, Y ., Wang, X., Cao, M., Zhang, S., Xiao, X., and Dong, J. ...

    1991

  15. [33]

    The image from December 18, 2000, shows a mix of cleared land and greener areas

    2 Life Does forest cover decrease in Argentina’s Salta Province from December 2000 to December 2019? The images above show deforestation over a span of two decades around the Salta Province of northern Argentina. The image from December 18, 2000, shows a mix of cleared land an...

    2000

  16. [35]

    is a cloud-based platform that enables users to perform geospatial analysis at a planetary scale using Google’s computational infrastructure. It houses over 90 petabytes of analysis-ready satellite imagery and more than 1,000 curated geospatial datasets spanning 50+ years of h...

    2017

  17. [2005]

    R., Zhang, S., Sun, Y ., and Wang, W

    Wang, X., Hu, Z., Lu, P., Zhu, Y ., Zhang, J., Subramaniam, S., Loomba, A. R., Zhang, S., Sun, Y ., and Wang, W. Scibench: Evaluating college-level scientific problem- solving abilities of large language models.arXiv preprint arXiv:2307.10635,

    arXiv

  18. [2012]

    Campbell, Q., Cox, S., Medina, J., Watterson, B., and White, A. D. Mdcrow: Automating molecular dynamics workflows with large language models.arXiv preprint arXiv:2502.09565,

    arXiv

  19. [2014]

    4 Snow / ice Does Lake Erie have more ice coverage compared to the other Great Lakes in February 14, 2018 afternoon? On the same date last year, total ice cover was 9.7 percent

    Green shading marks areas where light usage increased in December. 4 Snow / ice Does Lake Erie have more ice coverage compared to the other Great Lakes in February 14, 2018 afternoon? On the same date last year, total ice cover was 9.7 percent. Lake Erie was the iciest of the ...

    2018

  20. [2017]

    The llama 3 herd of models.arXiv preprint arXiv:2407.21783,

    Grattafiori, A., Dubey, A., Jauhri, A., Pandey, A., Kadian, A., Al-Dahle, A., Letman, A., Mathur, A., Schelten, A., Vaughan, A., et al. The llama 3 herd of models.arXiv preprint arXiv:2407.21783,

    arXiv

  21. [2018]

    Yang, A., Yang, B., Zhang, B., Hui, B., Zheng, B., Yu, B., Li, C., Liu, D., Huang, F., Wei, H., et al. Qwen2. 5 technical report.arXiv preprint arXiv:2412.15115, 2024a. Yang, J., Jimenez, C. E., Zhang, A. L., Lieret, K., Yang, J., Wu, X., Press, O., Muennighoff, N., Synnaeve, ...

    arXiv

  22. [2019]

    Chen, M., Tworek, J., Jun, H., Yuan, Q., Pinto, H. P. D. O., Kaplan, J., Edwards, H., Burda, Y ., Joseph, N., Brockman, G., et al. Evaluating large language models trained on code.arXiv preprint arXiv:2107.03374,

    arXiv

  23. [2020]

    Measuring mas- sive multitask language understanding.arXiv preprint arXiv:2009.03300,

    Hendrycks, D., Burns, C., Basart, S., Zou, A., Mazeika, M., Song, D., and Steinhardt, J. Measuring mas- sive multitask language understanding.arXiv preprint arXiv:2009.03300,

    2009 arXiv

  24. [2021]

    Auto-bench: An automated benchmark for scien- tific discovery in llms.arXiv preprint arXiv:2502.15224, 2025a

    Chen, T., Anumasa, S., Lin, B., Shah, V ., Goyal, A., and Liu, D. Auto-bench: An automated benchmark for scien- tific discovery in llms.arXiv preprint arXiv:2502.15224, 2025a. Chen, X., Wang, T., Guo, T., Guo, K., Zhou, J., Li, H., Song, Z., Gao, X., and Zhang, X. Unveiling th...

    arXiv

  25. [2022]

    M., Janizek, J

    Laurent, J. M., Janizek, J. D., Ruzo, M., Hinks, M. M., Hammerling, M. J., Narayanan, S., Ponnapati, M., White, A. D., and Rodriques, S. G. Lab-bench: Measuring capa- bilities of language models for biology research.arXiv preprint arXiv:2407.10362,

    arXiv

  26. [2023]

    M., Bowman, S

    Kim, N., Htut, P. M., Bowman, S. R., and Petty, J. (qa)2: Question answering with questionable assump- tions.arXiv preprint arXiv:2212.10003,

    arXiv

  27. [2024]

    Sci- knoweval: Evaluating multi-level scientific knowledge of large language models.arXiv preprint arXiv:2406.09098,

    Feng, K., Ding, K., Wang, W., Zhuang, X., Wang, Z., Qin, M., Zhao, Y ., Yao, J., Zhang, Q., and Chen, H. Sci- knoweval: Evaluating multi-level scientific knowledge of large language models.arXiv preprint arXiv:2406.09098,

  28. [2025]

    Chen, B., Xiao, X., Li, X., Pan, L., Doughty, R., Ma, J., Dong, J., Qin, Y ., Zhao, B., Wu, Z., et al. A mangrove for- est map of china in 2015: Analysis of time series landsat 7/8 and sentinel-1a imagery in google earth engine cloud computing platform.ISPRS Journal of Photogr...

    2015

Pith tools

Reviewed August 16, 2026 · model on record in the stance chip above.