{"id":"a967b212-0286-4989-b4cf-61981bc09ede","arxiv_id":"2504.12110","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"On a new 140-question Earth observation benchmark, the best LLM agent scores 33% accuracy with Google Earth Engine access because generated code fails to run over 58% of the time.","lead":"This paper introduces UnivEARTH, a benchmark of 140 yes/no questions about Earth observation drawn from NASA articles, and tests whether AI agents can answer them by writing code for Google Earth Engine. Even the best AI model answered only about a third of the questions correctly, because generated code failed to run more than half the time.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Ground-truth labels are verified against article text and images, not against GEE computations, so the 33% accuracy figure may conflate LLM failure with benchmark answerability failure.","rationale":"The reader's weakest assumption is exactly the one I judge most load-bearing: the ground-truth answers are article-derived and reviewer-checked, but not validated by GEE computation. The paper's own description of verification (Sec. 2.2) supports this reading: dataset availability is verified, but recovery of the specific article answer is not demonstrated. Appendix E confirms that human review answers from text and images, so inter-reviewer agreement does not establish GEE answerability. The fine-tuning result and the 33% accuracy both inherit whatever label noise or answerability mismatch exists, so this one gap threatens the central negative claim more than the small dataset size or missing OpenEARTH description. I therefore agree with the reader's conditionality. I did not find a separate concern that would move the verdict: the evaluation is multi-model, multi-trial, and the failure-rate analysis is internally coherent. If the proposed reference-implementation check fails, a later pass should lower the verdict, but relative to the current CONDITIONAL recommendation no adjustment is needed now.","tokens_in":15113,"tokens_out":4512,"duration_ms":45208,"concrete_test":"For each of the 140 questions, have two independent human experts familiar with GEE write and execute reference Python code using only public GEE collections and explicit regions/thresholds, recording Yes/No/Inconclusive for each. Compare these reference outputs to the released ground-truth labels. Then recompute the zero-shot, few-shot, and reflexion accuracy and failure rates from Table 1 restricted to the subset of questions where both reference implementations agree with the label. If reference implementations cannot recover a substantial fraction (e.g., more than 10%) of the labels, the 33% headline and the 'guaranteed question answerability' claim should be revised; if they recover all labels, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central negative result rests on the claim (Sec. 2.2, contribution list) that all 140 UnivEARTH questions are 'reasonably answerable' via Google Earth Engine with verified answers. The verification described in Sec. 2.2 only says the authors wrote test implementations, verified dataset availability, and sometimes identified alternative data sources; it does not say they computed the article-derived answer from GEE for each question. The independent review (Appendix E) asks reviewers to answer from article text and images (Q1-Q3), with Google Maps only for locations (Q4), so reviewer agreement (90.1% on Q1) validates labels against the articles, not against GEE data. If many article conclusions cannot be recovered by any competent GEE query—because of transient events, mixed pixels, threshold sensitivity, or missing products—then the 33% accuracy and 58% failure rate measure benchmark ambiguity or an overly strict execution environment, not LLM incapability. This is an internal validation gap, not a disagreement with external consensus, and it is load-bearing because the headline claim is precisely about grounded, executable GEE analysis.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces UnivEarth, a benchmark of 140 yes/no questions derived from NASA Earth Observatory articles, spanning 13 topics and 17 satellite sensors. The authors evaluate LLM agents that must answer the questions by generating and executing Python code against the Google Earth Engine API. They report that the best agents achieve only 33% accuracy because, they claim, code fails to run over 58% of the time; they also show that fine-tuning a Llama-3.1-8B model on synthetic data improves its accuracy to about 25%, comparable to much larger models. The central claim is that current AI systems cannot yet reliably support Earth observation workflows when answers must be grounded in executable analysis.","tokens_in":15302,"tokens_out":3983,"duration_ms":42844,"significance":"If the central result holds, the paper makes a useful contribution: it is among the first benchmarks to require LLM agents to ground Earth-observation answers in executable Google Earth Engine code, rather than relying on parametric knowledge. The curation process is thoughtful in several respects: using NASA Earth Observatory articles as an authoritative source, combining LLM-assisted question generation with manual editing, and running an independent human review that reached full agreement on the yes/no labels. The evaluation covers a broad set of commercial and open models, and the synthetic fine-tuning experiment is a constructive step toward improving smaller models. The dataset could be a valuable resource for future work on tool-using agents for scientific data analysis, provided the validation issues described below are addressed.","major_comments":[{"comment":"The benchmark's central claim that all 140 questions are 'reasonably answerable' via Google Earth Engine is not actually validated. Section 2.2 states that the authors 'wrote test implementations,' 'verified dataset availability,' and 'sometimes identified alternative data sources,' but it never states that they computed the article-derived ground-truth answer from GEE data for each question. The independent review described in Appendix E asks reviewers to answer from the article text and images, with Google Maps only for location checks; it does not validate that the ground-truth answer is recoverable by any competent GEE query. If many article conclusions cannot be reproduced from GEE data because of transient events, mixed pixels, threshold sensitivity, or missing products, then the reported 33% accuracy and 58% failure rate measure benchmark ambiguity or execution-environment strictness rather than LLM incapability. This is load-bearing because the headline claim is precisely about grounded, executable GEE analysis. The paper should provide, for each question, a reference GEE implementation that reproduces the ground-truth label, or otherwise demonstrate that the labels are computable from GEE data.","section":"2.2, Appendix E"},{"comment":"The abstract states that accuracy is low 'because the code fails to run over 58% of the time,' but the failure rate in Table 1 is defined more broadly. Section 3.1 says failure is recorded when 'the code was incorrect, or the data requested by the code was not available,' which includes cases where code executed but accessed unavailable data. It also includes cases where the model produced no answer because it could not complete the task. The paper should decompose the failure rate into code execution errors, wrong asset or data errors, and other causes, and should rephrase the abstract and conclusion so that the causal claim is not stronger than what Table 1 actually measures.","section":"3.1, Table 1"},{"comment":"One of the showcased examples is internally inconsistent. The Atmosphere row asks 'Did nitrogen oxide concentrations in the Northern Hemisphere increase from 2019 to 2020?' but the supporting sentence is about methane measurements: 'The annual growth rate for 2020 was the highest scientists had recorded since systematic annual methane measurements began in 1983.' If this is a typo, it should be corrected; if not, it indicates that some question-answer pairs may not be grounded in the cited article text, which would undermine the integrity of the benchmark labels. Please clarify.","section":"Table 3"}],"minor_comments":[{"comment":"The phrase 'an novel evaluation benchmark' contains a grammatical error; it should be 'a novel evaluation benchmark.'","section":"1. Contributions"},{"comment":"The correlations (r = 0.87 and r = -0.88) are computed over only ten model points; the paper should report confidence intervals or a permutation test to show that the correlation is not driven by one or two outliers.","section":"Figure 2"},{"comment":"The sentence 'even the best models achieving only 49.0% accuracy without internet access' should specify that this is the three-option no-internet scenario; in the two-option scenario the best model achieves 83.8%, so the unsupported reading is misleading.","section":"4. Conclusion"},{"comment":"The claim that all questions are 'reasonably answered' via GEE would benefit from a precise definition of what counts as reasonable, including how thresholding ambiguities and transient phenomena are handled.","section":"2.2"}],"recommendation":"major_revision","confidential_remarks":"The core issue is fixable but requires real work: the authors should provide a reference GEE script and recomputed answer for every benchmark question, and release those scripts alongside the dataset. If that validation is done and the example inconsistency in Table 3 is resolved, the paper would make a solid contribution. The omission is not a minor oversight; it directly affects the interpretation of the headline accuracy figure."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Read Kao et al. The punchline: UnivEARTH is the first benchmark I've seen that combines NASA Earth Observatory article QA with executable Google Earth Engine (GEE) code as required evidence. The design is sound: yes/no questions with an abstain option, local execution of agent-generated code, and answers parsed from output. The headline result is that the best agent (Claude-3.7-Sonnet with reflexion) gets only ~33% accuracy, mostly because code fails to run over 58% of trials. That's a stark, useful signal for anyone building scientific agents.\n\nThe strongest part, besides the benchmark itself, is the analysis showing a strong correlation (r=0.87) between zero-shot accuracy and the number of distinct GEE imagery collections used, and a negative correlation (r=-0.88) with 'wrong asset name' errors. That's a concrete, plausible explanation for the failure mode, even if it's based on only ten model points. The fine-tuning result—Llama-3.1-8B reaching 25% accuracy after training on synthetic data—is interesting, but the synthetic data (OpenEARTH) is described so vaguely that I can't reproduce or fully trust it.\n\nNow the soft spot, and it's a real one. The paper's central interpretive claim is that the 33% figure shows LLMs cannot reliably produce executable, grounded EO analysis. But Section 2.2's 'Question Verification' says only that the authors 'wrote test implementations... verified dataset availability, and in some cases identified alternative data sources.' It does not say they computed the ground-truth answer from GEE for each of the 140 questions. The independent review (Appendix E) asks reviewers to answer from article text and images (Q1) with Google Maps only for location (Q4), so the labels are validated against articles, not against GEE computation. If a meaningful fraction of the questions are not actually recoverable from GEE—transient events, mixed pixels, thresholds, missing products—then the 33% accuracy conflates LLM failure with benchmark ambiguity. This isn't a fatal flaw: the authors clearly did hands-on GEE testing (the snow example in Figure 1 shows numeric comparison), and the 58% code-failure rate is about execution, not labels, so the qualitative conclusion probably survives. But it's an internal validation gap that needs to be closed in the paper or the released artifacts.\n\nWho should read this: people building LLM agents for scientific data analysis, benchmark designers, and remote sensing folks. It deserves a serious referee—send it out, don't desk-reject. I'd make acceptance conditional on releasing the dataset, evaluation code, and per-question GEE verification evidence, plus error bars and a real description of the synthetic data. My own verdict is conditional: the benchmark could be a solid community resource, but I wouldn't fully trust the headline number until the verification gap is addressed.","headline":"A useful new benchmark for LLM agents in Earth Observation, but the headline accuracy figure rests on a verification step the paper doesn't actually describe.","tokens_in":15853,"tokens_out":4743,"would_cite":true,"duration_ms":45088,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":null,"created_at":"2026-08-16T12:37:19.485575+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":null,"supporting_citations":[],"review_version":1}