{"id":"73e90820-795d-45e4-a554-7dd1cfb8ca1b","arxiv_id":"2506.10365","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"AutoGEEval++ is a 6,365-case automated benchmark and evaluation pipeline that ranks 24 LLMs on Google Earth Engine code generation across unit, combo, and theme tasks.","lead":"AutoGEEval++ automatically tests how well large language models write Google Earth Engine code, using 6,365 tasks across 26 data types. It ranks 24 models on simple, multi-step, and full-scenario coding, giving geospatial developers a common benchmark.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Ground-truth answers are authored by DeepSeek-V3, which is itself among the 24 evaluated models; the §3.4.1 assertion that this choice introduces no bias is unargued, and if wrong it directly undermines the reported rankings.","rationale":"The reader identified the same load-bearing assumption I would: the benchmark's ground truth is partly authored by DeepSeek-V3, one of the evaluated models, and the paper's only response is a flat assertion in §3.4.1 that no bias is introduced. This is the correct target because every number in Tables 9–12 is a comparison against that ground truth; if the reference code embeds DeepSeek-V3's conventions, the unit-test leaderboard is not measuring geospatial coding ability so much as stylistic similarity to the test generator. The paper does contain real assets—a released repository, a 295,943-script corpus, a 24-model comparison, and detailed error logging—so the issue is not fabrication or negligence but missing validation. A contamination analysis and independent re-derivation of ground truth would settle the question. I also considered the 'first automated assessment framework' novelty claim, but that is secondary: even if prior work (including the authors' own AutoGEEval and the GeoSpatial-Code-LLMs Dataset) undercuts 'first,' the benchmark and rankings could still be valid, whereas reference-bias undermines the core measurements. The theme-test set is small (n=88) and the headline Theme_SA differences of a few points may be within noise, but the reference-authoring concern is more fundamental. For these reasons I keep the reader's CONDITIONAL verdict: the framework is promising and publishable after the bias check and ranking uncertainty analysis are supplied.","tokens_in":31246,"tokens_out":5156,"duration_ms":65940,"concrete_test":"Re-score a stratified 10% sample of the 5,078 unit tests and all 88 theme tests using independently authored reference implementations: recruit two external GEE programmers, give them only the function declarations and prompts, ask them to write and execute reference code and produce expected outputs without seeing any DeepSeek-V3-generated references, then run the existing Judge on all 24 models under both the original and the independent ground truths. If agreement between original and independent ground truth is below 95%, or if any top-5 Unit_SA or Theme_SA ranking changes, the §3.4.1 no-bias assertion fails and the rankings should be re-reported with uncertainty intervals.","verdict_should_be":"UNCHANGED","load_bearing_attack":"AutoGEEval++'s central claim is that 6,365 test cases and an execution-based judge give an unbiased, standardized measurement of GEE code-generation ability. The load-bearing condition is that the reference implementations and expected answers are correct, neutral ground truth, because Eq. (4) and Table 6 score every model by comparing its output to the expected_answer produced by executing the reference code. Section 3.4.1 dismisses the generator-bias risk in one sentence: 'the final evaluation is based on execution success rather than code quality—this model choice does not introduce bias.' That sentence is not an argument. A reference implementation that executes but is wrong, overfitted to DeepSeek-V3's parameter conventions, or contains a silent bug will mark an alternative correct solution as incorrect. The same model was used to generate combo-test questions (§3.5.3) and to translate theme-test code (§3.6), so the ground truth in all three task types is partly authored by one of the evaluated systems (DeepSeek-V3 and its 0324 variant appear in Tables 9–12). No contamination check, no inter-annotator agreement measure for the three-expert review, and no sensitivity analysis are reported. Since DeepSeek-V3 ranks first on Unit_SA, this is not a pedantic point: the headline ranking may reflect test-author style rather than geospatial coding ability. The small theme-test size (n=88) makes the Theme_SA rankings additionally fragile to a few mislabeled references.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents AutoGEEval++, an automated evaluation framework for large language models generating Google Earth Engine (GEE) Python code. The framework consists of a benchmark (AutoGEEval++-Bench) with 6,365 test cases across 26 GEE data types and three task levels (unit, combo, theme), a submission program that prompts models to generate and execute code, and a judge program that compares outputs against reference answers. The authors evaluate 24 LLMs covering general-purpose, reasoning, code-specialized, and geoscience-specific models, reporting accuracy (pass@n and stability-adjusted accuracy), resource consumption, efficiency, boundary-test pass rates, and error-type distributions. The headline results include rankings such as DeepSeek-V3 first on Unit_SA, GPT-4.1-mini first on Combo_SA, and o4-mini first on Theme_SA.","tokens_in":1866,"tokens_out":3026,"duration_ms":83645,"significance":"If the benchmark's ground truth is unbiased, this is a substantial contribution to geospatial code-generation evaluation. The benchmark is large, publicly released (GitHub link provided), and spans multiple task complexities and data modalities. The end-to-end execution-based judge and multidimensional metrics (accuracy, resource use, efficiency, error types) are a practical and reproducible protocol that goes beyond prior manual evaluations and the earlier AutoGEEval. The paper also ships a detailed construction methodology (AST parsing, PrefixSpan mining, expert review) and a broad model comparison. However, the validity of every reported number rests on the correctness and neutrality of the reference implementations, and the unresolved generator-bias issue (DeepSeek-V3 authored the references and is itself an evaluated model) means the rankings may be artifacts of test-author style. The significance is therefore conditional on addressing that concern and on correcting the reproducibility contradictions in the combo-test construction.","major_comments":[{"comment":"The reference implementations and expected answers for all three task types are authored or translated by DeepSeek-V3, which is itself one of the 24 evaluated models. The sentence in §3.4.1 — \"the final evaluation is based on execution success rather than code quality—this model choice does not introduce bias\" — is asserted rather than demonstrated. Since Eq. (4) scores every model by comparing its output to the expected_answer produced by executing the reference code, any systematic convention or error in DeepSeek-V3's reference code will directly affect all accuracy and SA numbers. Because DeepSeek-V3 ranks first on Unit_SA (Table 12), this is not a purely theoretical concern. Please provide a contamination analysis (e.g., check whether the benchmark cases appear in evaluated models' training data), a differential analysis on a subset of cases with independently written references, or a quantitative inter-annotator agreement measure for the three-expert review in Table 4. Without one of these, the unbiasedness claim is unsupported.","section":"§3.4.1, §3.5.3, §3.6; Tables 9–12"},{"comment":"The text states: \"the support threshold for official combinations was finally set to 0.10, resulting in 531 frequent combinations; for user-defined combinations, the threshold was set to 0.04, extracting 484 frequent combinations.\" Table 3, however, places the checkmark under User-Defined Function at support 0.10 (531 sequences) and under Official Function at support 0.04 (484 sequences). These are mutually contradictory. Since the 1,199 combination patterns are the union of the three mined sets, this contradiction prevents a reader from reproducing the combo-test construction. Please correct the text or the table and confirm the actual thresholds.","section":"§3.5.2, Table 3"},{"comment":"The theme test set contains only 88 cases, so Theme@5 for each model is based on 440 executions but only 88 independent tasks. The Theme_SA rankings in Table 12 that decide the headline result (o4-mini at 55.639, Claude3.7-Sonnet at 55.308) correspond to a difference of a few cases. The paper provides no confidence intervals, bootstrap estimates, or sensitivity analysis for the theme-level results, despite the claim in §3.3 that theme tests are the \"highest complexity and most realistic\" task. Please quantify the fragility of these rankings or explicitly acknowledge this limitation in Section 7.3.","section":"§6.1, Table 11"}],"minor_comments":[{"comment":"All three tables are captioned \"Accuracy results for unit tests.\" Table 10 reports combo tests and Table 11 reports theme tests; the captions should be corrected.","section":"Tables 9–11"},{"comment":"The text says the GEE Reference covers 1,374 function entries and 43 deprecated entries were excluded, yielding 1,325 functions; 1,374 − 43 = 1,331, so the numbers need reconciliation.","section":"§3.4.1"},{"comment":"The pass@n formula is garbled in the manuscript and the meaning of N and c is not fully defined; please rewrite with clear binomial coefficients and specify how n = 1, 3, 5 is used in the evaluation.","section":"§5.3.1, Eq. (7)"},{"comment":"The Coefficient of Variation (CV) is defined as the ratio of standard deviation to mean, but it is not specified over which set of measurements the standard deviation is computed (e.g., pass@1/3/5 scores, or repeated runs). Please clarify.","section":"§5.3.1"},{"comment":"The text says \"Figure 18 shows the distribution of Coefficient of Variation (CV)\", but Figure 18 is the stacked accuracy plot and the CV distribution appears in Figure 19. The citation should be corrected.","section":"§6.1"},{"comment":"The table caption says \"AutoGEEval-Bench\" instead of \"AutoGEEval++-Bench\", and the column abbreviation \"S.T.\" is undefined (elsewhere the paper uses \"T.T.\" for theme tests).","section":"Table 5"},{"comment":"The model name \"Owen-3-Thinking\" should be \"Qwen-3-Thinking\".","section":"Table 7"},{"comment":"The paper claims AutoGEEval++ is \"the first automated assessment framework tailored for LLMs performing geospatial code generation on the GEE platform\", yet §2.3 states that AutoGEEval is \"the first automated evaluation system for geospatial code targeting the GEE\". Please clarify what exactly AutoGEEval++ is first to do to avoid an internally inconsistent novelty claim.","section":"Abstract, §1, §2.3"},{"comment":"The limitations section does not mention the generator-bias risk for the reference implementations or the small size of the theme test set; both should be explicitly acknowledged given their impact on the reported rankings.","section":"§7.3"}],"recommendation":"major_revision","confidential_remarks":"The paper is within scope for a software-engineering or AI-evaluation venue and the authors have built a large, open, and potentially reusable resource. The main risk is the unresolved benchmark-author bias: DeepSeek-V3 authored the references and is also the top-ranked model on Unit_SA. This is not a fatal flaw in principle, but the current one-sentence dismissal in §3.4.1 is insufficient and should be addressed with additional analysis or a carefully worded limitation. The internal contradiction in Table 3 regarding the PrefixSpan thresholds is also easily fixable but currently blocks reproducibility. I recommend major revision rather than rejection because the load-bearing issues are addressable within the paper's scope."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know before you read it. This is a real contribution: a 6,365-case benchmark for GEE code generation with unit, combo, and theme tasks, an execution-based judge, and a 24-model comparison. It also has a load-bearing weakness: the reference code under every test case was written by DeepSeek-V3, the same model that tops the headline ranking, and the paper's only defense is one sentence asserting no bias.\n\nWhat's good: the combo-test construction is genuinely interesting—295k real-world GEE scripts parsed into ASTs, frequent patterns mined with PrefixSpan, reconstructed into test cases. The three-tier task structure is well designed and the difficulty gradient (unit < combo < theme) shows up in the results. The judge program's type-specific comparison logic is thoughtful. The error taxonomy (parameter, output type, wrong answer, syntax, network) and the resource metrics add useful diagnostic depth. This goes well beyond their own AutoGEEval, which was single-function only.\n\nNow the soft spots, in proportion. The abstract's \"first automated assessment system\" claim directly contradicts their own Section 2.3, which credits AutoGEEval with being the first automated evaluation system for GEE. That's an easy fix: say \"first multimodal, multi-tier\" or drop \"first.\" More serious is the DeepSeek-V3 issue. Section 3.4.1 says the choice \"does not introduce bias\" because evaluation is execution-based rather than code-quality-based. That is not an argument. If DeepSeek-V3's reference implementations have systematic quirks—parameter conventions, error handling, output formatting—models that resemble DeepSeek-V3 will be favored. Since DeepSeek-V3 tops Unit_SA, you can't rule out that the ranking is partly a test-author effect. No contamination analysis, no sensitivity analysis, no inter-annotator agreement for the three-expert review. That's not fatal to the benchmark as a resource, but it is fatal to taking the rankings at face value. The theme test is only 88 cases; a few mislabeled references could shift the ordering, and there are no error bars. Tables 9–11 are mislabeled (each says \"Accuracy results for unit tests\"), and Eq. (7), the pass@n formula, is garbled.\n\nBottom line: this is a useful benchmark for geoAI and code-generation evaluation, worth having despite the issues. It deserves a serious referee, and a competent one will send it back for revisions: fix the \"first\" claim, provide bias analysis or at least a sensitivity check, correct the tables and formula. If you work on geospatial code evaluation, cite it.","headline":"Useful benchmark resource with a genuine design contribution, but the 'first' claim is overstated and the DeepSeek-V3-authored ground truth puts the headline rankings on shaky ground.","tokens_in":32171,"tokens_out":3878,"would_cite":true,"duration_ms":42903,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that LLM-generated Google Earth Engine code can be evaluated automatically, end to end, with a 6,365-case benchmark and an execution-based judge that ranks 24 models.","keywords":["geospatial code generation","large language models","Google Earth Engine","automated evaluation","LLM benchmark","code generation evaluation","execution-based testing"],"falsifier":"Re-run the benchmark with reference implementations written independently, for example by human experts or by a different LLM, and compare the rankings; if DeepSeek-V3's relative standing drops when the reference generator changes, the reported leaderboard is a reference-bias artifact. A simpler check is to measure whether DeepSeek-V3-generated code adopts parameter conventions from the reference code more often than other models' code does.","tokens_in":31037,"feed_emoji":"🛰️","tokens_out":6812,"duration_ms":72929,"temperature":0.7,"pith_summary":"AutoGEEval++ is presented as the first automated assessment framework for LLMs generating geospatial code on Google Earth Engine. It contributes a benchmark of 6,365 test cases—unit calls, multi-function combinations mined from real scripts, and full research-style workflows—plus an execution-based judge that compares generated output against reference answers without human inspection. The authors run 24 models released before June 2025 and report that DeepSeek-V3 leads unit tests, GPT-4.1-mini leads combo tests, and o4-mini leads theme tests on a stability-adjusted accuracy metric. If the framework is sound, it would make geospatial code generation a measurable, reproducible benchmark task rather than a manual review exercise, with concrete resource and error diagnostics for model comparison.","feed_headline":"Automated judge evaluates 24 LLMs on 6,365 Earth Engine coding tasks","feed_subtitle":"Execution-based checks against reference answers replace manual review for GEE code benchmarks.","key_machinery":"The load-bearing mechanism is the type-aware Judge program combined with the six-tuple test-case schema. Because GEE's 26 data types share numerical representations, the judge maps each output type to a comparison strategy—array element comparison, key-value dictionary comparison, GeoJSON geometry consistency, or float equality with tolerance—which turns raw execution results into automated pass/fail decisions. That mapping is what lets the pipeline run end to end without human judgment. Reference implementations generated by DeepSeek-V3 and reviewed by three experts supply the ground truth that every model is graded against.","core_discovery":"The central claim is that a fully automated, execution-based evaluation of LLM-generated GEE code is feasible and produces a usable model ranking. Each test case is a six-tuple specifying a function declaration, a reference implementation, a parameter list, an expected output type, an output path, and a ground-truth answer; the judge program chooses a comparison strategy based on the output type's representation and verifies numeric or structural equality with tolerance. The framework also records token usage, inference time, and error categories. Across the three task levels the paper finds a consistent combo-over-unit-over-theme performance hierarchy, with multi-round generation helping mainly between the first and third attempts.","pith_inferences":["The paper does not test whether DeepSeek-V3's role as reference-code generator leaks into the rankings; comparing model outputs' parameter conventions and formatting against the reference style would reveal whether the reported leaderboard is partly a reference-bias artifact.","A natural extension is to port the type-to-representation judge mapping to other cloud geospatial APIs or domain-specific code platforms, using this benchmark's coverage pattern as a template.","Because the theme-test set has only 88 cases, its accuracy estimates carry wide confidence intervals; expanding it would likely reshuffle the theme ranking more than the unit or combo rankings.","The reported error-type patterns, such as the high syntax-error share in one reasoning-model family, are actionable training signals for model developers rather than just evaluation outputs."],"forward_implications":["GEE code generation can be benchmarked reproducibly without manual annotation, enabling direct comparison of future models on the same 6,365 cases.","Model rankings differ by task level: the paper's data imply that no single model dominates, with reasoning models excelling on theme tasks while lightweight general models can lead on combo tasks.","Multi-round sampling improves accuracy mostly from pass@1 to pass@3; further rounds yield diminishing returns, so evaluation and deployment budgets should reflect that.","Error-type logging separates parameter errors, type mismatches, wrong answers, syntax errors, and timeouts, giving model developers a targeted list of failure modes.","The 7B-parameter domain-tuned model evaluated in the paper ranks fourth in the composite ranking, suggesting vertical fine-tuning can compete with much larger general models."],"supporting_citations":[{"why":"The AutoGEEval predecessor that AutoGEEval++ extends; it supplies the original automated unit-level evaluation approach and the baseline the new framework claims to improve.","marker":"Hou et al., 2025b"},{"why":"The GeoSpatial-Code-LLMs Dataset, the prior automated geospatial code evaluation, which is limited to 40 samples and motivates the larger benchmark.","marker":"Gramacki et al., 2024"},{"why":"GeoCode-Bench, a manual-evaluation baseline that formally defines geospatial code generation as a task and provides the contrast for the automated approach.","marker":"Hou et al., 2024b"},{"why":"DeepSeek-V3 technical report; this model generates the reference implementations and ground-truth answers used throughout the benchmark.","marker":"Liu et al., 2024"},{"why":"HumanEval, the general function-level code benchmark whose evaluation paradigm the paper contrasts with execution-based GEE testing.","marker":"Li and Murr, 2024"},{"why":"LiveCodeBench, a contamination-conscious code evaluation benchmark that represents the general-purpose evaluation standards the framework aims to extend to a vertical domain.","marker":"Jain et al., 2024"},{"why":"Qwen2.5-Coder technical report, one of the evaluated model series and a representative code-specialized baseline in the comparison.","marker":"Hui et al., 2024"}],"fun_headline_variants":["AutoGEEval++ auto-judges 24 LLMs on 6,365 Earth Engine tasks","Execution-based judge ranks 24 LLMs on GEE code generation","Automated benchmark for LLM geospatial code on Earth Engine","AutoGEEval++: 6,365 GEE coding tests scored automatically"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The framework's validity rests on the assumption that the reference implementations and ground-truth answers—drafted by DeepSeek-V3 and reviewed by three experts—are unbiased, so every model is measured against a fair standard rather than a style the reference model happens to share.","fun_headline_variants_meta":{"raw":{"variants":["AutoGEEval++ auto-judges 24 LLMs on 6,365 Earth Engine tasks","Execution-based judge ranks 24 LLMs on GEE code generation","Automated benchmark for LLM geospatial code on Earth Engine","AutoGEEval++: 6,365 GEE coding tests scored automatically"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000456,"raw_usage":{"total_tokens":2307,"prompt_tokens":979,"completion_tokens":1328,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":595,"completion_tokens_details":{"reasoning_tokens":1244}},"tokens_in":595,"tokens_out":1328,"duration_ms":13361,"temperature":1.0,"reasoning_tokens":1244,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T04:28:05.643612+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the benchmark with reference implementations written independently, for example by human experts or by a different LLM, and compare the rankings; if DeepSeek-V3's relative standing drops when the reference generator changes, the reported leaderboard is a reference-bias artifact. A simpler check is to measure whether DeepSeek-V3-generated code adopts parameter conventions from the reference code more often than other models' code does.","supporting_citations":[],"review_version":1}