{"id":"fde1c971-ad47-484a-8642-50cc551d8a39","arxiv_id":"2505.12207","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"AgroMind introduces a 28,482-pair agricultural remote sensing benchmark spanning 13 tasks and 4 ability dimensions, showing current large multimodal models lag on spatial and fine-grained agricultural understanding.","lead":"AgroMind is a new benchmark for testing large multimodal AI models on agricultural remote sensing, built from 10 datasets and 28,482 image-question pairs across 13 task types. Testing 24 AI models, the best model, GPT-4o, beat a student human baseline overall, but all models struggled with spatial reasoning and fine-grained crop recognition.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"No contamination analysis means public-source test images may already be in LMM training corpora; the GPT-4o-vs-human result could reflect memorization.","rationale":"I independently converged on the same load-bearing assumption as the reader: benchmark validity conditional on no train/test contamination. The public-source composition makes this a first-order risk, and no per-source analysis exists to rule it out. The proposed split by private vs. public source is decisive because the Global Parcel Dataset cannot have been in the models' pretraining corpora. I see the open-ended cosine threshold (Appendix C.4, threshold fitted on a benchmark subset) and the difficulty-level circularity (Appendix C.3 defines difficulty from the same student-human accuracies used as baseline) as secondary issues that affect specific analyses but not the main rankings. The count mismatch between the abstract (27,247 QA/19,615 images) and main text (28,482 QA/20,850 images) is a reporting defect that should be corrected, but it does not by itself undermine the central comparison. The reader's CONDITIONAL verdict is appropriate; my concern does not move it.","tokens_in":33139,"tokens_out":6021,"duration_ms":61391,"concrete_test":"Split Table 2 by data source: compute GPT-4o and human accuracy separately on the private Global Parcel subset and on each public-source subset, controlling for task type and difficulty bucket. If GPT-4o's margin over humans shrinks to zero or reverses on the private subset while remaining large on public subsets, contamination of public images is the likely explanation; if the margin persists on private data, the concern is weakened. For open-source models, additionally run perceptual-hash or CLIP-based near-duplicate retrieval of AgroMind images against the models' documented training sets (e.g., LAION-5B, DataComp) and report the overlap rate.","verdict_should_be":"UNCHANGED","load_bearing_attack":"AgroMind's central claim — that current LMMs show large performance gaps and that GPT-4o surpasses human performance (43.14% vs 33.15%, Table 2) — presupposes that scores measure agricultural understanding rather than familiarity with the test images. Nine of the ten constituent sources are public datasets (IP102, Agriculture-Vision, PhenoBench, CropHarvest, OilPalmUAV, ACFR, OAM-TCD, 2018 AI Challenge, Maize Tassel; Section B.1, Table 4). The paper provides no contamination analysis: no deduplication against known pretraining corpora, no near-duplicate check, and no accuracy breakdown by public vs. private source. Section 5.1 describes the evaluation protocol but never tests whether the evaluated LMMs saw these images during pretraining or instruction tuning. The single private source (Global Parcel Dataset) is the natural control, yet Tables 2, 7, and 8 report only aggregate, scene-level, and sensor-level results. If GPT-4o's 9.99-point margin over the student-human baseline is concentrated in public-source subsets, memorized image-label associations could explain it. Section 5.2 itself states that models 'tend to reproduce answers seen in their training data rather than understand the agricultural scene itself,' but this check is not applied to AgroMind's own test set.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"AgroMind is a benchmark for evaluating large multimodal models (LMMs) on agricultural remote sensing. The authors combine nine public datasets and one privately collected global parcel dataset to produce a test set that, in the main text, contains 20,850 images and 28,482 QA pairs organized into 13 task types across four dimensions: spatial perception, object understanding, scene understanding, and scene reasoning. A pipeline is presented for data pre-processing, question generation, and quality control. The authors evaluate 20 open-source and 4 closed-source LMMs in a zero-shot setting, along with a random baseline and a baseline of student volunteers, and report that leading closed models such as GPT-4o exceed the human baseline overall, while most open-source models underperform. The paper identifies spatial reasoning and fine-grained recognition as the main weaknesses of current LMMs and releases data and code.","tokens_in":33374,"tokens_out":9121,"duration_ms":80546,"significance":"If the benchmark's measurements are valid, AgroMind would fill a genuine gap: existing remote sensing benchmarks are urban or domain-general, and agricultural benchmarks are smaller and narrower. The paper's strengths are its large scale, multi-sensor coverage, hierarchical task design, and the inclusion of a human baseline. The authors also provide reproducible evaluation protocols and a clear annotation pipeline, and they commit to public data and code. However, the headline results (e.g., GPT-4o outperforming humans by 9.99 points) rest on the assumption that the public-source test images were not memorized during LMM pretraining, and this assumption is untested. The open-ended scoring threshold is also tuned on a subset of the same benchmark, adding a circular element. These issues must be resolved before the benchmark's validity claims are fully convincing.","major_comments":[{"comment":"No contamination analysis is provided even though nine of the ten source datasets (IP102, CropHarvest, Agriculture-Vision, PhenoBench, OilPalmUAV, ACFR, OAM-TCD, 2018 AI Challenge, Maize Tassel) are public and may appear in the pretraining or instruction-tuning corpora of the evaluated LMMs. The paper's own Section 5.2 observation that models 'tend to reproduce answers seen in their training data rather than understand the agricultural scene itself' makes this risk concrete. The absence of a per-source accuracy breakdown means the reported GPT-4o advantage over humans (Table 2) could be partly an artifact of memorization. Please add (i) near-duplicate checks against common multimodal training corpora, (ii) accuracy reported separately for the private Global Parcel Dataset and for each public source, and (iii) a rerun of the model-versus-human comparison on the private subset alone. This is necessary to support the claim that AgroMind measures agricultural understanding rather than exposure to familiar images.","section":"Section 5.1 / Appendix B.1 (Table 4)"},{"comment":"The paper reports inconsistent dataset statistics: the abstract states 27,247 QA pairs and 19,615 images, while the main text (Section 3.1, Table 1, Figures 3 and 4) consistently reports 28,482 QA pairs and 20,850 images. The abstract also says 'eight public datasets,' whereas Section 3.1 and Appendix B.1 say 'nine public datasets and one private.' The dataset scale is a primary contribution, so these counts must be harmonized to a single, verifiable version.","section":"Abstract; Section 3.1; Table 1"},{"comment":"The correctness criterion for open-ended questions is a semantic-similarity threshold (cosine > 0.60) computed with paraphrase-MiniLM-L6-V2, and the threshold is described as determined through 'preliminary experiments on a dataset subset' 'under which all models achieve reasonable accuracy.' This is a free parameter fitted to the benchmark itself, and it directly affects reported open-ended accuracies (e.g., 15.14% for GPT-4o and 5.80% for humans in Table 6). Please report the proportion of open-ended questions, provide a sensitivity analysis of overall accuracy and model rankings across thresholds such as 0.50-0.70, and, if possible, validate the threshold against expert human judgments of semantic equivalence.","section":"Appendix C.4"}],"minor_comments":[{"comment":"The sentence 'All close-sourced models run on machines equipped with NVIDIA A800 GPUs' seems inaccurate for API-access models such as GPT-4o; please clarify the computational setup.","section":"Section 5.1"},{"comment":"The human baseline is reported only as a point estimate; given the claim that GPT-4o surpasses humans, please report the variance of human accuracy (e.g., standard deviation across the ten subsets or volunteers) and, if practical, confidence intervals for model scores.","section":"Table 2"},{"comment":"The text refers to a 'three-dimensional evaluation framework (Spatial–Object–Scene)' and also to 'three-dimensional Spa tial–Object–Scene framework,' which conflicts with the four dimensions (Spatial Perception, Object Understanding, Scene Understanding, Scene Reasoning) used elsewhere in the paper; please reconcile the terminology.","section":"Appendix B.1"},{"comment":"The geographical coverage map appears to contain garbled text (e.g., '/uni00000013/...'), likely a rendering error; please provide a legible version.","section":"Figure 8"},{"comment":"For single-choice parsing, the rule to select the last mentioned option when no option marker is present may bias results for long outputs; please report how often this fallback is used.","section":"Appendix C.4"},{"comment":"The 'Difficulty Level' column in Table 1 is blank for all datasets; either fill it in or remove it to avoid confusion.","section":"Table 1"},{"comment":"Minor grammatical issue: 'Large Multimodal Models (LMMs) has demonstrated' should be 'have demonstrated.'","section":"Abstract"},{"comment":"All multi-image questions are concatenated into a single image before evaluation; this is a reasonable way to equalize input formats, but the assumption that this preserves task difficulty is untested. Please provide a small-scale comparison of multi-image models on the original multi-image inputs versus the concatenated versions.","section":"Section 5.1"}],"recommendation":"major_revision","confidential_remarks":"The benchmark is potentially useful, and the evaluation is extensive. The contamination issue is the main risk; if the authors can show that the private Global Parcel subset reproduces the model-human ordering, the paper would be much stronger. The inconsistency in dataset counts should also be fixed before publication. The paper fits the scope of the journal, but the current version needs substantial revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know two things before reading. AgroMind is a real, substantial benchmark: ten datasets, 13 task types, 24 LMMs, a human baseline, and a detailed appendix on preprocessing. The second thing is that the headline claim about GPT-4o beating human performance is weaker than it looks, because the human baseline is a group of undergrad students and nobody checked whether the benchmark's public images are already inside the models' training corpora. The benchmark is worth engaging with; the specific numbers are not.\n\nThe integration is genuinely broader than what's out there. CDDMBench, AgriBench, and AgMMU focus on narrower task sets, whereas AgroMind adds spatial perception and reasoning tasks like boundary detection, spatial relationship, and planning. The task taxonomy—four dimensions, 13 tasks—is sensible and should be useful to the community. They evaluate a wide model zoo, and the per-task, per-scene, per-sensor breakdowns in the appendix are a nice touch. If the data and code are actually released as promised, that is a reproducible asset.\n\nNow the soft spots, in rough order of importance.\n\nFirst, no contamination analysis. Nine of the ten sources are public datasets (IP102, Agriculture-Vision, PhenoBench, CropHarvest, etc.). The paper never checks whether the LMMs saw these images during pretraining or instruction tuning, and it does not report results separately for the one private dataset, which is only about four percent of the QA pairs. Without that, the GPT-4o versus human margin is not trustworthy. The paper even says in Section 5.2 that models \"tend to reproduce answers seen in their training data\"—that concern should have been applied to their own test set.\n\nSecond, the human baseline is weak. Twenty undergrad and grad students, no agronomy training. That is fine if framed as \"non-expert humans,\" but the paper's language overstates it.\n\nThird, inconsistent counts. The abstract in the full text says 28,482 QA pairs and 20,850 images, but the abstract as given on arXiv says 27,247 QA pairs and 19,615 images. Figure 3 says 28,482. These need to be reconciled.\n\nFourth, the open-ended scoring threshold of 0.60 was tuned on a subset of this benchmark. That is fitting to the test set. At minimum, report sensitivity to that threshold.\n\nMinor: no error bars or statistical significance across runs for any of the 24 models. Not essential for a benchmark, but helpful.\n\nAll of these are fixable. The construction is sound enough to deserve a serious referee. My recommendation: engage with the paper, send it to review, but require the contamination analysis, the count fix, and a more careful framing of the human comparison.","headline":"AgroMind is a genuinely useful new benchmark for agricultural remote sensing, but the headline 'GPT-4o beats humans' claim needs a contamination check and a stronger human baseline before I'd trust the specific numbers.","tokens_in":33960,"tokens_out":4288,"would_cite":true,"duration_ms":41439,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"AgroMind benchmarks 24 large multimodal models on 13 agricultural remote sensing tasks and finds most lag a human baseline, with spatial reasoning and fine-grained recognition the weakest areas.","keywords":["agricultural remote sensing","large multimodal models","visual question answering","benchmark","spatial reasoning","crop disease diagnostics","zero-shot evaluation","scene understanding"],"falsifier":"Hold out a private set of agricultural images from the same sensors and crops that no evaluated model has seen, then compare model accuracy on it with accuracy on AgroMind's public-dataset questions; if public-question accuracy is substantially higher and private accuracy collapses especially on fine-grained recognition, the reported rankings partly measure memorization rather than agricultural understanding.","tokens_in":32909,"feed_emoji":"🌾","tokens_out":9099,"duration_ms":90479,"temperature":0.7,"pith_summary":"AgroMind is a benchmark for agricultural remote sensing that assembles 20,850 images and 28,482 question-answer pairs into 13 task types organized by four ability dimensions: spatial perception, object understanding, scene understanding, and scene reasoning. The paper's aim is to decide whether large multimodal models can genuinely interpret agricultural scenes, and it reports that most cannot: 17 of 24 evaluated models score below a human baseline of 33.15%, with the worst performance concentrated in spatial localization, boundary detection, counting, and visual prompt reasoning. GPT-4o reaches 43.14% overall, nearly ten points above the human baseline, but it still lags on the spatial tasks. If the benchmark is valid, it establishes a standardized way to measure progress in agricultural AI and points future work toward spatial and fine-grained recognition rather than broad knowledge recall.","feed_headline":"Most AI vision models fail agriculture's spatial reasoning tests","feed_subtitle":"AgroMind's 28,482 questions show GPT-4o leading while counting and boundary detection stump the rest","key_machinery":"The load-bearing machinery is the AgroMind task hierarchy and its QA generation-and-scoring pipeline. The benchmark defines four dimensions — spatial perception, object understanding, scene understanding, and scene reasoning — subdivided into 13 tasks; each image is paired with rule-generated or human-generated questions built from pixel-level masks, instance boxes, and parcel polygons, with a logic-enhancement step that inserts step-by-step reasoning paths into the answers rather than using raw labels. Evaluation is zero-shot: fixed-format answers are scored by strict string or set matching, localization answers by IoU above 0.5, and open-ended answers by cosine similarity from a Sentence-BERT model above a threshold of 0.60. Multi-image questions are concatenated into single images so that models cannot exploit multi-image input advantages. This pipeline is what turns heterogeneous public data into comparable task scores.","core_discovery":"On its own terms, the paper claims that AgroMind provides the first broad, multi-scene, multi-task evaluation of large multimodal models in agricultural remote sensing. The central empirical claim is that current models show significant performance gaps in spatial perception and fine-grained recognition while doing comparatively well on knowledge-driven scene reasoning: models score much higher on object classification, pest and disease diagnostics, and climate type reasoning than on spatial relationship, boundary detection, counting, or visual prompt reasoning. The paper also claims that the best closed model, GPT-4o, surpasses a human student baseline overall (43.14% vs 33.15%) but falls behind on boundary detection and counting, and that 17 of the 24 models underperform the human baseline. Across difficulty levels, the authors find that models outperform students on hard questions while struggling on easy and medium ones, and that architecture matters more than parameter count.","pith_inferences":["Because most test images come from public datasets that predate model training, the reported numbers may include a memorization component; a test on a private counterpart dataset would isolate genuine agricultural understanding.","The difficulty labels are assigned by student-human accuracy, so 'hard' denotes hard for students, not hard for agronomists; an expert panel might order the tasks differently and could change the human-versus-model comparison.","Strict exact matching for multiple-choice and count questions may mark correct-but-differently-worded answers wrong; a sampled human equivalence check would quantify this and could move the rankings.","If the spatial-perception gap persists on fresh data, it suggests a testable training recipe: augment multimodal models with spatially supervised agricultural data such as masks, boxes, and parcel polygons, then measure whether the 13-task scores close the gap."],"forward_implications":["A standardized scorecard now exists for agricultural remote sensing: any future model can be compared with the 24 evaluated models on the same 13 tasks, which should make gap-filling work easier to target.","The spatial tasks isolate a concrete failure mode: if models cannot localize objects, judge spatial relations, detect boundaries, or count, they cannot support precision agriculture operations such as yield mapping and targeted weeding.","The relative strength on climate type reasoning and planning suggests models compensate for weak visual grounding with broad world knowledge, so gains on those tasks should not be read as gains in agricultural perception.","The difficulty-level result implies that improving easy and medium questions — ordinary farm-scene situations — is a more urgent direction than chasing expert-level hard questions for real-world deployment.","Architecture and training recipe appear to matter more than parameter count, since InternVL2-26B outperforms LLaVA-NeXT-34B and LLaVA-NeXT gets worse from 8B to 13B."],"supporting_citations":[{"why":"Supplies GPT-4o, the strongest evaluated model and the source of the claim that a closed multimodal model can exceed the human baseline.","marker":"[11]"},{"why":"Supplies global tree cover imagery and annotations used for spatial distribution, area statistics, and scene comparison tasks.","marker":"[48]"},{"why":"Supplies UAV oil palm imagery with instance-level boxes used for counting and growth status recognition tasks.","marker":"[49]"},{"why":"Supplies multispectral images and anomaly masks used for anomaly reasoning, spatial relationship, and boundary detection tasks.","marker":"[50]"},{"why":"Supplies crop and weed semantic and instance annotations used for plant counting, occlusion analysis, and planning tasks.","marker":"[51]"},{"why":"Supplies 102 pest categories and VOC-format annotations used for pest and disease diagnostics questions.","marker":"[52]"},{"why":"Supplies global crop-type satellite samples and labels used for cropland classification and area estimation tasks.","marker":"[54]"},{"why":"Supplies the multi-view evaluation protocol and answer parsing conventions that AgroMind follows.","marker":"[17]"},{"why":"Supplies the MMMU-style answer parsing and evaluation setup used for judging model responses.","marker":"[44]"},{"why":"Supplies the Sentence-BERT model used to score open-ended answers by semantic similarity.","marker":"[64]"}],"fun_headline_variants":["AgroMind benchmark: AI stumbles on counting crops and spatial tasks","GPT-4o tops AgroMind, but all models fail at boundary detection","New ag benchmark reveals AI blind spots in spatial reasoning","AgroMind: 28k questions expose LMM limits in farm scenes","Why AI can't count crops: AgroMind benchmark findings"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper's conclusions stand on the assumption that the 20,850 test images, most drawn from public datasets that are available online, were not memorized during pretraining by the models being evaluated; the paper never tests this.","fun_headline_variants_meta":{"raw":{"variants":["AgroMind benchmark: AI stumbles on counting crops and spatial tasks","GPT-4o tops AgroMind, but all models fail at boundary detection","New ag benchmark reveals AI blind spots in spatial reasoning","AgroMind: 28k questions expose LMM limits in farm scenes","Why AI can't count crops: AgroMind benchmark findings"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00048,"raw_usage":{"total_tokens":2402,"prompt_tokens":997,"completion_tokens":1405,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":613,"completion_tokens_details":{"reasoning_tokens":1314}},"tokens_in":613,"tokens_out":1405,"duration_ms":10161,"temperature":1.0,"reasoning_tokens":1314,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T20:38:25.490284+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Hold out a private set of agricultural images from the same sensors and crops that no evaluated model has seen, then compare model accuracy on it with accuracy on AgroMind's public-dataset questions; if public-question accuracy is substantially higher and private accuracy collapses especially on fine-grained recognition, the reported rankings partly measure memorization rather than agricultural understanding.","supporting_citations":[{"cited_title":"Hello gpt-4o,","cited_arxiv_id":null,"evidence_quote":"Supplies GPT-4o, the strongest evaluated model and the source of the claim that a closed multimodal model can exceed the human baseline."},{"cited_title":"OAM-TCD: A globally diverse dataset of high-resolution tree cover maps","cited_arxiv_id":"2407.11743","evidence_quote":"Supplies global tree cover imagery and annotations used for spatial distribution, area statistics, and scene comparison tasks."},{"cited_title":"Growing status observation for oil palm trees using unmanned aerial vehicle (uav) images,","cited_arxiv_id":null,"evidence_quote":"Supplies UAV oil palm imagery with instance-level boxes used for counting and growth status recognition tasks."},{"cited_title":"Agriculture- vision: A large aerial image database for agricultural pattern analysis,","cited_arxiv_id":null,"evidence_quote":"Supplies multispectral images and anomaly masks used for anomaly reasoning, spatial relationship, and boundary detection tasks."},{"cited_title":"PhenoBench — A Large Dataset and Benchmarks for Semantic Image Interpretation in the Agricultural Domain,","cited_arxiv_id":null,"evidence_quote":"Supplies crop and weed semantic and instance annotations used for plant counting, occlusion analysis, and planning tasks."},{"cited_title":"Ip102: A large-scale benchmark dataset for insect pest recognition,","cited_arxiv_id":null,"evidence_quote":"Supplies 102 pest categories and VOC-format annotations used for pest and disease diagnostics questions."},{"cited_title":"Cropharvest: A global dataset for crop-type classification,","cited_arxiv_id":null,"evidence_quote":"Supplies global crop-type satellite samples and labels used for cropland classification and area estimation tasks."},{"cited_title":"Urbench: A comprehensive benchmark for evaluating large multimodal models in multi-view urban scenarios,","cited_arxiv_id":null,"evidence_quote":"Supplies the multi-view evaluation protocol and answer parsing conventions that AgroMind follows."}],"review_version":1}