REVIEW 3 major objections 5 minor 25 references
Embeddings based Anomaly Detection for Cleaning Global Crop Type Reference Datasets
T0 review · 3 major / 5 minor · reviewed 2026-07-31 · deepseek-v4-flash
Pith's one-line read The paper claims that a locality-aware anomaly detector on Earth-observation embeddings identifies genuinely mislabelled crop-reference samples, and that conservatively using its flags improves crop-type mapping across five macro-regions.
desk verdict A useful, practical cleaning pipeline with an honest limitations section, but the claim that two independent tests establish genuine mislabeling is only half true: the synthetic test works, the test-view lift is confounded. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The EBA detector. Its core object is the 'local slice': reference samples sharing a spatial cell (a hexagon in a hierarchical global grid), a label in the chosen legend (crop type or land cover), and optionally a source and year. Within a slice, each sample gets an anomaly score from the equally weighted average of three views — a percentile-normalized cosine distance to a trimmed, contamination-resistant centroid; a percentile-normalized mean distance to k nearest neighbours; and a robust median-absolute-deviation z-score — combined with a rank-based score. A MAD-based rule with a per-slice flag cap grades samples as normal, flagged, suspect, or candidate, and a confidence mapping assigns a
What would settle it
On a trusted dataset containing rare but correctly labelled fields (e.g., verified unusual management practices), measure the flag rate on those rare-valid points; if the detector flags them at a rate comparable to its enrichment of injected label errors, or if removing them fails to improve (or degrades) a downstream model, the 'flagged equals mislabelled' premise is refuted.
Extended reading notes
Core claim
The central claim is that embedding-based, locality-aware anomaly scores separate label errors from legitimate variation in global crop-type reference data. Each labelled sample is judged inside a 'local slice' — the set of samples sharing the same spatial cell and the same declared crop or land-cover label — using a score that combines cosine distance to a contamination-resistant trimmed centroid and mean distance to k nearest neighbours, normalized per slice and fused with rank and robust z-score views. The detector concentrates injected label errors 2.5–5× above chance in its flagged set (detection AUROC up to 0.84 for gross errors, ~0.72 for subtle confusable-class errors), and a model-i
Load-bearing premise
A sample whose embedding lies far from its same-crop, same-locality neighbours is more likely an error than a rare-but-valid example; if this fails, cleaning would remove legitimate signal and the reported accuracy gains would be an artefact.
Editorial extensions
If this is right
- Point-wise detection fails against whole-dataset corruption: if an entire source dataset is consistently mislabelled against the wrong legend, per-slice scores stay at chance and group-level aggregation is required to surface systematic errors.
- Conservative cleaning beats aggressive cleaning: dropping only the most extreme candidates or confidence-weighting yields a mean +2.1 crop-type macro-F1 gain, while removing all flagged points is the worst policy and discards hard-but-correct boundary samples.
- The test-view lift shows that measured accuracy of a fixed trained model rises when flagged held-out points are down-weighted, so flag quality is not an artefact of retraining on the cleaned set.
- Using a fine-tuned encoder sharpens error recovery (gross-error enrichment rises from 4.3× to 4.9×), indicating the detector's power is bounded by how well the embedding separates classes in a given region.
- The pipeline is encoder-agnostic and runs on any fixed-size embedding, making it a template for cleaning large Earth-observation reference datasets beyond crop mapping.
Reading between the lines
- The paper does not run the full iterative loop its own ablation implies: cleaning with a frozen encoder, fine-tuning on the cleaned set, then re-running detection with the improved encoder. A testable extension would be to check whether this loop converges to a stable, cleaner reference set and larger accuracy gains.
- Because the detector is per-slice and label-consistent, it could be transplanted to other domains where labels come from heterogeneous sources, such as medical image collections with site-specific class definitions; the enrichment test used here would quantify whether embeddings in those domains separate error from signal as cleanly as they do for crops.
- The confidence weights produced by the detector could be fed directly into a label-noise-aware training loss — for instance, as soft labels or per-sample loss reweighting — rather than being used only for filtering; this might recover signal from ambiguous boundary samples that removal discards.
- The enrichment metric (planted-error concentration in the flagged set) could serve as an extrinsic, label-agnostic benchmark for comparing geospatial foundation-model embeddings, since a better encoder should yield higher enrichment without any downstream training.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes an embedding-based anomaly detection (EBA) framework for cleaning crop-type reference data, using locality-aware slices (H3 cell + label) and a combination of centroid distance, kNN distance, and robust scores to flag suspicious samples. The authors evaluate the detector in three ways: (1) synthetic injection of label errors, showing 2.5–5× enrichment of injected errors in the flagged set and AUROC up to 0.84; (2) a 'model-independent test' where removing or down-weighting flagged points from a fixed model's held-out test set raises measured accuracy; and (3) downstream WorldCereal model training, where conservative cleaning improves crop-type macro-F1 in all five regions (mean +2.1 points). The paper also includes ablations on detector operating points and encoder choice. It concludes that the flagged points are genuinely mislabelled or misplaced, and that conservative cleaning helps while over-cleaning hurts.
Significance. If the central claim holds, the work provides a practical, reproducible tool for a real bottleneck in global crop mapping: cleaning heterogeneous reference data. The strengths are the transparent experimental setup (fixed held-out split, multiple views, operating-point sweep), open code, and the synthetic-injection evaluation that convincingly shows the detector can concentrate scattered label errors. The paper is honest about limitations, including the failure on whole-dataset corruption and the possibility that legitimate rare points are flagged. However, the significance is tempered by the fact that the real-data 'independent' test does not isolate mislabeling from model difficulty, and the downstream gains could in principle arise from removing hard-but-valid samples. The paper is therefore promising but does not yet fully establish that the flags are genuine label errors.
major comments (3)
- [§4.1 Test 2] The 'model-independent test' is not independent evidence of mislabeling. For a fixed model, removing or down-weighting points with high anomaly scores will raise measured accuracy whenever those points are ones the model tends to misclassify, regardless of whether their labels are correct. The paper's own Discussion (§5) admits that legitimate rare management practices or mixed pixels can be flagged. Thus Test 2 only shows that EBA scores correlate with model difficulty; it does not establish that flagged points carry wrong labels. This undermines the abstract's claim of 'two independent ways' and should be addressed, e.g., by a human audit of a random sample of flagged points, comparison with higher-confidence reference sources, or an analysis showing the lifted points are inconsistent with external metadata.
- [§4.2 downstream training] The downstream training improvements are consistent with the detector removing hard-but-valid training samples rather than correcting label errors. Removing difficult training points can improve generalization even when those points are correctly labeled, especially if they cause the model to overfit. The paper's class-wise analysis (Fig. 6) shows gains on rare classes, which is encouraging, but it does not rule out the alternative mechanism. Provide evidence that the removed training points are actually mislabeled—for example, by analyzing their source-dataset agreement or by showing that confidence-weighting (which retains all points) produces gains comparable to removal on the full test view.
- [§4.1 Test 1 / Table 1] The whole-dataset corruption case sits at chance (AUROC ≈ 0.50), as the paper acknowledges. While this is a reasonable scope limitation for a per-slice detector, it is relevant to the claimed practical utility: 'systematically mislabelled whole datasets' are one of the error modes listed in the Introduction (§1) as coming from map-derived labels and heterogeneous sources. The paper mentions group-level aggregation as a robustness measure (§3.6) but does not evaluate it against whole-dataset corruption. Please add a quantitative evaluation of the group-level aggregation, or at least clarify how common whole-dataset corruption is in WorldCereal and whether the proposed tool is intended to address it.
minor comments (5)
- [Abstract] The phrase 'two independent ways' is too strong given that Test 2's interpretation is confounded (see major comment). Consider rephrasing to 'two complementary evaluations' or qualifying the second test.
- [§4.1 Table 1 caption] The caption omits the whole-dataset corruption row; make explicit that it is not shown because it sits at chance, to avoid readers assuming it was not tested.
- [§3.3–§3.5] The notation is dense and some quantities (e.g., epsilon, q_min/q_max, tau) are not formally defined in the equations. A table of symbols would improve reproducibility.
- [§4.4] The fine-tuned encoder ablation (Fig. 9) is referenced in §4.1 and §4.4; consider moving the reference to the ablation section for clarity.
- [General] Minor typographical issues: 'cand. removed' and 'conf. weighted' are used without definition in the caption of Table 2; 'WorldCereal style model' appears in the conclusion without a hyphen. Also, the GitHub link is given inline but the repository name is not repeated in the references.
Circularity Check
No meaningful circularity: the EBA detector's scores are computed from embeddings and local slice statistics, not from the downstream outcome, and the main claims are supported by synthetic ground-truth recovery and a fixed held-out split. The test-view lift is a confound but not a definitional circularity.
full rationale
The central derivation chain is not circular. Anomaly scores are constructed from cosine distances to a trimmed centroid and kNN structure within H3/label slices (Secs. 3.2–3.5); neither the scores nor the flagging rule are fitted to downstream crop-type accuracy. Test 1 injects synthetic label noise and measures enrichment/AUROC of the flags against that planted ground truth, an external, parameter-free check. Test 2 re-scores a fixed trained model after removing/down-weighting flagged held-out points; while this confounds label errors with hard-but-valid points (the paper itself concedes in Sec. 4.1 that a valid out-of-distribution point may be flagged, and the Discussion acknowledges legitimate rare management practices and mixed pixels can be flagged), the lift is not forced by construction: removing points a model finds difficult can raise accuracy regardless of label correctness, but that is an evidence-interpretation weakness, not an equation-level circularity. The held-out split is fixed and spatial, and downstream training treatments are applied only on the training split, giving independent grounding. Self-citations to WorldCereal [21,25], the RDM [1], crop-calendar alignment [14], and deployment lessons [5] are contextual and not load-bearing for the claim that EBA flags mislabels. There is no uniqueness theorem, no renamed fit, and no ansatz smuggled in via self-citation. The most that can be said is a minor presence of the authors' own prior work in the framing; this does not reduce the central result to its inputs.
Assumptions & free parameters
free parameters (8)
- H3 grid level L (e.g., 1-3)
- Minimum slice size =
50
- MAD flag threshold k =
4
- Flagged share cap =
10%
- Percentile normalization q_min/q_max and epsilon
- Trimmed-centroid discard fraction tau
- Confidence mapping operating points =
default t=0.975, gamma=0.30; tightened t=0.95, gamma=0.23
- Nearest-neighbour k in local score
assumptions (5)
- domain assumption Frozen Presto embeddings separate crop classes and observation conditions well enough that anomaly distances track label errors
- domain assumption H3-cell + label slices provide an unbiased reference population for each sample
- domain assumption Outliers relative to local class are predominantly label errors rather than legitimate rare conditions
- domain assumption Held-out test labels are clean enough to serve as ground truth for downstream accuracy measurements
- domain assumption Injected corruption models (gross, subtle, whole-dataset) represent real WorldCereal error modes
Cite this review
Pith. "Pith review of Embeddings based Anomaly Detection for Cleaning Global Crop Type Reference Datasets." pith.science (2026). https://pith.science/paper/5LZHULBN
@misc{pith2026260723908,
author = {Pith},
title = {Pith review of: Embeddings based Anomaly Detection for Cleaning Global Crop Type Reference Datasets},
year = {2026},
howpublished = {\url{https://pith.science/paper/5LZHULBN}},
note = {Machine review of arXiv:2607.23908}
}
read the original abstract
High quality reference data remain a critical bottleneck for crop-type mapping at any spatial and temporal scale. Operational systems such as WorldCereal aggregate labels from heterogeneous sources such as parcel registers, national databases, field surveys, and map-derived products, each with their own biases, coverage gaps and unknown label noise. Simple global rules are inadequate, since crop phenology and observation conditions vary strongly across regions and seasons. In this study, we focus on a single, operationally relevant question: whether embeddings produced through geospatial foundation models are a viable basis for cleaning the reference data. We propose a practical, locality-aware, embedding-based anomaly (EBA) detection framework that operates on the embeddings of a pretrained Earth-observation encoder. We score each labelled sample against other samples of the same crop in the same area using a pretrained embedding, flag the ones that stand out, and test whether removing or down-weighting them before training yields a better model. We establish that the flagged points are genuinely mislabelled or misplaced in two independent ways: against synthetic ground truth, the detector concentrates injected label errors 2.5-5x above chance in its flagged set (detection AUROC up to 0.84); and on real data, a model-independent test shows that removing or confidence-weighting the flagged held-out points raises measured accuracy in trained models, for both crop type and land cover. Acting on the flags then improves the WorldCereal crop-type model across five macro-regions, evaluated on a fixed held-out split under three views. We find conservative cleaning helps while over-cleaning hurts. The EBA detector approach is designed to be reproducible and extensible, and can serve as a template for cleaning large, noisy Earth observation reference datasets beyond crop mapping.
Figures
Figures from the paper (6 more)
Reference graph
Works this paper leans on
-
[1]
PLoS ONE18(7), e0287731 (2023)
Boogaard,H.,Pratihast,A.K.,LasoBayas,J.C.,Karanam,S.,Fritz,S.,VanTricht, K., Degerickx, J., Gilliams, S.: Building a community-based open harmonised refer- ence data repository for global crop mapping. PLoS ONE18(7), e0287731 (2023). https://doi.org/10.1371/journal.pone.0287731
-
[2]
Geocarto International26(5), 341–358 (2011).https://doi.org/ 10.1080/10106049.2011.562309
Boryan, C., Yang, Z., Mueller, R., Craig, M.: Monitoring us agriculture: The us department of agriculture, national agricultural statistics service cropland data layer program. Geocarto International26(5), 341–358 (2011).https://doi.org/ 10.1080/10106049.2011.562309
arXiv 2011
-
[3]
In: ACM SIGMOD International Conference on Management of Data
Breunig, M.M., Kriegel, H.P., Ng, R.T., Sander, J.: LOF: Identifying density-based local outliers. In: ACM SIGMOD International Conference on Management of Data. pp. 93–104 (2000).https://doi.org/10.1145/342009.335388
arXiv 2000
-
[4]
arXiv preprint arXiv:2507.22291 (2025)
Brown, C.F., Kazmierski, M.R., Pasquarella, V.J., Rucklidge, W.J., Samsikova, M., Zhang,C.,Shelhamer,E.,Lahera,E.,Wiles,O.,Ilyushchenko,S.,etal.:Alphaearth foundations: An embedding field model for accurate and efficient global mapping from sparse label data. arXiv preprint arXiv:2507.22291 (2025)
arXiv 2025
-
[5]
In: Proceedings of the TerraBytes ICML Workshop: Towards Global Datasets and Models for Earth Observation
Butsko, C., Tseng, G., Van Tricht, K., Milli, G., Rolnick, D., Cartuyvels, R., Becker-Reshef, I., Szantoi, Z., Kerner, H.: Deploying geospatial foundation models in the real world: Lessons from WorldCereal. In: Proceedings of the TerraBytes ICML Workshop: Towards Global Datasets and Models for Earth Observation. Proceedings of Machine Learning Research, v...
2025
-
[6]
In: Advances in Neural Information Processing Systems (NeurIPS) (2022)
Cong, Y., Khanna, S., Meng, C., Liu, P., Rozi, E., He, Y., Burke, M., Lobell, D.B., Ermon, S.: SatMAE: Pre-training transformers for temporal and multi- spectral satellite imagery. In: Advances in Neural Information Processing Systems (NeurIPS) (2022)
2022
-
[7]
Scientific Data7, 352 (2020).https://doi
d’Andrimont, R., Yordanov, M., Martinez-Sanchez, L., Eiselt, B., Palmieri, A., Dominici, P., Gallego, J., Reuter, H.I., Joebges, C., Lemoine, G., van der Velde, M.: Harmonised LUCAS in-situ land cover and use database for field surveys from 2006 to 2018 in the european union. Scientific Data7, 352 (2020).https://doi. org/10.1038/s41597-020-00675-z
-
[8]
Remote Sensing12(6), 1034 (2020).https: //doi.org/10.3390/rs12061034
Elmes, A., Alemohammad, H., Avery, R., Caylor, K., Eastman, J.R., Fishgold, L., Friedl, M.A., Jain, M., Kohli, D., Laso Bayas, J.C., Jepsen, T., Raveling, J., Wang, J., Welch, B., Wrona, D., Zhu, Z.: Accounting for training data error in machine learning applied to earth observations. Remote Sensing12(6), 1034 (2020).https: //doi.org/10.3390/rs12061034
Show all 25 references
-
[9]
Earth System Dynamics8(3), 677–696 (2017)
Flach, M., Gans, F., Brenning, A., Denzler, J., Reichstein, M., Rodner, E., Bathi- any, S., Bodesheim, P., Guanche, Y., Sippel, S., et al.: Multivariate anomaly de- tection for earth observations: a comparison of algorithms and feature extraction techniques. Earth System Dynam...
2017
-
[10]
Frénay, B., Verleysen, M.: Classification in the presence of label noise: A sur- vey. IEEE Transactions on Neural Networks and Learning Systems25(5), 845–869 (2014).https://doi.org/10.1109/TNNLS.2013.2292894 16 Embedding-based Outlier Detection for Crop Reference Data
2014
-
[11]
arXiv preprint arXiv:2310.18660 (2023)
Jakubik, J., Roy, S., Phillips, C.E., Fraccaro, P., Godwin, D., Zadrozny, B., Szwar- cman, D., Gomes, C., Nyirjesy, G., Edwards, B., Kimura, D., Simumba, N., Chu, L., Mukkavilli, S.K., Lambhate, D., Das, K., Bangalore, R., Oliveira, D., Muszyn- ski, M., Ankur, K., Ramasubraman...
2023 arXiv
-
[12]
Journal of Experimental Social Psychology49(4), 764–766 (2013).https://doi
Leys, C., Ley, C., Klein, O., Bernard, P., Licata, L.: Detecting outliers: Do not use standard deviation around the mean, use absolute deviation around the median. Journal of Experimental Social Psychology49(4), 764–766 (2013).https://doi. org/10.1016/j.jesp.2013.03.013
2013 doi
-
[13]
In: IEEE International Con- ference on Data Mining (ICDM)
Liu, F.T., Ting, K.M., Zhou, Z.H.: Isolation forest. In: IEEE International Con- ference on Data Mining (ICDM). pp. 413–422 (2008).https://doi.org/10.1109/ ICDM.2008.17
2008
-
[14]
Ecological Informatics p
Moletto-Lobos, I., Franch, B., Guillem-Valls, A., Cyran, K., Kalecinski, N., Van Tricht, K., Vermote, E., Becker-Reshef, I., Nair, S., Degerickx, J., et al.: Enhancing worldcereal crop calendars with land surface phenology and machine learning. Ecological Informatics p. 103742 (2026)
2026
-
[15]
In: NeurIPS Datasets and Benchmarks Track (2021)
Northcutt, C.G., Athalye, A., Mueller, J.: Pervasive label errors in test sets desta- bilize machine learning benchmarks. In: NeurIPS Datasets and Benchmarks Track (2021)
2021
-
[16]
Journal of Artificial Intelligence Research70, 1373–1411 (2021).https://doi.org/10.1613/jair.1.12125
Northcutt, C.G., Jiang, L., Chuang, I.L.: Confident learning: Estimating uncer- tainty in dataset labels. Journal of Artificial Intelligence Research70, 1373–1411 (2021).https://doi.org/10.1613/jair.1.12125
2021 doi
-
[17]
ACM Computing Surveys54(2), 1–38 (2021).https://doi.org/ 10.1145/3439950
Pang, G., Shen, C., Cao, L., van den Hengel, A.: Deep learning for anomaly detec- tion: A review. ACM Computing Surveys54(2), 1–38 (2021).https://doi.org/ 10.1145/3439950
2021 doi
-
[18]
Remote Sensing9(2), 173 (2017).https: //doi.org/10.3390/rs9020173
Pelletier, C., Valero, S., Inglada, J., Champion, N., Marais Sicre, C., Dedieu, G.: Effect of training class label noise on classification performances for land cover mapping with satellite image time series. Remote Sensing9(2), 173 (2017).https: //doi.org/10.3390/rs9020173
2017 doi
-
[19]
John Wi- ley & Sons (1987).https://doi.org/10.1002/0471725382
Rousseeuw, P.J., Leroy, A.M.: Robust Regression and Outlier Detection. John Wi- ley & Sons (1987).https://doi.org/10.1002/0471725382
1987 doi
-
[20]
ISPRS Journal of Photogrammetry and Remote Sensing169, 421–435 (2020).https://doi.org/10.1016/j.isprsjprs.2020.06.006
Rußwurm, M., Körner, M.: Self-attention for raw optical satellite time series clas- sification. ISPRS Journal of Photogrammetry and Remote Sensing169, 421–435 (2020).https://doi.org/10.1016/j.isprsjprs.2020.06.006
2020 doi
-
[21]
Nature Food4, 736–737 (2023).https://doi.org/ 10.1038/s43016-023-00841-7
See, L., Gilliams, S., Conchedda, G., Degerickx, J., Van Tricht, K., Fritz, S., Lesiv, M., Laso Bayas, J.C., Rosero, J., Tubiello, F.N., Szantoi, Z.: Dynamic global-scale crop and irrigation monitoring. Nature Food4, 736–737 (2023).https://doi.org/ 10.1038/s43016-023-00841-7
2023 doi
-
[22]
IEEE Transactions on Neural Networks and Learn- ing Systems34(11), 8135–8153 (2022).https://doi.org/10.1109/TNNLS.2022
Song, H., Kim, M., Park, D., Shin, Y., Lee, J.G.: Learning from noisy labels with deep neural networks: A survey. IEEE Transactions on Neural Networks and Learn- ing Systems34(11), 8135–8153 (2022).https://doi.org/10.1109/TNNLS.2022. 3152527
2022 doi
-
[23]
arXiv preprint arXiv:2304.14065 (2023)
Tseng, G., Cartuyvels, R., Zvonkov, I., Purohit, M., Rolnick, D., Kerner, H.: Lightweight, pre-trained transformers for remote sensing timeseries. arXiv preprint arXiv:2304.14065 (2023)
2023 arXiv
-
[24]
https://h3geo.org(2018) Embedding-based Outlier Detection for Crop Reference Data 17
Uber Technologies: H3: A hexagonal hierarchical geospatial indexing system. https://h3geo.org(2018) Embedding-based Outlier Detection for Crop Reference Data 17
2018
-
[25]
S1:Examples of outliers surfaced by the EBA detector
Van Tricht, K., Degerickx, J., Gilliams, S., Zanaga, D., Battude, M., Grosu, A., Brombacher, J., Lesiv, M., Laso Bayas, J.C., Karanam, S., Fritz, S., Becker-Reshef, I., Franch, B., Mollà-Bononad, B., Boogaard, H., Pratihast, A.K., Szantoi, Z.: WorldCereal: a dynamic open-sourc...
2023 doi
Reviewed July 31, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.