REVIEW 3 major objections 7 minor 27 references
MESSI: A Multi-Elevation Semantic Segmentation Image Dataset of an Urban Environment
T0 review · 3 major / 7 minor · reviewed 2026-08-15 · deepseek-v4-flash
Pith's one-line read A new 2,525-image drone dataset maps how altitude changes semantic segmentation accuracy, with matched views of the same urban areas from 30 to 100 meters and vertical descents down to 10 meters.
desk verdict MESSI is a genuinely useful multi-elevation dataset for aerial segmentation, but the paper's headline altitude-effect experiment (Fig. 7) is confounded by the evaluation mask and needs to be redone or reframed. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The dataset structure itself is the load-bearing mechanism: horizontal trajectories at four fixed altitudes combine with vertical descent sequences over the same urban locations to create matched image sets in which scale and view angle vary while scene content stays roughly fixed. This is paired with per-image pose and camera intrinsics and an evaluation protocol that scores accuracy over the overlapping field of view shared by altitudes; that protocol is what turns the raw images into measurements of altitude transfer.
What would settle it
Compare per-altitude mIoU on fixed ground-projected footprints of equal real-world area, selecting the same physical region in each image rather than fixed pixel crops; if the accuracy gap between training and testing altitudes disappears or reverses under this metric, the altitude-transfer result is an evaluation artifact.
Extended reading notes
Core claim
MESSI is a benchmark of downward-looking drone photographs over dense urban neighborhoods, annotated pixel-wise with 15 classes. Its distinguishing design is that flight paths revisit the same ground areas at 30, 50, 70, and 100 meters, and add 15 vertical descents from 120 to 10 meters, so every region appears at several scales and viewpoints. Each image carries the drone's position and orientation and the camera's intrinsic parameters. The authors claim that because the altitude variation is built into the capture design, MESSI can train and evaluate semantic segmentation across elevations and can quantify how training altitude transfers to test altitude; they report a best out-of-distribution mIoU of 52.6 for SegFormer-B3 and evidence that extrapolating across altitude degrades accuracy.
Load-bearing premise
The claim that training altitude degrades performance at other altitudes rests on evaluating accuracy over the overlapping field of view, where higher-altitude scores are computed on smaller central crops; if that evaluation mask itself favors or penalizes certain altitudes, the observed transfer curve may be an artifact of the mask rather than a real altitude effect.
Editorial extensions
If this is right
- A segmentation network trained on MESSI's multiple altitudes should generalize better to unseen drone elevations than a network trained on a single altitude, because the training distribution covers the scale and viewpoint range of a 3D flight.
- The altitude-transfer effect reported in the paper means deployers can expect accuracy to degrade when a model trained at high altitude is applied to low-altitude imagery, and the dataset provides a controlled way to measure the magnitude of that drop.
- Because pose and intrinsics are included, MESSI also supports downstream tasks beyond segmentation, such as localization, navigation, and tracking.
- The public test benchmark with undisclosed ground truth allows the community to compare segmentation architectures and class-weighting strategies on a common drone-imagery evaluation.
Reading between the lines
- Beyond the paper, the matched multi-scale views make MESSI a natural probe for separating scale effects from viewpoint effects in aerial recognition; cropping fixed ground footprints would factor out resolution and isolate view angle.
- Beyond the paper, the reported altitude-transfer asymmetry suggests that altitude-aware augmentation, such as random zooming during training, may be more effective than training at a single altitude; this is a testable hypothesis the paper does not itself test.
- Beyond the paper, the availability of pose and intrinsics could enable cross-altitude ground-truth propagation, where labels from a low-altitude pass are projected onto higher-altitude images to reduce annotation cost; the geometry needed is present in the dataset, but the paper does not attempt this.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper presents MESSI, a drone-captured urban aerial image dataset of 2,525 images at 5472x3684 resolution, with 15 semantic classes, per-image pose and intrinsics, horizontal multi-altitude trajectories, and vertical descent sequences. It describes the annotation workflow with double-independent quality control, class statistics, training/validation/test splits, and reports benchmarks for several segmentation models under three class-weighting schemes. The best result is SegFormer-B3 with Sqrt weighting reaching 52.6 mIoU on the out-of-distribution test set. The paper also presents an experiment (Fig. 7) on how training altitude affects accuracy on vertical descent sequences, claiming that training at higher altitudes degrades performance at lower altitudes.
Significance. MESSI fills a real gap in the aerial semantic segmentation literature: publicly available datasets generally lack systematic revisits of the same area at multiple altitudes together with per-image pose and intrinsic parameters. The annotation workflow, with nearly 100% rechecking in the final QC phase, is a methodological strength, and the decision to withhold test ground truth for an online benchmark follows good practice. The multi-architecture, multi-weighting baselines are useful for the community. If the Fig. 7 altitude-transfer claim is made rigorous, the dataset would enable a genuinely new class of experiments on altitude and scale robustness of segmentation models.
major comments (3)
- [Section 4.2, Fig. 7] The altitude-transfer experiment is confounded by an altitude-dependent evaluation mask. The text states that accuracy was calculated on the overlapping field of view, with the full image scored at the lowest altitude and progressively smaller central crops at higher altitudes. This simultaneously varies the number of evaluated pixels, the class distribution, and the object scale relative to the network's receptive field, so the reported degradation with test altitude may be an artifact of the evaluation region rather than a property of the training altitude. The curves also appear to come from a single run with no error bars. To support the claim, the authors should evaluate on a common ground-area intersection across altitudes (or a fixed central crop) and report mean and standard deviation over multiple training runs.
- [Section 4.2, Table 5] The text says that the Water category "was omitted from Table 5 and from the mIoU calculations," yet Table 5 includes a Water row and reports both "mIoU with water" and "mIoU without water." This internal contradiction affects the reported evaluation protocol and must be corrected, either by removing Water from the table and recomputing the mIoU values, or by revising the sentence to state that Water was included and that an additional mIoU without Water is also reported.
- [Section 4.1] The description of the Fig. 7 training conditions is incomplete. The authors do not specify the exact sets of training altitudes used for each curve, the number of images per training condition, or whether the "all altitudes" model includes all Ir Yamim horizontal images at each altitude as well as the Agamim paths. Without these details, the experiment cannot be reproduced or interpreted, even after fixing the evaluation-mask issue.
minor comments (7)
- [Section 3.2, Fig. 3 caption] The caption "per attitude interval" should read "per altitude interval."
- [Section 3, Table 2] The table header uses "Agamin" while the text and Figure 1 use "Agamim"; please standardize the spelling.
- [Section 4.2, results text] The abbreviation mAcc is used without being defined in Section 4.1; please define it as mean pixel accuracy alongside the mIoU definition.
- [Section A.3, Table 8] The table lists 16 entries including "void," but the paper refers to 15 classes; please add a note that void is an unlabeled catch-all class and not one of the 15 semantic classes.
- [Section 2, Table 1] The openDD row contains a stray fragment "Segmentation search sensor" that appears to be a formatting error; please clean up the table cell.
- [Section 3.1] The phrase "double-independent checking" is used once; please clarify in one sentence whether this means two annotators independently label each image or another protocol.
- [Appendix, Tables 10-14] The sample-image tables are hard to evaluate in the PDF because the images are small; consider providing higher-resolution crops or a zoomed inset for at least one row.
Circularity Check
No circularity: MESSI is an empirical dataset paper; its benchmark numbers and altitude-statistics plots are derived from data and external tools, not from the paper's own assumptions.
full rationale
This paper is a dataset contribution with empirical benchmarks, not a derivation. The main claims are: (1) MESSI contains 2,525 images at multiple altitudes with annotations, pose, and intrinsics; (2) standard segmentation models can be trained and evaluated on it, with reported mIoU numbers; and (3) Fig. 7 illustrates altitude-dependent accuracy. None of these claims is obtained by fitting a parameter and then renaming the fit as a prediction. The model accuracies in Table 4 are computed by training on the stated splits and testing on the out-of-distribution Ir Yamim and Ha-Medinah Square sets, with standard metrics; there is no equation in the paper whose output is identical to an input by construction. The only self-citation, Pinkovich et al. (2022), is used as background motivation for the emergency-landing application and for the choice of BiSeNetV1 as one of several baselines; it is not invoked to define the dataset, set constants, prove uniqueness, or otherwise force any experimental result. The Fig. 7 protocol, in which accuracy is computed on the overlapping field of view shared by altitudes so that the lowest altitude uses the full image and higher altitudes use smaller central regions, raises a legitimate question about whether the reported degradation is attributable to the evaluation mask rather than to training altitude. That is a correctness/validity concern about an empirical comparison, not a circularity concern: the experiment does not reduce to its inputs by definition, and the paper does not claim to derive Fig. 7 from a first-principles model. The dataset itself is externally anchored (publicly released images, annotations, pose files, standard tooling), and the benchmark results are falsifiable against the released test benchmark. Under the rule that a self-contained empirical paper with no derivation-to-input reduction should receive 0-2, the appropriate score is 0, with the Fig. 7 confound noted as a non-circular scientific weakness.
Assumptions & free parameters
free parameters (3)
- Training descent exclusion set =
Scenarios 1, 35, 36
- Altitude bins for statistics =
(10,33], (33,60], (60,88], (88,125] m
- Subsampling ratio for descent training images =
1 of every 5 images
assumptions (4)
- domain assumption Pixel annotations produced with the V7 Darwin tool and double-independent checking are accurate and consistent.
- domain assumption The 15-class taxonomy is adequate for the emergency-landing application and for benchmarking semantic segmentation.
- domain assumption Imagery from Netanya and Tel Aviv neighborhoods is representative of dense urban environments for drone perception.
- domain assumption The MMsegmentation training protocol with Cityscapes pretraining is a competent evaluation of segmentation models on this dataset.
Cite this review
Pith. "Pith review of MESSI: A Multi-Elevation Semantic Segmentation Image Dataset of an Urban Environment." pith.science (2026). https://pith.science/paper/6V4XKTIC
@misc{pith2026250508589,
author = {Pith},
title = {Pith review of: MESSI: A Multi-Elevation Semantic Segmentation Image Dataset of an Urban Environment},
year = {2026},
howpublished = {\url{https://pith.science/paper/6V4XKTIC}},
note = {Machine review of arXiv:2505.08589}
}
read the original abstract
This paper presents a Multi-Elevation Semantic Segmentation Image (MESSI) dataset comprising 2525 images taken by a drone flying over dense urban environments. MESSI is unique in two main features. First, it contains images from various altitudes, allowing us to investigate the effect of depth on semantic segmentation. Second, it includes images taken from several different urban regions (at different altitudes). This is important since the variety covers the visual richness captured by a drone's 3D flight, performing horizontal and vertical maneuvers. MESSI contains images annotated with location, orientation, and the camera's intrinsic parameters and can be used to train a deep neural network for semantic segmentation or other applications of interest (e.g., localization, navigation, and tracking). This paper describes the dataset and provides annotation details. It also explains how semantic segmentation was performed using several neural network models and shows several relevant statistics. MESSI will be published in the public domain to serve as an evaluation benchmark for semantic segmentation using images captured by a drone or similar vehicle flying over a dense urban environment.
Figures
Figures from the paper (4 more)
Reference graph
Works this paper leans on
-
[1]
Ballesteros, German Sanchez-Torres, and John W
John R. Ballesteros, German Sanchez-Torres, and John W. Branch-Bedoya. Hagdavs: Height-augmented geo-located dataset for detection and semantic segmentation of vehicles in drone aerial orthomosaics. Data, 7 0 (4), 2022. ISSN 2306-5729. doi:10.3390/data7040050. URL https://www.mdpi.com/2306-5729/7/4/50
-
[2]
A critical evaluation of aerial datasets for semantic segmentation
Bianca-Cerasela-Zelia Blaga and Sergiu Nedevschi. A critical evaluation of aerial datasets for semantic segmentation. In 2020 IEEE 16th International Conference on Intelligent Computer Communication and Processing (ICCP), pp.\ 353--360, 2020. doi:10.1109/ICCP51029.2020.9266169
arXiv 2020
-
[3]
opendd: A large-scale roundabout drone dataset
Antonia Breuer, Jan-Aike Termöhlen, Silviu Homoceanu, and Tim Fingscheidt. opendd: A large-scale roundabout drone dataset. In 2020 IEEE 23rd International Conference on Intelligent Transportation Systems (ITSC), pp.\ 1--6, 2020. doi:10.1109/ITSC45102.2020.9294301
arXiv 2020
-
[4]
Vdd: Varied drone dataset for semantic segmentation
Wenxiao Cai, Ke Jin, Jinyan Hou, Cong Guo, Letian Wu, and Wankou Yang. Vdd: Varied drone dataset for semantic segmentation. arXiv preprint arXiv:2305.13608, 2023
arXiv 2023
-
[5]
Encoder-decoder with atrous separable convolution for semantic image segmentation
Liang-Chieh Chen, Yukun Zhu, George Papandreou, Florian Schroff, and Hartwig Adam. Encoder-decoder with atrous separable convolution for semantic image segmentation. In Proceedings of the European conference on computer vision (ECCV), pp.\ 801--818, 2018 a
work page 2018
-
[6]
Large-scale structure from motion with semantic constraints of aerial images
Yu Chen, Yao Wang, Peng Lu, Yisong Chen, and Guoping Wang. Large-scale structure from motion with semantic constraints of aerial images. In Pattern Recognition and Computer Vision: First Chinese Conference, PRCV 2018, Guangzhou, China, November 23-26, 2018, Proceedings, Part I 1, pp.\ 347--359. Springer, 2018 b
work page 2018
-
[7]
Masked-attention mask transformer for universal image segmentation
Bowen Cheng, Ishan Misra, Alexander G Schwing, Alexander Kirillov, and Rohit Girdhar. Masked-attention mask transformer for universal image segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp.\ 1290--1299, 2022
work page 2022
-
[8]
MMSegmentation : Openmmlab semantic segmentation toolbox and benchmark
MMSegmentation Contributors. MMSegmentation : Openmmlab semantic segmentation toolbox and benchmark. https://github.com/open-mmlab/mmsegmentation, 2020. Apache 2.0 License
work page 2020
Show all 27 references
-
[9]
The cityscapes dataset for semantic urban scene understanding
Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele. The cityscapes dataset for semantic urban scene understanding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern ...
2016
-
[10]
S egmentation B enchmark, 2019
DroneDeploy. S egmentation B enchmark, 2019. https://github.com/dronedeploy/dd-ml-segmentation-benchmark
2019
-
[11]
Mid-air: A multi-modal dataset for extremely low altitude drone flights
Michaël Fonder and Marc Van Droogenbroeck. Mid-air: A multi-modal dataset for extremely low altitude drone flights. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp.\ 553--562, 2019. doi:10.1109/CVPRW.2019.00081
2019
-
[12]
Geiger, P
A. Geiger, P. Lenz, and R. Urtasun. Are we ready for Autonomous Driving? The KITTI Vision Benchmark Suite . In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp.\ 3354--3361, 2012
2012
-
[13]
Performance analysis of semantic segmentation algorithms for finely annotated new uav aerial video dataset (manipaluavid)
S Girisha, Manohara MM Pai, Ujjwal Verma, and Radhika M Pai. Performance analysis of semantic segmentation algorithms for finely annotated new uav aerial video dataset (manipaluavid). IEEE Access, 7: 0 136239--136253, 2019
2019
-
[14]
xbd: A dataset for assessing building damage from satellite imagery
Ritwik Gupta, Richard Hosfelt, Sandra Sajeev, Nirav Patel, Bryce Goodman, Jigar Doshi, Eric Heim, Howie Choset, and Matthew Gaston. xbd: A dataset for assessing building damage from satellite imagery. arXiv preprint arXiv:1911.09296, 2019
1911 arXiv
-
[15]
A comprehensive review for typical applications based upon unmanned aerial vehicle platform
Yuqi Han, Huaping Liu, Yufeng Wang, and Chunlei Liu. A comprehensive review for typical applications based upon unmanned aerial vehicle platform. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 15: 0 9654--9666, 2022
2022
-
[16]
Kitti-360: A novel dataset and benchmarks for urban scene understanding in 2d and 3d
Yiyi Liao, Jun Xie, and Andreas Geiger. Kitti-360: A novel dataset and benchmarks for urban scene understanding in 2d and 3d. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45 0 (3): 0 3292--3310, 2022
2022
-
[17]
Uavid: A semantic segmentation dataset for uav imagery
Ye Lyu, George Vosselman, Gui-Song Xia, Alper Yilmaz, and Michael Ying Yang. Uavid: A semantic segmentation dataset for uav imagery. ISPRS journal of photogrammetry and remote sensing, 165: 0 108--119, 2020
2020
-
[18]
Semantics through time: Semi-supervised segmentation of aerial videos with iterative label propagation
Alina Marcu, Vlad Licaret, Dragos Costea, and Marius Leordeanu. Semantics through time: Semi-supervised segmentation of aerial videos with iterative label propagation. In Proceedings of the Asian Conference on Computer Vision, 2020
2020
-
[19]
Unmanned aerial vehicles (uavs): Practical aspects, applications, open challenges, security issues, and future trends
Syed Agha Hassnain Mohsan, Nawaf Qasem Hamood Othman, Yanlong Li, Mohammed H Alsharif, and Muhammad Asghar Khan. Unmanned aerial vehicles (uavs): Practical aspects, applications, open challenges, security issues, and future trends. Intelligent Service Robotics, 16 0 (1): 0 109...
2023
-
[20]
Icg drone dataset
Christian Mostegel, Michael Maurer, Nikolaus Heran, Jesus Pestana Puerta, and Friedrich Fraundorfer. Icg drone dataset. https://dronedataset.icg.tugraz.at, 2019
2019
-
[21]
Ensemble knowledge transfer for semantic segmentation
Ishan Nigam, Chen Huang, and Deva Ramanan. Ensemble knowledge transfer for semantic segmentation. In 2018 IEEE Winter Conference on Applications of Computer Vision (WACV), pp.\ 1499--1508, 2018. doi:10.1109/WACV.2018.00168
2018
-
[22]
Finding a landing site in an urban area: A multi-resolution probabilistic approach
Barak Pinkovich, Boaz Matalon, Ehud Rivlin, and Hector Rotstein. Finding a landing site in an urban area: A multi-resolution probabilistic approach. Sensors, 22 0 (24): 0 9807, 2022
2022
-
[23]
The multi-temporal urban development spacenet dataset
Adam Van Etten, Daniel Hogan, Jesus Martinez Manso, Jacob Shermeyer, Nicholas Weir, and Ryan Lewis. The multi-temporal urban development spacenet dataset. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp.\ 6398--6407, 2021
2021
-
[24]
Samrs: Scaling-up remote sensing segmentation dataset with segment anything model
Di Wang, Jing Zhang, Bo Du, Minqiang Xu, Lin Liu, Dacheng Tao, and Liangpei Zhang. Samrs: Scaling-up remote sensing segmentation dataset with segment anything model. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and S. Levine (eds.), Advances in Neural Information P...
2023
-
[25]
Segformer: Simple and efficient design for semantic segmentation with transformers
Enze Xie, Wenhai Wang, Zhiding Yu, Anima Anandkumar, Jose M Alvarez, and Ping Luo. Segformer: Simple and efficient design for semantic segmentation with transformers. Advances in Neural Information Processing Systems, 34: 0 12077--12090, 2021
2021
-
[26]
Bisenet: Bilateral segmentation network for real-time semantic segmentation
Changqian Yu, Jingbo Wang, Chao Peng, Changxin Gao, Gang Yu, and Nong Sang. Bisenet: Bilateral segmentation network for real-time semantic segmentation. In Proceedings of the European conference on computer vision (ECCV), pp.\ 325--341, 2018
2018
-
[27]
write newline
" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...
Reviewed August 15, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.