REVIEW 3 major objections 3 minor 3 cited by
Cooperative Aerial Robot Inspection Challenge: A Benchmark for Heterogeneous Multi-UAV Planning and Lessons Learned
T0 review · 3 major / 3 minor · reviewed 2026-08-10 · deepseek-v4-flash
Pith's one-line read The paper claims that CARIC, a simulation benchmark for heterogeneous multi-UAV inspection, provides a ready-to-use platform for developing and comparing planning algorithms, and that its first competition shows no single approach yet…
desk verdict Useful benchmark artifact, but the lessons-learned section is compromised by undisclosed author-team overlap. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is the benchmark itself: a simulation stack built on a widely used open-source robotics simulator, with a heterogeneous fleet, a line-of-sight communication router that drops messages between occluded drones, and an evaluation function $Q = \sum_{i} \max_j q_{i,j}$, where $q_{i,j} = \max_k (q_{\mathrm{seen}} \cdot q_{\mathrm{blur}} \cdot q_{\mathrm{res}})$. The load-bearing identity is the multiplicative quality score: motion blur depends on pixel displacement during exposure (Equation 3), and spatial resolution is expressed as millimeter-per-pixel compared with a desired threshold (Equation 6). This score is what forces algorithms to balance speed, viewpoint distance, and camera aim, and the absence of any prior map is what makes exploration, mapping, and inspection need to be coordinated online under intermittent line-of-sight communication.
What would settle it
Run the same algorithms in the field on a structure with known defect locations, have experts label which captured images are usable for defect detection, and check whether the order of teams by CARIC score matches the order by expert-rated detection accuracy; a mismatch would refute the benchmark's validity.
Extended reading notes
Core claim
CARIC operationalizes heterogeneous multi-UAV inspection as a scoring problem: explorers with LiDAR plus a camera build an online point-cloud map of structures enclosed by given bounding boxes, while photographers with only a camera fly to viewpoints to capture interest points; each image earns a quality score $q = q_{\mathrm{seen}} \cdot q_{\mathrm{blur}} \cdot q_{\mathrm{res}}$, where $q_{\mathrm{seen}}$ requires line-of-sight and field-of-view, $q_{\mathrm{blur}}$ penalizes pixel motion during exposure (Eq. 3), and $q_{\mathrm{res}}$ penalizes spatial resolution coarser than a desired millimeter-per-pixel (Eq. 6). The total score is the sum over interest points of the best image any drone captured. The authors' finding from the first competition is that the three top solutions embody a fundamental trade-off between exploration thoroughness and inspection time, between fast continuous scanning and sharp stop-and-scan imaging, and between simple volume-based task partitioning and workload-aware allocation, and that no single design wins all scenarios.
Load-bearing premise
The benchmark's ranking assumes the simulated line-of-sight, motion-blur, and spatial-resolution metrics faithfully capture whether images are good enough for real structural defect analysis; until those metrics are calibrated against real inspection outcomes, a high CARIC score may not mean a high real-world inspection quality.
Editorial extensions
If this is right
- Treating exploration and inspection as separate sequential phases can leave photographers idle for long stretches; the fastest mapper in the competition won two scenarios but missed thin structures in a third.
- Task allocation based on bounding-box volume is a poor workload proxy; the team that allocated by number of inspection viewpoints achieved more consistent coverage.
- Continuous fast camera motion detects more candidate interest points, but the blur metric penalizes it heavily, so stop-and-scan yields lower recall but higher per-point quality and suggests role specialization among photographers.
- Run-to-run variance of over 1000 score points in some scenarios shows that robustness of pathfinding and initialization is as important as the planning strategy itself.
- No single submitted approach dominated all three scenes, so the benchmark can differentiate future multi-UAV inspection planners.
Reading between the lines
- The multiplicative score could be used as a differentiable objective in trajectory optimization: since $q_{\mathrm{blur}}$ depends on image-plane velocity, a planner could trade speed against expected blur directly rather than using waypoint stops as a proxy.
- The benchmark's line-of-sight communication model invites communication-aware task allocation as an explicit scoring axis, such as counting messages or map freshness, which the current score ignores.
- If the simulated blur and resolution metrics are not calibrated against real defect-detection performance, CARIC rankings may not transfer to field deployments; a validation study comparing CARIC scores with expert crack-detection rates on captured images would settle this.
- The observation that volume-based workload estimation fails suggests inspection workload should be measured by surface area and viewpoint count, a metric the paper's own analysis supports but does not formalize as a benchmark statistic.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces CARIC, a Gazebo/RotorS-based simulation benchmark for heterogeneous multi-UAV inspection planning, in which teams of LiDAR-equipped explorers and camera-equipped photographers must inspect interest points on structures known only by bounding boxes, subject to line-of-sight communication and image-quality scoring (blur and resolution). The paper reports the CARIC competition held at CDC 2023, presents the approaches of the top three teams (KIOS CoE, XXH, STAR), and draws lessons about exploration-vs.-inspection trade-offs, inspection quality vs. detection rate, task allocation, and performance variance. The central empirical claim is that no single approach achieved robust performance across all scenarios.
Significance. If the benchmark is adopted by the community, CARIC would provide a useful standardized testbed for a previously underexplored problem: cooperative heterogeneous UAV inspection without a prior structural model. The paper's strengths include a publicly available, ready-to-use simulation stack with clearly defined scoring metrics (Eqs. 1–6), a concrete competition protocol with multiple runs and two hardware configurations, and an explicit statement that rankings were consistent across those configurations. The authors also provide qualitative per-scenario analyses with illustrative trajectory and detection maps. However, the empirical lessons are substantially weakened by two issues: all three analyzed teams are from the authors' own institutions, and per-run scores are not released. These concerns must be addressed before the paper's conclusions about community-wide performance can be accepted.
major comments (3)
- [III and V] The three 'top teams' analyzed in Sections IV and V — KIOS CoE (University of Cyprus), XXH (Nanyang Technological University), and STAR (Sun Yat-sen University) — all include members of this paper's author list, as shown by the affiliations in the author block. The manuscript nowhere discloses this overlap. Because the lessons in Section V are drawn exclusively from these self-affiliated teams, the central claim that no single approach has achieved robust performance across scenarios (Section VI) is not an independent assessment of the broader participant pool. The paper should (a) explicitly disclose the author–team overlap, (b) release per-run scores for all teams or at least for the analyzed teams, and (c) provide a separate analysis of non-organizer teams if such data exist. Without this, the lessons learned are better characterized as a self-assessment of the organizers' own algorithms.
- [V-D2] The claim that no single approach is robust across all scenarios rests on box plots for only three teams, with no statistical tests, effect sizes, or baseline comparisons. This is especially concerning because Section V-D1 reports within-team variance of over 1000 points between runs for XXH and STAR; the observed cross-scenario differences may be partly due to run-to-run noise. The paper should report per-run scores, indicate the number of runs used for each box in Figure 9, and either add significance testing or explicitly state that the cross-scenario comparison is anecdotal given n=3 teams.
- [II-D3 (Eq. 6)] The resolution metric q_res uses the parameter rdes as the desired MMPP, and the introduction to Section II states that the metrics 'ensure the captured data meets standards for structural analysis.' However, no calibration or validation is provided linking q_blur and q_res to actual defect detection or structural inspection outcomes. As a result, the benchmark's rankings may not transfer to field deployment, and the claim that the scores reflect inspection quality for structural analysis is currently unsupported. Either provide a calibration study against real inspection data or soften the wording to describe these as heuristic image-quality proxies.
minor comments (3)
- [II-D2, Eq. (4)] There is a typo in the definition of v1: it is printed as 'v1 = f · z1/z1', which should presumably be 'v1 = f · y1/z1'.
- [III and Figure 9] The text says teams are ranked by the max score across five runs, while Figure 9 shows box plots of 'overall scores across tests.' Please clarify whether the box plots summarize the five runs per team, and indicate the number of runs and whether any runs were excluded (e.g., due to crashes).
- [III] The statement 'The rankings were consistent across both hardware setups' is not backed by data. A small table or correlation coefficient showing per-team ranks on each machine would support this claim.
Circularity Check
No significant circularity: the benchmark claim is supported by a public codebase and external competition runs, while the author-team overlap is a reporting/fairness concern rather than a definitional or self-citation circularity.
full rationale
The paper contains no derivation chain in which a predicted quantity is equivalent to a fitted input by construction. The central claims are that CARIC is a ready-to-use benchmark and that the CDC 2023 competition results show no single approach is robust across all scenarios. The first claim is supported by the publicly available simulation stack, and the second is an empirical observation of competition scores, not a quantity defined by the scoring equations. The evaluation metrics in Eqs. (1)-(6) are explicitly stated design choices; they compute scores from simulated observations rather than fitting parameters and then renaming them as predictions. The self-citations that appear, such as [16]-[20] for mTSP, FUEL, RACER, Star-Searcher, and Fast-Planner, describe algorithmic components used by the competing teams and are not the sole justification for CARIC's validity or for the lessons learned. The undisclosed overlap between the top three teams and the authors' institutions is a legitimate concern about external validity and transparency, but it is a conflict-of-interest and reporting issue, not a circularity of the kind defined in the review criteria. No equation reduces to another by construction, no fitted parameter is presented as a prediction, and no load-bearing claim depends on an unverified self-citation. Therefore, no circular step is identified.
Assumptions & free parameters
free parameters (4)
- c (pixel width)
- tau (exposure duration)
- rdes (desired resolution)
- mission time budgets =
600s for MBS and Crane, 240s for Hangar
assumptions (5)
- domain assumption Ground truth odometry is available to every UAV in the global frame.
- domain assumption An underlying multi-robot localization and mapping system is assumed available, giving UAVs neighbor odometry and keyframe point clouds under line of sight.
- domain assumption Interest points are sampled on the surfaces inside the given bounding boxes.
- domain assumption Communication between UAVs is restricted to line of sight and messages are dropped otherwise.
- domain assumption Only UAVs that complete the mission without collision are counted in the final score.
Cite this review
Pith. "Pith review of Cooperative Aerial Robot Inspection Challenge: A Benchmark for Heterogeneous Multi-UAV Planning and Lessons Learned." pith.science (2026). https://pith.science/paper/P3CE6OTU
@misc{pith2026250106566,
author = {Pith},
title = {Pith review of: Cooperative Aerial Robot Inspection Challenge: A Benchmark for Heterogeneous Multi-UAV Planning and Lessons Learned},
year = {2026},
howpublished = {\url{https://pith.science/paper/P3CE6OTU}},
note = {Machine review of arXiv:2501.06566}
}
read the original abstract
We propose the Cooperative Aerial Robot Inspection Challenge (CARIC), a simulation-based benchmark for motion planning algorithms in heterogeneous multi-UAV systems. CARIC features UAV teams with complementary sensors, realistic constraints, and evaluation metrics prioritizing inspection quality and efficiency. It offers a ready-to-use perception-control software stack and diverse scenarios to support the development and evaluation of task allocation and motion planning algorithms. Competitions using CARIC were held at IEEE CDC 2023 and the IROS 2024 Workshop on Multi-Robot Perception and Navigation, attracting innovative solutions from research teams worldwide. This paper examines the top three teams from CDC 2023, analyzing their exploration, inspection, and task allocation strategies while drawing insights into their performance across scenarios. The results highlight the task's complexity and suggest promising directions for future research in cooperative multi-UAV systems.
Figures
Figures from the paper (7 more)
Forward citations
Cited by 3 Pith papers
-
FleetScape: A Mixed Reality Sandtable for Spatial Supervision and Control of Scalable Drone Fleets
A mixed-reality 3D sandtable lets single operators supervise fleets of 5–15 simulated inspection drones, with situational awareness degrading beyond roughly 10 drones.
-
Audio Array-Based 3D UAV Trajectory Estimation with LiDAR Pseudo-Labeling
An audio-only neural network predicts 3D UAV trajectories with 0.48 m average position error on MMAUD, trained with unsupervised LiDAR trajectory estimates as pseudo-labels.
-
Multi-agent Embodied AI: Advances and Future Directions
A survey that maps multi-agent embodied AI methods and benchmarks across control, learning, and generative-model categories, and lists open challenges.
Reference graph
Works this paper leans on
-
[1]
A. Bircher, K. Alexis, M. Burri, P. Oettershagen, S. Omari, T. Mantel, and R. Siegwart, “Structural inspection path planning via iterative viewpoint resampling with application to aerial robotics,” in 2015 IEEE International Conference on Robotics and Automation (ICRA) . IEEE, 2015, pp. 6423–6430
work page 2015
-
[2]
Towards fully automated unmanned aerial vehicle- enabled bridge inspection: Where are we at?
C. Zhang, Y . Zou, F. Wang, E. del Rey Castillo, J. Dimyadi, and L. Chen, “Towards fully automated unmanned aerial vehicle- enabled bridge inspection: Where are we at?” Construction and Building Materials , vol. 347, p. 128543, 2022. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S0950061822022036
work page 2022
-
[3]
An autonomous aircraft inspection system using collaborative unmanned aerial vehicles,
A. Saha, L. Kumar, S. Sortee, and B. C. Dhara, “An autonomous aircraft inspection system using collaborative unmanned aerial vehicles,” in2023 IEEE Aerospace Conference, 2023, pp. 1–10
work page 2023
-
[4]
Cooperative coverage path planning for visual inspection,
S. S. Mansouri, C. Kanellakis, E. Fresk, D. Kominiak, and G. Nikolakopoulos, “Cooperative coverage path planning for visual inspection,” Control Engineering Practice , vol. 74, pp. 118–131, 2018. [Online]. Available: https://www.sciencedirect.com/science/article/pii/ S0967066118300315
work page 2018
-
[5]
Bim- supported drone path planning for building exterior surface inspection,
X. Huang, Y . Liu, L. Huang, S. Stikbakke, and E. Onstein, “Bim- supported drone path planning for building exterior surface inspection,” Computers in Industry , vol. 153, 12 2023
work page 2023
-
[6]
Re- ceding horizon path planning for 3d exploration and surface inspection,
A. Bircher, M. Kamel, K. Alexis, H. Oleynikova, and R. Siegwart, “Re- ceding horizon path planning for 3d exploration and surface inspection,” Autonomous Robots, vol. 42, pp. 291–306, 2018
work page 2018
-
[7]
C. Papachristos, M. Kamel, M. Popovi ´c, S. Khattak, A. Bircher, H. Oleynikova, T. Dang, F. Mascarich, K. Alexis, and R. Siegwart, Autonomous Exploration and Inspection Path Planning for Aerial Robots Using the Robot Operating System . Cham: Springer International Publishing, 2019, pp. 67–111. [Online]. Available: https://doi.org/10.1007/978-3-319-91590-6 3
-
[8]
View path planning via online multiview stereo for 3-d modeling of large-scale structures,
S. Song, D. Kim, and S. Choi, “View path planning via online multiview stereo for 3-d modeling of large-scale structures,” IEEE Transactions on Robotics, vol. 38, no. 1, pp. 372–390, 2022
work page 2022
Show all 20 references
-
[9]
Multi-uav trajectory planning for 3d visual inspection of complex structures,
S. Ivi ´c, B. Crnkovi´c, L. Grbˇci´c, and L. Matlekovi´c, “Multi-uav trajectory planning for 3d visual inspection of complex structures,” Automation in Construction, vol. 147, p. 104709, 2023
2023
-
[10]
Multi-uav coverage path planning for the inspection of large and complex structures,
W. Jing, D. Deng, Y . Wu, and K. Shimada, “Multi-uav coverage path planning for the inspection of large and complex structures,” in 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 2020, pp. 1480–1486
2020
-
[11]
A multirobot system for 3-d surface reconstruction with centralized and distributed architectures,
G. Hardouin, J. Moras, F. Morbidi, J. Marzat, and E. M. Mouaddib, “A multirobot system for 3-d surface reconstruction with centralized and distributed architectures,” IEEE Transactions on Robotics , vol. 39, pp. 2623–2638, 8 2023
2023
-
[12]
Rotors—a modular gazebo mav simulator framework,
F. Furrer, M. Burri, M. Achtelik, and R. Siegwart, “Rotors—a modular gazebo mav simulator framework,” Robot Operating System (ROS) The Complete Reference (Volume 1) , pp. 595–625, 2016
2016
-
[13]
Autonomous underground flight with m300 rtk and the emesent hovermap
Emesent, “Autonomous underground flight with m300 rtk and the emesent hovermap.” [Online]. Available: https://enterprise-insights.dji. com/blog/m300-rtk-emesent-hovermap-autonomous-underground-flight
-
[14]
Rapid in-flight image quality check for uav-enabled bridge inspection,
F. Wang, Y . Zou, X. Chen, C. Zhang, L. Hou, E. del Rey Castillo, and J. B. Lim, “Rapid in-flight image quality check for uav-enabled bridge inspection,” ISPRS Journal of Photogrammetry and Remote Sensing, vol. 212, pp. 230–250, 6 2024
2024
-
[15]
Evaluation and enhancement of resolution-aware coverage path planning method for surface inspection using unmanned aerial vehicles,
W. Wu, Y . Funabora, S. Doki, K. Doki, S. Yoshikawa, T. Mitsuda, and J. Xiang, “Evaluation and enhancement of resolution-aware coverage path planning method for surface inspection using unmanned aerial vehicles,” IEEE Access, vol. 12, pp. 16 753–16 766, 2024
2024
-
[16]
Swarm path planning for the deployment of drones in emergency response missions,
A. Anastasiou, P. Kolios, C. Panayiotou, and K. Papadaki, “Swarm path planning for the deployment of drones in emergency response missions,” in 2020 International Conference on Unmanned Aircraft Systems (ICUAS). IEEE, 2020, pp. 456–465
2020
-
[17]
Fuel: Fast uav exploration using incremental frontier structure and hierarchical planning,
B. Zhou, Y . Zhang, X. Chen, and S. Shen, “Fuel: Fast uav exploration using incremental frontier structure and hierarchical planning,” IEEE Robotics and Automation Letters , vol. 6, no. 2, pp. 779–786, 2021
2021
-
[18]
Star-searcher: A complete and efficient aerial system for autonomous target search in complex unknown environments,
Y . Luo, Z. Zhuang, N. Pan, C. Feng, S. Shen, F. Gao, H. Cheng, and B. Zhou, “Star-searcher: A complete and efficient aerial system for autonomous target search in complex unknown environments,” 2024
2024
-
[19]
Racer: Rapid collaborative exploration with a decentralized multi-uav system,
B. Zhou, H. Xu, and S. Shen, “Racer: Rapid collaborative exploration with a decentralized multi-uav system,” IEEE Transactions on Robotics, 2023
2023
-
[20]
Robust and effi- cient quadrotor trajectory generation for fast autonomous flight,
B. Zhou, F. Gao, L. Wang, C. Liu, and S. Shen, “Robust and effi- cient quadrotor trajectory generation for fast autonomous flight,” IEEE Robotics and Automation Letters , vol. 4, no. 4, pp. 3529–3536, 2019
2019
Reviewed August 10, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.