{"id":"d1b98f56-15ee-4f27-996b-9746a12337f8","arxiv_id":"2412.02316","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"Deep Q-learning with separate scout and cleaner policies outperforms greedy, PSO, and random baselines for simulated plastic-waste collection, with the largest gains in complex port layouts.","lead":"This paper uses deep reinforcement learning to coordinate two teams of autonomous boats, scouts that map floating plastic and cleaners that collect it, in a simulated port environment. The authors report that the learning-based approach collects more trash than standard greedy, particle swarm, and random-search baselines, especially in complex channel layouts.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Policy selection uses the same 100 episodes that produce the reported metrics, so the claimed DRL advantage may be inflated by selection bias; a separate validation/test split is needed.","rationale":"The reader identifies the same selection-bias issue in the rationale, but selects the sensor/communication idealization as the weakest assumption. I agree that idealizations matter, but the more immediately load-bearing concern is internal to the reported evaluation: the policy used for Table I is chosen on the same episodes used to produce the table, so the reported DRL gains and confidence intervals are not a fair comparison against one-shot baselines. This affects the strongest claim directly. The concrete test is straightforward and does not require reimplementing the environment: split the episodes, reselect checkpoints on validation only, and recompute the table on a fresh test set. If the gap persists, the central claim is substantially supported; if it shrinks, the paper needs a corrected protocol and possibly more modest conclusions. The verdict should remain CONDITIONAL rather than REJECT because the issue is fixable and the paper provides code and training details that would allow the test to be run.","tokens_in":11876,"tokens_out":2617,"duration_ms":33427,"concrete_test":"Re-run the evaluation with a strict train/validation/test split: train for 60,000 episodes, select the best checkpoint on a held-out validation set (e.g., 200 episodes), then compute Table I means and 95% confidence intervals on a fresh test set of 100 episodes never used for selection. Report how many checkpoints were considered during selection. If the Scenario B PTC gap between DDDQL+Greedy and Greedy shrinks materially or loses statistical significance, the current headline should be qualified to reflect that the advantage depends on the selection protocol.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central empirical claim rests on Table I, where DDDQL+Greedy reaches 91.13% PTC in Scenario B versus 72.11% for Greedy. Section IV states: \"The policy chosen is the one that obtains the best performance on average. For the evaluation process, the average results of the same 100 episodes have been recorded for each algorithm.\" If the same 100 episodes are used both to select the DRL checkpoint and to compute the reported metrics, then the DRL numbers are maxima over however many checkpoints were evaluated, while each baseline receives a single score. The confidence intervals in Fig. 6 and Table I are computed on the very episodes used for selection, so they understate variability. This protocol alone can manufacture or inflate the headline gap; the size of the effect depends on the number of checkpoints considered, which is not reported. A separate validation set for checkpoint selection and a fresh test set for final metrics is therefore a precondition for the claim that DRL-based algorithms \"outperform other benchmark heuristics.\" The reader's rationale mentions this selection-bias concern, but the paper's Section IV wording makes it a load-bearing threat to the strongest claim, independent of the sensor and communication idealizations also flagged.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This manuscript proposes a model-free deep reinforcement learning framework for coordinating two teams of autonomous surface vehicles: scouts that maintain a trash-density model and cleaners that collect waste. The method uses DDDQL with team-shared neural networks, an image-like state representation, and role-specific reward functions. Experiments compare two DRL training variants against lawn mower, random walker, PSO, and greedy baselines in two grid scenarios, reporting PTC and MSE over 100 episodes. The authors conclude that DRL-based algorithms outperform the benchmark heuristics, with the largest gap in the narrow-corridor Scenario B (DDDQL+Greedy reaches 91.13% PTC versus 72.11% for the greedy baseline).","tokens_in":12138,"tokens_out":3712,"duration_ms":46179,"significance":"If the empirical comparison is valid, the paper makes a useful contribution to multi-robot informative path planning: it demonstrates a concrete heterogeneous ASV coordination mechanism, provides a detailed simulation setup, and makes the code publicly available. The comparison against external heuristics gives the claim independent grounding, and the inclusion of confidence intervals is a positive feature. However, the headline quantitative claim rests on an evaluation protocol that appears to use the same 100 episodes for both checkpoint selection and final metric reporting, which can inflate the DRL advantage. The simulation also makes strong idealizations about perception and communication. The work is therefore of interest to the community, but the central claim needs a sounder evaluation before it can be accepted.","major_comments":[{"comment":"The checkpoint selection protocol appears to use the same 100 episodes that are later used to compute all reported metrics. Section IV states: \"The policy chosen is the one that obtains the best performance on average. For the evaluation process, the average results of the same 100 episodes have been recorded for each algorithm.\" If the 100 evaluation episodes are also used to select the DRL checkpoint, then the DRL results in Table I are maxima over however many checkpoints were considered, whereas each baseline receives a single score. This alone can inflate the reported DRL advantage and shrink the confidence intervals in Fig. 6, because the variability across checkpoints is not included. Please separate a validation set for checkpoint selection from a fresh test set for final metrics, and report the number of checkpoints considered and the variance over training seeds.","section":"Section IV, Table I and Fig. 6"},{"comment":"The baseline algorithms are not specified sufficiently for reproduction. There is no description of the PSO swarm size, update rule, objective function, or replanning horizon, and the greedy baseline's behavior in the heterogeneous multi-agent setting is not defined (for example, how it breaks ties, how it coordinates the four agents, and how it uses the shared trash model). This matters because the claimed margin over these baselines is the main evidence for the DRL advantage. Please provide precise parameter settings or pseudocode for all baselines, including the lawn mower and random walker route-generation details.","section":"Section IV, baselines"},{"comment":"The simulator assumes that every vehicle within radius rho updates the trash model exactly, with Y_hat[i,j] = Y[i,j], and that centralized communication has no delay or range limits. This is stated in Section III-B, but the introduction and contributions claim operation \"under realistic constraints\" and the conclusion generalizes to real ASV deployments. Because detection errors, missed detections, false positives, communication latency, and water currents beyond wind/random fluctuations are absent, the reported PTC and MSE margins should be interpreted as upper bounds for an idealized setting. The generalization claim should be tempered, or supported by a sensitivity analysis or a set of physical trials.","section":"Section III-B and Section I"}],"minor_comments":[{"comment":"The text uses the acronym \"PCT\" in one sentence (\"both in terms of average MSE and PCT\"), while the metric is defined and used elsewhere as PTC.","section":"Section IV, paragraph after Table I"},{"comment":"The target Q-value expression has an extra closing parenthesis after the argmax; please check the notation: Q_target(s', argmax_{a'}(Q(s'; theta)); theta-) should be written unambiguously.","section":"Equation (1)"},{"comment":"The trash count K is sampled from N(60, 10^2), but a normal distribution can produce negative values; please specify how negative samples are handled.","section":"Section III-B"},{"comment":"The reward weights c_alpha, c_beta, c_gamma, c_delta and the greedy buffer prefill ratio are tuned by trial and error on the two test scenarios; a sensitivity analysis or validation on at least one unseen scenario would make the generalization claim more convincing.","section":"Section IV, hyperparameters"},{"comment":"The caption contains a grammatical error: \"two discretized scenario maps which differences in complexity\" should read \"with differences in complexity.\"","section":"Figure 2 caption"}],"recommendation":"major_revision","confidential_remarks":"The paper is in scope for a robotics journal and the open-source code is a strength. The main issue is the checkpoint-selection protocol, which is fixable within the manuscript's scope by introducing a separate validation/test split. I would not reject on the basis of the simulation idealizations alone, but the authors should also provide fuller baseline specifications."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nYou should know this paper before citing it: the headline claim—DRL beats greedy and PSO on plastic cleanup in a complex harbor—rests on a checkpoint-selection protocol that can manufacture part of the gap. The text says the policy chosen is the one with best average performance on the same 100 episodes used for final metrics. That turns the DRL numbers into a max over checkpoints while each baseline gets one score, and the reported CIs understate variability. The fix is standard: hold out a validation set for selection, then evaluate on fresh test episodes.\n\nWhat is genuinely new: the application of the authors' DDDQL framework (from their earlier water-quality work) to heterogeneous scout/cleaner teams, with role-specific rewards and a dynamic trash model. The two scenarios—open port and narrow wharf—are sensibly chosen, and the comparison includes four baselines plus two DRL variants. Code is released. That is real work.\n\nBeyond selection bias, the perception model is generous: detection is exact within sensor radius, no communication delays, no missed detections. These idealizations are stated, but they matter if the claim is about real fleets. Baselines are also under-specified; PSO and greedy details are not enough to reproduce exactly.\n\nThe central comparison is plausible, and the DRL-plus-greedy-training improvement is a nice practical finding. But the selection-bias issue is load-bearing because it directly affects the size of the headline advantage. A referee should ask for a proper validation/test split and a sensitivity check on the number of checkpoints considered.\n\nFor you: worth a look if you work on multi-robot informative path planning or sim-to-real transfer for cleanup. I would not cite it until the evaluation protocol is fixed. It deserves a serious referee, not a desk reject.","headline":"A useful simulation study of DRL for heterogeneous cleanup fleets, but the headline performance gap may be inflated by selecting the DRL checkpoint on the same 100 episodes used to report final metrics.","tokens_in":12644,"tokens_out":2007,"would_cite":false,"duration_ms":21934,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A deep-reinforcement-learned fleet of scout and cleaner boats collects more floating plastic than greedy, PSO, or pattern-based planners, especially in narrow, obstacle-heavy ports.","keywords":["heterogeneous multi-robot systems","autonomous surface vehicles","deep reinforcement learning","informative path planning","Dueling Double Deep Q-Learning","plastic waste collection","multi-agent coordination","waterway monitoring"],"falsifier":"A concrete test: run the trained policy on a physical test course with known trash placements and measure PTC against both the simulated value and a greedy baseline; if detection misses or radio delays push the real learned-policy PTC at or below the greedy policy's in a corridor layout, the adaptability claim fails. A cheaper simulation-based check is to inject a realistic detection probability (for example 90% within radius $\\rho$, consistent with the stated vision-model accuracy) and re-run the 100-episode evaluation; if the DDDQL+Greedy margin over greedy collapses, the result depends on the exact-perception assumption rather than on the learned coordination.","tokens_in":11702,"feed_emoji":"🚤","tokens_out":12420,"duration_ms":110344,"temperature":0.7,"pith_summary":"This paper tries to establish that a heterogeneous fleet of autonomous surface vehicles—fast scouts that map floating plastic with wide-range cameras and slower cleaners that collect it—coordinates more effectively under deep reinforcement learning than under classical heuristics. The proposed framework, Dueling Double Deep Q-Learning with a shared per-team network, uses the trash model as the nexus between teams: scouts are rewarded for updating the map and covering new ground, cleaners for collecting trash and staying near known waste. In the harder of two simulated layouts, a narrow-corridor sport wharf, the best learned policy cleans 91.13% of the trash against 72.11% for the greedy heuristic and 67.19% for particle swarm optimization, with smaller confidence intervals across random episodes. The authors conclude that model-free learned policies generalize across layouts without per-scenario heuristic tuning, which matters for real cleanup missions where waste drifts with wind and currents.","feed_headline":"91% of floating plastic cleaned by a learned two-team boat fleet","feed_subtitle":"Scout and cleaner ASVs trained with Dueling Double Deep Q-Learning beat greedy and PSO baselines in cluttered waterways.","key_machinery":"The load-bearing mechanism is the shared contamination model $\\hat{Y}$, updated at every step by any vehicle whose sensor radius $\\rho$ reaches a node, and this model is part of the state input to both teams' neural networks, making it the nexus of cooperation. Policies are learned by Dueling Double Deep Q-Learning: two estimates of the action-value function reduce Q-value overestimation, a dueling stream separates state value from action advantage, and prioritized experience replay, shared within each team, stabilizes training. The reward design encodes the division of labor—cleaners receive a large weight for trash collected and a penalty proportional to distance to known trash, while scouts receive bonuses for covering unexplored nodes and for producing changes in the model. The state representation is a stack of six image-like matrices: navigability with coverage, the trash model at times $t$, $t-1$, and $t-2$, the observing agent's fading ten-step trail, and the positions of the other agents.","core_discovery":"The paper's central claim is that inter-team cooperation between scouts and cleaners can be learned end-to-end with model-free deep reinforcement learning, and that the learned policy outperforms benchmark heuristics on both the percentage of trash cleaned (PTC) and the accuracy of the shared contamination model. Cooperation is deliberately engineered through the trash model $\\hat{Y}$: the cleaner team's decisions depend on the map the scout team produces, so the better the scouts map, the easier the cleaners' job becomes. In the challenging Scenario B, the DDDQL variant trained with greedy actions reaches 91.13% PTC with a 95% confidence interval of $\\pm 2.38$, versus 72.11% ($\\pm 7.10$) for greedy and 67.19% ($\\pm 6.80$) for PSO, while also achieving the lowest final mean squared error (MSE) of all compared methods. The authors further claim that training with greedy actions and pre-filling the replay buffer improves final performance in intricate layouts, whereas in the open Scenario A the plain DDDQL already matches it, and that the millisecond-level inference time is negligible at real ASV speeds.","pith_inferences":["Because the simulation treats detection within radius $\\rho$ as exact, an immediate extension is to feed the scout reward with detector confidence or false-negative rates; if the 91% result relies on perfect perception, the gap over greedy may close when detection noise is injected.","The MSE advantage may partly be a side effect of faster cleaning—fewer remaining trash items make the map trivially easier to estimate—so an ablation with scouts mapping but cleaners disabled would isolate genuine mapping quality.","The greedy-seeding training recipe generalizes beyond trash collection: for off-policy DRL in sparse-reward robotic tasks, pre-filling the replay buffer with a strong heuristic's trajectories may speed learning more than random exploration.","The problem is framed as a finite cleanup, but persistent waste influx (a river continuously depositing plastic) would turn it into sustained monitoring; the current reward structure would need a coverage-ratio term to avoid the fleet quitting after the initial batch."],"forward_implications":["A fixed, model-free DRL policy trained once transfers across structurally different port layouts without per-scenario tuning of heuristics.","In corridor-heavy environments the learned policy beats greedy by up to 26 percentage points in the percentage of trash cleaned (PTC), so cluttered waterways are exactly where this approach earns its complexity.","Fleet size can grow with little added learning cost, since agents within a team share one network and one experience replay.","The narrower confidence intervals of the DRL results mean more predictable performance across random starting positions, trash distributions, and wind conditions.","The roughly 19 ms per-fleet decision time is far below the second-scale movement of real ASVs, so the computational overhead does not block real-time use."],"supporting_citations":[{"why":"Direct precedent: applies DDDQL with prioritized experience replay to heterogeneous ASV fleets; the architecture and evaluation style this paper extends to scouts-plus-cleaners cooperation.","marker":"[15]"},{"why":"Supplies the multi-agent DRL framework for water monitoring with shared networks and safety coordination, adapted here for two specialized teams.","marker":"[11]"},{"why":"Defines Double Q-Learning, the base algorithm whose two estimators mitigate Q-value overestimation.","marker":"[22]"},{"why":"Defines the dueling network structure that splits value and advantage streams, completing the DDDQL variant used.","marker":"[25]"},{"why":"Defines prioritized experience replay, which stores and samples the transitions both teams train on.","marker":"[24]"},{"why":"Grounds the greedy algorithm's near-optimality guarantee, motivating both its use as a benchmark and as the source of seeding actions.","marker":"[28]"},{"why":"The closest prior heterogeneous system (drones detect, surface vehicles collect) that this paper contrasts its fully learned coordination against.","marker":"[20]"},{"why":"Provides the floating-waste detection dataset that backs the perception model assumed in the sensor-radius update rule.","marker":"[21]"}],"fun_headline_variants":["AI boat duo cleans 91% of floating waste in cluttered waters","Two-team deep RL boats pick up 91% of plastic","Learned scout-cleaner boats beat heuristics in trash collection","Deep RL cleans 91% of litter with cooperating boat teams","Autonomous boat fleet uses deep RL to clean 91% of waste"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that every piece of trash inside a vehicle's sensor radius is detected exactly and shared instantly, since the simulation ignores detection errors, missed detections, communication delays, and range limits; if the real perception and networking chain deviates from this, the reported cleaning and mapping advantages may not reproduce on physical boats.","fun_headline_variants_meta":{"raw":{"variants":["AI boat duo cleans 91% of floating waste in cluttered waters","Two-team deep RL boats pick up 91% of plastic","Learned scout-cleaner boats beat heuristics in trash collection","Deep RL cleans 91% of litter with cooperating boat teams","Autonomous boat fleet uses deep RL to clean 91% of waste"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000584,"raw_usage":{"total_tokens":2753,"prompt_tokens":959,"completion_tokens":1794,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":575,"completion_tokens_details":{"reasoning_tokens":1703}},"tokens_in":575,"tokens_out":1794,"duration_ms":12680,"temperature":1.0,"reasoning_tokens":1703,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T23:35:37.653774+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A concrete test: run the trained policy on a physical test course with known trash placements and measure PTC against both the simulated value and a greedy baseline; if detection misses or radio delays push the real learned-policy PTC at or below the greedy policy's in a corridor layout, the adaptability claim fails. A cheaper simulation-based check is to inject a realistic detection probability (for example 90% within radius $\\rho$, consistent with the stated vision-model accuracy) and re-run the 100-episode evaluation; if the DDDQL+Greedy margin over greedy collapses, the result depends on the exact-perception assumption rather than on the learned coordination.","supporting_citations":[{"cited_title":"Informative deep reinforcement path planning for heterogeneous au- tonomous surface vehicles in large water resources,","cited_arxiv_id":null,"evidence_quote":"Direct precedent: applies DDDQL with prioritized experience replay to heterogeneous ASV fleets; the architecture and evaluation style this paper extends to scouts-plus-cleaners cooperation."},{"cited_title":"Deep reinforcement multiagent learning framework for infor- mation gathering with local gaussian processes for water monitoring,","cited_arxiv_id":null,"evidence_quote":"Supplies the multi-agent DRL framework for water monitoring with shared networks and safety coordination, adapted here for two specialized teams."},{"cited_title":"Deep reinforcement learning with double q-learning,","cited_arxiv_id":null,"evidence_quote":"Defines Double Q-Learning, the base algorithm whose two estimators mitigate Q-value overestimation."},{"cited_title":"Dueling network architectures for deep reinforcement learning,","cited_arxiv_id":null,"evidence_quote":"Defines the dueling network structure that splits value and advantage streams, completing the DDDQL variant used."},{"cited_title":"Prioritized experience replay,","cited_arxiv_id":null,"evidence_quote":"Defines prioritized experience replay, which stores and samples the transitions both teams train on."},{"cited_title":"Greed is good: Near-optimal submodular maximization via greedy optimization,","cited_arxiv_id":null,"evidence_quote":"Grounds the greedy algorithm's near-optimality guarantee, motivating both its use as a benchmark and as the source of seeding actions."},{"cited_title":"Automatic collaborative water surface coverage and cleaning strategy of UA V and USVs,","cited_arxiv_id":null,"evidence_quote":"The closest prior heterogeneous system (drones detect, surface vehicles collect) that this paper contrasts its fully learned coordination against."},{"cited_title":"Flow: A dataset and bench- mark for floating waste detection in inland waters,","cited_arxiv_id":null,"evidence_quote":"Provides the floating-waste detection dataset that backs the perception model assumed in the sensor-radius update rule."}],"review_version":1}