Towards Bio-inspired Heuristically Accelerated Reinforcement Learning for Adaptive Underwater Multi-Agents Behaviour

Antoine Vivien; Benoit Clement; Eva Artusi; Karl Sammut; Matthew Stephenson; Paulo Santos; Thomas Chaffre

REVIEW 4 major objections 5 minor 48 references

Towards Bio-inspired Heuristically Accelerated Reinforcement Learning for Adaptive Underwater Multi-Agents Behaviour

T0 review · 4 major / 5 minor · reviewed 2026-08-08 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that adding Particle Swarm Optimization to the epsilon-greedy exploration of Multi-Agent Soft Actor-Critic lets three simulated underwater vehicles learn a 2D coverage task with fewer environment interactions, which it…

desk verdict A clear, honest HARL extension whose central training-time claim is unsupported because the paper never measures wall-clock cost. read the letter →

arxiv 2502.06113 v1 pith:5BSM74E6 submitted 2025-02-10 cs.RO cs.SYeess.SY

Antoine Vivien , Thomas Chaffre , Matthew Stephenson , Eva Artusi , Paulo Santos , Benoit Clement , Karl Sammut This is my paper · ORCID

classification cs.ROcs.SYeess.SY

keywords coverageplanningmulti-agentreinforcementlearningParticleSwarmOptimizationheuristicallyacceleratedMASACautonomousunderwatervehiclesexplorationstrategy

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper tries to establish that swapping random exploration for Particle Swarm Optimization (PSO) proposals inside Multi-Agent Soft Actor-Critic (MASAC) shortens the learning phase of a multi-agent coverage task. The motivation is that MARL's long training times confine underwater autonomy to simulation, so anything that reduces the number of environment interactions could move these systems closer to real deployment. In a customizable 2D coverage environment with three agents, the authors report that PSO-guided epsilon-greedy exploration reaches higher mean rewards earlier in training than the same algorithm with random exploration. The paper presents these as preliminary results supporting the hypothesis that heuristics in the exploration strategy lead to lower training time.

What carries the argument

The central object is a PSO-guided epsilon-greedy exploration rule. Whenever the policy chooses to explore, instead of sampling a uniform random action, the agent executes the best candidate found by PSO, which in the reported runs uses 50 particles and 100 iterations to maximize a fitness function identical to the RL reward function. This mechanism is attached to MASAC, a multi-agent variant of the Soft Actor-Critic actor-critic algorithm with stochastic policies and a replay buffer; the intended effect is to bias exploration toward high-quality regions of the action space from the first episode, sharpening the exploitation-exploration trade-off and accelerating policy-gradient updates.

What would settle it

A direct falsifier is a wall-clock comparison: run the same 2D coverage task with and without PSO, counting not just episodes but total computation time including PSO's 100 iterations over 50 particles for every exploration step; if the PSO-trained agent reaches the same coverage in fewer episodes but not in less time, the central claim of reduced training time fails. A second check is to compare PSO's proposed actions with actions found by a fine exhaustive search over a small action grid, since the proposed mechanism requires those proposals to actually be high-reward.

Watch

Extended reading notes

Core claim

The central claim is that heuristically accelerated exploration, specifically PSO embedded in epsilon-greedy action selection, lowers the training time of MASAC on a continuous 2D coverage task. Because the PSO fitness function is defined to be the same as the MARL reward function, the actions PSO proposes are associated with high rewards from the very beginning of training. Since policy-gradient updates work by comparing actions through the advantage function, feeding the agent better-than-average actions early lets it improve its return immediately rather than waiting to stumble on such actions by chance. The paper states this as: "These preliminary results support the hypothesis that the addition of heuristics within the exploration strategy leads to lower training time."

Load-bearing premise

The acceleration claim assumes that PSO's proposed actions are cheap enough and good enough that the total training time really drops, but the paper neither measures PSO's computational cost nor verifies that its proposals are near-optimal.

Editorial extensions

If this is right

MASAC should reach the same coverage performance in fewer episodes when exploration is guided by PSO, reducing the number of simulator interactions needed before the policy is usable.
Because the PSO fitness is the RL reward itself, the same insertion can in principle be applied to any epsilon-greedy MARL algorithm without changing the reward design.
Providing high-reward actions early makes each policy-gradient update more informative, since advantage estimates are computed relative to better-than-average actions.
If the interaction savings survive the overhead of running PSO, the approach is a step toward learning policies on-board real autonomous underwater vehicles, where each interaction costs battery and mission time.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The paper counts episodes, not wall-clock time, so the logical next test is to measure total computation time including PSO's 100 iterations over 50 particles at every exploration step; the headline claim stands or falls on that comparison.
The reported acceleration was measured with three homogeneous agents on a 30-by-30 grid; a natural extension is to check how the advantage behaves as the number of agents grows, since PSO's search space expands with the joint action dimension.
PSO's own hyperparameters (particles, iterations, inertia, cognitive and social coefficients) are part of the training cost, so an adaptive or learned PSO configuration might amplify the acceleration or erase it entirely.
The paper itself notes that switching to the automatic-entropy version of MASAC creates stability challenges; a fair extension would test whether PSO-guided exploration also stabilizes that variant.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

The short version: this is a straightforward but legitimate extension of HARL — replace epsilon-greedy random actions with PSO-guided actions in MASAC, for a 2D coverage task. It is not a groundbreaking idea, but it is a sensible combination that I do not think is in the literature, and the paper is refreshingly clear about its own limits. The problem is that the central claim — "reduces training time" — is not supported by the evidence they present.

What is good: The related work on HARL and bio-inspired methods is competent and appropriately cites Bianchi et al. and the HARL line. The environment is described in enough detail to be reimplemented (3 agents, 30x30 grid, 60-degree forward sensor, reward function). The authors are explicitly honest that these are "preliminary results" and they list sensible future work. The idea of injecting domain knowledge through PSO into the exploration policy is a reasonable research direction.

What is soft: The results section shows only two qualitative learning curves, no numbers, no error bars, no seed counts. More importantly, the headline metric is training time, but they only report environment interactions (episodes or steps). They never measure wall-clock time or compute the PSO overhead. Section 3.1 gives 50 particles and 100 iterations per PSO call; that is a lot of computation inside the learning loop, and it could easily outweigh the savings in environment steps. The stress-test note is right: "fewer interactions" is not "lower training time." On top of that, the PSO fitness function is identical to the reward function, so PSO is effectively a local planner optimizing the same objective. That is not circular in a logical sense, but it makes the acceleration unsurprising and raises a fairness question about whether you are comparing RL to RL plus a planner.

I agree with the reader's verdict: reject as-is. But it is not a worthless paper. The kernel is worth pursuing, and with wall-clock time measured, PSO cost reported, and multiple seeds with variance, it could become a decent conference paper. The citation pattern looks fine; the self-citations are relevant and the HARL citations are proper.

Who is it for? People working on HARL, multi-agent coverage, or sim-to-real for AUVs might skim it. I would send it to review if I were the editor, because the idea is reasonable and the authors are clearly able to respond to feedback. It just needs substantially more evidence.

My recommendation: if you are reviewing it, reject with clear, constructive comments focusing on the missing wall-clock measurement and the reward-function confounding. If you are the editor, send it out — a good referee can push it toward something publishable.

Referee Report

4 major / 5 minor

Summary. The paper addresses the coordination of a multi-agent system (MAS) of autonomous underwater vehicles (AUVs) for coverage planning and detection of objects of interest. It proposes combining Multi-Agent Soft Actor-Critic (MASAC) with Particle Swarm Optimization (PSO) as a biologically inspired heuristic: during epsilon-greedy exploration, the random action is replaced by a PSO-output action that maximizes a fitness function set equal to the MARL reward. The authors report preliminary results on a 2D coverage task with three agents and claim that the method reduces training time by requiring fewer environment interactions.

Significance. The problem is practically important: MARL methods for underwater and space multi-agent missions suffer from long training times, and a method that reliably accelerates learning without sacrificing performance would be valuable. The paper also contributes a customizable coverage environment and a clear integration of PSO into the epsilon-greedy scheme. However, the significance as demonstrated is limited: the central claim of reduced training time rests on qualitative plots with no quantitative support, and the PSO's computational overhead is not measured. The design choice of using the reward function as the PSO fitness function makes the 'heuristic' effectively an online optimizer of the target objective, which weakens the claim that bio-inspired heuristics per se improve MARL. If the authors can provide rigorous evidence and an honest accounting of computational cost, the idea could be a useful step toward reducing MARL training time in simulation-to-real settings.

major comments (4)

[Section 4, Figures 2 and 3] The central claim that 'the addition of heuristics within the exploration strategy leads to lower training time' is not supported by the presented evidence. The results are two qualitative learning curves with no numerical reward values, no error bars, no number of independent random seeds, and no definition of what constitutes convergence or 'training time.' Without these, the reader cannot assess the variance of the results or verify that the observed improvement is statistically meaningful.
[Section 3.2 and Section 4] The paper equates fewer environment interactions with lower training time, but it never measures wall-clock training time or the computational cost of the PSO. Each PSO evaluation uses 50 particles and 100 iterations (Section 3.2), and if invoked on exploration steps, this overhead can easily dominate the savings in environment steps. The claim of 'lower training time' is thus unsubstantiated; the correct metric would be total computation time or a cost model that includes PSO's internal optimization.
[Section 3.1] The PSO fitness function is defined to be identical to the MARL reward function. This means the 'heuristic' is not an independent source of prior knowledge but an online maximizer of the same objective the agent is learning. The observed superiority of PSO-guided exploration over random exploration is therefore unsurprising and confounded: the comparison effectively pits RL exploration against a reward-based planner. To support the claimed value of bio-inspired heuristics, the authors should compare against other action-selection methods (e.g., a random action with equal compute, a separate heuristic not derived from the reward, or a planning baseline) and discuss the distinction between a heuristic and an optimizer.
[Section 3 (overall)] The experimental setup is insufficiently specified for reproducibility. Missing details include MASAC hyperparameters (learning rates, network sizes, batch size), the replay buffer size, the epsilon schedule, the number of training episodes, and the precise integration of PSO (how often it is called, whether it is applied to all agents or a subset, and how its output is combined with the epsilon-greedy probability). These details are necessary for the results to be independently verified.

minor comments (5)

[Section 3.2, reward function] The reward function equation is garbled with non-ASCII characters and is not legible; it should be typeset properly using LaTeX math notation.
[Figures 2 and 3] The figures lack axis labels, units, legends, and a description of what the curves represent (e.g., mean reward per episode over how many runs).
[Section 5, Conclusion] The conclusion says the method 'reduces the training time,' while Section 4 only claims 'these preliminary results support the hypothesis.' The language in the abstract and conclusion should be aligned with the level of evidence actually presented.
[Section 2.2] The related work on Heuristically Accelerated Reinforcement Learning (HARL) is cited but not discussed in enough depth; the novelty of the proposed PSO-as-heuristic approach relative to existing HARL methods should be stated explicitly.
[References] There are several incomplete references (e.g., [39] has no year or venue, [41] is missing page numbers) and inconsistent formatting throughout. The reference list should be cleaned up.

Circularity Check

1 steps flagged · score 4.0 of 10

Partial circularity: the PSO heuristic's fitness is defined as the MARL reward, so the observation that PSO-guided actions yield high rewards is built into the design; the training-time claim itself remains an empirical comparison.

self definitional [Section 3.1 (novel strategy), and Section 4 (Results); fitness-function definition and explanation of speedup.]
"The optimization problem (fitness function) is defined in the same way as the reward function of the MARL in order to guarantee this potential acceleration during learning. This ensures that the actions outputted from the PSO will conduct the agents to high-quality rewards during the learning. ... This can be explained by the fact that the actions resulting from the PSO are on average associated with higher reward values, which would otherwise be experienced only later during the training without heuristics."

The PSO fitness is set equal to the MARL reward function, so an action selected by PSO is, by construction, one that scores highly on the training objective. The Results section then offers as the explanation for faster training that 'actions resulting from the PSO are on average associated with higher reward values' — a property that is the objective of the construction rather than an independent empirical discovery. Thus the part of the argument saying 'the heuristic provides high-reward actions' reduces to the definition of the heuristic. The headline conclusion 'lower training time' is not itself entailed by that identity; it rests on the comparative episode-reward curves, so the circularity is partial.

full rationale

No fitted-parameter circularity, renamed known result, or load-bearing self-citation chain is present. The self-citations in the reference list ([21], [39], [42]) are used as related work or future directions, not to justify the acceleration claim. The only definitional shortcut is that the PSO fitness function is explicitly the same as the MARL reward function, which makes the observation that PSO-guided actions achieve high reward a consequence of the design rather than independent evidence. This weakens the explanatory force of the reported speedup and creates a legitimate fairness concern relative to the random-exploration baseline, especially because PSO is run for 100 iterations with 50 particles per exploration step and its wall-clock cost is never measured. However, the central comparative claim about reaching better mean rewards in fewer episodes is empirical and not logically forced by the reward-fitness identity, so a moderate partial-circularity score of 4 is appropriate.

Assumptions & free parameters 7 free parameters · 4 assumptions · 0 invented entities

The central claim rests on hand-chosen PSO parameters, a hand-designed reward, and the assumption that PSO solutions are good guides. The paper does not provide sensitivity analyses or cost accounting for the heuristic.

free parameters (7)

PSO particles = 50
Choice of swarm size for exploration guidance; not justified by sensitivity analysis.
PSO iterations = 100
Number of swarm updates per exploration step; affects cost and quality of heuristic actions.
PSO time-steps = 50
Planning horizon for the heuristic; the paper does not explain how action sequences over 50 steps are mapped to agent actions.
PSO inertia = 0.8
Set following a modified PSO from reference [37] without ablation.
PSO cognitive and social components = 2.0
Chosen from reference [37] without tuning for this environment.
Reward weights = -5, -5, +1
Hand-chosen weights for collisions, out-of-bounds, and new covered pixels; no sensitivity analysis.
Environment sensor field = 60 degrees, radius 2 units
Chosen to mock a forward-looking sonar/camera; no real sensor model.

assumptions (4)

ad hoc to paper PSO solutions approximate good actions for the coverage task.
Section 3.1 defines the PSO fitness as the reward function, so the heuristic is assumed to find high-reward action sequences, but no evidence of PSO planning quality is provided.
domain assumption The simplified 2D grid environment represents the underwater coverage planning problem.
Section 3.2 describes a custom 2D environment with homogeneous agents and no currents, communication constraints, or sensor noise; the paper assumes conclusions transfer to the described MCM underwater scenario (Section 3.3).
standard math Standard RL convergence assumptions for MASAC.
Policy-gradient updates are assumed to improve with higher-reward samples (Section 4).
ad hoc to paper The number of environment interactions is a valid proxy for training time.
The paper claims reduced training time but only reports episode counts, ignoring the computational cost of running PSO with 50 particles and 100 iterations.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Towards Bio-inspired Heuristically Accelerated Reinforcement Learning for Adaptive Underwater Multi-Agents Behaviour." pith.science (2026). https://pith.science/paper/5BSM74E6

@misc{pith2026250206113,
  author       = {Pith},
  title        = {Pith review of: Towards Bio-inspired Heuristically Accelerated Reinforcement Learning for Adaptive Underwater Multi-Agents Behaviour},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/5BSM74E6}},
  note         = {Machine review of arXiv:2502.06113}
}

read the original abstract

This paper describes the problem of coordination of an autonomous Multi-Agent System which aims to solve the coverage planning problem in a complex environment. The considered applications are the detection and identification of objects of interest while covering an area. These tasks, which are highly relevant for space applications, are also of interest among various domains including the underwater context, which is the focus of this study. In this context, coverage planning is traditionally modelled as a Markov Decision Process where a coordinated MAS, a swarm of heterogeneous autonomous underwater vehicles, is required to survey an area and search for objects. This MDP is associated with several challenges: environment uncertainties, communication constraints, and an ensemble of hazards, including time-varying and unpredictable changes in the underwater environment. MARL algorithms can solve highly non-linear problems using deep neural networks and display great scalability against an increased number of agents. Nevertheless, most of the current results in the underwater domain are limited to simulation due to the high learning time of MARL algorithms. For this reason, a novel strategy is introduced to accelerate this convergence rate by incorporating biologically inspired heuristics to guide the policy during training. The PSO method, which is inspired by the behaviour of a group of animals, is selected as a heuristic. It allows the policy to explore the highest quality regions of the action and state spaces, from the beginning of the training, optimizing the exploration/exploitation trade-off. The resulting agent requires fewer interactions to reach optimal performance. The method is applied to the MSAC algorithm and evaluated for a 2D covering area mission in a continuous control environment.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

48 extracted references · 46 canonical work pages

[1]

It refers to the process of determining a path or set of movements, that ensures a robot or swarm thoroughly covers a given area or surface

Introduction Coverage planning is an important task that is required in various environments. It refers to the process of determining a path or set of movements, that ensures a robot or swarm thoroughly covers a given area or surface. This is a fundamental problem in r obotics, especially for tasks requiring complete or efficient space exploration, such a...

work page
[2]

This environment can be either known or unknown

Related Works 2.1 Multi-Agent Reinforcement Learning In Reinforcement Learning (RL) [4], an environment is modelled as a Markov Decision Process (MDP) where an agent learns to make decisions by experiencing different states, taking actions and receiving rewards. This environment can be either known or unknown. Thus, RL allows an agent to learn through tri...

work page
[3]

Then, the epsilon-greedy method is chosen to improve exploration during training and facilitate the incorporation of the PSO with the first version of the MASAC

Methodology 3.1 A novel strategy to MARL coverage planning The research method in the domain of MARL started with a benchmark of the different suitable methods for coverage path planning, including a comparison between some recent MARL algorithms and more precisely, some recent actor-critic methods Among the recent actor-critic methods for underwater appl...

work page
[4]

One of the chosen metrics for the evaluation is the reward value

Results When the learning begins, a starting point is defined at each episode from which all the agents start one beside the others. One of the chosen metrics for the evaluation is the reward value. The desired result is first, an increasing value along the training, and second, higher values with the PSO compared to values obtained without it. The shown ...

work page
[5]

Conclusion This paper has described the problems related to the coordination and collaboration of an autonomous MAS performing coverage planning in a complex environment, particularly in the underwater context of detecting and identifying objects of interest while surveying an area. Key challenges include managing interactions between agents, ensuring rel...

work page
[6]

Deep reinforcement learning for multiagent systems: A review of challenges, solutions, and applications,

T. T. Nguyen, N. D. Nguyen, and S. Nahavandi , “Deep reinforcement learning for multiagent systems: A review of challenges, solutions, and applications,” IEEE Trans. Cybern., vol. 50, no. 9, pp. 3826–3839, 2020

work page 2020
[7]

Multi-Agent Generative Adversarial Interactive Self-Imitation Learning for AUV Formation Control and Obstacle Avoidance

Z. Fang, T. Chen, D. Jiang, Z. Zhang, and G. Li, “Multi -agent generative adversarial interactive self-imitation learning for AUV formation control and obstacle avoidance,” CoRR, vol. abs/2401.11378, 2024

work page Pith review arXiv 2024
[8]

A multi -agent framework with moos-ivp for autonomous underwater vehicles with sidescan sonar sensors,

L. Paull, S. S. G., M. L. Seto, and H. Li, “A multi -agent framework with moos-ivp for autonomous underwater vehicles with sidescan sonar sensors,” in Autonomous and Intelligent Systems - Second International Conference, AIS 2011, Burnaby, BC, Canada, June 22-24, 2011. Proceedings (M. Kamel, F. Karray, W. Gueaieb, and A. M. Khamis, eds.), vol. 6752 of Lec...

work page 2011

Show all 48 references

[9]

A survey on swarm robotics for area coverage problem,

D. K. Muhsen, A. T. Sadiq, and F. A. Raheem, “A survey on swarm robotics for area coverage problem,” Algorithms, vol. 17, no. 1, 2024

2024
[10]

R. S. Sutton and A. G. Barto, Reinforcement learning an introduction - Second edition. MIT Press, 2018

2018
[11]

Buşoniu, R

L. Buşoniu, R. Babuška, and B. De Schutter, Multi-agent Reinforcement Learning: An Overview, pp. 183–221. Berlin, Heidelberg: Springer Berlin Heidelberg, 2010

2010
[12]

Multi -agent reinforcement learning: A selective overview of theories and algorithms,

K. Zhang, Z. Yang, and T. Başar, “Multi -agent reinforcement learning: A selective overview of theories and algorithms,” 2021

2021
[13]

An empirical investigation of the challenges of real -world reinforcement learning,

G. Dulac-Arnold, N. Levine, D. J. Mankowitz, J. Li, C. Paduraru, S. Gowal, and T. Hester, “An empirical investigation of the challenges of real -world reinforcement learning,” CoRR, vol. abs/2003.11881, 2020

2003 arXiv
[14]

Multi -robot path planning using an improved self -adaptive particle swarm optimization,

B. Tang, K. Xiang, M. Pang, and Z. Zhanxia, “Multi -robot path planning using an improved self -adaptive particle swarm optimization,” International Journal of Advanced Robotic Systems, vol. 17, no. 5, p. 1729881420936154, 2020

2020
[15]

Group decisions in humans and animals: A survey,

L. Conradt and C. List, “Group decisions in humans and animals: A survey,” Philosophical transactions of the Royal Society of London. Series B, Biological sciences, vol. 364, pp. 719– 42, 12 2008

2008
[16]

From animal collective behaviors to swarm robotic cooperation,

H. Duan, M. Huo, and Y. Fan, “From animal collective behaviors to swarm robotic cooperation,” National Science Review, vol. 10, p. nwad040, 02 2023

2023
[17]

Sample efficient multi-agent reinforcement learning with masked reconstruction,

J. I. Kim, Y. J. Lee, J. Heo, J. Park, J. Kim, S. R. Lim, J. Jeong, and S. B. Kim, “Sample efficient multi-agent reinforcement learning with masked reconstruction,” PLOS ONE, vol. 18, pp. 1–14, 09 2023

2023
[18]

Multi-agent reinforcement learning: a critical survey,

Y. Shoham, R. Powers, and T. Grenager, “Multi-agent reinforcement learning: a critical survey,” 2003

2003
[19]

A survey and critique of multiagent deep reinforcement learning,

P. Hernandez-Leal, B. Kartal, and M. E. Taylor, “A survey and critique of multiagent deep reinforcement learning,” Autonomous Agents and Multi-Agent Systems, vol. 33, p. 750–797, Oct. 2019

2019
[20]

Value-based methods [22] estimate the value of each state or state -action pair to derive optimal policies by selecting actions that maximize the cumulative reward

which combine value -based and policy- gradient methods, using an actor to propose actions and a critic to evaluate them [21]. Value-based methods [22] estimate the value of each state or state -action pair to derive optimal policies by selecting actions that maximize the cumu...
[21]

Multi-agent reinforcement learning: A review of challenges and applications,

L. Canese, G. C. Cardarilli, L. Di Nunzio, R. Fazzolari, D. Giardino, M. Re, and S. Spanò, “Multi-agent reinforcement learning: A review of challenges and applications,” Applied Sciences, vol. 11, no. 11, 2021

2021
[22]

Decentralised learning in systems with many, many strategic agents,

D. Mguni, J. Jennings, and E. Munoz de Cote, “Decentralised learning in systems with many, many strategic agents,” Proceedings of the AAAI Conference on Artificial Intelligence, vol. 32, no. 1, 2018

2018
[23]

An efficient centralized multi-agent reinforcement learner for cooperative tasks,

D. Liao, Z. Zhang, T. Song, and M. Liu, “An efficient centralized multi-agent reinforcement learner for cooperative tasks,” IEEE Access, vol. 11, pp. 139284–139294, 2023

2023
[24]

Fully decentralized cooperative multi-agent reinforcement learning: A survey,

J. Jiang, K. Su, and Z. Lu, “Fully decentralized cooperative multi-agent reinforcement learning: A survey,” 2024

2024
[25]

Entropy regularized actor-critic based multi-agent deep reinforcement learning for stochastic games,

D. Hao, D. Zhang, Q. Shi, and K. Li, “Entropy regularized actor-critic based multi-agent deep reinforcement learning for stochastic games,” Information Sciences, vol. 617, pp. 17– 40, 2022

2022
[26]

Soft actor-critic algorithms and applications,

T. Haarnoja, A. Zhou, K. Hartikainen, G. Tucker, S. Ha, J. Tan, V. Kumar, H. Zhu, A. Gupta, P. Abbeel, and S. Levine, “Soft actor-critic algorithms and applications,” CoRR, vol. abs/1812.05905, 2018

2018 arXiv
[27]

Evaluation of a deep reinforcement-learning-based controller for the control of an autonomous underwater vehicle,

Y. Sola, T. Chaffre, G. le Chenadec, K. Sammut, and B. Clement, “Evaluation of a deep reinforcement-learning-based controller for the control of an autonomous underwater vehicle,” in Global Oceans 2020: Singapore – U.S. Gulf Coast, pp. 1–7, 2020

2020
[28]

Leveraging world model disentanglement in value-based multi- agent reinforcement learning,

Z. Wang and D. Meger, “Leveraging world model disentanglement in value-based multi- agent reinforcement learning,” 2023

2023
[29]

A collaborative multiagent reinforcement learning method based on policy gradient potential,

Z. Zhang, Y.-S. Ong, D. Wang, and B. Xue, “A collaborative multiagent reinforcement learning method based on policy gradient potential,” IEEE Transactions on Cybernetics, vol. 51, no. 2, pp. 1015–1027, 2021

2021
[30]

Learning to walk via deep reinforcement learning,

T. Haarnoja, S. Ha, A. Zhou, J. Tan, G. Tucker, and S. Levine, “Learning to walk via deep reinforcement learning,” in Robotics: Science and Systems XV, University of Freiburg, Freiburg im Breisgau, Germany, June 22-26, 2019 (A. Bicchi, H. Kress-Gazit, and S. Hutchinson, eds.), 2019

2019
[31]

Sampling efficient deep reinforcement learning through preference-guided stochastic exploration,

W. Huang, C. Zhang, J. Wu, X. He, J. Zhang, and C. Lv, “Sampling efficient deep reinforcement learning through preference-guided stochastic exploration,” IEEE Transactions on Neural Networks and Learning Systems, pp. 1–12, 2023

2023
[32]

Heuristic-guided reinforcement learning,

C.-A. Cheng, A. Kolobov, and A. Swaminathan, “Heuristic-guided reinforcement learning,” NIPS ’21, (Red Hook, NY, USA), Curran Associates Inc., 2024

2024
[33]

Heuristically accelerated reinforcement learning: Theoretical and experimental results,

R. A. C. Bianchi, C. H. C. Ribeiro, and A. H. R. Costa, “Heuristically accelerated reinforcement learning: Theoretical and experimental results,” in 20th European Conference on Artificial Intelligence (L. D. Raedt, C. Bessiere, D. Dubois, P. Doherty, P. Frasconi, F. Heintz, an...

2012
[34]

Transferring knowledge as heuristics in reinforcement learning: A case-based approach,

R. A. Bianchi, L. A. Celiberto, P. E. Santos, J. P. Matsuura, and R. Lopez de Mantaras, “Transferring knowledge as heuristics in reinforcement learning: A case-based approach,” Artificial Intelligence, vol. 226, pp. 102–121, 2015

2015
[35]

Heuristically Accelerated Reinforcement Learning by Means of Case-Based Reasoning and Transfer Learning,

R. A. C. Bianchi, P. E. Santos, I. J. da Silva, L. A. Celiberto, and R. Lopez de Mantaras, “Heuristically Accelerated Reinforcement Learning by Means of Case-Based Reasoning and Transfer Learning,” Journal of Intelligent & Robotic Systems, vol. 91, pp. 301–312, Aug. 2018

2018
[36]

Ant system: optimization by a colony of cooperating agents,

M. Dorigo, V. Maniezzo, and A. Colorni, “Ant system: optimization by a colony of cooperating agents,” IEEE Trans. Syst. Man Cybern. Part B, vol. 26, no. 1, pp. 29–41, 1996

1996
[37]

The bees algorithm and mechanical design optimisation,

D. T. Pham, M. Castellani, M. Sholedolu, and A. Ghanbarzadeh, “The bees algorithm and mechanical design optimisation,” in ICINCO 2008, Proceedings of the Fifth International Conference on Informatics in Control, Automation and Robotics, Intelligent Control Systems and Optimiza...

2008
[38]

Firefly algorithms for multimodal optimization,

X. Yang, “Firefly algorithms for multimodal optimization,” in Stochastic Algorithms: Foun- dations and Applications, 5th International Symposium, SAGA 2009, Sapporo, Japan, October 26-28, 2009. Proceedings (O. Watanabe and T. Zeugmann, eds.), vol. 5792 of Lecture Notes in Comp...

2009
[39]

Brain storm optimization algorithm,

Y. Shi, “Brain storm optimization algorithm,” in Advances in Swarm Intelligence - Second International Conference, ICSI 2011, Chongqing, China, June 12-15, 2011, Proceedings, Part I (Y. Tan, Y. Shi, Y. Chai, and G. Wang, eds.), vol. 6728 of Lecture Notes in Computer Science, p...

2011
[40]

Group search optimizer: An optimization algorithm inspired by animal searching behavior,

S. He, Q. H. Wu, and J. R. Saunders, “Group search optimizer: An optimization algorithm inspired by animal searching behavior,” IEEE Trans. Evol. Comput., vol. 13, no. 5, pp. 973– 990, 2009

2009
[41]

Solving engineering design problems by social cognitive optimization,

X. Xie and W. Zhang, “Solving engineering design problems by social cognitive optimization,” in Genetic and Evolutionary Computation - GECCO 2004, Genetic and Evolutionary Computation Conference, Seattle, WA, USA, June 26-30, 2004, Proceedings, Part I, vol. 3102 of Lecture Not...

2004
[42]

Particle swarm optimization,

J. Kennedy and R. Eberhart, “Particle swarm optimization,” in Proceedings of International Conference on Neural Networks (ICNN’95), Perth, WA, Australia, November 27 - December 1, 1995, pp. 1942–1948, IEEE, 1995

1995
[43]

Distributed 3-d path planning for multi-uavs with full area surveillance based on particle swarm optimization,

N. Ahmed, C. J. Pawase, and K. Chang, “Distributed 3-d path planning for multi-uavs with full area surveillance based on particle swarm optimization,” Applied Sciences, vol. 11, no. 8, 2021

2021
[44]

Particle swarm optimization algorithm and its applications: A systematic review,

A. Gad, “Particle swarm optimization algorithm and its applications: A systematic review,” Archives of Computational Methods in Engineering, vol. 29, p. 2531–2561, 04 2022

2022
[45]

Sim-to-real transfer of adaptive control parameters for auv stabilisation under current disturbance,

T. Chaffre, J. Wheare, A. Lammas, P. Santos, G. L. Chenadec, K. Sammut, and B. Clement, “Sim-to-real transfer of adaptive control parameters for auv stabilisation under current disturbance,” The International Journal of Robotics Research, vol. 0, no. 0, p. 02783649241272115, 0
[46]

A deeper look at experience replay,

S. Zhang and R. S. Sutton, “A deeper look at experience replay,” 2018

2018
[47]

Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor,

T. Haarnoja, A. Zhou, P. Abbeel, and S. Levine, “Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor,” in Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsmässan, Stockholm, Sweden, July 10...

2018
[48]

Learning adaptive control of a uuv using a bio-inspired experience replay mechanism,

T. Chaffre, P. E. Santos, G. Le Chenadec , E. Chauveau, K. Sammut, and B. Clement, “Learning adaptive control of a uuv using a bio-inspired experience replay mechanism,” IEEE Access, vol. 11, pp. 123505–123518, 2023

2023

Pith tools

Reviewed August 8, 2026 · model on record in the stance chip above.

[1] [1]

It refers to the process of determining a path or set of movements, that ensures a robot or swarm thoroughly covers a given area or surface

Introduction Coverage planning is an important task that is required in various environments. It refers to the process of determining a path or set of movements, that ensures a robot or swarm thoroughly covers a given area or surface. This is a fundamental problem in r obotics, especially for tasks requiring complete or efficient space exploration, such a...

work page

[2] [2]

This environment can be either known or unknown

Related Works 2.1 Multi-Agent Reinforcement Learning In Reinforcement Learning (RL) [4], an environment is modelled as a Markov Decision Process (MDP) where an agent learns to make decisions by experiencing different states, taking actions and receiving rewards. This environment can be either known or unknown. Thus, RL allows an agent to learn through tri...

work page

[3] [3]

Then, the epsilon-greedy method is chosen to improve exploration during training and facilitate the incorporation of the PSO with the first version of the MASAC

Methodology 3.1 A novel strategy to MARL coverage planning The research method in the domain of MARL started with a benchmark of the different suitable methods for coverage path planning, including a comparison between some recent MARL algorithms and more precisely, some recent actor-critic methods Among the recent actor-critic methods for underwater appl...

work page

[4] [4]

One of the chosen metrics for the evaluation is the reward value

Results When the learning begins, a starting point is defined at each episode from which all the agents start one beside the others. One of the chosen metrics for the evaluation is the reward value. The desired result is first, an increasing value along the training, and second, higher values with the PSO compared to values obtained without it. The shown ...

work page

[5] [5]

Conclusion This paper has described the problems related to the coordination and collaboration of an autonomous MAS performing coverage planning in a complex environment, particularly in the underwater context of detecting and identifying objects of interest while surveying an area. Key challenges include managing interactions between agents, ensuring rel...

work page

[6] [6]

Deep reinforcement learning for multiagent systems: A review of challenges, solutions, and applications,

T. T. Nguyen, N. D. Nguyen, and S. Nahavandi , “Deep reinforcement learning for multiagent systems: A review of challenges, solutions, and applications,” IEEE Trans. Cybern., vol. 50, no. 9, pp. 3826–3839, 2020

work page 2020

[7] [7]

Multi-Agent Generative Adversarial Interactive Self-Imitation Learning for AUV Formation Control and Obstacle Avoidance

Z. Fang, T. Chen, D. Jiang, Z. Zhang, and G. Li, “Multi -agent generative adversarial interactive self-imitation learning for AUV formation control and obstacle avoidance,” CoRR, vol. abs/2401.11378, 2024

work page Pith review arXiv 2024

[8] [8]

A multi -agent framework with moos-ivp for autonomous underwater vehicles with sidescan sonar sensors,

L. Paull, S. S. G., M. L. Seto, and H. Li, “A multi -agent framework with moos-ivp for autonomous underwater vehicles with sidescan sonar sensors,” in Autonomous and Intelligent Systems - Second International Conference, AIS 2011, Burnaby, BC, Canada, June 22-24, 2011. Proceedings (M. Kamel, F. Karray, W. Gueaieb, and A. M. Khamis, eds.), vol. 6752 of Lec...

work page 2011