Pith. sign in

REVIEW 4 major objections 4 minor 33 references

Swarm Behavior Cloning

T0 review · 4 major / 4 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Penalizing hidden-feature differences among ensemble policies reduces action divergence and improves imitation returns in eight control environments.

desk verdict A simple, plausible regularizer for ensemble BC; the empirical claim is suggestive but not statistically secured, and the theory doesn't prove what it claims. read the letter →

arxiv 2412.07617 v1 pith:7ORFRSRH submitted 2024-12-10 cs.AI

Jonas Nü{ss}lein , Maximilian Zorn , Philipp Altmann , Claudia Linnhoff-Popien This is my paper · ORCID
classification cs.AI
keywords SwarmBehaviorCloningImitationLearningEnsembleActiondifferenceHiddenfeatureregularizationMeanepisodereturnReinforcement
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Behavior cloning with an ensemble of neural policies suffers from a failure mode the paper calls increasing action differences: in states underrepresented in the expert data, the policies disagree and their averaged action can land in low-value regions. Swarm Behavior Cloning changes the training loss so that policies are not only trained to match expert actions but also penalized for having dissimilar hidden-feature activations, encouraging the ensemble to behave like a coherent swarm. Across eight benchmark control environments the method reduces mean action difference and improves mean episode return relative to both single-policy BC and standard Ensemble BC, with the largest gains in high-dimensional environments such as HalfCheetah. The paper also argues that this regularization concentrates the ensemble on the mode of the hidden-activation distribution, giving a theoretical justification for why forcing agreement helps.

What carries the argument

The central object is the coupled ensemble loss (equation 2): the standard per-policy behavior-cloning term plus a pairwise penalty $\tau \sum_{i<j} \|h_i^k(s) - h_j^k(s)\|^2$ at each hidden layer $k$, for a user-chosen coefficient $\tau$. This term is what turns independently trained policies into a swarm: it pushes the internal feature representations toward one another while leaving each policy's output head free, preserving some diversity in computation. The theoretical complement is a density-concentration argument: the distribution of $N$ independently sampled hidden activations $h_k$ is proportional to $p(h_k)^N$, whose mass shifts to the global mode as $N$ grows, and the paper claims that optimizing loss (2) realizes this concentration.

What would settle it

Estimate $p(h_k)$ for a fixed state $s$ by training many independent BC policies on the same dataset and histogramming one-layer hidden activations; train a Swarm BC ensemble with that same setting and compare its converged hidden activations to the estimated global mode. If the Swarm activations do not track the mode as $\tau$, $N$, and the number of estimation samples grow, the paper's theoretical equivalence between loss (2) and mode-seeking is false. A simpler empirical falsifier: on any of the eight environments, set $\tau$ so large that the BC term is negligible; the proposition predicts the ensemble still finds the mode of $p(h_k)^N$, whereas in practice the policies would just collapse to identical weights, which can be checked directly.

Watch

Extended reading notes

Core claim

On the paper's own terms, Swarm Behavior Cloning establishes that a simple additive regularizer on hidden-layer activations of an ensemble of behavior-cloned policies reduces the mean action difference between members and improves the quality of the averaged action. The combined loss is $L(s,a) = \sum_i (\pi_i(s)-a)^2 + \tau \sum_k \sum_{i<j} \|h_i^k(s)-h_j^k(s)\|^2$, where the second term couples all pairs of policies at every hidden layer. With $\tau = 0.25$ and $N = 4$ policies, the method matches or beats both BC and Ensemble BC in all eight tested environments; for HalfCheetah with 8 expert episodes the scaled return is 0.72 versus 0.17 for Ensemble BC. The paper additionally claims that training such a coupled ensemble approximates sampling from $p_N(h_k) \propto p(h_k)^N$, so that as $N$ grows the shared features concentrate on the global mode of the single-policy activation density.

Load-bearing premise

The theoretical justification rests on a claim that the paper asserts but does not prove: that minimizing the coupled loss is the same as drawing hidden-feature vectors from a distribution that concentrates on the single most probable feature vector.

Editorial extensions

If this is right

  • Swarm BC reduces the mean action difference between ensemble members in every environment tested, by up to about 44% in BipedalWalker.
  • Its mean episode return is nearly never worse than BC or Ensemble BC, with the largest gains in higher-dimensional environments such as HalfCheetah.
  • The method is fully offline, requiring no environment rollouts beyond the expert demonstrations.
  • The additional hyperparameter $\tau$ controls the trade-off; $\tau = 0.25$ worked best on Walker2D, and too large a value degrades performance.
  • Enlarging the ensemble beyond $N = 4$ yields little extra return but linearly increases training time.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the mode-concentration argument is right, the same hidden-feature coupling should also suppress action variance in out-of-distribution states that never appear in the training data, a point the experiments do not directly test.
  • Because the regularizer only couples features, it could in principle be combined with diversity-promoting terms to prevent all policies from collapsing to identical networks, though the paper does not explore that.
  • A direct test of the theory would be to estimate $p(h_k)$ empirically from a large pool of independent BC policies and check whether Swarm BC's converged activations coincide with its mode.
  • The feature-coupling idea is architecture-agnostic and could be applied to other offline imitation variants such as implicit behavior cloning, which the paper does not consider.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper proposes Swarm Behavior Cloning (Swarm BC), a variant of ensemble behavior cloning in which N policies are trained jointly with an additional regularizer that penalizes pairwise differences between their hidden feature activations. The objective is Eq. (2): the standard per-policy MSE plus tau times the sum over layers and policy pairs of squared hidden-feature differences. The ensemble action is the average of the N predicted actions. The authors evaluate the method on eight OpenAI Gym environments, comparing scaled episode returns and mean action differences against BC and Ensemble BC, and include ablations over tau and N. Section 7 presents a theoretical analysis intended to show that the regularized ensemble concentrates on the mode of the hidden-activation distribution.

Significance. If the empirical claim holds, Swarm BC is a simple, fully offline regularizer for ensemble imitation learning that requires no additional environment interaction, and the paper would provide a useful baseline for a known failure mode of ensemble BC. The evaluation is not circular: it uses external benchmark environments and independent Stable-Baselines3 expert policies, and the paper explicitly reports ablations for both introduced hyperparameters. However, the load-bearing empirical evidence is currently weak (five seeds, no significance tests or confidence intervals), and the theoretical analysis in Section 7 does not bridge the actual loss in Eq. (2) to the mode-concentration argument it proves. The central idea is plausible and worth further scrutiny, but the manuscript in its present form does not convincingly establish the claimed improvements.

major comments (4)
  1. [Section 6, Figure 3] The central empirical claim that Swarm BC significantly improves episode returns is not statistically supported. The paper reports means over 5 seeds and 20 evaluation episodes, but never reports per-seed values, confidence intervals, effect sizes, or a significance test; the word 'significantly' appears repeatedly without a test. With n=5, the HalfCheetah contrast (0.72 vs 0.17 scaled return at 8 episodes) is not interpretable without knowing seed-level variance, and the shaded 'standard deviation' bands in Figure 3 are not a substitute for a paired comparison. Please provide per-seed results and appropriate significance tests or bootstrap confidence intervals, or soften the conclusions.
  2. [Section 6, Figure 5] Hyperparameter selection undermines the generality claim. The values tau=0.25 and N=4 are chosen from the Walker2D ablation (Figure 5) and then fixed for all eight environments, yet the method's behavior depends directly on the strength tau of the regularizer. The ablation itself also uses 'significantly' without a statistical test. Please add a sensitivity analysis on more than one environment, or demonstrate that the reported ranking is stable across reasonable tau values.
  3. [Section 7] The theoretical section does not prove what it claims. The text states that 'training an ensemble with similar feature activations corresponds to finding the global mode' of p(h_k), but Section 7 only proves concentration of the product density p(h_k)^N; it never derives this product form from the actual training objective in Eq. (2), nor does it show that SGD on Eq. (2) samples from p_N(h_k). Moreover, the Proposition is stated in the limit tau -> 0 and N -> infinity, whereas the experiments use finite tau=0.25 and N=4. Please supply a derivation connecting Eq. (2) to the mode-seeking argument, or explicitly label Section 7 as intuition and remove the claim that it is a proof.
  4. [Section 6, Figure 4] The reduction in mean action difference is quantified for only four of the eight environments (Figure 4), and the only numerical reductions given (44% for BipedalWalker, 11% for Ant) are not accompanied by errors or tests. The paper's second hypothesis is therefore only partially verified. Please report aggregate reductions and variability across all environments, or restrict the claim accordingly.
minor comments (4)
  1. [Section 7, final display] The final expression for N uses p(phi+)/p(phi#) where the densities at stake are p(h+_k)/p(h#_k); the notation should be consistent, and 'deviation' should be 'derivative'.
  2. [Section 6, experimental protocol] The text says datasets D contain x in [1,8] episodes, but the x-axes in Figure 3 begin at different values for different environments (e.g., HalfCheetah at 3, CartPole at 1); please clarify the exact protocol used for each environment.
  3. [Section 3, Definition 1] The definition is numbered 'Definition 1' but referred to later as 'Definition 3.1'; use consistent numbering.
  4. [Algorithm 1] The Input line should use set notation D = {(s, a = pi_e(s))} rather than D = (s, a = pi_e(s)), and the phrase 'in Chapter 5' should be 'in Section 5'.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: empirical benchmark is external and the theory gap is a missing derivation, not a reuse of inputs.

full rationale

The paper's central claims are empirical: Swarm BC reduces mean action difference and improves mean episode return relative to BC and Ensemble BC. These claims are evaluated on eight external OpenAI Gym environments using expert policies from Stable-Baselines 3, an independent library, so the performance comparison is not derived from the method's own fitted values. The proposed loss (2) is an explicit new objective; Definition 1 defines mean action difference, and the reduction of that quantity follows directly from penalizing pairwise hidden-feature distances, while the return comparisons use external baselines. No fitted parameter is renamed as a prediction, and no self-citation carries the argument: the only self-citations (Nusslein et al. 2022; Phan et al. 2023) appear in background material and are not load-bearing. The theoretical analysis in Section 7 is disconnected rather than circular: it proves concentration of the product density p(h)^N, but it never derives loss (2) from that density, so the claimed bridge from the loss to the global mode is an unproved assertion, not an equation that reduces to its own input. The ablation-based choice of tau and N is an experimental limitation, not circularity. Thus no identified step exhibits self-definition, fitted-input prediction, or citation-dependent forcing.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The empirical claim rests on two hand-chosen hyperparameters (tau and N), a domain assumption about expert quality, and a structural assumption that MLP activations are comparable across policies. The theoretical section adds three mathematical assumptions and one unproved bridge between the regularized loss and the product density p_N; that bridge is the most fragile part of the theory.

free parameters (2)
  • tau (tau) = 0.25
    Regularization coefficient for hidden-activation alignment. Chosen via ablation on Walker2D (Figure 5) and fixed for all other environments; central results depend on this choice.
  • N = 4
    Number of policies in the ensemble. Chosen via ablation on Walker2D (Figure 5) as a tradeoff between performance and training time; all experiments use N=4.
assumptions (4)
  • domain assumption Expert policies can be obtained from Stable-Baselines 3 SAC (continuous) and PPO (discrete) and provide optimal demonstrations for the dataset D.
    Assumed in Section 6; if the experts are suboptimal, the relative comparison to BC baselines may not transfer to real expert demos.
  • domain assumption All ensemble policies are standard multilayer perceptrons with the same hidden layer sizes, so hidden activations h_i^k(s) can be compared with an L2 penalty.
    Section 5 states this assumption. The loss (2) requires the layers of different policies to be architecturally aligned.
  • domain assumption The hidden activation density p(h_k) is continuously differentiable, has a single mode, and lives on a bounded hypercube.
    Stated in the Proposition in Section 7. These conditions are used in the concentration proof and are not justified for neural network activations.
  • ad hoc to paper Optimizing the Swarm BC loss (2) corresponds to sampling hidden activations from p_N(h_k) proportional to p(h_k)^N.
    Section 7 asserts this correspondence but does not derive it from the loss. The subsequent proof is about concentration of measure and never references loss (2).

how reviews work

0 comments
Cite this review

Pith. "Pith review of Swarm Behavior Cloning." pith.science (2026). https://pith.science/paper/7ORFRSRH

@misc{pith2026241207617,
  author       = {Pith},
  title        = {Pith review of: Swarm Behavior Cloning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/7ORFRSRH}},
  note         = {Machine review of arXiv:2412.07617}
}
abstract

In sequential decision-making environments, the primary approaches for training agents are Reinforcement Learning (RL) and Imitation Learning (IL). Unlike RL, which relies on modeling a reward function, IL leverages expert demonstrations, where an expert policy $\pi_e$ (e.g., a human) provides the desired behavior. Formally, a dataset $D$ of state-action pairs is provided: $D = {(s, a = \pi_e(s))}$. A common technique within IL is Behavior Cloning (BC), where a policy $\pi(s) = a$ is learned through supervised learning on $D$. Further improvements can be achieved by using an ensemble of $N$ individually trained BC policies, denoted as $E = {\pi_i(s)}{1 \leq i \leq N}$. The ensemble's action $a$ for a given state $s$ is the aggregated output of the $N$ actions: $a = \frac{1}{N} \sum{i} \pi_i(s)$. This paper addresses the issue of increasing action differences -- the observation that discrepancies between the $N$ predicted actions grow in states that are underrepresented in the training data. Large action differences can result in suboptimal aggregated actions. To address this, we propose a method that fosters greater alignment among the policies while preserving the diversity of their computations. This approach reduces action differences and ensures that the ensemble retains its inherent strengths, such as robustness and varied decision-making. We evaluate our approach across eight diverse environments, demonstrating a notable decrease in action differences and significant improvements in overall performance, as measured by mean episode returns.

Figures

Figures reproduced from arXiv: 2412.07617 by the authors.

Figure 1
Figure 1. This figure visualizes schematically the predicted actions of three different Behavior Cloning approaches, repre [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. This figure visualizes exemplarily the mean action difference for an entire episode of an ensemble containing N = 6 policies. We used the LunarLander-continuous environment since it has a 2-dim action space that can be easily visualized. The x-axis in the left plot represents the timestep in the episode. For two interesting timesteps, we have visualized the predicted actions of the N policies {a i t = πi(st)} (gray … view at source ↗
Figure 3
Figure 3. These plots show the mean normalized test returns of our approach [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: In this figure we are evaluating whether [PITH_FULL_IMAGE:figures/full_fig_p008_4.png]
Figure 5
Figure 5. Figure 5: To examine the sensitivity of the two hyperpa [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

33 extracted references · 22 canonical work pages

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION fin.entry add.period write newline FUNCTION new.block output.state before.all = 'skip after.block 'output.state := if FUNCTION new.sentence output.state after.block = 'skip output.state before.all = 'skip after.sentence 'output.state := if if FUNCTION not #0 #1 if FUNCTION and 'skip pop #0 if FUNCTIO...

  2. [2]

    and Doshi, P

    Arora, S. and Doshi, P. (2021). A survey of inverse reinforcement learning: Challenges, methods and progress. Artificial Intelligence , 297:103500

    work page 2021

  3. [3]

    and Sammut, C

    Bain, M. and Sammut, C. (1995). A framework for behavioural cloning. In Machine Intelligence 15 , pages 103--129

    work page 1995

  4. [4]

    D., Monfort, M., Muller, U., Zhang, J., et al

    Bojarski, M., Del Testa, D., Dworakowski, D., Firner, B., Flepp, B., Goyal, P., Jackel, L. D., Monfort, M., Muller, U., Zhang, J., et al. (2016). End to end learning for self-driving cars. arXiv preprint arXiv:1604.07316

    arXiv 2016

  5. [5]

    Brantley, K., Sun, W., and Henaff, M. (2019). Disagreement-regularized imitation learning. In International Conference on Learning Representations

    work page 2019

  6. [6]

    Brockman, G., Cheung, V., Pettersson, L., Schneider, J., Schulman, J., Tang, J., and Zaremba, W. (2016). Openai gym. arXiv preprint arXiv:1606.01540

    arXiv 2016

  7. [7]

    M., and Gaidon, A

    Codevilla, F., Santana, E., L \'o pez, A. M., and Gaidon, A. (2019). Exploring the limitations of behavior cloning for autonomous driving. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 9329--9338

    work page 2019

  8. [8]

    Dietterich, T. G. et al. (2002). Ensemble learning. The handbook of brain theory and neural networks , 2(1):110--125

    work page 2002

Show all 33 references
  1. [9]

    Dong, X., Yu, Z., Cao, W., Shi, Y., and Ma, Q. (2020). A survey on ensemble learning. Frontiers of Computer Science , 14:241--258

    2020

  2. [10]

    Eschmann, J. (2021). Reward function design in reinforcement learning. Reinforcement Learning Algorithms: Analysis and Applications , pages 25--33

    2021

  3. [11]

    Finn, C., Levine, S., and Abbeel, P. (2016). Guided cost learning: Deep inverse optimal control via policy optimization. In International conference on machine learning , pages 49--58. PMLR

    2016

  4. [12]

    A., Wahid, A., Downs, L., Wong, A., Lee, J., Mordatch, I., and Tompson, J

    Florence, P., Lynch, C., Zeng, A., Ramirez, O. A., Wahid, A., Downs, L., Wong, A., Lee, J., Mordatch, I., and Tompson, J. (2022). Implicit behavioral cloning. In Conference on Robot Learning , pages 158--168. PMLR

    2022

  5. [13]

    C., He, F.-L., Rodr \' guez, J

    Giusti, A., Guzzi, J., Cire s an, D. C., He, F.-L., Rodr \' guez, J. P., Fontana, F., Faessler, M., Forster, C., Schmidhuber, J., Di Caro, G., et al. (2015). A machine learning approach to visual perception of forest trails for mobile robots. IEEE Robotics and Automation Lette...

    2015

  6. [14]

    and Ermon, S

    Ho, J. and Ermon, S. (2016). Generative adversarial imitation learning. Advances in neural information processing systems , 29

    2016

  7. [15]

    Hussein, M., Crowe, B., Petrik, M., and Begum, M. (2021). Robust maximum entropy behavior cloning. arXiv preprint arXiv:2101.01251

    2021 arXiv

  8. [16]

    B., Allievi, A., Banzhaf, H., Schmitt, F., and Stone, P

    Knox, W. B., Allievi, A., Banzhaf, H., Schmitt, F., and Stone, P. (2023). Reward (mis) design for autonomous driving. Artificial Intelligence , 316:103829

    2023

  9. [17]

    Y., Russell, S., et al

    Ng, A. Y., Russell, S., et al. (2000). Algorithms for inverse reinforcement learning. In Icml , volume 1, page 2

    2000

  10. [18]

    u lein, J., Illium, S., M \

    N \"u lein, J., Illium, S., M \"u ller, R., Gabor, T., and Linnhoff-Popien, C. (2022). Case-based inverse reinforcement learning using temporal coherence. In International Conference on Case-Based Reasoning , pages 304--317. Springer

    2022

  11. [19]

    u lein, J., K \

    Phan, T., Ritz, F., Altmann, P., Zorn, M., N \"u lein, J., K \"o lle, M., Gabor, T., and Linnhoff-Popien, C. (2023). Attention-based recurrence for multi-agent reinforcement learning under stochastic partial observability. In International Conference on Machine Learning , page...

    2023

  12. [20]

    Raffin, A., Hill, A., Gleave, A., Kanervisto, A., Ernestus, M., and Dormann, N. (2021). Stable-baselines3: Reliable reinforcement learning implementations. Journal of Machine Learning Research , 22(268):1--8

    2021

  13. [21]

    and Rokach, L

    Sagi, O. and Rokach, L. (2018). Ensemble learning: A survey. Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery , 8(4):e1249

    2018

  14. [22]

    M., Cui, Z., Altanzaya, A

    Shafiullah, N. M., Cui, Z., Altanzaya, A. A., and Pinto, L. (2022). Behavior transformers: Cloning k modes with one stone. Advances in neural information processing systems , 35:22955--22968

    2022

  15. [23]

    Smith, M., Maystre, L., Dai, Z., and Ciosek, K. (2023). A strong baseline for batch imitation learning. arXiv preprint arXiv:2302.02788

    2023 arXiv

  16. [24]

    Sutton, R. S. and Barto, A. G. (2018). Reinforcement learning: An introduction . MIT press

    2018

  17. [25]

    Torabi, F., Warnell, G., and Stone, P. (2018). Behavioral cloning from observation. arXiv preprint arXiv:1805.01954

    2018 arXiv

  18. [26]

    Torabi, F., Warnell, G., and Stone, P. (2019a). Adversarial imitation learning from state-only demonstrations. In Proceedings of the 18th International Conference on Autonomous Agents and MultiAgent Systems , pages 2229--2231

    2019

  19. [27]

    Torabi, F., Warnell, G., and Stone, P. (2019b). Recent advances in imitation learning from observation. arXiv preprint arXiv:1905.13566

    2019 arXiv

  20. [28]

    Webb, G. I. and Zheng, Z. (2004). Multistrategy ensemble learning: Reducing error by combining ensemble learning techniques. IEEE Transactions on Knowledge and Data Engineering , 16(8):980--991

    2004

  21. [29]

    Wen, C., Lin, J., Darrell, T., Jayaraman, D., and Gao, Y. (2020). Fighting copycat agents in behavioral cloning from observation histories. Advances in Neural Information Processing Systems , 33:2564--2575

    2020

  22. [30]

    Yang, Z., Ren, K., Luo, X., Liu, M., Liu, W., Bian, J., Zhang, W., and Li, D. (2022). Towards applicable reinforcement learning: Improving the generalization and sample efficiency with policy ensemble. arXiv preprint arXiv:2205.09284

    2022 arXiv

  23. [31]

    Zheng, B., Verma, S., Zhou, J., Tsang, I., and Chen, F. (2021). Imitation learning: Progress, taxonomies and challenges. arXiv preprint arXiv:2106.12177

    2021 arXiv

  24. [32]

    and Meng Joo, E

    Zhifei, S. and Meng Joo, E. (2012). A survey of inverse reinforcement learning techniques. International Journal of Intelligent Computing and Cybernetics , 5(3):293--311

    2012

  25. [33]

    and Zhou, Z.-H

    Zhou, Z.-H. and Zhou, Z.-H. (2021). Ensemble learning . Springer

    2021

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.