pith. sign in

arxiv: 2606.17624 · v1 · pith:URPHQOZInew · submitted 2026-06-16 · ❄️ cond-mat.stat-mech

Work Extraction via Backward Motion in Optimal Closed-Loop Stochastic Control

Luis Frieder Reinalter , Emanuele Panizon , Lokesh Muruga , Clemens Bechinger This is my paper

Pith reviewed 2026-06-26 22:57 UTC · model grok-4.3

classification ❄️ cond-mat.stat-mech
keywords stochastic thermodynamicsfeedback controlwork extractioncolloidal particlereinforcement learningoptical tweezersthermal fluctuationsclosed-loop control
0
0 comments X

The pith

Optimal feedback policies in colloidal systems extract work by moving the trap backward to exploit thermal fluctuations.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper shows how real-time reinforcement learning can discover optimal closed-loop control protocols for displacing an optical trap in an overdamped colloidal particle. As protocol duration increases, the learned strategy crosses over from simple deterministic dragging to policies that deliberately move the trap against the particle's instantaneous position, harvesting energy from thermal kicks. These policies match the exact analytic optimum and lower the net work cost until it becomes negative. The method also succeeds in heterogeneous force landscapes where analytic solutions are unavailable.

Core claim

In finite-time stochastic control of a colloidal particle, the optimal closed-loop policy identified by reinforcement learning transitions from deterministic dragging to feedback-assisted backward motion that exploits thermal fluctuations, reducing the required work and eventually enabling net work extraction while agreeing quantitatively with the exact optimal solution.

What carries the argument

The optimal closed-loop policy, found via in situ reinforcement learning on the colloidal particle, that switches from forward dragging to backward trap motion to exploit fluctuations.

If this is right

  • For short protocol times the optimal strategy remains deterministic dragging with no backward steps.
  • For sufficiently long protocol times the optimal policy produces net work extraction by using position-dependent backward moves.
  • The same RL procedure yields near-optimal policies in spatially varying external force fields where closed-form solutions are intractable.
  • The quantitative match between RL policies and the exact optimum validates that the learned crossover is not an artifact of the learning algorithm.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same backward-motion principle may appear in biological molecular motors that operate far from equilibrium.
  • Extending the protocol to multiple particles could reveal collective strategies for fluctuation harvesting in interacting systems.
  • The approach offers a practical route to design stochastic heat engines or information engines when analytic optima are unknown.

Load-bearing premise

Real-time reinforcement learning converges reliably to the globally optimal feedback policy despite experimental noise, limited sampling, and the chosen reward function.

What would settle it

A direct experimental measurement showing that the average work for the RL-derived protocol exceeds the analytic minimum work for the same protocol duration, or violates the known bounds for closed-loop work extraction.

Figures

Figures reproduced from arXiv: 2606.17624 by Clemens Bechinger, Emanuele Panizon, Lokesh Muruga, Luis Frieder Reinalter.

Figure 1
Figure 1. Figure 1: Average work ⟨W⟩ versus the protocol duration tf . Symbols are experimental values obtained from RL-identified optimal policies, the solid and dashed lines denotes the ana￾lytical solution for the optimal closed-loop protocol and that without feedback. Insets: representative trap trajectories λt and corresponding particle trajectories xt for tf = 0.2 s and 3 s. For tf = 0.2 s, the particle systematically l… view at source ↗
Figure 2
Figure 2. Figure 2: (a)–(c) Representative action distributions for [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: (a) RL-optimized trap trajectory for tf = 1 s in the presence of an externally imposed, time-dependent perturbation. The blue curve shows a single realization of the optimal feedback protocol identified by RL, while the gray band indicates the scatter between 100 individual experiments. The trap trajectory first approaches the perturbed region, remains there for an extended part of the protocol, and then r… view at source ↗
Figure 4
Figure 4. Figure 4: Schematic of the experimental optical-tweezer [PITH_FULL_IMAGE:figures/full_fig_p006_4.png] view at source ↗
read the original abstract

We experimentally realize finite-time feedback control in an overdamped colloidal system using real-time optical tweezers with in situ reinforcement learning (RL). By varying the protocol duration tf for displacing the optical trap between prescribed positions, the optimal strategies identified by RL reveal a crossover from deterministic dragging toward the target to feedback-assisted exploitation of thermal fluctuations, reducing and eventually overcoming the energetic cost. The resulting policies agree quantitatively with the exact optimal closed-loop solution. By extending the approach to spatially localized external forcing, we further show that RL can identify optimal feedback strategies in heterogeneous stochastic environments where direct analytical control design is challenging.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

1 major / 1 minor

Summary. The paper experimentally realizes finite-time feedback control of an overdamped colloidal particle using real-time optical tweezers and in-situ reinforcement learning (RL). Varying the protocol duration tf, the RL-derived policies exhibit a crossover from deterministic dragging to feedback-assisted exploitation of thermal fluctuations, eventually enabling net work extraction; these policies are reported to agree quantitatively with an independently derived exact optimal closed-loop solution. The approach is further applied to a heterogeneous forcing landscape where analytic design is difficult.

Significance. If the RL policies are verifiably globally optimal and the quantitative match to the exact solution holds, the work supplies a direct experimental test of optimal closed-loop stochastic control in the presence of thermal noise and demonstrates that RL can discover non-intuitive strategies (including backward motion) that outperform naive protocols. The extension to spatially heterogeneous environments also illustrates a practical route for control design when closed-form solutions are unavailable.

major comments (1)
  1. [Abstract and Results] Abstract and Results: The central claim of quantitative agreement between RL policies and the exact optimal closed-loop solution, together with the reported deterministic-to-fluctuation-exploitation crossover, requires that the in-situ RL procedure has converged to the globally optimal policy. No diagnostics (multiple independent runs with different random seeds, policy variance across trainings, or explicit convergence metrics under the experimental noise and finite sampling) are described that would rule out a good but suboptimal local policy whose performance happens to lie within the reported error bars for the tested tf values.
minor comments (1)
  1. [Abstract] The abstract states that the policies 'agree quantitatively' with the exact solution; a table or figure explicitly comparing the RL-extracted work, position trajectories, and control forces against the analytic expressions for several tf would strengthen the claim.

Simulated Author's Rebuttal

1 responses · 0 unresolved

We thank the referee for their thorough review and for highlighting the importance of verifying global optimality in the RL procedure. We address the single major comment below and will strengthen the manuscript accordingly.

read point-by-point responses

  1. Referee: [Abstract and Results] Abstract and Results: The central claim of quantitative agreement between RL policies and the exact optimal closed-loop solution, together with the reported deterministic-to-fluctuation-exploitation crossover, requires that the in-situ RL procedure has converged to the globally optimal policy. No diagnostics (multiple independent runs with different random seeds, policy variance across trainings, or explicit convergence metrics under the experimental noise and finite sampling) are described that would rule out a good but suboptimal local policy whose performance happens to lie within the reported error bars for the tested tf values.

    Authors: We agree that additional convergence diagnostics would make the claim of global optimality more robust. The quantitative match to the independently derived exact solution across a range of tf already provides strong supporting evidence, as a merely local policy would be unlikely to reproduce the exact optimal performance (including the backward-motion regime) within experimental error bars. Nevertheless, to directly address the concern we will revise the manuscript to include: (i) results from five independent RL trainings with distinct random seeds, (ii) the observed variance in extracted work and policy parameters, and (iii) training curves of the cumulative reward under the experimental noise level. These additions will be placed in a new supplementary section and referenced in the main text. revision: yes

Circularity Check

0 steps flagged

No circularity: experimental validation against independently derived exact solution

full rationale

The paper's core claim is an experimental realization of feedback control via in-situ RL in a colloidal system, with policies shown to agree quantitatively with a separately derived exact optimal closed-loop solution. No load-bearing step reduces a prediction or result to a fitted parameter, self-citation chain, or definitional equivalence. The exact solution is treated as an external benchmark, and RL performance is presented as empirical agreement rather than a constructed identity. This matches the default expectation of a self-contained experimental demonstration.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 0 invented entities

Based solely on the abstract; the central claim rests on the validity of the overdamped colloidal model and the assumption that RL finds the global optimum under the chosen experimental conditions.

axioms (1)
  • domain assumption The colloidal particle dynamics are overdamped.
    Explicitly stated in the abstract as the physical regime under study.

pith-pipeline@v0.9.1-grok · 5633 in / 1172 out tokens · 28475 ms · 2026-06-26T22:57:47.395341+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

32 extracted references · 5 canonical work pages · 2 internal anchors

  1. [1]

    Schmiedl and U

    T. Schmiedl and U. Seifert, Phys. Rev. Lett.98, 108301 (2007)

    2007

  2. [2]

    Aurell, C

    E. Aurell, C. Mej´ ıa-Monasterio, and P. Muratore- Ginanneschi, Phys. Rev. Lett.106, 250601 (2011)

    2011

  3. [3]

    T. R. Gingrich, G. M. Rotskoff, G. E. Crooks, and P. L. Geissler, Proc. Natl. Acad. Sci. U.S.A.113, 10263 (2016)

    2016

  4. [4]

    M. C. Engel, J. A. Smith, and M. P. Brenner, Phys. Rev. X13, 041032 (2023)

    2023

  5. [5]

    Alvarado, E

    J. Alvarado, E. G. Teich, D. A. Sivak, and J. Bechhoe- fer, Annual Review of Condensed Matter Physics17, 327 (2026)

    2026

  6. [6]

    Bechhoefer,Control Theory for Physicists, 1st ed

    J. Bechhoefer,Control Theory for Physicists, 1st ed. (Cambridge University Press, 2021)

    2021

  7. [7]

    L. K. Davis, Optimal multi-parameter control of trapped active matter (2026), arXiv:2603.16778 [cond-mat.soft]

    work page arXiv 2026

  8. [8]

    Seifert, Rep

    U. Seifert, Rep. Prog. Phys.75, 126001 (2012)

    2012

  9. [9]

    Sagawa and M

    T. Sagawa and M. Ueda, Phys. Rev. Lett.104, 090602 (2010)

    2010

  10. [10]

    Toyabe, T

    S. Toyabe, T. Sagawa, M. Ueda, E. Muneyuki, and M. Sano, Nature Phys6, 988 (2010)

    2010

  11. [11]

    T. K. Saha, J. N. Lucero, J. Ehrich, D. A. Sivak, and J. Bechhoefer, Phys. Rev. Lett.129, 130601 (2022)

    2022

  12. [12]

    T. K. Saha, J. Ehrich, M. Gavrilov, S. Still, D. A. Sivak, and J. Bechhoefer, Phys. Rev. Lett.131, 057101 (2023)

    2023

  13. [13]

    S. A. Loos, S. Monter, F. Ginot, and C. Bechinger, Phys. Rev. X14, 021032 (2024)

    2024

  14. [14]

    L. K. Davis, K. Proesmans, and E. Fodor, Phys. Rev. X 14, 011012 (2024)

    2024

  15. [15]

    Oikawa, Y

    S. Oikawa, Y. Nakayama, S. Ito, T. Sagawa, and S. Toy- abe, Nat Commun16, 10424 (2025)

    2025

  16. [16]

    Whitelam, Phys

    S. Whitelam, Phys. Rev. X13, 021005 (2023)

    2023

  17. [17]

    Abreu and U

    D. Abreu and U. Seifert, EPL94, 10001 (2011)

    2011

  18. [18]

    J. M. Horowitz and J. M. R. Parrondo, New J. Phys.13, 123019 (2011)

    2011

  19. [19]

    Admon, S

    T. Admon, S. Rahav, and Y. Roichman, Phys. Rev. Lett. 121, 180601 (2018)

    2018

  20. [20]

    R. S. Sutton, A. G. Barto,et al.,Reinforcement learning: An introduction, Vol. 1 (1998)

    1998

  21. [21]

    Xu, Phys

    R. Xu, Phys. Rev. E105, 054123 (2022)

    2022

  22. [22]

    Nasiri and B

    M. Nasiri and B. Liebchen, New J. Phys.24, 073042 (2022)

    2022

  23. [23]

    Rengifo and G

    D. Rengifo and G. T´ ellez, A machine learning approach to fast thermal equilibration (2025), arXiv:2504.08080 [cond-mat]

    work page arXiv 2025

  24. [24]

    See Supplemental Material at [url] for details on exper- imental feedback implementation, calibration and the reinforcement-learning framework

  25. [25]

    Sekimoto, Prog

    K. Sekimoto, Prog. Theor. Phys. Suppl.130, 17 (1998)

    1998

  26. [26]

    Speck and U

    T. Speck and U. Seifert, Phys. Rev. E70, 066112 (2004)

    2004

  27. [27]

    V. Mnih, K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare, A. Graves, M. Riedmiller, A. K. Fidjeland, G. Ostrovski, S. Petersen, C. Beattie, A. Sadik, I. Antonoglou, H. King, D. Kumaran, D. Wier- stra, S. Legg, and D. Hassabis, Nature518, 529 (2015)

    2015

  28. [28]

    Proximal Policy Optimization Algorithms

    J. Schulman, F. Wolski, P. Dhariwal, A. Radford, and O. Klimov, arXiv preprint arXiv:1707.06347 (2017)

    work page internal anchor Pith review Pith/arXiv arXiv 2017

  29. [29]

    Optimal Control of a Mesoscopic Information Engine

    E. Panizon, arXiv preprint arXiv:2603.29804 (2026)

    work page internal anchor Pith review Pith/arXiv arXiv 2026

  30. [30]

    L. F. Reinalter, E. Panizon, L. Muruga, and C. Bechinger, Supporting data for ’Work Extraction via Backward Motion in Optimal Closed-Loop Stochastic Control’ available at 10.5281/zenodo.20597766 (2026)

    work page doi:10.5281/zenodo.20597766 2026

  31. [31]

    P. H. Jones, O. M. Marag` o, and G. Volpe,Optical tweez- ers: principles and applications(Cambridge university press, Cambridge, 2015). END MA TTER Experimental apparatus—The optical trap was gener- ated by a laser with wavelengthλ trap = 532 nm, focused through a 100×, 1.45-NA oil-immersion objective (Olym- pus). Lateral control of the trap positionλ t w...

    2015

  32. [32]

    This term recovers the known open-loop Schmiedl-Seifert result⟨W ol⟩= κλ2 f 2+tf /τr in the ∆t→0 limit

    evaluated atn=N, and α= exp(−κ∆t/γ). This term recovers the known open-loop Schmiedl-Seifert result⟨W ol⟩= κλ2 f 2+tf /τr in the ∆t→0 limit. The second termg N(Σ) needs to be solved recursively, noting that the variance varies deterministically during a trajectory. It starts at Σ N =k BT /κand, while at each step it is set to zero by the observation, it a...