pith. sign in

arxiv: 2605.27643 · v1 · pith:GPDGZPENnew · submitted 2026-05-26 · 💻 cs.RO · physics.optics

Agentic Language-to-Objective Synthesis for Optofluidic Assembly

Pith reviewed 2026-06-29 16:53 UTC · model grok-4.3

classification 💻 cs.RO physics.optics
keywords Speak-to-Objectiveoptofluidic assemblylarge language modelsdifferentiable objectivesmicroparticle patterningnatural language interfaceslaser-induced flowsinverse optimization
0
0 comments X

The pith

An LLM pipeline turns spoken commands into differentiable objectives that assemble microparticles via laser-driven fluid flows.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper introduces Speak-to-Objective, a modular agentic pipeline that conditions a large language model to translate user commands into fully differentiable objective functions. These objectives feed a constraint-aware inverse solver (SLSQP) and an experimental optofluidic platform that uses laser-induced thermoviscous flows to move microparticles. The pipeline composes separate geometry, spacing, and assignment or topology terms to produce robust objectives that succeed from partial traces, recover after perturbations, and support both descriptive patterning and precise placement. The design keeps the specification of what to assemble separate from the actuation method, creating an actuator-agnostic interface between natural language and physical assembly.

Core claim

Speak-to-Objective uses a perceive-compose-propose-act-report-and-learn loop in which a conditioned LLM generates composite objective functions from geometry, spacing, and assignment terms; these functions are executable by SLSQP and, when paired with laser-induced thermoviscous flows, produce natural-language-programmable microscale particle assemblies in a microfluidic environment that remain robust to incomplete information and external disturbances.

What carries the argument

The Speak-to-Objective pipeline, which composes geometry, spacing, and assignment/topology terms into composite differentiable objectives that interface user intent with an inverse solver and physical actuation.

If this is right

  • Objectives generated by the pipeline can assemble particle patterns from partial traces without requiring complete initial specifications.
  • The same objectives enable recovery of the target pattern after external perturbations during assembly.
  • Explicit objectives produced by the pipeline support precise placement of individual particles in addition to descriptive patterning.
  • Because the objectives are actuator-agnostic, the same language-to-objective translation can be reused with different physical actuation methods.
  • The closed loop that incorporates user feedback allows the system to refine objectives over repeated interactions.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The separation of intent specification from actuation details could allow the same pipeline to control assembly in other light-based or robotic manufacturing settings.
  • Repeated user feedback in the report-and-learn step may gradually improve the LLM's ability to produce objectives that require less solver iteration.
  • If the objective composition generalizes, the approach could reduce the expert knowledge needed to program new microscale assembly tasks.

Load-bearing premise

A conditioned large language model can reliably output fully differentiable objective functions that the SLSQP solver can execute to produce successful assemblies on the optofluidic platform without any post-hoc manual adjustments.

What would settle it

Run the pipeline on several distinct natural-language commands, feed the generated objectives directly to the SLSQP solver, and check whether the resulting particle patterns on the experimental platform match the commanded geometries and placements with no manual editing of the objectives.

Figures

Figures reproduced from arXiv: 2605.27643 by Elena Erben, Eric Lauga, Fan Nan, Gerhard Neumann, Ivan Saraev, Moritz Kreysing, Weida Liao.

Figure 1
Figure 1. Figure 1: Figure1: Agentic intent-to-light-based-assembly framework: [PITH_FULL_IMAGE:figures/full_fig_p011_1.png] view at source ↗
Figure 5
Figure 5. Figure 5: Descriptive-objective control for natural-language-programmable [PITH_FULL_IMAGE:figures/full_fig_p016_5.png] view at source ↗
read the original abstract

Light-based advanced manufacturing increasingly requires programmable, closed-loop tools that translate human design intent into executable operations at small length scales. Yet a key bottleneck persists across robotic and manufacturing modalities: turning user intent into machine-readable objectives that are reliably executable. While micro-robotics offers versatile manipulation via optical actuation of fluids, mathematically tractable goal specification remains manual and hard to reuse. Here, we introduce Speak-to-Objective, a modular agentic pipeline that uses a conditioned Large Language Model (LLM) to translate spoken or written commands into fully differentiable objective functions for assembling microparticles in a constraint-aware inverse solver (SLSQP) and on an experimental optofluidic platform. The approach employs a compact loop - perceive -> compose -> propose -> act -> report & learn - that treats the objective as the interface between intent and actuation, separating what to assemble or pattern from how to actuate, while learning from user feedback. The pipeline composes geometry, spacing, and assignment/topology terms to generate robust descriptive objectives that assemble from partial traces and recover after perturbations, as well as explicit objectives for precise placement, all in an actuator-agnostic fashion. Using laser-induced thermoviscous flows as the physical actuation modality, we demonstrate natural-language-programmable, light-based microscale assembly of particle patterns in a microfluidic environment. Beyond its immediate impact on programmable microassembly, and using laser-induced optofluidic actuation as a reduced-complexity experimental platform, our work points toward self-driving, AI-assisted optical manufacturing platforms in which natural language, differentiable objectives, and laser-based actuation are coupled into a reusable digital workflow.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

2 major / 1 minor

Summary. The paper introduces Speak-to-Objective, a modular agentic pipeline that uses a conditioned large language model to translate spoken or written commands into fully differentiable objective functions. These functions are optimized via a constraint-aware SLSQP inverse solver for microparticle assembly and validated experimentally on an optofluidic platform using laser-induced thermoviscous flows. The pipeline employs a perceive-compose-propose-act-report&learn loop that separates user intent from actuation; it composes geometry, spacing, and assignment/topology terms to produce objectives claimed to support assembly from partial traces, recovery after perturbations, precise placement, and actuator-agnostic operation, with a demonstration of natural-language-programmable microscale assembly.

Significance. If the claims hold with supporting quantitative evidence, the work could advance programmable micro-manufacturing by providing a reusable natural-language interface to differentiable objectives and physical actuation. The explicit separation of intent from actuation, combined with LLM-based composition and closed-loop feedback, offers a pathway toward self-driving optical manufacturing platforms. The reduced-complexity optofluidic testbed supplies a concrete experimental anchor, though the absence of metrics in the abstract limits evaluation of the robustness assertions.

major comments (2)
  1. [Abstract] Abstract: The central claims that the composed objectives are 'robust' and enable 'assembly from partial traces and recover after perturbations' are presented without any quantitative results, success rates, error bars, validation data, or statistical analysis, which is load-bearing for the assertion that an LLM can reliably produce fully executable objectives for the SLSQP solver without post-hoc manual adjustments.
  2. [Abstract] Abstract: No experimental details are supplied on trial counts, perturbation types, recovery metrics, or comparisons to manual objective specification, undermining assessment of the actuator-agnostic and perturbation-recovery claims that form the core of the pipeline's contribution.
minor comments (1)
  1. [Abstract] The abstract would benefit from inclusion of at least one key quantitative performance metric to ground the robustness claims.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the constructive feedback on the abstract. We agree that quantitative support for the robustness and recovery claims is needed in the abstract itself to strengthen the presentation. We will revise the abstract accordingly while preserving the manuscript's core contributions. Point-by-point responses are below.

read point-by-point responses
  1. Referee: [Abstract] Abstract: The central claims that the composed objectives are 'robust' and enable 'assembly from partial traces and recover after perturbations' are presented without any quantitative results, success rates, error bars, validation data, or statistical analysis, which is load-bearing for the assertion that an LLM can reliably produce fully executable objectives for the SLSQP solver without post-hoc manual adjustments.

    Authors: We agree that the abstract would be strengthened by including quantitative metrics. The full manuscript reports experimental results with trial counts, success rates, and recovery metrics in the results section. We will revise the abstract to add a concise summary of these quantitative findings (e.g., success rates for partial-trace assembly and perturbation recovery) to better support the claims. revision: yes

  2. Referee: [Abstract] Abstract: No experimental details are supplied on trial counts, perturbation types, recovery metrics, or comparisons to manual objective specification, undermining assessment of the actuator-agnostic and perturbation-recovery claims that form the core of the pipeline's contribution.

    Authors: We acknowledge that the abstract lacks these experimental details. We will update the abstract to incorporate key information on trial counts, perturbation types tested, and recovery metrics drawn from the experimental validation in the manuscript. This revision will address the concern without requiring changes to the underlying results or methods. revision: yes

Circularity Check

0 steps flagged

No significant circularity detected

full rationale

The paper describes an agentic LLM-based pipeline (Speak-to-Objective) that composes geometry/spacing/assignment terms into differentiable objectives for SLSQP optimization and experimental optofluidic actuation. No equations, fitted parameters, or derivation steps are presented that reduce to self-definition, fitted-input-as-prediction, or self-citation load-bearing. The central claim rests on external experimental validation with thermoviscous flows rather than internal reduction to inputs. The pipeline is framed as separating intent from actuation with reported robustness to partial traces and perturbations, making the work self-contained against external benchmarks.

Axiom & Free-Parameter Ledger

0 free parameters · 0 axioms · 0 invented entities

Only the abstract is available; no free parameters, axioms, or invented entities are described in sufficient detail to populate the ledger.

pith-pipeline@v0.9.1-grok · 5840 in / 1250 out tokens · 55285 ms · 2026-06-29T16:53:46.272007+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

46 extracted references · 7 canonical work pages · 1 internal anchor

  1. [1]

    Institute of Biological and Chemical Systems, Karlsruhe Institute of Technology, Germany

  2. [2]

    Department of Applied Mathematics and Theoretical Physics, University of Cambridge, UK

  3. [3]

    Department of Mathematics, Imperial College London, UK

  4. [4]

    action-at-a-distance

    Institute of Anthropomatics and Robotics (IAR), Karlsruhe Institute of Technology, Germany *e-mail: moritz.kreysing@kit.edu Abstract Light-based advanced manufacturing increasingly requires programmable, closed- loop tools that translate human design intent into executable operations at small length scales. Yet a key bottleneck persists across robotic and...

  5. [5]

    M.; Bjorkholm, J

    Ashkin, A.; Dziedzic, J. M.; Bjorkholm, J. E.; Chu, S., Observation of a single-beam gradient force optical trap for dielectric particles. Opt. Lett. 1986, 11 (5), 288-290

  6. [6]

    G., A revolution in optical manipulation

    Grier, D. G., A revolution in optical manipulation. Nature 2003, 424 (6950), 810-816

  7. [7]

    G., Optical forces arising from phase gradients

    Roichman, Y.; Sun, B.; Roichman, Y.; Amato-Grill, J.; Grier, D. G., Optical forces arising from phase gradients. Phys. Rev. Lett. 2008, 100 (1), 013602

  8. [8]

    T., Optical pulling force

    Chen, J.; Ng, J.; Lin, Z.; Chan, C. T., Optical pulling force. Nat. Photon. 2011, 5 (9), 531-534

  9. [9]

    M.; Jones, P

    Maragò, O. M.; Jones, P. H.; Gucciardi, P. G.; Volpe, G.; Ferrari, A. C., Optical trapping and manipulation of nanostructures. Nat. Nanotechnol. 2013, 8 (11), 807-819

  10. [10]

    Light Sci

    Gao, D.; Ding, W.; Nieto-Vesperinas, M.; Ding, X.; Rahman, M.; Zhang, T.; Lim, C.; Qiu, C.-W., Optical manipulation from the microscale to the nanoscale: fundamentals, advances and prospects. Light Sci. Appl. 2017, 6 (9), e17039-e17039

  11. [11]

    Li, X.; Dan, D.; Zavatski, S.; Gao, W.; Zhang, Q.; Zhou, Y.; Qian, J.; Yang, Y.; Yu, X.; Yan, S.; Xu, X.; Martin, O. J. F.; Yao, B., Optical tweeze-sectioning microscopy for 3D imaging and manipulation of suspended cells. Sci. Adv. 2025, 11 (27), eadx3900

  12. [12]

    C.; Pané, S.; Zhang, L.; Pumera, M., Magnetically Driven Micro and Nanorobots

    Zhou, H.; Mayorga-Martinez, C. C.; Pané, S.; Zhang, L.; Pumera, M., Magnetically Driven Micro and Nanorobots. Chemical Reviews 2021, 121 (8), 4999-5041

  13. [13]

    J.; Choi, H., Fabrication and Characterization of Magnetic Microrobots for Three- Dimensional Cell Culture and Targeted Transportation

    Kim, S.; Qiu, F.; Kim, S.; Ghanbari, A.; Moon, C.; Zhang, L.; Nelson, B. J.; Choi, H., Fabrication and Characterization of Magnetic Microrobots for Three- Dimensional Cell Culture and Targeted Transportation. Advanced Materials 2013, 25 (41), 5863-5868

  14. [14]

    G.; Gibson, G

    Būtaitė, U. G.; Gibson, G. M.; Ho, Y.-L. D.; Taverne, M.; Taylor, J. M.; Phillips, D. B., Indirect optical trapping using light driven micro-rotors for reconfigurable hydrodynamic manipulation. Nat. Commun. 2019, 10 (1), 1215

  15. [15]

    B.; Padgett, M

    Phillips, D. B.; Padgett, M. J.; Hanna, S.; Ho, Y. L. D.; Carberry, D. M.; Miles, M. J.; Simpson, S. H., Shape-induced force fields in optical trapping. Nature Photonics 2014, 8 (5), 400-405

  16. [16]

    Light: Science & Applications 2020, 9 (1), 84

    Li, D.; Liu, C.; Yang, Y.; Wang, L.; Shen, Y., Micro-rocket robot with all-optic actuating and tracking in blood. Light: Science & Applications 2020, 9 (1), 84

  17. [17]

    K.; Liu, L.; Huang, Q.; Wu, M

    Zhang, S.; Xu, B.; Elsayed, M.; Nan, F.; Liang, W.; Valley, J. K.; Liu, L.; Huang, Q.; Wu, M. C.; Wheeler, A. R., Optoelectronic tweezers: a versatile toolbox for nano-/micro-manipulation. Chem. Soc. Rev. 2022, 51 (22), 9203-9242

  18. [18]

    The Journal of Physical Chemistry C 2025, 129 (11), 5502-5510

    Dara, P.; Käll, M., Bubble Dynamics and Directional Marangoni Flow Induced by Laser Heating of Silicon Nanodisk Arrays. The Journal of Physical Chemistry C 2025, 129 (11), 5502-5510

  19. [19]

    The Journal of Physical Chemistry C 2023, 127 (35), 17454-17459

    Dara, P.; Shanei, M.; Jones, S.; Käll, M., Directional Control of Transient Flows Generated by Thermoplasmonic Bubble Nucleation. The Journal of Physical Chemistry C 2023, 127 (35), 17454-17459

  20. [20]

    S.; Anderson, L

    Nedev, S.; Carretero-Palacios, S.; Kühler, P.; Lohmüller, T.; Urban, A. S.; Anderson, L. J. E.; Feldmann, J., An Optically Controlled Microscale Elevator Using Plasmonic Janus Particles. ACS Photonics 2015, 2 (4), 491-496

  21. [21]

    Science Robotics 2017, 2 (4), eaam6431

    Li, J.; Esteban-Fernández de Ávila, B.; Gao, W.; Zhang, L.; Wang, J., Micro/ nanorobots for biomedicine: Delivery, surgery, sensing, and detoxification. Science Robotics 2017, 2 (4), eaam6431

  22. [22]

    M., Optofluidic microsystems for chemical and biological analysis

    Fan, X.; White, I. M., Optofluidic microsystems for chemical and biological analysis. Nature Photonics 2011, 5 (10), 591-597

  23. [23]

    M.; Braun, D., Optically driven fluid flow along arbitrary microscale patterns using thermoviscous expansion

    Weinert, F. M.; Braun, D., Optically driven fluid flow along arbitrary microscale patterns using thermoviscous expansion. J. Appl. Phys. 2008, 104 (10)

  24. [24]

    M.; Kraus, J

    Weinert, F. M.; Kraus, J. A.; Franosch, T.; Braun, D., Microscale fluid flow induced by thermoviscous expansion along a traveling wave. Phys. Rev. Lett. 2008, 100 (16), 164501

  25. [25]

    W.; Iserman, C.; Kar, M.; Munder, M.; Voigt, A.; Alberti, S.; Grill, S

    Mittasch, M.; Gross, P.; Nestler, M.; Fritsch, A. W.; Iserman, C.; Kar, M.; Munder, M.; Voigt, A.; Alberti, S.; Grill, S. W.; Kreysing, M., Non-invasive perturbations of intracellular flow reveal physical principles of cell organization. Nat. Cell Biol. 2018, 20 (3), 344-351

  26. [26]

    D.; Klykov, S.; Maghelli, N.; Kreysing, M., Feedback-based positioning and diffusion suppression of particles via optical control of thermoviscous flows

    Erben, E.; Seelbinder, B.; Stoev, I. D.; Klykov, S.; Maghelli, N.; Kreysing, M., Feedback-based positioning and diffusion suppression of particles via optical control of thermoviscous flows. Opt. Express 2021, 29 (19), 30272-30283

  27. [27]

    D.; Seelbinder, B.; Erben, E.; Maghelli, N.; Kreysing, M., Highly sensitive force measurements in an optically generated, harmonic hydrodynamic trap

    Stoev, I. D.; Seelbinder, B.; Erben, E.; Maghelli, N.; Kreysing, M., Highly sensitive force measurements in an optically generated, harmonic hydrodynamic trap. eLight 2021, 1 (1), 7

  28. [28]

    Liao, W.; Erben, E.; Kreysing, M.; Lauga, E., Theoretical model of confined thermoviscous flows for artificial cytoplasmic streaming. Phys. Rev. Fluids 2023, 8 (3), 034202

  29. [29]

    Light Sci

    Erben, E.; Liao, W.; Minopoli, A.; Maghelli, N.; Lauga, E.; Kreysing, M., Opto-fluidically multiplexed assembly and micro-robotics. Light Sci. Appl. 2024, 13 (1), 59

  30. [30]

    Small 2025, e01039

    Erben, E.; Saraev, I.; Liao, W.; Nan, F.; Lauga, E.; Kreysing, M., Optical Micromanipulations Based on Model Predictive Control of Thermoviscous Flows. Small 2025, e01039

  31. [31]

    P.; Parlane, F

    MacLeod, B. P.; Parlane, F. G. L.; Morrissey, T. D.; Häse, F.; Roch, L. M.; Dettelbach, K. E.; Yunker, L. P. E.; Hein, J. E.; Berlinguette, C. P., Self-driving laboratory for accelerated discovery of thin-film materials. _Sci. Adv._ 2020, _6_ (20), eaaz8867

  32. [32]

    M.; Gusev, V

    Burger, B.; Maffettone, P. M.; Gusev, V. V.; Aitchison, C. M.; Bai, Y.; Wang, X.; Li, X.; Alston, B. M.; Li, B.; Clowes, R.; Rankin, N.; Harris, B.; Sprick, R. S.; Cooper, A. I., A mobile robotic chemist. _Nature_ 2020, _583_, 237-241

  33. [33]

    M.; Donina, L.; Dragone, V.; Long, D.-L.; Cronin, L., Controlling an organic synthesis robot with machine learning to search for new reactivity

    Granda, J. M.; Donina, L.; Dragone, V.; Long, D.-L.; Cronin, L., Controlling an organic synthesis robot with machine learning to search for new reactivity. _Nature_ 2018, _559_, 377-381

  34. [34]

    Kalhor, P.; Jung, N.; Bräse, S.; Wöll, C.; Tsotsalas, M.; Friederich, P., Functional Material Systems Enabled by Automated Data Extraction and Machine Learning. _Adv. Funct. Mater._ 2023, _34_ (20), 2302630

  35. [35]

    W.; Thomas, D

    Coley, C. W.; Thomas, D. A.; Lummiss, J. A. M.; Jaworski, J. N.; Breen, C. P.; et al., A robotic platform for flow synthesis of organic molecules informed by AI planning. _Science_ 2019, _365_ (6453), 557-561

  36. [36]

    F.; Aspuru-Guzik, A., Autonomous chemical research with large language models

    Boiko, D.; Chan, B.; Lu, C.; Jensen, K. F.; Aspuru-Guzik, A., Autonomous chemical research with large language models. Nature 2023, 624, 570–578

  37. [37]

    Agentic Exploration of Physics Models

    Nägele, M.; Marquardt, F., Agentic exploration of physics models. arXiv preprint arXiv:2509.24978 (2025)

  38. [38]

    Eureka: Human-Level Reward Design via Coding Large Language Models

    Ma, Yecheng Jason; Liang, William; Wang, Guanzhi; Huang, De-An; Bastani, Osbert; Jayaraman, Dinesh; Zhu, Yuke; Fan, Linxi (“Jim”); Anandkumar, Anima. Eureka: Human-Level Reward Design via Coding Large Language Models. In: Proceedings of the 12th International Conference on Learning Representations (ICLR 2024)

  39. [39]

    DrEureka: Language Model Guided Sim-To-Real Transfer

    Ma, Yecheng Jason; Liang, William; Wang, Hung-Ju; Wang, Sam; Zhu, Yuke; Fan, Linxi; Bastani, Osbert; Jayaraman, Dinesh. DrEureka: Language Model Guided Sim-To-Real Transfer. In: Proceedings of the Robotics: Science and Systems (RSS) Conference 2024

  40. [40]

    arXiv preprint arXiv:2508.17921 (2025)

    Paul, D.; Milosevic, N.; Scherf, N.; Cichos, F., Physical embodiment enables information processing beyond explicit sensing in active matter. arXiv preprint arXiv:2508.17921 (2025)

  41. [41]

    de Bakker, V.; Hejna, J.; Lum, T. G. W.; Celik, O.; Taranovic, A.; Blessing, D.; Neumann, G.; Bohg, J.; Sadigh, D., Scaffolding Dexterous Manipulation with Vision- Language Models. arXiv preprint arXiv:2506.19212 (2025)

  42. [42]

    M.; Häse, F.; Kreisbeck, C.; Tamayo-Mendoza, T.; Yunker, L

    Roch, L. M.; Häse, F.; Kreisbeck, C.; Tamayo-Mendoza, T.; Yunker, L. P. E.; Hein, J. E.; Aspuru-Guzik, A., ChemOS: An orchestration software to democratize autonomous discovery. _Chem. Sci._ 2018, _9_, 6091-6098

  43. [43]

    Flows inside Cells: From Natural Cytoplasmic Streaming to Microfluidic Systems

    Liao, Weida. Flows inside Cells: From Natural Cytoplasmic Streaming to Microfluidic Systems. Apollo - University of Cambridge Repository, 2025, doi:10.17863/CAM.122076

  44. [44]

    E.; Chang, B.; Mitchener, L.; Yiu, A.; Szostkiewicz, C

    Ghareeb, A. E.; Chang, B.; Mitchener, L.; Yiu, A.; Szostkiewicz, C. J.; Shved, D.; Gyimesi, G. J.; Laurent, J. M.; Wright, S. M.; Razzak, M. T.; White, A. D.; Finnemann, S. C.; Hinks, M. M.; Rodriques, S. G. A multi-agent system for automating scientific discovery. Nature 2026, doi:10.1038/s41586-026-10652-y

  45. [45]

    Y.; Norgaard, P.; Shamsi, Z.; Smalling, D.; Thompson, J.; Venugopalan, S.; Williams, B

    Aygün, E.; Belyaeva, A.; Comanici, G.; Coram, M.; Cui, H.; Garrison, J.; Johnston, R.; Kast, A.; McLean, C. Y.; Norgaard, P.; Shamsi, Z.; Smalling, D.; Thompson, J.; Venugopalan, S.; Williams, B. P.; et al. An AI system to help scientists write expert-level empirical software. Nature 2026, doi:10.1038/s41586-026-10658-6

  46. [46]

    Accelerating scientific discovery with Co- Scientist

    Gottweis, J.; Weng, W.-H.; Daryin, A.; Tu, T.; Sirkovic, P.; Myaskovsky, A.; Glowaty, G.; Weissenberger, F.; Orlandi, A.; Tomašev, N.; Zverinski, D.; Rendulic, I.; Vedadi, E.; Hasler, F.; Rimanic, L.; et al. Accelerating scientific discovery with Co- Scientist. Nature 2026, doi:10.1038/s41586-026-10644-y