pith. sign in

arxiv: 2606.12048 · v1 · pith:KR5LFBCRnew · submitted 2026-06-10 · 💻 cs.RO

Point Cloud Segmentation for Autonomous Clip Positioning in Laparoscopic Cholecystectomy on a Phantom

Bal\'azs Gyenes , Nikolai Franke , Paul Maria Scheikl , Pit Henrich , Rayan Younis , Gerhard Neumann , Martin Wagner , Franziska Mathis-Ullrich This is my paper

Pith reviewed 2026-06-27 09:25 UTC · model grok-4.3

classification 💻 cs.RO
keywords point cloud segmentationautonomous robotic surgerylaparoscopic cholecystectomyclip positioningsynthetic data pre-trainingdata augmentationphantom evaluationminimally invasive surgery
0
0 comments X

The pith

A point-cloud segmentation model trained on 60 real examples plus synthetic pre-training enables the first autonomous robotic clip positioning on a laparoscopic surgery phantom at 0.75 mm precision.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper demonstrates the first robotic system that performs autonomous clip positioning on a physical phantom for laparoscopic cholecystectomy. After segmenting a colorless point cloud captured by a single camera, the system uses spline interpolation to locate clip targets that a human operator can adjust. The segmentation model overcomes data scarcity by pre-training on 128,000 synthetic point clouds and applying two novel augmentation techniques before fine-tuning on the 60 labeled real examples. In physical robot trials this yields target localization at the required 0.75 mm precision with 95 percent success and fully autonomous clip placement with 100 percent success while keeping every motion visually verifiable. A reader would care because the approach shows how limited surgical data can still support safe, interpretable automation in a high-stakes domain.

Core claim

The central claim is that a segmentation model trained on only 60 hand-labeled real point clouds, after pre-training on 128,000 synthetic point clouds and two new augmentation methods, produces segmentations accurate enough to extract clip targets via spline interpolation and to drive a robot to those targets with 0.75 mm precision at 95 percent success while executing the full autonomous clip-positioning task at 100 percent success on a physical phantom.

What carries the argument

Colorless point-cloud segmentation model that identifies anatomical structures so spline interpolation can compute clip target positions from a single-camera view.

If this is right

  • Target positions are localized to 0.75 mm at 95 percent success rate in robot experiments.
  • Autonomous clip positioning completes with 100 percent success on the phantom.
  • End-effector motion remains within minimally-invasive surgery constraints and is shown to the operator for verification.
  • The same pipeline supplies insights for other surgical or non-surgical tasks that require precise target identification and navigation.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the synthetic-plus-real training recipe generalizes, similar systems could be built for other clip or staple placement tasks without collecting hundreds of labeled surgical point clouds.
  • The interpretability requirement (visualized motion paths) may become a standard filter for any autonomous surgical robot that must remain under human oversight.
  • Extending the method to stereo or RGB-D input could reduce reliance on perfect point-cloud quality while preserving the low-data training strategy.

Load-bearing premise

The segmentation accuracy achieved on the phantom will remain high enough for safe clip placement when the same model is later used on real human tissue that differs in lighting, blood, and deformation.

What would settle it

A drop in segmentation accuracy below the level needed for 0.75 mm target localization when the trained model is tested on real-patient point clouds recorded under variable lighting and tissue deformation.

Figures

Figures reproduced from arXiv: 2606.12048 by Bal\'azs Gyenes, Franziska Mathis-Ullrich, Gerhard Neumann, Martin Wagner, Nikolai Franke, Paul Maria Scheikl, Pit Henrich, Rayan Younis.

Figure 1
Figure 1. Figure 1: We present a robotic system for autonomous clip [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Schematic of inference pipeline, showing how the input point cloud is segmented to isolate the target regions. Liver ( [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: In laparoscopic surgery, instruments are inserted into [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: The clip is applied onto the target structure in three [PITH_FULL_IMAGE:figures/full_fig_p005_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Different gallbladder phantoms used for real world [PITH_FULL_IMAGE:figures/full_fig_p006_5.png] view at source ↗
read the original abstract

High-risk applications in robotics, such as robot-assisted surgery, present unique challenges. These systems must be both highly precise and interpretable in order to be deployed in environments with very low tolerance for error or unsafe exploration. We present the first robotic system to demonstrate autonomous clip positioning on a physical phantom in laparoscopic surgery, one of the most common interventions in general surgery. After segmentation of a colorless point cloud from a single camera, target positions for the clips are extracted using spline interpolation, and can then be adjusted by the human operator. The segmentation model is trained on only 60 hand-labeled real point clouds, reflecting data scarcity in the surgical domain. We overcome this with a combination of pre-training on 128,000 synthetic point clouds and two novel data augmentation techniques. The motion of the end-effector to each target is visualized for the operator, satisfying the unique motion constraints of minimally-invasive surgery while ensuring that the robot's actions are verifiable and interpretable. In real robot experiments, our system localizes targets with the required precision of 0.75mm at a 95% success rate and executes autonomous clip positioning with a 100% success rate. We provide insights that are applicable to many other surgical and non-surgical tasks that require identifying and navigating to a precise target. Source code and project page: https://github.com/balazsgyenes/kirurc

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

2 major / 1 minor

Summary. The paper presents the first robotic system for autonomous clip positioning on a physical phantom for laparoscopic cholecystectomy. It performs segmentation on colorless point clouds from a single camera using a model pre-trained on 128,000 synthetic clouds and fine-tuned on 60 hand-labeled real ones with two novel augmentations; targets are then extracted via spline interpolation (with optional human adjustment), and the end-effector motion is visualized for interpretability. Physical robot experiments on the phantom report 95% success localizing targets to the required 0.75 mm precision and 100% success executing autonomous clip positioning.

Significance. If the reported success rates hold under the described experimental conditions, the work supplies a concrete, reproducible demonstration of precise autonomous action in a data-scarce surgical robotics setting on a phantom, together with an interpretable visualization step that respects MIS constraints. The public release of source code and project page is a clear strength that supports verification and extension.

major comments (2)
  1. [Experiments / Results] Experiments / Results section: the manuscript states a 95% success rate at the 0.75 mm threshold and a 100% clip-positioning success rate but provides neither the number of trials performed, error bars or confidence intervals, nor an explicit definition of what constitutes a failure or how the 0.75 mm threshold was selected; these omissions are load-bearing for the central empirical claim.
  2. [Methods] Methods section on data augmentation: the two novel augmentation techniques are described only at a high level; without their precise algorithmic definitions or pseudocode (even though code is released), it is difficult to assess how they address the domain gap between synthetic and real point clouds that underpins the reported segmentation accuracy.
minor comments (1)
  1. [Abstract / Discussion] The abstract and title correctly limit all quantitative claims to the phantom; the discussion of applicability to real tissue should be framed explicitly as future work rather than as a basis for the reported results.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the constructive feedback. We address each major comment below and will revise the manuscript to strengthen the reporting of experiments and the description of the augmentation methods.

read point-by-point responses

  1. Referee: [Experiments / Results] Experiments / Results section: the manuscript states a 95% success rate at the 0.75 mm threshold and a 100% clip-positioning success rate but provides neither the number of trials performed, error bars or confidence intervals, nor an explicit definition of what constitutes a failure or how the 0.75 mm threshold was selected; these omissions are load-bearing for the central empirical claim.

    Authors: We agree that these statistical details are necessary to substantiate the central claims. In the revised manuscript we will explicitly state the number of trials performed for both localization and clip placement, report error bars or binomial confidence intervals, provide a clear definition of failure (localization error > 0.75 mm or any clip placement that does not meet the target), and justify the 0.75 mm threshold by reference to the clinical tolerance required for safe clip application on the cystic duct in laparoscopic cholecystectomy. revision: yes

  2. Referee: [Methods] Methods section on data augmentation: the two novel augmentation techniques are described only at a high level; without their precise algorithmic definitions or pseudocode (even though code is released), it is difficult to assess how they address the domain gap between synthetic and real point clouds that underpins the reported segmentation accuracy.

    Authors: We accept that a high-level description is insufficient for reproducibility and evaluation. The revised Methods section will include the precise algorithmic definitions of both augmentation techniques together with pseudocode, explicitly showing the operations applied to synthetic point clouds to reduce the domain gap before fine-tuning on the 60 real samples. revision: yes

Circularity Check

0 steps flagged

No significant circularity

full rationale

The paper reports empirical success rates (95% localization at 0.75 mm, 100% clip positioning) measured directly in physical robot experiments on a phantom. The segmentation model is trained on 60 real + 128k synthetic point clouds with data augmentation, but the central claims are experimental outcomes on the physical setup rather than any derivation, equation, or prediction that reduces to its own fitted inputs by construction. No self-citations, uniqueness theorems, or ansatzes are invoked to support the reported performance; the results stand as direct measurements independent of any circular reduction.

Axiom & Free-Parameter Ledger

1 free parameters · 1 axioms · 0 invented entities

The central claim rests on the transferability of a segmentation model from synthetic data plus 60 real labels to physical phantom trials; no new physical constants or mathematical axioms are introduced beyond standard assumptions of camera calibration and rigid phantom geometry.

free parameters (1)
  • number of real labeled point clouds
    The choice of exactly 60 hand-labeled examples is a data-selection parameter that directly affects reported performance.
axioms (1)
  • domain assumption A single colorless point cloud from one camera is sufficient to identify clip target locations on the phantom anatomy.
    Invoked when the segmentation output is used to drive spline interpolation for clip targets.

pith-pipeline@v0.9.1-grok · 5810 in / 1502 out tokens · 18494 ms · 2026-06-27T09:25:11.080868+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

27 extracted references · 1 canonical work pages · 1 internal anchor

  1. [1]

    Epidemiology of gallbladder stone disease,

    E. A. Shaffer, “Epidemiology of gallbladder stone disease,”Best Practice & Research Clinical Gastroenterology, pp. 981–996, Jan. 2006

    2006

  2. [2]

    Autonomy for Surgical Robots: Concepts and Paradigms,

    T. Haidegger, “Autonomy for Surgical Robots: Concepts and Paradigms,” IEEE Transactions on Medical Robotics and Bionics, 2019

    2019

  3. [3]

    Point cloud models improve visual robustness in robotic learners,

    S. Peri, I. Lee, C. Kim, L. Fuxin, T. Hermans, and S. Lee, “Point cloud models improve visual robustness in robotic learners,”ICRA, 2024

    2024

  4. [4]

    DexPoint: General- izable Point Cloud Reinforcement Learning for Sim-to-Real Dexterous Manipulation,

    Y . Qin, B. Huang, Z.-H. Yin, H. Su, and X. Wang, “DexPoint: General- izable Point Cloud Reinforcement Learning for Sim-to-Real Dexterous Manipulation,” inCoRL, Mar. 2023

    2023

  5. [5]

    Real-time deep learning semantic segmentation during intra-operative surgery for 3D augmented reality assistance,

    L. Tanzi, P. Piazzolla, F. Porpiglia, and E. Vezzetti, “Real-time deep learning semantic segmentation during intra-operative surgery for 3D augmented reality assistance,”IJCARS, Sept. 2021

    2021

  6. [6]

    Autonomous Tissue Retraction in Robotic Assisted Minimally Invasive Surgery – A Feasibility Study,

    A. Attanasio, B. Scaglioni, M. Leonetti, A. F. Frangi, W. Cross, C. S. Biyani, and P. Valdastri, “Autonomous Tissue Retraction in Robotic Assisted Minimally Invasive Surgery – A Feasibility Study,”RAL, Oct. 2020

    2020

  7. [7]

    A Comparative Study of Spatio-Temporal U-Nets for Tissue Segmentation in Surgical Robotics,

    A. Attanasio, C. Alberti, B. Scaglioni, N. Marahrens, A. F. Frangi, M. Leonetti, C. S. Biyani, E. De Momi, and P. Valdastri, “A Comparative Study of Spatio-Temporal U-Nets for Tissue Segmentation in Surgical Robotics,”IEEE Trans. on Medical Robotics and Bionics, Feb. 2021

    2021

  8. [8]

    An Automatic Path Planning Method of Pedicle Screw Placement Based on Preoperative CT Images,

    X. Qi, J. Meng, M. Li, Y . Yang, Y . Hu, B. Li, J. Zhang, and W. Tian, “An Automatic Path Planning Method of Pedicle Screw Placement Based on Preoperative CT Images,”IEEE Trans. on Medical Robotics and Bionics, May 2022

    2022

  9. [9]

    Autonomous Needle Manipulation for Robotic Surgical Suturing Based on Skills Learned from Demonstration,

    K. L. Schwaner, D. Dall’Alba, P. T. Jensen, P. Fiorini, and T. R. Savarimuthu, “Autonomous Needle Manipulation for Robotic Surgical Suturing Based on Skills Learned from Demonstration,” inCASE, Aug. 2021

    2021

  10. [10]

    A Learning-Driven Framework with Spatial Optimization For Surgical Suture Thread Reconstruction and Autonomous Grasping Under Multiple Topologies and Environmental Noises,

    B. Lu, W. Chen, Y .-M. Jin, D. Zhang, Q. Dou, H. K. Chu, P.-A. Heng, and Y .-H. Liu, “A Learning-Driven Framework with Spatial Optimization For Surgical Suture Thread Reconstruction and Autonomous Grasping Under Multiple Topologies and Environmental Noises,” inIROS, Oct. 2020

    2020

  11. [11]

    Estimating Reference Bony Shape Models for Orthognathic Surgical Planning Using 3D Point-Cloud Deep Learning,

    D. Xiao, C. Lian, H. Deng, T. Kuang, Q. Liu, L. Ma, D. Kim, Y . Lang, X. Chen, J. Gateno, S. G. Shen, J. J. Xia, and P.-T. Yap, “Estimating Reference Bony Shape Models for Orthognathic Surgical Planning Using 3D Point-Cloud Deep Learning,”IEEE Journal of Biomedical and Health Informatics, vol. 25, no. 8, pp. 2958–2966, Aug. 2021

    2021

  12. [12]

    6DOF Needle Pose Estimation for Robot-Assisted Vitreoretinal Surgery,

    M. Zhou, X. Hao, A. Eslami, K. Huang, C. Cai, C. P. Lohmann, N. Navab, A. Knoll, and M. A. Nasseri, “6DOF Needle Pose Estimation for Robot-Assisted Vitreoretinal Surgery,”IEEE Access, 2019

    2019

  13. [13]

    SuPer: A Surgical Perception Framework for Endoscopic Tissue Manipulation With Surgical Robotics,

    Y . Li, F. Richter, J. Lu, E. K. Funk, R. K. Orosco, J. Zhu, and M. C. Yip, “SuPer: A Surgical Perception Framework for Endoscopic Tissue Manipulation With Surgical Robotics,”RAL, vol. 5, Apr. 2020

    2020

  14. [14]

    Learning Visual Shape Control of Novel 3D Deformable Objects from Partial-View Point Clouds,

    B. Thach, B. Y . Cho, A. Kuntz, and T. Hermans, “Learning Visual Shape Control of Novel 3D Deformable Objects from Partial-View Point Clouds,” inICRA, May 2022

    2022

  15. [15]

    Autonomous robotic laparoscopic surgery for intestinal anastomosis,

    H. Saeidi, J. D. Opfermann, M. Kam, S. Wei, S. Leonard, M. H. Hsieh, J. U. Kang, and A. Krieger, “Autonomous robotic laparoscopic surgery for intestinal anastomosis,”Science Robotics, p. eabj2908, Jan. 2022

    2022

  16. [16]

    PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation,

    C. R. Qi, H. Su, K. Mo, and L. J. Guibas, “PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation,” inCVPR, 2017

    2017

  17. [17]

    Point Trans- former,

    H. Zhao, L. Jiang, J. Jia, P. H. S. Torr, and V . Koltun, “Point Trans- former,” inCVPR, 2021

    2021

  18. [18]

    Dynamic Graph CNN for Learning on Point Clouds,

    Y . Wang, Y . Sun, Z. Liu, S. E. Sarma, M. M. Bronstein, and J. M. Solomon, “Dynamic Graph CNN for Learning on Point Clouds,”ACM Trans. Graph., vol. 38, no. 5, pp. 146:1–146:12, Oct. 2019

    2019

  19. [19]

    Masked Autoencoders for Point Cloud Self-supervised Learning,

    Y . Pang, W. Wang, F. E. H. Tay, W. Liu, Y . Tian, and L. Yuan, “Masked Autoencoders for Point Cloud Self-supervised Learning,” inECCV, 2022

    2022

  20. [20]

    Point-BERT: Pre- Training 3D Point Cloud Transformers With Masked Point Modeling,

    X. Yu, L. Tang, Y . Rao, T. Huang, J. Zhou, and J. Lu, “Point-BERT: Pre- Training 3D Point Cloud Transformers With Masked Point Modeling,” inCVPR, 2022

    2022

  21. [21]

    Improved Regularization of Convolutional Neural Networks with Cutout

    T. DeVries and G. W. Taylor, “Improved regularization of convolutional neural networks with cutout,”arXiv preprint arXiv:1708.04552, 2017

    work page internal anchor Pith review Pith/arXiv arXiv 2017

  22. [22]

    LapGym - An Open Source Framework for Re- inforcement Learning in Robot-Assisted Laparoscopic Surgery,

    P. M. Scheikl, B. Gyenes, R. Younis, C. Haas, G. Neumann, M. Wagner, and F. Mathis-Ullrich, “LapGym - An Open Source Framework for Re- inforcement Learning in Robot-Assisted Laparoscopic Surgery,”JMLR, 2023

    2023

  23. [23]

    Point Cloud Skeletons via Laplacian Based Contraction,

    J. Cao, A. Tagliasacchi, M. Olson, H. Zhang, and Z. Su, “Point Cloud Skeletons via Laplacian Based Contraction,” in2010 Shape Modeling International Conference, June 2010, pp. 187–197

    2010

  24. [24]

    Point Cloud Augmentation With Weighted Local Transformations,

    S. Kim, S. Lee, D. Hwang, J. Lee, S. J. Hwang, and H. J. Kim, “Point Cloud Augmentation With Weighted Local Transformations,” inICCV, 2021

    2021

  25. [25]

    PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space,

    C. R. Qi, L. Yi, H. Su, and L. J. Guibas, “PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space,” inNeurIPS. Curran Associates, Inc., 2017

    2017

  26. [26]

    Stratified transformer for 3d point cloud segmentation,

    X. Lai, J. Liu, L. Jiang, L. Wang, H. Zhao, S. Liu, X. Qi, and J. Jia, “Stratified transformer for 3d point cloud segmentation,” inCVPR, 2022

    2022

  27. [27]

    Stereo reconstruction from microscopic images for computer-assisted ophthalmic surgery,

    R. Peter, S. Moreira, E. Tagliabue, M. Hillenbrand, R. G. Nunes, and F. Mathis-Ullrich, “Stereo reconstruction from microscopic images for computer-assisted ophthalmic surgery,”IJCARS, Mar. 2025

    2025