Leveraging vision and kinematics data to improve realism of biomechanic soft tissue simulation for robotic surgery

Jie Ying Wu, Peter Kazanzides, Mathias Unberath

Research output: Contribution to journalArticlepeer-review


Purpose: Surgical simulations play an increasingly important role in surgeon education and developing algorithms that enable robots to perform surgical subtasks. To model anatomy, finite element method (FEM) simulations have been held as the gold standard for calculating accurate soft tissue deformation. Unfortunately, their accuracy is highly dependent on the simulation parameters, which can be difficult to obtain. Methods: In this work, we investigate how live data acquired during any robotic endoscopic surgical procedure may be used to correct for inaccurate FEM simulation results. Since FEMs are calculated from initial parameters and cannot directly incorporate observations, we propose to add a correction factor that accounts for the discrepancy between simulation and observations. We train a network to predict this correction factor. Results: To evaluate our method, we use an open-source da Vinci Surgical System to probe a soft tissue phantom and replay the interaction in simulation. We train the network to correct for the difference between the predicted mesh position and the measured point cloud. This results in 15–30% improvement in the mean distance, demonstrating the effectiveness of our approach across a large range of simulation parameters. Conclusion: We show a first step towards a framework that synergistically combines the benefits of model-based simulation and real-time observations. It corrects discrepancies between simulation and the scene that results from inaccurate modeling parameters. This can provide a more accurate simulation environment for surgeons and better data with which to train algorithms.

Original languageEnglish (US)
Pages (from-to)811-818
Number of pages8
JournalInternational Journal of Computer Assisted Radiology and Surgery
Issue number5
StatePublished - May 1 2020
Externally publishedYes


  • Deep learning
  • Error correction
  • FEM
  • Robotic surgery
  • Simulation
  • Soft tissue deformation

ASJC Scopus subject areas

  • Health Informatics
  • Radiology Nuclear Medicine and imaging
  • Computer Vision and Pattern Recognition
  • Surgery
  • Biomedical Engineering
  • Computer Science Applications
  • Computer Graphics and Computer-Aided Design


Dive into the research topics of 'Leveraging vision and kinematics data to improve realism of biomechanic soft tissue simulation for robotic surgery'. Together they form a unique fingerprint.

Cite this