Towards Autonomous Retinal Microsurgery Using RGB-D Images

Ji Woong Kim, Shuwen Wei, Peiyao Zhang, Peter Gehlbach, Jin U. Kang, Iulian Iordachita, Marin Kobilarov

Research output: Contribution to journalArticlepeer-review


Retinal surgery is a challenging procedure requiring precise manipulation of the fragile retinal tissue, often at the scale of tens-of-micrometers. Its difficulty has motivated the development of robotic assistance platforms to enable precise motion, and more recently, novel sensors such as microscope integrated optical coherence tomography (OCT) for RGB-D view of the surgical workspace. The combination of these devices opens new possibilities for robotic automation of tasks such as subretinal injection (SI), a procedure that involves precise needle insertion into the retina for targeted drug delivery. Motivated by this opportunity, we develop a framework for autonomous needle navigation during SI. We develop a system which enables the surgeon to specify waypoint goals in the microscope and OCT views, and the system autonomously navigates the needle to the desired subretinal space in real-time. Our system integrates OCT and microscope images with convolutional neural networks (CNNs) to automatically segment the surgical tool and retinal tissue boundaries, and model predictive control that generates optimal trajectories that respect kinematic constraints to ensure patient safety. We validate our system by demonstrating 30 successful SI trials on pig eyes. Preliminary comparisons to a human operator in robot-assisted mode highlight the enhanced safety and performance of our system.

Original languageEnglish (US)
Pages (from-to)3807-3814
Number of pages8
JournalIEEE Robotics and Automation Letters
Issue number4
StatePublished - Apr 1 2024


  • Computer vision for medical robotics
  • medical robots and systems
  • vision-based navigation

ASJC Scopus subject areas

  • Mechanical Engineering
  • Control and Optimization
  • Artificial Intelligence
  • Human-Computer Interaction
  • Control and Systems Engineering
  • Computer Vision and Pattern Recognition
  • Biomedical Engineering
  • Computer Science Applications


Dive into the research topics of 'Towards Autonomous Retinal Microsurgery Using RGB-D Images'. Together they form a unique fingerprint.

Cite this