Orsi Academy brings real-time artificial intelligence to the OR for a first-of-its-kind augmented reality surgery
-
A successful use case to enhance the safety of augmented reality (AR)-guided surgery
-
Development of a deep learning algorithm to detect all non-organic items. More than 130 000 manually labeled instruments on 31 812 frames, an effort duration estimated over 2000 hours.
-
Real-time instrument detection during AR-guided robot-assisted partial nephrectomy
-
The best of both worlds: Orsi’s datasets, AI networks and in-house knowledge combined with dedicated hardware and software by NVIDIA & DELTACAST
In February 2023, Orsi Academy integrated the NVIDIA Holoscan platform & the 12G-SDI DELTACAST capture card in the operating room to support an in-human robot-assisted surgery for the first time. The partial nephrectomy performed at the OLV Hospital Aalst in Belgium was definitely a first for surgery and the healthcare industry to use NVIDIA Holoscan in the operating room.
To treat a kidney cancer patient, urology surgeon Dr. Ruben De Groote, assisted by Orsi’s engineers Jente Simoens & Jasper Hofman, successfully performed kidney tumor removal using the Intuitive DaVinci robot with the help of augmented reality (AR). This AR overlay was derived from the patient’s pre-operative CT scan and was rendered with real-time AI-assisted instrument detection. The video feed overlay allowed the surgeon to more clearly see the patient’s vascular and tissue structures and interact with them, as instruments were now no longer obstructed by the AR overlay.
What made this procedure a landmark step?
- First use of NVIDIA Holoscan in the operating room for in-human robot-assisted surgery
- Real-time detection of up to 37 types of non-organic structures. No instrument obstruction by the 3D model during the procedure.
- Low-latency patient-specific AR overlay, derived from the pre-operative CT scan
- All rendered in real-time during robot-assisted partial nephrectomy surgery
Pushing boundaries
Our Surgical AI Lead, Dr. Pieter De Backer, led his team’s collaboration with NVIDIA and Deltacast: “Exploring the full potential of augmented reality and artificial intelligence has been a long-standing challenge in surgery. Robotic surgery is particularly suited to push these boundaries due to the integration of different technologies. Orsi’s state-of-the-art datasets, AI networks and in-house knowledge in combination with the dedicated hardware and software by NVIDIA and Deltacast has had a tremendous multiplicative effect. A true team effort!”
Orsi Academy previously solved the problem of offline instrument detection during surgery. However, the approach was insufficient for real-time use due to high latency. This in turn hindered its potential to be used in live operations and its utility to surgeons. “While important challenges still lie ahead, we now know that computational resources are no longer the crucial bottleneck in the process. We are entering an exciting time for surgery!” Dr. De Backer concluded.
Seeing through organs
Coming out of this uniquely augmented surgery, this is what Dr. De Groote had to say: “With Orsi managing to reduce latency to an absolute minimum, augmented reality is ready to be used in clinical practice to optimally guide surgeons during challenging surgical cases. It allows you to navigate with higher precision and uncover hidden vulnerable anatomy, which is a significant asset for the surgeon.”
Previous research on this subject by the Orsi innovation team
- Improving Augmented Reality Through Deep Learning: Real-time Instrument Delineation in Robotic Renal Surgery (open access!)
Would you like to be part of pioneering research?
We are looking for a Senior Computer Engineer and an NLP Engineer!