Surgeon from a Distance

“We can imagine that with today's network bandwidth, the console could easily be positioned anywhere in the world, no matter where,” says computer scientist Anirban Mukhopadhyay of the Junior Research Group Medical and Environmental Computing (MEC-Lab) at TU Darmstadt “And if the bandwidth is stable enough, it is also possible to operate remotely. But where will we find these trained surgical assistants who are on-site for the procedure and will help?” So the researchers working with Mukhopadhyay are developing artificial intelligence that is able to take over these tasks in remote and rural areas or to support less qualified personnel.

Previous research in the field of artificial intelligence, in particular in image analysis, is not sufficient for this. For example, in order to to analyse a surgical workflow, the classical methods from computer vision and deep learning algorithms are adapted with minimal changes so they can use the surgical image data. In particular, the pixels of the videos are considered in order to make a classification. But this doesn't do justice to surgery. It's a complex process. It requires a lot of communication, because changes are constantly occurring in the operating theatre. Plus there is a lack of data that can be used to train artificial intelligence. Data sets in the five- or six-digit range are often used for image analysis, but they are not available during operations. You'll never get lots of examples in surgery because no two operations are completely the same.

“We've looked at how we can start with shapes in the image instead of pixels,” says Mukhopadhyay. “Surgeons are interested in the anatomical shape, the elasticity of these different shapes, and how to operate, what we can touch and what we shouldn't touch. So we tried to incorporate these ideas in the deep learning algorithms.” Shapes only occur with a specific topology or geometry. If an AI can recognise these aspects, then it will require less data for learning.

One example is cochlear implants, hearing prostheses that are inserted in the inner ear. The prostheses consist of two components, the actual implant and a speech processor that is worn behind the ear like a hearing aid. It contains a microphone that picks up acoustic signals. These are converted into electrical signal patterns in the speech processor and sent through the skin to the implant via a transmitter coil. The implant passes them on to electrodes that stimulate the auditory nerves.

The researchers use preoperative computer tomography scans of the head to train artificial intelligence. These are made to plan operations – such as how to insert the probe and place the electrode in the cochlea. So the patient does not also have to be irradiated in order to create training data. Once the algorithms have found the upper, middle and lower parts of a nerve in these scans, the robot knows where to make the connection with the implant. This means the system does not have to take every pixel in the image into account, but only has to identify the shapes. The advantage of this is that fewer training images are required for deep learning. “We were able to show that our algorithms already work robustly with 15 to 20 examples,” says Mukhopadhyay.

Building on this idea of shape consistency, the researchers introduced supervised contrastive learning – in which the AI learns to segment surgical instruments from real laparoscopic abdominal surgery solely by viewing simulated images.