Augmented Reality, Superhuman Abilities and the Future of Medicine

Earlier this month, I participated as a panelist at the Digital Orthopedics Conference in San Francisco (DOCSF 2022), where a key theme was envisioning the medical profession in 2037. In preparation for the event, a small group of us reviewed the latest research on the clinical applications of virtual and augmented reality and critically assessed the current state of the field.

I have to admit, I was deeply impressed by how far augmented reality (AR) has come for use in medicine in the last 18 months. So much so that I don’t expect we’ll have to wait until 2037 for AR to have a major impact on the field. In fact, I expect augmented reality to become a common tool for surgeons, radiologists, and many other medical professionals by the end of this decade. And by the early 2030s, many of us will be going to the family doctor and being examined by a doctor wearing AR glasses.

The reason is simple:

Augmented reality will give doctors superpowers.

I’m talking about superhuman abilities to visualize medical images, patient data and other clinical content. The costs associated with these new capabilities are already quite reasonable and will fall quickly as augmented reality hardware is produced at higher volumes in the coming years.

The first superpower is x-ray vision.

Augmented Reality will give doctors the ability to look directly inside a patient and see signs of trauma or disease at the exact spot in their body where they are. Of course, there is already the ability to look under the skin with tools like CT and MRI scans, but currently doctors are looking at these images on flat screen monitors and have to imagine how the images relate to the patient on the table. This type of mental transformation is an impressive skill, but it takes time and cognitive effort, and is nowhere near as informative as it would be if doctors could simply look inside the human body.

With AR headsets and new techniques for registering 3D medical images to a patient’s real body, the superpower of x-ray vision is now a reality. A stunning study from Teikyo University School of Medicine in Japan tested an experimental emergency room with the ability to take full-body CT scans of trauma patients and allow the medical team, all wearing AR headsets, an instant look inside the patient Examination table and see the trauma exactly where it is. This allowed the team to discuss injuries and plan treatment without having to resort to flat screens, saving time, reducing distractions and eliminating the need for mental transformations.

In other words, AR technology takes medical images off the screen and places them in 3D space exactly where it’s most useful for the doctor – perfectly aligned with the patient’s body. Such an ability is so natural and intuitive that I foresee that it will quickly permeate all medical applications. In fact, I expect doctors in the early 2030s will look back at the old way of doing things, glancing back and forth at flat screens, as awkward and primitive.

Augmented reality technology goes beyond x-ray vision and provides physicians with assistive content that is superimposed on (and into) the patient’s body to assist them in clinical tasks. For example, surgeons performing a delicate procedure will see real-time navigation cues projected onto the patient, showing the exact location where procedures need to be performed with precision. The aim is to increase accuracy, reduce mental effort and speed up the procedure. The potential value of surgery is extreme, from minimally invasive procedures such as laparoscopy and endoscopy to more hands-free surgical procedures such as the placement of orthopedic implants.

The concept of augmented surgery has been a goal of AR researchers since the invention of core technologies. In fact, it dates back to the first AR system (the Virtual Fixtures platform) developed at the Air Force Research Laboratory (AFRL) in the early 1990s. The aim of this project was to show that AR can improve human dexterity in precision tasks such as surgery. As someone involved in this early work, I have to say that the progress the field has made in the decades since has been remarkable.

Consider this: When we tested this first AR system with human subjects in 1992, we asked users to move metal pins between holes two feet apart to quantify whether virtual overlays could improve manual performance. Now, thirty years later, a team from Johns Hopkins, Thomas Jefferson University Hospital and Washington University performed delicate spinal surgery on 28 patients with AR to aid in the placement of metal screws with a precision of less than 2mm. As published in a recent study, the screw placement system achieved such accurate registration between the real patient and the virtual overlays that surgeons achieved 98% of standard performance metrics.

Looking ahead, we can expect augmented reality to impact all aspects of medicine as precision has reached clinically feasible levels. In addition, important breakthroughs are in the works that will make AR faster and easier to use in medical environments. As described above, the greatest challenge for any precision augmented reality application is the accurate registration of the real world and the virtual world. In medicine, this currently means attaching physical markers to the patient, which is time-consuming and laborious. In a recent study from Imperial College London and the University of Pisa, researchers tested a “markerless” AR system for surgeons that uses cameras and AI to precisely synchronize the real and virtual worlds. Their method was faster and cheaper, but not quite as accurate. But that’s just early days – in the coming years, this technology will make AR-assisted surgery viable without the need for costly markers.

Additionally, camera-based registration techniques will take AR out of highly controlled environments like operating rooms and bring it into a wider range of medical applications. In fact, by 2030, I expect GPs will be treating patients with the benefits of AR headsets.

Which brings me to another superpower that I expect doctors will have in the near future – the ability to see into the past. That’s because doctors will be able to take 3D images of their patients with AR headsets and later view those images aligned with their patients’ bodies. For example, a physician could quickly assess the healing progress of a skin lesion by examining the patient through AR goggles and interactively flipping back and forth over time to compare the current view with how the lesion appeared on previous visits.

Overall, the advances researchers have made in the medical use of virtual and augmented reality are impressive and exciting, and have significant implications for both medical education and medical practice. To Dr. To quote Stefano Bini of the UCSF Department of Orthopedic Surgery: “The beneficial role of AR and VR in the upskilling of healthcare workers cannot be overstated.”

I agree dr Bini and would go further as I see augmented reality impacting the workforce well beyond healthcare. Eventually, the superpowers of x-ray vision, navigational clues, dexterity assists, and the ability to see into the past will come in handy in everything from construction and auto repair to engineering, manufacturing, farming, and of course, education. And with AR glasses being developed by some of the biggest companies in the world, from Microsoft and Apple to Meta, Google, Magic Leap, HTC and Snap, these superpowers will almost certainly become mainstream within the next five to 10 years. consumers come. Improving all aspects of our daily lives.

Louis Rosenberg, Ph.D is the CEO and chief scientist of Unanimous AI and has received more than 300 patents for his work in VR, AR and AI.