How Microsoft HoloLens helps the blind navigate complex buildings

Researchers have discovered how the Microsoft HoloLens can be used to help blind people navigate their way through complex buildings.

The augmented reality headset's primary use is to supplement what a sighted wearer can already see with a virtual interface, adding digital objects to real walls in a way which blends the real world with computer software.

To do this, the HoloLens uses cameras and an onboard computer to understand the world around it, recognize objects, and build a digital map of its surroundings. But now, researchers from the California Institute of Technology have used this to help direct the visually impaired through complex buildings.

The headset asks wearers to "follow me", and because HoloLens uses spatial audio — where sounds can be made to come from a specific direction and distance — the voice can be placed exactly where the person should be walking. The HoloLens then recognizes corners in corridors, and tells the wearer when they are approaching a staircase. It even tells them where the handrail is, and if there is one available on both sides of the stairs, by using object recognition.

A video published by the researchers shows how a blind man with a cane was able to navigate a complicated building, walking along corridors, turning corners, climbing stairs and arriving at the right room, all thanks to directions delivered by the HoloLens.

The man was one of seven visually impaired people to test the system, all of whom completed the route on their first attempt, although one took a wrong turn before finding their way. The test also found users were able to navigate more quickly with the HoloLens than by just using their walking cane.

When asked to locate a chair in a room, test subjects took eight times as long when using only their walking aid, compared to when also using the HoloLens. They also covered 13 times as much distance when not using the HoloLens. When the test was repeated with obstacles blocking their path, the HoloLens helped them navigate to the chair without any collisions.

"That was pretty cool," one tester said on reaching his destination.

As well as precise directional instructions, the system also assigns certain noises to objects. For example, walls are made to hiss with white noise when the person gets too close. The wearer can also scan the environment around them, with the headset saying what is in front of them at each turn. A single object, like a chair, can be selected and will repeatedly call out to help the wearer find their way to it, the voice raising in pitch the closer the wearer gets to the object.

The systems proved to be accurate, as when they were asked to point their head towards an object, the test subjects scored an accuracy of between three and 12 degrees. "These results show that users can accurately localize the virtual voices generated by HoloLens," the researchers - Yang Liu, Noelle Stiles and Markus Meister - said in a report of their work.

It was also found that subjects were able to accurately remember the location of objects identified audibly by the HoloLens. The researchers said the wearers' ability to remember the position of objects in a room using their ears was similar to that of sighted people using their eyes.

The researchers behind this project say HoloLens headsets could be handed out at the front desk of a hotel or shopping mall to help blind or visually impaired visitors find their room or locate a specific store. (If interested in buying Microsoft Hololens, click here.)