Zum Hauptinhalt springen
TU Graz/ TU Graz/ Services/ News+Stories/

From the Foot Perspective: TU Graz Develops Algorithm for Shoe-Based Blind Assistance System

05/05/2021 | TU Graz news | Research

By Susanne Eigner

The shoe from the company Tec-Innovation warns blind and visually impaired people of obstacles thanks to ultrasonic sensors. Computer scientists at TU Graz have now developed a camera-based AI image recognition system for this purpose.

David Schinagl in front of three shots from the perspective of shoes. The algorithm developed by TU Graz (excerpts of which are on the far right of the screen) recognizes and marks areas that can be walked on without danger. © Lunghammer - TU Graz

A shoe that tells you where to go. The Lower Austrian company Tec-Innovation has developed an intelligent shoe for detecting obstacles. The shoe, known as "InnoMake", has recently been put on the market as an approved medical device and is intended to make the personal mobility of blind and visually impaired people safer. "Ultrasonic sensors on the toe of the shoe detect obstacles up to four meters away. The wearer is then warned by vibration and/or acoustic signals. This works very well and is already a great help to me personally," says Markus Raffer, one of the founders of Tec-Innovation and himself visually impaired.

The team led by Raffer and his founding partner Kevin Pajestka determined early in the development phase that two pieces of advanced information are extremely important to usability: the nature of an obstacle and its directional path, especially if it is downward facing, such as holes or stairs. "Not only is the warning that I am facing an obstacle relevant, but also the information about what kind of obstacle I am facing. Because it makes a big difference whether it's a wall, a car or a staircase," says Raffer.

AI recognizes walkable areas

After an active search, Tec-Innovation was able to win over Graz University of Technology (TU Graz) as a cooperation partner in 2016. Since then, the Institute of Computer Graphics and Vision has been working on a camera-based addition to the first version of the product, as computer scientist Friedrich Fraundorfer explains: "We have developed state-of-the-art deep-learning algorithms modelled on neural networks that can do two main things after detecting and interpreting the content of the image. They use camera images from the foot perspective to determine an area that is free of obstacles and thus safe to walk on. And they can recognize and distinguish objects." The algorithms trained using machine learning can already be operated on a specially designed mobile system. Thanks to the latest powerful special processors, mobile use of the complex AI algorithms is now also possible. "This can be attributed to the enormous processor development of the past few years," adds Fraundorfer's colleague David Schinagl. The TU Graz algorithm is patented and has been transferred to Tec-Innovation.

Merging data into a navigation map

Tec-Innovation is now working on integrating the system into a prototype; the camera plus processor must be robustly and comfortably integrated into the shoe. Friedrich Fraundorfer and his team at TU Graz, in turn, have already turned to the next stage as the logical continuation of the project. They want to combine the information collected while wearing the shoe in terms of swarm knowledge into a kind of street view navigation map for visually impaired people. "As it currently stands, only the wearer benefits in each case from the data the shoe collects as he or she walks. It would be much more sustainable if this data could also be made available to other people as a navigation aid," says Fraundorfer.

For the conception and prototypical implementation of such a street view map for blind and visually impaired persons, a funding application is currently being submitted to the Austrian Research Promotion Agency FFG. Because of the enormous challenges it faces, this type of navigation support is still in the distant future. According to Fraundorfer, the biggest sticking points are the continuously needed updating and expansion of the map, linking it with previous data, and the IT connection of the shoe system. For the researcher at TU Graz, however, one thing is certain: “We will definitely continue to stay on the topic. After all, in our highly innovative world, an alternative to the 70-plus-year-old cane for the blind must also be possible."

This research is anchored in the "Field of Expertise Information, Communication & Computing", one of the five research foci at TU Graz.

Contact

Friedrich FRAUNDORFER
Assoc.Prof. Dipl.-Ing. Dr.techn.
TU Graz | Institute of Computer Graphics and Vision
Phone: +43 316 873 5020
fraundorfernoSpam@icg.tugraz.at

Markus RAFFER
Mag.
Tec-Innovation | CEO
Phone: +43 1 226 44 66
m.raffernoSpam@tec-innovation.com
www.tec-innovation.com

David Schinagl in front of three shots from the perspective of shoes. The algorithm developed by TU Graz (excerpts of which are on the far right of the screen) recognizes and marks areas that can be walked on without danger. © Lunghammer - TU Graz
The Innomake shoe as it is now already available on the market. The ultrasonic sensor is attached to the toe of the shoe. In the future, a camera plus a processor running the algorithm will be integrated there. © Lunghammer - TU Graz
Thanks to the camera-based AI image recognition algorithm penned by Friedrich Fraundorfer and his team at TU Graz, the Innomake shoe can be further developed into a "seeing" shoe. © Fraundorfer
Markus Raffer is managing director of Tec-Innovation. © Lunghammer - TU Graz
A camera shot from the perspective of shoes: the area that can be walked on without danger is delimited by color, recognized and interpreted by the image recognition algorithm of TU Graz. © TU Graz