A new way to ‘see’
SmartSight project unleashes power of on-device AI, edge and cloud computing
Posted on: June 18, 2021; Updated on: June 18, 2021
By Chris Horn, chorn@sc.edu, 803-777-3687
Close your eyes, step outside and listen to the cacophony of sounds — traffic, voices and distant noises. With your eyes closed, safely navigating the surrounding environment would be challenging, to say the least. That’s what life is like 24/7 for the 285 million blind and visually impaired people around the world.
Now imagine being one of them but wearing special glasses with sensors that take in your surroundings and in real time describe them to you. A computerized voice whispers in your ear that a person walking a dog is approaching from behind and a car is about to whiz past on the street. The voice describes the sidewalk stretching out before you and even tells you the name of a co-worker standing nearby. In an instant, you have become more independent because your surroundings, once shrouded in darkness, are illuminated by the power of technology.
Pooyan Jamshidi, an assistant professor of computer science and engineering, is a principal investigator on a three-year $500,000 NSF collaborative grant with Mohsen Amini Salehi at the University of Lousiana Lafayette to develop the intelligence and computing capabilities for such a smart device. Dubbed SmartSight, the platform will enable on-device artificial intelligence to improve real-time perception for blind and visually-impaired users.
“The goal is to develop AI assistance to increase perception for blind and visually impaired people,” says Jamshidi, who joined the College of Engineering and Computing in 2018. “Ideally, they should be able to use the device to ‘see’ what’s going on around them similarly to, or perhaps even better than, people with unimpaired vision.”
The concept might sound simple, but the computing firepower needed to perform such tasks is extraordinary and energy intensive, Jamshidi says.
The goal is to develop AI assistance to increase perception for blind and visually impaired people.
Pooyan Jamshidi
“Algorithmic capabilities for such a device are computationally complex,” says Jamshidi. “Perception models typically rely on deep neural networks, which require large matrix-to-matrix computations every time we run them. For perception tasks we may need to run them thousands of times a minute so they become unusable for on-device deployment.
“One of our goals is to make the real-time, on-device computing more efficient, so it’s not so energy intensive. You don’t want a mobile device that needs recharging all the time.”
To that end, the SmartSight system would have its own built-in computing capability, relying on model compression and quantization to make on-device model inference energy efficient, but also offloading a task to an an accompanying device like a smartphone or an edge (such as exist in stores and other environments) to distribute the computing load and reduce energy consumption.
Still in its first year, the project eventually will progress to a large-scale validation with blind and visually impaired individuals testing the technology for usability, robustness, convenience and discreteness. In the meantime, Jamshidi and Salehi will coordinate their respective research teams, including four Ph.D. students and a postdoctoral fellow, on a rigorous production schedule that will include weekly progress reports.
“We think development of SmartSight will not only help pioneer scientific discovery but also improve the quality of life and social inclusion of blind and visually impaired people,” Jamshidi says. “I have always aspired to align my research toward ‘AI for social good,’ a research that makes a difference for our society and hopefully makes a real impact beyond publications. Even modest progress toward addressing this challenge will bring about remarkable cost savings in health care.”