A platform for computer vision accessibility technology – Google Research Blog

Two years in the past we introduced Undertaking Guideline, a collaboration between Google Analysis and Guiding Eyes for the Blind that enabled folks with visible impairments (e.g., blindness and low-vision) to stroll, jog, and run independently. Utilizing solely a Google Pixel telephone and headphones, Undertaking Guideline leverages on-device machine studying (ML) to navigate customers alongside outside paths marked with a painted line. The expertise has been examined everywhere in the world and even demonstrated in the course of the opening ceremony on the Tokyo 2020 Paralympic Video games.

For the reason that authentic announcement, we got down to enhance Undertaking Guideline by embedding new options, comparable to impediment detection and superior path planning, to soundly and reliably navigate customers via extra complicated eventualities (comparable to sharp turns and close by pedestrians). The early model featured a easy frame-by-frame picture segmentation that detected the place of the trail line relative to the picture body. This was adequate for orienting the consumer to the road, however supplied restricted details about the encircling atmosphere. Bettering the navigation indicators, comparable to alerts for obstacles and upcoming turns, required a significantly better understanding and mapping of the customers’ atmosphere. To unravel these challenges, we constructed a platform that may be utilized for a wide range of spatially-aware purposes within the accessibility house and past.

At present, we announce the open supply launch of Undertaking Guideline, making it obtainable for anybody to make use of to enhance upon and construct new accessibility experiences. The discharge contains supply code for the core platform, an Android utility, pre-trained ML fashions, and a 3D simulation framework.

System design

The first use-case is an Android utility, nevertheless we wished to have the ability to run, check, and debug the core logic in a wide range of environments in a reproducible manner. This led us to design and construct the system utilizing C++ for shut integration with MediaPipe and different core libraries, whereas nonetheless having the ability to combine with Android utilizing the Android NDK.

Underneath the hood, Undertaking Guideline makes use of ARCore to estimate the place and orientation of the consumer as they navigate the course. A segmentation mannequin, constructed on the DeepLabV3+ framework, processes every digicam body to generate a binary masks of the rule of thumb (see the earlier weblog submit for extra particulars). Factors on the segmented guideline are then projected from image-space coordinates onto a world-space floor airplane utilizing the digicam pose and lens parameters (intrinsics) supplied by ARCore. Since every body contributes a unique view of the road, the world-space factors are aggregated over a number of frames to construct a digital mapping of the real-world guideline. The system performs piecewise curve approximation of the rule of thumb world-space coordinates to construct a spatio-temporally constant trajectory. This permits refinement of the estimated line because the consumer progresses alongside the trail.

Undertaking Guideline builds a 2D map of the rule of thumb, aggregating detected factors in every body (crimson) to construct a stateful illustration (blue) because the runner progresses alongside the trail.

A management system dynamically selects a goal level on the road a long way forward based mostly on the consumer’s present place, velocity, and course. An audio suggestions sign is then given to the consumer to regulate their heading to coincide with the upcoming line phase. By utilizing the runner’s velocity vector as an alternative of digicam orientation to compute the navigation sign, we eradicate noise attributable to irregular digicam actions widespread throughout working. We will even navigate the consumer again to the road whereas it’s out of digicam view, for instance if the consumer overshot a flip. That is doable as a result of ARCore continues to trace the pose of the digicam, which may be in comparison with the stateful line map inferred from earlier digicam photos.

Undertaking Guideline additionally contains impediment detection and avoidance options. An ML mannequin is used to estimate depth from single photos. To coach this monocular depth mannequin, we used SANPO, a big dataset of outside imagery from city, park, and suburban environments that was curated in-house. The mannequin is able to detecting the depth of assorted obstacles, together with folks, automobiles, posts, and extra. The depth maps are transformed into 3D level clouds, just like the road segmentation course of, and used to detect the presence of obstacles alongside the consumer’s path after which alert the consumer via an audio sign.

Utilizing a monocular depth ML mannequin, Undertaking Guideline constructs a 3D level cloud of the atmosphere to detect and alert the consumer of potential obstacles alongside the trail.

A low-latency audio system based mostly on the AAudio API was applied to offer the navigational sounds and cues to the consumer. A number of sound packs can be found in Undertaking Guideline, together with a spatial sound implementation utilizing the Resonance Audio API. The sound packs had been developed by a crew of sound researchers and engineers at Google who designed and examined many various sound fashions. The sounds use a mixture of panning, pitch, and spatialization to information the consumer alongside the road. For instance, a consumer veering to the proper might hear a beeping sound within the left ear to point the road is to the left, with rising frequency for a bigger course correction. If the consumer veers additional, a high-pitched warning sound could also be heard to point the sting of the trail is approaching. As well as, a transparent “cease” audio cue is all the time obtainable within the occasion the consumer veers too removed from the road, an anomaly is detected, or the system fails to offer a navigational sign.

Undertaking Guideline has been constructed particularly for Google Pixel telephones with the Google Tensor chip. The Google Tensor chip permits the optimized ML fashions to run on-device with increased efficiency and decrease energy consumption. That is important for offering real-time navigation directions to the consumer with minimal delay. On a Pixel 8 there’s a 28x latency enchancment when working the depth mannequin on the Tensor Processing Unit (TPU) as an alternative of CPU, and 9x enchancment in comparison with GPU.

Testing and simulation

Undertaking Guideline features a simulator that permits speedy testing and prototyping of the system in a digital atmosphere. Every thing from the ML fashions to the audio suggestions system runs natively throughout the simulator, giving the total Undertaking Guideline expertise without having all of the {hardware} and bodily atmosphere arrange.

Screenshot of Undertaking Guideline simulator.

Future course

To launch the expertise ahead, WearWorks has change into an early adopter and teamed up with Undertaking Guideline to combine their patented haptic navigation expertise, using haptic suggestions along with sound to information runners. WearWorks has been creating haptics for over 8 years, and beforehand empowered the primary blind marathon runner to finish the NYC Marathon with out sighted help. We hope that integrations like these will result in new improvements and make the world a extra accessible place.

The Undertaking Guideline crew can also be working in the direction of eradicating the painted line utterly, utilizing the newest developments in cellular ML expertise, such because the ARCore Scene Semantics API, which might establish sidewalks, buildings, and different objects in outside scenes. We invite the accessibility group to construct upon and enhance this expertise whereas exploring new use instances in different fields.

Acknowledgements

Many individuals had been concerned within the improvement of Undertaking Guideline and the applied sciences behind it. We’d prefer to thank Undertaking Guideline crew members: Dror Avalon, Phil Bayer, Ryan Burke, Lori Dooley, Music Chun Fan, Matt Corridor, Amélie Jean-aimée, Dave Hawkey, Amit Pitaru, Alvin Shi, Mikhail Sirotenko, Sagar Waghmare, John Watkinson, Kimberly Wilber, Matthew Willson, Xuan Yang, Mark Zarich, Steven Clark, Jim Coursey, Josh Ellis, Tom Hoddes, Dick Lyon, Chris Mitchell, Satoru Arao, Yoojin Chung, Joe Fry, Kazuto Furuichi, Ikumi Kobayashi, Kathy Maruyama, Minh Nguyen, Alto Okamura, Yosuke Suzuki, and Bryan Tanaka. Because of ARCore contributors: Ryan DuToit, Abhishek Kar, and Eric Turner. Because of Alec Go, Jing Li, Liviu Panait, Stefano Pellegrini, Abdullah Rashwan, Lu Wang, Qifei Wang, and Fan Yang for offering ML platform assist. We’d additionally prefer to thank Hartwig Adam, Tomas Izo, Rahul Sukthankar, Blaise Aguera y Arcas, and Huisheng Wang for his or her management assist. Particular due to our companions Guiding Eyes for the Blind and Achilles Worldwide.