Researchers from the United States, in collaboration with engineers at Meta, have developed an algorithm that enables smart glasses to anticipate human attention trajectories in a three-dimensional environment. This innovative technology can predict where a user will focus their attention several seconds in advance.
The new system represents a shift from analyzing two-dimensional static images to a comprehensive modeling of human behavior in real-world settings.
The study was led by Fiona Ryan, a graduate student at Georgia Tech’s School of Interactive Computing. She created the first 3D platform for predicting so-called “scan paths” (the trajectories of eye movement) based on first-person video footage.
Ryan explains, “Since humans operate in a three-dimensional world and are constantly in motion, standard 2D metrics for image analysis are not effective for portable devices like smart glasses.”
The algorithm calculates attention vectors as a sequence of gaze fixations that directly depend on the user’s current goal. For instance, if the system detects a hand moving toward a coffee cup, it automatically predicts the next step—searching for a location to place the cup.
Visualization of the algorithm’s operation (screenshot: Techxsplore)
The majority of the practical work was conducted during Ryan’s internship at Meta.
To train the artificial intelligence, a specialized dataset called Aria Digital Twin was utilized. This dataset comprises thousands of hours of first-person video recordings capturing everyday interactions between people and objects within home settings, combined with high-precision 3D reconstructions of the entire environment.
This approach allowed developers to obtain precise coordinates of actual gaze directions and correlate them with the geometry of the space.
Currently, the software can reliably predict gaze direction on average three seconds ahead, with some simpler scenarios achieving predictions of up to ten seconds.
This timeframe is sufficient for the graphics processor in augmented reality glasses to proactively generate (render) necessary virtual cues or interface elements in the area where the user is about to look.
Ryan notes, “This completely eliminates the effect of image lag.”
Looking ahead, the developers plan to integrate contextual scenarios into the model, which will help refine predictions during prolonged focus on a single object.
Beyond consumer electronics and smart glasses, the technology holds significant potential in robotics: it could be used to train algorithms in robots to mimic natural human perception while performing household or industrial tasks.
A new algorithm developed by researchers in collaboration with Meta enhances smart glasses by predicting user attention in 3D environments. This advancement could improve user experience in various applications, including robotics.