Researchers at Cornell University conducted a significant study on the capabilities of modern multimodal AI models, specifically focusing on their ability to interpret social intelligence. The findings revealed that while AI can effectively predict physical accidents, it struggles to comprehend human emotions and facial expressions.
The primary aim of the study was to equip future domestic and industrial robots with the ability to understand social cues—such as reading gazes, inferring intentions, and anticipating human needs without verbal communication.
To achieve this, the researchers utilized a collection of short, tense video clips featuring scenarios such as:
- A toddler carrying an overflowing cup of hot coffee;
- A man speeding recklessly on a lawnmower;
- A humanoid robot attempting to jump between high blocks.
The team tested six leading models of computer vision and language, including proprietary systems like OpenAI’s GPT-4o and Google Gemini 2.0 Flash, as well as promising open-source alternatives like DeepSeek.
The testing was conducted in two phases:
Initially, the models analyzed the actions depicted in the videos and attempted to predict the outcome of each scene—whether it would end in success or disaster.
In the second phase, the AI was shown videos or photographs of the faces of people watching the same clips and was tasked with predicting the likelihood of an accident based solely on human facial expressions, such as fear, squinting, or smiling.
The results indicated a stark contrast between the AI’s performance in direct video analysis and its ability to interpret human emotions. During the direct analysis, the AI demonstrated exceptional predictive capabilities.
The top-performing open-source model achieved an accuracy rate of 70%, while the leading proprietary system recorded a 63% accuracy rate—both figures aligning with or even surpassing the attentiveness of an average human. The AI effectively grasped the principles of physics and the dangers inherent in the situations presented.
However, when the task shifted to evaluating human expressions, the AI’s accuracy plummeted to a concerning range of 44.5% to 53.5%. Some advanced neural networks began providing identical responses regardless of the emotional state of the observer, whether they appeared shocked or calm.
According to the lead author of the study, Maria Teresa Parreira, who presented the findings at the International Conference on Human-Robot Interaction (HRI 2026) in Edinburgh, these results highlight a significant deficit in social intelligence among contemporary AI systems.
“Robots do not understand the non-verbal signals that humans emit during their interactions with the world,”
Parreira emphasized.
Wendy Ju, a professor at Cornell, noted that the study’s results point to a broader flaw in current engineering practices. Many laboratories have spent years confining robots to controlled environments, striving to achieve an idealized version of them before releasing research findings.
“When they finally test these robots in real-world scenarios, they are often surprised by how different the context is from the lab and how unpredictably humans react,”
Ju remarked. She argued that robots should learn directly in real-world settings, adapting to human emotions and modifying their software through live interactions.
The research team plans to investigate the reasons behind AI’s inability to recognize human facial expressions and aims to adjust the training algorithms used for neural networks.
A study from Cornell University highlights significant limitations in AI's ability to interpret human emotions, revealing that while AI excels in predicting physical events, it struggles with social intelligence. This research underscores the need for robots to learn in real-world contexts to better understand human interactions.