Conference on Cognitive Computational Neuroscience

Humans possess the remarkable ability to detect and identify social interactions in visual scenes, but the computations that underlie this ability remain largely unknown. Some argue that humans make these judgements by inverting generative models of the physical and social world. However, evidence from recent behavioral, neural, and computational work suggest that bottom-up visual processes alone are sufficient to capture human behavior. To compare these two alternatives, we compare our recent bottom-up visual graph-neural-network model (SocialGNN) and a generative inverse planning model (SIMPLE) with human ratings of social interactions in animated videos resembling real-life social scenes, using representational similarity analysis. Both models were significantly correlated with human judgments. Further, each model uniquely explained a significant amount of variance in human judgments suggesting that humans recognize social interactions using both bottom-up and generative processes. Preliminary investigations show that SocialGNN matched human judgments better on "friendly" and "neutral" videos, whereas SIMPLE performed better on videos depicting "adversarial" relationships or "chasing". Together these results suggest humans employ both purely visual computations and generative inverse inference, and may switch between strategies depending on stimulus characteristics. Future work will investigate ways these two model classes can be combined to match behavior and neural data.