Look, Then Speak: Social Tokens for Grounding LLMs in Visual Interactions Kathy Garcia, Vighnesh Subramaniam, Boris Katz, Brian Cheung
Accepted at UniReps Workshop, NeurIPS 2025

PDF Project Page BibTeX Discussion

We introduce social tokens: visual features extracted from video frames & strategically inserted into language models to enhance their comprehension of social interactions. This method significantly improves LLM performance on social tasks by enabling richer integration of social cues. Our results demonstrate that social tokens help bridge the gap between human and model understanding of complex social dynamics.

Aligning Video Models with Human Social Judgments via Behavior-Guided Fine-Tuning Kathy Garcia, Leyla Isik
arXiv preprint, 2025

PDF arXiv Project Page

We introduce behavior-guided fine-tuning to align video models with human social perception. Using 49,000+ human similarity judgments, we fine-tune video models via a hybrid triplet-RSA loss, significantly improving alignment with how humans perceive social interactions while maintaining performance on downstream tasks.

A Large-Scale Study of Social Scene Judgments: Alignment with Deep Neural Networks and Social-Affective Features Kathy Garcia, Leyla Isik
Vision Sciences Society (VSS), 2025 (Talk Presentation)

DOI Project Page

We benchmark how closely deep neural networks capture the social understanding reflected in human judgments and brain responses to real-world interactions. Our dataset and framework reveal that while AI models are closing the gap, humans still rely on uniquely social cues to interpret complex social scenes.

Modeling Dynamic Social Vision Highlights Gaps Between Deep Learning and Humans Kathy Garcia, Emalie McMahon, Colin Conwell, Michael F. Bonner, Leyla Isik
International Conference on Learning Representations (ICLR), 2025

PDF Poster Project Page Code & Data BibTeX

We present a dataset of natural videos and captions involving complex multi-agent interactions, and we benchmark 350+ image, video, and language models on behavioral and neural responses to the videos. Together these results identify a major gap in AI's ability to match the human brain and behavior and highlight the importance of studying vision in dynamic, natural contexts.

Large-scale Deep Neural Network Benchmarking in Dynamic Social Vision Kathy Garcia, Colin Conwell, Emalie McMahon, Michael F. Bonner, Leyla Isik
Vision Sciences Society (VSS), 2024 • Cognitive Computational Neuroscience (CCN), 2024

DOI Project Page

Large-scale benchmarking of 300+ DNNs with diverse architectures, objectives, and training sets, against fMRI responses to a curated dataset of 200 naturalistic social videos, with a focus on the "lateral" visual stream.

Predicting Dimensional Symptoms of Psychopathology from Task-Based fMRI using Support Vector Regression Kathy Garcia, Zach Anderson,  Iris Ka-Yi Chat,  Katherine S.F. Damme,  Katherine Young, Susan Y. Bookheimer,  Richard Zinbarg,  Michelle Craske,  Robin Nusslock 
SfN Global Connectome, 2021 (Virtual poster presentation)

This study develops a novel machine learning approach using Support Vector Regression (SVR) to explore potential biomarkers in fMRI data for symptoms of anxiety and depression, finding that MID task-fMRI data does not accurately predict these symptoms, with results indicating a poor model fit.