Human Similarity Judgments Reveal Fine-Grained Representations of Relationships and Actions in Dynamic Social Scenes

00Overview

How does the mind organize dynamic social scenes? Instead of assuming a taxonomy, we learned one from behavior. The resulting dimensions explain twice as much variance as theory-based annotations, combine relationships with actions (who is doing what with whom), and predict visual cortex — including the lateral pathway and STS — better than those annotations.

Existing accounts organize social scenes along pre-defined dimensions: social-affective axes like warmth, dominance, valence, and arousal, or action categories and action-related features. Both specify the candidate dimensions before measuring the representational space. We instead collected odd-one-out similarity judgments on 250 naturalistic two-person videos and trained a sparse positive similarity embedding (SPoSE) to recover the latent dimensions directly from behavior.

01Approach

Try a trial yourself — this is what participants saw:

How the model learns Methods figure

Each video starts as a point in a 90-dimensional non-negative space.

For a triplet (i, j, k), each pair's similarity is the dot product of its vectors; a softmax over the three gives the probability each pair is “most similar.”

The model minimizes the negative log-likelihood of human choices plus an L1 sparsity penalty. After training, near-empty dimensions are pruned, leaving 40.

02Learned dimensions explain twice the variance

All 13 pre-labeled annotations — scene, body configuration, action and interaction, and social-affective features — together explained only r² = 0.25 of held-out human similarity (split-half reliability 0.67). Individually, only dominance, spatial expanse, and intimacy were significant.

Variance partitioning

Combining both sets shows the annotations add almost nothing beyond the learned dimensions, while the learned dimensions carry substantial unique variance.

03Explore the dimensions

A greedy forward-selection curve has an elbow at 11 dimensions, which carry 79% of the full model's predictive power (r² ≈ 0.41). Each is stable across retraining (mean reliability 0.60–0.88). Labels come from the top videos and a contrastive caption analysis: words that appear in descriptions of high- but not low-loading videos. Pick a dimension:

Word colors: noun verb adjective adverb. JS = Jensen–Shannon divergence between high- and low-loading caption vocabularies (chance 0.61 bits).

04Broad categories split into distinct social contexts

Related content is often fractionated rather than lumped together. Infant scenes separate into caregiving routines, crying, and affectionate play; “play” itself splits by who is playing — parent–child versus child–child — even though the top verb for both is “play.” Adult scenes separate into conversation, cooking and eating, sport, and more.

The annotations describe these scenes almost identically, yet the embedding learned from people's judgments treats them as different situations.

05No dimension maps onto a single annotation

Correlations between the top 11 dimensions and the 13 pre-labeled annotations. Hover a cell for details; click a dimension to focus on it. Dimensions that correlate with annotations usually span several families (scene, body, action, social-affective), and three (7, 17, 8) match none.

06The same dimensions predict visual cortex

In fMRI from four people who passively watched the same 250 videos — never making similarity judgments — the learned dimensions explained more variance than the annotations across occipitotemporal cortex in every participant. Effects were strongest on the lateral surface (overlapping EBA and pSTS) and in ventral temporal cortex. The advantage held, reduced, even with only the 11 core dimensions — fewer predictors than the annotation model.

Whole-brain searchlight Lateral-stream ROIs Which dimension wins where

07Robustness

Choosing 11 dimensions Recoverability

08Takeaways

09Citation

@misc{garcia2026finegrained,
  title     = {Human Similarity Judgments Reveal Fine-Grained Representations
               of Relationships and Actions in Dynamic Social Scenes},
  author    = {Garcia, Kathy and Isik, Leyla},
  publisher = {PsyArXiv},
  note      = {Preprint},
  year      = {2026}
}