⭐️ Presented by the Office of Academic Diversity, part of Diversity, Equity and Inclusion

Fernanda Ferreira & Karina Tachihara - Psychology

“Multi-Utterance Language Production as Foraging”

How do speakers plan complex descriptions and then execute those plans? In this work, we attempt to answer this question by asking subjects to describe complex visual scenes while simultaneously monitoring their eye movements. We posit that speakers begin planning by organizing the scene into meaningful clusters or groupings of objects centered around both physical and semantic distance. Speakers then describe the scene cluster by cluster, exploiting the content available at one cluster before moving on to another. To test these ideas, in a preregistered study 30 participants described 30 indoor and outdoor scenes while their eye movements and speech were recorded. Physical distance was calculated by identifying the centroid point of each object and then computing the Euclidean distance between centroid points for every object pair. Semantic distance was calculated using ConceptNet Numberbatch to obtain the semantic similarity between object labels. A clustering algorithm was then applied to establish the appropriate number of clusters per scene and to assign objects to each physical or semantic cluster. We observed that, consistent with our hypothesis, objects separated by shorter physical distances and objects that are semantically more similar were discussed in closer temporal proximity in the verbal descriptions. In addition, word productions that involved jumping from one predefined cluster to another took longer to initiate than those associated with the same cluster. We conclude that speakers solve the linearization problem by establishing physical and semantic clusters and exploiting the information contained at each one. This approach treats multi-utterance language production as a type of foraging behavior, where people balance exploration and exploitation by locating an object cluster, exploiting the contents of that cluster, and exploring the environment for new clusters once the current one is depleted.