Close
The ECOLANG Multimodal Corpus enables a new approach to the study of language in its core ecological niche: face-to-face, multimodal environments.
Context
To account for the human capacity for language, we must study language in the settings that represent the conditions in which language has evolved, is learned and is commonly used. Here, communication comprises a wealth of multimodal signals – such as gestures, eyegaze, and intonation – alongside speech. However, most of the research concerning language processing, learning, and the neurobiology of language has focused on speech or text only. The ECOLANG Multimodal Corpus of adult-adult and adult-child conversation enables a new approach to the study of language in its core ecological niche: face-to-face, multimodal environments.
Gabriella Vigliocco presents Ecological Language: A Multimodal Approach to Language Learning and Processing (Abralin ao Vivo, 2021) with Susan Goldin-Meadow.
Corpus
The corpus provides audiovisual recordings and annotation of multimodal behaviours (speech transcription, gesture, object manipulation, and eye gaze) by British and American English-speaking adults engaged in semi-naturalistic conversation with their child (N = 38, children 3-4 years old) or a familiar adult (N = 32). Speakers were asked to talk about objects (familiar or unfamiliar) to their interlocutors both when the objects were physically present or absent. Thus, the corpus characterises the use of multimodal signals in social interaction and their modulations depending upon the age of the interlocutor (child or adult); whether the interlocutor is learning new concepts/words (unfamiliar or familiar objects) and whether they can see and manipulate (present or absent) the objects
Application
The corpus provides ecologically-valid data about the distribution and cooccurrence of the multimodal signals for cognitive scientists and neuroscientists to address questions about real-world language learning and processing; and for computer scientists to develop more human-like artificial agents.
Ecological Language: A multimodal approach to language and the brain is funded by the European Research Council (grant ID 743035) from January 2018 to June 2024. The project is led by the Language and Cognition Lab (PI: Gabriella Vigliocco) with the Language, Action and Brain Lab (PI: Jeremy Skipper), at University College London, UK.
