Situated Spoken Language Comprehension
Speaker: Matthew Crocker
Institution:
Saarland University
Abstract:
There is increasing evidence that human language comprehension is highly adapted both to long-term experience and to immediate linguistic and non-linguistic context. In this talk, I will present a collection of recent experiments focusing on adaptation to immediate context in situated language comprehension. Using the "visual world" paradigm, which monitors eye movements in scenes during spoken comprehension, we have demonstrated the ability of the human language processor to rapidly adapt to and exploit diverse linguistic cues, prosody, as well as information from the immediate scene. These findings have led to the proposal of the "coordinated interplay account" of scene and utterance processing (Knoeferle and Crocker, Cognitive Science, 2006), which argues for the rapid use, and even priority, of scene information during spoken comprehension.
In order to model the rapid use of both experience and immediate context, we have developed a family of connectionist models based on Elman's Simple Recurrent Networks. These networks have been modified to include inputs not only for the utterance, but also for the current (visual) context, and have been trained to model the findings of five experiments. In addition to the general ability of the networks to successfully learn the task, they also model on-line behavior: (a) anticipation of role and filler, (b) frequency biases derived from training, (c) early influence of depicted events, (d) delayed disambiguation without the scene, (e) relative priority of scene information and (f) attention in the visual scene (Mayberry, Crocker & Knoeferle, 2006). In conclusion, I argue that the mechanisms that underlie language comprehension are fundamentally adaptive in nature: optimized to recover the most likely interpretation based on relevant linguistic and non-linguistic information sources.