Human language understanding in visual contexts

Seminar: Computerlinguistik, B.Sc./M.Sc.

When understanding language in a particular situation, people are often provided with relevant visual information from the immediate scene, in addition to the linguistic input. Such visual information can be exploited to ground linguistic references, resolve ambiguities or to form expectations about the next thing to be mentioned. In this seminar, we will review the core findings from visually situated language processing that has emerged from psycholinguistic eye tracking studies. Our focus will be on how we mentally represent the visual, spatial and linguistic information and how different sources of information interact. In addition to the experimental side, we will also consider several computational models of visually situated language processing.

timetable (temporary version)

References

Huettig, F., Rommers, J., and A. Meyer (2011). Using the visual world paradigm to study language processing: A review and critical evaluation. Acta Psychologica.

Altmann, G. T., and Y. Kamide (1999). Incremental interpretation at verbs: restricting the domain of subsequent reference. Cognition, 73(3):247-264.

Altmann, G. T., and Kamide, Y. (2007). The real-time mediation of visual attention by language and world knowledge: Linking anticipatory (and other) eye movements to linguistic processing. Journal of Memory and Language, 57, 502−518.

Altmann, G. T. and Y. Kamide (2009). Discourse-mediation of the mapping between language and the visual world: Eye-movements and mental representation. Cognition, 111, 55-71.

Knoeferle, P. and M. W. Crocker (2006). The coordinated interplay of scene, utterance, and world knowledge : Evidence from eye tracking. Cognitive Science, 30, 481-529.

Mayberry, M., M.W. Crocker and P. Knoeferle (2009). Learning to attend: A connectionist model of the coordinated interplay of utterance, visual context, and world knowledge. Cognitive Science, 33, 449-496.

Tanenhaus, M. K., Magnuson, J. S., Dahan, D., and Chambers, C. (2000). Eye movements and lexical access in spoken-language comprehension: evaluating a linking hypothesis between fixations and linguistic processing. Journal of Psycholinguistic Research, 29, 557−580.

Tanenhaus, M.K., Spivey-Knowlton, M.J., Eberhard, K. M., and Sedivy, J. C. (1995). Integration of visual and linguistic information in spoken language comprehension. Science, 268, 632-634.