Gaze as function of instructions - and vice versa


When people hear instructions about what to do next or where to go, they will try to comprehend and ground the instruction by looking at the potential targets or destinations. Thus, the listeners' eye-gaze is a good indicator for what they have understood - which can then inform the speaker/instructor about the success of their instruction. Due to this tight coupling of language comprehension and the listener's eye-gaze, recorded eye-movements during comprehension of (spatial) descriptions serve to reveal how such descriptions are processed incrementally and which ones are more efficient than others (e.g., Eberhard et al., 1998). Such results can contribute, for instance, to improving the efficiency of systems that provide navigational instructions during driving or walking. Further, a speaker or generation system is able to exploit the eye-movements of a listener in order to monitor the listener's understanding and quickly adapt or revise an instruction (see e.g. Clark & Krych, 2004, Koller et al., 2012). Exploring the dynamics of spoken interaction and the role that eye-movements play not only helps to make generation systems (such as navigation systems) potentially more efficient, it also raises the questions whether the listener's (in particular, the driver's) eye-movements can be used pro-actively. That is, can and should an instruction account for the listener's current focus of attention, e.g. "Do not turn left /there/." or "Turn at /that/ corner." ? In this seminar, we will review literature that addresses the different stages of spoken, gaze-augmented interaction with the aim to answer the question on pro-active gaze-based instructions. We will discuss the contributions and limitation of the literature to a more complete understanding of the speaker-listener interplay so far and, if possible, develop ideas for an experimental design that can be used to explore gaze-driven, pro-active instructions empirically.

Literature

• Listener gaze: Attention during comprehension of spatial descriptions


• Chambers et al. 2002. "Circumscribing Referential Domains during Real-Time Language Comprehension"
  Journal of Memory and Language, 47, 30-49. [pdf]


• Eberhard et al. 1995. "Eye Movements as a Window into Real-Time Spoken Language Comprehension in Natural Contexts"
  Journal of Psvcholinguistic Research, Vol. 24, No. 6. [pdf]


• Spivey et al. 2002. "Eye movements and spoken language comprehension: Effects of visual context on syntactic ambiguity resolution"
  Cognitive Psychology 45, 447-481. [pdf]




• Listener gaze in live (face-to-face) instruction-giving


• Clark and Krych 2004. "Speaking while monitoring addressees for understanding"
  Journal of Memory and Language 50, 62-81. [pdf]


• (Alexandr) Hanna and Brennan 2007. "Speakers' eye gaze disambiguates referring expressions early during face-to-face conversation"
  Journal of Memory and Language, 57, 596-615. [pdf]


• Keysar et al. 2000. "Taking Perspective in Conversation: The Role of Mutual Knowledge in Comprehension"
  Psychological Science, 11, 32. [pdf]




• Listener gaze: Understanding and reacting to generated instructions


• (Nikolina) Garoufi and Koller 2010. "Automated planning for situated natural language generation"
  Proceedings of the 48th ACL, Uppsala. [pdf]


• (Nikolina) Koller et al. 2012. "Enhancing referential success by tracking hearer gaze"
  Proceedings of the 13th Annual SIGdial Meeting on Discourse and Dialogue (SIGDIAL), Seoul. [pdf]






• Evaluating generated directions/referring expressions


• Andersen et al. 2012. "Eye tracking, strategies, and sex differences in virtual navigation"
  Neurobiology of Learning and Memory 97 (2012) 81-89. [pdf]


• (short paper) Gargett et al. 2010. "The GIVE-2 Corpus of Giving Instructions in Virtual Environments"
  Proceedings of the 7th International Conference on Language Resources and Evaluation (LREC), Malta. [pdf]


• (short paper) Kiefer et al. 2012. "Towards Location-Aware Mobile Eye Tracking"
  Proceedings of the Symposium on Eye Tracking Research and Applications (ETRA '12), Santa Barbara. [pdf]


• Lovelace et al. 1999. "Elements of Good Route Directions in Familiar and Unfamiliar Environments"
  LNCS Lecture Notes in Computer Science Volume 1661, 1999, pp 65-82 . [pdf]


• Paraboni et al. 2007. "Generating Referring Expressions: Making Referents Easy to Identify"
  Computational Linguistics, 33 (2), 229-254. [pdf]


• Roger et al. 2011. "Landmarks' use in speech map navigation tasks"
  Journal of Environmental Psychology 31, 192-199. [pdf]