Beyond Language-Only Dialogue

Unless restricted by the situation in which interaction takes place, dialogues among humans naturally involve multiple modes: speech, drawings, facial expressions, pointing and other gestures, etc. Including multiple modes in a dialogue system obviously raises the issues of how to integrate the inputs in interpretation (fusion) and how to distribute and coordinate output in production (fission); cf. for example the SmartKOM project http://smartkom.dfki.de/. When designing a dialogue system, one question is which modes are technically feasible, but it is also a matter of determining which modes would be (most) useful. The latter decision may depend on the on the user(s), on the domain and task(s), and/or on the environment/situation(s) in which the system is used.

Multimodal Interfaces and beyond: Embodied Conversational (Interface) Agents: ``I argue that, since conversation, anthropomorphization, and social interfaces in general are so popular in the interface community, attention needs to be paid to how they are implemented. That is, embodiment needs to be based on an understanding of conversational function, rather than an additive - and ad hoc - model of the relationship between non-verbal modalities and verbal conversational behaviors.'' [Cassell2000]

Reading: [Cassell2000], [Cassell2001]