Competition and cooperation in integrated visual and linguistic processing
Speaker: Moreno Coco
Institution: University of Edinburgh
Abstract:
Recent studies in psycholinguistics have investigated the relation between language processing and other areas of cognitive processing. A prominent example is work on the connection between language and visual processing utilizing the visual world paradigm. These studies have adopted a linguistic perspective, mostly asking how and to which extend language processing is influenced by visual context. However, to achieve full understanding of the language-vision interaction, we must also ask how visual processing influences language processing.
We present an eye-tracking experiment that addressed this issue by
investigating the interaction (cooperation or competition) between
visual and linguistic processing in ambiguity resolution.
The
materials used were globally ambiguous in both modalities: a sentence
with a PP attachment ambiguity ('the girl will put the orange on the
tray in the bowl') was concurrently presented with a globally
ambiguous visual scene (each constituent or combination of constituent
corresponded to a visual object). Cues for the resolution of ambiguity
were given in both modalities. Visual disambiguation was cued using
low-level visual features (saliency), while linguistic disambiguation
was cued using intonational breaks. The experiment was set up such
that the linguistic and the visual cues were either in cooperation or
in competition.
The results showed that both saliency and intonational breaks were used during integrated visual/linguistic comprehension at different points of processing. Saliency was shown to have anticipatory effect; it enabled the prediction for upcoming verbal arguments. Intonational breaks control the compositional time between constituents; therefore depending on where the break was placed we observed differential matching between partial linguistic structures and visual objects. Moreover, when intonational and saliency cues agreed, we found a cumulative effect. When they competed, there was no clear preference of one modality over the other.
