Abstract
Background: Gesture and speech are theorized to form a sin-
gle integrated system of meaning during language produc-
tion (McNeill, 1992), and evidence is mounting that this in-
tegration applies to language comprehension as well (Kelly,
Ozyurek & Maris, 2010). However, it is unknown whether
gesture is uniquely integrated with speech or is processed like
any other manual action. To explore this issue, we compared
the extent to which speech is integrated with hand gestures
versus actual actions on objects during comprehension.
Method: The present study employed a priming paradigm
in two experiments. In Experiment 1, subjects watched multi-
modal videos that presented auditory (words) and visual (ges-
tures and actions on objects) information. Half the subjects
related the audio information to a written prime presented be-
fore the video, and the other half related the visual informa-
tion to the written prime. For half of the multimodal video
stimuli, the audio and visual information was congruent, and
for the other half, incongruent. The task was to press one but-
ton if the written prime was the same as the visual (31 sub-
jects) or audio (31 subjects) information in the target video or
another button if different. RT and accuracy were recorded.
Results: In Experiment 2, we reversed the priming se-
quence with a different set of 18 subjects. Now the video
became the prime and the written verb followed as the target,
but the task was the same with one differenceXto indicate
whether the written target was related or unrelated to only the
audio information (speech) in preceding video prime. ERPs
were recorded to the written targets.
In Experiment 1, subjects in both the audio and visual tar-
get tasks were less accurate when processing stimuli in which
gestures and actions were incongruent versus congruent with
speech, F(1, 60) = 22.90, p
<
.001, but this effect was less
prominent for speech-action than for speech-gesture stimuli.
However, subjects were more accurate when identifying ac-
tions versus gestures, F(1, 60) = 8.03, p = .006.
In Experiment 2, there were two early ERP effects. When
primed with gesture, incongruent primes produced a larger
P1, t (17) = 3.75, p = 0.002, and P2, t (17) = 3.02, p =
0.008, to the target words than the congruent condition in the
grand-averaged ERPs (reflecting early perceptual and atten-
tional processes). However, there were no significant differ-
ences between congruent and incongruent conditions when
primed with action.
Discussion: The incongruency effect replicates and ex-
tends previous work by Kelly et al. (2010) by showing not
only a bi-directional influence of gesture and speech, but also
of action and speech. In addition, the results show that while
actions are easier to process than gestures (Exp. 1), gestures
may be more tightly tied to the processing of accompanying
speech (Exps. 1 & 2). These results suggest that even though
gestures are perceptually less informative than actions, they
may be treated as communicatively more informative in rela-
tion to the accompanying speech. In this way, the two types
of visual information might have different status in language
comprehension.
