de.mpg.escidoc.pubman.appbase.FacesBean
English
 
Help Guide Disclaimer Contact us Login
  Advanced SearchBrowse

Item

ITEM ACTIONSEXPORT

Released

Conference Paper

Referring to Objects with Spoken and Haptic Modalities

MPS-Authors
http://pubman.mpdl.mpg.de/cone/persons/resource/persons96341

Romary,  Laurent
Max Planck Digital Library, Max Planck Society;

Locator
There are no locators available
Fulltext (public)
Supplementary Material (public)
There is no public supplementary material available
Citation

Landragin, F., Bellalem, N., & Romary, L. (2002). Referring to Objects with Spoken and Haptic Modalities. In Fourth IEEE International Conference on Multimodal Interfaces (pp. 99-104). Los Alamitos, Calif.: IEEE Computer Society.


Cite as: http://hdl.handle.net/11858/00-001M-0000-0013-8777-3
Abstract
The gesture input modality considered in multimodal dialogue systems is mainly reduced to pointing or manipulating actions. With an approach based on the spontaneous character of the communication, the treatment of such actions involves many processes. Without any constraints, the user may use gesture in association with speech, and may exploit the visual context peculiarities, guiding his articulation of gesture trajectories and his choices of words. The semantic interpretation of multimodal utterances also becomes a complex problem, taking into account varieties of referring expressions, varieties of gestural trajectories, structural parameters from the visual context, and also directives from a specific task. Following the spontaneous approach, we propose to give the maximal understanding capabilities to dialogue systems, to ensure that various interaction modes must be taken into account. Considering the development of haptic sense devices (as PHANToM) which increase the capabilities of sensations, particularly tactile and kinesthetic ones, we propose to explore a new domain of research concerning the integration of haptic gesture into multimodal dialogue systems, in terms of its possible associations with speech for objects reference and manipulation. We focus in this paper on the compatibility between haptic gesture and multimodal reference models, and on the consequences of processing this new modality on intelligent system architectures, which is not yet enough studied from a semantic point of view.