HCI International 2011

T09: Multimodal Information - Collection and Analysis of Interactive Data

Full Day Tutorial

Kristiina Jokinen (short bio)
University of Tampere and University of Helsinki, Finland

Objectives:

The tutorial aims at providing a state-of-the-art overview in the issues related to the collection and analysis of conversational interactive data. The goal is to study methods and challenges in the collection of multimodal corpora, and to review tools and techniques for the analysis and for the automatic processing of data, especially from the point of view of dialogue activity and natural communication. The full day tutorial will also aim at deepening the participants' skills in the analysis by allowing the participants to analyse and experiment with the data themselves.

Content:

The tutorial deals with theoretical and practical aspects related to corpus collection and annotation. It focuses on human conversational interactions, and studies communication that goes above and beyond the propositional information contained in the speech. The discussion concerns especially such topics as the function and use of gestures, facial expressions, and eye-gaze as conversational means, as well as their alignment with spoken interaction. The tutorial is grounded in research that applies a special multilevel approach to modelling of feedback, turn-taking, and social affect that these modalities signal in interaction management: the top-down human analysis of interactive situations is combined with the bottom-up signal analysis of the multimodal activities which the participants are engaged in. The tutorial also deals with the modelling of these issues in computational contexts, and discusses technological challenges in deploying research results in interactive applications, such as appropriate annotation levels, and recent standardisation and infrastructure efforts.

The tutorial will consist of lectures and group discussions related to the core topics. There will also be real data analysis of the selected data so as to provide hands-on exercises and practise on the analysis and annotation of the data. The participants are also encouraged to contribute to discussion sessions with their own experience.

Target audience:

The tutorial will be relevant to researchers and students working on multimodal communication and on interaction technology. It will provide a compact and easy-to-digest overview of the emerging fields of intelligent interaction informatics, and it will also serve as an update with respect to current practices and methodologies in data collection, analysis, and application.

Brief Biographical Sketch:

Kristiina Jokinen is Adjunct Professor of Interaction Technology at the University of Tampere, and her research focuses on nonverbal communication, spoken dialogue modelling, and multimodal interaction (esp. gestures and eye-gaze). She also holds a post of Adjunct Professor at the University of Helsinki where she leads the Nordic collaboration project NOMCO (Multimodal Corpus Analysis in the Nordic Countries). She currently is also Visiting Professor at University of Tartu, Estonia, where she has given regular courses and initiated a video corpus collection of conversational Estonian. In 2009-2010 she was the NICT Invited Scholar at the Doshisha University in Kyoto, Japan, and was in charge of eye-gaze corpus collection on conversational data. She has played a leading role in several academic and industrial research projects concerning natural language interaction, spoken dialogue systems, and multimodality, and she is a member of the Management Committee for the EU COST Action 2102 Cross-Modal Analysis of Verbal and Non-verbal Communication. Her book "Constructive Dialogue Modelling - Speech Interaction and Rational Agents" was recently published by John Wiley & Sons. She is the secretary of SIGDial, the ACL/ISCA Special Interest Group for Discourse and Dialogue.