SmartKom: Foundations of Multimodal Dialogue Systems

Prof. Dr. Dr. h.c. mult. Wolfgang Wahlster is the Director and CEO of the German Research Center for Artificial Intelligence (DFKI GmbH) and a Professor of Computer Science at the Universitat des Saarlandes, Saarbrucken. In 2000, he was coopted as a Professor of Computational Linguistics at the same university. In addition, he is the Head of the Intelligent User Interfaces Lab at DFKI. He was the Scientific Director of the VERBMOBIL consortium on spontaneous speech translation (1993-2000) as well as the SmartKom consortium on multimodal dialog systems (1999-2003) and currently serves as the Scientific Director of the SmartWeb consortium on mobile multimodal access to semantic web services (2004-2008). He has published more than 150 technical papers and 6 books on language technology and intelligent user interfaces. His current research includes multimodal and perceptive user interfaces, user modeling, embodied conversational agents, smart navigation systems, semantic web services, and resource-adaptive cognitive technologies. He is the editor of the book "Verbmobil: Foundations of Speech-to-Speech Translation" (Springer) and the Co-Editor of the Readings in Intelligent User Interfaces.

• Dialogue Systems Go Multimodal: The SmartKom Experience.- Facts and Figures About the SmartKom Project.- An Exemplary Interaction with SmartKom.- Multimodal Input Analysis.- The SmartKom Architecture: A Framework for Multimodal Dialogue Systems.- Modeling Domain Knowledge: Know-How and Know-What.- Speech Recognition.- Class-Based Language Model Adaptation.- The Dynamic Lexicon.- The Prosody Module.- The Sense of Vision: Gestures and Real Objects.- The Facial Expression Module.- Multiple Biometrics.- Natural Language Understanding.- The Gesture Interpretation Module.- Multimodal Dialogue Processing.- Modality Fusion.- Discourse Modeling.- Overlay: The Basic Operation for Discourse Processing.- In Context: Integrating Domain- and Situation-Specific Knowledge.- Intention Recognition.- Plan-Based Dialogue Management for Multiple Cooperating Applications.- Emotion Analysis and Emotion-Handling Subdialogues.- Problematic, Indirect, Affective, and Other Nonstandard Input Processing.- Multimodal Output Generation.- Realizing Complex User Wishes with a Function Planning Module.- Intelligent Integration of External Data and Services into SmartKom.- Multimodal Fission and Media Design.- Natural Language Generation with Fully Specified Templates.- Multimodal Speech Synthesis.- Scenarios and Applications.- Building Multimodal Dialogue Applications: System Integration in SmartKom.- SmartKom-English: From Robust Recognition to Felicitous Interaction.- SmartKom-Public.- SmartKom-Home: The Interface to Home Entertainment.- SmartKom-Mobile: Intelligent Interaction with a Mobile System.- SmartKom-Mobile Car: User Interaction with Mobile Services in a Car Environment.- Data Collection and Evaluation.- Wizard-of-Oz Recordings.- Annotation of Multimodal Data.- Multimodal Emogram, Data Collection and Presentation.- Empirical Studies for Intuitive Interaction.- Evaluation of Multimodal Dialogue Systems.

From the reviews: "The book is organized into six parts, each of which contains several chapters. ... This book should be useful for many types of readers. Those who want to get a sense of the state of the art of multimodal communication ... will find it invaluable. Students who have had a good grounding in artificial intelligence or natural language processing should read the book to get an overview of how the techniques from these fields can be integrated into a comprehensive system." (J. P. E. Hodgson, ACM Computing Reviews, Vol. 49 (4), April, 2008)