We begin with multimodal signal processing, architectures, and machine learning. It includes recent deep-learning approaches for processing multisensorial and multimodal user data and interaction, as well as context-sensitivity. A further highlight is processing of information about users' states and traits, an exciting emerging capability in next-generation user interfaces. We discuss real-time multimodal analysis of emotion and social signals from various modalities and perception of affective expression by users. Then we discuss multimodal processing of cognitive state using behavioral and physiological signals to detect cognitive load, domain expertise, deception, and depression. This collection of chapters provides walk-through examples of system design and processing, information on tools and practical resources for developing and evaluating new systems, and terminology, and tutorial support for mastering this rapidly expanding field. Finally, we look at experts' exchange views on the timely and controversial challenge topic of multimodal deep learning. The discussion focuses on how multimodal-multisensor interfaces are most likely to advance human performance during the next decade.

This seminar is most appropriate for graduate students and of primary interest to students studying computer science and information technology, human-computer interfaces, mobile and ubiquitous interfaces, affective and behavioral computing, machine learning, and related multidisciplinary majors.

Central part of the seminar is the reference book "The Handbook of Multimodal-Multisensor Interfaces: Signal Processing, Architectures, and Detection of Emotion and Cognition - Volume 2" (https://dl.acm.org/doi/book/10.1145/3107990). At the beginning there will be an introduction to the subject. Everyone will receive a chapter, for which a presentation (30 min. + 30 min. discussion) and a written elaboration (5-10 pages) are to be prepared.

Contact: Hannes Kath,