Institute for Psychoacoustics and Electronic Music (IPEM)

MEGA: Multisensory Expressive Gesture Applications

Project Description:

The project is centered on the modeling and real-time analysis, synthesis, and networked communication of expressive and emotional content in non-verbal interaction by multi-sensory interfaces, from a multimodal perspective. Music (including singing voice) and movement (including dance) are first class channels for conveying expressive and emotional content.

Many research projects are currently focused on developing applications in mixed reality (MR), virtual environments (VEs), and inhabited information spaces (IISs). In these environments the extraction and synthesis of expressive content are of paramount importance for enhancing usability and communication in computer multimedia where inputs and outputs can be represented by music signals, visual media, or robot movement in robot-human interaction. This project is centered on the modeling and communication of expressive and emotional content in non-verbal interaction by multi-sensory interfaces. In particular the project focuses on music performance and full-body movements as first class conveyors of expressive and emotional content. Real-time, quantitative analysis and evaluation of expressive content in different performances of the same musical score, or in different performances of the same dance fragment are examples that will result from the project. Our approach implies a new and original consideration of the role that the physical body and the physical world play with respect to interaction. Such interaction mechanisms are not limited to single humans, but they may be extended to groups of humans, such as a group of actors on stage or a group of visitors in a museum or a group of music performers, considering the group as a whole entity.

Project objectives

  1. Design of novel paradigms of interaction, mapping strategies, and multimedia interfaces grounded on the real-time analysis and synthesis of expressive content in music, gesture, and visual languages, and their multimodal integration
  2. Implementation of prototype modules developed into real-time working modules
  3. Development of products for end-users by incorporating prototype modules into a flexible working environment enabling specific applications and services.

International website

http://www.megaproject.org