Interactive Virtual Environments (IVEs) provide unique new possibilities to study human perception and socio-cognitive processing in complex scenes. In IVEs, ecologically valid environmental conditions can be combined with a high degree of experimental control. Although researchers often included audio as a modality, the integration of highly realistic audio is still an exception. The overarching goal of this project is to identify the requirements regarding binaural audio rendering to achieve highly realistic audiovisual presentations of virtual social scenes and to characterize the effects of the audio rendering on (social) presence and socio-cognitive processes such as emotion, empathy, and (social) anxiety. Whereas in the first funding phase, the focus was on achieving the highest possible realism, in the second funding phase we will concentrate on characterizing the interaction between top-down effects and stimulus-related bottom-up effects in virtual social interaction scenarios on a self-report, behavioral, and on a neural level (measured via EEG). This interaction will be investigated by considering and analyzing expectations, emotion, familiarization, social stress, and presence as well as higher-order social cognition (e.g., empathy) in virtual social scenarios with different degrees of complexity regarding the binaural rendering in these scenarios: the (high realism) rendering developed in phase 1 is the starting point, which will be extended to multiple speech and non-speech auditorium-related sources as well as to a more advanced own-voice renderings in order to enable close-to-real-life interactive social scenes. Habituation and training, as well as alternatives to head-tracking, will be investigated to facilitate the use of audiovisual IVEs and to enable further clinical applications such as treating social anxiety disorders. Overall this project will contribute to better understand psychological and physiological processes and their functional relevance to provide plausible representations of real-life environments.
