-
Considerable progress has been made over the past years in the understanding of auditory cognitive processes and capabilities - from perception, attention and memory to complex performances such as scene analysis and communication. The first phase of AUDICTIVE focussed mostly on porting the well-controlled but often unrealistic stimulus presentations used in auditory cognition research to more comprehensive virtual or mixed reality environments. Here, recent developments in hard- and software technologies were reflected, with audiovisual virtual and mixed reality (VR, MR) reaching a high level of perceptual plausibility. In the first phase, the research in AUDICTIVE aimed at converting and/or applying interactive VR and/or MR technology to explore auditory cognition in audiovisual scenes that are closer to real life. With a variety of interactive virtual environment technology and scenes, controlled research was conducted on how acoustic and visual components and further contextual factors affect perception, cognition, and interaction. Here, we focused on the applicability and transfer of established paradigms from auditory cognition with rather simple audiovisual environments, significantly expanding beyond previously used basic audio or visual-perception-based approaches.
- To what extent are recent theories of auditory cognition and related empirical findings applicable in a more life-like environment created by interactive audiovisual VR?
- How can the realism and vibrancy of audiovisual virtual environments be brought to a higher level, both in terms of audiovisual representation as well as user interaction, based on the findings from auditory cognition research?
- What are suitable and novel quality evaluation methods that enable the systematic and efficient assessment of VR systems (cf. b) in light of the targeted auditory cognition research (cf. a)?
- What novel auditory cognition research paradigms, methods, models, and theories can be established that consider the relevant aspects and features of interactive (virtual) environments (as determined in the first phase) as key components?
- To what extent do audiovisual components (identified and/or developed in the first phase) need to be provided and integrated to achieve a new level of vibrancy and fidelity of virtual environments? How can audiovisual cues in social VR and, in particular, for conversational virtual humans, be realized to significantly improve user experience and social presence, with outreach to real-life situations?
- How can instrumental models be developed that predict VR quality or QoE in terms of auditory cognition performances? What guidelines need to be met by audiovisual VR systems and environments to validly enable specific auditory cognition research?
The results of the first phase have laid the foundation for the second phase of AUDICTIVE, which aims to identify or (further) develop suitable paradigms to use them in more realistic scenes, aiming to elicit close-to natural perception, experience, and/or behaviour.
Hence, in the second phase of AUDICTIVE, we seek to further expand the scientific level of knowledge, theories, and models that have been developed within basic auditory perception and cognition research to even more realistic daily-life situations. Here, new knowledge, methods, and techniques shall be generated (e.g., psychometric and cognitive assessment, QoE evaluation, physiological or behavioural analysis, signal acquisition and analysis, VR/MR technology enhancement), for richer and more complex scenarios, involving interactive VR and/or MR technology. This reflects that virtual reality technology is continuing to mature, opening up new areas of application that include acoustics as a key component, such as immersive social MR (various parties being co-present in one environment); health-promoting measures through research on the effects of complex acoustic environments and/or noise (e.g., classroom, working environment); training, therapy and/or rehabilitation requiring (e.g., social anxiety, speech training after sudden hearing loss); assistive systems, possibly for user groups with special needs such as being hard of hearing (e.g., telepresence designed for elderly).
AUDICTIVE targets the three main research priorities:
Auditory cognition (a): Projects in AUDICTIVE will investigate, which auditory cognition research paradigms, methods, models, and theories can be established that consider the relevant aspects and features of interactive (virtual) environments (as determined in the first phase) as key components. Here, projects aiming for further advancement of the VR and/or MR-extended, more basic cognitive research paradigms from Phase 1 are funded, as well as novel research paradigms specifically catering for the opportunities brought along with VR and/or MR technology.
Interactive virtual environments (b): Regarding VR or MR technology, projects in AUDICTIVE will investigate to what extent audiovisual components (identified and/or developed in the first phase) need to be provided and integrated to achieve a new level of vibrancy and fidelity of virtual environments. Further, research in AUDICTIVE projects will address how audiovisual cues in social VR and, in particular, for conversational virtual humans, can be realized with different instances of VR techniques to significantly improve user experience and social presence, with outreach to real-life situations.
Quality evaluation methods (c): Using cognitive performances to assess auditory and audiovisual cognition based on VR and/or MR scenes can be considered as a cognition-based assessment of the VR and/or MR technology employed. In this second phase, projects are sought to investigate evaluation methods more systematically, answering questions such as how instrumental models may be developed that predict VR and/or MR quality or QoE in terms of auditory cognition performances, or what guidelines need to be met by audiovisual VR and/or MR systems and environments to validly enable specific auditory cognition research.
Specific questions to address research priorities of Phase 1 were:
Specific questions to address research priorities of Phase 2 are:
Insights from AUDICTIVE are expected to advance new virtual and augmented reality technologies involving acoustics as a key component, for the user-centered development and design of applications such as immersive teleconferencing, health-promoting measures, rehabilitation, or assistive systems supporting users with special needs.
