Auditory Cognition in Interactive Virtual Environments

    Considerable progress has been made over the past years in the understanding of auditory cognitive processes and capabilities - from perception, attention and memory to complex performances such as scene analysis and communication. To this aim, well controlled but often unrealistic stimulus presentations were used, involving simple instances of virtual environments.
    With recent developments in hardware- and software technologies, audiovisual virtual reality (VR) has reached a high level of perceptual plausibility that overcomes the limitations of simple laboratory settings. Applying interactive VR technology is expected to help understand auditory cognition in complex audiovisual scenes that are close to real life, including acoustically adverse situations such as classrooms, open-plan offices, multi-party communication or outdoor scenarios with multiple (and moving) sound sources. In particular, VR enables controlled research on how acoustic and visual components and further contextual factors affect the ability to interact with the scene, e.g. freely moving in it.
    To best integrate research in auditory cognition and VR, it is required that aspects from the three scientific disciplines of acoustics, cognitive psychology, and virtual reality/computer science are addressed. The cooperation between researchers from these fields will allow synergetic effects which cannot be achieved by a single discipline.
    Hence, by linking the three disciplines, the priority program "Auditory cognition in interactive virtual environments - AUDICTIVE" will significantly extend the knowledge of hearing-related cognitive performances in real-life scenes and to enable creating "auditory-cognition-validated" VR technology. AUDICTIVE targets fundamental research across the three disciplines and addressing the three research priorities (a) "auditory cognition," (b) "interactive audiovisual virtual environments," and (c) "quality evaluation methods," the latter being located at the interface between (a) and (b).

    Specific questions to address research priorities are:
  1. To what extent are recent theories of auditory cognition and related empirical findings applicable in a more life-like environment created by interactive audiovisual VR?
  2. How can the realism and vibrancy of audiovisual virtual environments be brought to a higher level, both in terms of audiovisual representation as well as user interaction, based on the findings from auditory cognition research?
  3. What are suitable and novel quality evaluation methods that enable the systematic and efficient assessment of VR systems (cf. b) in light of the targeted auditory cognition research (cf. a)?

  4. Insights from AUDICTIVE are expected to advance new virtual and augmented reality technologies involving acoustics as a key component, for the user-centered development and design of applications such as immersive teleconferencing, health-promoting measures, rehabilitation, or assistive systems supporting users with special needs.