Heard but not seen: The importance of spatial audio elements for Immersion in XR environments
Sound in XR environments: an overlooked tool
If you were to ask a virtual production team what creates immersion in an XR environment, the answer will most probably be connected to something visual. The LED XR stage, the rendered visual that is being displayed, or maybe the lighting that finishes of the feel of the environment. If sound even gets mentioned, it will be most likely be towards the end of the list.
This was the starting point for this research, conducted in collaboration with AVENUE as part of a graduation project for Breda University of Applied Sciences. The research question was as followed: Which spatial audio design elements most effectively facilitate audience immersion in XR environments?
The study used a Dutch folklore De Witte Wieven (The Witty Women) as an example story for a spatial audio XR experience. Twelve participants attended individual listening sessions at the XR stage of AVENUE where they compared 3 variation of six spatial audio elements. A simple supporting visual was displayed on the XR screen while the participants listened to the audio. After the listening session a semi-structured interview was conducted, of which the data was analysed using a thematic analysis.
The spatial audio elements put to the test!
The spatial audio elements tested were: directionality, movement/drift, vertical space, reverb, distance/proximity, and environmental density. Each participant heard three variations per element that ranged from minimal to very heavy implementation of the spatial audio element. After listening to the diferent versions the participants were asked to state the version in which they felt most immersed.
The information that emerged from the interviews wasn’t just data. It were descriptions from participants about physical reactions such as, the urge to turn around, a feeling in their stomach, or a tingle in their ears. These physical responses clearly highlighed which elements genuinely improved immersion and the feeling of presence within the participant.
Which spatial audio elements create immersion?
The strongest drivers of immersion turned out to be movement/drift, where participants felt as if the sound circled around them or moved through them. And distance/proximity, where close sound made the participants feel consumed by the sound where as far sound made them feel like observers of a situation.
Evironmental density turned out to be a vital spatial audio element for placement. Rich ambience layers allowed the participants to create a vivid imagination of a place where the story was taking place.
Reverb should be used with cation since it should always match the environment since if it is overdone it disoriented the participants.
Vertical space ended up being the least effective element since it was the hardest to perceive by participants, especially through headphones.
Some quotes from the participants that shows their experience:
“The distance element. It was inside me. It was definitely there. It just felt like the spirit consumed me.”
“Environmental sounds really make me immersed in where I am. Movement in the sounds made me feel like there is actually something there. But the environmental sounds gave more context to the story.”
”The movement did make me think a little bit, should I turn around to the left or right, because they’re moving around you and it feels like they were behind me.”
Visuals need to cooperate with audio for it to work!
When researching audio you don’t expect to find results on visuals. However, when conducting this research it became clear that the visual displayed on the XR screen helped in making the participants feel immersed. Participants did however mention that this immersion could only happen when the visual was coherent with the audio they were hearing. This shows how important the collaboration between visual and audio is, and that if you truly want an immersive experience audio should not be an afterthought but and equal consideration along with the visual!
The takeaway for XR sound designers
The findings of this research can be used as guidelines for XR sound designers. They come in a practical hierarchy so anyone designing Spatial Audio for an XR environment knows what to prioritise. The most important Spatial Audio elements to include are movement and distance, since these elements created immersion and presence the most. Then comes environmental density, it is important to build a rich ambience layer that gives the listeners enough information about the intended environment. Reverb should be used with caution since overdoing it could lead to disorienting the listeners. Lastly, vertical spacial audio can be left on the background since it showed to not be perceived by participants. Therefore as a sound designer it is best to focus your energy on the other elements mentioned above.
On a general note, it is important to think about the balance across all elements since extremes in both directions like too much reverb or a too dense environment have been noted to break the immersion. The goal should not be to make use of the spatial elements to the maximum but to use them just enough to tell your story well, keep the listener interested, and of course immersed!
For AVENUE and also the virtual production community that work in XR environments this research has a clear message and opens a new path forward. XR technology has been proven as a successful filmmaking tool. But if this study shows one thing it is the potential XR can have when sound and its design is treated with the same attention and precision as the visuals. If this is done the XR industry can flourish and grow into an even stronger and more immersive tool for filmmaking and much more!