Surround Sound: Understanding the audio challenges of VR and AR
Over the years, audio in video games too rarely found the attention it deserves. Often the most nuanced sonic work enjoys little more than a hat tip in game reviews, while an emphasis on photorealism and polished graphics can arguably feel overstated by contrast.
As VR and AR arrive, where audio is more explicitly vital in bolstering immersion and presence, that may be set to change.
Audio veterans like Dolby Laboratories' Director of E-Media Technology Nicolas Tsingos are helping shape what is possible in the emerging field of sound production for VR and AR.
"Despite the fact directors in cinema claim that audio is 50% – or even 80% – of the experience, historically audio has been seen as less important than the visuals by the consumer," muses Tsingos. "There's a natural reason for that, of course. We rely much more on our vision than our hearing on a day-to-day basis."
Tsingos is quick to add, however, that the role sound plays in our experience of navigating day-to-day life makes it particularly relevant in virtual realms. And that is in part because VR environments so deftly mimic our own perspective of reality.
"Contrary to games before VR, where you had the limitation of the framing of a screen to focus the player, in VR there's the fully immersive environment that's all around you, so audio will be very important to carry the cues needed to guide the player, or control player reactions."
It’s something we've discussed on this blog before, when Otherworld Interactive's Robyn Gray looked at directing the player's journey through VR and AR experiences.
In real life, where vision is limited to an individual's field of view and look-direction, our ability to hear in full 360 means audio can serve as warning or guide to what is out of sight, be it through the sound of a fire alarm or the noise of dropped coins scattering from view.
In VR experiences, the same is true. Sound is a 360 degree constant, while visuals are limited to those objects the user looks toward. As with so much in VR and AR, that attracts as many challenges as it does opportunity.
"The challenge with audio in VR is that you really want a very precise match of the 3D space you are in, and how the acoustics of that virtual space impacts the audio," explains Tsingos.
It's certainly a tricky business. Without delivering the VR user full 3D audio that is impacted by the shape and form of the scene, the sound can conflict with even the most refined, high-framerate experience.
Improper soundscapes can break the illusion of VR and undermines its knack for engendering presence. And without presence, the various head mounted displays lose the viewer.
All it takes is a sound effect that feels like it is mismatched with the acoustics of the physical space, or blurred and smudged audio. Fortunately, Dolby and many others are doing a great deal to enable quality VR and AR audio.
"Dolby is putting a lot of effort into letting content creators get the acoustics and spatial feel to the audio that they want, but without sacrificing the integrity and naturalness of those sounds," Tsingos confirms. "That's a huge challenge."
Traditionally audio in games and related content is produced and rendered within an engine, and mixed and compiled for a given VR experience.
"That's the traditional way, where people are recording their sounds, or pulling them from a library, and then bring them together," Tsingos offers. "For game-like applications, that process can work fairly well, but there are challenges in bringing that altogether to make a consistent soundscape.
Single Point Audio Capture
"Another type of VR sound is captured alongside 3D video or spherical video. There are other challenges there too. The way people tend to do this today is they record everything from a single point of view aligned with the camera, with a good quality microphone. Then they fuse that sound with what the camera has captured, and that can work quite well."
However, Tsingos says, that approach can limit the flexibility of playback, and encumber how much freedom content creators can give the user to move around.
"If you do that single point audio capture, the audio will sound bad if you move away from that point in a scene," he says.
Headphones vs. Loudspeakers
Over in AR, where virtual audio must recognize the acoustics of a real-world space, the challenges are even more significant. But they do bring about an unlikely solution: using loudspeakers instead of headphones.
Headphones, of course, are synonymous with VR, and often earmarked as pivotal to true immersion. But they aren't perfect. They isolate the user, and provide different audio on a per-user basis, where ordinary variations in human ear shape mean different users will hear an identical sound differently.
"If you use loudspeakers, many of the problems headphones bring about just go away, and especially with AR," reveals Tsingos. "The speakers are in the room with you, and they sound like they are in the acoustics of the room the user is in. There's no need to adapt to a room when the sound is already there, existing in the physical world."
Loudspeakers are also more suited to social VR gaming, where a user wearing a head mounted display can still hear other players playing on a traditional screen.
"There's a lot to say for a more shared playback of audio in making VR and AR experience social," Tsingos concludes.
Indeed, as the social elements of many different interactive entertainment forms continue to grow in importance, so does the value of counteracting the isolating nature of many VR solutions.
As with every discipline and specialty relevant to crafting VR and AR content – there remains plenty to learn and refine in the sphere of sound design and production. But that is exactly what makes the field so exciting.