The big buzzword which has been growing in popularity at GDC the last few years has been the return of VR (Virtual Reality) with a smaller subset AR (Augmented Reality).
VR first appeared back in the 90’s as low resolution, slower response screens headsets – and boy were they heavy! I was lucky enough to have a friend who owned a VR hardware business, MindFlux.com.au, and was able to actually try these first generation headsets on games such as Quake and Descent!
A good simple overview of the technology can be found here in this article “For VR to be truly immersive it needs convincing sound to match“. The author mentions one key element which has been touted and preached for many many years before VR made it’s return.
“The premise of VR is to create an alternate reality, but without the right audio cues to match the visuals, the brain doesn’t buy into the illusion.”
This rings true in non-VR games but is quite the understatement in my opinion in regards to 3D experiences. Not only does the audio need to match – but since the audio is now spatialized, the focus shifts to audio feature implementation and a good quality ambisonic to binaural audio decoder engine is required to really sell the immersion. Traditional mixing goes out the door – so implementation is the mix. The trick is to find the right balance between realism and cinematic implementation tricks to totally immerse the listener emotionally and functionally within the new environment.
Both VR and AR have their own subsets of problems associated with how audio is presented. My thoughts are a lot clearer on implementing and mixing VR – due to the separation of audio and visual from the real world.
With AR, one question about creating an immersive audio experience is how to merge the sounds from the virtual world with those in the real world. Overlaying imagery may work with color analysis adjustments to the overlay just like in 3D movies so that your eyes are fooled. However fooling your ears may be a little more difficult. Consider this, if you are in a large room and you have a VR sound source appear – your ears would expect it to behave acoustically like the environment your in. If that acoustic placement does not match, then it will stand out and destroy the immersion.
There are many techniques to implementing audio to trick the ears into believing something. However this really depends on the application – a static view point in 3D Film vs moving around a 3D game environment.
One of the important aspects to both these is some sort of centering of your self within this world. Our ears pick up tiny reflections off objects within the acoustic space from sound emitters. These reflections, combined with the HRTF behavior of spatial recognition helps your mind place you in an acoustic space and thus determine the location of another object within that room. Without a grounding reference point your ears would be disconnected from the visuals at a subconscious level. Infact I experienced this at GDC2016 last week. The post VR effects on me was slight disorientation and a little nausea. I checked out a tech demo by the TwoBigEars guys for their 3DCeption tool. This tool does a very good job of combining a few different spatial binaural calculations to provide not only room reverb, but also HRTF so that you are aware of your position and bearing in the virtual world. I did not feel any disorientation or nausea afterwards.
Recent Comments