My research draws in ideas from computational physics, psychophysics, graphics and acoustics to make games and Augmented/Virtual Reality more believable. Over the past decade, I've developed wave modeling techniques for sound synthesis and propagation for improving the realism of interactive spatial audio. My work balances research advances with practicability: it has been published at premiere venues while shipping in multiple major products being experienced by millions of people.
Nikunj Raghuvanshi is a senior researcher at Microsoft Research. Over the past decade, he has created Triton, the world's first accurate and designable wave acoustics system. Triton has shipped in multiple AAA games and VR apps: Gears of War 4, Sea of Thieves, Windows 10 mixed reality portal and AltSpaceVR. He has also made contributions in interactive sound synthesis, such as data-driven impact sound variation shipped in the AAA game Crackdown 2, and virtual wind instrument modeling on graphics processors. Nikunj has published and given talks at top academic and industrial venues across disciplines: ACM SIGGRAPH, Audio Engineering Society, Acoustical Society of America, and Game Developers Conference and served on the SIGGRAPH papers committee. Before Microsoft, he did his PhD studies at UNC Chapel Hill, where he invented the ARD wave solver and his work helped initiate sound as a new research direction. His thesis work was licensed by Microsoft.
Triton is the first wave acoustics engine for games and VR/AR. It automatically models sound wave propagation (like diffraction) on full 3D game maps for moving sources and listener. A key novel aspect is providing the sound designer intuitive controls to dynamically tweak the physical acoustics. This results in believable environmental effects that transition smoothly as sounds and player move through the world, such as scene-dependent reverberation, and smooth occlusion, obstruction and portaling effects.
Triton is the first demonstration that high-quality wave acoustics can be made feasible for production games and Augmented/Virtual Reality.
Nikunj Raghuvanshi and John Snyder, ACM Transactions on Graphics (SIGGRAPH), 37(4), August 2018
We precompute immersive spatial wave-acoustic effects, like diffracted sound arriving around doorways, directional reflections, and anisotropic reverberation within RAM and CPU footprints immediately practical for games and VR.
Andrew Allen and Nikunj Raghuvanshi, ACM Transactions on Graphics (SIGGRAPH), 34(4), July 2015
This paper describes the first real-time technique to synthesize full-audible-bandwidth sounds for 2D virtual wind instruments. The user is presented with a sandbox interface where they can draw any bore shape and create tone holes, valves or mutes. The system is always online, synthesizing sounds from the drawn geometry as governed by wave physics. Our main contribution is an interactive wave solver that executes entirely on modern graphics cards with a novel numerical formulation supporting glitch-free online geometry modification.
Nikunj Raghuvanshi and John Snyder, ACM Transactions on Graphics (SIGGRAPH), 33(4), July 2014
This paper presents a precomputed wave propagation technique that is immediately practical for games and VR. We demonstrate convincing spatially-varying effects in complex scenes including occlusion/obstruction and reverberation. The technique simultaneously reduces the memory and signal processing computation by orders of magnitude compared to prior wave-based approaches. The key observation is that while raw acoustic fields are quite chaotic in complex scenes and depend sensitively on source and listener location, perceptual parameters derived from these fields, such as loudness or reverberation time, are far smoother, and thus amenable to efficient representation.
This paper provides the framework underlying Triton.
Shipped in Crackdown 2 (with Guy Whitmore and Kristofor Mellroth at Microsoft Game Studios)
Brandon Lloyd, Nikunj Raghuvanshi, Naga K. Govindaraju, ACM Symposium on Interactive 3D Graphics and Games (I3D), 2011
Video games typically store recordings of many variations of a sound event to avoid repetitiveness, such as multiple footstep sounds for a walking animation. We present a technique that can produce unlimited variations on an impact sound while usually costing about the same memory as a single clip. The main idea is to use an analysis-synthesis approach: a single audio clip is used to extract the resonant mode frequencies of the object and their time-decay, along with a fixed residual signal in time domain. The modal model is then amenable to on-the-fly variation and re-synthesis.
Nikunj Raghuvanshi, John Snyder, Ravish Mehra, Ming C. Lin, and Naga K. Govindaraju, ACM Transactions on Graphics (SIGGRAPH), 29(3), July 2010
This paper presents the first technique for precomputed wave propagation on complex, 3D game scenes. It utilizes the ARD wave solver to compute acoustic responses for a large set of potential source and listener locations in the scene. Each response is represented with the time and amplitude of arrival of multiple wavefronts, along with a residual frequency trend. Our system demonstrates realistic, wave-based acoustic effects in real time, including diffraction low-passing behind obstructions, sound focusing, hollow reverberation in empty rooms, sound diffusion in fully-furnished rooms, and realistic late reverberation.
Nikunj Raghuvanshi, Rahul Narain and Ming C. Lin, IEEE Transactions on Visualization and Computer Graphics(TVCG), 15(5), 2009
We present a technique which relies on an adaptive rectangular decomposition of 3D scenes to enable efﬁcient and accurate simulation of sound propagation in complex virtual environments. It exploits the known analytical solution of the Wave Equation in rectangular domains, allowing the field within each rectangular spatial partition to be time-stepped without incurring numerical errors. The spatial partitions communicate using finite-difference-like linear operators, that do incur numerical error as weak artificial reflections. The use of analytic solutions allows this technique to provide reasonable accuracy at low spatial resolutions close to the Nyquist limit.
Nikunj Raghuvanshi and Ming C. Lin, ACM Symposium of Interactive 3D Graphics and Games (I3D), 2006
We present various perceptually-based optimizations for modal sound synthesis that allow scalable synthesis for virtual scenes with hundreds of sounding objects.