I work at the intersection of computational acoustics and graphics, researching fast sound simulation techniques for games and Augmented/Virtual Reality. This includes wave modeling techniques for sound synthesis and propagation for increasing the realism of interactive spatial audio. My broader interest is to improve the believability of virtual worlds. I like to balance research contributions with practical impact: my work has been published at premiere venues while shipping in multiple major Microsoft products being experienced by millions of users.
Nikunj Raghuvanshi is a researcher at Microsoft Research where his primary focus has been Triton. It is the first spatial wave acoustics system to provide immersive physical acoustic effects within the stringent demands of games and AR/VR. Triton is now a proven solution having shipped in AAA Microsoft games, Gears of War 4 and Sea of Thieves. It holds significant promise for improving spatial audio in AR/VR, with initial adoption in the Windows 10 mixed reality portal. He has also made research contributions in interactive sound synthesis, for instance a technique for impact sound variation shipped in the AAA game Crackdown 2, and virtual musical instrument modeling on graphics processors. Nikunj has published and given talks at premiere academic and industrial venues across disciplines: ACM SIGGRAPH, Audio Engineering Society, Acoustical Society of America, and Game Developers Conference, where he has also been invited to provide reviews and serve on program committees.
Before joining Microsoft in 2009, he did his PhD studies at UNC Chapel Hill with a focus on Computer Graphics where he researched efficient sound synthesis and propagation techniques for interactive audio-visual applications. His research and code commenced sound simulation as a new direction in the UNC Gamma research group. His entire PhD codebase was licensed from UNC by Microsoft.
Triton is a first-of-a-kind wave acoustics engine that I've researched for a decade. It automatically models sound wave propagation on full 3D game maps for moving sources and listener, while also providing the sound designer artistic control on the acoustics. This results in believable environmental effects that transition smoothly as sounds and player move through the world, such as scene-dependent reverberation, and smooth occlusion/obstruction effects.
Triton is the first demonstration that high-quality wave acoustics can be made feasible for production games and Augmented/Virtual Reality.
Andrew Allen and Nikunj Raghuvanshi, ACM Transactions on Graphics (SIGGRAPH), 34(4), July 2015
This paper describes the first real-time technique to synthesize full-audible-bandwidth sounds for 2D virtual wind instruments. The user is presented with a sandbox interface where they can draw any bore shape and create tone holes, valves or mutes. The system is always online, synthesizing sounds from the drawn geometry as governed by wave physics. Our main contribution is an interactive wave solver that executes entirely on modern graphics cards with a novel numerical formulation supporting glitch-free online geometry modification.
Nikunj Raghuvanshi and John Snyder, ACM Transactions on Graphics (SIGGRAPH), 33(4), July 2014
This paper presents a precomputed wave propagation technique that is immediately practical for games and VR. We demonstrate convincing spatially-varying effects in complex scenes including occlusion/obstruction and reverberation. The technique simultaneously reduces the memory and signal processing computation by orders of magnitude compared to prior wave-based approaches. The key observation is that while raw acoustic fields are quite chaotic in complex scenes and depend sensitively on source and listener location, perceptual parameters derived from these fields, such as loudness or reverberation time, are far smoother, and thus amenable to efficient representation.
This paper provides the framework underlying Triton.
Shipped in Crackdown 2 (with Guy Whitmore and Kristofor Mellroth at Microsoft Game Studios)
Brandon Lloyd, Nikunj Raghuvanshi, Naga K. Govindaraju, ACM Symposium on Interactive 3D Graphics and Games (I3D), 2011
Video games typically store recordings of many variations of a sound event to avoid repetitiveness, such as multiple footstep sounds for a walking animation. We present a technique that can produce unlimited variations on an impact sound while usually costing about the same memory as a single clip. The main idea is to use an analysis-synthesis approach: a single audio clip is used to extract the resonant mode frequencies of the object and their time-decay, along with a fixed residual signal in time domain. The modal model is then amenable to on-the-fly variation and re-synthesis.
Nikunj Raghuvanshi, John Snyder, Ravish Mehra, Ming C. Lin, and Naga K. Govindaraju, ACM Transactions on Graphics (SIGGRAPH), 29(3), July 2010
This paper presents the first technique for precomputed wave propagation on complex, 3D game scenes. It utilizes the ARD wave solver to compute acoustic responses for a large set of potential source and listener locations in the scene. Each response is represented with the time and amplitude of arrival of multiple wavefronts, along with a residual frequency trend. Our system demonstrates realistic, wave-based acoustic effects in real time, including diffraction low-passing behind obstructions, sound focusing, hollow reverberation in empty rooms, sound diffusion in fully-furnished rooms, and realistic late reverberation.
Nikunj Raghuvanshi, Rahul Narain and Ming C. Lin, IEEE Transactions on Visualization and Computer Graphics(TVCG), 15(5), 2009
We present a technique which relies on an adaptive rectangular decomposition of 3D scenes to enable efﬁcient and accurate simulation of sound propagation in complex virtual environments. It exploits the known analytical solution of the Wave Equation in rectangular domains, allowing the field within each rectangular spatial partition to be time-stepped without incurring numerical errors. The spatial partitions communicate using finite-difference-like linear operators, that do incur numerical error as weak artificial reflections. The use of analytic solutions allows this technique to provide reasonable accuracy at low spatial resolutions close to the Nyquist limit.
Nikunj Raghuvanshi and Ming C. Lin, ACM Symposium of Interactive 3D Graphics and Games (I3D), 2006
We present various perceptually-based optimizations for modal sound synthesis that allow scalable synthesis for virtual scenes with hundreds of sounding objects.