Deep spherical harmonics light probe estimator for mixed reality games

Abstract

The recent developments in virtual and mixed reality by the video game and entertainment industries are responsible for increasing user’s visual immersion and provide a better user experience in games and other interactive simulations. However, the interaction between the user and simulated environment still relies on game controllers or other unnatural handheld devices. In the mixed reality context, the usage of more natural and immersive alternative to the game controllers, such as the user’s hands, may drastically increase the game interface experience, allowing a personalized visual feedback of the user’s interactions in the real-time simulation. There are basically two approaches for including the user’s hand. A 3D reconstruction based method, typically based on depth cameras, or an image-based approach, composing the virtual scene with the real images of the user’s hands. In the composition of the user’s hands and virtual elements, perceptual discrepancies in the illumination of objects may occur, generating an inconsistency in the illumination of the mixed reality environment. A consistent illumination of the environment greatly improves the user’s immersion in the mixed reality application. One way to ensure consistent illumination is by estimating the real-world illumination and use this information to adapt the virtual world lighting setting. We present the Spherical Harmonics Light Probe Estimator, a deep learning based technique that estimates the lighting setting of the real-world environment. The method uses a single RGB image and does not requires prior knowledge of the scene. The estimator outputs a light probe of the real-world lighting, represented by 9 spherical harmonics coefficients. The estimated light probe is used to create a composite image containing both real and virtual elements in an environment with a consistent illumination. We validate the technique through synthetic tests achieving an RMS error of 0.0573. We show the usage of the method in an augmented virtuality application.