Deep Light Source Estimation for Mixed Reality

Abstract

Mixed reality is the union of virtual and real elements in a single scene. In this composition, of real and virtual elements, perceptual discrepancies in the illumination of objects may occur. We call these discrepancies the illumination mismatch problem. Recovering the lighting information from a real scene is a difficult task. Usually, such task requires prior knowledge of the scene, such as the scene geometry and special measuring equipment. We present a deep learning based technique that estimates point light source position from a single color image. The estimated light source position is used to create a composite image containing both the real and virtual environments. The proposed technique allows the final composite image to have consistent illumination between the real and virtual worlds, effectively reducing the effects of the illumination mismatch in Mixed Reality applications.