Lighting Estimation for Mixed Reality.

Last updated on Aug 27, 2021

Consistent Lighting

Consistent rendering of lighting is a substantial part of the eXtended reality universe, particularly in mixed reality, where real objects are combined with virtual objects. The consistent lighting of real and virtual objects increases the user`s immersion and dramatically improves the user`s experience.

The rendering of consistent lighting requires previous knowledge of the environment lighting. This is a challenging task due to the unknown aspects of real-world lighting settings.

This research project aims to estimate real-world lighting using state-of-the-art machine learning techniques. Exploring Deep Learning methods, combining with computer graphics expertise, we aim to extract valuable information about the environment lighting to provide a consistent mixed reality world.

Deep Learning Mixed Reality Computer Graphics

Publications

Spatially and color consistent environment lighting estimation using deep neural networks for mixed reality

This paper presents a CNN-based model to estimate complex lighting for mixed reality environments with no previous information about the scene. We model the environment illumination using a set of spherical harmonics (SH) environment lighting, capable of efficiently represent area lighting. We propose a new CNN architecture that inputs an RGB image and recognizes, in real-time, the environment lighting. Unlike previous CNN-based lighting estimation methods, we propose using a highly optimized deep neural network architecture, with a reduced number of parameters, that can learn high complex lighting scenarios from real-world high-dynamic-range (HDR) environment images.

Bruno A D Marques, Esteban Clua, Anselmo Montenegro, Cristina Nader

Spatially and color consistent environment lighting estimation using deep neural networks for mixed reality

Deep spherical harmonics light probe estimator for mixed reality games

We present the Spherical Harmonics Light Probe Estimator, a deep learning based technique that estimates the lighting setting of the real-world environment. The method uses a single RGB image and does not requires prior knowledge of the scene. The estimator outputs a light probe of the real-world lighting, represented by 9 spherical harmonics coefficients. The estimated light probe is used to create a composite image containing both real and virtual elements in an environment with a consistent illumination.

Bruno A D Marques, Esteban Clua, Cristina Nader

Deep spherical harmonics light probe estimator for mixed reality games

Deep Light Source Estimation for Mixed Reality

We present a deep learning based technique that estimates discrete point light source position from a single color image. The estimated light source position is used to create a composite image containing both the real and virtual environments.

Bruno A D Marques, Rafael Rego Drummond, Esteban Clua, Cristina Nader

Deep Light Source Estimation for Mixed Reality