Reasoning realistic illumination and material using diffusion model

What is it about?

3D reconstruction aims to recover 3D scene properties such as geometry, material and illumination from 2D images. However, it is a tapical ill-posed propram due to occlussion or decomposition ambiguity, that one can find multiple diverse solutions to explain the same observation. One common solution is combine prior knowledge from human to narrow down the reasoning range. We propose to introduce a diffusion model based HDR environment map prior to the inverse rendering process, to refer diverse and realistic sampling of illumination and materials.

Featured Image

Photo by Ngô Thanh Tùng on Unsplash

Photo by Ngô Thanh Tùng on Unsplash

Why is it important?

Classical 3D reconstruction method tends to return deterministic and unrealistic results of illumination and material due to lack of prior knowledge such as how the nature illumination condition looks like. We solve this problem by introducing a powerful diffusion model based illumination prior. Further, our method finds multiple plausible solutions from the solution space, which is distinguish from deterministic methods that always converge to the same solution and can't tell you how ambiguous the problem itself is.