Repurposing a relighting network for realistic compositions of captured scenes

Baptiste Nicolet1,2,3 Julien Philip1 George Drettakis1
1Université Côte d'Azur, Inria 2École Polytechnique, IP Paris 3Télécom Paris, IP Paris

In Proceedings of the ACM SIGGRAPH Symposium on Interactive 3D Graphics and Games, 2020

Teaser
Example of a composition of two captured historical landmarks using our method. We extract the geometry of one(b) from a multi-view dataset, and import it into the other (a). Our method ensures coherent treatment of lighting and shadows,producing a realistic result (c).

Abstract

Multi-view stereo can be used to rapidly create realistic virtual content, such as textured meshes or a geometric proxy for freeviewpoint Image-Based Rendering (IBR). These solutions greatly simplify the content creation process compared to traditional methods, but it is difficult to modify the content of the scene.We propose a novel approach to create scenes by composing (parts of) multiple captured scenes. The main difficulty of such compositions is that lighting conditions in each captured scene are different; to obtain a realistic composition we need to make lighting coherent. We propose a two-pass solution, by adapting a multi-view relighting network. We first match the lighting conditions of each scene separately and then synthesize shadows between scenes in a subsequent pass. We also improve the realism of the composition by estimating the change in ambient occlusion in contact areas between parts and compensate for the color balance of the different cameras used for capture. We illustrate our method with results on multiple compositions of outdoor scenes and show its application to multi-view image composition, IBR and textured mesh creation.

Downloads

Reference

Baptiste Nicolet, Julien Philip, George Drettakis. Repurposing a relighting network for realistic compositions of captured scenes. Proceedings of the ACM SIGGRAPH Symposium on Interactive 3D Graphics and Games, May 2020. 
@inproceedings{nicolet2020repurposing,
    author = "Nicolet, Baptiste and Philip, Julien and Drettakis, George",
    title = "Repurposing a Relighting Network for Realistic Compositions of Captured Scenes",
    booktitle = "Proceedings of the ACM SIGGRAPH Symposium on Interactive 3D Graphics and Games",
    month = may,
    year = "2020",
    organization = "ACM",
    keywords = "Image Manipulation, Image-based rendering, Image processing",
    url = "http://www-sop.inria.fr/reves/Basilic/2020/NPD20a"
}