Deferred Neural Lighting: Free-viewpoint Relighting from Unstructured Photographs
Event Type
Technical Papers
Technical Papers Q&A
Registration Categories
TimeSunday, 13 December 20209:12 - 9:18 SGT
LocationZoom Room 2
DescriptionWe present deferred neural lighting, a novel method for free-viewpoint
relighting from unstructured photographs of a scene captured with
handheld devices. Our method leverages a scene-dependent neural
rendering network for relighting a rough geometric proxy with
learnable neural textures. Key to making the rendering network
lighting aware are radiance cues: global illumination renderings of a
rough proxy geometry of the scene for a small set of basis materials
and lit by the target lighting. As such, the light transport through
the scene is never explicitely modeled, but resolved at rendering time
by a neural rendering network. We demonstrate that the neural textures
and neural renderer can be trained end-to-end from unstructured
photographs captured with a double hand-held camera setup that
concurrently captures the scene while being lit by only one of the
cameras' flash lights. In addition, we propose a novel augmentation
refinement strategy that exploits the linearity of light transport to
extend the relighting capabilities of the neural rendering network to
support other lighting types (e.g., environment lighting) beyond the
lighting used during acquisition (i.e., flash lighting). We
demonstrate our deferred neural lighting solution on a variety of
real-world and synthetic scenes exhibiting a wide range of material
properties, light transport effects, and geometrical complexity.