NeRF: Neural Radiance Fields for Photorealistic View Synthesis | ECCV 2020 Talk

[00:00:00]hi i'm ben mildenhall from uc berkeley and i'll be talking about neural radiance fields a new scene representation for view synthesis this is joint work with bretool srinivasan matt tancik john baron ravi rama morsi and renaung this talk we're going to focus on the problem of view synthesis where you get multiple input images of scene along with their corresponding camera poses and you want to use them to render photorealistic new views of the same scene you can get a sense of how difficult this task is based on the jerkiness and the large pixel shifts between input views shown on the left our goal is to synthesize seamless novel views like shown on the right one approach that's been very successful so far is to start with your set of images and use some procedure typically a 3d cnn to predict a discrete volumetric representation like a voxel grid then render new views by simply compositing a long rays this rendering procedure is differentiable and is easy to optimize through so the network can be trained using just held out images as supervision though prior work has demonstrated impressive results computing and storing these large voxel grids can quickly become unwieldy for large or high resolution scenes instead of representing a scene on a discretized grid one recent trend of the computer vision community is to represent shapes as continuous functions instead the main idea here is to represent a shaped surface as the level set of a fully connected neural network that's defined on 3d space since the entire representation is just the network weights the compression benefit can be huge compared to voxel grids initial work trained these networks with ground truth 3d content but more recent methods have developed algorithms to train them using only observed 2d images however this approach hasn't yet been able to match the photorealistic quality of voxel grids in nerf we want to use the volume rendering model that has achieved great results for photorealistic view synthesis but avoid the limitations of discrete grids instead we use a neural network to encode a continuous and volumetric representation of the scene combining these two ideas allows us to achieve really impressive results significantly outperforming prior work that was based on either continuous or discreetly sampled scene representations these are the key details of nerve that we'll cover in this talk our main idea is to simply encode a continuous volumetric representation of the scene inside a neural network we're able to use techniques from classic volume rendering to synthesize new views from this representation and we can optimize the network's parameters with gradient descent using images of just a single scene finally we'll demonstrate how applying a simple positional encoding to the inputs can greatly improve the network's ability to represent high frequency scene details first let's look at our continuous volumetric scene representation we represent any arbitrary scene as a simple fully connected network just nine layers of 256 channels each this network takes in a continuous 5b coordinate that consists of a 3d location xyz and a 2d viewing direction theta phi the network outputs the volume density sigma of particles at that location as well as an rgb color that represents the radiance emitted by a particle at that input location and along that input direction instead of representing volumes as voxel grids here we're just fitting a continuous function approximator to the volume without ever instantiating a grid of individual samples this allows us to trivially add more input dimensions without incurring any extra storage cost imagine representing the same 5d function as a grid sampled array discretizing those extra additional dimensions would be prohibitively expensive but in nerf we get it almost for free by just concatenating more numbers to the network input now let's look at the rendering model we use to synthesize novel views from our representation to render an image we have to estimate how much light of each color makes it out of the volume along each camera array here's a visualization of two cameras viewing this lego bulldozer to render each view we query the network at a bunch of discrete points along each ray shown by the black dots on the left the network help is the corresponding view dependent colors and volume densities shown on the right we simply composite these colors along each ray to compute the final single output color for each pixel let's take a closer look so to compute the color of a camera array that passes through the volume we need to estimate a continuous 1d line integral along that ray and we do this by querying the mlp at a bunch of samples between starting and ending distances t1 and tn and use a quadrature estimate from traditional volume rendering to estimate the integral value this estimate computes the color c of any camera ray as a sum of contributions from each segment of the array each contribution consists of the segment's color c sub i weighted by an estimate of transmission t sub i which computes how much light is blocked before reaching the segment and the amount of light that's emitted by the segment alpha sub i which is a function of the segment's length and its estimated volume density sigma this is essentially the same as the opacity of the segment one additional important detail of our rendering model is that we allow the color of any 3d point to vary as a function of the viewing direction as well as 3d position if we change the direction inputs for a fixed xyz location we can visualize what view dependent effects have been encoded by the network here's a visualization for two different points in the synthetic scene we're visualizing the view direction dependent radians learned by the network both for a point on the side of the ship and a point on the water surface this visualization is similar to a lumisphere from a surface light field we can see that a specular highlight appears at grazing angles for each point which demonstrates how for a fixed 3d location adding view directions as an extra input allows the network to represent realistic view dependent appearance effects now let's take a look at how we optimize this network if we go back and take a look at the expression for accumulated color along each ray this computation is trivially differentiable with respect to the color and volume density outputs each sample and is therefore differentiable with respect to the parameters of the network that's outputting those values this lets us optimize the network parameters by using stochastic gradient descent to just minimize the error of re-rendering all the input images we have so over the course of training simple multi-view consistency encourages the network to allocate high volume density and accurate colors to the locations where the surfaces actually exist in the underlying scene if you implement what we described so far using a standard mlp the result is quite blurry while the results from our full method contain much sharper details we accomplish this by manipulating the input coordinates to let the network represent much higher frequency functions which in this case means representing scenes with complex geometry and texture interestingly this problem where the network outputs a blurred representation is not just specific to view synthesis it's actually emblematic of a more general challenge in training deep networks to represent high frequency functions to illustrate this issue we'll drop down to a simple toy problem of optimizing a deep rally network to memorize a 2d image the input is a 2d pixel coordinate xy and the output is the rgb color of the corresponding pixel in the image we'd think this would be super easy since networks are universal function approximators but if you train a standard fully connected network to memorize an image in this way it doesn't even converge to a sharp representation this is pretty surprising especially given that this network has more parameters than the image has pixels we found a simple and effective way to enable the network to represent high frequency functions like this one we just map each of the input coordinates to a higher dimensional space encoding them with a set of sinusoids that have exponentially increasing frequencies and we apply this transformation to each scalar coordinate before passing it into the network note that this is a completely deterministic mapping with no learned parameters and that it incurs negligible extra storage and compute cost after applying this encoding to the inputs the network is now easily able to memorize the image and applying the same exact mapping to our input 5d coordinates of location and direction gives us a similar improvement for nerf now let's take a look at some results from our system here we're visualizing results across a wide variety of real world scenes each captured with about 20 to 80 input images nerf is able to render photorealistic novel views with fine geometric details and realistic view dependent effects we're able to capture all of this visual complexity in just the weights of a simple fully connected network requiring only five megabytes of storage for each scene this idea outperforms prior work that also represents a scene as a continuous function encoded by an mlp here's an example on synthetic data where we see that scene representation networks can't represent the same level of detail nerf is more consistent across views than our own prior work called local light field fusion which has to blend between multiple discrete voxel grid representations to render each view and nerf is able to represent more detailed content than neural volumes which uses a single sampled volume but is limited by its discrete resolution in these examples of synthetic scenes we show how nerf can synthesize a full hemisphere of inward facing views from just 100 input images here we're comparing a normal rendering to one where we fix the camera pose and manipulate the viewing direction inputs to the network showing how nerf can encode realistic specularities like on a tv and table or on this stove we can visualize the geometry represented by nerf by computing the expected termination depth of each ray optimizing a nerf representation from images alone results in a really detailed scene geometry like we see in these red flowers and in this christmas tree in its ornaments we can use this detail geometry to augment a scene with virtual objects and render accurate occlusion effects as they interact with the scene we invite you to take a look at our project page where you can see additional results and download our code and data thanks for listening