Neural Radiance Fields (NeRF) Explained | 3D Scene Reconstruction

[00:00:00]what is going on guys in this video i want to um explain the paper uh behind the nerves or neural radiance fields and i want to introduce the concept of nerves uh basically you know i want to explain how they work um what they do um and so on and if this video is something that is interesting to people um and would be interesting to make a sort of from scratch implementation then let me know and i can get working on that depending on how on how popular that would be so i want to explain in a way that's easy that doesn't take too long hopefully i don't know how long this video is going to be but hopefully i'm aiming for 10 minutes we'll see if i get there it's probably going to be longer than 10 minutes but yeah all right so uh let's just start with um you know what is the goal of nerfs and uh the idea is uh that we we um we basically have some scene um in this case of drums and we take a bunch of input images like we take a bunch of different images of this drum from different viewing angles as you can see here and the idea is that we can then train nerf or run nerve optimize nerve and then we can synthesize a a view of this of this drum kit um or this scene in general it's not a particular object it's a scene so basically make a 3d reconstruction from a multi-view image input so from 2d images viewed from different angles we can reconstruct a 3d scene and we can render new views so here's an example they have from the paper where basically have like a dozen of photos that are very like jumpy in the representation but then when we uh use nerves we get this very smooth uh sort of transition uh and i don't know how smooth it looks like on the video but in in the paper it's supposed to be very smooth and i can also show you i thought this alright so i also thought i could show you this this is more recent uh research on nerfs which basically is instant nerve reduces the training time but the idea is the same so we have a bunch of input images and then from that it's able to make a complete 3d reconstruction uh from that so we can see that we can move around completely in this new uh in this new scene so that's the cool part nerves that what that's what they do now really let's get to uh basically you know how do they work and so this is the key key uh part uh and so i'll explain it's very simple actually when you actually get the idea of it so you can imagine you know for this 3d scene um if we were able to ask the model um you know what basically we have a point in this 3d uh environment we say you know we're interested in the coordinate x y z if it's viewed from particular viewing angle of theta and phi so you know as a single pixel point in in space basically we're interested in a single pixel point in space from a particular viewing angle uh we run that through our network f which is just a fully connected neural network like it's super super simple super basic um and then the uh the network outputs four values rgb the color of that particular pixel endpoint and then the volume density which is basically describing is there actually something in space here like is there actually an object here and what is the density of that object and um and so it is basically described also like the transparency of the of the material and so on hopefully that's an accurate explanation of what the volume density is but it basically describes is there something actually here and uh yeah so the network architecture in this case is uh just nine layers of fully connected neural networks with each node or each intermediate uh hidden nodes are 256 i believe so it's super simple um and so you can imagine you know if we can do this from if we if you can do this for all the points in space from all viewing angles then we can get a complete 3d representation of what that looks like so that's the network architecture um now i want to get to the key idea and the core concept of nerves which might not be immediately you know it might not be very obvious but the idea is that the network will actually be specific to the scene so we will over fit on our network to the particular scene that we're interested in and so i want to mark here that this is very unusual from how we usually how we usually do things in neural network and training um because you could imagine you know all right so we train this nerf uh we send in you know all these different environments and we train it on on this multi-viewed input images and then we can reconstruct the scene and now all we do is that we just send in any arbitrary environment with this multi-viewing images and then we get the scene representation um but that's not how nerves work the neural network is actually specific it's overfit to the particular scene that we're interested in so it basically the the network is garbage for other environments it's only for the specific scene that we're interested in um so essentially the network has condensed all the information from all different viewing angles into a complete 3d representation in the parameters of the network so this makes nerfs very costly because for each scene you have to train it and then so basically like to do something with nerves you have to train it completely there's no like pre-trained nervous models around because that's not how they work so let's answer the big question you know how does it all work because all right we just have this simple neural network but it doesn't you don't get the full picture so um step a all right step a so we have these input images uh where we've taken them right and so what we do is that we send out this array uh of um basically where we can sample um coordinates in space so we said we you know we assume that this is a from a 3d representation and we uh we send out an array from that and then we we we know the particular viewing angle of the camera right so the theta and phi are known uh and then we sample just random points along this particular uh line in space and we say um all right can you get the x y z coordinates of those points and uh obviously we don't know anything about those points right uh we don't know uh the color we don't know if there is anything there we don't know but what we do know is that we can send this to our neural network so we can send it uh in this intermediate step here we get those five values we send it to our network and we get rgb and density output now we know that these are going to be garbage right in the beginning it's going to be complete rubberish um and the example we or when we get to part b here it actually looks like it does something that's uh you know here it looks like it's been trained for a bit because it can detect the color of where there's actually an object but in the in the beginning it won't right um but so it let's just say that we're looking at ray 1 it outputs some guess of what the color and what the density is at each of these points all right so we get something from from those points and we we have multiple rays you know many many many many many rays because each you know you can imagine each uh input image sends out array or you know a lot of rays and then we have many points for each ray and then we have many viewing angles for the particular scene so what do we do next so we have these guesses for the output of each point for each array for each input image so we take those and now we basically use uh more traditional volume rendering methods and uh you know in this case here what we see is uh the density uh along each ray so for ray one for example in the beginning uh there's nothing there so the density is uh very low uh this is just air and then at some point we hit the particular object and then the density goes up because it tells us hey there's something here and then afterwards it goes down again because now we've passed the object or the the scoop or whatever um and so you know you can imagine that depends on the ray that we send out from the viewing angle but so basically what you can imagine um is that you know we do some way of um computing uh the the um all right so let me explain this in a better way so you know the label that we have right the only label that we have is the point for the particular point in the input image uh that's all we have right so we don't know the label for each of these points sent out from this ray but what we can what we can do is that we can sort of average them in a smart way using these traditional volume rendering methods and we get basically you can imagine we take a mean of all the points and then we say that the mean of these points for this ray should equal the point at this particular position in the input image so the rendering loss in this case is we get some color output of how this looks like when we take the mean of the array and then we subtract we take the l2 loss for uh the ground truth that we that we know and now you might be asking all right so you know how does it actually learn you know what is what in these particular points of the ray um and you know the answer here is that since we have multiple viewing angles uh the neural network can sort of figure it out and can know where um sort of where there's air or there's nothing and where there's an object so let's just give an example i don't know if you can actually see this but let's just say that it from from the second image ray two let's just say that it takes a point where um there's just empty uh empty space and it shoots out a shoots out array and that hits one of these dots here uh that's a white for the other ray when we do this uh this volume rendering for that particular point they're all going to be space right so each point along that ray should be just white and so in that way it can learn that all right this particular point in space should be white because we know that from this particular viewing angle and so we can sort of put that together and then when we view it from another angle uh which you know we don't know necessarily that it's space there uh we use the information from the other viewing angle and the neural network can sort of figure it out that this is uh just space and then i can uh do it in that way hopefully that gives some sort of intuition basically the neural network figures it out it's a magic thing we just send it in and it works um so uh you know hopefully everything is relatively clear maybe you know this volume rendering is confusing and so i'll explain and give some intuition around that um so the volume rendering i uh probably am confusing you more now when i throw this up but let's just give the intuition for this so what we want to the left here we want the color of a particular array right so we're viewing a particular uh image um and we put put a particular array from that image we want to get what is the color from from the ray that we send out so uh we take an integral basically to take the way of taking a mean um uh that i described simplistically from some part beginning of the ray to some ending point of the array so the ray is one-dimensional right and the way that we sort of illustrate how far we're going in the ray is with the parameter t so then we multiply three things here um and so the first one uh capital t of t which is how far we've gone along this ray is basically describing how much light has been blocked up to this particular point t um so you might imagine you know there might be a like an object blocking in the beginning and so then nothing will pass further on of the ray um so we want to know the information of that and then uh we want to know what is the density at the point uh t that we're looking at so we look at the ray for for how far we've gone of the of the ray which is at t now uh and then from the ray we get x y z coordinates uh and then we get the density at that particular point and then lastly we multiply with the color at point at r of t from a particular viewing angle d um and then this capital t of how much light has been blocked is an integral now i know that was fast i know you probably know like all you need to know okay is that this volume rendering thing is uh basically from the 1980s it's a traditional volume rendering method it's not specific to nerf uh but how it does it is that it multiplies it takes a mean basically as i described simplistically by looking at these three weighted uh how much light has been blocked up to point t the density at the particular point and then the color at that point and multiplies them and takes an integral and that's the color that's how we get the color for a particular array so that's all you need to know uh what we get with this traditional method is we get out a color for array that we've shot out from a particular point in this input image all right so now that we got this color you know what we pers what we think is the color of the point from all the points of the ray we need to compare it right so for the loss uh we take a a sort of l2 loss of um of the of so this is a little bit confusing i will get to this later on we have two terms here but just imagine we have this c hat which is the what we perceive to what we approximate or calculate to be uh the color of this ray that we've shot out and what the actual color of the ray is so we uh we can we compare those two and hopefully they should be the same right and that's the loss uh and i will get to this c and f it's a trick that they use i will explain it in just a moment so um let's get to the so that that's another trick to use two tricks um i'll get to that in a moment but first the first trick that they use um you know i showed you earlier that we have five parameters as input xyz the sort of the point in space and then we're interested in from a particular viewing angle um so it seems that neural networks don't work that well uh when we have like when this is this is maybe a little bit simplistic but basically networks don't work well when we have very low dimensional uh features because the weights are sort of um like basically they're not goods work better if we have high dimensional data so what they do is that they uh add these uh sine and cosines um like fourier features they just take these x y z viewing angle they do some transformation by doing sine and cosine of those values and then they multiply by two and four and basically some power of two and they do that uh a lot of times until they get a higher dimensional space so basically then we will have that neural network has more weights connected to the input and then it seems to work better um but we're not you know we're not adding any information this is a deterministic thing we're doing to applying to the input to get higher dimensions um yeah that's pretty much it so that's the first trick that they use uh they actually made a separate paper to explain why this works i haven't looked at it um but yeah anyways so that's the first trick that they use the second trick that they use is hierarchical sampling so um let's just look at this one to the left here we have this camera we send out this ray we have some some cloud which is uh some object right we uh we sample these rays at this point this ray point uh and then uh these are sort of at uniform distance but um we can then so okay we we get these points which are uniformly distant distant from each other uh we run that through the network and then we get the density and we see that all right the neural network thinks that we hit something here uh maybe it's a maybe maybe because the network believes it's hit something maybe it's more important to know the specifics of you know where we actually where there is an object so what they do is that they then sample a second time and they choose points close to the particular object that we perceive that we have found so the idea here is that we can use more efficient sampling by not doing uniform but rather that we have these coarse points these uniform distance point from each other we get something from the group where we get that there's an object and then we sample a second time you know where there's more importance because we know there's an object that we perceive that there's an object there sort of a second two times sampling method um and that is also when we go back i told you i would explain this uh where we have a color for the course point c and then the second term is uh the color for the uh definer points that we've sampled for the second time so that's the loss function um all right and that's it for this video it got oh it got 20 minutes all right longer than i thought but anyways hopefully that gave you a good understanding of first of all why why nerves right what they what they are and then also how they work and the tricks that they use to make it work better if you're interested in having a from scratch implementation let me know if that would be interesting and yeah thanks for watching the video hope to see in the next one [Music]