Image Geometry and Planar Homography | Computer Vision Lecture 9

[00:00:00]okay we might get we might get started uh this is the second lecture about image geometry so last Thursday we had a bit of a rave about the geometry of image formation we talked about three dimensions and two dimensions and I'm going to revisit some of the things that we did in that last lecture and then get through to one piece of information that you absolutely going to require to complete the crack right it's the mystery of the blue dots why have you got blue dots on the sheet today we explain why you've got blue dots all right so to recap what we did in the last lecture we talked about how we can represent threedimensional things that look three dimensional on a plane right and it's all to do with obeying the laws of the geometry of image formation if you don't obey these laws things are not going to look very three-dimensional right so it took a long time for human beings to be able to figure out the geometry and draw things on a plain surface that look like they've gone actual depth to them and has some street art versions that we showed talked about last Thursday as well um and we can create we can form an image and we talked a bit about pinhole image formation so if you've gone a dark room with a little tiny aperture in the wall then you get an inverted image formed of the world outside it's called a pinhole camera some of you may have made one when you were a kid uh and the geometry of this image formation is really pretty simple it's all about similar triangles we talked about similar triangles but we also talked about the fundamental problem with a pinhole camera and that is that it creates images which are very very dark yeah I can stand up but I can't walk uh this is progress uh that the images are very dark because not much energy not much radiant energy from the world can get through that tiny little hole most of it smacks against the into the wall so we form an image using a lens a lens being a big chunk of glass allows a lot more light to come through uh so we have this thin Lens model that you probably met in high school physics uh and it allows you to create much Brighter Images but it's got a focus problem right it's possible that the image will be out of focus and you have to adjust the position of the lens with respect to the surface on which the image is formed in order to achieve a beautifully focused image the sad thing is you can't have generally you can't have everything the image in Focus so you can have one thing in focus and the rest is blurry uh you can adjust what's in Focus but you can kind of everything in Focus simultaneously so we talked about some alternative sorts of cameras light field cameras last week which allow you to capture the whole L field and retrospectively focus the image anyone go out and buy a light field camera the price of them how much are they 400 bucks okay all right so there were some equations for the thin Lens model and so a consequence of forming an image of taking three three threedimensional world and crunching it down so that it lies that you have an image on a plane we've removed the dimension right we've uh removed we've gone from 3D to 2D we've removed the dimension of depth and the consequence of that is that things look a little odd so lines that are really parallel in the world become not parallel in the image things that are circles in the world become not circles in the image and this is so familiar to us that we don't pay it second thought we don't think that it's weird because we've got perspective cameras in our heads and this is the way things look to us uh but if you think about it it's a little odd and it's a consequence of losing a dimension you can't have a you can't throw away a dimension casually and not have consequences these are the consequences so in the in I want to touch briefly on some some really fundamental geometry stuff uh so again hopefully you know this from from high school and from your education here at quut some really familiar Concepts from geometry and we're just going to extend them a little bit so we have the concepts of what's called ukian geometry a ukian plane and that is a plane where the rules of ukian geometry appli and we have cartisian coordinates so this is the idea where you've got an x coordinate this way and a y-coordinate orthogonally to that and it was developed by a guy called Renee dear a famous philosopher and mathematician and mercenary it's an interesting job combination and uh he was the first person to realize you could you could represent a point uh as two as two coordinates against a pair of orthogonal axes and the story is that he was lying in his in his bed one morning and he liked to sleep in uh and he was lying and he saw a fly on the ceiling and he realized that he could represent the position of the fly by its distance from the two walls and so that then came up into this concept of cartisian geometry which is named after him so if we're doing cartisian geometry in the plane two-dimensional cartisian geometry we have a point and it's got an X and a y coordinate right so we can represent a point as a vector that goes from the origin to the point or we can represent it by two numbers it's x and y coordinate this should be very very familiar to you right so here's our point we can represent it by two coordinates so we say it exist in the space r s so it is represented by two real numbers that's what r s means we can also represent it in a homogeneous form so we represent a point which has got two coordinates by three numbers and you might think that this is not very useful but I'll show you in a few minutes why it's useful we touched a bit on it last week we're going to talk a bit more about it today so we we say that this number exists in the space p^ s it's a projective coordinate a two-dimensional projective coordinate is represented by three numbers now we so we to convert a number from cartisian to homogeneous we just append a one to it so we have X and Y is our cartisian coordinate the homogeneous coordinate is X comma y comma 1 what we do is whack a one on the anend our homogeneous coordinates I tend to represent with a little tier on top so here's a homogeneous coordinate of a point p with Tilda and it's represented by three numbers X Tilda y Tilda and Zed Tilda yeah to convert it back to a cartisian coordinate I divide the first two numbers by the third number this is really important okay we take the first two numbers divide them by normalize them by the last number and we get the cartisian equivalent so we can convert from cartisian coinness to homogeneous homogeneous back to cartisian whole lot of stuff becomes really easy when you uh when you think in terms of homogeneous coordinates and what's really interesting in homogeneous coordinates is that points and lines are almost the same we say that they're duels and I'll show you a few cheap tricks it's not examin a but it's really cute um here's a line in homogeneous coordinates I can represent a line by three numbers I can represent a point by three numbers I can represent a line by three numbers so there's a line represented by three numbers L1 L2 L3 and there's a point so I can Define the line as a set of all points for which the dotproduct of the line and the point is equal to zero so we call that the point equation of a line and it's Advantage compared to the form of the line that you probably probably know y = mx + C is that this form of a line allows you to have a line that's vertical in that cartisan representation of a line if the line's vertical M goes to Infinity it all gets a bit ugly in homogeneous coordinates we represent a line by three numbers you can represent a horizontal line a vertical line line of any slope at at at all so you can represent a line by three numbers as well now if I've got a one point represented in homogeneous coordinates and another Point represented in homogeneous coordinates then the equation of the homogeneous line that goes through them is the cross product of the two points in homogeneous coordinates this all becomes delightfully easy it's much more complicated to do this in the usual two-dimensional representation of lines and points and similarly if we've got two lines represented in homogeneous form and that point that they intersect at is the cross product of the two lines so there's a gorgeous symmetry here between lines and points uh we can connect lines that go join two points points that are formed by the intersection of two lines as I say not examinable uh but just to try and show the power of this homogeneous representation of points and lines and why don't we do that so we can represent the pinhole model of a camera this thin thin Lens model by this homogeneous equation and so we have the homogeneous representation of the points in the world which is X and Y and Zed with a one on the end so this is a three-dimensional cartisian point in homogeneous form so I've just put a one on the end of it so that's the vector that's on the right we have a matrix in the Middle with mostly fs and kns and a one in it and on the other side then we have the homogeneous coordinate of the point on the image and that's represented by three homogeneous coordinates yeah and to convert from those homogeneous coordinates on the image back to the homogeneous coordinates back to the cartisian coordinates on the image we use this relationship so what we've done is we've taken World Point XY Z turned it into a homogeneous Vector multiplied by a matrix and then converted it back to cartisian form and uh we can we can model the effect of a camera this perspective projection that crunches 3D into 2D so this is how we can represent the image formation process so there's the X and Y on the on the image plane so we've eliminated the the sort of the the divide by by Zed uh problem that normally comes when you represent these equations in cartisian in cartisian form now go back to this relationship that I just had before and we can we can factor that Matrix into two matrices actually The Matrix that's on the right does the scaling it's all about the focal length and you know with a camera in his lens the focal length is Big then it's going to do zooming it's going to it's going to zoom in on something as far away focal length is small it's going to take a very wide angle view things are going to appear smaller in the image and this is all stuff that we touched on in the last lecture that Matrix converts three dimensions to two Dimensions there's another way we can represent all this and it's What's called the central projection model so before we had a model where the light Ray came in through a lens focused on a plane on this side it's quite convenient to represent it in a slightly different form where the ray comes from the world pierces the image plane on its way to the origin so you draw a line from the world point to the origin it comes like this and it intersects the image plane the equations are exactly the same but it's quite common in many textbooks to write the image formation process using this Central projection model the maths is exactly the same but it's called the central projection model that's the Maths for up the top there now if we use this Central projection model and it's a little bit illustrative normally when the ray comes in right it hits the image plane at a particular point and the image plane is generally quite small in a camera you know the image plane might be a little square you know kind of that size if it's film it might be only 6 mm square if it's the if it's the sensorship in your camera yeah it's a very little sensor that the image is formed on so if something's maybe 2 meters high in the world on the image plane it's only going to be a few millimeters tall right it's it's it's it's shrunk everyone get that okay in a digital camera system right the image is going to fall onto an array of pixels and we've talked about pixels before and arrays of pixels so what we're going to do is consider that the image is now going to fall onto a grid and we want to know which grid cell does the ray fall on because that's the pixel that that light Ray is going to illuminate now we've talked about with images too that they have a different coordinate system with an image we have the coordinate 0 0 up in the top left corner right and the the U coordinate increases this way and the V coordinate increases down the image and we've done that when we've been doing image processing in PRS and CHS and and talking about it in the first few lectures so what we want to do is some transformation between the distance in meters uh at which the at which the ray lands on the image plane and it's coordinate in terms of pixels right so we want to scale it from meters into pixels and we do that by using this linear transformation at the top so what we do is we take the the size of the image and we divide it by the height of each pixel and that converts the height of the image in meters on the image plane which is really small into a number of pixels and then we also want to do a shift of the origin from the middle of the image off to the top left corner and so that Matrix up the top there does both of those things if you work in homogeneous coordinates it will both scale the image and it will shift it and and so that the the values there u0 and V 0 are the coordinates of the center of the image yep so if your image is 1024 by 1024 then U and v0 will be 512 right so what we're doing is we're building up a number of Transformations and we can stick them all together and what we end up with is this relationship here which I went over really quickly last time there are three matrices all in a row so we have on the right hand side the point in the world it's X Y and Z coordinate that we've turned into homogeneous form then we're going to multiply by three matrices and we're going to end up with the homogeneous coordinates of the pixel in the image so these matrices uh and this the those two are called the intrinsic parameters of the camera they are things that depend only on the camera so and there's two parts to that there's one part which depends on the focus length like how much Zoom your camera's got that's the middle Matrix The Matrix that's on the left hand side is doing this scaling that's that got information about how big the pixels are and where the camera where the pixel array is with respect to the lens so those two matrices describe things that are innate in the camera the fundamental characteristic of the camera the next Matrix along the one with the r and a t in it says whereabouts the camera is in the world so the r Matrix is rotation Matrix that you might have covered with Michael way back at the beginning says what's the orientation of the camera in the world which way is it pointing and T is where is the camera in the world so you think that the image that you form with your camera depends on a lot of things the image depends on where the camera is pointing where it's situated what the zoom what the focal length is and on the size of the pixel array right the image formation depends on all of those parameters and that's how it can be factored out now normally you don't know all uh so that they're called the extrinsic parameters they depend only on where the camera is not on the characteristics of the camera all together all those matrices together are called the camera Matrix you can it's very very difficult to derive this from first principles so generally what you can do is you can perform some calibration process and you can estimate it you just have a matrix with 12 numbers in it it's a 3x4 matrix with 12 numbers in it you put in the three-dimensional coordinates in one side on the other side comes the image plane coordinate so what I'm going to do is a little exercise for you saying you need some pencil and paper and here's a camera Matrix uh and I'm going to give ask you to determine what's the image plane coordinate of the world Point 40 0 so that's going x coordinate of four and the Y and Zed coordinates are equal to zero right if it's it's worth just doing this at your at your at your leisure all right so that's the camera Matrix and that's the calculation that you want to do if you're using mat lab you would have got an answer you would have got a three Vector that looks like that right but it's a homogeneous coordinate so when we want to convert it back to a cartisian coordinate we got to divide the first two numbers by the last number right divide them all by four and we end up then that that point in the world the point that's got an x coordinate of four Y and Z coordinates of 0o appears as a pixel at the horizontal coordinate 712 and the vertical coordinate 912 so if the image was a 1024 by 1024 a megapixel image then it would be down in the bottom right hand corner right that's where that point would be projected to so there you go uh that's how you can model the projection of a camera and that equation that we had back here we're going to revisit that again in a in a few moments so that's how you can do it in mat lab now what's interesting is what I've done now is I've introduced an extra scale factor here I've introduced a Lambda in the front there yeah so it's multiplying everything by Lambda so if we do that then the U tilder and the V tilder and the W tilder will all be scaled by Lambda yeah we agree on that so when we convert it back to a cartisian coordinate because we divide through by the by the by the W Tilda the Lambda disappears so you can multiply that matrix by any arbitrary scale factor and you'll get exactly the same results this is a slightly unusual thing about homogeneous coordinates is that they don't care about scale the result will be unchanged and it's because that's the case then what we can do is we typically write it like this that the there are 12 numbers in that Matrix but because the scale factor is completely arbitrary by convention we make the bottom right corner equal to one we can make it equal to anything but by convention we make that one equal to one so then there are only 11 other numbers that we need to that we need to figure out so that's uh a characteristic of homogeneous coordinates scale factors do not matter now in the in the very lovely Matlab toolbox there is a function that generates a camera object uh and here I've generated a camera uh that's situated uh at position with XY Z coordinates of 0 1 and 2 and then I've rotated the camera so that it's pointing along the xais of the world so I've got a camera which by default points up along the Zed axis I shifted it and I've rotated it so that it's uh it's looking along the x-axis of the world that's what those rotations on the end do and I've created a camera model in mat lab and then I can ask it for the C Matrix for the camera projection Matrix and there is the camera projection Matrix which is the one you just used in that exercise I can give it a point and I can uh call it project function and so it the project function of this object turns a 3D Point into a point on the image plane uh I can do this in mat lab I can ask it to plot the camera for me and plot the object and so you see a little tiny camera icon and a little icon representing the point uh so I can see where's the camera which way is it looking where's the point that that it's seeing yeah uh so I can visualize the camera observing the world in in mat lab amusing okay what we're going to do now is a simplification of what we just did and what I'm going to do now is consider that my camera is looking at a point the point's on a plane in the world and I can arbitarily choose where the origin of my coordinate of my coordinate system is so I'm going to put the the or of my coordinate system on the plane and the X and Y AIS of my coordinate system are aligned are in the plane yeah so Zed is pointing out of the plane X and Y are within the plane so that's that's it up the top there I can put the origin anywhere I like but I've decided I'm going to put it there on the plane so that means then for every point that lies on the plane its Z coordinate is equal to zero get that so down the bottom then is the projection Matrix that Matrix with a C11 C12 Etc and they can those numbers just depend on they depend on where the camera is looking and they depend on the coordinate frame yeah and now I've written instead of XY Z and one I've written xy0 and one because Zed is always equal to zero in this particular example now if that's the case if Z is if that's got a zero there then that's going to multiply out the third column of the Matrix is going to be multiplied by zero I really don't need that column or that or that number there right so I can remove a column of the C Matrix I can remove a row of the vector on the edge and now I'm left with a 3X3 system that Matrix is generally called H it's called a planer homography and what it does is it Maps points in a plane to points in the image there's no depth here anymore so if I know the two two dimensional coordinates of a point in a arbitrary plane in the world I can convert those just using that simple Matrix into the coordinates of a point in the image so this is going to be really useful to you homography is going to be your friend by the end of the PRACK uh I've written the homography here with a one in the bottom corner so it's a 3X3 Matrix got nine numbers in it one of the numbers is equal to one so there's only eight numbers that we need to figure so there it's got five it's got eight unknowns now if we've got a point on one plane and a point on the other plane that gives us two pieces of information so with four points four points on this plane and four points on this plane that I know that gives me enough information to be able to estimate this Matrix H so here's an example of uh of something you can do with this homography technique so I took a picture uh of nraam Cathedral as one does uh and it's big so you're standing there in the Square you're looking up at nram Cathedral you take a picture like that right and so it's pretty distorted because I'm standing on the ground looking up at something that's pretty massive and you can see that it's distorted right it's this the sides are actually vertical but it's got this uh foreshortening effect right so it appears to con uh the sides appear to converge right but I know some things about the cathedral I know that these four points here lie on a plane right they lie on the frontal plane of the cathedral more or less yeah so if I pick those four points and I say okay they should they should be on a plane but and I can draw a I can draw a shape through those and I can say that that shape really should be a rectangle yeah if I was looking at it properly that shape would be a rectangle so I've got four points on the front of the cathedral I've got four places where I'd like those points to be in my ideal image of the cathedral and and so two of the pl points haven't moved but two of the points do move so what I can do now is I can estimate a mography between one set of points and the other set of points I've got four points that belong to one plane and four points that belong to the ideal plane uh the ideal representation of the cathedral so in M lab is a function it allows me to estimate a homography I've got the two sets of points I picked them off pick the pixel coordinates off with the I disfunction uh and then I figured out where I'd like them to be where I'd like the red the red dots to be and I estimate the homography Matrix and there it is now I can use this homography Matrix then to warp the image so what I've done is I've applied that homography to every point in the image I've straightened it up now the straightening isn't entirely valid all I've said is that that frontal plane of the cathedral that I know is a plane and I picked four points that should be on a rectangle if I looked at it from a cherry picker they would have been a rectangle and I forced them to be that way but it's going to introduce some Distortion so for all the points that don't lie in the plane where I've done this business there's going to be quite a bit of distortion so if you look at this more closely uh you actually see that on the sides of the bell tower uh there's some Distortion because those points don't lie in the plane I don't my photograph doesn't have the full three-dimensional representation of the cathedral so there's no way I can get a perfect representation of the cathedral from a different viewing Point uh this is the best I can do but it's only correct for those points that lie in that frontal plane of the cathedral everything else will be somewhat distorted but it's not a bad trick uh so in Matt lab I can do that just using uh that function there called home warp so warping is an interesting business uh it's used a lot in uh in special effects and what I've done here is there are the two images uh the output image on the left and the input image on the right and what I've done is for every pixel coordinate in that leftand image every single Pixel coordinate I have pass I have multiplied That by The homography Matrix H and figured out what the coordinate should be in the input image so I've just given one example I picked a a pixel up the top there at coordinate 600 comma 100 I pass that through the homography Matrix and I end up with uh the results 757 and 51.3 so what's saying that the pixel in the left image belongs at that fractional coordinate in the input image right I can't look up a fractional coordinate in the input image but I can interpolate it I look at the neighbors I I work at the integer coordinates I then look at the neighbors and I do some interpolation some weighted averaging to figure out what would be the value at 757 comma 51.3 I do this process for every single Pixel in the input image right I take its coordinates put them through the homography do the interpolation uh and then set the pixel in the output image so it's a little bit timec consuming uh but on a reasonable computer it goes like that um so what I've done is used some very simple geometric information I've used this notion of an homography I've used my domain knowledge about Cathedrals and planes to be able to do this trick and rectify the MH what it's equivalent to is really moving my camera from the blue coordinate frame on the ground looking up like this to a virtual camera which is up in the air looking at the Cathedral straight on and actually from that homography Matrix I can actually decompose it in some ways and actually find out what was the angle that I was looking at the Cathedral at so after I've come home and I've got this image I can work out what was the angle at which I had my camera tilted up above the ground given that the front plane of the cathedral is is vertical it's an old Cathedral so that may not be a valid assumption uh so this is some some simple information that you can back out of just a digital image okay Olympics were on recently you sit there on the television you see pictures of swimming pools and you see amazing things you see world record lines moving down the swimming pool uh you see national flags and names of swimmers draped over Lanes right it's the homography trick so there's a swimming pool and there's a national flag and a name right it's a nice big rectangle and I want that to lay over that uh so the name is a plane and there's a plane in the pool between the two between the two swimming ropes so I can map those points to those points I can commute I can compute an homography I can warp that flag and name Banner right into a set of pixels that'll be kind of Twisted and like and then I can overlay it exactly on the swimming pool on the swimming pool swing pool Lane so that's how that's done if you know the geometry of the scene uh and everything's a plain your life becomes very very easy and that's that there are many cases like that you imagine a surveillance camera looking down at a flat car park right if you if you compute the homography then from any point that you see in the camera you can work out what's its coordinate within the car park uh so there's many cases where cameras are looking at environments that are a plain which brings us to the PRACK so in the PRACK way long time ago you built a robot right that worked on a little sheet it worked on a plane okay and you took you're taking pictures with a camera uh and that results in an image being formed on a plane the image plane within the camera so here we have the camera image it's got its coordinates u and v and there's the worksheet that your robot's going to sit on right and it's got coordinates X and Y and in the camera image we see some rotated Twisted skewed version of the sheet that the robot's sitting on we've got two coordinate systems we've got XY it's the coordinate system that your robot understands and we've got the UV coordinate of things that your camera can see so you already know how to figure I think the the unv coordinate of a Blue Block or a green blob or a red blob you know how to do that yeah the problem is how do you get your robot to reach that because they're in different coordinate frames all right so the answer is homography so we have a point in the image I got P coordinates of u and v another point on the robot worksheet I'm going to call it Q it's coordinates X and Y and I can compute homography that Maps uh P to Q or or the other way around interesting thing with the homography you take you multiply by the inverse of the homography Matrix and it does does the opposite translation the opposite mapping yeah so let's say we've got four points in the image plane P and sorry I've switched that should be Q right uh the Four Points X's which are in the robot plane I put those coordinates into columns uh so I build up a matrix with four columns uh before four image points before World points image points are measured in pixels the world points are measured in millimeters right what's really important is that the points correspond right so the First Column of this Matrix U1 V1 has to be the image coordinates of the point X1 y1 you get what I mean by correspondence yeah they have to be the image plan coordinates of that particular point in the world if you swap swap those around bad things will happen so on the sheets the worksheets that you've looked at so far there have been nine blue dots right these are the calibration dots so in your software if you can find these nine dots right you can measure the the coordinates of the nine dots with a ruler on the worksheet you can find the coordinates of the nine blue dots in the image once you've got those two sets of data you can estimate this homography Matrix once you've got the homography Matrix then you can map the coordinates of the green square into XY coordinates that your robot can reach now I put nine dots on the sheet you only need a minimum of four theoretically minimum is four but you get a better estimate the more the more points that you use so that's why there are nine dots on the sheet okay so this is a really important part of the lab right you're going to use the nine dots to estimate uh the homography between the camera image the robot worksheet and then you're going to use that homography then to map all those other shapes into the robot workspace and then you're going to use the robot kinematics to reach to that XY coordinate and uh put your position your pencil in the middle of the shape that I ask you to reach so is that all clear so the matb function that you need to do this there is the homography function it's in the toolbox got a help page uh there are two matrices uh so p is the Matrix that contains the image points X is the Matrix that gives you the the robot the workspace coordinates and the H trans function then Maps uh image points into into robot coordinates you got to get them the right way around if you compute the homography not from P to X but from X to P you'll end up Computing the inverse homography which will map in the opposite direction so when you compute the graphy as it is up the top that's the homography from P to X from image to robot and you then apply that homography to image and it will give you robot coordinates all right so summary of what we of what we've covered we've talked about mapping points from 3D to 2D we talked about how easy things are when you use homogeneous coordinates and always remember when you do the these things the result is a homogeneous coordinate you always need to divide by the the third the last number divide the last number into all the other numbers throw the last one away to get back to a cartisian coordinate this this trick from homogeneous and cartisian applies to two-dimensional cartisian numbers so X Y becomes X y1 also to threedimensional cartisian numbers XY Z becomes XY Z1 uh the toolbox functions do this transformation this conversion between cartisian and homogeneous automatically for you so you're not going to see it but you might see it in a pencil and paper exercise in a tutorial or an exam for instance homogeneous coordinates are scale invariant that's a very important property and we can map points from one plane to another using this thing called an homography Matrix so that's the sum