Haptic Rendering Explained: Robotics & Force Feedback (Computerphile)

[00:00:00]So we are going to talk about computer haptics today. Uh and I will give you an introduction of what haptics means.

[00:00:07]>> Okay. So I'm thinking of things like force feedback on a controller on a on a computer in a video game or something.

[00:00:14]Is is this a sort of thing we're talking about?

[00:00:16]>> Yes. Uh it is one aspect of haptics. So uh when you think about mobile phones or game controllers, there are typically vibrations. Uh and these are um about your tactile sensation or cutaneous sensations that are about fingertip receptors or >> so when you type on an Android phone it makes >> yes it makes little buzzes uh and when when it rings it buzzes as well. Uh this is one aspect of haptic feedback and the other aspect is canesthetic feedback or proper receptive feedback which uh talks to our senses of forces position. So it is how haptic feedback is transferred through our joints and muscles.

[00:00:58]>> So this this must be the sort of hard stuff to get right. Like when you shake someone's hand, you sort of feel how hard they're shaking and you shake back.

[00:01:04]But a robot, how does that do that? Is this the sort of thing we're going for?

[00:01:07]>> It is similar. But uh when you think about touching a human, it is actually very much complex. So the thing about haptics is is really distributed. So we get information from multiple channels because there are a lot of different sensors on our uh body especially in our hands and we combine them. So it's a combination of how forces for example how much you're squeezing which is about pressure >> and then how much you're moving for example which is about the forces that you're applying to. And robots today they are only capable of realizing some of it because we don't have uh sensory technologies that could uh sensory and actuation technologies that that could make it very comparable to human haptics.

[00:01:54]When I talk about computer optics it is a little bit different than how you would realize the sense of touch with a robot. is about how you can create a virtual world like a simulation a computer graphics game for example and you implement a device to feedback forces to you. So when I talk about haptics today uh what I refer to will be kinesesthetic feedback force feedback uh and what we are going to talk about is how these forces are computed when you see a scene and this scene would be a simulation. Uh what it can reflect to the real world as well. Uh and within this simulation we assume that there are objects that we want to touch. So this is if you can see a haptic interface >> and it moves in uh in the 3D space. So this is a uh grounded haptic interface which has actuators >> and what it reads is the position and the orientation although it doesn't uh really drive torqus at your hand. It only gives you uh force feedback. This is an impedance type device which means that it reads positions and gives you forces. There are other kinds of devices as well. Uh but this is the probably more uh most common of uh haptic force feedback interfaces that you can have.

[00:03:19]We also have different interfaces which you can for example wear like uh exhaust skeletons or force gloves u and they do operate on similar mechanisms as well.

[00:03:30]So we are going to talk about uh how we actually use this almost as a mouse and then go and touch an object in in a on a yeah on a virtual space on a 2D screen.

[00:03:44]Uh so I can um actually talk about this as a uh sensing actuation problem where you sense the data which is positions in this case and you have then a simulation loop which consists of a visual loop and a haptic loop and then you have the actuation. Uh the visual loop um operates typically at a very low frame rate like 30 Hz 60 Hz. Uh however the haptic loop in order to be stable it needs to operate at a much higher rate which is traditionally 1,000 hertz.

[00:04:20]>> Oh wow. Okay.

[00:04:20]>> So every millisecond you have to make a computation and send it back to the device so that the device would give you something meaningful and stable. Let's say we have a haptic device here which is similar to this that can read data from encoders. So we have encoder data which means I can read the state of the robot arm uh which is typically the joint angles.

[00:04:46]>> Mhm.

[00:04:46]>> And I know the model of the haptic device. So I can compute the position of the tool tip or the endector uh or the interface point as we as we call it. So the encoder data uh is fed into something that we call forward kinematics and this is very traditional in robotics. So you can get the state the joint angles of a robotic device and because you know the model you can compute where the tool tip is and this is the easiest uh easy problem. Then forward kinematics computes a position.

[00:05:24]That position is used to interface with a simulation. And within the simulation I have objects. These objects can move.

[00:05:32]These objects can be static. Uh what I need to do in order to enable haptic rendering is uh very similar to what we do in computer graphics to look into collisions. So the position of the tool tip is taken and that is checked for collisions with objects within the scene. So I will call this collision detection.

[00:06:00]>> People will be thinking of again video games with this won't they? You know >> it is the same principle.

[00:06:04]>> Same principle.

[00:06:05]>> Uh so haptics and computer graphics they are extremely similar. So many of the methodologies especially in haptic rendering uh is adopted I think from computer graphics. Some differences is in simulation you want to show visually something that is compelling. The aim of haptic rendering in contrast to visual rendering is to make someone feel something real >> physical. Yeah.

[00:06:31]>> Physical. And the one other difference is uh the frame rate. So this needs to be really very quick whereas this tolerates failure a little bit more.

[00:06:42]Tactile sensations or haptic sensations are much faster.

[00:06:46]uh we wouldn't really see if some frames are dropped or if they are injected with some information. Uh we would feel better discontinuities and they are what we uh put meaning into when understanding the world explore the world but also from a robotics point of view the discontinuities make the robot unstable. So the behavior is not realistic. So collision detection takes some information from simulation which is where the objects are for example the tetrahedral or triangular models that are cons comprising the uh simulation elements. We have the position of the uh of the device the tool tip and we compute a contact point. Once we have the contact point we need to compute a collision response. So in graphics a collision response could be you push one object and it moves and it touches something. So there's a cause and effect. So you put you issue a force and there's a physics uh model behind the simulation which makes things happen.

[00:07:50]And this is the same principle. You touch something and you generate a force. The object can move in response.

[00:07:56]The collision response would compute a force, a force vector to be felt at the tool tip or in my hand if I was pushing it, if the system was completely transparent.

[00:08:09]Uh, and then that force is sent to the simulation and to the haptic device. The force that you send to objects in the simulation and the force that you should send to the device may differ.

[00:08:22]So you could have some force modulation here which could be scaling or putting it into the ranges so that you don't you don't harm the device or >> it could be >> so it's a bit again in like graphics you know you scale to what the monitor is capable of displaying rather than I don't know burning it out.

[00:08:40]>> Yes, exactly. It is very similar. So you move mouse your mouse for example five pixels but then your agent moves five mters.

[00:08:47]>> So there's a scaling over there. So I will draw two arrows here. One is the force sent to the >> simulator if anything is moving and the other one would be uh the force that you should feel at the tool tip.

[00:09:03]>> When it comes to the device, what I can control is the torqus that I send to the motors. So there's another block here we call the inverse kinematics.

[00:09:17]Uh and this is the harder problem when you compare it with the forward kinematics because it solves the inverse problem of what should be the state of this robot. It's not a hard problem as it is uh MP hard or anything but uh it is less straightforward than forward kinematics and we do have very established methods for the inverse kinematics as well. uh and this computes torqus as I said this kind of operates at 30 htz let's say and this has to operate at at least 1,000 >> oh sorry so it can be more than that >> it can be more than that uh but if you drop down then you would start feeling it's not really stable and in uh force feedback robotics as well that becomes a bottleneck especially if you're doing if you're dealing with for example plate systems if you're sending a robot arm to a space station and we are doing a delayed teleoperation and so on. So this is our uh the bottleneck.

[00:10:19]>> Okay.

[00:10:20]>> Okay. So this is exactly what we are doing and uh I will now talk a little bit about how we do the collision detection and how we do the collision response. So we could feel some objects in a scene. So as you may know if you're interested in computer graphics uh objects are discretized. So you may have a polomial polygonal object which consists of triangles. Uh and the most naive way could be I have this point this mouse cursor and I have thousands of triangles. In order to check for a collision, I could just check with every triangle. But then this is very slow.

[00:11:04]>> Yeah.

[00:11:04]>> Especially if you have to do a thousand operations in a second. This gets very slow. So what we do in haptics is um we use bounding boxes and we use u hierarchical bounding boxes which is which is similar to this oct three algorithms where you have a bounding box for an object and then you check whether there is a collision with that bigger bounding box.

[00:11:26]If not you should feel nothing with a haptic device. It should be really transparent that as if you're holding nothing in your hand. And if you have a collision, you start dividing it into smaller sections. So you don't actually search the whole space, but you only look into a small set of triangles.

[00:11:46]Uh and why this works is um also because of the observation that uh once you detect where you are then the surf space really becomes very very small >> because humans actually can't move that that fast.

[00:12:04]Okay. So that makes the search problem uh very very quick.

[00:12:10]>> Uh and once we know we are reaching an object, we have a surface that we are testing against u the collision response the computation of the collision response the forces is the is the problem. So I'll go over the simplest example here um for a point to line point to surface um collision detection.

[00:12:35]Um you could know when you're not touching the object and this is your haptic interface point which I will write as hip. The force in this situation should be zero because you're outside and you shouldn't really feel anything. The device should feel really empty as if it's not uh existing which we call uh transparency in robotics. And when you touch the object in haptics we allow the hip to pierce into the object.

[00:13:05]In graphics you would see something on the object. So visually you should see the object as here. But in order to compute forces, you need some kind of formulation that allows you to compute uh >> to see how how much forces Yeah.

[00:13:22]>> Yes. you should apply to uh pull you away from the object. So in uh practice what you would do is you would allow the hip to pierce into the object and then you would place a little virtual spring here and then the force you compute would be simply the spring force. You would set a stiffness parameter K and that would define your material properties. So I can show you a very simple demo. So what you see here uh is now I I turned on this uh demo where you have three different surfaces.

[00:14:03]So one of the surfaces is characterized as being soft, the other one is medium and the other one is hot. Uh and if I so this is my this is a representation of my cursor here which is very similar to the uh endector >> and when it touches very slowly you could see a maybe something like a springy >> movement. Yeah. And if you uh look at the device, it's kind of really springy.

[00:14:28]You could see how the force is modeled.

[00:14:31]So it does act like a spring. If it's if I go to a harder object, then you could feel it more >> and the hardest object then you would really feel that your your accelerator is sliding on.

[00:14:46]>> So inside there, all the motors are basically saying don't let that probe go any lower down. Right. Yes, they are saying that because we are I am very slow. So I'll also show you something different here. Uh because this is all about movement, right? I had a haptic interface point and I'm moving it in space. So I'm really very slowly approaching this. So I'm not really piercing the object a lot. Right? So you don't actually see that this is really getting into the you know the you could see the the sphere pretty well. Maybe you see some kind of piercing motion that you see the bottom is getting in. But what happens when I become a little bit maybe faster?

[00:15:31]I'm not really holding it very much >> bouncing.

[00:15:34]>> So it's bouncing.

[00:15:35]>> Yeah.

[00:15:35]>> So I'm I'm actually making it go into something instable.

[00:15:41]Uh and you can also see it's sometimes popping through, >> right? So it's getting under the object where it shouldn't be allowed to. And why is this happening? Um because it's just positions the um the rendering algorithm doesn't know where I am. So what happened over there is I was moving the hip and then I was moving it very fast. So I was actually going really like here. And then the force that you compute gets quite large and it makes you pop out of the object, >> jump and bounce back. And then this can actually escalate as if you know it can escalate really badly. In this in this case it didn't, but it can also get very very unstable. So you have to terminate the simulation. Another issue uh that happens is for example when we have a very thin object uh which was also the case in this example uh where you would compute forces that would push you away from the body of the object when you approach from this direction and away from the body of the object if you approach from this direction. But if I'm coming and if I'm really fast, I could actually poke through. Another thing that could be a problem is if I have a complex object, then I would represent it as a few subsections.

[00:17:21]And in sub in this subsection, for example, if the hip is here, I would find the closest surface and compute a force that pushes me in that direction. So this is the forces that you should feel for this surface. And if I'm here, I should be actually feeling forces against this surface. So that my uh my interaction would feel like I am on these points. But if I am approaching and if I'm moving from here to here, then you would feel like somebody's popping me out from the site.

[00:18:01]>> So these are um not realistic. This is not something that we would expect from a good horse rendering implementation.

[00:18:08]>> Would that be helped by making it faster?

[00:18:12]um it could be helped but also it is about u the characteristics of the device. Each device has a limit on how much stiffness that they can tolerate and uh obviously about the uh the frame rate as well the server rate. Um so I can still pop through objects and it is not very desirable. Um so how we uh eliminate these in haptic rendering is through proxy algorithms where um I actually have almost shown you the first idea that comes to mind. If I am rendering force, instead of computing just the distance about this uh this piercing distance, I could compute an error and I could try to minimize that, right? And the error would be the error between where I should be and where I am literally. So instead of actually trying to represent the distance to the line, I could define a proxy point which would solve the issue of jumps and I could compute a spring between the proxy and the uh and the object. So at any time you could look into where the proxy is by looking into the collision detection and you could compute a point where the proxy should be residing and make the proxy walk over the object and make your hip only attracted to that so that you would eliminate the >> trying to get back to where the proxy is at all times.

[00:19:56]>> Yes. Okay. So this is sometimes called uh an god object algorithm uh sometimes a proxy based rendering um and it is a much better way to display forces. Um so that's all uh and this proxy proxy methods is a method for haptic rendering but they also uh are used for human robot collaboration. So uh in our research we typically use proxy points for human human or human robot collaboration where you have a robot interacting through an interface point a human interacting through another interface point and they get connected through a proxy which then connects to the task and operates it.

[00:20:42]And this way you could do a lot of uh intelligent things like assigning dominance behaviors to the human and the robot, separating their inputs from each other so you could analyze them better.

[00:20:53]Uh so that the proxy becomes not a proxy for rendering only but a proxy for agreement between the two actors.

[00:20:59]>> This is like an ongoing open problem.

[00:21:01]It's not something that's considered solved, is it? Or is it just there are different ways of approaching it?

[00:21:06]>> Um I think you you could consider haptic rendering. uh in some ways solved. So there are mechanisms that you could simply do things but there are still uh issues for example when you represent material properties such as uh what exactly I am feeling because we didn't really talk a lot about friction or uh material properties that make this feeling realistic. what what is it that I feel when I touch a sponge >> something like this or something like a table. So the there there are libraries to uh enable this kind of learning of object properties. Uh also transferring the haptics over distances and so on is a I think still an open problem. So there are coding mechanisms just as we do video coding for haptic coding but uh it is still still open. We can't really tolerate very delayed communication with haptics. So there's a lot of work going on in teleoperation architectures but also packaging the haptic information and sending sending in distances.

[00:22:23]If the goal is to move from start to goal and if I can't actually uh have an analytical solution, I'll do pink.