Numerical Analysis: Errors and Propagation in Calculations

[00:00:06]so we talked about numerical analysis and how important it is to investigate errors but the natural first question is what is an error so here is the situation we have some quantity let's call it X that we want to know but for some reason instead of X we actually arrive at some approximation let's call it X hat now now error intuitively would be somehow related to the difference between these two numbers or in other words the distance between these two numbers so for instance this could be a reasonable notion of error I would take the precise value and subtract the approximate value the one that I actually have on hand but of course somebody may say why not the other way around and that's a good idea as well so this could also be error and perhaps somebody says wait a second we are talking about distance here and distance that's an absolute value of the difference so that's another good candidate for the error and if you go through various books on numerical analysis you will find that authors actually pick out of these three of course you pick one and then you stay with it because you want to be consistent but you will encounter throughout the books all three possible versions of the definition of so-called absolute error now intuitively I am a little bit partial to the last one because it provides me with the information that I want how far is my guess from the actual value well absolute value is actually not really let's say convenient in calculations so I think that most authors including me checking out and do not use this one instead to use the version without absolute value and there it's more or less a toss of a coin uh there is there is not really much difference between these two approaches but of course you have to settle on one and I decided to settle on the first one so for me and in this course and therefore also for you this first expression will be the absolute error of some estimation we have a quantity that we want to know it's estimation and that's the absolute error I'm not actually losing on information because when we will be investigating absolute error I will be using this nice formula for calculations but then after I find the answer I just stick it in the absolute value and I get the information about the size of the error anyway okay so this is like a compromise the best of both both worlds good calculations and I get the answer is this NOA reasonable the answer is actually yes we do encounter it in many situations let me let me show you here is an example which could be also seen as experiment okay you have a stick and you want to know how long it is so you get a ruler and you line up one end of the stick with zero and you look at the other end of the stick uh here we go this is a ruler in case you didn't recognize it and let's say here is Mark 13 here is Mark 14 if I'm going by meters but you can pick whatever unit you want now in the middle somewhere here there's a mediumsized bar and then there should be four small ones uh let's see I guess that could be about it yeah that looks reasonably like a school ruler and now here at zero I line up one end and the other one comes I don't know here forance so I look at it and I would say okay I think that my stick has length I guess 13.6 CM or if you want L is 136 mm now obviously this is not the precise length of the stick perhaps I could also make a guess about another digit but it would be a guess but for with this I'm sort of reasonably content it would be dangerous to venture any farther and it's also equally obvious that I'm making an error now what kind of error that's a good question and of course if you don't know how to calculate something you go to definition so here's the definition uh to find the error of my guess I would have to take the actual length of the stick where do I get it now this is a problem fundamental problem if you want to find the absolute error you need to know the actual value the precise value and if you know the precise value why would would you guess another value which is not even precise so this doesn't really make much practical sense so instead what we very often do we try to find an upper estimate for the absolute error and that's something where we have a good chance why well let me see how did I get my measurement I looked at the end of the stick and I went to the nearest tiny bar now assuming that I'm not like totally clumsy and that I do it properly all the time it seems that the largest error that I can make is exactly half the distance between the two neighboring bars so the error is bounded by 0.5 mm and this is typical uh in several meanings actually first of all this is typical for applications we get an upper Bound for the absolute error and many authors but not all use lower case e for that estimate so that's something which is very typical and another typical thing is that you get this type of error when you are using measuring devices now if you are measuring something which is continuous in nature like weight length uh energy things like that temperature uh then you have to use a measuring device which has granularity it is never precise completely precise So when you buy uh let's say a slightly more advanced measuring device there should be a sticker which actually tells you what is the Precision in terms of absolute error so this absolute error that's a very useful notion which appears pretty much every time you are asking how large is it how you know and so on so uh this is natural now what kind of information can I get out of this well let's have a look at the formula here when I rearrange it I find that the actual precise value is the approximate value plus the error as we talked about it I unfortunately do not know the error so this is not practical but it will be helpful theoretically we will see soon instead what I have is this I know that the error in absolute value is bounded by some estimate of my X and the error is the absolute value of x Min - x head and it's not really difficult to rearrange this into the following form the precise value must be hidden between X hat plus the estimate ex x and x hat minus the estimate ex so we are getting in range for the precise value out of our measurement and the knowledge of the uh absolute error and this actually is very popular so there is a shortcut notation for it X is X head plus or minus e x and that's a very popular engineering notation uh for instance in my case I would say okay the length of the stick is 36.0 plus or minus 0.5 mm now Engineers they have some rules of how to write this properly and of course one of the most obvious rules is that the unit here and the unit here they have to agree so that these two numbers can be put together okay also visually you just look at it and you see what what's the range what's the uncertainty that's a good word there is actually a theory about it but we will not go into it here okay so it seems that absolute error is a very good thing to know but the key question is you do some measurement you learn about your absolute error and now you want to know whether your measurement is reliable or not is it good or is it worthless so let's have a look at the situation imagine that you still have some measurement measuring device which has absolute error at most 0.5 mm and now let's say that we measure something let's say something practical uh let's say the distance between my bed and the bus stop okay that's a very important piece of data uh when I'm optimizing my early morning routine so I look at it and um I don't know 400 23 M now following the advice that we obtain over here or over here I could write my findings as follows X is equal to uh I'm careful I need it uh in millimeters so it will be 423,000 plus or minus 0.5 mm that should be the conclusion and I look at it and I would say this is really precise yes in fact I would say this is really suspiciously precise because when I see those three zeros here this is very unlikely when I see a result like this I would rather guess that the Precision was something like a meter or let's say half a meter or something like that obviously somebody was just estimate well I was just guessing so I was suspicious rightly uh but if this really were a Precision of my measurement that this is even needlessly precise because Precision costs m the more Precision you need the more expensive measuring device you have to buy so I would say that here I was actually wasting my money if I really measured the distance between my bed and the bus stop with Precision 0.5 mm I strongly suspect there is some mixup in the data on the other hand imagine that I'm measuring the thickness of my hair now I actually haven't done it but I looked it up on the internet and it seems that if I obtained 0.08 mm then this is is not really Off the Mark could be something like that now again let's write it X is 0.08 plus or minus 0.5 mm now does it look like a good measurement definitely not because the error the width of the range for my value is much larger than the actual value itself the one which I calculated or measured so the error actually overwhelms totally the data which I obtained from my measurement I mean if you subtract it would even seem that the hair could have a negative thickness well that could explain a lot but still I really find it highly suspicious and in fact such a measurement would be considered highly unreliable and not usable at all in applications you definitely do not want the error larger than what you measure so the conclusion is that if you want to judge the quality of your measurement you shouldn't be looking just at the absolute error but compare the absolute error with the number that you obtain from your measurement that is the right measure of reliability of the result and I think that we are ripe for some slide here we go I can see the definition of the absolute error and I hope it's the same yeah it's the one that I picked here and then I can also see the definition of a relative error which which is exactly what we talked about here comparison in mathematics comparison is done usually by division so we ask how large the absolute error is that's the absolute value I've been talking about it and divided by the magnitude of the precise answer and there you can see the problem again typically it's impossible to find the relative error but very often we have good information about the upper bound of the relative error down here we can also see a note about the upper estimate made for the absolute error which is commonly denoted lowercase e uh but not universally and what we really want is the upper estimate this is common sense because we want to keep the error down and you keep things down by having good upper estimate uh it would be also nice to have a notation for some estimate of the relative error but I'm not aware of any notation which would be let's say wildly accepted so I didn't put there anything we will just say upper estimate things like that okay now let's have a look at the situation we will soon see that there are possible there are many settings where we actually have information about the relative error we have upper Bound for it so can I get some information from knowing the relative error and when I look at the formula I can see right away that the absolute error is the relative error times the precise value which is not exactly convenient as we already saw about twice so what can I do if I actually do some practical application some practical calculation and I want to get some information but I don't know the precise value X well there are two approaches the first one is very useful but one has to be very careful it's an engineering approach an engineer would say Okay I want to know the absolute error now what happens if I replace the precise value with my approximate value do I make a error the answer is yes but small if I'm lucky okay sometimes perhaps I have information that my error is small which after all I can see from the relative error the relative error should definitely be a small number unless I did something really bad okay and if the relative error is a large number then I can throw it out anyway so I can expect relative error to be tiny and therefore these two numbers should be close hopefully now as you can see this is dangerous but it's also very practical so it has to be done uh let's say with experience and with knowledge of what can go wrong of course for a mathematician this is not exactly okay so let's have a look at how mathematician would handle it so let's put it aside as a mathematician I want precise calculations and I want to pass from X to X head that's my motivation so I would say okay is there any connection between x and x head and the answer is yes I've got this formula the precise answer is the approximate answer plus the error and now I will apply the triangle inequality to this part and I will get that the absolute error an absolute value is Epsilon X and now the tri inequality forces me to put inequality here and I get X head Plus e x an absolute value each of them separately uh I had to put the bracket here because the relative error multiplies the whole group I have to make sure that it still multiplies the whole group now I multiply out and I move the absolute error to the left little trick what do I get I'm moving it to the left I try multiply it out and on the right there will be Epsilon x * X head and finally I factor out the absolute error and I divide by whatever is left I'll try uh factor out so I get Epsilon X over I pull it up I guess oneus Epsilon x x head is this valid the answer is uh well not in general because I've been dividing by this number and and I need this number to be positive so that I can keep the inequality uh is it reasonable to assume that this is positive the answer is yes because if the Epsilon X that's a relative error is one or even more then it means that my estimate X head is worthless and I don't want to work with it anyway so if this x head estimate is any good then definitely relative error should be close to zero and therefore this number is positive so under this assumption this calculation was correct this is mathematical formula which is reliable we just proved that it's true let's compare it with the engineering formula they are remarkably similar and if my measurement is really good then the relative error is almost zero and this denominator is almost one we can ignore it and we get the engineering formula okay so that's a nice relationship between a precise mathematics and Engineering point of view which is not precise but it sends the right message and it's important to understand things if are using numerical analysis in applications you should understand what it says okay so it's useful to also have this kind of information now we just saw that there is a relationship between the absolute error and the relative error because once we have the absolute error some Bound for it either the precise one or the engineering ones we can start talking about intervals of reliability things like that so let's ask crucial question let's say that I learned that Epsilon X is a specific number certain number what does it tell me about my measurement that's a very good question and as I say we now prefer to see intuitive information so let's skip precise mathematics for a while and let's have a look at an example imagine that I measured something and the answer which I obtained was let's say 2 3 4 5 6 6 7 8 uh n let's make it nice okay and then from some Source I learn that the relative error is okay I should write bound it but let me write equality for Simplicity 1 over 100 which is by the way 10 to -2 which we of course know but I think it's useful to see both informations in the picture uh so what does it tell me about the precise value that I I was after what is the range for it well let's have a look at it this is my measurement 2 3 4 5 6 7 I should have taken shorter number okay what is the absolute error because we need the absolute error to see the range over here or over here so the absolute error is bounded by a number which I obtain by taking the relative error and multiplying the estimate with it the approximation so I take this approximation and I multiply it by the relative error what does it do to it well it will shift the decimal place by two places to the left so the decimal dot goes to two uh goes two places to the left which means the number as a such shifts by two to the right and it will be 2 3 4 5 6 7.89 okay so that's the error so what is the range for the actual value for the precise value X well I look at this error and I add it or subtract it from this number and when I add it and subtract it it will change the digits here so these digits they can change I don't know anything about them but these two will not change definitely so these two digits are there and there are some other digits and this is something something that I know for sure so my conclusion is that if I have some measurement and I learn that the error is 1 over 100 or 10 to -2 then it seems that I can trust two digits of this answer the first two digits interesting what happens if I learn that the error relative error is 1 over th000 well it seems that I can repeat the same trick and I find that the error the absolute error will change it by okay let me shift it by [Music] three and some things are happening here I don't care and when I again repeat the same trick repeat the same reasoning I will find that the precise answer for sure starts with 2 3 4 and then there will be some digits so it seems that here I can trust three digits now can you see the pattern 10 to -2 two digits 10 to -3 three digits or two zeros three zeros two digits three digits this is very nice this is the intuitive meaning of relative error if it is in this form but since we are looking for upper estimate usually we can always perhaps add a little bit more so that the estimate for l has this nice shape another important thing is that if you can decrease relative error 10 times you will add another reliable digit to your result or vice versa if doce some let's say um accident your relative error is multiplied by 10 then your result actually loses one digit of reliability okay now I would like to write it as a theore because it's important but I cannot write it as a theorem because we don't have reliable digit as a mathematical notion and it is possible to Define it mathematically but when you try it you will find that it's actually very difficult I don't want to open this question even it's a kind of worms I I don't want to go into it so let's stay on the let's say intuitive level okay here we go if the relative error is bounded by number 1 / 10 to P which is 10 to P then my estimate X head has P reliable digits and this is really informal statement not a mathematical statement so I will just erase it quickly so that nobody can see it but let's appreciate it but before we appreciate it notice this big gap here this is not usual space between words I will have to do something there uh let's have a look at the situation where the relative error is one over and now I put four zeros when I put four zeros the situation goes like that what 2 3 4 5 6 7 8 9 I shifted by four 1 2 3 4 2 3 4 5 and now imagine what happens if you add if you add these two numbers then you will get something more than 10 you carry one it comes here things start getting really difficult and if you're unlucky well perhaps I will add another zero so we can see it better yeah it will be better so this will be 2 3 4 when you get unlucky notice that I have 1 2 3 4 five zeros so perhaps I could trust five digits but it's not true because there is this carry over one which comes here creates 10 carries over and suddenly I get seven here as a possibility so I can no longer trust five digits but only four but perhaps you can easily imagine if these were nines then this sare over one would make a domino effect and change all of them so that's another reason why I cannot make this into a mathematical statement because it's not true actually so just to be on the safe side I will put P minus one digits here so when you have relative error 10 to negative p you have P minus one to be on the safe side reliable digits this is still not true but it's almost true in most cases this will work for you so this is the intuitive meaning of a relative error it actually goes both ways and to see that we will need a lot of room so we talked about absolute errors being related to measurements now relative errors are closely related to the way numbers are stored in computers and calculators now you probably think that there is little problem in storing information in the form of integers and perhaps fractions and that's essentially true unfortunately for engineering calculations we have to go beyond rational numbers fractions things like that we also need some irrational numbers and for calculations to work well we need them stored in a decimal form now when you take an irrational number and write it as a decimal number it has infinite expansion so how do store it in a computer that's a problem there is a trick for it of course let's have a look at it so I have a number let's say 13 2567 and I want to store it in a computer so typically computer would first fix the position of the uh decimal point and there are actually two ways or two wi spread possible positions for the decimal point and for this course I chose the one which is behind the first significant digit so I would write it as 1.

[00:27:05]32567 and now I have to add time 10 to one the competitive uh school of thought thinks that the best position is in front of the first decimal digit the first significant digit and also you can develop the theory in a similar way I I think that this one is slightly more convenient and also slight slly more widespread so I'm sticking with this one okay so we have a digit nonzero digit that's important decimal point and some digits and the exponent which is Shifting which is positioning the decimal point to the right place and now here comes the trick we need to store only a specific part of this decimal expansion because it could be potentially infinite and perhaps even this one is too long it depends on the memory that we have if you have a very very tiny computer then it can perhaps store only four digits so if the computer can only store four digits it would go well that's a good question actually again there are two possible approaches in some systems they use so-called truncation and truncation works easily you just ignore whatever you don't want so I want four digits let me put it here p is equal to four I want four digits so it's 1.

[00:28:31]325 and I just ignore the rest * 10 to 1 the other possible way is rounding and I would say that this number is 1.32 and here I am going to round up to six times 10 to one now in some systems you can really find this truncation and the advantage is that it's quick simple no problem there uh disadvantage is that it's slightly less precise so for engineering calculations especially if they are important and they are all important of course you would prefer rounding now before we get into the format and so on let's address one interesting question uh what if I ask for this number to be stored or on an even tinier computer with a really tiny memory okay which which remembers just three digits perhaps I will start with slightly modified example let's say that I started with 1.

[00:29:33]325 what do I store when it comes to three digits well when I was at elementary school they told me that five which comes after the last storable digit should be rounded up all the time so at elementary school I would say 1.33 and the teacher would say okay that's nice and you know check the answer and I would get points or whatever however this this is dangerous why rounding works like that you have two digits or two numbers that you can remember I can remember 1.32 and I can remember 1.33 and that's all I can remember now if I get a number which is somewhere in between then if it's closer to 1.32 I will round it down if it's closer to 1.33 I will round it up so that's quite obvious however 1.

[00:30:28]325 is exactly in the middle so it has exactly the same distance from these two numbers now if I keep rounding these numbers all to the top then I'm actually introducing a bias into my calculations because this is unfair these numbers are exactly in the middle and I'm pushing all of them up and this can be also dangerous because if you have a bias in calculations it could actually shift the result in the wrong way this can be a serious problem so in engineering what they need is to round such numbers sometimes up and sometimes down ideally have the time up half the time down but this cannot be achieved in such a way because you would have to keep track of your roundings you would have to store somewhere in the memory information how many times you round it up how many times you round it down this is Impractical so instead Engineers came up with an interesting idea they said okay you round in such a way that the resulting number has an even digit at the end which is convenient you can divide by two it's very nice so actually this is not proper engineering rounding we should be rounding it to 1.32 by the engineering standard now quite frankly this is totally off the topic of this course we don't have to worry about it but I think it's curious that people in applications people who work with real life calculations they actually see things differently compared to what I learned as a student so yeah this is just an interesting side note let's not worry about it let's worry about this okay we have a number and we change it into a number which has a specific form and we keep just a part of it this is called floating Point representation and if I remember correctly what I was typing we should be seeing it right here and we do so when you have a number X in computer it is replaced with a different number which is called FX very often in in books and it has a specific form you specify at the start how many digits you want to store now these digits then can be anything between zero and N but the first digit cannot be zero obviously uh there is one exception of course if you want to store zero you simply store zero no problem there but for the other positive numbers for positive numbers this format is quite clear key question instead of this number I'm storing this number or this number perhaps I'm making an error definitely you can see it here how large is the error let's have a look at the situation closer I have my number and let's say that I already shifted to the decimal dot so it's in the following form there is the first digit dot the second digit the third digit and so on then there is the P minus first digit and I still remember this one p digit and P plus first digit and so on many many digits and then there's the shift by some exponent in my computer this gets stored or in my calculator as a floating Point number which starts in the same way now actually that need not be true because if we are using grounding then there should be there could be this car over one traveling and domino effect changes even this one so I'm going to simplify my life now I'm going to ignore the situations with carryover ones so so let's be optimistic I stored these P minus one digits which is what I did over here I stored the first three digits easily but the fourth digit there I have to worry about rounding up or down so there will be some digit let me call it capital DP and it could be either little DP or DP + one okay it depends on the tail here now what is the absolute error and there is of course this shift the absolute error of this number I obtain by taking the precise value and subtracting the approximate value so I'm subtracting I'm subtracting and of course I can see that all these places they cancel these digits so o o o o how many O's the answer is p minus one because I can see this is the first one second one third one and so on uh at the end of course there is this 10 to e now the question is what should I put here that's a very good question and to simplify my life I will ignore this decimal part here let's imagine that the decimal dot is somewhere here so I'm just comparing two numbers and I know I want to know the difference now there are two cases one of them is when I'm rounding down so Capital DP is the same as little DP and then when I'm subtracting then this taale actually tells me the error of this approximation now what can I say about this error well remember we are talking about the case when I'm rounding down if I'm rounding down then this digit can be one or zero 0 1 2 3 4 and even five if there are all zeros afterwards but not more so this digit cannot be more than five and therefore the error cannot be more than 05 and now I have shift the decimal dot back I don't have to look look at it intuitively anymore okay or the error could be less so let me put it this way that's the case when we are rounding down what is the error when you are rounding up in that case I'm increasing this digit by one but in order to do that this digit should be five and something behind or six or seven or eight or nine so it's actually really close to this new digit as a number okay if you imagine there is this number DP there is this number DP Let's ignore whatever comes in front of these two guys okay because it's the same so we can ignore it just let's focus on this if I look at this taale in order to do the rounding up I have to be in this range so the error which I make by rounding up cannot be more than half the distance between these two guys and half the distance between these two guys is 0.5 okay this fits uh if you do not see it try it with with some numbers with some particular numbers concrete numbers and you will see how it works okay anyway this is the absolute error what is the relative error well by definition this should be the absolute error divided by the relative error uh sorry by the actual value the precise value let's estimate in the numerator I have an upper estimate for the absolute error 0.0 O's how many O's the answer is p of them first second third P so there is p Zer and five like that time 10 to e divided by and I don't want to put X there because it would be in the answer and I don't don't want it there I just want one nice real number preferably so what I'm going to do I will replace this x with a lower estimate okay I want to increase the fraction and increase the fraction by making the denominator smaller so what is the natural lower Bound for this number I will just take all these digits and I push them down as low as they go so these digits can be pushed down to zero but this digit can be only pushed down to one so the natural lower estimate is 1 point and some zeros perhaps time 10 to e now let's see 10 to e cancels here I'm dividing by one so I'm essentially not dividing and here is five times and how many times do I have to shift the decimal dot so that it appears here behind P behind five and the answer is p i I said it uh I've been thinking about it already so it should be like that and again if you don't see it right away try it with some concrete numbers 0.05 and you try to shift the decimal dot you try it few times you will see that it works okay so what we have here is an upper Bound for the relative error when numbers are stored with P digit precision and that's exactly the relationship I've been talking about every time you enter some number in a computer or calculator this computer or calculator has some P digits this is a fixed parameter of the system that it remembers and therefore you can count on making an error relative error of this size now let's see Vis it this intuitively it seemed that if the relative error is of the form 10 toga P then I can trust P digits uh okay P minus one but let's be generous now I can trust P digits here we found that if we can store P digits and when I'm storing them I'm trusting them of course so if I can store or trust P digits then the relative error is 10 to negative P again I'm losing actually one power here this is less than a Reco and if I put 10 instead of five to increase it it cancels here and I get this so again there is a shift by one okay so it doesn't work precisely both ways but each way I'm essentially losing one digit or one one power of 10 so this confirms that there's a really close both-sided relationship between how far I can trust a specific number and the relative error so that's a very important confirmation of our intuitive feeling but it's also a very important information when you store a number with a computer there is a certain precision and you cannot really hope that you will get anything better this is the Baseline Precision relative error Precision for numbers that you store in your computer of course if you are storing integers that's another story but once you store decimal numbers you are in trouble when I was young and naive I thought that perhaps if I store 1.8 then actually I'm not making any mistake well actually my calculator when it sees number 1.8 it changes it into a binary code and when it changes it into a binary code 1.8 happens to have infinite expansion so there will be infinitely many zeros and once and the computer or calculator stores just a few of them and here is the error which I'm making so even if I put a really cute number into my calculator I'm still making an error immediately with this relative error as an upper bound unfortunately of course if I chose a different number which is cute perhaps it would be cute also after transforming into binary code but I'm not good enough to do the calculations in my head immediately so as essentially the bottom line is when you are putting numbers into a computer this Epsilon this Epsilon that's something that you have to count on this should be appearing immediately most likely in your calculations this error okay which brings me to the last topic of this error chapter or the error topic uh we put some numbers into our calculators oring to computers and those numbers gets distorted by some relative error that you have control fortunately but then you start calculating with them how do these errors on the input influence the outcomes of your calculations that's a very good question and very important question so that's our next topic let's make some room imagine that you need to add 4 over9 and 4 over9 now if you do this calculation with a pen and pencil you will get 8 over9 and that's a precise answer answer if for some reason you need to do this calculation in our calculator and let's say this is a very tiny calculator with a tiny memory just two digits 4 over9 would transform into 0.44 and that's it now we know that the actual value is 0.4444 44 and so on it's a periodic number but the computer doesn't know it or calculator it's just 0.44 and another 0.44 and when this is added you will get 0.88 however the precise answer should be 0.89 so you can see that the answer is not precise and one of the things where things got wrong was already here on the input When you entered number 4 over9 into the calculator or a computer there was a tiny mistake and these two mistakes they sort of added up and created this this mistake so that's the key question for this moment what happens to errors on the input when we are applying algebraic operations to these numbers on the input let's have a look at the slide here we can see that we have two numbers X and Y and let's assume that they are positive this will make our life easier a little bit and on the left we can see what happens when we try to add them subtract multiply divide and there is a special operation and we can see the absolute errors of the outputs so let me focus on the first Formula to appreciate the meaning of that formula I have two numbers and I want to add them so this is the ideal outcome but unfortunately due to working in a computer I'm instead adding some approximations and when I add them I get approximation of the sum and I would like to compare these two outcomes so what is the absolute error of this outcome it's the absolute error of number x + y and now when you are reading definitions it's good not to worry too much about individual letters but rather think of Notions so what is the absolute error what does the definition say it says you take the precise value and you subtract the approximate value so the precise value is x + y and the approximate value is X head plus y head that's what we calculated instead of what we should have done okay uh you could take this and claim that it's the answer yeah that's a formula it's a mathematical formula so what else would you want uh actually our main motivation is to somehow find the error of the outcome as it depends on the error on the input so I would like to re arrange it in such a way that I can see the absolute error of X and absolute error of Y that's my main motivation uh here it's actually quite simple because I can just reorganize open this bracket and I get x - x head plus y minus y head and I can introduce new brackets and I look at it and I say oh wait a second this is obviously error of X plus error of Y that was easy now uh let me put it here that's a nice mathematical formula which takes the error on the input and shows me the error on the output this error can be positive negative so they may cancel it may happen if you get lucky that the outcome is actually more reliable than the input but you cannot count on it in numerical analysis we should be pessimists and make sure that the pessimistic case is still good enough that's the idea so let's be pessimists the largest possible error that we can make when adding two numbers like that is bounded by and now again triangle inequality our friend in need it's ex plus e y here we go and that's the formula that we can see in the left column on the top then we can see formula for subaction and the proof is actually the same more or less it's just a minor modification uh the third formula for multiplication looks uh less easy less nice and it actually is not nice at all because the X and Y uh numbers are appearing in the answer here this is beautiful because we just look at the information about the errors on the input combine them and get the information about error on the output but there X and Y interfere but we cannot do better uh let me show you the proof so we are interested in the error of the number x * Y which is the actual precise value minus what we calculated in our calculator instead now you can see the difference between the two because here I could just rearrange and I got exactly these subtractions that I wanted to see but here there's no way to somehow factor out y for instance to get x - x head so if I want to get something let's say reasonably nice I would have to do some trick here is a favorite mathematical trick I would really love to see x - x head if I want to see x - x head I need to get rid of Y most likely by factoring out so why don't I put minus X head time y now I can factor out Y and I get exactly what I want unfortunately there is equality here which pre prevents me from just putting some extra term in order to fix it I also have to cancel it immediately and now it's not there at all and I can keep going with the above formula so in fact these two terms in the middle they are not there at all I have I I have a right to put equality but they're also there now I factor out y from the first one and I get x - x head * Y and now this is for things get tricky this part I did because I wanted it but here I didn't have a choice I had to add the same expression as is here and there is no guarantee that this other part will work out nicely so we have to keep our fingers crossed let's see I can factor out X hat and I get y minus y hat wow we got lucky so this is error of x * y plus X head * the error of Y and now if we apply the absolute value here we use triangle inequality and then distribute the absolute value inside the products we should get exactly that formula is that it yep there are two versions actually one of them has head with X and the other one has head with Y it depends whether you put a head here or here it can come in two positions and both of them eventually work out more or less the same okay so that's the multiplication trick is slightly more devious but it's more or less the same story uh the fifth formula is like a bonus it's a special version of division where the numerator is actually known precisely the number one it's perfect it's integer so there is no uh source of error with this number one just in the denominator okay so this was just an aside let's focus on the addition because absolute error is useful we couldn't do without knowing it but for interpretation we w't want to know relative error so let's have a look at the first line on the right relative error okay relative error of x + y by definition is the absolute value of error the absolute error of x + y divided by the actual value x + y okay now let's see for the numerator we have a nice upper estimate and for the denominator we do not have anything yet unfortunately okay uh well again I could keep this as the answer but I would love to see relative error of X and relative error of Y somewhere in the mix here so I have to introduce it there well the first obvious step is to break it into two fractions so we are getting e ex over x + y+ e y over x + y everything in absolute value now here comes a dirty trick I will break this fraction into two which are multiplied and of course I want to keep things the same so I put ex here and x + y here and I will do the same thing with the other fraction so far I didn't really change anything this is just a different weight of writing I imag that there is one here one here one here one here it's the same but now here comes the trick for some reason I decide to put X here and also here then and that they cancel and I'm still keeping the same equality here running here for some reason I put Y in absolute value and now I look at it and say oh wait a second this is the relative error of X wow aren't we lucky um I talked about an assumption that X and Y are positive and and this I will use here to get rid of absolute values I prefer to have it like X over x + y and the relative error of x + y over x + y * the relative error of Y here we go is it nice uh not yet because I can see X and Y in the answer I really like those epsilons here but not this part U I would like to factor out EP but they need not be the same however another trick I'm happy with an upper estimate so I can replace these two relative errors with the larger of them which will make things larger so what will appear there well instead of Epsilon X I will put larger of Epsilon X and Epsilon y so if Epsilon X is larger it will be here and it's the same and if Epsilon Y is larger it will replace this guy and the inequality works and I will do same exactly the same with the other relative error I replace it with the maximum now this maximum is just one number just one symbol it's just one object I can Factor it out maximum Epsilon X Epsilon Y and when I factor it out there will be X over x + y + y over x + y and this adds up to 1 and disappears and that's the happy end that's what we can see on the right okay the other five formulas can be done using similar tricks so this there is really nothing more to talk about when it comes to this theorem and its proof so from the mathematical point of view we have it licked let's talk about the interpretation and for that we will pass to engineering approach we will sort of like uh let's say close one eye so that we can see it a little bit fuzzy and say oh it's about like this uh you can see that the mathematical Precision forced us to do some complicated stuff on the right uh when you look at uh the second last the last row uh you can see something with 1 / 1 - Epsilon X we saw this before this is exactly the result of trying to make engineering let's say guesses precise passing from relative error to Absolute error and vice versa things like that uh if Epsilon X is really really tiny then it can be ignored in those ratios and those formulas become much more uh Pleasant much more readable so let's put another fact here which is an engineering fact this is not really a truly mathematical fact although okay it pretends to be so but okay I'm making it clear that it's not so imagine that you have some general rounding error Epsilon or the upper Bound for it Epsilon is the upper Bound for rounding error like we talked about it computers calculators they have this rounding error based on the Precision number of digits that they remember and it's common and now you have two numbers and you decide to add the multiply them and so on and you want to know how reliable the outcome is now remember relative error tells you about the number of digits that you can trust roughly the first first Formula it says that when you add two numbers then the relative error cannot grow which we also see here the relative error of the summation cannot be larger than the relative error on the input and there is also multiplication by a and this a is an integer it's a number which does not have an error coming with it so this is very nice let's let's go down let's make a chart Theory addition addition is okay we love it Division and multiplication you can see that they behave in the same way when you divide two numbers or multiply two numbers then the relative error doubles which is not too bad multiplication division okay but let's ask this question what happens if there are four multiplications in your calculation each multiplication increases the relative error by two with four multiplications you are getting the original relative error multiplied by 2 by 2 by 2 by two which is 16 * Epsilon and 10 times Epsilon means that you lost one digit of precision in fact for every three and 13 of multiplications or divisions you can potentially lose one digit of precision and then you have to start worrying so when you have many multiplications many divisions you should be careful about keeping Precision of our calculations reasonably good so I have to put here it's okay when there is just few of them and for the last I left the second operation which is subtraction this one is exceptional you can see that there is X and Y appearing in the result which is something that we do not want to see but there is no way to avoid it it must be there unfortunately and let's see what the coefficient is now here the coefficient was two for multiplication and division and we already observed that it could complicate things there the number x + y over x - y could be as large as you want if X and Y are close so the relative error could blow up totally if you're subtracting two numbers that are very similar and that's a frowny here when close I mean the numbers I don't have too much room let me give you a common sense explanation why subtraction of similar numbers is a big problem let's say that you have a number and you can trust four digits why not a b c d these are the digits that you trust then there is e f g h i let's say and you are subtracting another number where you can also trust four digits and this number is very similar and because it's very similar then the first digits are the same but the last three digits are different and I will subtract them and when you are subtracting then these digits which are the same they cancel each other out and the result is actually determined by this part of the numbers and that's exactly the part that you do not trust at all which is influenced by the error that you have here so you can see really without any knowledge of theory that when you are subtracting similar numbers you shouldn't really trust the result much unless you have it under control this is very difficult and very dangerous so when you look at this let's say intuitive version or engineering version of the theorem or we look at this chart this is what we call propagation of ER error in operations you start with some errors on the input and you're asking what happens to them how do they influence my result and when it comes to addition for instance you can see that actually the answer is as reliable as the question essentially it may even get better if you get lucky and those errors cancel each other out which is the case also for multiplication division subtraction if you get lucky it's okay but there is no guarantee that you will get lucky unless you are really good uh okay uh this is the theory the propagation of error in calculations uh by the way you can also ask about propagation of Errors when you substitute into exponential into s into tangent into logarithm things like that uh I made a little note about it in the lecture note so if you want look it up uh by the way tailor expansion is a very good tool there we will talk about tailor expansion here a lot very good tool in numerical analysis let's talk about practice addition addition was really good theoretically what happens when we try it in an actual computer actually turns out that there may be a problem and the problem happens when those numbers are very different and I should have some room here yeah I remember it correctly let's say that you have 1300 and you want to add 13 on a computer so the computer stores these two numbers it stores 1.3 time 10 to 1 2 3 10 cubed plus 1.3 * 10 to 1 and it's a computer which stores only two digits it's a very basic computer so these numbers are stor Pro properly two digits here two digits here now here comes the problem if you want to add two numbers in a floating Point form you need to have the same exponential term here you need to unify the exponents and this is done by going to the larger so you still keep 1.3 * 10 to 3 and here you want to see at the end 10 to 3 how do you get 10 to three by moving the decimal dot two places to the left so what you should put here is 0.013 really but the computer only remembers two digits which is these two guys so the computer puts Z 0.0 here and looks at it and says okay this is 1300 as you can see there is an error here at the end very unpleasant and this can get out of hand if you are doing a lot of calculations when you're adding a lot of really tiny numbers this can be fatal there is a real world application where people who were programming the system overlooked this uh let's say this trouble this potential trouble and it actually had disastrous consequences the system wasn't workable in applications uh there were big troubles with it so one has to keep this in mind when programming when programming systems when you are adding numbers which are very different uh you may introduce an extra error which is not uh included in the theory of error propagation the same thing is happening when you're subtracting numbers so for subtraction you are in trouble when the numbers are too close and you you are in trouble when they are too different so when you are subtracting numbers ideally you would have numbers which are not quite the same but not too far sort of middl midh apart and then you're okay multiplication here the problem practical problem lies in the floating Point format when you write a number in a floating Point form do we have some yes here the exponent also needs some room in the memory and this is limited for instance in my calculator I cannot make a larger exponent than 99 which means there is a largest possible number and if your result exceeds this number the computer just gives up or calculator this is called overflow error so we have to really worry about overflow errors likewise there is the largest negative number which can be put into the exponent on my calculator is stand to 99 so when I look at the real line as my calculator sees it here is the theoretical real line here is zero but on my calculator there is a smallest positive number which can be entered and stored in memory and nothing smaller can be put into my computer or calculator and there is also the largest possible number this is the range of numbers that my calculator or a computer can store if it if some result of calculation goes above the computer starts complaining and just stops calculating or calculator that's the Overflow error if some result Falls below the smallest possible expressible number it automatically becomes zero this is the underflow error and you often accounter it with division now you might say okay if this number is almost zero then there is no harm in replacing it with zero the problem is this can be an intermediate step in calculation so forance imagine that you are calculating uh one of those combinatorial numbers so we have some big number and factorial on the top and factorial on the bottom how do you implement it well if you implement it by calculating the factorial and then dividing it by the factorial of the denominator then this number will be much larger than the eventual outcome and if you get overflow error here you will never get to the answer even though the answer could be reasonably reasonably large on the other hand if you implement it like that you first divide by the factorial of the denominator and then multiply by n factorial then here we may run into underflow problem you replace this with zero and you will never get the correct answer so one thing that people who program software need to do is to look at calculations and arrange them in such a way that they mitigate the possibility of these two problems so for instance when you're calculating something like that a good idea is to start with one or start with n and do something like n ided by K * nus1 / by Kus one and things like that so they do keep multiplying dividing and in that way you are keeping the intermediate results happy more or less so there are also tricks how to handle situations when you're adding very tiny numbers with very huge numbers things like that I mean this has been studied because it's very important so there are some results there are some procedures that can be followed and when you using as an engineer some software for critical real life critical calculations you are really hoping that those who programmed it knew their business knew their job and did the numerical calculations properly but you never know and second uh sometimes there are problems which are really I mean there is no uh overall solution that would handle all the problems it may always happen that there will be a situation that nobody counted on and then you are in trouble so it's uh it's a good idea to develop intuition engineering intuition that you look at a problem and you sort of make a guess educated guess how the situation will develop and then you ask computer for prediction and if they are the same you may say okay I was good you know I did the same prediction as this computer which costs I don't know how many how many millions and if those predictions differ then you can ask okay was it me who made a mistake or the computer who made a mistake and sometimes it's one sometimes it's the other but either way if the predictions do not agree this shows that you should look at this situation closer which is always good anyway uh let's have a look okay let's make room and let's have a look at an example okay what we see here on the top is a nice looking polinomial essentially because we can see powers like y to power 6 with integer coefficients all these multiplicative numbers they are just integers cute integers three digits four digits nothing bad just at the end there is one division so it's not actually a polinomial but almost a polinomial apart from the last expression and the last expression is not also too bad because this is just a simple division now this expression was invented by a guy called R I think it was a German and this surprised a lot of people when they tried playing with it uh which we will do as well because I'm going to evaluate it I'm going to substitute two numbers for x and y and these numbers are integers when you as you can see but I will treat them as decimal numbers forcing computer to work to do the calculations in the floating Point format so let's see what the answer is the answer is 10 to 29 is this in some way interesting not yet not yet so the answer is 10 to 29 now the trick is I'm using Maple here but this is not really important the important thing is uh the Precision of this calculation was 10 digits now my favorite number is 13 so I just wonder whether something would change if I increase Precision of calculations to 13 which on the system can be done so let me ask pretty please do the calculation again with Precision 13 digits and the answer is 2345 Now isn't that quite the big change from 10 to 29 I mean this is a huge number do we have a name for it million billion trillion quadrillion pent million whatever Google 10 to 29 this is insanely huge number and suddenly it's 23 just by changing Precision of calculation by three digits let's ask about 15 so if we ask the system to perform this calculation with Precision 15 the answer is - 10 to 24 now this is a big surprise because we are huge number here but on the negative half of the real line so these two numbers are really far apart and this one is somewhere in the middle okay this looks like an interesting sport how about 20 digits Precision with 20 digits Precision the answer is 2345 so this is becoming the candidate or the favorite uh if we should guess the answer but we are not going to guess we will try higher Precision 25 digits with Precision 25 digits the result is okay I would love to write 10 to 13 I would really love it but this is almost 10 so it's almost -10 to 14 14 is also a nice number okay let's not be prejudiced against numbers okay 30 digits Precision with with 30 digits Precision the outcome is 23 45 and so on um 1 2 three three times this number appeared this is interesting let's see what happens with Precision 35 wow we didn't have this one before either so if we ask for PR Precision 35 digits the answer is uh let's say minus 10,000 quite a list of values to choose from how about 40 digits Precision with 4 digits Precision the alleged answer is - 16 uh let's say 55 I rounded this one to two digits let me do it here as well um this is a actually the correct answer good question what Precision do you have to use in order to obtain the correct answer and the answer is I tried it of course but just to show you how this works with 38 digits I am already getting it but with 37 digits Precision I'm still getting this familiar 2345 so uh this shows that the smallest Precision which gives me the precise answer 38 digits precision and nobody uses it in calculations real life calculations are done at much much lower precisions which means that operating systems and the softwares for engineers that work in the usual way they would actually yield totally wrong answers for this calculation by the way in electrical engineering field there is an International Association whatever and they have a guidelines and the guideline for elect electrical engineering real life electrical engineering is approximately 15 digits Precision so they would get this answer officially this is a problem when this guy room came up with this example um a lot of people were unhappy and they tried to cure their systems of such trouble here the problem is that when you do the calculations you are subtracting numbers which are very similar and we saw it here subtraction of close numbers that's a problem and here we can see it nicely okay this is a really really big problem so people try to fix their systems and this uh room guy and some other people just tweaked this example a little bit and they got the wrong answers again so uh yeah it's really hard to make a calculating system which is totally foolproof okay you can always find some sort of calculation which squeezes through those guidelines and those those defenses and uh yeah this is difficult so so this is a really nice example a very famous room example which shows that the calculations even with relatively high precisions can give you totally totally wrong answers and they even can suggest things that are sort of uh I mean this is Devious you know repeating these answers all the time these wrong answers so yeah that's uh this is not a happy end to this lecture so let's try some happy end okay so cases like this are extremely rare they they are artificially prepared and most real life engineering calculations they are much safer fortunately for us and anyway we should be really careful and check the answers if the computer tells us something don't trust it use our Common Sense look at it and ask does it make sense does it fit with a picture of the situation I have uh things like that anyway this is essentially the end of the chapter called error propagation we will be drawing on some conclusions later on when we will uh investigate numerical methods and we will ask is it numerically stable which means do the errors on the input totally spoil the answer or not that's a good question and this knowledge will come definitely handy okay so we will refer to it but the chapter is over that's also happy end let's have a look at our first numerical method we mentioned that absolute errors are closely related to error in measurements now relative errors they are closely related to working with computers and calculators okay let me start again we talked about absolute errors being related to measurements now relative errors are closely associated with working in computers and calculators no okay last try the third time we talked about relative errors okay we talked about absolute errors being related to measurements now relative errors are closely related to computers and calculators which also means irrational numbers no no I'm mixing it up we talked about absolute errors being related to measurements now relative errors that are closely related to the way that's in which ah Jesus I got totally derailed okay let me try it for the last time one more time we talked about absolute errors being closely related to measurement no that's okay one more time we talked about absolute errors being related to measurements now relative errors they are closely related to the way numbers are stored in computers and calculators and we will ask is it it numerically stable which means do the errors on the input throw out or do the inputs on the on the input no do the errors on the input totally spoil the answer or not