Building Anomaly Detection Systems: Essential Guide

[00:00:00]Uh welcome everyone to this uh uh webinar. Uh I have with me my two good friends who I've known for several years and uh we'll I'll tell you a little bit more about their background in a second.

[00:00:16]But uh this is uh going to be a great opportunity to hear from two people who are have been thinking deeply about these topics for several years. Um I think uh Arun more on the user uh side and uh and uh and uh Ira more on the builder and vendor side. So it'll be great to have two different perspectives. Uh as you know uh machine learning and AI are both hot topics uh in general but uh more recently for enterprise and one thing that enterprises uh share and have a lot of are temporal data and time series. So uh that's why I'm super excited to uh talk to these two gentlemen today because uh uh time series and temporal data are usually great starting use cases for enterprise who are interested in getting serious about uh machine learning and AI. And um so without uh further ado, I wanted to uh introduce our two uh main uh speakers today. So I'm more more of a in the background here. Uh I am Ben Laura. I am the chief data scientist at uh Oral. I am also currently now chairing three big uh conferences uh strata data uh the strata data conference the O'Reilly artificial intelligence conference and tensorflow world. So uh maybe I'll have you guys do a quick intro of uh yourselves. Uh so basically just at a high level talk a little bit about your education and experience.

[00:02:18]um and what led you to this uh uh particular set of topics around time series theories and anomaly detection in particular. So Arun, why don't we start with you?

[00:02:30]>> Uh sure. Thanks Ben. So I finished my graduate studies back uh in 2008. It has been a long journey. So since my my graduate school, I've been interested in u performance analysis and diving more into the data. Uh and anomaly detection is a topic which uh is an integral part of performance analysis and anomalies surface in pretty much every domain as we shall talk about later in the webinar. So over the course of my uh industry career, I've been working on anomaly detection in several places. Uh back at Twitter um along with my interns, we develop an open-sourced anomaly detection system or technique which we use in a infrastructure context.

[00:03:29]And uh um my team has been conducting extensive research and development on various fronts to address this challenge of anomaly detection. More recently uh at machine zone uh we were looking at um fraud detection in the context of performance marketing. So fraud is just another manifestation of anomaly in the realm of marketing data. So you can have an anomalies uh in impression data, click data, install data or engagement data. So we built a lot of novel techniques to address anomaly detection and uh I mean I've known Iro and uh Ben for a long time and we shall talk more about the general applicability of anomaly detection techniques uh as we move along in this webinar. So Ira very briefly.

[00:04:27]>> Yeah. So thanks a lot uh Ben and Arun for for having this panel. Uh yeah so my background is in uh machine learning but I actually started from uh single processing which is why I like time series data so much. Um I worked a lot on video but then uh moved to time series analysis and my work at HP labs uh and then HP software. uh and five years ago started a company around uh anomaly detection for time series data which is what what anod does.

[00:05:00]>> All right, let's let's uh let's get started uh with uh uh the webinar itself with a series of short presentations and then followed by hopefully Q&A with all of you in the audience. So let's start by asking you all to raise your hand. Um, so we have a couple of questions here for you the audience.

[00:05:30]Uh, is there a slide? Uh, oh yeah. Yep.

[00:05:34]Here you go. So, uh, how many of you have attempted to build an anomaly detection system in your company or your organization? So, raise your hand. So, there's a feature uh that allows you to to do so.

[00:06:00]>> So, yeah, it'll be interesting to see actually. Uh >> yeah, so I see there are 15 people who already raised their hands.

[00:06:09]15, sorry, 17.

[00:06:12]That's pretty cool. Pretty cool. Yeah.

[00:06:15]Next. Um, did uh did what you do uh involved any of these things, statistical or machine learning or more specifically deep learning or were you mostly using uh other techniques? I guess uh we should have probably narrowed this down to machine learning and deep learning, right?

[00:06:44]>> Yeah, >> that's a huge huge part. So 11 out of the 17.

[00:06:56]Very cool. So um I guess uh the first question is why care about anomaly detection at all? Right. So uh and uh who should you be looking to in your business to drive uh an anomaly detection initiative? So Ira, you want to uh take a stab at answering these questions?

[00:07:19]>> Sure. And and of course I'm going to give it from the perspective of uh of the some of the work that uh we do with our customers and what we see them getting value out of. Um so so really the type of anomaly detection we are looking at is on time series data. Uh and what is a time series? It's a lot of different measurements that you have from your business. Uh and there are the measurements are you know they start from the revenue and how many visitors you have on your site and and how many people are buying on your on your e-commerce site. And it could be your telos uh looking at uh uh how many calls are being made and how many calls are being dropped and you know that there are a lot of a lot of examples from from industry um I think we can we can show I have a slide here. So the type of of incidents that you find uh using anomaly detection in organization are you know cost a lot of money to organizations. So, uh, Gartner said that an incident cost on average $300,000 an hour if you're a large enterprise. Um, 7% reduction in your latency already creates a big gap in in revenue. Uh, you know, 100 milliseconds basically creates 7% loss of revenue or, you know, it takes 30 31 hours a week loss to incidents in organization. So it costs a lot of money uh to have incidents and anomaly detection has really been the way to the new way using machine learning to find all these incidents as fast as possible before they cost so much. Um so that's that's really the type of uh things and we have you know there are plenty of examples I can give from our customers but of course I can't share them but there are a lot of them that you can see out there um that are published and uh you know if we go to the next slide then I actually prepared a few a few examples uh for example uh you know just recently uh Dutch emergency line was hit by a telco outage which cost a lot of money in in reputation of course uh app stores applications going down target was hit by outages uh that cost them millions of dollars. So these type of failures if they're not detected quickly uh and fixed quickly then they tend to cost quite a lot of money to any any company and anybody running a business especially online businesses uh know know that this is very costly.

[00:10:02]Yeah. All right. So then uh uh it's good to kind of uh put in context why uh even care about anomaly deduction and uh >> to also kind of emphasize that uh uh anomaly detection applies to businesses that collect metrics which is basically every business and doesn't necessarily have to be confined to IT metrics as Ira has emphasized. It could be business metrics. So now let's uh try to be more specific as to what it is that we're talking about here. So uh Arun, can you kind of uh help us kind of narrow down the scope of what is an anomaly detection uh system?

[00:10:47]>> Sure. So I mean as Ben said that anomaly detection need not be limited to IT metrics. It can also be business metrics. And let me give you a specific example. So back at Twitter where we developed one of the early techniques u there were many business metrics or KPIs you know such as number of tweets per second or number of uh follow followers per second or number of retweets per second. So these are business metrics and if you see a sudden drop from what you typically expect then that constitutes an anomaly. So uh at a high level an anomaly corresponds to a deviation from an expected pattern. Of course there are too a lot of details um which one needs to dive into. So we uh employed uh or we developed novel techniques to be able to detect any anomaly in a business metric or in our uh infometrics for that matter. Uh why automated? So the key thing is that historically people have been uh detecting these anomalies via dashboards. Now if you go if you uh attend conferences such as O'Reilly AI or O'Reilly Stata data it is pretty routine for speakers from leading companies such as Facebook, Netflix, Google and so on so forth to share that in the interest of uh observability uh they monitor hundred hundreds of millions of time series. Now at that scale is practically not viable to have to monitor and detect anomalies via dashboarding. So that uh necessitates organically to build automated anomaly detection whereby you detect anomalies algorithmically and and then marry that with the context and only surface those which warrant action.

[00:12:54]>> Yeah. So if I can add maybe to to what Arun said. So yeah a lot I've seen it also in my previous companies uh the way people detect anomalies uh today or basically detect unusual situation. What is an anomaly? Anomaly only is something unexpected that happens and the context of a of a of a business it's an incident or an opportunity that if you don't pay attention to it uh it might cost you or you have a missed opportunity. Uh and the way as Arun said the way traditionally uh it's been done is either having sitting people sitting down in front of dashboards all day long and actually looking for those unusual situations. uh but that doesn't scale and when you have hundreds of millions I think Facebook has billions of these time series collected all the time. Um so that's one way and the other way is to try to create uh alerts uh and and static alerts uh basically saying if it crosses some threshold then send me a notification because I can't just look at everything. So that tends to work in either very small scale where you know your data really well or if you have a very good understanding of what those thresholds should be. For example, if I'm monitoring a server, I know my CPU should not reach 100%. Let me detect let me find whenever it crosses 90%. Or my disk space has cannot be uh more than 95% full otherwise it might reach its capacity. Uh but when you're talking about a lot of application KPIs or time series uh business metrics, it's really not clear what those thresholds are. There isn't really a good way to know. Uh but people have still have done a lot of automatic uh uh threshold or thresholding using static thresholds that tends to create a lot a lot of false positives or missed mis detections. uh and it's also becomes a system that you have to maintain very very constantly and very hard uh to do.

[00:15:04]>> So how does how does anomaly detection uh IRA relate to machine learning?

[00:15:11]>> Ah so that's uh that's a that's a great question. So so basically anomaly detection is a field in machine learning right? It uses a lot of the algorithms for machine learning. It's an unsupervised uh machine learning uh uh area uh uh and it's basically looking for the outliers.

[00:15:30]So the way I like to think of it uh if in unsupervised learning you do clustering oftentimes this is the extra step of saying oh something is not related to any common cluster that I can find in my data. Uh and now the question is how do you define a cluster? uh and that depends on the type of data that you have. If it's a time series, then you have to model it one way. If it's if it's static data or non non-time related data, then you you use other techniques.

[00:15:59]But it's basically part of what we call unsupervised uh learning. I also think of it as uh um as as something that uh that even in regular machine in in in in other parts of machine learning you need to apply uh in order to get good training. For example, if I'm a machine learning practitioner and I want to get and I get I get a training set to do even a supervised task, I might want to discover my anomalies in the data so it doesn't bias or corrupt my uh my training data. But that's another area of anomaly detection.

[00:16:36]>> Yeah. Yeah. Yeah. Yeah. Actually, uh that's an area I'm super interested in around uh basically data quality, data cleaning and the these sorts of things.

[00:16:44]So, Arun, uh uh so what are some of the common techniques that uh uh people should uh be looking towards uh uh implementing if they're for example getting started?

[00:16:59]>> Yeah. So as uh Ira mentioned that in most cases uh one should uh explore using unsupervised techniques uh to build uh anomaly detection system because in uh in a uh real world scenario it's very hard to get labels and hence leveraging any supervised uh learning based techniques is not viable.

[00:17:25]Now talking about in the unsupervised realm there is a pretty extensive literature but to get started there are very simple techniques such as uh mean plus minus 3 sigma. So it's just as a starter one can leverage rules like these. These are still far better than using a static threshold because um mean and standard deviation they are uh function of the underlying distribution. Now having said that uh there are many uh underlying assumptions which uh a simple rule like mean plus minus 3 sigma uh makes. So it's very important that even when one uses offtheshelf rules or techniques like these that the underlying assumptions are validated. So for instance in this case it assumes that the time series is stationary. it is assumes that there is uh uh no seasonality and it assumes a normal distribution. So these assumptions should be validated otherwise you may uh uh end up with false positives or false negatives.

[00:18:39]>> So actually uh you know I wanted to add to to what Arun said. So of course there are a lot of unsupervised techniques and one of the questions that Arun alluded to is why not use supervised techniques.

[00:18:50]Um and and you know when we talked about the in the intro how do they get to anomaly detection I actually when I started uh my work at HP uh we faced a similar problem but I wanted to solve it using supervised techniques. meaning I wanted to know I mean I know that if I do supervise and I have enough labels the chances of me catching things and measuring how accurate I am is much better uh than unsupervised techniques because there the the target is not very clear and you make a lot of assumptions on your data as Arun said the problem is uh that that in a lot of the cases that we're talking about there aren't really there isn't a way to get labels of anomalies and what I found when I tried to do this And I tried to do it for a year questioning people around it more the IT domain questioning what are the common uh anomalies that happen in the world. Uh let me find those label them and train a supervised model and maybe capture 80% of things. But I quickly found out that that there aren't a good way to capture uh these labels from from people because uh uh all these systems that we're monitoring, all these applications when they work okay, they all look alike, but they when they start becoming faulty or there are all sorts of things that go wrong, every everything that uh everything that goes wrong may look different than others. So it's very hard to actually even get the top 10 cases of things falling apart. Um so it's like Dtorik's uh book, you know, good families all look alike, but uh uh uh broken families are all different.

[00:20:31]And this is the same way with what we're dealing with, which is why we have to go to these unsupervised techniques and learn what is normal because that is possible and then find these abnormal patterns out of those.

[00:20:44]>> Great. So uh we have a quick poll before we go to the next section. Um and so basically uh it we want to kind of take your temperature as to what is the most important component of an anomaly detection system.

[00:21:03]So give you give you folks a few seconds here to answer and then we'll go on to the next section where we'll do uh a more detailed discussion as to requirements of what makes a successful anomaly detection system.

[00:21:18]>> Oh I see there is at least one person who wants to find the anomalies in their significant other.

[00:21:27]Those are the most interesting things about the significant other.

[00:21:42]That is cool.

[00:21:59]Yeah, this is super interesting. Uh, it actually is in line with what we what we see.

[00:22:10]All right.

[00:22:16]False positives obviously number one.

[00:22:20]And >> real time then real time.

[00:22:22]>> Yep. Real time. Now that's getting becoming more and more important again and it will become more important with 5G.

[00:22:29]>> Oh yeah.

[00:22:30]>> And then scale. Very cool. All right. So let's because we're in a a tight schedule here. Let's uh jump into the next section which is basically uh system requirements of a successful anomaly detection system. Um so here we we'll just kind of have a uh a free flowing discussion and uh uh Ira and Arun feel free to jump in but basically what are some of the requirements that you think are uh uh critical uh for people and I think uh we also should probably uh factor in that uh the audience here has many different entry points. Some probably are just getting started and as we noted in the raise your hands earlier some have already begun building u anomaly detection system. So uh requirements uh Aroon you want to take a first stab?

[00:23:31]>> Uh sure. So as the audience also highlighted that real time and scale are pretty key to any uh anomaly detection system which one may want to use in a industry environment. Um as uh data availability uh becomes more and more ubiquitous you know uh there'll be more and more data streams which will be flowing through the system. uh so to be able to extract anomalies in real time so that appropriate action can be taken uh uh can be taken uh will hold key. So for instance we had Amazon Prime day yesterday and the day before. So imagine that if you are at Amazon Prime and then there are certain products which are uh selling like hot cakes then Amazon would want to know these uh trending uh items so that they can uh uh revamp their inventory in time uh and minimize the loss sales. So surfacing these real-time insights is very important and scale goes without saying that if you are a big enterprise you know um being able to scale um because you are monitoring hundreds of millions of time series is also very critical.

[00:24:55]>> So let me uh let me actually interject myself here quickly. So when we talk about real time, there's the notion of being able to process the data in real time, surface anomalies in real time, but uh IRA and and I guess Arun as well.

[00:25:12]Uh what about the anom the ability to uh adapt and and learn from new from a possible new changes in the state of the world. So real time or online learning Yeah. So, so I think that is actually critical uh especially for this type of systems measuring businesses if you're measuring system metrics or business metrics that state of the world changes and you have to have algorithms that are able to adapt adapt quickly enough detect the anomalies on one hand but adapt to changes in the world uh fast enough to be able to catch new situations. Um it relates to the false positive false negatives as well because uh if you uh if you have so the the the most important thing in the poll uh people shared are false positives and I think this is really the perception that we also see false positives is something critical to minimize because like any system that you might must act on if it tells you something that is wrong and you waste time on it then you lose your confidence and uh and you and start stopping using it. Uh and that happens a lot in the old static threshold because uh people get tons of alerts and get alert fatigue. So uh adapting to changes that are happening in the world is critical to reduce false positives and to still be able to catch the right things. And changes happen a lot in businesses. Uh it's not like it's they're not closed systems that are designed once, built once and are known to operate in the same way like a like a let's say an airplane engine uh or a big machine they it keeps changing so it has to adapt. It has to it has to use new data constantly to change what it knows about the world.

[00:27:10]>> So uh Arun are there different types of anomaly detection systems? uh uh is there a uh simple taxonomy that people should keep in mind?

[00:27:21]>> Sure. So um as we had one of the earlier slides that uh broadly speaking one can either employ techniques rooted in statistical learning or machine learning or deep learning. uh there is a vast amount of literature uh in all the three uh subdomains and I as I mentioned earlier that anomalies manifest in different forms you know it can be from astronomy to manufacturing to operations or networking or what you have so it has been studied for more than a century uh now this relates the taxonomy relates to uh the third bullet here we have on the slide which is explanability So it's the selection of the technique is should be guided by your use case.

[00:28:13]You know uh someone working in academia trying to or some pursuing research in anomaly detection system is it's probably all right to just surface anomalies and show that it is that the false positive rate is lower than state-of-the-art and then move on. But for instances like uh uh what anod deals with because they have real clients you know explanability is very important. So from my own personal experience back at machine zone where we uh we used to um dispute fraud in our marketing campaigns. So fraud is for the benefit of the audience is one of the most rampant uh pain points uh in the realm of uh digital marketing. So whenever we used to surface fraud, it was imperative that we were we could explain why what we have surfaced is is indeed fraudulent. So explanability to our uh um ad partners was pivotal to get the money back. So and and that's where you know approaches like deep learning which have shown lot of promise in different in other domains such as uh speech synthesis or uh uh audio to text uh text to audio conversion and so on so forth.

[00:29:40]In the realm of anomaly detection when you're deploying it in a uh in a industry context uh they they tend to fall short because uh uh there is lack of interpretability slash uh explanability. There is although there has been recent work or recent focus on improving the explanability of uh deep learning models but they still have a way to go. So uh leveraging the taxonomy depending on the use case uh uh can be very helpful.

[00:30:17]>> Yeah I I think that another aspect of of that uh besides explanability is what we that the fourth the fifth bullet I think multivariate or univariate. So actually when you go from uh you can have a multi a full multivariate technique where you have a model that learns normal based on the inputs from many many different sensors uh and then it output is there is I found an anomaly in all these sensors or not. The problem with that approach that if you need to explain why it said that there was an anomaly it can be very very challenging to do it. uh as as opposed for example to univaria techniques that look at time series uh each time series at its own variable and find anomalies in them. uh those are easier to explain because you can show oh here this time series value is now outside the norm and I can I can draw the norm for you so you can see it and what I mean by normal uh and that's really a lot that's bes relates to the explanability as well uh and it kind of hampers uh the ability to use some multivariate techniques or more uh uh deeper techniques like deep learning uh in some cases some use cases of anomaly detection and there are other use cases as Arun mentioned that you know you can use the multivari techniques because you don't necessarily need to explain the outcome maybe there is an automated action taken after an outcome uh that says there is an anomaly and uh as long as it's mapped out correctly beforehand you can take those those actions. So it it really depends on the use case. um uh there is a lot of different techniques that you can use based on uh whether you need to explain it or not.

[00:32:05]>> So what about um so uh I recently been thinking a lot about uh the infrastructure uh ecosystem around machine learning itself for which obviously a lot of that will feed into anomaly detection. So there people are starting to think about uh tools for model governance u model operations. So obviously uh uh in a system like anodot you can have many many models out there in production. Um and so what uh what sorts of uh best practices do you have in terms of uh of uh introducing new models, retiring old models uh right >> or basically QAing uh new models, right? So before you even introduce them uh uh to your system. So um how do you know that uh how do you know that uh these new techniques that Arun mentioned around deep learning are even worth the cost of uh of of uh deployment.

[00:33:19]>> All right. So so I can talk about some of the models we have and maybe Arun can speak about some of his background around it. So definitely uh um the when we have models that we want to deploy there is of course a whole phase of of developing it and testing it on some data that we put aside. Uh but let's say we passed all of that and that is there are a lot of best practices for that for that. Uh if we pass all of that then uh we still have to deploy these models into production and then send them uh um and then send them on their way and see what happens. The only way that we found to actually make sure that they still function in production is to monitor them uh directly as well and actually use anomaly detection to find when they're starting to behave differently than before. Um I'll give you example not from anodot uh I like it's a it's it's an it's an example I really like to give uh for other types of machine learning algorithms. Uh there I think in December there was a glitch at the Google ad uh one of their software for ads uh that actually somebody introduc there was some bug in their training software and their algorithm their bidding algorithm started bidding on many many ads on the internet and actually won a lot of them because it was really way bidding way too high. Um and basically these these bidding algorithms they are usually based on machine learning techniques and if something goes wrong with that uh model that is now doing actual bidding uh it's going to overbid or uh create a lot of wrong bids in wrong places and it's going to cost a lot of money to Google it actually cost uh I read in the articles anywhere from a million dollars to $10 million in in one hour of that glitch. uh and the way to catch it is to actually monitor what these models are doing uh and catching when there are starting to behave differently and this is the you know the anomaly detection.

[00:35:22]So what you see here on the slide is actually an example of a time series that you might measure could be revenue and what what you're seeing here on the screen is is how does it how does how does a mo how do we learn uh what is normal about it. So we start out by very wide distribution in the beginning of the time series because we don't know what happens with it. Uh eventually we realize that there is some seasonal pattern to it. Uh first daily then weekly and then we have a very good understanding of the normal behavior and we can detect anomalies. Now when you're modeling when to your question when you're tracking your machine learning models in production they're actually going to produce a lot of these time series. Some may look like this one some may look differently. uh may have different behaviors, non-stationary, stationary behaviors, but they are still going to exhibit some normal pattern.

[00:36:14]And that's the way to that's the way we found is the best way to to track them.

[00:36:19]And it's very similar to tracking your business. I mean, I think of machine learning models and their governance.

[00:36:25]Machine learning models in production running in production, they're like uh I think somebody from Gartner or Forester called them AI workers. So they are just like any other part of uh the working environment of something doing doing constantly doing taking actions and making decisions based on input data that it gets. So you have to monitor just like just like you monitor the performance of any worker. You know there's also the the uh argument and the philosophy I think uh and in some regulated interest industries even very strong guidelines around uh making uh how you organize your teams. In other words, you may have uh one team that develops the anomaly detection system and one completely independent team that uh just monitors it. Um Arun, I wanted to actually very quickly get your take on uh since you've mostly been uh uh on the side of someone who either builds or evaluates these systems. So any kind of high level advice for our listeners as to how you would uh evaluate uh a system an anomaly detection system if you were on the side of someone who is deciding uh whether or not to uh use a specific anomaly detection system.

[00:37:57]What are some of the things you would look for?

[00:38:00]>> Yes. So uh you know the devil is always in the details. So if I if I were to evaluate a anomaly detection system is to get uh some feel for what are the different models uh they support and more importantly that what are the underlying assumptions of the uh the models make and are those assumptions satisfied uh by the the the data at hand. So now like a company like Anodot who who has invested lot of uh time and effort in building such a system. So it's not particularly uh uh relevant or needed to dive into the technical details but at as long as as a potential user of such a system you know uh it's as long as like basically doing a due diligence that the product actually validates the assumptions of the models uh the model makes with the data and if if it is so and the other thing is that one model doesn't fit all. So, uh that's one of the nice things about u a system like anodot that they have a ensemble of models. They don't uh rely on a single model because the data characteristics are very different uh depending on the data source. So uh making sure that there is an ensemble of models so that uh the final recommendation of the system will be much more robust than using one or two models.

[00:39:42]>> So you're on the so let's say Arun you're on the uh technical evaluation team right? uh now you kick the tires on a system. Now you have to convince uh some business decision makers. So uh how would you go about convincing them that a system is is solid and good and so is this where kind of a systems ability to explain how uh results are being uh decisions are being made becomes important? So what are some of the things that are critical to technical teams who want to convince uh their bis business decision makers? So um so that's a great question you know so so if you roll back time let's say I was still at Twitter and then if if we were evaluating a system uh to find anomalies in business metrics then you know we would pilot the system and we'll see that okay how many times that whenever we had a uh an incident or there was some other u due to some other uh reason our business metrics nose dived or there was a sudden spike maybe because of DOS attack. How often how many times was the system under evaluation was able to detect and what what was the business value you know so was it able to catch the the key uh uh instances where our business uh could have been impacted materially. So that would make a very convincing business argument that look for the smaller incidents even you know it's not as material but for the bigger ones uh uh like if you are at Amazon and you expect sales of north of 100 million but you see that hey for the first six hours my sales are only 2 million so it's a negative anomaly and the system was able to catch it you know so that would be a very convincing case that look uh our business was was getting uh impacted uh in a material fashion and the system was able to catch it >> and obviously I think I guess Ira uh there's certain operating requirements right so the ability to integrate ability to ease of uh getting started so uh how easy is it to feed data into a system how how soon before I get results and things like that >> yeah I mean so so yeah when we designed uh kind of the system. Um then you know we had to think about a lot of things from the algorithmic and product requirement what is uh you know what does actually have to have uh before it becomes really useful. So false positive minimizing false positive as as the poll said it's obvious. Uh there was another part that 40% said it's important which is scale. uh and the way I think about scale and this is part of the requirements both from the algorithm but also from the product itself. Uh the scale scalability is not just the ability to process a lot of data and process it fast and real time uh but also to uh to do to to do this processing and analysis and finding of anomalies uh without uh any human intervention. Uh because if you're collecting millions of these time series and you want to find anomalies in all of them um if you start introducing if you start asking people questions oh what is this uh time series is this stationary or not is this uh seasonal yes or no then you can't really scale with that. So it has to have um so scalability is is both from the algorithm perspective being really autonomous learning everything that it can by itself uh as much as possible with minimum to zero human intervention.

[00:43:39]Um uh and then the product requirement around that is to have the ability to explain what the algorithms found. So finding an anomaly without explaining it and without having a product or having some way to explain why it was an anomaly and usually the way to explain it is using using some visualization. Um show a graph with normal how it looks normally and then show that it's outside. It's very easy for us as humans to understand why it was an anomaly. Uh so these so from the algorithm perspective being very autonomous being able to cope with a lot of different types of data distributional data distribution season patterns uh ver uh you know adapting to varying situations.

[00:44:26]All of those are critical and then from the product is being able to show the result in a way that the user any user can understand it why it said it said so. uh and that's really a lot of the requirements we put ourselves and scale is one of them. Um and scale actually uh the algorithms and the scale of them uh feeds into minimizing false positives skill feeds into being able to do it in real time. So that's uh that's part of the requirements. I don't know maybe Arun when you were working on the Twitter open source uh or before it became an open source did you get requirements from the business of what it should be? No, it's it um the requirements are essentially what you just outlined that uh in the infrastructure space, you know, think I mean if we put ourselves in the shoes of a SR, you know, an S sur would not want to get a uh false positive uh pager at 3:00 in the morning. If it happens even twice, you know, the the S sur is not going to use the system, you know, going forward. So minimizing false positives is extremely important in that context.

[00:45:38]Now the flip side is let's say if you have if you deploy the system in healthcare space then you would rather have false positives than false negatives because false negative if you have false negatives you know you're putting someone's life at risk you know so it is very context dependent so yeah back at Twitter false positives was the like minimizing false positives was the key requirement uh because otherwise you will have very low adoption and then of course it has to be more real time that if if uh the system catches a anomaly which is indeed anomaly and warrants action and but if you only suffice it two hours later than the actual incident then uh that would uh result in a very poor user experience you know people are not able to tweet or retweet and you don't want that incident to prolong for two hours so you know timeliness is very important >> all right so we have a few sections left I want to go through these fast because I do want to get through some of the questions from the audience. So first uh uh resources. So Arun so what are some what are the resources needed to support uh successful implementation in terms of people and budget and uh are there any open source that are open source tools that are available? So to get started you know uh there are uh lot of resources available out there currently in the realm of unsupervised learning uh the default languages people use is either Python or R. So there are plenty of packages in both languages and in fact language is not a barrier. Um uh R is more popular in statistics, Python and CS but you can use packages like RP2 to to call a R package from Python. So there are a lot of uh packages uh in both u uh languages one can leverage to get started.

[00:47:38]So and then uh since uh uh people who are considering uh buying a system from a vendor uh or a uh cloud provider uh it's also not just a question of budget but processes as well right so you have to have uh a process for how to evaluate uh uh systems or even models from third parties.

[00:48:04]>> Yeah. Uh that's a good point. So uh this uh relates to what we discussed earlier that to have uh uh integration support that are there APIs uh which these uh vendors support and as Ira mentioned that um being being able to explain why an anomaly is an anomaly via uh dashboarding. So if uh you have if someone has 10,000 time series can uh the dashboard surface only relevant time series which are actionable and then highlight why it is a it is anomalous uh that is very critical for evaluating the system. So someone which doesn't have a rich set of APIs or connectors to different data sources be it cfka v hive or or what or pulser uh that would be very limiting. So, Ira, I I bet uh you get this question a lot, right? So, which is basically uh your system is cool, but uh maybe we'll just build it ourselves.

[00:49:06]>> So, >> yeah. So, so that that's a it's always a valid question. Um so so what we've seen or what I've seen over the past uh five years and even before that uh so it's it's quite easy to get something going uh something working on some of the data uh using a lot of these open sources.

[00:49:26]You still in my opinion need to have if you want to seriously build a system that can handle you know both scale and a wide variety of signals you need to have data scientists that understand time series on board. um it's not so they they have to understand this unsupervised techniques and understand the special thing about time series as opposed to non-time series anomaly detection. So the time series means that you have to have models that that take into account the temporal dynamics of different of of of what you're measuring. If you don't take into account temporal dynamics in some way within your algorithms, you're actually losing out and you're going to create a very a lot of false positives and temporal dynamics is is is not easy to catch. Now, the way we've seen it and and we've seen a few examples of companies that a lot of them are our clients today uh that you know started out as small startups um when they were in startup mode, you know, they did not invest anything in anything related to anomaly detection. maybe some basic alerting and dashboarding, but it's okay as long as you have, you know, run a lot of data. Uh your incident cost is probably uh a lot less than what you're making and you don't have time or resources to invest in anything. Uh so you're just you're just handling it and then we've seen that you know after the first year year and a half you start getting a critical issue that costs you a lot and you know then the CEO and everybody starts getting nervous. Uh but then it gets fixed and gets fixed and you know things move along until the next big one comes. Uh usually something that is overlooked uh something not doesn't start as a major outage but it starts as something much smaller and then you know kind of slowly creates a big damage and you lose a lot of money and then everybody freaks the board freaks everybody freaks out and they want and now comes the moment that you decide to to build something of it. Um and at that point you have a decision you can start building your your own.

[00:51:30]You can buy it from a vendor. Uh if you start if you decide to build then you have to create a team around it uh with some with a data scientist or two or three depending on the size of the company uh and how ambitious the plan is uh developers. You need backend. You need some testing. You need somebody to define the product if you're serious about it. And that takes time. It costs money. uh and at that time you're spending money but it's not solving your incident. So you still lose money and you hope you don't get another big one uh before you finish it. Uh unfortunately a lot of times people think of it as a project that starts and end and not as a product uh that you have to keep and maintain because you're building a product for internal product to do something just like you would any other internal product. uh and if you don't continue to invest in it when the project is done, you end up seeing that it doesn't work. And we've seen it happening a lot of times uh where it worked on some data, it worked in the lab, it worked in the while the development of the product, it works on a small subset of the data, but then if the investment doesn't continue constantly, uh it's not a project, it's a product. So, so you have to >> and there's also I think uh there's also the question for data scientists right so uh so in in terms of like you said so first of all uh it's a somewhat of a specialized topic temporal dynamics and and time series but also uh data scientists might start questioning should I be working on this or should I should I be working on something that uh that uh business people in my company really care about, right? So, whatever the core business it is that uh uh we're involved in, be it finance or logistics, maybe I should be using my machine learning skills uh directly for the business. So, uh let's jump to the next section. And here I want really brief answers because uh I do want to get to some of the questions. So, one minute each. Of course, the big topic uh of this uh webinar is anomaly detection, but I think uh actually a uh an equally massive topic for businesses these days is forecasting and forecasting of all sorts, right? So nowadays uh demand forecasting for uh retailers and logistics providers, everyone is thinking about forecasting. So uh will these any of these uh topics or things that we talked about or systems that uh we built do any of these help me with forecasting? So one minute each haroon.

[00:54:15]>> Yeah, sure. So uh this is a very um tightly coupled topic with anomaly detection.

[00:54:26]uh any business you know be it business metrics like uh your traffic volume or your sales you know and if you're a public company then you need to uh provide guidance to the wall street and forecasting is very uh intimately tied to anomaly detection because the data you're using your forecast is as good as your model but also the data. So if your data historical data has a lot of anomalies your fidelity of your forecast will be pretty low. So uh it's very common that before you uh employ uh uh the state-of-the-art forecasting technique whatever it may be that you cleanse your data by leveraging uh anomaly detection techniques to ensure that the input to your forecasting algorithm uh is um is uh is clean of anomalies uh otherwise your forecast uh would not be as robust. uh Ira one minute.

[00:55:26]>> Yeah. So, so basically a lot of the techniques that find anomalies are also are forecasting techniques because what are they doing? They're learning the normal pattern and they're saying okay I expect the normal pattern in an hour or in or in a day or in a minute to be this and then if the when the data arrives and it's not based on that forecast that's an anomaly that's the definition of that anomaly in that sense. So the techniques are tightly coupled and the interesting part is what Arun said correctly and I couldn't agree more um is that you know the when you're when you're trying to train forecasting models the anomalies in your data can really hurt you uh can really hurt how you how you train your forecasting models the accuracy of those forecasting models and we've seen a lot of examples of that uh with lots of different types of data Um by >> all right so uh >> you have to do something >> uh before we jump to the Q&A we do have one poll question um so if uh we can bring that up which is basically how important do you believe having anomaly detection is for your company based on uh uh Iris quick primer in the beginning and all the things we've talked about.

[00:56:47]So uh let's uh give you guys maybe 30 to 45 seconds to answer this and then we'll I'll uh um I'll go through some of the questions from the audience.

[00:57:21]All right.

[00:57:22]>> So, yeah, this is this is interesting.

[00:57:24]So, I thought maybe you know it would have been nice if we gave only critical super critical and really really critical.

[00:57:35]>> All right. So, now to audience questions. Um, so I'm going to skip around because some of these we may have addressed more during the presentation, but uh, uh, I'll try to get to as many of these as possible. Uh, we only have a couple of minutes left here. So, first of all, to both of you, what is a reasonable false positive rate? I guess it really depends, right? So, but uh, yeah.

[00:58:05]>> So, Arun, >> so yeah. So uh can you repeat the question?

[00:58:09]>> What is a reasonable false positive rate?

[00:58:14]>> Yeah. So again it depends on the context but let's say if you if you are in a in the ops team and uh if you get more than 5% false positive rate and I it also depends whether it's you know uh >> what time of day you're getting.

[00:58:36]Exactly. Exactly. So uh now without putting too many uh uh you know uh if that if that yeah I think anything more than 5% uh would probably uh have a low adoption rate. I mean and I'm being very conservative here. Yeah.

[00:58:57]>> So actually uh uh >> let's uh let's move forward the IRS so that we can answer more more questions.

[00:59:04]So I'll just I'll have one one of you ask answer each of these questions. Um so Ira, so which criteria can we use to determine which anomaly detection method is more likely to be suitable in each case?

[00:59:22]>> H so that's uh yeah that's a that's a great question. the the best way is to actually have some known anomalies in the data and run multiple methods and see which one produces the lowest uh false positive rate and catches the the true uh the true ones with the highest.

[00:59:43]So do uh you know any type of ROC analysis or uh or any other type of analysis. If there is no label data at all, uh then really you're left with some measures that are proxies of the quality. And what we found to work best is to first look at how well do we forecast does the algorithm forecast the next samples um on one hand uh and then on the other hand, how many anomalies does it produce? And you want to do a trade-off between the two because you can get a very accurate and narrow forecast but it might produce a lot of anomalies and most likely a lot of them are not correct. So there is a trade-off between uh between those two and that that would be if you have no labels that would be the selection criteria.

[01:00:33]So Arun very briefly uh what are some of the pros and cons of open-source code using open source projects for for something mission critical like this?

[01:00:45]>> Yeah. So uh I can speak from my own experience you know what we built at Twitter it was we never uh claim that it will work in every context. So it's very important to understand that people have this tendency to use things off the shelf and expect it to work especially in the realm of anomaly detection.

[01:01:04]Anomalies are very contextual in nature.

[01:01:06]So uh uh it's very unlikely that something you uh uh grab from the internet unless it was built for the same domain will work out of the box. So you will either have to tune the models or you may have to develop new models.

[01:01:23]So keep that in mind. you know, as you're exploring, uh, the use of models out there in open source >> and also as Ira alluded to earlier, uh, it's one thing to build an anomaly detection system for one time series.

[01:01:39]>> Yeah.

[01:01:39]>> Uh, but uh to really do it systematically at scale uh somewhat semiautonomously with uh lower operating costs uh is another matter. So Ira um question from the audience which is uh do you have any suggestions as to h uh how people can reduce false positives?

[01:02:06]>> That's a kind of >> wow that's a long uh it can be a very very long answer. So the way we found the best way we found to do it is to actually get feedbacks on alerts uh from on anomalies that we find for different I mean if you're building it for some some group collect their feedback when they get the alert itself and then uh when you collect it. So Debbie if you can mute your line. Um so when you collect it then you can then you can actually go back and fix what's wrong with it. That's the only way we found to really reduce false positives especially because of context. A lot of time there is context around what is an anomaly. Uh and even if mathematically mathematically it looks like an huge anomaly from a contextual point of view your users might not view it as an anomaly at all. um or might not view it as an actionable anomaly that they want to get. So you have to collect that feedback and that's the way we we build it uh to to really fix things. It still has to work with minimizing false positive out of the box as much as possible, but if you want to get the extra mile uh and to get to that 5% that Arun was alluding to, that's the that's the way to go. Without it, we've seen we've seen it go around 20%, 15% false positive, 10%. And it's still much better than a lot of the static threshold methods out there. Uh much much lower. And it also depends on the quantity. So if you send a 100 alerts with 20% false positives, it's not the same as sending 10 alerts with 20% false positives a day. So it depends on that as well. So this might be a question back to you Ira but uh uh Arun feel free to weigh in but very briefly uh how do you tackle the curse of dimensionality when you are using unsupervised anomaly detection in a multivariant time series.

[01:04:13]>> Oh yeah that's a that's a tough one. So a lot of regularization. Uh so you have to have regular regularizers in every type of model that you're using. Um for example in some of the models that we're using even the univariate ones we have uh either basian prior that regularize the output of the models. Um and if you have other types of models you want to regularize if you're using LSDM you want to do dropout. So each model has its own technique. Uh but it must be there otherwise it overfits like crazy.

[01:04:47]>> So I think uh that's all the time we have. I'd like to thank Ira and Arun once again and uh uh on a personal note both Ira and Arun are are give the most popular talk at our AI conference. Uh they usually give a joint presentation on on deep learning uh for time series.

[01:05:10]So, it's uh it's uh great to do this with you, too. And uh it's funny because I think I'm the one who brought you guys together to begin.

[01:05:19]>> Yes, you did. That's right. That's right.

[01:05:21]>> Forced marriage. But uh uh yeah, so this this has been great. And uh uh our organizer coordinator Debbie has her email up. So if you want if you have further questions, feel free to ping Debbie. But other than that uh thank you all for joining us today.