Graph Analytics and ML for Fraud Detection Using TigerGraph

[00:00:00]all right I think it's time to start otherwise we are going to run out of time so good morning good afternoon good evening thank you for joining this graph gurus Workshop I'm SAR barani a Senor solution architect at Tiger graph emia today I will show you how to build improve Business Solutions with graph and machine learning in our case this approach allowed us to double the performance of a fraud detection system this increase in efficiency was achieved by my manager Richard Henderson our lead solution architect inia whose webinar recording I hope most of you watched prior to this Workshop Richard is also on this call he will support me with answering questions during and the end and at the end of the workshop talking of questions and answers um little housekeeping first all attendees are muted but you can ask questions at any time using the Q&A tab in the zoom menu all other comments please sent to the chat of course this Workshop is being recorded and the link to it will be emailed to you with the slides an extract of the slides is already shared you can find the link in the chat window of Zoom you can use it in case you missed the step and during the exercise and you need to go back if you have any issues with the zoom please contact the organizer via the chat the exercises will be done using a tiger graph database instance running on tiger graph Cloud if if you have already created your instance and I hope most of you already did please make sure it's up and running if not go to my solution page on tiger of cloud and start it and actually I have to do it myself so I just come here select solution operations and click on start if you haven't created the solution yet please do know please do so now uh so that it could be ready by the time we start the exercise as it of humans to provision the resources in addition to this you will need the schema export that I have mentioned in the prerequisites email downloaded there is a link in the chat window to all of the resources that we are going to use today that you might need today you will find um the schema export or graph export file as the first link we will use a python Notebook on Google's collab this has been shared with you as well but if you want to use it you locally for example in your jython notebook you can download it using the link sent in the email or shared in the uh chat window finally we are going to import data from a CSV file stored in a AWS S3 bucket uh but the file also has been shared with the link to it in the [Music] chat as I mentioned at the very beginning my goal is to show you how to build an improved Business Solution with and machine learning in our case a solution allowed to double the performance of a fraud detection system of course your use case will be most likely different but the concepts and most of the steps will be still applicable because I'm not a data scientist and I do not have the domain knowledge not at your level anyhow I won't be able to tell you exactly what application to build but I can show you how to build it so here is the agenda how to achieve that first we will review the ways we can work with the Tiger product and ecosystem this part will be interwoven by the first dose of Hands-On exercises then I will review our use case defro detection and what Richard did and what results he got this will be followed by the remaining exercises where we will feed the results coming from the graph into the machine learning model and as usual the workshop will end with a Q&A section when we can answer all of the questions that could not be answered during the presentation and exercises uh if you would like to know more more about our Origins and achievements you will find the information in the slide de that will be sent to you and that can be now access through the link shared in the chat window now um a very brief introduction to graph databases and graph analytics for those who are completely new to these subjects a graph database stores and processes vertices or vertexes and edges this is based on the concept of graph Theory who Origins can be traced back to Leonard O's 1736 paper on the Seven Bridges of kbur so the science behind graph processing analytics is actually ordered on the science behind relational databases that really started with ef cod's 1970 publication vertices are somewhat similar to database tables or data frames in programming languages and data processing Frameworks in do aare text just like a row in the database table or data frame contains information about a real life en like people companies products or about more abstract Concepts but instead of columns vertices have attributes or properties that could be of simple or complex data types while a table or data frame is a main unit of storage in an other system in Tiger graph the vertex instances are Standalone storage units vertices are often visualized as a circle or a dis although the shapes or images are also regularly used in graph Theory they are also called nodes but that could be ambigious in a computer science environment where components of a network or a cluster solution are also typically called nodes so I will stick to calling them vertices edges to me are the real differentiators that make graph databases completely different from other data storage and processing Technologies edges represent relationships between real life entities I the vertices this is the important part of graph analytics is mainly about discovering and exploiting the value of these relationships that often are not trivial not direct but require multi hope tipling analysis through these relationships a graph database can provide context for the analytics utilizing the value of relationships these ways is very difficult costly and simply impossible uh with other Technologies edges can be considered similar to a join or a primary key for increas relationship in a relational database but there is a significant difference in a relational database a foreign key only describes a possibility of a relationship between two tables and the join in a query needs to find out if there was actually a link that is costly this is why relational data warehouses use a data modeling technique called star schema which reduces the number of joints on the other hand in Tiger graph the relationships are created at the time the data is inserted and can be seen literally as you experence it soon um Can immediately there is no need for extra work to follow the links in fact if it is very easy to follow any number of links or hops without any additional work or guessing this is what we call Deep link analyzis we can also say that edges are first class citizens in graph databases unlike for increase in relational ons an edge can be undirected directed or directed with a reverse Edge some relationships like friendship are not directed or the direction is not meaningful or relevant after all friends are all equal in other situation though you need to express a hierarchy or a literal Direction like ownership or events following each other to Ure that your graph carry follows these relationships properly you want to Define directed edges to implement them but in case you might need to go backwards for example to find owners of a property you might also so want to add reverse edges edges are generally visualized as line or Arrow depending on whether they are directed or not edges can also have attributes or properties just like vertices and this flatter feature makes tagap a property graph database these properties on both vertic and edges can be used for filtering sorting aggregations and in various buil-in and user defined functions and algorithms usually exploring a graph is one of of the possible first steps to recognize the often not obvious and longdistance relationships and their values but operationalizing these findings and insights is the ultimate goal this can be done by developing queries which are our kind of store procedures they are a piece of code stored and executed in the database at highest possible speed in a parallel fashion and without the need to move data anywhere we will see an example of this during the exercise the four main tools to work with tiger graph are graph Studio which is a web interface to design populate and explore graph and to develop Solutions the GSL command line utility that allows you to enter and execute arbitrary gql statements gsq graph structure query language is the query and programming language of tiger graph it's an easy to learn the powerful language model on the name suggest the structured query language of relational databases we will take a closer look at it later uh the next is the rest API that is mainly used for integration and operationalization and aim mostly at data manipulation and downstream consumer applications and finally byph a community backed open source python connector intended to be used for data science and data engineering your tiger graph instance should be available by now so let's get started with graph Studio yes it's up and running ready so I click here on this applications icon and the first link is graph Studio which opens a new tab okay the homepage shows every functionality graph Studio provides the ones that we will not study today are visual query building us or build graph patterns here where is my mouse csor this one and um migration from relational databases we have uh good webinars if you want to learn more about these we will also not export the schema although that's effectively just saving the database metadata into a file so let's start with schema design on the left side we are going to create a simple schema for start with two vertices and two edges the vertices will be the person vertex and a movie vertex and the the edges will be a friend Edge linking person to person and a likes Edge linking a person to a movie to keep it simple let's have only one attribute for each vertex types a name for person and the title for a movie so let's create the person first let's click on the plus icon here to create it oops I'm not sure why do I have here this it's not supp closed to be here okay so let's create the um person vertex first uh when you create a when you define a Vertex you need to First specify a name which must be uni unique within all of the vertices and edges uh then you need to specify a primary ID which is which has the same function as the primary key in relational data databases it can be called anything it's typically ID or or something like the back them combined with the ID C for example part ID the ID should have one of the available data types Additionally you can specify the styling of the vertex Pro Prov picking a color and potentially assigning an icon to this and finally you can specify the attributes like case of an idid you have to give it a name uh you have to pick an attribute type and you can specify what would be the default value if this data is not provided uh during loading or or not available in any way Additionally you can indicate if you wanted to index we wanted an index on this attribute indexing just like in case of relational tables improves lookup speed but comes with maintenance cost so index only those attributes that are frequently used in queries so uh let's create the vertex as I said it is called person the ID is a integer value I leave the color as it is uh as I said this person will have a name which is a string attribute I'm not going to index or provide a defa for you for this let's create the other one the movie it has an ID again an integer and it has one attribute called title which will be obviously a string now that we have two vertices let's create the edges um to that to do that let's click on this Arrow icon on the top and then click on the first or uh Source uh vertex type and then on the target vertex type so this will be a like relationship between the person and the movie um let's call it likes it's a directed relationship after all a movie cannot like a person but I don't need a rever so I disable that so it's a relationship between person and movie and let's give it one attribute called first watched a simple integer to indicate the year when person watch. movie for the first time and finally let's create the last Edge for this exercise by clicking on the arrow button again and clicking on the person twice after it both the sort and the target of this uh Edge and it's called friendship from person to person and again it will have one attribute called since again a simple integer to indicate the year and here we go we have our first uh graph or schema designed now that we have this we need to publish it to the database because at this St it's only Lees in the graph Studio editor so by clicking on this up Arrow I'm going to publish this uh schema which will take a few seconds and now that we have these vertices edges we can create a graph for this clicking on this graph I can give it a name simply my graph which is a typical name for uh graphs uh in taret graph and I choose all the vertex and Edge types to be included this might take a while because tiger graph is going to create the story structures and optimizes a few things so let's move on in the me oh actually it's already done very good let's move on anyhow um and let's jump a little bit ahead as you can see on the left side The Logical Next Step would be to map data to graph and then load it but let's Jump Ahead a little bit to graph exploration to get a little bit more familiar with what we have created oh sorry I should have cleared this um so graph exploration is us name implies mostly about exploring the graph to find out what we have in the graph but this graph is brand new it has nothing so let's create a few uh instances of vertices edges first again very simply you click on the plus button and you pick which type of vertex or you want to create in case of vertices I'm picking person I gave it an ID one and I call it after myself uh let's create another one ID which must be unique will be two and let's call it Albert after myON so we have now two people two person vertices let's link them through friendship I'm a friend of my son and since the day he was born all right so far so good and this is a French relationship so it's not an arrow but obviously if it was a relationship called parent or child it would be a a directed Edge Let's create another uh vertex this time a movie ID number one this is a different vertex type so I can use ID one there will be no collusion with the ID one of of myself as a person vertex and let's call this movie Star Trek back the first the original movie that I'm a big fan of and let's create a relationship between these two so sard likes Star Trek and I funly enough only have seen this movie last year for the first time so we have now a very simple graph instance with three W stack instances and two edges OB viously in production environments graph data is not entered manually this is for for illustration purposes so the two steps that we will soon review are defining data sources and data loading rules plus the data loading itself um but before we do that let's have a look at another function that is accessible from the graph Studios homepage import ing a solution a solution includes metadata the definition of the vertices and edges the schema the graph itself the definition of data sources data loading JV saries and so on these are stored and from the database but the solution also contains information that is only used by gr Studio are not stored in the database like vertex and H colors and icons the layout of the schema design the current and sa layouts of grph exploration and so on in any case data is not included that needs to be exported in different ways that we are not going to cover today the reason we are taking a det tour here is that I provided you with a predefined schema that I would like you now to import into your graphic stest this saves us time I we don't have to create a more complex graph from scratch and we have data for the schema so that we will use later and we can load now importing a solution it's important to remember importing a solution clear the database it removes all of the data and metadata already there and creates new object and new metadata but not data because it's not included so use it carefully exporting a solution is a great tool by the way to back up your work for example exporting B your work from a development environment or to share a solution with others but yeah as I mentioned we are not going to cover this today so click on import an existing solution and navigate to the file do I have shared the link to and this is called graph fror export.

[00:20:36]thatg open as I mentioned it will erase everything you get this warning but this is actually what we want so I allow it to overwrite this import might take a few minutes perhaps um maybe not let's see how long will it take don't forget after after all it's a cloud instance very potentially very far from you so pushing through the the export file might take a bit Next Step that we are going to do is actually reviewing what we have just loaded almost there sorry this is a step that we couldn't accelerate much it really depends on the infrastructure between yourself and Digraph Cloud okay so let's see what we have here so we created a new graph called loaded a new graph called FR graph so let's have a look at the design it's more complex but still not a very complex schema um the it's it's it supports use case which is about customer buying various products in various categories from Merchants through transactions which have a number of attributes like obviously the amount dates um and and various graph related uh or gr specific information and these are linked together with these edges so um let's have a look at the data mapping that we have skipped previously tiger graph can load or inest data from a number of sources the most obvious of this is loading from files uploaded from your local computer but this is typically done in development and test environments you can also import data stored in AWS S3 Cloud blob storage the equivalent of aure blob storage or Google cloud cloud storage which are not yet supported but soon will be loading from s is what we will do in a minute in production environment it's typical to load streaming data using kka it is also possible to import data directly from relation databas databases as I mentioned earlier you can also use the rest API that we will be discussing a bit later to observe data and there are plans to implement data sources such as the previously mentioned Cloud story Services hop hdfs FTP and HTTP sources what you can see on the screen now let's zoom out a little bit is a data source this file here and data mapping arrows let's start with reviewing the data source first as I mentioned we are going to use a data file that is stored in U AWS S3 it's a CSV file this is already preconfigured so you don't have to do anything but I quickly show you how this was set up and how alternatively you can load data from local files let's start with this later one because it's bit simpler or easier so if you want to upload a file locally from your computer you click on this plus button here and then pretend it's not here and then you click on the plus button again and then you navigate to the file that you want to upload this is effectively the same file that we are going to load from AWS just it's a smaller one too not to waste too much time on on data loading itself so if I open this file this is uploaded to the server to the tiger graph database server and now by clicking on this we can tell tiger graph what it has to deal with there are uh a number of options on the top first the file format whether it's text or compressed by any of these listed format but it's so I'm not going to change it what is the delimiter it is a comma automatically detected but you can pick something else including top characters if necessary what is the end of line character I just leave it on back sln which is the natural line and character depending on your platform and enclosing characters this one is enclosed by the strings are enclosed by single clot so I I use this one and obviously this file has a Fed header so I click this one has header to avoid loading the header row as as data so this is how you define the structure or or tell the structure to Tiger graph of your uh Source data file I'm not going to do anything here because we have already an S3 based file definition that I'm going to show you oh I shouldn't have picked on that so if you want to load data from an S3 bucket then you click here and then again you click on the plus sign to set up a data source a data source will be a description which you will instruct tiger graph what what to connect to in AWS you have to give it a name it could be anything as you see here I created minus S3 DS S3 data source and you have to provide the AWS access key and secret key which can be provided to you by your AWS administrator I'm not going to show this because it's already set up but I show you that once it's set up tiger graph will show all of those buckets in S3 to which you have access to going to use this one where I have only one object or data file this sorry CSV file that we are going to use actually it has the same structure I'm not going to set it up and anyhow it's already set up but you could configure it as if it was just a file um as you saw tiger can read data from CS and similar separated by files like top separated where use tsvs but also from Json files Json is flexible technology independent way for data exchange but this will not be covered today once you have uploaded and configured the source files you can map map their fields and columns to vertex and Edge attributes and Edge primary IDs this is done by clicking on this double arrow icon and then clicking on the data source and then on the uh um and then um on the target Vertex or Edge so let's have a look at it this orange arrow indicates that this is the mapping that I'm working on and all you have to do here really is is simple you just click on the on this first uh table uh which shows the columns or or or fields of the source data the data source and some sample data and then on the other side you will see the structure of your Vertex or Edge including the primary ID and the and the attributes so for example this is a transaction I know that the sequence is the transaction ID and I could map various values here obviously it's I'm not going to do this because it's not a valid one but mapping is this simple I delete this to make sure I'm not going to ruin the experience later um when you as as you can see you can use one source data source to provide data for any number of vertic and edges and conversely you can populate a Vertex or an edge type from any number of uh sources so I could have two files here mapping to transaction for example if I have two different sources a few other things you can do here um so it's not just a simple transfer of data from source two vertices but you can also use a number of functions for example string concatenation to create a composite ID or for something similar and there are number of string um um and date manipulation and the data type conversion and other functions at your disposal further more you can also specify filters you can tell which columns or sorry which rows to be uh loaded or which ones to be skip but we don't unfortunately have time for that now I would like you to give it a try I would like to ask you to delete this edge here between the data source and the customer to transaction Edge it's a very simple Edge as you can see here we are mapping the ID of the transaction and the IDE of the customer to the Target and Source vertex and and uh and do creates The Edge itself during the data load you after all you define the Edge by five things the type the source uh vertex type and Source vertex ID and the T Target vertex type and Target ID so I would like to ask you to select the mapping click on the delete button or toubin button and and redo it as it is shown on the next slide oops sorry go back so simply create on sequence and map it to the transaction uh vertex and then click customer column and M it to the customer vertex very simple I give you a little time to do this in fact actually I have to redo it myself sorry for this but you will see what I'm doing so I'm clicking on this Double R icon mapping the data source to on edge SE goes to transaction customer go to customer once you have done this you once you have finished the data mapping you need to make it available for the database again this at this stage in the editor of of of graph Studio only so I'm I'm publishing this data mapping in the background it creates procedures that we will execute next time data load uh jobs actually not that it went fast so let's have a look at the data loading itself in fact actually there is not much to do here really because all you have to do is click on this play button to start data loading and actually I started and I describe the rest for you why it's it's being being loaded yes I want to load this so um data loading is is done in parallel fashion is possible you can also pose and resume data loading and you can clear the data store it will only remove the vertex and Edge instances but it will not actually remove the definition so it's it's it clears the data store but doesn't um clear the metadata It's Already Done um important note here when you load data to a Vertex with the same primary ID or an edge with the same type same source and Target vertex then the existing instance will be overwritten without warning this is called upsert if the new data is not there yet it will be inserted if it's there it will be updated this could be a good or bad thing you have to decide what you want and make sure that you know what's going to happen um now that the loading is finished we can click on the data source and we can have a quick look at the statistics uh this is a good first tool to check data quality because it shows how many lights were rejected for various reasons it shows also the loading speed and and and you can see a graph showing the activities and the outcome so far so good so let's move on and go back to graph exploration that we very briefly touched previously um there are a number of ways to sorry to explore your graph again it shouldn't be here so first of all if you know uh a if you know the primary IDE of a u vertex you can quickly look it up by specifying the type the primary IDE and clicking on this button but you can also brow the vertex by picking up a number of vertices of the selected types and you can even specify filters which ones you want to pick up so let's just quick do a quick browsing here so we can see here Merchants categories uh transactions and customers so let's do the next exporation step double clicking on a transaction double clicking shows theing displays the first hope relationships the first hope edges between the selected Edge and it will show which um vertices it is uh linked to through the edge so very often it is enough just to start browsing and then expanding oh interestingly enough these two um um transactions um were related to transportation and maybe this one yes as well so it's very easy to start to understand what what is the data we are facing here another thing that you can do if double clicking is too much that you can expand uh by selecting a number of it little bit too much perhaps but let's say let's select these two by selecting these two transactions I can say that I want to expand it to any other uh vertex types through any other edges although obiously you can select which ones you want you can even specify filters so let's expand this and it shows me what are the the relationships and then I can continue expanding this as long as my browser can handle the number of entities to be displayed first and more very quickly you can use P finding algorithms finding the shortest pass one of the shortest passes all of the shortest pass were all passes between two Vex sky typ vertex entities or you can find connections between entities we don't have time for this today unfortunately and and you can actually run queries which will return um subgraphs a combination of vertices and edges we will talk about this a little bit later but we need to move on now uh and talk a little bit about queries as I mentioned the is are the the next natural step and the way to operationalize our findings uh queries need to be deployed in the database which can take some time so as a first step I would like to ask you to install or deploy these queries by clicking on this up Arrow icon in a little box it will take a while but um because tiger graph is doing a lot of optimization in the meantime it's not just uploading the file so let's let's install it and I'm going to provide you a few details about queries while it's it's happening so queries in Tiger graph are what you would call uh stored procedures in in relational databases it's a piece of code name saved and executed within the database operating over the data stored in the database and they have a number of benefits first they run in the graph so there is no need to move the data for example to pull it out from your database into your python notebook for analytics they save time and simplifies operation second they can be compiled into machine code this is what actually we are currently doing uh at the time of of deployment uh so that they can be executed as fast as possible without the need to understand and interpret The Code by the database every time third the query code is executed in a parallel fashion diagraph is an MPP system a massively parallel processing solution meaning that the database might consist of multiple Computing nodes sharing the task of data storage retrieval and processing this is transparent to you no matter if you have only one node in your database or hundreds the same code runs everywhere and you will see one database only you don't need to know where specific pieces of the graph are stored the system will find them for you finally queries installed in the graph have a rest API endpoint so that they can be easily put into production with minimal effort to integrate them with consumer applications queries are written in a language called gsq or graph structured query language again as I mentioned earlier it's it's closely most on the relational databases SQL language so you will find a lot of familiar keywords and constructs if you have relational database um experience let's have a quick look at it um of course due to the unique nature of graph data structure some specific modification additions needed to be introduced we don't have time to dive into the language but we have a lot of resources starting from documentation to recorded webinars where you can learn about it and we would be happy to help you with us the one thing that I would like to say now is that the logic in a gsq query is very simple you start with the seed vertex set one or more vertices and then expand from these vertices into other vertices through edges between them and while you're traversing or moving uh from one Edge to one verx to the other you can specify conditions which edges or vertices should be touched during this traversal uh pretty much it's same thing that you would do manually in the graph exploration part from one set to vertices to the other and then the next one and so on the result of this traversal as I said is another ver set which could be the base for the next hope so this way hope by hope you can reach your destination um one hope in in our query is Express us a select statement then here is one we start with a starter set which is all of the customers uh and through the customer transaction age we find all of the transactions and we do some manipulation in the meantime obviously this is a slightly more complicated query than the usual uh but if you have a look at Richard's presentation I'm fairly sure you will understand what he's doing here um gsq is a called it's called a touring complex language which means in practice it's not just a query language like SQL but it has procedural statements for example as you can see here we have ifs and loopings and so on so you can write actually very complex um real programs in it to in the database um and then this is executed in the graph um in fact the language is so rich that we have a great number of sophisticated graph algorithms developed in uh gsq and and shared as a library it includes functions like centrality scores including page rank CL that was invented by Larry Page of Google bu finding Community detection similarity similarity algorithms and and so on these are ready to cook templates that you can easily adjust to your graph and needs uh we are not going to gql I mentioned here and we are also not going to cover the gql common line utility here as it is not needed for this Workshop so let's have a quick look at the next way to work with tiger graph and that's the rest API the as I mentioned earlier um the rest API is a PR interface for the database uh we provide HTTP end points.

[00:41:41]return Json responses and um these these functions cover a number of the things like uh data manipulation metadata retrieval buil-in graph Al and so on and as I mentioned in addition to this built-in default uh rest end points all the queries that are comped in the database provide their own endpoint so you can immedately use it through various tools uh including even your Ro browser or assuming that you have the authorizations to do so um this is all very useful I mean it's very easy to operationalize Tiger graph and and and access it with great number of tools uh but per is not the most commoning to use to to use Curl or web browser um this is why we work on language conductors the first among those being the python conductor called py tiger graph um it's a it started as a community initiative with tiger now contributing a lot it's not officially supported but you can get a lot of help from Community website not just from Tiger graph employees but other uh python users as well it started as a wrapper over the rest API but we added a lot more functionality and now it has access to all of the database function through arbitrary GSL commands um the purpose of this is to bring graph analytics results into python but don't misuse it don't use tiger graph as a bit bucket by pulling out everything into python let tiger graph do the heavy graph analytics and use the results the for example for care queries to end your python model and obviously PPH being python you can use it from juper or uh call or or any python applications to get the data and visualize your findings and now finally reached the point where we can talk about the use case I cover the tools let's look at the use case and the implementation I hope that you had the time to watch the recording of the presentation by Richard but for those who couldn't watch it here's a very brief summary um so Richard was using a public data set and built a machine learning based R scoring application to valid that using tiger graph or graph analytics gives materially significant improvements his goals were to make the results explainable and the data and the MEO method replicable to make sure that the runtime performance is the same or better done the Legacy non graph based solution and to minimize rocket science and maximum data science and the initial outcomes show around 50% increase in fro detected with a 60% reduction in for positive um the key behind this Improvement was realtime generation of new features from the graph generating these using other Technologies would be difficult too long or impossible um as we saw the schema contains a few vertices we describing retail transactions and those vertices and let me let's have a quick look at back at this they have a lot of attributes um or properties and Richard used these and the relationship uh between these um vertices or transactions and and and person to extract a number of new features in real time over a parameter history of days and here you can see the list I'm not going to go through this and you can anyhow get more information better information from Richard's presentation um so he used these and the results are very convincing as you can see using R data only around half as much fraud use cases went undetected um and there were nearly 60% less false positives and the results are also explan so you can see using real time Aggregates and graph context there is a more detailed picture of what really influence of of Fraud detections and then you can see some of them some of these uh generated newly generated features have a lot more importance than the the traditional traditional delegacy was coming from a um a trans from transaction data only um let's bring this to your notebook um Richard had to you had to develop two functions to prepare the data for analytics so let's go back to this quickly these are the two functions the fix statistics and fix fixed transaction Aggregates so let's quickly execute this by selecting the function and clicking on this run button the first one should complete in no time and inde did while the F transaction Aggregates probably will take a little bit more but not excessively long amount of time and while it's it's running it might be finishing soon let's have a look at the notebook I have shared the notebook with you and you should have the link um let me find the link in actually I can copy the link from either you can copy the link from your from the mail that has been sent to you I can actually just copy paste from the chat window that where I shared it previously C here we go I said here we go come on [Music] Google yes so this is a a standard J pyter call of notebook um if you want to use your local Jupiter instance you can download this notebook or in case you don't have a Google account uh but I'm going to use it in in callab right now um first of all you have to say if you want to change make any modifications and we will have to we need to save it locally save a copy on your drive I think I get a new one yes so I can focus on this one and um one of the first things we have to do is install a few things that are not uh by default available in collab and probably not available in your computer might not be on your computer uh if you are using a local jupyter first of all we need to install pyer graph and Richard are using two modules category booster and these IPI vets that we need to install and some modification needs to be made so let's execute this first cell it's it takes some time while it's running let's check back to okay don't worry about this um in in the D environment we limit the for for for a free instance um the execution time for a query is limited but the query has been executed in the background so we will be able to use it so the two prep ation steps has been done and we are going to run from The Notebook this main function which returns over a period of time the new features so coming back to this yeah yeah everything is installed let's hide this okay so now that we have everything is ready there is one more thing that we need to do we need to set up the connection to Tiger graph and the details are sent here but I'm going to show you what you need to do with a little description first of what we need to do so obviously uh tiger Enterprise solution it comes with Enterprise level and Enterprise solution that comes with all kind of security features and the instances in pyer graph are secured to be able to get access to this we need to get a secret a secret is a small random identifier that are typically provided to you by your database administrators but because you created the instance and you're the administrator you have to do it you have to create it yourself this secret will be then used by py graph to create a session token the session token is the the authentication token for the session for the execution of the um notebook don't worry about all of these details they are partly described in the um notebook and then obviously we have detailed documentation describing all of the security features and in most cases for example if you create a local virtual machine or Docker image you probably don't have to worry about any of this at so two things that we have to specify here we obviously need to tell the notebook where to connect to and I provided all of the details except the actual URL uh that you can get by simply copy pasting the first part of the URL from your notebook which is in my case this St g- Workshop so I just copy paste this this contrl C so this is the the the unique identifier that the email ask you to create it has to be unique because this is now an Internet domain name and you have to make sure that you are not using each other's uh the I database accidentally so I copy paste this and replacing this replacing this placeholder here and the next thing is to generate a secret for this we have to go back to graph studio and have a look at one feature that I didn't describe in detail and it is the administration Pages mostly you will use graph studio for Designing scheme of populating it exploring the graph and and and right queries but you can also uh do some Administration here mostly monitoring um but also you can manage users you can add users and specify their access details here what we will do here though is only generating a secret a secret is a named random piece of string that I'm going to create now and I just um create give it a name my secret it could be anything as long as it it meets certain syntactical syntactical requirements and I have to then click on this plus button the database is going to create this random string which be linked directly to you within the database this link sorry this information can be only seen now as you can see a previously uh created uh secret is is off cased so it cannot be used any longer so make sure that when you are creating a secret you copy paste it and save it somewhere I'm going to copy paste this newly generated secret into my notebook here to replace the second place folder and and that's it that's all we need to do to set up the connection and now we start executing the notebook again as I mentioned earlier I'm not a data scientist I won't be able to tell exactly what Richard has done this I more or less understand what's happening but I can't explain why exactly he chose what he chose to do um uh you can ask him you can probably get a lot of other lot of information right out of his presentation I know actually because I have since presentation so I'm what I'm going to do here is just to show you the first few um steps and uh to show that that it works and then we can discuss the rest later in the Q&A part or or offline so now that everything has been installed I have to import certain things that we are going to use there are some settings here finally we are going to establish the connection specifying the token and here in the first statement and the only statement that that is needed we are going to run this installed query going back [Music] to here we are going to in use this query installed which actually as you can see here comes with this rest end point so this rest end point is wrapped into this python function run installed query we specify the name and the parameters 14 days of History the default is one this history is 14 and then you can specify some additional parameters which is not relevant now and we get back a Json document but we have a function to convert that into python uh pandas data frame I actually make it shorter because I think I'm running out of time let's say 3 days I'm not sure how it will impact the validity of the data but it's probably not that relevant so when we are executing this this is running over I don't honestly know how many but several actually I can't tell exactly how many because if we go back to load data we can have a look at the statistics we have uh nearly 600,000 transactions so this function is running over running through the 600 and transactions plus all of the links uh all of the the edges uh and it should finish soon but again because it's running in the cloud I can't really control can't really tell in advance how long it will take okay it has been executed so let's move on to the next step next few steps which will allow us to see the data and here we go so there is 600,000 uh rows of data returned and the new um um features calculated new features calculated over a 3 day period and the rest is is complicated data science complicated to me but I really hope that you have the experience and the skills and then we'll be able to uh understand Richard's intention here coming back to my slides now that we actually build the solution I would like to draw your attention to Our Community Resources we have a number of resources for the community itself we have a a forum um we have a chat channel on Discord we have a size for developers and we are on most important um um social media channels and that actually concludes my presentation at the very end of the hour I hope that you will still have a few minutes to um for a Q&A session Richard derria did we have any questions that went unanswered or do we have any questions that needs to be discussed right now yes of course the presentation will be shared you are going to get an email with the link for the presentation and all of this information that I have sent to you all of the download links will be included in that any questions any other questions regarding the solution or the data science part I have server a number of questions answered during the presentation so I um not sure if we need to wait for any other answers or any questions that's fine zadia I know Richard covered a lot a lot of questions during the actual session um and you've also shared details of where you can we can all be reached so um another question's just popped up yes I see it where does the jupy a notebook PR so you can use it in in two ways if you have jupite there install locally for example part of the Mona distribution then you can run it on your notebook or any hardware virtual or physical we use we chose to use it run it on collab Google's implementation of a dupy notebook like um data science collaboration platform we chose this to make sure to minimize the the need for you to download and work with files in long term we try to elate all of the need so it will be all cloud-based uh so we use collab but my understanding is there are other platforms and we are looking at those if you have some preferences if you would like to see this running on different environments pleas let us know any other questions oh let me know if I can help it actually that would be really nice as I mentioned earlier we definitely want to expand into Azure in fact tiger cloud is already implemented on Azure and we are working on expanding to the rest of the ecosystem and I'm I'm honestly not that familiar with with Azure but I'm fairly sure that there is a similar platform and and we would be happy to um to work with aure it's obviously very very important platform to us to us thanks Linda and I think because so many answers were answered so many questions were answered during the um session I think we can thank you now and uh your participation uh as I said you will be given all of the resources that we use today and here on this last slide you can see my contact details um yes the tiger is part of my Discord handle um but you will be sure find me by my first name and thank you very much for your attention have a good day and hope to see you back soon on the community uh channels or on social media or the next gra grp sessions thank you goodbye