Mastering Parser Combinators in Rust with Nom | Comprehensive Guide

[00:00:00][Music] [Music] [Music] [Music] [Music] [Music] [Music] [Music] [Music] [Music] [Music] [Applause] [Music] [Music] [Music] [Music] [Music] [Music] [Music] [Applause] [Music] [Music] [Music] hi uh welcome to the March talk for Russ Dublin Meetup this is our first talk of uh 2024 bit of a slow start to the year which hopefully we have another talk wind up for um next year uh for the moment we're going to stay remote we did a few uh onsite last year but the numbers were low uh but we definitely want to have a few more this year on site so if anybody's got any interest just let us know in the chat etc etc H to get into the topic today's talk is about kind of paring data for want of a better phrase I think you know we've all come across Json XML and well structured data and there's lots of libraries out there for us to be able to par that however at times you're going to encounter custom data types or data that's not well structured and one approach I suppose is to use a Rex for things like this and we've all encountered Rex hell towards the end of a lot of code so another approach to that is to use parsers of combinators and that's what the talk today is about and I'm going to let our speakers talk about that so just to quickly introduce luano I think we all know he's kind of been a great uh speaker for us over the years and today he's accompanied Roberto who has worked a lot with Lano on the the Advent of code and there is a lot of other content from luano and Roberto available if you want to look that up afterwards uh really good talks on their personal channels and blogs um this video will be available um uh on our on our channel in a couple of days Brian's going to do all the editing etc etc but for now we're going to kick off if you have any questions just pop them into the chat as they as they pop up and we're going to take all the questions uh towards the end but we'll leave plenty of time for that so I'm going to hand over to luano and Roberto thanks very much awesome thank you Alan for the intro I share my screen and hopefully you can see that and Roberto whenever you're ready I'll leave it to you to start yeah tell me when the screen is shared I think it is now yeah okay so hello I'm Roberto I'm lazy by Nature ponic by choice old enough to be rusted and I principally Engineer here and there and we can connect on link free SL gambus my S then I leave it to you luano all right hello everyone I'm Luciano by the way before I tell you more about myself all the slides are already available online and you can get the QR code there or the link we will also share the link at the end but it's because we have a few Cod Snippets that maybe you want to check out later so it might be handy just to get the slides okay again my name is luano I have a few budges that are totally not relevant for this talk so I'm gonna skip them I work as a senior architect at for theorm it's a cloud consulting company we do lots of AWS so if that's your thing definitely feel free to check us out and reach out to me and also totally not relevant but something I have to say every single time I am the co-author of this book about nodejs and the same pattern so if that's also your thing please check it out and let it let me know what you think and you can find all my contact details there I'm generally available on Twitter LinkedIn m on Blue Sky and whatnot so whichever is your channel of choice you can reach out to me and I'm more than happy to have a chat so let's start the talk what the heck is a parer and one I guess not necessarily academic definition is a parer is a program that can turn text or bites into structured information and to really understand what that means we can take this example so we can imagine we have a string that is describing some kind of point in a 3D space so this can be a potential way to represent that information as a string but as it happens we might need to use that information in a program so we might want to pass that information and extrapolate some elements and maybe copy over this element into a struct that might look like this because of course this kind of struct is something more useful because we can access the individual items in the struct and do something more useful with it so how can we actually pass a string like that and build such a struct now there is a very naive approach that I'm sure everyone has used at least once in their lifetime probably more than once and I like to call it string split like a madman because you you start to split stuff and discard stuff and eventually you come up with some information you can use and if we want to use some pseudo code it might look like something like this so this is a variable that that contains our input then we might use some kind of information called some kind of function called split that says okay take this string and split in two parts looking for a delimiter that is effectively a column and a space and then the two parts we are going to call them on one side label and on the other side remainder so we are basically separating the string in two parts removing that separator in the middle and we have a label on the left side and a remainder on the right side now we can keep going and say okay we don't really care about the label so let's look at the remainder and we split again this time looking at a parenthesis open parenthesis and what we are doing we are just trying to get rid of that parenthesis and you can imagine we can keep going we split again and remove the closing parenthesis and then we split again this time we can split three times so we want to get three segments and we just split based on comma and a space and at this point we have x y and Z which is the information that we really care about now this is an approach that works you can use it and it's mostly fine I guess for the most part but there are a few problems it's not always suitable because you might uh need to have very good delimiters and not always you have delimits that you can rely on in this particular case we have for instance parenthesis and quotes and sorry we have parentheses and commas and columns so that that's a would use case for that but in other cases you might not have such delimiters so it can be very simple for simple stuff but it might get increasingly hard for more realistic use cases and the other thing is that it's not something where you can easily handle errors like you need to write a lot of code to be able to handle errors consistently and probably that's not something you want to do especially if you have a complex protocol that you need to pass it might get very out of end very quickly instead you can think of use a regex we have regx and we take the same input so the same string as before then we say okay what should I match with ARX to be able to extract those three numbers that are there because I just care about them I will say that we have a string that starts and then it starts with the label Point column space then I want to match an open round bracket I have to escape it with the back SL because the round bracket is a special Charter for the regx I have cre capture group the capture group says there be there could be a minus sign then there will be a number of numbers at least one or more numbers and that is our definition of numbers so we also match the comma and we match another Capal group that is the second number of our point then we have a third capture group that is the third number of our point and then we match the closed round bracket and the dollar as you can see this is a little bit complicated to read and complicated to write and we Define like I said before the capture groups so we will have three capture groups one for each number that we want to capture they map to the Rel an number and we can name them we can through every reex library that is out there you can name a group so it will you will give an alas to this group that is indexed by number uh we have x y and Zed because those are the three coordinate of our 3D Point uh yes those underline yeah X Y and Z but there is a problem problem there is a something to be aware of rust doesn't have in the core libraries a regex library so we have to add this regex library to be able to use regex in Rust to do so we need to use the carad regex command and uh yes sadly it's not something that comes out of the box we have to install it not a big deal is is easy but something that you have to remember to do so let's look a little bit at some Rust code for real not s code this time on how to pass our 3D point in a 3D Point structure first of all we have to import our crate that we installed uh before we have to define the struct of our 3D point it will be 3 e32 x y and Zed nothing complicated then we start our main F main function defining the reg X itself the unrap here is safe because unrap in this case fails only if we Define the regx that is not correct but in our case this regx is correct because is are coded and because we wrote it and we tested it uh notice that there is an r in front of the open double quote because this is a row string to avoid to have to double escape the back slashes because the back slashes are are not escaped in this uh string because they are used to escape the special Charter for the regex itself then we capture on top of a string with our uh reg X that we just compiled let's say uh this unup is a little bit less safe because if uh the string is not matching that will be that will throw a fatle that will error out because in this case it's not because the string is are Cod and then it's valid so it will gone but remember when you capture on top of ofx H some input that can be an input from a file an input from the user always check if there is an error and always handle the error here for the sake of Simplicity we skip at that so the output of this will V captures data type that will contain the full match the key zero the first one that doesn't have anas then it will contain another three groups one two and three that will be our numbers 22 17 and minus 11 and they will also have an alas x y and Zed that is then used to address them so when you have this capture group you can address them by number or by alas in our case here in this line 14 where we create the point 3D instance of this data structure we refer to the number by to the capture Group by Alias we pass it as an integer and we unrap here thep is safe because of our capture group that only capture numbers so it capture something that is always a valid number otherwise it's failing before arriving here so is safe to unwrap in this place we unwrap X we unrap Y and we unwrap Zed we have our point and then we print it out so what are the problems with this approach some people when confronted with a problem think I know I will use x and now they have two problems H this is a famous say that is going around so yes Dr X have they have a pair but they also have their problems point one they are hard to the bug H they match everything or they match nothing they don't tell you why they did match they just don't match so when you start having a strange behavior in your software you have to think oh could that mean that the regx was not matching some kind of strange input that I was not expecting and instead of giving you an error you have to try and the bug then also learning to write X is a complicated require a lot of practics and uh the final result is not the most readable stuff on Earth H and they have limitation for example they are not very well suited for Pars in nested structures there are languages like CP where they have their own kind of regex that can match nested structors but by the book reex Forex is hard that to match the co the closing kly bracket of an Open kly Bracket if there are more nested kly brackets in a string for example adjacent so it's hard to pass a nest adjon with regx more than hard is impossible so what is the next natural step after this so the next natural step is what we can call consuming parser sometimes also called parser combinators and the idea is that you want to have a a mechanism that allows you to consume a little bit of information and match that the information is actually what you expect and probably try to convert that little bit of information into something useful and again let's see an example because it makes it a little bit easier to understand the concept and it's always the same example and we'll start with some pseudo code so let's imagine that we have a function that just says okay you have an input I just want you to read meaning consume the first part of the string and we expect that part to be a labeled so we don't really know the code of this function at this point but just imagine that somehow we implemented this piece of functionality that knows how to consume the initial part of the string until it looks like a label and the next step so once we do that we are effectively splitting the string in Al if you want the first part is the past data so the data that we already consumed and the rest is the remainder string so we still need to process the rest of the data now we have another function let's just call it read separator that just looks for it only consumes the remainder data so we kind of discarded already the first part and it looks for a separator in this case a column and effectively again we have other two parts as a result the first part is what we just matched so what we consumed and match so the the column itself and the yellow part is the remainder and we can keep going on the next one is going to look for a space same idea we match the space we have a remainder the next one maybe it's a little bit more sophisticated we can say read a tle of integers so you can already see that you can make these smaller parses as simple or as complex as you want and it's on you to effectively Define how to make them easily composable and usable but let's just say that we have this information this function that allows us to process an information that looks like a table of integers what it's going to do it's just going to take all the information so it's going to look for parentheses for numbers for separators and it's just going to give us three numbers already looking as an integer and at this point the remainder is the empty string now this is the idea of consuming parsers and one of the obvious advantages is that once you start using this approach handling errors become so much easier because as soon as you fail for instance imagine that now the input is Hell word and you try to read a number from h word is not going to match immediately and it can stop right there and it's just going to tell you well I couldn't match a number on the remainder string which is a l word so when you see an error like this it's just much easier to relate what was the input what was the specific part of the matching that you were trying to do and why did it fail which is not something that you get with reject you might get it with string splitting but it's on you to write all the error code and the error reporting now one library that allows you to implement this idea is n we will talk at the end of the talk about some Alternatives but it's kind of a default as far as I know in the rust ecosystem has been at least for the last couple of years and again is not something that you get by default in Rust so we need to install it using cargo and winom is going to look more or less like I described before so here we have a little bit of realistic rust code again the input is always the same and we can assume that we have created a function called par Point what we are going to do we're just going to say part Point using the current input of course this operation can fail if it doesn't match it's just going to give us a result that looks like an error here we know it's going to be fine so we can easily unwrap and then it's going to turn as a tle with two items the first item that we are just ignoring with the underscore is the actual uh um remainder string so we don't really care because we know that pass point is going to pass the entire string and the rest the second element in the table is the actual point as a structure information now this is of course a little bit vague because to really understand this we need to look into the par Point function so the par Point function function could look like this and this is actually one of the distinctive features of n that all the parsers that you write have more or less this kind of signature so the first thing that you have is one argument and this argument is generally called the input so what is it that you want to pass and of course in this case we are passing a string but n is not necessarily limited to Strings you can also pass bites you could even pass your own sequence of tokens if you wanted to implement something a little bit more High Lev for instance generally speaking strings is the most common that I've seen but again Nome is very flexible on that the other thing is that the result um here is an alas that Noms provide you which is effectively abstracting the error type there is a specific G error type and then there are two generics and this generics are again the input type so what is going to be the remainder after you pass and and then what is going to be the data type that you are trying to parse with this specific parser function in our case we assume we defined this point 3D struct so this is going to be the output that we expect if we are able to pass the string correctly so somewhere down there we are going to have all the passing logic and then at the end right now line four we are going to have to return again this kind of weird structure it might look a little bit weird but in N you are always thinking in terms of t where the first element is always the input or the reminder if you want and the second element is the value that you actually passed right now okay so what is the first step the first step is that we want to for instance match on point and I have again the example at the bottom right of the slide so this is the string we are trying to pass just as a as an example so the first thing that we want to make sure is that the string starts with Point column and space and N gives us a a parer combinator this is the way that they call the built-in functions that you can easily compose that is called tag and what tag does is effectively an exact match so it's just going to succeed if the remainder string starts with that particular string that you are putting in the tag the other thing that is interesting to note here is that there is a very functional approach so this looks like cing for instance if you come from ascal or other functional languages so it's the idea that tag itself is a function which can produce another function so it's a function where you can pass an argument in this case the exact string that you want to match and it returns another function which is the actual parser that can match exact that string and to use that parer we have to call the function with an input so we could have splitt this operation into two separate lines so we could have said TG point and name that I don't know prefix part or something like that but it's generally more common when you see n examples to see this kind of use case where people are just saying I'm going to create the tag parser in line and then I'm going to call immediately uh the specific input that I want to pass and this is just because generally these kind of passes are one shot you are not probably going to reuse the tag Point somewhere else in the same function block so this is effectively what we want to match and this is the value we want to match against finally of course every parer can fail because what if the string doesn't start with point maybe it starts with a number maybe starts with something else this is not going to match and the passess is going to stop right there with an error so here we can use the question mark Operator just to effectively escalate that error and return an error type now in this particular case remember that we always get the remainders which now we are calling input again again so we are kind of shadowing the original input as we consume all the string we got as an input we keep calling the remainder as input just for Simplicity the other element that we get is what we just matched and we will be matching a string or a string slice in this case that it only contains Point column space and we don't really care about that value we just want to make sure it matches exactly that but it's not information that we need to retain so that's why here is kind of safe to just disregard it with an underscore okay then the next step we can do the same thing to just match on a parenthesis then the next step is a little bit more interesting because this time we know we pass Point column space open parenthesis there will be a number there or at least we expect to be a number there and we expect the number to be a sequence of digits if you look at it as text but we should be able to convert that sequence of digits into a valid it type rust type so this is another rust sorry n combinator that you have it built in as part of the library and you can use it every time you need to pass a number it's just going to make your life easier because it gives you already a number type and if it fails to pass the sequence of Digit or if it fails to convert that sequence of Digit into a valid either to2 you're going to get an error and that error is going to be escalated so now we have everything else that we need to complete this function so we need to match a comma in a space another number a comma in a space another number and a closing parenthesis and at this point we were able to extract x y and Zed as either to choose and it's very easy to build our instance of that particular struct now with n you generally see people breaking down the problem into smaller and more usable parsers we had a little bit of duplication in our code so there is ways to factor that in a way that there is less duplication basically and we can make things a little bit more reusable as well for instance one thing is that we might have a separator maybe we are building a larger parcel that parses not just points but maybe also triangles rectangles other shapes other information metadata so maybe you might have this concept of separator in different places in in the paring uh logic that you're writing so it might be use useful to just write a small parsers that can only match separators and actually we can make the idea of a separator a little bit more flexible before we were just saying it's always a comma separated by one space maybe we can extend the idea and say well there needs to be a comma but I can have zero spaces or n spaces it doesn't matter we can make our passsing a little bit more lenient that way so here we are introducing a new uh combinator to do that that again provided by Nome and this is called pay and what it does it basically allows us to combine two different parer and it's going to try to apply them in sequence it's going to apply the first one and then the second one and it's going to if everything succeed if everything matches it's going to give us back a table that contains the result of the first one followed by the result of the second one now another example that we might have is Parts coordinates maybe this idea of having a sequence of three integers it's something that I we might want to make it reusable so we can also build a parsel for that specific part of the information so we are only trying to match for instance 22 comma 17 comma minus 11 and we can use another combinator this time is called tble and it's kind of a generalized version of the pay one where pay only allows you to concatenate two parsers here you can concatenate as many parsers as you want you just provide all the parsers in a t so in our case our sequence of parsers is going to be I to2 separator I to2 separator and IU again and notice that we are reusing that separator that we just defined ourselves in the previous slide so this is the idea that as you start creating your own little parsers you can easily combine them together and finally once we have those two parsers we can create the version two of our pars point and the first part is always the same we just want to match on on an explicit tag that is point column and space honestly I was a little bit lazy here we could have said spaces but you get the idea and then here what we are doing we are combining other passers using another combinator here called The Limited which what it does is basically allowing you to say I want to pass some information but I know that that information needs to have the limiters for instance an open parenthesis at the beginning and a closing parenthesis at the end I don't really care about the result of those two delimiters they just need to be there so just give me back what is being passed in the middle so we are just using pass coordinates to really extrapolate information from from our parser and at that point this is maybe a a little bit more advanced version of the pass point but we are making it a little bit uh more usable and we have less duplication of code now at this point we learned just as a summary some useful Nome combinators and I think n takes a little bit of practice just to get used to the different combinators that you have available so it's probably a good idea to do a recap the first one we saw is called tag and allows you to do an exact string match then we have iter to2 which allows you to to check if whatever is in the remainer of the string looks like a number and as long as it looks like a number it's going going to be consumed and when everything is going to be consumed is going to try to pass that sequence of digits into an actual itue and if succeeds you actually get out a value that is a number and it's not just a string slice anymore then we have space zero which can match zero or more spaces in a sequence then we have pair so you can take two parsers and apply them in sequence and you get back a tle with two values which are effectively the result of the two pares individually then we have tle it's kind of a general version of the previous one where you can effectively concatenate and different parsers in sequence and then we have delimited where you basically say parer number one is the initial delimiter par number two is what we really want to process as useful information parel number three is the closing the limiter there is a very good n combinators cheat sheet on the repository when you get get the slide you'll be able to click there and the idea is that you have a list of all the most common uh combinators and that list also tells you they are grouped in kind of logical groups and it tells you more or less when do you need to use that particular combinator so let's uh look at the RAS protocol itself ER res protocol is the radi serialization protocol and um what is it exactly uh is a binary protocol that is defined as a binary protocol even if is very human readable that is used to control uh is control code Ashi standard chars uh to Define sequence of informations uh this sequence of information always end with a CRF a carage return and a line feed uh and is also used in to separate the different parts of our data types the first bite of every payload is identifying what will follow so um there are different kind of types uh in these R protocol there are simple bul aggregate so simple strings bu strings arrays MKS sets all kind of stuff so this first Char is telling us which one is the following type and here we have a table that show us what are those types we have the plus that is a simple string the minus is a simple error the integer is a column the dollar is a bull string array noes bans double big numbers big errors verba strings Maps set and pushes we will explain some of them later on so let's look at an example message let's start with something simple we start with plus okay that is a simple string with okay written in it and you can see that it starts with the plus then there is the string and it terminates SLR sln so then we have another simple example that is the simple error that starts with a dash then there is something written in it because of the protocol normally there is a some kind of code at the beginning and then a fre text string this is not implemented in the protocol itself this is just a convention and then again it ends with SLR sln and this is a simple error then column from the table before if you remember is the integer and we have the integer but we have the integer as a sequence of ASI Char so we have 1,000 here we have column 1,000 but this will be four by one z0 Z in ASI so it will not be a H binary encoded 1,000 bit by bit and this is an integer again we can go with the more complicated data structure for example if we want to have a multi-line string or we want to have a bigger string we have a bu string a bu string starts with a dollar then there is a number a separator sln and the string itself that five is the length of the string that will follow so this tell how many bites we need to pass going on five in this example to get the v string in this way is possible to add Carriage rard inside the string that otherwise will break our protocol but knowing the length of what we have to read there is no need of doing anything special there then we can go with aggregate types more complicated one this one is an array uh you can see there is a star CH that CH is the number of element of the array SL the Shen then dollar five so the first element of this array is a bu string of length five that contains a LW then there is another element of length five containing w and this is an array with two elements and then going on we can have doubles again represented as with AI chars and not with binary Charters we can have big integers or big numbers we can have Maps where we have for example here we have a map with two element then we have the key of that is first that is a simple string the first value is a integer with value one then we have a second key that is called it second again as a simple string and it has an integer value of two but we wrote a parel for this with not what we did we started defining the data structure itself so we defined an enumeration of all these possible data types all this possible data types ER contains sometimes a string a slice of a string to be honest an integer 64 for the integers the null contains nothing the N is just a null the bull itself contains if it's true or false double we kept the string because we don't want to use decimal so enforce a external library and we don't use don't want to use float to avoid to lose Precision same for big numbers we keep an Str Str because uh we don't want to force a library for big numbers handling because there's not a library in the core of rust for this you have to install a crate H but is a important to notice that for example for array we have a vector of value so we have a vector our of ourself and there are a bunch of types that have this there is the array there is the map there is the set there are the pushes pushes are basically arrays that come from the server to the client so these types are this type is basically a recursive type because it can contain references to other instances of itself this made some stuff a little bit more complicated for example if you see the set is not a Nash set but is a B3 set because to be a n set you should be hashable obviously but because you can have mutable values in value those are not hashable how do you Ash an ash map how do you Ash an array that can change so for this reason we use the even for the map we don't have asmap but we have a vector of keys and Vector of values and so on so we tried to stay in types that were H not enforcing any kind of implementation to the user of this Library we first wrote uh a first parer for passsing a value in uh of this and here we use a combinator called oneoff oneof say take from the input one of these chars and we provided all the list of the possible Char that the resp protocol is supporting then we said okay let's make a match based on this type on this Char and based on the Char that tell us what kind of type is following we select the right parsel for our input so we selected the right parer for that type of value we use it and we return it directly because it has has the same signator right result as the uh value because we always return a value from everything so and this is the first idea that we had to to pass the resp protocol but then we went for an alternative approach the alternative approach is to use the alt combinator from uh n what is the alt combinator doing the combinator is taking a couple of and is trying all of them until it find the first that is not failing so because of the resp protocol this is quite fast because if a if a parer fails it fails at the first Char because it will find plus okay is is not starting with plus I fail and then alt will take the second one starts with the minus yes no and go it goes on like this and our function our value now is simpler because it's just alt of this list of parsers of the input done no intermediate variables of any kind uh yes we are here but to do so yeah sorry to do so we had to Define some um the limiter and some functions to well pass like we did before so we created the CR LF uh combinator that is basically just a tag that match slen this will fail if the input doesn't start with that and then here there is an example of one of this parsel the parel the par simple string so this is our crlf and then we have the par simple string par simple string again is taking a slice Str Str as an input is giving back a result first of all is checking that the input starts with a plus if we are facing not a simple string this will immediately fail and ALT will go to the next parser let's say that for example our simple string example down there at the bottom of the slide we match so we go on now we use another n combinator called terminated let's say and also take while let's say take from this string while the Char is not AAR anden so while the string is something else take that until you are terminated by a crlf so it will consume the string in this value that n value will be as in this case will be a slice of Str Str yes of this input there is the question mark So if there is a something wrong or there's no Terminator this will error out and when we have our slice in our value variable we return the remainer of the input and a simple string enumeration type with value inside it and this is how we passed a simple string another example can be to Parts a bu string like we said before there is a you can see the examp below there is a dollar a number sln and then there is 19 chars then another back SLR back sln how we did this again the signature is always the same the input say that the tag say that the first Char of the input should be a dollar we then pass a u32 terminated by crlf u32 is not the r of u32 is the n u32 combinator so we pass a u32 number length will be a number at that point we use again determinated by crlf take take takes a number of chars that you provide and we provide length casting that as U size because we pass the u32 but take once a your size so now this is take 19 chars terminated by crlf and if one of these exraction fails this parer is not the right one to use if we arrive to this line it means that this was the right par to use and we return our B string enumeration with the value that we just extracted again we have the array let's go a little bit faster on this because same signat we match a star we want to know how many elements we have in the array here we use a another n combinator that is called count and count applies another parer it's not like take that takes a number of character count takes a number of parsers value so here there is basically a recursive call to our pass value function and say I want to call this parer length times an array of two elements here so we will call Par value two times and we will get two values out of it values will be already a vector so a vector of values so we'll be already the right type to use inside our return return value for the numeration array then obviously we wrote parel for all of these types but we are not going to show you all of them yeah otherwise we will kill you by boredom anyway yeah I think it's good now to do a good recap of the other combinators that we learned so far so one off you can match one Char or one car I guess I should say from a list of allowed ones terminated basically what it does allows you to apply one parser and then it's going to look that the remainder string starts with the content of a second parser so we use this to make sure that once we pass something successfully that is actually follow followed by a Terminator in our case SL sln then we have eof so end of file we can say is is an interesting one because it basically doesn't really match anything it actually checks that the remainder string has length zero so you can really use this end of file only with input types that Implement length so that that are implemented tra size then we have alt the one we used at the very beginning of this protocol where we try to effectively say try to match a sequence of parsers and stop as soon as the first one matches the string then we have take while so it's going to allow you to keep consuming cars and basically um stop only when a specific condition applies and then we have take which basically says take a certain number of characters finally we have count which is useful when you know you need to repeat a part a certain number of times so you can just say count this parer and times and it's going to give you back a vector of all the Matched values one for every repetition of that parser now if you want to look a little bit more how we implemented all of this again we're not going to show you all the examples but the library is already published it's called tiny rasp we haven't used it for anything useful yet we plan to do more coding challenges maybe where where we try to build our own redies and then at that point maybe this library is going to be useful but right now now is just there for people to use it so if you find it useful let us know if you want to contribute and make it better of course all of that is welcome and we recorded in a series of live stream our exercises in trying to make all of this happen so if you want to see the you of us screaming at the RAS compiler for like three hours and a half which is probably not a great thing to do but if you crazy that way you can check out the recording on YouTube there is a playlist with all of that and that's the link now I have one last bit before we go into the conclusion but I leave Roberto to describe this part here I will fly over this because we don't have a lot of time but basically can we use n to pass binary formats the answer is yes we took a very simple binary file format the STL file format this is from the Wiki page this is what the Wiki page describ the format so there is 80 bytes that are an another there is a u32 of four bytes 32 of number of triangles that will be described then there is this number of blocks of 50 bytes and every block of 50 byte is made of a normal Vector made of free float three vertices of the triangle in a three dimensional space and a u in attribute by count that is something that I always found zero in the STL I check it but some editor use it to store color of the triangle and other properties that are not uh yeah not standardized so it's just an integer but ah yeah and there is a a monument in Dublin with this teot in Smithfield this Midfield Square because this is called utas Midfield uh pot how is called this one channel yeah I never understood by the way why this Monument was in Dublin it's literally the blender example that you that the the first 3D example that you see anywhere and then I realized while I was researching it that the square where it is is called Midfield and this teot is called the Utah teot and apparently one of the biggest cities in Utah is called Midfield so somebody made this weird connection and they decided to put this monument in Dublin so if you are in Dublin it might just be a fun thing to know and see yeah but we did to pass this binary format again we imported a bunch of stuff from n some uh consumer that we already saw some combinator that we already saw before but we imported the binary ones l e underly F32 Le underline u16 and u32 Le stands for little andian because when you pass a binary number you need to know if the most significative and less sign significative fight is first or last last in the file and STL is working in lalan Lan is basically the encoding that also the Intel CPU are using Motorola was using big big andian arm can use both but is mainly little andian so we import the library we Define our structure that we will answer from these parser so we have an STL file that will be a slice an that is a slice of u8 there will be a number of triangle that will be a u32 there will be a vector of triangles every triangle is made of three ve free vertex one vector for the normal of the triangle that that tells where the triangle is looking at basically and our attribute the 16 bit from before here is the par that is parsing the binary block of 50 byte that is the triangle itself so it's taking as input not an Str strr but is's taking a slice of u8 an array of u8 is returning the same and a triangle structure line 27 oops sorry line 27 we basically create a free float parser that is a simpler version is a just a uh way of not writing tle a222 four times but we created this uh utility basically and we use it to pass the normal the vertex and the attribute in the end we return through the okay same as before we return our leftovers and one triangle structure extracted from our tle that we uh extracted with the Dom after this we have the parel of the main file that again takes this slice u8 and R this slice of u8 and an STL file first line is pars take 80 bytes because we don't care about what they are they are always 80 bytes then we pars with Lan u32 how many triangles we will have so we have at most four four billions andever triangles then with count like we did before we get that number of triangles that will already be a vector of triangles and we just return it the next step is to use this barer in the main function of our software that is reading ex sample STL from the dis getting the slice of B using the parser line 65 we discard the remainder there should be no remainder but STL some editors leave metadata at the end of the file for extra information informations so we are not strict about this we just discard whatever is after that and we print what it was inside the file inside the file we will have a another with our at bytes we will have a u32 with our number of triangles this example was 2,466 triangles and then we will have that number of triangles inside an array inside a vector with all their door numbers and attribute by count like I said I always found zero in these examples but I think we a little bit over time so I leave it to you Chan I'm going to quickly try to wrap things up so if you need to do parsing the first option is probably string splitting it's a good option but it get messy really quick so probably good for simple stuff but don't overuse it Rex is another valid approach but there are pros and cons they can be hard to learn the bug and to get right but sometimes they're just good enough to do to get the job if you have confidence in r exess n and the idea of parsers combinators in general it's probably a more interesting approach it's a little bit more involved because again you need to learn that the way of thinking of parer combinat you in the case of Nome you need to learn a little bit of the library and the helpers pars that you have built in but once you are confident with them it's definitely a good option to use and you can build fairly sophisticated parsers that are still pretty easy to read and also when you have an error that generally that the reporting of those errors is quite clear so if you have an issue when you are in production or when you're testing it is easier than things like Rex to figure out exactly what's going wrong and what do you need to fix and the other cool thing is that we Nom with the same ideas and concept you can also pass binary data which is not always obvious when for instance you're using something like string splitting or R access now I have a little bit of a bonus content you might be wondering why it is called n and this is actually in the rmy of the repository this is the logo the official logo of N and it's basically saying n will Happ take all the bites and basically eat them right now one thing that came to our attention in the last week basically when we already prepared most of this talk is that there are actually a couple of interesting alternative parser combinator crates in the rust EOS system that are emerging and getting a lot of traction the first one is called wino which is actually started as a fork of gnome and it seems to be a lot more up to dat than n is and is also trying to focus a lot more on the developer experience so they are trying to kind of iron out some of the edge cases that are not super clear with Nome and G try to give developers a better experience in terms of the API itself some issue with error reporting try to make that even better so wi know seems to be kind of a natural Contender to to gnome but it's pretty much the same approach because it started as a fork of gome then there is chamski which I think has a slightly different API so it favs more the idea of function chaining rather than a more functional style where you try to wrap functions one inside the other so function composition so you might like that or not depending on what which kind of programming style you prefer I also had the luck to chat with Ed pay with the author of wo on Mastodon and his take is that N is a little bit unmaintained at this stage because the last release was years ago at this point and the new release was promised one year and a half ago and it's not there yet there is kind of a beta version that you can find with another name I think it's called N8 like there is a crate called N8 which is not the official one but it's kind of a placeholder for people that want to experiment with the current working progress but apparently somewhere there was a quote of the author of gome saying that it's not necessarily going to happen anytime soon and it's not blocking anyone so apparently wi know now has a little bit more weight in terms of if you're looking for a crate that is going to be well maintained probably it's uh it's a more obvious choice if you're starting a new a new project and the repository also list a bunch of famous projects that recently migrated from n to Wi just because there is a little bit of uncertainty around the N project so everything we said I think is still very relevant just because we know it's going to be very very similar to n but just be aware that if you are starting a new project today maybe it's better to look at the two and decide exactly which one you want to use going forward now here we have a bunch of links with things we mentioned today the Nom documentation the combinator cheat sheet there is another cheat sheet which is very good if you want to understand how the error management works and if you need to create custom errors how you can do that there are very good official examples in the repository so definitely check those out our own library and the playlist with our live coding and if you want to follow some of our Mya Roberto live streams we do that on Twitch and YouTube and there are links there and that's all we have for today so I think I don't know how much time we have left but I think we we can be open for questions if you would like thanks Luana thanks Roberto another a great talk really well presented I love your slides they're excellent um we don't have many questions I think it's because you're so good at explaining things we had one from Tim early on I'm not sure to if it's still relevant it was sorry I hav noted here you often expect a second generic to be an error type is there one general error type under lying the eye result do you have a context for that good question I'm trying to find this question so I can highlight it was at the very beginning is after Ras Dublin Say the video is real oh yeah so it was it the the beginning is H below that one there you go okay yeah so basically my understanding and I I think you're better off checking out the actual signature of that ey result but my understanding is just a type alas that effectively hides the error type and defaults that to a n specific error type so you you are kind of forced to use the N error type type and actually my feeling using n for a little bit is that they try to abstract as much as possible the error reporting so you are better off just using the combinators the basic combinators and they will do the correct error reporting for you in case of an issue another thing that we wanted to show but then probably there was too much stuff in our slide is another combinator called verify that you can use when for instance you matched a string and then maybe you have a little bit of complex logic you are not able to abstract with the default combinators or with another custom parser then with verify you can just run arbitrary code and if that arbitrary code returns an error type then everything errors out with an error of type verify failed something like that so there are kind of Escape hatches if you want to do a little bit of custom error typing otherwise you might need to redefine this ey result type and be explicit about the result type but I think at that point you you might have to map all the errors when you use the standard combinator so it might get a little bit messy but again feel free to experiment I think there might be room there for for custom stuff I don't know if Roberto you want to add anything there no no you explaining it very well you have another question there Li if you want to take a look at it and highlight it second from the [Music] bottom so is there a easy way to pass Json with Dynamic key value types uh I think this is actually one of the examples that you have in the repository so if I understand the question is could you write a parer for Json using something like n and the answer is definitely yes and I believe that one of the examples you find in the N repository is exactly to Parts Json and I think there is even the Json 5 extension the one that supports comments so it's even a little bit more involved than basic Json we can have the special guest Roberto Scott yeah wants some attention yeah I'm not really seeing any more questions pop up here the Tiana yeah in any case our contacts are there online so if you happen to to review the slides at a later time and you come up with more questions feel free to reach out and ask I'm not going to promise we have all the answers but we we'll try our best yeah feel free to open request feel free to tell us that we don't write good rust is true we are still beginners so any suggestion is well accepted I saw a compliment flying through the text messages there halfway through it yeah here there is a comment saying that M from mat saying he will probably use S for Json I I probably agree with that I think the example is just more of an exercise if you want to understand how to parse recursive parsing effect recursive data structure like Json or like the one we presented with the Rus protocol so yeah definitely use serd for everything that is Json or Json like that is supported in said lots of emojis my cat is telling us that no more questions are there other questions it was a enough okay I think uh We've answered all the questions that the audience has um thanks a million again Roberto and Tiano way presentation as usual um and this video will be available uh on the on the channel in a couple of weeks or probably sooner than that but thanks everybody for coming along today I think we can kick off a a countdown to exit L okay thanks everyone for being here and thanks R for hosting us have a good one bye I