The Rust programming language ecosystem features a rich collection of community-developed crates (libraries) that address specialized programming challenges, ranging from code generation tools like 'pretty-please' and 'derive_where' to advanced concurrency verification with 'loom', demonstrating how Rust developers leverage these specialized libraries to solve complex problems efficiently while maintaining high standards of correctness and performance.
Top Rust Crates for Error Handling, Parsing, and More
Added:Proficiency in Rust's core syntax, including ownership, borrowing, and lifetime management.
Familiarity with Cargo, Rust's build system and package manager, to understand how external crates are integrated.
Understanding of standard library error-handling patterns, specifically the use of `Result`, `Option`, and the `?` operator.
Basic knowledge of Rust traits and generics, particularly standard library traits like `std::fmt::Display` and `std::error::Error`.
Developing custom procedural macros and code generation tools to automate boilerplate reduction.
Designing robust Domain-Specific Languages (DSLs) using advanced parser combinators (e.g., `nom` or `pest`).
Implementing structured logging, distributed tracing, and advanced observability in production systems using crates like `tracing`.
Exploring high-performance asynchronous runtime architectures (such as `tokio`) and safe concurrent programming patterns.
1.7K views36likes1:33:13Original Release: 2025-01-14
now you can say something funny yes now we can okay thank God yes uh you know actually was what a great episode last week by the way that was that was uh that was tons of fun with the the predictions episode it it was great and uh a lot of people seem to enjoy it so that was great I'll tell you the one person who did not enjoy it lives with me because it went two hours yeah the prediction episode's always gonna be long though I'll just tell her that that's fine yeah exactly you know it's like listen like once a year they're just going to go they're going to go a little long I'll try that yeah give that a try um but no it was a great one and and and uh great job getting Simon and and Mike on good those guys are delightful um so I did see a headline on Thursday that I'm like oh my God this prediction is wrong already Intel's CEO search heats up as leadership shakeup drives turn around hopes and so I'm literally like this story is going to be like a rumor about a candidate that they're going to be announcing in like like this is going to be done by the weekend uh but then I go to the so I'm thinking like oh great like well fine I I knew the risks uh the story is that City analyst Christopher dley says Intel might name a permanent CEO in the next few months I mean that's talk about like a prediction oh no I'm like wow wow that's how how do you get to be an analyst that sounds hard yeah uh conversely they might not like that's the alternative yeah sorry I mean take nothing away from the superlative work of City analyst Christopher dley who I'm sure does very thoughtful research they but it is not a a great Insight that they might name a permanent CEO the next couple of months yeah but Adam I decided that I'm going to take uh zero I will take no credit if they name this in January but if it's February I will take one 12th credit and I you know if it's March 2 12th credit one sixth credit and so on that that' be very little nice little way of you know because I know we keep score so closely yes some of us do and then the other thing that that I just wanted to mention to you because so uh the my uh I got a high school senior um he's got a terrific English teacher a shout out to miss Foster and she has given him a very good assignment she given a Class A very good assignment which is uh you need to seek out an adult and you need to ask that adult for three books that had an impact on them as an adult they can't be kids books but books written for adults um and then you student will read one of these books and I I the teacher I miss Foster will decide one of these you read and uh Alexander picked me my my 17-year-old my which is great I I think I think my wife is a little hurt but you know we've got other kids you know we can you can focus your reference on them I bet he asked their both I bet he both he told both of you that he was only asking you because he's a pro um I think so he he is I agree with you he is a pro to the point where I think he might have concocted that story if he had such a story he would concoct it only for my wife's benefit I think he and I both know that I was his I I actually was his first call I really do appreciate it um and so uh the the three books that I named I'm like this is a great question what are the three books that are that have had an impact on me professionally have changed changed me um or changed me in in in adulthood uh you have read all three and and I would with the advice of don't overthink it um I was wondering how many of them you could rattle off of the three I think you can three for three how many of the three books that you and I have both read have most influenced you as an adult that's right I feel that you you I mean you can I think you can go at least two for three again don't overthink it I like my I'm like Reeling I feel like I've this is a quiz I did not prepare for listen it's a pop quiz okay everybody take take out your paper no I feel like the okay can I have a hint can I phone a friend on this one give so are we talking about like in the technology domain is that what you're saying yes okay in actually our shared professional lives okay I'm gonna say Steve Jobs in the next big thing that is one and I think you you I'm going to give you the hint that I think you've gotten the hardest one the other two are easier than that holy [ __ ] I mean I like I don't think this is right but I I feel like the quantum dot uh was deep pull and no that's not right that is a very deep pull okay so what was my the title of my block entry when he launched the company oxide oh oh oh oh right sold the new machine right of course so the new machine yeah and then what did we name our group after it at son there we go Ben Ben Rich's uh Skunk Works all right you're right it was easy you're right I I got the hardest one first the quantum dot is definitely an example of overthinking it that is uh that's a very deep pull by the way that's that that's an obscure book I I don't know why that like that's the only thing my literally the only book I could think of in that moment there like I name another book couldn't I couldn't name your brain just went white and all I could think of was this relatively obscure book published in 1998 or whatever it was that's all that was left yeah well uh yeah you know I tried to prevent you from overthinking it but I you know sometimes sometimes the overthinking is gonna happen on its own um anyway I thought that was a great assignment so I thought it was and I will be uh he'll be reading Skunk Works by Ben Rich so oh ni do that with him yeah it's G be fun um so rain Eliza welcome sorry welcome to uh us talking the three books that have influenced us professionally um very exciting about this um and a shout out to Chris kco because I you know uh he had had a tweet of like oh you know crazy you should know it's great to see a resurrection of this because I did this on the new R station podcast and I that must have been lodged somewhere in my subconscious um but I did you ever listened to the new station Adam I have I but not that episode like that doesn't I didn't realize that we were uh stealing from another great artist yeah that's right exactly um but very uh very excited for this and because I also feel like I mean and this is true of not just rain and Eliza but I feel like rain and Eliza are two that definitely are are constantly pulling out crates that I have never heard of um that are extremely useful and I'm wondering like why haven't I heard of them should we start with how do we want to do this at because do we online you impli that we should have a dton cap um I well I just I just me I look at list and and David is here in the audience I assume just to to bask in my my fanboying for him but um you know I look at my list there's there's so many crates that that David to has made that I appreciate and and I'm gon to kick it off just with one of those because um one of the ones that I stumbled on you know I write a bunch of macros here and there um I am frustrated with rust format Brian and not in the way that you're frustrated with rust format but like I want to use it in kind of a library context and that's challenging to do what do mean go on that's interesting well I so I stumbled I was like surely David writer of all macros has stumbl has like done something for this and he has so there's a crate called pretty please and it is for doing rust formatting like formatting of code and what I really love about it is it um kind of uh turly opinionated that is to say look I'm not trying to be rust format I'm just trying to make things better like pretty like I'm pretty printing the thing I'm not formatting the thing and if you don't like it well you know get out of here and there are a bunch of like um I think there a bunch of been some PRS and issues of the form like could you do it a little bit differently and I really appreciate that Dave is kind of like no like it you take it or leave it like if you don't like the way it's formatted then you know maybe format it differently that's fine um but it has been a godsend for a lot of the the testing that I've done for these code generation crates for okay that's interesting um because so youve got a lot of crates that that generate a huge amount of of code yeah like so for for example progenitor you have like three lines of macro that it poop out like 60,000 lines of code and uh and you want the the code that it emits you want to be readable well in particular when um I'm dumping that like into a file for uh test automation or whatever yeah I want it to be like at least vaguely readable and um I've used rust format in the past but like there's a bunch of challenges associated with using rust format in a programmatic context like that and pretty pleased it's been fantastic like exactly what I needed you know maybe I should be using this okay this is this is already paying dividends actually because I I I've got in my crates that generate code I have just I've kind of manually made the code that I generate rust format clean oh yeah um pretty pleas is gonna help you yeah pretty pleas is I think exactly what you want in that like you mean you your code generation is like emitting new lines and stuff like that yeah yes that's right it's doing all that yeah tot and you know that that's interesting because it has made the co I mean of course I I I I kind of believe that code that generates other code there's like a balance that must be achieved in the universe and if the the code that you're going to emit is going to look the code that emits that code has to be filthy um but maybe that maybe that's too my code that emits rust format clean code is filthy and I would this I think would allow me to clean it up quite a bit totally this is what I've fallen in love with with regard to like rust macros which is you can use uh another David crate the the quote the quasy quote quasi quoting system so you you quote quote code that looks like rust code and then pretty please well just clean it up so you don't have to like just live in this cave person era of like strings on strings and doing your own semi formatting here and there and the beautiful thing too is like your code generation in macro context can be exactly the same code that if you want to generate code and dump it into files and I think it just allows for really like debuggable testable understandable code as opposed to as you're saying Brian like kind of this swirly code generation that is also interspersed with formatting Brian you might be pleased to know that I got out in front of this a few months ago and ripped out a bunch of string based code generation from Idol which may have been Cliff's doing rather than yours but uh now that uses quote and pretty pleas oh that's you know Ela yeah that is Cliff doing not mine but I would I can go look at that as a model because I what I'm thinking of is in particular is the PM bus crate which is just MH uh there's there's some Grime that could be cleaned up in there um for sure yeah this looks great God you know there's always there's always a you know they always as I think we've said before here you know they always say that there's a chat that includes everyone except for you I always feel like there's always a detail night crate you haven't heard of and I that's what I'm saying I mean that that was the Tweet right like I feel like uh there David has done so much stuff in this kind of domain too that surely David has found this problem in fact I I'm GNA cast this open to David and to rain who's bumped into this you know one of the things that I struggle with rain is a problem I saw you working on in dropshot which is one of the things that sin another day toate crate doeses very nicely is like turning uh you know errors in the rust macro context into uh code generated errors to help debug and stuff and one of the things I saw you do in drop shot was like collect a pile of Errors to then emit all at once and I'm sort of surprised that there wasn't something you reach for to say um you know as you encounter problems and errors along the way accumulate this list so that you're not just failing on the first problem but actually emitting a bunch of Errors for the user to then handle all at once yeah is there anything like that um so I spent it's funny because after you know after we talked about it last time I ended up spending a little while looking at it and um there are a bunch of great libraries and actually some of them that I wanted to talk about here um there wasn't quite anything that I notice kind of hit that exact spot um and partly because I think one of the things that kind of becomes challenging is that you're um if you want to do good error handling and this kind of goes into that uh the cosmic balance thing that you're talking about Brian if you want to do good error handling uh often like you can no longer use like like good type system um things so so as an example uh you know one of the ways you might model something in Rust is with like a result of you know the okay value or the error value right um but if you want to like collect errors then often something you will do is you will pass in like an erson mute error collector or something like that and the value that you returned like is an option and now you have to know that if you know there's kind of this this implicit invariant here that if there is an option then that means that you had at least one error go in and so on um a crate that I did actually want to call out though and and something that um doesn't quite solve this uh specific problem even though I wish it wouldn't um is uh miet so miet is uh a really really cool crate it is um kind of uh so if you're familiar with rust and and of course dto is great verse uh you'll have come across uh this error and anyhow right uh m is kind of a combination of uh this error and any and it kind of meets both of those things uh but uh another crate that it actually meets uh is uh code span um so if you're familiar like um one of the things that's really interesting about rust is that it uh rust C has like great error messages and I think I think you know I think that's one of the reasons that all of us feel uh feel pretty good about rust right is that fair to say um and with uh the error messages I think I think you know one of the things that's really nice is like you know there's this lovely like syntax like highlighting where it'll like show you the exact like you know things that were wrong and it will give you a suggestion of what to do instead and like all of those things amazing yeah it's so good it's uh and so there's actually a few crates that uh do that so so the Russ C's own error thing is uh is extracted out into a crate um that I don't remember the name of off the top of my head um uh then there's another crate called gold span but uh M also captures all of that and meet actually one of the things it can do is it can store a list of Errors um so what you can do is um and I have used this pattern some places is that you actually store a list of errors and then you uh and then you have me at report that with um so you can provide the source code uh that those errors associated with uh and the bite offset and so you kind of provide that source code and then metat will kind of render that in a nice way um so it's uh I think that kind of style of like high quality error reporting is actually something that is really really cool about rust um and I don't know if there's any other ecosystem that has paid this much attention to like how your error messages look right rather than just reporting like your uh you know your line number or whatever yeah this looks really cool this looks so good yeah I've not seen this have you seen this before i' never seen this no never h actually you know what R may have pointed me to this a while ago and but R and just to be clear this is not in like macro context this is and when you say it's kind of draws inspiration from Russ C yes it's not like it's for you know if you're processing some other kind of document or whatever and you want to draw on that that kind of concept yeah okay cool yes uh yes it is right um so so as an example actually one of the examples is that you can integrate it with sird Json so you can actually get like great highlighting for like which bit of a sird Json uh thing failed and and I think that's really cool oh that is really good yeah that is because you know I this I really like Ron a lot Ron not the humans named all the humans named Ron although I guess I like all of you to but the Ron the the rust object notation I like a lot but man the error handling is the error messages are really not very good um and it's which is frustrating and boy this would be an opportunity to really improve them Brian it is with a heavy heart that I must inform you that if you have anytime in the last 6 months or so made a typo in an idol Ron file and gotten an error that has like some RW Source in it uh that's thanks to me at okay so so you've integrated M into the Ron parsing in Idol that is correct um okay i' got to do the same thing for you yeah okay I I I need to do the same thing this is this is really uh but this is already paid enormous dividends and we're only like 20 minutes in or whatever do want to sound a little less surprised that 20 minutes like seven minutes was us screwing around I mean this is amazing I think I think in general though there is uh some value in having you know the kind of thing I wrote which is like you have like a nested tree structure right and you are parsing through the nested tree structure and you want to actually not just fail on the first error globally but you have kind of a notion of like you know want to go through as as much as possible and you want to collect as many errors as possible um and I have had to do that a few times and I've pretty much handwritten something uh each time so that kind of suggests that maybe uh if I don't know if if there if the audience has a a suggestion for something that kind of does that uh otherwise uh that might actually be worth doing and kind of putting out as a separate thing uh me is kind of a much bigger scope thing but yeah this is where Deton tells us to reach under our chairs and we've we've all got that crate sitting right there exactly I know uh I have a I have a couple of I have like a pair of really really cool sery related crates uh um so there is uh a crate called Seri ignored and there's another crate called sery path to error so uh I think both of these are are really good and again kind of coming at it from the you know you want to produce good error messages kind of thing right um so one of the things that I've noticed in like when defining say a configuration file is that people will often misspell things right um and um CI has this really cool uh deny unknown Fields I love this um so and and deny Fields is great uh but sometimes you don't want an error you instead want a warning and uh 30 ignored actually kind of lets you get that warning so it's kind of somewhere in the middle between like the silently accepting the uh you know maybe the typo um or failing and I really like cly ignored for that because um often like you know you want to support like some kind of forward compatibility and if you have that forward compatibility then you don't just want to choke if you see like a new Option or whatever right um and so sir ignore does a really really good job of kind of reporting that um and then kind of paired with that um but kind of solving a slightly different problem is uh ser pots to error and uh what c pot to error does is it will uh it will try and Report the nearest part of uh the uh kind of what failed so it'll it'll kind of maintain some State and like you know which Keys have you traversed into and so on and it does a pretty good job with that um so uh you know there's there's a there's one specific asterisk which uh can you know we can don't really want to kind of get into right now now because it attracts but uh overall like these this spare of crates has has just kind of been I feel like this has like really elevated the you know like kind of error handling experience around configuration files for me yeah this is great I not used actually either of these and um again we're you know I but Adam I'm also standing by my my uh belief that we should not have any bag limit on dto N crat no spot on maybe like 50 or 100 or something um but yeah path to error we've definitely used I mean I'm sure you've seen those Json errors that are like yeah no I failed to parse uh bite 6015 is that helpful no okay yes yeah I mean it's not helpful at all it's very very unhelpful the opposite of helpful that's right in fact so unhelpful because you're like maybe I could figure out the 615th bite like that wouldn't be the hard thing in the world would it I I have done that I me that has been like my kind of immediate go-to sadly as opposed to being like we should actually you deserve a better error message this is like we deserve nice things I need to totally it's a long path to really adjust to that yeah those are great what else is on your list rain oh boy um uh that was actually uh as far as uh I mean I couple other detto n crates but they'll come up later um another crate that I wanted to call out uh and kind of you know as a cool proc macro crate is uh derive where um so um so one of the things that you know kind of people run into sometimes is that uh you want to do say like a derived debug right um or like a derive clone or something and like uh if you have a a crate which has a generic like you know tle and clone um then the implementation that rust generates is um you uh it only implements like you only derive clone if T also uh implements clone right like you you only get to implement clone that way um that is like mostly what you want but sometimes not and uh so you know one option is you if you don't want that clone bound right like sometimes you're not actually storing a te in there you are storing say some kind of derivative type of T um then you do that um but uh the one I really like uh that kind of automates this is derive we um so what derive wear lets you do is it lets you say derive clone where um some bound so you could say derive derive t clone uh sorry derive my type clone where T is you know like some implementation some some implementation of some trait that you've defined uh but you can also save like you know you don't you don't have any restrictions on T so you can just do dve wear clone um I remember like showing uh something else at a demo day and then everyone else was like what's this derve wear thing and then uh that ended up turning into dve Weare demo it was very funny but uh it's a it's a crate I really like and I and I kind of reach end up reaching for um as far as I can tell it just generates a brock macro that iterates over the the fields um but it just puts different set of bounds on them so it's it's just a proc macro in that sense that's really cool yeah definitely one of my my favorite little little proc macros that kind of help out yeah so and how were you using it in the the thing that you were demoing rain what was the specific use case where you using it I think uh what I ended up I think what I ended up having was like I was storing like a Phantom data of t or something like that so what I was so I was storing like um so I had these uh I had this thing which only stored the T as a marker so it didn't actually store any concrete values of T um and um implementing like something like clone for that should not require that t Implement clone right like just logically it is um uh that is not a requirement so I ended up reaching for dve Weare with that that's a pretty common thing I end up having to do that's right NE it seems very useful yeah how you discover that crate that's this is a good question we're like how did it occur to you to go look for a crate that does that um I think uh I think what I ended up doing God this is you have telepathic crate Powers I mean you can you don't have to tell us if you don't want to but I this is I this these are there are these like these rain I'm like how does rain know about this okay so rain while you think of the answer I would just say have you ever done this bra I go to chat GPT and I think like surely there's a crate for this and I describe the crate that I want and every into crate a thousand per. it's like all you have to do cargo ad schoras board then schoras board you know colon colon whatever and it'll do exactly what you want I'm like thank you so much J GPT like great doesn't exist or it just or exists and it does something totally unrelated to it but I've never got it I've never had it say sorry like nope nothing I do think it's funny that Chachi PT is most likely to hallucinate in my experience when you yourself think that this thing should exist you know what I mean where Chachi p is like no there should be totally a CR like that in fact there is it's this it's smus Sport and the uh and on the one hand it's like somewhat vindicating that it's like because there are you know I I had I went this I chpt had a very Vivid hallucination around uh organizational gists so organizational you do you use gists in get all sure yeah and organizational gists exist but I can't see how to create them or edit them and chat gbt also believes that you should be able to do the things that I believe you should be able to go do to create or edit them and it's like no you can't do any of those things actually they do not it was uh you know you know reminds me the the organizational gists are like inside UFO 5440 have we talked about inside UFO 5440 here maybe not this year but like several times right right exactly maybe not in the last like last six weeks I don't know uh inside UFO 5440 was a Choose Your Own Adventure which okay fine choose your own Adventures were you you read choose your own Adventures yes yes but I think but rain you and like Eliza You' not Steve did you re choose your own Adventures oh excuse me I i r sorry this is great to hear this is great to hear Eliza so this is good choose adventurers have left the generational chm this is a relief I'm glad I used to like hold my finger at a place when I wasn't sure what decision I wanted to make and then like choose one and be like nah never mind and like back up one and then like I started getting a stack of like I can't hold three in my fingers in three different places in this book at the same time that's right this amazing that Eliza you also R your adventure I am just old enough to remember save scum in a book yeah okay well so there was okay well great this is great this is something that really can bring all generations together there was one of the early chose Adventures was inside UFO 5440 and inside UFO 5440 in every ending you died except for one ending where you you you got to Utopia and but there was no page that actually sent you there was no path that sent you to Utopia and I just remember my little 9-year-old brain just absolutely smoldering on this and not being able to and of course in hindsight as an adult going back and reading inside u440 in addition to the warning of like you know don't read this book straight from cover to cover blah blah blah there is a special warning that this book that this choose J Adventure may may require you to think in a very unorthodox way which was basically their way of saying that you needed to go straight anyway it was it was way too meta also as an adult I'm sure you appreciate the like maybe this will keep this kid occupied for an hour and a half uh aspect to definitely appreciate that and it uh may have overshot the mark because like I'm pretty sure my brain like just absolutely seized up on it I I don't I mean clearly I'm talking about I mean I'm I'm I'm a 51-year old man talking about this Choose Your Own Adventure so like I I think you can safely say that it overshot the mark how did you not reference this to Alexander when he was asking for a a book that influenced you most significantly like like fish Skunk Works like give me a break oh my God why was not inside UFO 5440 I mean look they asked for three not four okay that's why they obviously they asked for four that's that I mean clearly has influenced me a great deal where were we I'm so sorry I feel like I I feel I've done the thing where we are we now we've ended up on like the ending that you can't have any way of getting I'm so sorry I think um you know I I was thinking about how I'd do that and I feel like I don't there's no like magic right like I in this case I ended up Googling for like I think Russ derive custom trade bounds or something and like derive where is like in the first page of results um but like you know I'll like spend a little time looking on Google and cre Sayo and um and I'll also like you know maybe like ask some people who are there like you know I know do you know something and then um sometimes I can't it um it's just uh you know it's really hard like I think one of the ways uh one of the more I think structured ways that has helped is like like if you have a a particular code base you like which you feel like might use something like that then kind of stick around in that codes B source code I think that is kind of you know that that feels like a good way and I've discovered a whole bunch of crates that way yeah and it's funny R I think that you mentioned that because I kind of feel like this has been this glorious positive feedback loop of the rust ecosystem getting larger is that there are more kind of programs you can go to to say how did this thing do this that I like I mean certainly that's how I discovered 2 RS now ratu was from like using something that used it I'm like this is amazing like what did this thing actually use to do this um and I guess Adam is now a good time to say that teaser for our our Ratatouille episode coming up in two weeks right I think uh orhun who uh is a maintainer for Ratatouille met over at rust lab um in Florence Italy which I've mentioned a couple times on the PO and uh he's GNA be joining us in two weeks at 9:00 a.m. Pacific which is some other time in in Turkey so yeah really excited to be talking about ratatouille with him it' be great but I definitely found that from doing exactly that R of going to looking at other programs and seeing what they were using on the subject of finding crates I was thinking a bit this afternoon about you know I have personally had a lot of experiences with there are many categories of crate where there are a bunch of implementations of basically the same thing with basically the same API and wildly different performance characteristics uh my classic example is like async channels like mpsc's there's the Tokyo One there's the Futures one there are a bunch of crates that are just channels and having done a bunch of benchmarking and digging into into their implementations a few years back when I was writing my own channel they all have very different performance characteristics and they're not really there isn't a good one you know you don't just pick the good one you pick the right one for the job and it depends pretty substantially on the usage pattern and I think kind of the big lesson that I learned from that is that as a great author I found it very useful to sort of upfront like at the very top of the read me front and center when you've gone and done something that there are you know four different versions of on crates.io like a channel why should you use this why should you not use it how does it compare with other Crites that maybe Implement something similar and I found it really useful to do that and I hope that when people go searching for these libraries and encounter something that I've written they read this section and know when you know maybe this is actually not the appropriate implementation for the specific problem they're trying to solve or when it is and I want to kind of encourage others to think about doing this because I think it's it's a really valuable exercise I can maybe scare up some of the examples of times I've written that yeah for sure and is now so as I know at some point we're going to talk about bitfield crates uh is now a a good time to talk about that because that that is an example where there are a bunch of different crates I know you as well well so my bitfield crate actually um I I know I promised you some some bitfield crate opinions my bitfield crate has such a section in its read me and it basically leads with there's like no reason you should choose this um I wrote it for fun because I wanted to write it for fun uh and the one interesting thing about it relative to other bitfield crates is that mine is a declarative macro rather than and um I'm a big fan of the bitfield crate called modular bitfield um which allows you to sort of have a struct and annotate various fields in the struct with attributes and you generate this very nice um you know packed within one word bit field thing and I think that it presents kind of the nicest interface for doing this uh and mine is just kind of worse in every possible way except that it doesn't use a procedural macro uh because I thought that it would be fun to see if I could get it to work without using a procedural macro okay so I'm you know I not use modular bitfield and this is not some like this bitfields are a really good example of something where it can be hard to find the I mean there are a bunch of different crates they're all like named bitfield or have like bitfield or bitfields or they've got bitfield in the name somewhere and it can be hard to sort out what's what um so I did not I've never discovered modor bitfield and this but this is actually much closer to the interface that I had kind of wanted to find in other bit field crates this feels like a really a much a much more kind of natural interface yeah I think it's I think it's really nice it's by far the nicest interface to this sort of thing that I've seen and I just pasted in chat the um comparison with other crates section for my sort of C generis bitfield crate that says basically don't use this use modular bit but I've used my own thing in all my projects because I want to make my own I thought it would be fun yeah this is actually a great guideline in terms of comparisons with the crates I also think it's something that I like about the rust ecosystem too is that it's not like I it doesn't feel like it's a popularity contest people are just trying to find the right tool for the job and it's like hey you don't like if my crate is not the right fit for you let me actually go s you to the other crates that maybe better fit for you I mean to the contrary of being a popularity contest I mean how many crates have you gone to that have a really thoughtful like see these other crates section and I really appreciate it because not that they're apologizing for existing but really explaining like no this is not an IH like I built this because I looked at these other things and I evaluated them and they didn't meet my needs but they probably do meet your needs so like don't use my thing just because I built it like go use the thing that meets your needs it's incredibly thoughtful it's also it's very nice as a maintainer to not have to like respond to the issues of people who are just constantly showing up to say can you make this other crate well no I can't because I didn't set out to do that and you know not to sound too much like a member of a cult but Brian this is um why it's important to have upfront values for one's technical projects uh yes uh in terms of being very explicit I do think it's very helpful to be explicit about the things I care about and the things I don't care about or and because I think it's it's okay that the things that that matter to a crates author don't need I mean I and just just what you're saying a lot is about being upfront about you know maybe it's uh maybe performance is a primary consideration maybe it's a secondary consideration but being explicit about that is actually um really really helpful I think yeah and the big lesson from the channel thing that I learned from doing a bunch of benchmarking of channels is that there isn't really just a box that says performance on it that you can click right because for instance I in the read me to my channel crate which I posted in Discord chat um I discuss you know there's this question of like an npsc Channel and async channel has to store the messages in the queue someplace um and there are channel implementations that will allocate and deallocate chunks of buffer as you are like sending and receiving messages so that like the the memory usage of the channel is proportional only to the number of messages currently in the queue versus Channel crates that will allocate the whole buffer once when you make the channel and there isn't necessarily one of those is not good and the other is bad it's a question of is this channel like something that is structurally integral to the program and it lives for the entire time that the program exists and all of the messages go in that channel and is bound on that channel like extremely large and the message very big and that means that if you keep it fully allocated all the time that's like a very large amount of memory or is it something where you are making these channels and you're having a bound of like eight or 10 messages and now there's just extra overhead of of doing this like I'm going to allocate another chunk of buffer as I need it and it really depends pretty substantially on the usage patterns and there is no sort of the right move uh so it's useful to document those sort of performance tradeoffs and the ways in which like this might be suitable for one type of use but not for another yeah totally and I think that the rust EOS system tends to pretty good about talking explicitly about what those trade-offs are but um certainly excellent advice to create authors to be very explicit about those tradeoffs um I think uh uh uh a place where it kind of that's kind of organically arose is um is with um CLI um bar sync crates because there's a whole bunch of CLI bar sync crates so so I'm talking about things like clap uh which you know many if you're written a r s tool you've almost already come across clap um but there's a whole bunch of other points in this design space that people have hit with various trade-offs and um I was really appreciative of like people really put together like benchmarks for like you know you're considering things like um how long a build takes and like how many bites get added to the final binary right um versus like error handling and so on and I think you know different projects can reasonably make different trade-offs here and um one of the things I um uh this stable was like when I saw this table and when I saw like you know the amount of work put into it it was it was just very very impressive to me and so you you've dropped uh the uh ARG parse Rosetta RS repo into the chat that is looking at the uh benchmarking all of the the the argument paring libraries which I think is really interesting um truthfully I've been although actually now looking at the times it's kind of interesting look at the build time versus the overhead um because I I have used clap for most things and I've been especially with the when clap merged with Ro opt I found it to be um really pretty terrific but I also think it's great that there are other approaches out there it's not the only one out there yeah in and in particular like I mean clap has a couple different ways to use it you can use it with or without the uh proc macro but then uh there's a bunch of others like uh so actually another one that I really like um that is much lower level than clap uh is Lex opt um so the goal of leopt is like all you were like all it gives you is an iterator over the options right you know uh so you're getting an iterator and in the iterator you get like a little bit of structure so you get whether it's like a single a dash or a Double Dash so you get like very very basic things like that um and you know some if you really want that low level of control then uh then Lex soft is great uh but the trade-off there is that you need to write your help yourself and you need to remember that each time you add you know a thing you also need to add the help for that and you know maybe the error messages aren't as good and so on so so you know these are these are the kinds of things that uh that you have to consider so you know I I I recommend clap as the thing to go to right if you want to if you want to start but but these are all things that are uh you know worth considering for things like embedded binaries and so on I want to point out that Lex opt has in its read me a very nice why and why not section in which it says basically everything the range has told us yeah interesting that just looking at Lex Ops read me now um and it I mean it wants to be like small correct pedantic imperative minimalist unhelpful I feel like this is a description of many of [Laughter] us that's right look in the mirror you may be looking at Lex hoft you may be looking at Lex hoft exactly uh but and I I I it also means it's making less decisions for you I got to say like the thing that drives me uh that is annoying about clap and maybe this is something that has been fixed I should go see if they and I should probably get an actual issue open on this but there is and Steve I can't remember if if we actually opened an issue on this or not with clap that there is basically no way to have a a minus H option like a minus H it is going to take for help no matter what like if you're like no no I don't want that to be help it's like too bad clap is like it's nope that's help yeah I remember that being something I'm not sure if we did open an issue or not but yeah um and but but honestly clap is so useful and helpful in so many other regards I'm like okay you know what I actually really appreciate it but it's a good example where it's making it is not uh I think it's fair to say not small um and not minimalist um and uh it is it is different from Lex opt um so I think it's you know Eliz just you're saying what you're saying about like the being very upfront about about who you are as a crate and kind of what what the rubric is going to be for the way a crate decides to integrate additional work or not I think is extremely helpful yeah lexo is Lex Hop's nice I like that yeah it's it's it's really cool I have I've actually used it in combination with clap so I was like you know there were places where I had clap to a first level and then um and then I wanted like a second level of parsing for like something more detailed and then I used Lex up for that so you ultimately like it takes a bunch of strings right like it is a thing that takes of strings and produces output right so you can it's it's a primitive that is generally useful I think yeah and I also love that the the why not under Lex object too um so that's that yeah that's pretty great um all right what else is on is on everyone's list well I do really feel like I will be sad if I don't get the opportunity to plug what I feel is the crate that has had the biggest and most profound impact on my life personally uh and that crate is Loom which is pretty different from everything we've discussed so far this is a crate that Carl era wrote while he was working on uh the Tokyo Schuler and what Loom is it's a model checker for concurrent rust programs and the way that it works is it gives you um sort of a set of all of The Primitives in standard thread standard sync atomics and standard sync mutex and so on and a sort of simulated unsafe cell and the way these things work is that they have basically the same API as the standard Library functions but rather than actually being like you're spawning a real thread or you're just creating a real single word that you're doing Atomic compare and swap operations on instead what they do is they um deterministically simulate all of the potential interleavings of concurrent operations that are permitted by rust's memory model or the C++ memory model which rust inherits uh and this is sort of based on a a paper I believe that describes a sort of model Checker like this for C++ and so what you can do is you can have like using some conditional compilation you can say normally I want to actually spawn threads or use real atomics or what have you but when I'm running my tests I want to be able to write these um these deterministic models that will exhaustively explore all of the permitted interleaving like the rust compiler is allowed to emit or allowed to allow the oper scheduler to Emit and then you know it's um it if you use the loom unsafe cell it will check like okay if I have a immutable access from one of the simulated threads and then the this like thread yields and now I'm like executing some other thread and now there's a a mutable access to that same unsafe cell it will then you know generate a reasonably nice Panic uh and when you do this you sort of have to sit and run this task for tens of thousands of iterations because you know this is sort of a a combinator explosion of potential like paths that the model permits through this like test that you've written but the reward for that is that if you've written complex concurrent code like a lock free data structure you get to learn all of the ways in which you've done it wrong yeah which is I would say deeply and profoundly humbling um you learn the ways that like perhaps you are executing this code in real life on an x86 machine and you've never seen any of these possible data raises because you're running on an x86 machine but you know someday your code might be cross-compiled for arm and it just so happens that like you've used sufficiently Advanced or sufficiently relaxed Atomic orderings that when compiling for arm you will actually see like loads and stores reordered in ways that will result in this data race that you've never SE in real life and you've used Loom it sounds like to actually debug weight and lock free data structors I have learned used it not to debug weight and lock free data structures so much as learn that my weit and lock free data structures wrong you okay so I was gonna ask so all right so Loom has found an inner leaving which now has incorrect behavor avior yeah what does it what happens now does it in terms of like getting from that inner leaving to understand were you able to relatively easily get from from lom's discovery of an inner leaving to be able to be able to wrap your brain around what it actually happened so Loom will log it's like it will log um you know I'm doing this operation at this time and it will try to tell you it its logging is like somewhat useful it will try to use trap collar a lot so that it like captures like where was this mutex constructed in the program at what line uh where was this Atomic constructed at what line was it accessed at what line was this um this unsafe cell access and which thread did that um or which simulated thread in this test that you've written and it will try to sort of give you some help information about that but honestly it also is just sort of very useful as a trial and error mechanism that yeah sometimes you just sort of end up going oh I think I understand what the problem is and I'm going to kind of permute the program a little bit and I'm going to run it through Loom again and maybe now this model will actually you know after running through tens of thousands of iterations I've actually not found anything that causes a data race or a deadlock it also does deadlock detection and it has a a leak detection facility similarly um if you also use looms wrappers around box or other ways of allocating and deallocating tell you leaked a box or an arc uh and again the thing about this is that it sounds at the surface level similar to tools like uh tan or asan or valren uh but it's actually quite different because it's a model Checker rather than a sanitizer that you run your program under and then get back oh while it was executing it did a bad thing but you know it's possible that you'll just never see the bad thing happen during that execution whereas with this sort of deterministic model checking of course there might be bugs in the model Checker or you might have set bounds on how much it can explore the state St Spa uh wow I can't talk today the state space and you might have missed a bug but if you set aside those things you know that you've actually deterministically explored everything that compiler is allowed to generate so any anything that is outside of that is not permitted by the model this is a life-changing crate for you in part because it highlighted the uh the challenges in your own wait and log free data structur way that describe a little bit how how this impct your yeah this stuff is incredibly hard to reason about um and every time you think that you're actually good at it uh that's that's very dangerous right because it's just this this stuff is incredibly difficult for us to deterministically explore all of the interl permitted by the model in our head um and so it's just sort of like it has has really kneecapped me every time I've used it and uh just sort of taught me about my own insignificance and how small my my mind is relative to what is permitted by this like kind of extremely complex uh memory model and really the way that it has impacted me is that I will never write lock free weight free or even concurrent code that uses locks that is of sufficient complexity without using loom and I try very hard to avoid anyone else's code that has not either been tested using loom or tested using another similar model checking because I don't think that human beings unassisted do I think that it's sort of like C versus rest right it's sort of like there are plenty of C programs that have run in production and are you know thus far we have not seen the lurking memory errors in them that's great but this is a way of exhaustively proving the correctness of our programs and it has taught me that I don't like this is not me saying like like y'all don't know what you're doing because I don't trust myself to do this unassisted either I think that it is just fundamentally like you will regret not using these tools and you will regret using any library that implements a complex concurrent data structure that is not tested using a tool like I'm not saying it does it certainly does not have to be boom in particular but something of this nature is just kind of a necessary tool through write this kind of software um I C I certainly was there for the gradual sort of push to cover all of Tokyo's internals with loom Carl developed this while he was sort of rewriting the schedule and over time we sort of push to get it into more and more of the various synchronization Primitives and like other uh Tokyo internals and we found just a kind of devastating amount of bugs by requiring that any new or changed code have test um and many of those bugs had not been discovered directly but they probably sort of fixed a lot of the like weird inexplainable behaviors that there were GitHub issues that nobody what the answer to was yeah that is wild and I mean I you kind of like chilling when you start seeing all the also when you have these issues where you realize like God the the symptoms of this problem would be really far removed from the root cause would be really difficult to De bug presumably um if if seen in the wild it would just kind of be you would die on some State inconsistency and presumably um and then try to reason about how the hell you could possibly end up in that state um that yeah that seems great um and I I love the fact so so was Loom done by Carl as part of the work on Tokyo I it was kind of loom born out of the need to be able to to better understand or validate the Tokyo changes yeah at it came out of I believe between some of you might be old enough to remember Tokyo 1.0 or Tokyo 0.1 uh where like Tokyo was split into like Tokyo core and uh Tokyo IO and various other other crates um and in the the sort of process of writing Tokyo o.2 uh Carl rewrote the entire multi-threaded runtime more or less on his own and in the process of doing that he realized that this was just like extremely difficult and somewhere along the line found found the paper um I believe the original uh the the paper is called a CDS Checker and it describes a very similar thing in C++ uh and Carl basically said to himself uh you know I can't keep continuing the schedule or rewrite without this I have to stop what I'm doing and go and implement it and you know I'm sure Carl can recount this story much better than I can but he sort of stopped everything he was doing and went and materialized this thing uh since then it has been kind of improved substantially uh in particular with regards to actually being able to tell what went wrong in your program instead of just sort of well you did a data race good luck um and uh also it's performance has been kind of optimized substantially because we might not generally think like oh it's you know it's a testing tool um performance is like very very important but it's a testing tool that will execute a test potentially hundreds of thousands times um sometimes you're really sitting there for like an hour waiting for thing to run one test uh so so a great deal of perf work was sort of done uh more recently to try and make it not just mind-numbingly slow but yeah that's that's really that's its Heritage uh and I mean that's great obious improving the performance is terrific um I do well but it's also just extremely satisfying when you got the computer just working so hard to I I I mean I love it when the computers are working and we're you know we can get to uh we get to come back in an hour and see what the computer has found in terms of these of these subtle issues um it's very satisfying but that's that's great Eliza that's metal level one last note on just how long it takes for this thing to run at a meta level I would add that the length of time that the loom model of a concurrent data structure takes to run is sort of a good like earning metric too like if it takes if it takes an hour to test this thing maybe this thing is actually too complicated and you could make it yeah interesting and trying to it just driving towards something that is simpler um and then um uh someone in the chat asks about postcard um I we should use postcard in in humility we uh humility H use postcard um as a is that a c serialization form y yeah and but one that is uh pretty tight um and pretty straightforward so yeah well I'm a I'm a postcard fan for sure postcard is memory serves is very similar to huis um sort of indigenous serialization format but with a couple of key differen um I think that Cliff skipped the um varant everything in postcard every integer in post card I believe is the barend I don't know about the speciic difference is but Cliff definitely looked at Postcard whenever we were doing the Hub sterilization stuff um and definitely took a lot of inspiration from it but did like ultimately decided to design his own thing I had on my list go I had on my list another one of uh postcard is at James mun's thing uh and I had one of his other other projects on my list of crates uh which is BBQ which is like uh Q like the data structure and BBQ is a um a multi-c consumer multi-rod bite Q uh that allocates exclusively in contiguous regions in memory uh and the idea is that this is a queue that you can grab sort of a chunk of bites of a given size off the front of and then you can do a dma directly into that lead and release it to the queue and then you can like wake up the other end and uh he's got like a bunch of different uh the interface for it is like kind of hairy but it allows you to say I want this like static region that I've declared as the backing storage for the queue or I want to be able to dynamically allocate a bite buffer that is the backing storage for the queue so that you can use it really in both um like embedded projects where where you don't have any capacity to do Dynamic allocation uh you can make them on the stack you can make them on the Heap and uh they're they're really nice and they're they're dma safe so you can just like have your your Nick or whatever write directly into the region and Q that will then be consumed by somebody else uh it's quite nice it's also based on a paper I believe uh called bit buffers and I think that that's kind of an underappreciated crate that I have I have really enjoyed using yeah and as someone in the chat points out has a 90-minute guided tour um that is included go uh watch the guided tour of BBQ U yeah that looks that looks good um I have a great um uh so um is bgra do mainstream to talk about here or um no so I I I'm not sure yeah what is it sorry I'm embarrassed it's too mainstream like I no I don't think so ever heard of YouTube Brian like it's pretty B I don't know um so uh uh so pcra is uh it's a creat I've Had The Good Fortune to use um a few times um uh in my career and uh it is a crate that that uh lets you represent graphs right um so it is a crate that essentially has a bunch of graph data structures and you know you can represent your things in there and um one of the things you know and I was thinking about why I like pet graph so much and like you know there's some there's some other places where I will like handr write my own representations rather than using some framework someone has provided and like you know for this in this case pet graph is like it is a whole framework right so you you kind of model your data you put it into their data structures right um and for me like I think the distinguishing thing is um that pet graphs gives you a lot of value uh from that thing so there is like a wealth of graph algorithms that are included in pet graph um so you know there's like there's like two different um SEC algorithms there's uh there's a bunch of different like you know like men uh you know the max cut Min flow stuff like there's um there's a lot of really careful handling and um so you know at this point it's like okay you know if I have a graph and one way I could use do a graph is like you know the simplest way as you can imagine like a a a a node with like a Arc of node of children or something right like or something like that um and I think um you know what better kind of what then you end up having to write your own algorithms uh on top of that uh but bcph just kind of you know you have to do a little bit of work to fit into it but it just gives you all of these algorithms and like there have been times where I have thought that all I want is like a DFS and you know you could probably write a DFS by yourself but then I realized oh you know in some cases the graphs can have Cycles so I need an algorithm to kind of convert the graph into like what is called a condensation graph which is the same graph but without cycles and uh and then you know so it kind of gives you all of these things and um I don't know it's just there's something very satisfying about bcra in a way that um you know I I really like it yeah this seems cool yeah this great you know I've known about this for a while unlike some folks um but uh but I've sort of resisted using it just because it felt heavyweight if you know what I mean I think is exactly as you're saying Ryan about this this kind of dichotomy between kind of big framework versus kind of lean and mean um but the a great endorsement and and a good reminder to go at least kick the tires next time I come across a problem that feels like it might be up petr's graphs um alley the fact you've heard of it I mean this explain explains rain's concern that it was too mainstream for this that uh so so where did you Mr I've already heard of this thing where did you hear about Petra where where is one yeah so where where did I hear about it yeah I don't know I don't know I mean I guess just other podcasts I don't know but I I I you know I think I was looking for it agre to see other podcasts I didn't realiz we' do we have an open relationship like that I didn't realiz you're the one who appears on all these other podcasts not okay okay oh here we are now we're here so uh I in the typify uh crate that I wrote to do Json schema to rust um like code generation or I would there's a bunch of like compiler I mean um graph like problems to it in particular you got to find Cycles if you find a cycle you want to break it with a box as you generate this kind of containment cycle um as you look at derives you kind of maybe want to look at strongly connected component so I started looking for rust crates that implemented these SEC algorithms and that's where I came across pet graph pet graph interesting P graph is like yeah I think pretty well known on the forums because when people say like oh rust can only handle tree shaped data structures a very common thing is like well did you try P graph because it's it's like old it's not old as may be wrong it's been around for a long time and therefore is well known largely because it was kind of the first like you want a graph likee data structure like okay here's like a good easy one to use thing yeah interesting and and it's actually doing this by actually properly managing adjacency lists as opposed to actually like having references to nodes right I mean I presumably yeah um so bedgraph is really interesting because it actually presents like um five four or five different uh representations of a graph um so there's the adjacent CS graph which is the default graph right like you know if you want a graph then you probably want to reach for an adjacency list graph right uh there is also an adjacency Matrix graph um uh which you know if your graph is uh uh you know in some cases you want to use the Matrix representation of things and you can do fancy things with I vectors and so on um um there's um also like this other one that lets you kind of uh so the the the first two representations only let you use integer Keys um if I remember correctly then there's also one that lets you use your own Keys anything that implements copy and I think hash and EQ or whatever um uh but then it also provides uh essentially like an abstract interface so it provides a bunch of traits and uh if you have your own graph and so you can bring your own graph and you can Implement you know those traits for your graph and if you do that then you get access to like Thea set of algorithms uh to the extent that your create support that uh to the extent that your graph supports that's awesome that's really great well then I also love the fact that you can easily output it as graph is so you can actually go right that is your love language you know G plot is my love language first of all not graph is you know my bad sorry please he us yeah exactly um but yeah this is uh this is neat this is neat and just of course laying eyes on on the reference to dyra gives me that flashes me back to dy's dy's tweet actually Adam and your your masterful work there oh yeah that was the way back yeah I'll put that in the the way back exactly from the from the true Twitter spaces era H okay so rain not apparently too mainstream for Adam and for Steve perhaps but not for me I I I you could just take me as a as as a complete Neo fight with respect to some of these these crates but uh looks great I'm going to name another mainstream crate that everyone knows about but I'm going to and it is a detonate crate but I am going to give a particular shout out in it which is sin SN the the syntactic crate now the shout out I'm going to give in it is that like the more you can do things like every sin has thought of more things than you think of like so for example uh I've had a path I'm like oh okay I want if the path is exactly of length one and if it matches the string than doing thing there's a built-in for that uh if you ever find yourself dealing with like a a a function that or a uh a structure that has a bunch of generic parameters there's a function that splits it up and exactly the way that you want for doing a drive macro so this is only to say like spending a a quiet time in the tub or whatever like reading the docs for sin is time well spent and there's like lots of stuff built in there that uh anticipates the things that you think you might need to build by yourself yeah the was another Deton crate that kind of sin reminded me of is paste Adam um and I was looking for the equivalent of the uh do we say for of the the the OCTA Thorp character do do you do we say hash or or or or pound I think I think I say pound yeah I think I did say pound and I'm worried I now say hash in any case the the so in CPP CP processor there is the pound pound operator does not Google well very hard like I didn't even know what that thing was called all I know is that I had used it in CPP and I wanted an equivalent in Rust and I could not go I mean I it's like I I I didn't even know what to search for I was just I felt so helpless um and I don't think I think you bailed me out of that one at some point I think you're just like I was describing my Agony of try I I couldn't even search for the thing that I was trying to replace in terms of what is I now I now know is called the token concatenation operator terms of pound pound um but the so I couldn't even Google the thing I wanted to replace let alone a way to replace in Rust and I think you I think you would put me on to paste um but I no that that that pace is now read only I'm not sure if that's because it's done or if I should be using something else um the I I do love the fact that oh my God paste has donon has added pound pound as a as a GitHub Topic in the past crate has it been done for my benefit is amaz that was quick incredible yeah exactly um but another another great detto a crate um the another crate that I wanted to get in there ad him is um the uh uh Goblin for Elf um and gimy for dwarf um elf is much simpler than dwarf um and and I think she like lib elf is actually a pretty good library in C um but goblin you you know what's not a pretty good library is lip dwarf and C lip dwarf is not a pretty good Library that's exactly right that's exactly right lib elf is a good library and lib dwarf is really not a good library at all that's exactly right um the U but goblin makes it super easy to rip apart elf binaries um and gimle makes it as easy to go through dwarf as dwarf is gimy has done a good job of like gimy is basically like look like dwarf's problems are not my problems gimy gimy does a good job as good a job as it can do I I really like uh gway quite a bit but that was another and those are like relatively easier to find because you're looking for a dwarf a d a dwarf crate or an elf crate you kind of know what you're searching for you'll find those but they're both very good I have a shout out for a crate that uh that's sitting in a in a sea of um undifferentiated crates more more so that is to say if you're searching for I want the dwarf parcer like you're going to find it um I really like HTTP mock there are a bunch of HTTP mocking crates out there and in fact I think in our open cron repo we use all of them um by accident uh but HTTP mock uh is the one that I really enjoy the most and in particular gives you a little closure with a structure called uh when and then another structure called then and then you do kind of manipulation on when to find the um the kind of predicates of like when you want the response returned and then the then is the actions taken as a result of the HTTP query um I really like it I really like the the way it you know I think that that there are some crates that kind of like vomit their guts out and this is one where it really presents a nice user experience a nice user interface and there's a bunch of complexity underpinning that that that allows for that nice interface and I I really enjoy that one it's my my favorite HTTP mocking crate if that doesn't make me the world's biggest dork and that is HP mock right HTTP mock exactly yeah okay yeah yeah um yeah interesting so and what have you used this for so uh we use it in the um so I want so I wrote the uh progenitor CLI generator and I wanted to have um end to end validation of like running CLI commands the the CLI is built in clap as well so um I wanted to do that but not against a real you know oxide server so we actually autogenerate additional traits for HTTP mock so then you can make type checked um mocks against our API so like the the CLI is banging against this mock server to validate all the different you know CLI sub commands that we that we emit or that we create yeah wow that's really cool yeah it's a just it's just a nice interface I just really appreciate like the way that it the way that it operates there's some limitations like um you know I think there are other mocking crates where you have maybe more flexibility or like you just get a generic you know function where you can respond with whatever you want I think the constraints associated with this allow you to build something that's um a little more type safe yeah that's neat I'm just looking in the chat there is uh rhdl a a rust based HDL for fpga development we not that's that's very spicy to go look at that one yeah that sounds really cool um that that is totally new to me but then we but but I think we've already established lots of things that are apparently very main stream are are are new to me um rain Eliza would you do you have some other uh other shout outs of the crates that yeah I got um so I got one uh so this is a crate that um so um so I maintained this crate but I didn't write it I just happened to have been the one that manages its crate cyber releases um so this is a crate called uh Camino so this was originally written by um boats uh who uh was a rust project um alumna um they had uh so they were the one who drove like asyn await for example in Rust um and so they've done a ton of work and and so one of the the things that they did was um you know it if you've done anything around Bots uh in Rust like file pots and file names and stuff then it's always been bothersome because in the very very typical you know rust will be like anally correct about everything as far as possible um so that mindset kind of gets reflected in the way the B libraries are designed so they will handle weird things like um uh unpaired surrogates on Windows or like non utf8 pots on unix's and like they will and and so that ends up being like if you want to write um a tool that is as correct as possible and it handles as many files as possible then you probably need to take care of all that um but in reality most of the time you don't right most of the time like if imagine you're like at oxide right and you're writing a simple um uh like a simple like server or whatever like you are going to like the files you're going to get and the files you're going to use are like are they're going to be like well structured right in some way right so BS uh wrote uh a library called Camino which um uh essentially replaces uh OS string uh as the base with uh string as the base for things so uh so these are poths that behave like strings so so you know they don't handle every possible path but they handle like basically every like realistic bath that like most programs are ever going to see um so this is uh this is a crit that like I use for like pretty much everything that um like I end up writing and I think most people should use it um there's a uh there actually some P now this sounds like you know there sounds like a tradeoff in some cases right like you're you know you're losing some functionality or whatever um but one of the things I've realized um from my time working on this stuff is that actually that trade-off was always false and so as an example like you know if you say if you have a path puff and that path puff has a path that isn't a valid string um then that path does not get serialized as Json properly right as an example um or like if you or if you get a string it won't you know won't get serialized properly so if you are ever deserializing pots you are already putting in a restriction that those spots must be valid strings right so you are not adding anything new here and I think you know I think Cino kind of is is a real Improvement to anyone who does that um so I know that like at you know at oxide we use it a bunch um I used it a bunch but yeah I think if you if you want um if you want to handle pots and you don't already know that you need to handle like every possible path then you should consider uh using Cino well Kino does a good job in in the kind of the crate read me of explaining why it exists and and when you should use it and when you shouldn't use just to your your analy's earlier point about um I think they do an excellent job yeah about what the problem that it's solving yeah I definitely need to be using this in like three different places thank you so much rain yeah kamino rocks um I do have one last raw I have a hard stop uh 6:30 so I just really wanted to get this one in crate that I really like because of its sort of implementation and its sort of cleverness and Beauty um and it is also sort of an example of a thing where there right design for this category of of data structure and instead you like really have to pick the correct one for your use case which this may or may not be uh which is concurrent hash maps and my personal favorite concurrent hash map is John gang sets uh EV Janu uh EV Maps uh which is an eventually consistent hashmap and the way it works is it's just sort of got two hashmaps uh and one of them you read from uh and that allows you to read from it without acquiring like a any kind of lock right and then there's one that you write to um and periodically you swap them uh and this is quite nice because you know there's actually like nothing scary going on in have two two maps and a read write lock um and if you choose to have them be only eventually consistent you don't refresh the read replica on every right and if you do you still have something nicer than just naively sticking a one hash map inside of a read right lock because sometimes doing the right operation to the map will do a bunch more uh you might have to allocate something inside the map you fill up a bucket and have to move things around uh and all of that happens in a right lock that's only contended by writers right and then the lock that also contends withs just swaps two pointers right so the amount of like the amount of time that aend with that lock substantially reduced relative to just putting like one hashmap inside of a uh but you're still contending with the reader because you have said like I want to do this refresh operation on every right but you also can tune the consistency of the map and say I actually don't want to do that I want to do it periodically and now you have a situation where you've reduced the contention readers substantially by every five or 10 or 25 rights you refresh the replica that's read from uh and this is just kind of neat because I I find it very beautiful in its sort of conceptual elegance and depend depending on the particular need you have for a concurrent hashmap it could be the right one or it could be wildly incorrect for your use case I just think it's fun yeah and in particular so this when this is going to be especially a good fit if you've got many many many readers right um a read and performance is important um and it's it's something that you want to update structure you want to update but you're willing to to have some control over when those updates are seen by the readers right um You don't need them to be always the eventual because the thing I also like I mean know I correct if I'm wrong but just from reading the the the description it sounds like you've got some control over eventually consistent is not just like well it may be a day or two um it it you got some control over over when that actually happens yeah I the thing that I neglected to mention is believe way to explicitly say right now I want to the two replic as well as you can set like an interval or uh a number of rights after which you will refresh I haven't used this in quite some time I don't remember the API for it uh but the idea of it has stuck with as long as I've known about it yeah know I like it I like it I I actually I also um although nowhere near as sophisticated as this I also do love uh index maps and multimaps are two very very simple crates that are uh very useful uh index Maps being where you can actually iterate over things in the order in which they were put in the hashmap which I think is very helpful uh and then multimap allows you to have multiple values for particular key which is also very helpful um without um and again very simple crates but very very useful this looks neat uh well Liza especially if you got to run at 6:30 I um did you did you get all your crates out there do you have any last crates you need to to to get in there um that's most of my list the rest was um oh I wanted to mention uh the btes crate which is a terrible name for a wonderful library that many of you already encountered be or perhaps unknowingly uh because if you use hyper you actually are secretly using this uh and btes is something from the Tokyo project and what it is oh Sean's here in the chat uh Sean can talk Lots about bytes bytes is essentially it's a reference counted bite buffer uh so it's like an arc V u8 except that you can take slices of it and the slices are also owned objects that participate in the reference count of the whole buffer uh so this is very nice if you want to read data from the network and then you know parse it into something and you want to take slices out of it uh for like here's the the h request's path and its its headers can all be sub slices of one buffer all the bites into um and I think btes is just sort of A really lovely Library really nicely uh and also is the fun foundational building block under uh creative the rain and ey collaborat in past uh which is buff list which is just some code that um was that I think uh rain asked me how to do something and I referenced some code that had been written probably by Shan MacArthur uh within an application and that ranges went and turned into a library that used at oxide um and I'm going to let brain talk about sure yeah um so B this was just something that actually it was your code I'm pretty sure it blamed you Eliza but um so I love I love bites um because it kind of presents this unified interface um over um so so bytes comes with a type called byes which is represents um this uh so it is kind of a it uses Dynamic dispatch under the hood but it it is um a type that represents a contiguous sequence of bytes um bytes also comes with a trait called buff and that buff trait uh does not require the sequence of bytes to be contiguous um so um so you can imagine uh a different implementation which actually is a segmented list or segmented queue uh which which ends up being the right data structure for this uh off um off bite sequences uh so buff list is actually that segmented CU and um I might have talked about it I I think I talked about it in the episode uh where we talked about prop test and uh verification but like um that was where uh I ended up writing like a curer type over it one that can essentially navigate uh this uh que and um and you know use prop test for that and ended up finding six different bugs because like Eliza I find it very very hard to reason about these uh these things uh by myself wow that's cool um and uh so that's the buffalus crate right yeah that is that is great that eler I ended up writing the uh very incorrect at first but now hopefully correct uh cursor implementation that was my contribution to it that's very cool uh Ry are there any other crates that that you got on your West chair um yeah uh the last one I actually wanted to mention uh because I think it it deserves a real real shout out is uh wnow so um maybe this came up in the in the chat earlier but um so uh so I got a you know I got I got a degree in computer science and like you know one of the requirements as a compiler class and like I hated writing compilers and I hated writing parsers that was my least favorite class out of the whole thing um and uh since then I've had to implement parsers a few times and each and every time I've just like it's been been miserable um and Nom so I ended up using uh nom for something and Nom I think is a great Library uh there's a whole bunch of trade-offs across all the different libraries uh but I ended up using nom for something and I thought n was okay um winow is feels like the first time where writing a parser was like a joyful experience which is not something I ever thought uh I would say about a paral library um so uh I did want to make a special shout out to wnow um say uh Ed page has done a lot of work um in you know on this stuff and wno is is absolutely like I think you know if you want to write something parer shaped then you should probably take uh you should either use wi or if you want to do your own thing you should like look very heavily at wiow and see what it does and and and you know um kind of use that as inspiration well just looking at I mean there's a a really complete tutorial on it I it's very like this is a very kind of full complete crate here yeah I it's one of those things right where it's like it says 6 or whatever but like it it it is like too high quality to be like you know just kind of uh treated that way I think it is like it is a very very mature crate um I've used that I know a bunch of other oite have used it um pretty sure Ry I think I pointed Ry to it and he was really excited and he ended up using it and he was pretty happy with it um so yeah winow is my my shout out yeah that's a that's a great one that's a great one and maybe a good one to end on there um I love their the chapter on on also debugging is uh for winow is very cool um yeah I mean obviously I'm a I'm a sucker for for anyone talking about the debugging of their crate or their parser um well rain thank you very much Eliza thank you in in absentia um and uh thanks for vo in the chat we had a Adam this this is great we ended up with a with a lot of crates here this great I I kind of can't believe we haven't done this before and uh certainly going to do it again I feel like uh a good pairing with like our books in the Box uh annual tradition but this is a good one to come back to I I I think you're right I think this is this is one we we got to come back to uh and next time I will have heard of Petra so I I I get to be with the with with with the cool kids which is very nice um and we can do we'll do some out loud readings from inside UFO 5440 perhaps um uh well R thanks again uh Steve thank you as well of course um and yeah so Adam NE in two weeks it's going to be uh r y um and and say I'm not sure if we're going to do an episode next week or not we need to it's a holiday here in the US so we'll figure that out but stay tuned all right uh thanks everyone
Up Next
42:10
DWARF5 and GNU Extensions: Mapping Binary to Source in Modern Debugging
@UnofficialFOSDEMStreamdump
172 views•2018-02-03
11:23
Solving the Heat Equation with DeepXDE and PINNs
@Dr.Mohammad_Samara
8.5K views•2023-07-17
4:55
HTTP Requests Explained: GET, POST, PUT, DELETE
@codecademy
103.1K views•2021-10-07
19:25
Enigma Machine Mechanics: WWII Encryption Explained
@JaredOwen
13.2M views•2021-12-11
Related Study Plans & Knowledge Roadmaps
Structured learning paths in Computer Science
computer-science
Building a Rust Debugger: ptrace, ELF & DWARF
01Systems Programming Foundations & Rust
02Process Control and the Linux Ptrace API
03ELF Layout: Inside Linux Executables
+ 3 more learning modules
38h6 modules
Explore Roadmap
computer-science
Container Runtimes: Namespaces, Cgroups & Go
01Linux Systems & Go Programming Foundations
02The Core of Isolation: Chroot and Filesystems
03Linux Namespaces: Process Isolation
+ 3 more learning modules
32h6 modules
Explore Roadmap
computer-science
Building a CHIP-8 Emulator: Rust & CPU Basics
01Computer Architecture Foundations (Bits & Bytes)
02Rust Programming for Systems Development
03CHIP-8 Virtual Hardware Specifications
+ 2 more learning modules
32h5 modules
Explore Roadmap
computer-science
Building BitTorrent: Go, P2P & Concurrency
01Go Fundamentals & Concurrency
02Networking Foundations & P2P Architecture
03Bencode Parsing & Torrent Metadata
+ 3 more learning modules
26h6 modules
Explore Roadmap