Go Tools in Action: An Overview of the Go Developer Toolkit

[00:00:00]Hi, this is Go tooling in action. I'm Francis Campo, a developer advocate for the Go team at Google. And in this video, I'm going to show you some of the Go tools that I use in my regular [Music] day-to-day. I'm going to start with a completely empty directory. This directory is completely empty. As you can see, there's nothing inside. Really, really nothing inside. And it is under the environment variable named go path.

[00:00:26]and I'm inside of this directory here.

[00:00:30]Cool. So, how do I write Go code? Well, let's start with something very simple.

[00:00:37]And I'm going to use no tools at all. Uh to the point that I'm actually going to use cat. A Go program always starts with a package statement. And then a couple of import statements. And then my function is going to be function main. And when you call it, it will print hello YouTube. And that is a whole Go program.

[00:01:04]How do you write it now? Well, very easy. Go run my Go. Great. It works.

[00:01:10]Now, if you know anything about Go, you'll notice that that code right here, it looks pretty ugly. It's not exactly what I could expect when I write Go. And that's something that Goof can fix. So if I do go fumpt main.co I will get the output that this code should look like.

[00:01:28]If I do goofump d I will get the difference between both. This is basically just adding those two blanks right there. And if I do goofamp-w main go I will rewrite the file. So now main go looks as it should.

[00:01:45]For the rest of the talk I'm going to be using VS Code. VS Code is an open-source piece of software created by Microsoft and it's pretty awesome. It has some really cool Go extension that allows me to basically every single time I change my code and I save Goof will be executed directly. So I can forget about Goof.

[00:02:06]Now that we have this code here, we can pay attention a little bit more to what it is it's actually doing. You can see that thumb.printlen is calling the function print defining the package f and we're importing it here. What happens if we remove it? Well, go will complain. If we try to compile this, an error will say that font in line four is undefined or is unknown. So, how do we fix that? We need that import fa statement. But actually there's a tool called go import that every single time I save is also run. So when I save it will appear automatically. Same way if I add another hello world but now I'm using the log package that will print it to uh to the standard error instead when I save it will appear too and if I remove one of them they will remove it again. So basically with this go imports you're able to forget a little bit about uh the import statements they just manage directly. Let me go back to the previous version with hello YouTube. I showed that you can run go run main.go and get the output that we desire. But there's more than that. You can also do go build. And when you do go build, it's actually generating that demo there. That demo is a executable binary. And if you run it, you get exactly what you expected.

[00:03:29]That binary is a static binary that contains everything you need to run that file in in a Mac architecture. So I could compile this code and send the binary to someone else that has a Mac OS like me and be able to run it directly without any extra dependencies. But what if my friend instead of using Mac was using Linux or Windows? Well, very easy.

[00:03:52]Just say go equals Windows and go build again.

[00:03:57]Now we're doing cross compilation of our code and voila, we just created demo.exe. So demo.exe is indeed um Microsoft Windows mono.net assembly. And we can try to run it, but that should not work very well because it's actually only working for Windows, not for Mac. That is pretty cool. Another option to compile your code is to use go install. And what go install is doing, it's compiling your code and installing the binary, the resulting binary in the gopath /bin directory. So if I run go path /bin demo, I will get again hello YouTube.

[00:04:41]Cool. What else we could do? Well, go get go get given a package. In this case, I'm just giving the current package, but I could give something else like github.comolangexamples/hello. It will go to GitHub, download that code, uh, example. There you go. it will it went to GitHub downloaded the co the corresponding code compiled it and installed the binary right here in my bin directory inside of go path so far we've seen a lot of the different things that the go tool can do there's more than that let's see one [Music] more go list if I run it here it just says the import path of the current package not really cool doesn't do much but you can find more interesting things like you could say you know I want to print and this is a go template I want to print the name of the package and the name of the package is main because it's a binary I could also say you know I want that and documentation and you will see that there's no documentation um let's fix that so I'm going to add some documentation saying uh demo is an awesome demo and now run it again And you will see that that documentation appears here. Demo is an awesome demo. I could also do other things like I want to see all the import statements and it says that my package my package only depends on font which is normal because that's the only thing that we're doing f.len. But we can also see things like okay so what does thumb depend on? And now there's more of them.

[00:06:30]That's pretty interesting. Let's join all of those all of that list with bra lance in the middle. There you go. Now we got a list of the packages that f depends on. Pretty cool. You can imagine how easy it is with this to construct a tool that could generate a dependency graph for your own code.

[00:06:51]Okay, so I've shown one way of accessing the documentation for the current package which is right here doc and you can do the same thing for font and see the documentation for the thumb package but there is better ways to do it than that. Uh my favorite one is go doc. So if you do go doc of this package you will get the same message.

[00:07:15]If you do go doc of font you get much more than that. you will get actually the full documentation which is pretty big with all the different functions that appear in thumpt. What if I care about a specific function? Well, I can do go do fumpt printf and that will give me the documentation for that specific function. Another option is to run go do the binary with - http and give it a port. Now when you're running this, what you're doing is actually listening on port 6060. So when I visit it, you can see that the packages we can see are all the packages in my machine. Not only the packages in the standard library, that is cool, but all the packages that I've ever downloaded. So for instance, we can find Gorilla and Gorilla Max, which is something that I I regularly use, which means that I have access to all of this documentation even when I'm traveling around and I'm in an airplane without connectivity.

[00:08:15]Okay, so let's make our demo a little bit more [Music] complex. We're going to modify our main.go and instead of being just hello world, it's going to be a web server. So I'm going to say that all the requests are handled by this handler that I'll define in a minute.

[00:08:40]And I can do listen and serve to start listening on port 8080.

[00:08:49]And I finally need to define my handler. So that is a response writer and an HTP request. And basically everything I'm going to do is just print hello YouTube on the web. Cool. This compiles. It looks pretty good. So, let's try running it. Go run.

[00:09:14]Go. Huh. Nothing happens. That is surprising. Well, actually, not really.

[00:09:22]What is going on is that something here is failing. And Go doesn't have exceptions. Go uses sometimes panics when something exceptionally bad happens. But if there's an problem that is something expected, instead what happens is that we return an error. So one of these functions has an error that we're ignoring and that's why we're failing silently. But which one?

[00:09:49]Well, fortunately, there's a tool that that tell us exactly that and it's called erot.

[00:09:56]When I run it, it will tell me that on line 11, HTTP listen and serve is not being checked. And as you can see here, H listen and serve has indeed the type of a function that given a string and HTCP handler returns an error and that error is not handled. So if the error is not nil, whoops, misspelled nil, log fatal of error.

[00:10:25]Cool. Let's try it again now. Hopefully. Yeah, there you go. So, now it fails, but at least it tells me what's going on. And the problem is that there's there's already one server that is bound to 8080. So, I'm going to go here, kill that server, run again, and I'm going to get this message saying, "Hey, uh, are you sure you want to accept in incoming connections?" Which is which means that our server is actually running. So I can go to localhost880 and I get hello YouTube on the web. Cool. So now I got my first web server already completely written and everything works. For the next step, what I'm going to do is I'm going to add something that is probably a bad idea, but I'm going to check if uh the path that I'm getting uh is an email and that email finishes with atgoland.org. How I'm going to do that?

[00:11:18]I'm going to do it with regular expressions.

[00:11:20][Music] So I'm going to import regax and I'm going to write it first. So then I get the auto completion which is always nice. So reg dot uh I'm going to use mass compile and that gets the string itself. So, I'm going to say that this starts with the beginning of the line.

[00:11:44]Then whatever, which is what I care about add.org and then the end of the line.

[00:11:54]And that's going to be my regular expression. My path, I'm going to get it from path uh URL. That is the path of the request.

[00:12:07]And I'm going to check if something matches by calling find all string submatch. There you go. Uh this has two parameters. The first one is the string to be analyzed.

[00:12:24]So in this case we call it path and the second one is how many of them you want to match. Uh since I'm going to be matching all of them, I can pass just minus one. And now I can say if match is not nil then it means that that that was actually an email and I'm going to print hello gopher and the name uh the name is going to be from actually from the regular expression. So I'm going to get the second part. The first part is the whole thing. The second part is this piece here which is the username in the email.

[00:13:06]Cool. And I save. Uh that does not compile. That is sad. Let's see what's complaining. Uh does not support indexing. Oh yeah, not regular expression match. There you go. Now this complaints because it says hey possible formatting directive in print in franken call. And this is not the compiler. If it was the compiler, we could get a red uh squiggly line, but we're getting green. That is because it is not failing to compile. But go vet, which is different tool, is complaining about it. And govet is basically something that it's going to give you hints on code that might be wrong. It could have some false positives, but occurs rarely enough that for me it's something that you should always have as part of your editing tool. Cool. So, how do we fix that? Uh, yeah, that's actually fprint f. There you go. And let's add one slashn there. Also, if I'm print this one, I don't want to print the next one.

[00:14:13]Cool. So, now we have our whole code.

[00:14:15]our code what it does is it says okay I'm going to handle all web requests with handler and then listen on port 8080 and then the handler what it does is it creates this regular expression it checks if the path of the request matches the regular expression if it does it says hello gopher with a name otherwise it will say hello dear uh and then just use the name and oh and same problem uh if print f there you go I'm going add the dash end right here. Cool. So now we have our code. It compiles. It runs. Uh doesn't run because it's already running here. So let's run it again. It runs and it says hello dear nothing. Or if I tryoland.org, it says hello dear campoang.org.

[00:15:14]Oh, so that didn't match. And that didn't match because there is that slash there. Let's drop that slash. So, I'm going to do it by skipping the first character of the path. We know that all the paths will start with slash. So, this is safe. Let's run it again. Hello, dear.org. So, that's still not good.

[00:15:36]What is going on? Oh, that's the end of the line, not the beginning. So let's kill it. Let's run it again. Oh, that is worse. Okay, let's try without uh if I say okay, dear fu.

[00:15:53]But when I pass something that matches the regular expression, it crashes and it crashes with a panic. Something pretty serious says index out of range somewhere. You know what? Instead of trying to find what's going on by reading the logs, why don't we set a [Music] breakpoint? So, I'm going to set a breakpoint. I'm going to start the debugger and the debugger. I'm going to say going to debug some go and I'm going to launch it. So, it's going to emote debug. Okay. So that is my configuration that is now stored here and I can try to just run it. I'm going to stop the server first and then run it. That should request for the permission to Oh, that is new. Okay. Uh that is requesting for the permission to connect. That looks that looks good for now. And now if I refresh, there you go. I got here and we're on the first step. I'm going to go next step. I'm going to go next step.

[00:17:06]And I feel like match here. There's something wrong because that looks like the line that is failing. So what is match? Well, match according to this, it has two elements. Oh no, it has one element. It has one element and the first element has two elements. And that is the problem. We're actually accessing the second element of a slice that only has one element. So actually what we want to do is we want to access the first element and then the second element of the first element. Kind of confusing but if you look at this makes sense. We're trying to get this campo here. So zero and then one. Cool. So let's just play. Will this work? There you go. Now it's done. Let's try it again.

[00:17:55]Allow.

[00:17:56]Refresh. And hello go for campo.

[00:17:59]Awesome. Okay, so that works. Okay, so testing like this, but basically just sending requests to our running server and see what goes on is a way of testing, but it's definitely not something that we want to have to do every day. So let's build something that actually verifies the validity of our code. And that's what we call unit tests.

[00:18:21][Music] So to create a new unit test, I'm going to create a new file and I'm going to call it main test.go. And by saying it's underscore test, uh this file will be ignored by go build, go install, go get, etc. But it will be taken into account when we do go test. So that test what it's going to do is it's going to be package main too.

[00:18:51]And I'm going to create my function test handler that receives a testing t. And for now, if I do go test, it runs that test. And it says that everything went went well. Cool.

[00:19:07]Well, it went well because nothing it did not do anything. But we can make it fail.

[00:19:13]RF something went really wrong. And I run it. go test will fail saying oh yeah something we're really wrong cool so let's use that to verify that our code is doing what we expect so first of all I'm going to want to call handler so I need a response writer and an HTTP request okay so let's start with the request I'm going to create a new request so new request and I have a couple parameters to pass The first one is the method. So I'm going to use method get and then the path it's going to be http localhost 8080. This could be whatever you want. I'm going to try then campo.org and then the body. The body is going to be empty. So I can pass nil.

[00:20:12]Okay, we have the request. And now what about the response writer? Well, let's see a little bit more about this.

[00:20:19]So handler has this definition. We can go to the definition and then we have this response writer. And a response writer is an interface, right? It's an interface that has header, right? And write header. So basically what I want is a fake response writer. And luckily for us, there's one of those and it's in HTTP test. And it's what we call a new recorder.

[00:20:46]So that recorder will res return a response recorder. I'm going to call w and wreck has a couple methods but the important ones has write write header and header which are the ones that uh response writer needs which means that w is a response writer. So now I can call handler wreck and wreck both pronounced exactly the same way and that error should be handle if the air is done fail the test saying could not create request and that is the error. So now we were able to call the function handler and that is pretty cool and now we want to make sure that the response we got was actually correct. If the status code is not status okay or 200, I'm going to error the the test by saying expected status 200 got this other one. And if going to check that the response which is body dot string if that response doesn't contain the word gopher actually go for campo that that should include that then I'm going to say RF unexpected body in response response and I'm going to say that is the button that response. So that is this one here. Cool. So we got our test. Our test basically what it does is send that request and then verifies that the answer is correct. Let's test it. Go test. And it fails because it says an expected body response. Hello gopher campo. Oh it not string contains. There you go. say it doesn't contain hello then it should fail.

[00:23:08][Music] Great. So now we have everything we need. We have our test. And what is even cooler is that you can see both pieces of code one next to each other and run the tests by running test go test package. And I will run all the tests in the package.

[00:23:35]There's only one. So that's pretty much it. But now I can do test with coverage.

[00:23:42]And when I do test with coverage, you can see that we're running this test handler on the on the left side and we're measuring what are the lines that are covered by that test in the main go in this file right here. So we can see that we actually never executing this line. So let's add one more case, one more test case to make sure that that use case is also covered.

[00:24:08]So I could copy paste this code but instead what I'm going to do is I'm going to use what we call in go a tabledriven test. So I'm going to define a new variable which is a slice of strcts. Uh those trucks have input and output which are strings. And the first one is going to be exactly what I had. So goland.org and I expect to find gopher campo.

[00:24:32]The second one is going to be something else. And I expect to find actually let's use something like this. And I expect to find dear something.

[00:24:51]Cool. Now I'm going to iterate over those cases. And here instead of camp.org I'm going to use C.IN. in and here instead of go for compo I'm going to use C.out. That's it. So now we're executing the same code but we're checking for two different values and we can easily add more if we need to. Let's rerun our tests with coverage and you will see that that line change color and now everything is written. Okay. So now we have something that runs and it is valid because our test pass. That is pretty good. But is it fast? Well, let's see how we can tell if something is fast or [Music] not. What I'm going to do is I'm going to run my server. Go run main.go. Now my server is running and I'm going to use go work. that will do basically 5 seconds of load testing over my server and I'm going to try with campaolon.org. So now it's going to during 5 seconds going to use tango routines and after 5 seconds it will gives us it will give us the result. it will say, "Okay, so you were able to run 13,000 requests per second, which is a decent amount of them if you think about it, but maybe you can do better. But how can we do better?" Well, we're not really sure, but we're going to save this for later. Let's save this as perf.0 and we we we'll check that out later.

[00:26:43]So, how do I know how can I get more information about what my server is actually doing? One of the ways is by instrumenting your code. And my favorite way of instrumenting the code is by importing http puff. And as you can see, I'm importing it with an underscore as the name for the package because I want to make sure that uh go imports doesn't delete it.

[00:27:09]So now without doing anything else, I'm just going to run my code again. And everything looks the same, let's see if it's slower than before. So before we're able to do 13,000 requests per second. Now we're able to do 12,000 825. So there's around 500 requests less per second. But that's actually really not that bad. Now the good thing about this is now all of a sudden we're able to understand better what's going on. Something that we can do is we can visit localhost debug/prof. And when we visit this page you're going to see that you can see that you can list all the routines all the go routines and you also have access to the heap. And on the heap at the end you have a lot of information of the memory, how many freeze, how many hip allocations, but also how many times the garbage collection run so far, which is 2004. And what were the latencies for the pauses in nanconds for the last last 256 pauses? That is cool. And you could probably write some script that parses this and obtains the result. But there's actually already something that does exactly that and that is pro. So you can run go tool pro say I want to run it for 5 seconds and the endpoint to hit is debug p prof profiles. I'm going to run for this. Oh uh let's see what did I do wrong.

[00:28:54]dcs uh profile. There you go. Now we're running it. So go to pro during 5 seconds and in 5 seconds we we'll get to puff. PRF is a tool that uh if you never used it before basically what it does it it analyzes some binary uh the execution of some binary and then it allows you to investigate what happened. So if you do top oh the profile is empty. So, it actually did not measure anything and that is normal. Why? Well, because there was no traffic going on. So, let me run go work. But I'm going to run it for one minute this time. And let's do it again for 5 seconds. I'm going to record. I'm going to uh get the profiling working. And now if I do top, there you go. And I can see what pieces uh what functions are actually the most uh or we spending most time running them. But it is actually not that useful for me. Uh instead what I would like to see is this graph here by running web. We get this huge SVG and we can log into it. We can zoom into it and see what's going on. Okay. So that is a Cisco pars [Music] get. Okay, that is main handler. This one here, this is the one that we wrote.

[00:30:25]This is the one where we care about. You can see that it's calling print f and compile and all matches and stuff like that. So that actually kind of makes sense that that looks like pretty much what we wanted, but it's kind of hard to understand exactly how much every single one of the those boxes is taken.

[00:30:42][Music] Something that I really like is if you run that again, you can do go torch for 5 seconds and go torch is something that developed by by Uber that will do something quite similar that go tool puff. will measure all the performance of your binary and basically uh taking snapshots of what your process is doing at every single time and then it will generate that torch. SVG. So let's open that torch.svg. I really like this one because it's much easier to understand what's going on. The horizontal axis, it's basically how much time everything takes in percentage. So we know that connection serve takes 81 81% of the whole time that we measured but the one we care about is this main handler and now in main handler we can see that there is three things fprint fals all string sub match and mass compile mass compile calls compile that calls compile that call compile and pass that goes on and on and on cool so now we understand better what we're spending our time doing the first thing that seems pretty obvious is that we're doing this compilation of the regular expression every single time. And that's probably something that we don't need. So I'm just going to go to my code and say, you know what, this regular expression, I'm just going to move it out and basically do it only once. The rest of the time, we're just going to reuse it. So going to run my code again.

[00:32:26]I'm going to run some requests and going to do go torch again. Now hopefully torch.svg the new one will not contain that compilation step anymore.

[00:32:42]Let's see it. Okay, so we have actually became much smaller which is pretty good. Main handler is here and now we don't have the compilation step.

[00:32:54]Instead we have fprint f final string some match and uh then some com t2e that we can we'll see later. Cool. So is this faster? Well we're not really sure but go work knows. So let's run it.

[00:33:19]So if we run it for only 5 seconds, we get that now we're able to do 20 almost 22,000 requests per second while before we're able to do 13,000. So we want 9,000 requests per second. That's pretty good. Okay, let me save this for later, too. I'm going to save it as perf one.

[00:33:43]Okay, at this point I could continue doing this, but instead what I'm going to try to do is I'm going to try to understand better what's going on with that specific function, the mainh handler. I want to do a benchmark for that function rather than running the whole [Music] server. How do you do that in Go? Well, actually, it's very similar to what you could do if it was a test, right? So everything that I'm going to do is I'm going to copy this test and I'm going to use always the same one. Let's move to what we had before. So that was here. And instead of repeating it per case, we're gonna call this benchmark. Benchmarks don't get T's.

[00:34:42]Benchmarks can get piece. And we're going to repeat this B dot N times. That's it. Now this what we what used to be Oh, that's not it. We need to rename the T as a B. Okay. So now it now that is uh that is it. We have our our new benchmark handler which pretty much does the same as as our code but it does it as many times as we need. And if we want to do it, if we do go test.bench with this, I'm gonna run my benchmark and get the fact that it's taking 4,367 nconds per operation, which is pretty good. So, can we do better?

[00:35:33]Again, we need we need profiling. So, I'm going to enable more profiling. CPU profile is going to call prof. CPU and I could open it with go torch binary name is demo test and the input is prop.cpu I can create my torch. Now you can see that this is running my demo.andler handler many times and you can see exactly how much time uh that is taking.

[00:36:13]So find all string so much is what is taking the most. It's taking 41.3%. So how could we improve that?

[00:36:22]Well, at this point probably is when we should reconsider using regular expressions at all because basically what we're doing here is we're making sure that this is something that finishes with goland.org. work. So what if instead we did something like if strings has suffix uh so we're going to check that path has the suffix at golang.org Then we're going to use this could be faster, but I'm just going to use trim suffix and we can drop completely this.

[00:37:10]So now what I'm doing is if uh my path finishes with agoland.org then just remove that suffix aoland.org and use it. Um actually let's declare as an as a variable. sign on.

[00:37:27]Cool. First things first, go test. Go test passes which means that our code is still correct. That is cool.

[00:37:37]Go test that bench. And let's see. Before we were able to run it was taking four n four milliseconds. Now it's taking 3 milliseconds. That is pretty good.

[00:37:48]That's a pretty good improvement.

[00:37:51]Can we do better than that? Well, let's see a little bit more in detail. Let me create that CPU profile proc. And this time I'm going to use the tool p to open demo test and prov.cpu. And instead of doing web, I'm going to do list of my handler. And I can see that I'm spending 420 milliseconds uh on this function here. But I would like to know how much time I'm actually spending on string trim suffix. You know what? Let's go back to our go torch. So we have our demo handler.

[00:38:47]fprint f is the one taking the most now right header here it's taking a very long time and most of it is taken by the tech content type so why is that well go when you don't set the content type on the web response it tries to figure out so it's actually analyzing what we're writing to figure out if it's text or if it's JSON or if it's an image so instead of doing that why don't we just say it?

[00:39:18]So we can say header set content type is text plane. Cool. Let's see if that is faster or not. So from 3,00 to 2600 and now we went down to oo, 1900. That is really good. We're under 2,000 now. That is great.

[00:39:42]Why don't we open the torch again? And you can see the our handler now. It's mostly spending time writing the header, but that's actually really fast. So I think that we're pretty much okay. Let's see how much faster it is. So we're going to do it as we did before by using go work. Let's see. While that's running, we started with 13,000 requests per second, then 21,000 requests per second, and now we're up to uh that's not crazy.

[00:40:23]22 almost 23,000 requests per second.

[00:40:26]But still, we went from 13,000 to 23,000. So, we get 10,000 requests per second just by doing some those little tweaks.

[00:40:35][Music] So, does that mean you should always be using these tools to change every single piece of your code to make sure everything is faster? Well, no. Only if it's needed. Only if you actually need that uh extra performance. But it is important to think about the fact that you should measure that performance before you start trying to make any other tweaks. Why? because maybe you're going to spend hours improving some piece of code on which you're only spending 1% of your time. So the increase on performance going to be pretty negligible. So it's better to first do your monitoring, use go torch, use go tool, prov, use whatever tool works for you. But at the end what you need is first analyze your code then improve it.

[00:41:29][Music] I hope you enjoy this talk. Thank you very much for paying so much attention to such a long video. And you can ask me any questions on my email which is by the way campo.org or you can also find me on Twitter as Francesque. Thank you.

[00:41:48][Music]