Implementing User-Defined Functions and Closures in a Programming Language

[00:00:00]back to my Channel today we're going to be finally diving back into the series in doing user defined functions so let's just get started and dive on in if you need the code for this episode it'll be in the description down below so let's get started by first defining what the Syntax for functions is going to be in our makeshift language I decided I'm just going to do the FN keyword to denote the start of a function the name of the function we'll go after and in this case it'll be add and then we'll have an arguments list where we'll pass the parameters so for example X and Y and then we'll open up braces and we can do whatever we want here like let's results equal X Plus y prints results and then I'm just going to pass back results like so so since our language implicitly Returns the previous the last evaluated element in a block this would be valid just by doing this so let's now go ahead and actually get started by implementing this first starting with the um the lexer right so what we want to do is make sure we actually have a keyword specifically for a function so I'm going to do the FN keyword and this is going to be just FM now let's go ahead and make sure we also have everything we need we need commas four comma separated lists parentheses it looks like everything else is perfect so that's good let's now go into the bottom of our lexor and just really quickly make sure that we add in the token one second there we go uh FN is going to be a token type dot FN there we go okay so that's it for the lexor let's now move on to the AST types and come in here and right below variable declaration I'm going to create a new type for function declaration like so because again this is going to be a statement so syntax like this like const Foo is equal to this isn't going to be valid because this is a statement in itself you could make it valid but you would have to define a function as an expression for that to work so we have variable declaration I'm going to come down here and right below the variable declaration I'm actually going to copy this and change it to be a function declaration change it in here as well and we have a few things right we're going to need the parameters now the parameters is simply just an array of strings right when you declare a function it's a string separated array since we're in a language that's 100 Dynamic there's no like you know I don't have to put that this is a number or whatever neither do I have to do that here right this is just going to be a string separated array so that's perfect this is going to be the parameter list then we also want the name of the function right I mean that's really important we need to know what the name of the function is so we can Define it in a local and Global scope and then lastly um we need to know the body and the body right here is just going to be a statement right so the body of a function is simply a statement like so so this is all we need actually for a function now if your language has extra keywords that are like really important right like maybe in between here you wanted to you want to Define that this function is like asynchronous right or maybe you want to Define that here like this is an asynchronous function then you can hold on to those values inside of here like async could be a Boolean right um in fact if you look at the JavaScript syntax um there's things like Arrow which is a Boolean and this will state whether this is an arrow function or an anonymous function all those kind of things for now that we're going to keep it really simple and just Define a function declaration like so okay so that is almost it now we just had to define the parser rule for this statement so I'm going to come up here and right below variable declaration I'm going to just add in our function declaration right we got to import it okay and then inside of the statement right we have a check we have a check for a let and a const where we parse variables and we also want to have a check case for token type dot FN right we want to build a handle function declaration so we're going to return this dot parse function declaration like so and I need to see if I can quick fix change spelling declare method declare yep so we're going to declare the method right here and here is our method so it's perfect let's now go ahead and actually implement this right so we know at this point in time if we've made it right here we know for a fact the current token the token we're at is of type um function right like just like when we're here we know that the token is of type const or let so let's come in here and let's just eat that token so let's just do this dot um we will do eats which just eats the token eats FN keyword perfect so we've eat the keyword right now what we need to do is actually get the name so we're going to do const name is equal to what we're going to do is this dot expect because we want to expect a token of the type so we're going to do expect and we're going to do token type dot identifier right I expected expected choke function name following FN keyword and I'm going to just get the value like so so this will double check that we have the name of the function and throw an error if it doesn't exist okay so now what we need to do is we need to handle the arguments list and this is tricky because um we don't want them to be args like in the call Expressions where it's an expression array we instead want a string array two handles so we're going to do const orgs is equal to this dot parse and we have a parse args function which returns an array of arguments now just in case you don't remember what this function does I'm actually going to come in here and go to definition and we can see it expects a parenthesis and then we read in the arguments until we get to the parentheses right so it's just going to read in and parse the argument list which parse argument list simply just reads comma separated arguments however however however however according to our AST we're not trying to pass args right arguments are a runtime semantic we're trying to pass um parameters which is just the name so we just want to know the variable names so at runtime we can Define them and all that good stuff so what we're going to do and I had to find where we're at it's parse function declaration right what we want to do is we want to map over all of these and just double check that they are all strings so we're going to do cons or uh params is equal to this Dot orgs args not this sorry args actually we'll just do this we'll set it equal to an array like so and what we're going to do is we're going to say it's a string array like so and what we're going to do is we're just going to do a const for Loop uh ARG of arcs and we're going to just double check that the ARG right which is an expression has the type of identifier right so we're going to do if org dot kinds is not equal to and we're going to double check that it is an identifier they all have to be identifiers right if it's not an identifier we're going to throw an error I'm going to say expected uh inside function declaration expected parameter to be of type string I don't know something like that right um and we could log the actual thing and that's just what I'm gonna do I'm gonna log the argument like so okay and then we're just gonna do params.push org um dots uh we'll have to destruction this looks like so we'll do ARG as identifier dot um symbol like so and there we go this is going to actually push uh the arguments into the parameter list right so now we'll have a valid parameter list so if we look at what our code is at this point in time we've already parsed all of this which means now all we need to do is parse um this right here so that's really cool we actually already have that handled as well right if you remember inside a parse statements let me see what we did for parse statements yep so let's go ahead and now we're going to do a this dot expects we're going to expect a token type dot we're going to expect a open uh is it open bracer bracket I always get these confused I think it's Bryce um open brace open brace open brace one second I always forget which one's which uh let's come in here what is a bracket and what is a brace this is print open brace poof I got it okay so expected ah ick function body dick declaration right so we're gonna expect the body and then we're just gonna do a while uh actually I'm pretty sure we already have let me look up here real quick pretty sure we already have a block expression do we not okay so what we're going to do is we're going to come in here and we're gonna do a um const body like the body is equal to we're going to give it an empty array we're going to type it as a statements array like so right so this is going to be the actual body what we're going to do is we're just going to do a nice little while loop so we're going to do while this dots um at Dot token type is not equal to eof I think it is dot dot type is not equal to token type dot e of and this dots at dots let's function DOT type it's not equal to token type Dot close brackets and we'll do a const uh actually we'll just do this dot body not body this sorry body dot push this dot parse and we're going to just parse a statement right parse statements like so so we're going to be pushing on to the statement we're going to continually parse and parse and parse and then we're going to do this dot expects token type Dot close brace um closing closing there we go all right so now we have everything we need except um right we've parsed at this point in time we've gone ahead and parsed the name the function the arguments and the entire body now so that's awesome now all we need to do is just construct the type so const function is equal to uh I guess we can't call it function we'll just call it FN is equal to it's going to take in the body the name the uh parameters and the type is a function declaration right there we go as a function declaration like so uh is equal to body is a statement array let me just double check the AST so we expect a kind to be function declaration body to be a statement of array [Music] um let's see conversion of type string params as a string array type string how does it type let me double check this uh it's kind it's kind that's why there we go now all we have to do is come down here and return FN like so uh actually just kidding there is still errors so body um look at this again okay parameters is a string array oh parameters is equal to params there we go I'm just a dead there we go okay so now we have our function let's quickly try this out and just see if this works right we should at least be able to parse a function like this uh and I'm going to just create the subtract function I did not into the sub takes in a and takes in nothing right and I'm just going to copy add paste it inside as well just to like show um that this is working correctly right so there we go just to add it spice it up a little bit so let's now go ahead and try to run this and we would expect that we'll get an error when we get into the interpreter.ts we should see down here the SD node has not yet been set up for interpretation so let's quickly go ahead and actually try this out so we're going to do Dino or run Dash a main.ts we get unexpected token found during parsing value is a closing um bracket type 14 okay so let me just quickly get rid of that just real quick make sure it's not because of uh okay yeah so let me actually go ahead now let's look in the parser and let's try to figure out where this happens so it's because it's a BRAC it's I think closing brackets bracket bracket bracket is it a bracer bracket oh wait it's um closing Grace everything's brace brace brace this dot is it expected Advance always let's see yep um args parse args params cool that looks beautiful um let's just come in here in console.log args and we should see hopefully oh there we go I actually okay perfect um so we get the error message we were expecting to see we run it we get this St has not been set up for interpretation right and it tells us that there is a kind of variable declaration um that we're not able to handle yet so that's really cool let's uh go ahead and you can see it's a function called add it takes in an X and A Y so we have them as a string and we can see the body contains some other stuff so that's really cool let's now go ahead and actually try to use this now and handle definitions so now we're done with the front end side of things we need to actually create a type for functions right we already have native functions added so I'm going to go into the values and we can see we have this native function type I'm going to come up in here and create a function type as well so we have native functions and functions right I'm going to copy this right here like so and this is just going to be a function value function value we don't really need a make Native function yep that looks fine okay so it's not going to have a call method actually instead what we're going to do is we're going to have the name which is going to be a string we're going to have the parameters which is a string array um right we're going to need to know this at runtime so we can build the environment speaking of the environment we need to know the Declaration um environments which is going to be an environment right we need to know where we defined this function if we Define this function here the environment is going to be the global environment and that's important because we need to know that so we can capture scope and provide closures so we have the Declaration environments and then we'll also have the body which is going to be a stmt array and I should be able to yep up in here Imports import statement from dot slash dot dot slash front ends slash AST I think yay okay so we're gonna have the body the Declaration environments the parameter list and the name and that's all we need to create a function so let's handle function declarations now so let's go into The Interpreter let's come in here and let's handle instead of variable declarations we'll also handle function declarations so this is going to be function declaration and this is going to be a function declaration like so this is going to be as a function declaration there we go okay so now let's go ahead and create this function right here so I'm going to go into eval eval statements I'm going to copy the variable declaration and we're going to change this to match function declaration declaration this is going to be of type function declaration I need to update the Imports update Imports uh we'll also fix this here because the Imports all screwed up missing import there we go right so now we can come in here and start getting this working right so what we want to do okay so to actually Define a function what we need to do is just create our function like so so we're going to create if we were calling the values what do we need to create a function value well we need all of these properties right here so I'm going to just copy those real quick and we're going to say as function uh value like so okay so what do we need we need the name well the name is the Declaration dot name that's simple the parameters is the Declaration dot parameters the environment is just going to be EnV because we're just going to capture our current environment right this is where we're declaring the function so that makes sense next what we need is the body so the body is going to be the Ian declaration dot buddy and there we go this is really it for our function and instead it's going to be function like so and I'm also going to come into the values and just change this from instead of saying native FN this is going to just say a function and there we go so in here we've now created a function value so now what we need to do is given this declaration environment we need to define the function so we're just going to do e and V Dot declare variable right what we're going to do is we're going to declare the variable as a declaration dot name so this is the name of the function we're going to declare it as FN and we're going to give it a constant value of true because we we want to declare the value like so and all this is going to do is then return FN so we're just gonna do return declaration bar and there we go that's all we have to do to actually create our function so what will happen is is we'll create this and whatever scope we're running in it'll capture the body it won't actually execute anything in the body right this is these are all just statements but it'll be ready for when we have a call expression so let's now actually go ahead and try this out it won't work but there's only one little fix we have to do so I'm going to come in here and just run it and see what happens and we shouldn't get an error we run it we don't get an error so that's actually really cool right we're seeing that we're not getting an error at all so I'm going to do prints and again we're not calling it we're just printing the value and what we can see is we get our reference to our function we can see that the environment it's declared in has the variable map with all the global scope right it has that this is a constant we can see that the body contains the actual stuff we need so that's really cool now let's go ahead and try to call it so it's call add with the value of 10 and 4. right if we recall correctly actually I'm not even a print because um it prints in here for us so let's just do const results is equal to add add a semicolon prints and we're just going to print results like here so we should see the value result printed twice now what will happen is this is going to throw an error because we haven't handled this yet right and there we go error uncut in promise cannot call function cannot call a value that is not a function this is a custom error message we created so we actually can see where we need to fix this is in the Expressions right in the evaluation why why in here because in here we have evalcolic spur and what we do is we get the function right if the function type is equal is not equal to a native function we just throw an error so instead what we're going to do is we're going to change this so if the function is a native function we'll do this this logic right here beautiful then we'll do else if uh FN DOT type DOT type is equal to a function we'll do some other stuff here and then else actually we don't even need to throw the else we'll just copy this throw statements right here um and I'm going to get rid of this else if if we get to this point in time right here then we know we're not a function or that now what's really important here is we can't just do what we did before right there's no call method on a native function so what we're going to do is we're going to do const FN is equal to and we're just going to do um I'm just gonna call it funk I guess const Funk is equal to FN as uh function function value right here right we just want to get the function um destructured like so okay what we now want to do is we want to construct the arguments list right it's really important that we actually are able to construct um yeah so or sorry we need to construct the new environment so we're going to construct the scope for the new function this scope mind you is the scope that the function is going to have at runtime and the parent is going to be not this environment at all actually it's going to be the environment of the function so we're going to do scope equals new environment and the parent is going to be function dot the Declaration environment right this scope right here will inherit from the function itself creating a closure of sorts so now we'll have the scope what we want to do is we want to create right creates um the variables for the perimeter parameters list right we need to create the variables so that way they're actually accessible right if we don't actually assign X and Y to this new scope we created this would throw an error so what we're going to do is we're going to come in here four const uh param of uh parameters actually we'll do it this way we'll do four um 0i is less than and we're going to go into scope dots not scope sorry funk Funk dots and we're going to go through the parameter list dot length I plus plus so what we're going to do is we're going to do scope dot declare variable and we're going to declare a variable with the variable name of the parameters at I so we're going to do I'm just going to do const bar name is equal to Funk dot parameters at I so this should be a string right this is the variable name so we're going to do VAR name and the value is going to be args of said values so args of I and this is going to be a false it's not going to be constant we could make it so it's constant and you couldn't assign it to the Constructor but I don't really like that now I am going to put something here um to keep in mind to do right um check the bounds here right we know for a fact we're never going to miss the sponge but there might be less orgs than the print the function so we need to verify airity of function so please do that on your own make sure that's correct right make sure that it is correct handle it how you would once we come down here right we know the scope is now populated with all of these arguments which is good ants um we have the function right here so all we have to do is do evaluates actually we'll just do a return on evaluates what are we going to do we're going to evaluate the funk right the funk dot body and we're going to pass in the environments of scope so Funk dot buddy is a statement array um oh yeah that's actually not what we do sorry what we're gonna do is we're gonna do the results is equal to an array and we're going to say this is a runtime value right run time not like that so result is going to be a runtime value we're going to do four const um uh statements of uh Funk dot body so we're going to evaluate each statement in the function and at the very end we'll actually return results right the last evaluated statement whether there's a return statement or not doesn't matter so what we're going to do is we're going to do um statements of function body so what we're going to do is because there could be an empty statement is equal to make null because there might be a chance that there's nothing in this Loop right what we're going to do then is we're going to do const uh we're going to do results is equal to evaluates on stmt and we're going to pass in the scope like so and what this is going to do is actually evaluates the function body 1 by a line right statement by statement line button so that is what we're going to do this is if we're at the type of function if it's a native function it's still super simple and if it's not then we're going to throw an error so let's actually go ahead and try this out now so we're going to come in here and try to run it and that's pretty cool we can see that we get both of our logs right inside of this function we print the results that's why we're seeing that I can also get rid of this right here and we can see that we have results and I can do results uh plus five for some you know and we'll do add 10 and 4 so we should see 19 come out of here and we get 19. so we can see that we have a scope we can also in here what's really cool is we can create a function so we can do function subtract uh actually let's just demonstrate this by creating uh we'll just have our add function stay like this and what we'll do is we'll create a function called make adder and it's going to return a function right so just to show that we can do this uh we're gonna do this like so and we're just gonna do add like so now what's interesting is we can pass in a um offsets like so and in here we can do X Plus y plus oh so now what this is going to do is this is going to create a closure so we'll create a function which we can call so we'll call Adder right Adder is equal to make Adder we'll pass in an offset of one so whatever we give our Adder right is going to be at like X Plus y plus that one so now we can do prints and we can do Adder we can call adder with 10 and 5. so we'd expect to get 16 here which is really cool so let's see if this works we run it um and I need to const that or I need to see that is that and that's really cool we have created the ability to return a function from a function so we can just double check that that's what we're doing um we can call adder I'm going to just pass this in right here there we go we can call Adder print it and we should see that we actually get a function back so when we call adderm or print it we get a type of function right so that's really cool um yeah that is how you can Implement functions inside of a language that is like this again there's a lot of moving Parts here so if you're confused um or would like any extra explanations check out the code in the GitHub description down below also feel free to join our Discord server and I would love to help one-on-one in a Discord call or something kind of clear up the air with any of these things so I'll talk to you guys in a later episode peace