Building a C++ Reflection Engine: Static & Dynamic Systems

[00:00:00]thank you for coming and menu sent it and we will talk about waiting for flexion engine from scratch okay so first of all what's reflection reflection let's say reflection is the ability to raise or manipulate the Parana structure at runtime or a compile time more of this okay so you know what classes your program has declared or defined but functions namespaces alike so you have the disability to to query your language and say hmm what types are the cliff in my library in me in my API what feels that that las heras what methods and Jim can do it you do this at one time it's called dynamic reflection and if you do it at compile time it's called static reflection so what are the differences here in this example this stupid sample okay we have an image space then a class a class with a default constructor a method and a public member variable okay so starting with reflection in my case is something like this okay an API based on templates that takes a function and has information about for example the source location of the entity the name of the entity the pointer to the function cetera so the place of static reflection compared to dynamic is it's completely typesafe because do you use it with templates you know the type of the entities okay there is no overhead no runtime overhead the councii is that it's everything based on templates okay template metaprogramming but I think those concepts very could help okay about the API you see in templates but I think something like boost Hana or something like that could have killed her Paulo to to clean the API okay on the other hand dynamic reflection it's something like this it has the very same information but you can manipulate and clean it at one time with one time parameters on the like it's not user friendly okay you as you can see you can directly ask the API for types and namespaces directly using strings but also the entity ceremony palpable at one time you can for example rather methyls on okay did you know the new method name at runtime and the like but the bonus performance okay this under the hood is uh and any basically so there is a penalty on this okay okay our goal is to write in one hour more or less and static and dynamic reflection system for C++ okay so let's list with the API to something simple okay something about object-oriented programming object-oriented programming about classes and relations public member functions and public member variables okay so what we need to achieve that what we need is information about the places or enumeration Signet in a header file okay with informations of the information about the class members of active in class of the hither the source file the source file path sorry the surf line were the class or the method declared and pointers to member so you can you can then involve these members like then when you have all these metadata we need an APA hopefully and user-friendly API to create this metadata or take that metadata from an entity say you pass a name in space and it says okay there are this class this an emergency on this function okay so collecting metadata okay the first thing we have to do is to parse C++ in some way to circulate this metadata so but we have what we will to do is something very simple okay just per hitter information it's it's really simple so here is our header file you process it you know put generated code with the metadata okay there is no global code base information so you don't have cross references of namespaces at compile time okay but but in the the DNG in the dispute to do that is really simple compared to to the alternative and in our case metadata sistered superflat generated header files okay so important the metadata to your API is our simplest is included in the file and it just works okay late clock for well you can buy your own c++ person okay that's not up to you okay bad a paper to use Nick Lang Nick Lang is a seeing interface to the client library okay it keeps j an APA to pass filed as it was plus file and take it's a steal bc transverse in the EI still processing it etc and live cut is really simple it's just that the client compiler it's written as a series of C++ models okay the series of C++ libraries and the intent is just C interface and the stubble bit is emergency interface to the C++ clock IPA so you're guaranteed if you update your clang installation your codebase no work with note explode okay you can use the disciplines class I peel all so that it's more powerful but it's not quite guaranteed to work in following in follow questions okay so here's our hello world program with Python using lip clan so it's really simple I was amazing it did this these fits in one slide okay in these actually works so you have to import the the client API okay this is a PI pi packets called clang which has these this IPA in the packets syndics and you have the most important classes are index cursor cursor kind index is just kind of combination database okay you file compilation from these classes and pulsar is a visitor or unknown of the ast okay and cursor kind is a type of note you are visiting in that cursor for example it says it will be a function declaration a function definition and function argument whatever okay so to actually pass as it was passed by do you only have to create an index but your input file I'm using here the file you can pass literally a CPP file and with your compilation arguments okay then the D index returns an est actually a cursor object pointing to the translation you need compiled and you can visit it recursively for example in the example printing information about the about the ast location the location of the entity the file the line the name of the entity and the type of entity if you run these do you take something similar to this very very similar to this this this one okay the upper one is my process it tasty but you can see yeah okay the room to notice a translation unit then there is a global namespace some type verbs are class the clarity on etc okay that corresponds to this file okay so you can see this project name is Paris type their normal class etc and so yeah this is a there is a field and this is a entities not supported by the lipliner API but it it's it gives you a lot of information okay so so the neckline Python API just watched the DC Visitacion API into into a curse or class as I said I call surplus or a percent say the current year or visiting in the ast and exposes some properties actually a lot of properties about the cursor the most important one wants the kind of the cursor that is the the type of note it could associate it could be a namespace and in this particular issue and a name is a classic relation our class template declares he own a class template argument well okay this poem in Luke length is the simple of the entity for example you declare a class full the spelling is full okay if you declare a function f with no arguments this Pauline is f but on the other hand if you declare a function f with more arguments the spelling is it's still F but the display name is more complete it shows you the arguments of the function so you can different in it but okay location provides information about the location of the entity and the source code gives you the position in the file the complete parts of the file and the range of text that takes that entity for example random spaces okay from line 2 to 200 for example things like that okay there are many more properties I recommend you to to always take the fact this file ok it's documented there is no documentation of the Python API so the the best one the best things you can do is to to have these this fight ok so to read the metadata we have to take only the most interesting interesting cursors for you ok do you start with in the east irreducibly and you if you are interested only in in north case for example and object oriented things they are not interested in function declarations in namespace you are not interested in templates ok we are only interested in classes and structs that are clustered fully and Maurizio's methods and fields ok more or less yeah there could be a couple of properties more that this this is fine and then when you when you think their only that that cursors you can expose more properties that you might need okay so for this is an example from from a process that takes the this Pauline ok and builds a full qualified name for the entity so you can reproduce in the generated called the the entity in an eunuch you in that unique way okay okay to delete C++ code we will write templates to generate a C++ model of the entity of the military so metadata we will post okay then we can pass the process it est to the template engine you who you like we have used ginger I think and then run it and you have a pretty and cool C++ hitter with your metadata okay the format of the generated code depends on your IP a completely okay depending on what things you want to do the the API will the API sorry will be different and the generated code with its metadata will be different I really recommend you to to read at the attitude generated code from the cute compiler okay we're waiting and static reflection API so the only alternative was to write all the metadata form in templates okay everything is written as a template okay this means we need to play with template metaprogramming a bit okay as I said I think this experience conical help okay maybe it's easier to build first version of the of the metadata in templates in wrong templates and then to write an API with Hana that looks nicer okay so what will be your metadata format my dear was to you select completely the metadata from the API and from the axis so to the API okay so these things are completely sobriety we can throw and we have the the API and the generation in the future so in my case I have a meta name space that has templates which store the the representation of different metadata with for example the the template source info this is like as information about the the common information about an entity its name its Pauline its its file path the line where it's clear okay then these this meta a test exposed at especially static conscious power functions okay so everything is done at compile time you have all the information available at compile time you have to do some tricks okay to take so to take at at least are translated into conscious query string but you can do that okay I'm at the point of the format is to expose sequences of metadata as public aliases that earliest to tab lists okay so you have lists of for example methods and method is another instance of a template of the metadata template okay okay to break the metadata we have to to beta in this case and India template to tactics or process it ast we will write a template that matches a template metadata format and there is the define now C++ code okay this is why I said first that the data is per head only okay because it's much easier to just pick a file process is it's a AST and output are here filed on having a global database with racing conditions and like it's it's much easier to children like this so this is an example of one of me my templates okay Sabine an India template that takes one of me of my processes notes okay and generates a sourcing for template distance okay it directly takes the Lakeland data process if for example this generates parallel templates of chars okay by this pack of packs of choice and the final code has a heavy ball template pack or representing the path for example to my father okay for example if you look at the at the generated code if I put the the part of a file maybe it ends like there okay but it's fine you will not read that code with the compiler so it's fine okay you have the whole templates in guy in this dilatory in the a github repository okay you can look at it the vision API once we have the the all the metadata first all the code with the metadata there anything we have to bright an API to access to that matter attacked by time okay so all those again okay to make to remember if it was too bright a metadata API that can be read at compile time so you have my goal was to to mark directly from your C++ entities for example from their class to the metadata of that class okay so you put directly the the class and the class identifiers and you get the the class metadata okay so to do that we an API that takes that with center points sorry directly templates for example this is a class metadata entry point you know directly put the class there and yeah and it returns the the metadata instance okay okay in the astronomy because it's really windows okay that's a broad its Linux is just too I'm a hacker with Windows so in the derivative code we have we have instance of the templates of static reflection meter we say we see before and we have to bright the mapping from that template entry point to the metadata okay so the only way to do that is to first instance entry points with each each entity which each entity metadata and then collect everything and say this is a class references everything declared variously in the idiot is fight okay so but the dating what the cogeneration is reduce is to to first instance entry points in the code gen namespace which directly inherit from meta instance so they that classes which are the metadata and then the API access he called Jen because Jen namespace okay in its current state is its one-to-one mapping from the user API to Cochin okay I wrote it this way so I can get rid for from all the elite class stuff from when the the started perfection proposal gets accept this is an example of the genetic code you can as you can see this is the start of my home path and I think I took these two night I think it's and more something like here okay we can see the code so as you can see there is a template field declared in the in the Kotian namespace and what ETA to reduce is to specialize the fill template with the entity as parameter so you diddly join the API and the entity together okay and the path to that file is something like this somebody seen okay okay here's one example of using this tactic of reflection API write in JSON generally sir okay so what we will do is to write a serialize function then takes c++ value and outputs at least an update okay and deserialize function that takes a type a C++ type I mean it's on addition object and returns the C++ object with that data okay these functions are completely generic okay type safe and completely automatic okay again wait and we will be using the yes informal in C++ API okay I think's wait so so the format of the JSON series Asian is more likely like this it is not a JSON object having a type key with the time name of the value of the yeah the type name of the value and then Bailey you key with this Ariel I see its value in whatever format it depends on the on the data okay so the type of the original value sister as a string in the jetsons and you can take the type I did this deserialization point okay so you can throw on a section like this is some string not an integer for example okay in the format of the JSON doesn't match the the C++ class you do pass to is DC réaliser okay and technically it's kind of C 2's plus value is really sit in an optimal way because you write an overload for this kind of C++ class or type okay this is a DC réaliser for fundamental types okay it just takes the type name this whole line is to just to take the time name you can do it in many ways you can write a generic version of the demand roll API of your compiler you can do anything the point is to take the t-time name okay and then the value this is a fundamental value maps want to want to do some value okay it could be an integer or a floating point or whatever okay and you just return the deviation when duty serializes okay you do the episode boot back sorry you can check the type at the pointer of DC else Asia so it's false and says okay this this has to be under string sorry and mental type for example a ninja okay and you return the deviation ballads compared to the type and just works okay this user is easy enough another another this map okay the mapping is is simple it's again the type the value and the value is an object that's that has a key value map okay yes only sum up mapping a map to a JSON is direct okay so during both this recursively on the on the different keys and values of the map okay as you can see here you serialize the key and you say realize the value okay then the type and then return the days to deserialize is exactly the same okay to check the type it's an order mah blah blah blah okay and you DC realize a map and map he deserve a lies it it's equal to mod value this utilizes okay for this key value per and finally the the comment the object serialization okay objects has are serialize it as a map of public member variables okay so what we do is to do a for it over the fields of the class okay and fruits filled with serialize the the member variable using it's spelling ASCII and its value as as Mali okay as we can see we to get the value of the member we invoke the pointer the member pointer on the object okay this doesn't fit on this light bad habit right here here okay object object here okay this looks better okay so you what you do is to to take this palette of the field as key and serialize it value and to serve as a pilot you have to assistive to access the field okay that's why why we do something similar to histidine book okay then the type and then return the deviation to deserialize is mostly the same okay the switch again to the code here okay Jews Jews in Europe aside operation okay for its public member variable of the type you'd be serialized value do you get the the spelling of the member variable and and then I think that to the to the simplest plus object okay okay the fall sample is is that file it is in the github repository so you can you can check it out so next example google port whopper mapping okay we have we will map Google put about data model communication model to a simple or C++ business model okay directly and ultimately okay why we have a by tech in my team we have a 2000 file more or less file of just called doing this okay there's a protocol of communication between the all the devices in the system and everything it's the same allocated new protobuf as a new value in the protobuf return prata okay everything from our c++ data model okay this is releasing the same get from the prot about put it in c++ object so what we will do is to bribe to function similar to the jason example ones that take a C++ type and NASA Bluffs re and put about C++ generated code type okay we see the last means to map from the protobuf object to the or C++ object model of the value and the simulation means to take a proton and give us or our C++ object and as I said it's very very similar to the addition example there's only one difference you can define a combustion a custom combustion from the total of API types to your type so it's easier to to do our one-to-one mapping of German okay the algorithm is really simple marvelous okay for its fields in the C++ class in the C++ object you search for equivalent or matching setter or getter in the C++ sorry in the C++ per turn off generated code for example a proverb defines our said whatever member function to assign values to a whatever field in the protobuf okay a set allocate is one cetera so you you do reflection on the on your C++ mobile and on the Pro Tour of gold okay and you try to map try okay the example is this one this is our communication model network settings an API to change the the network settings of our device okay we have an IP address a gateway paint interval and your server settings pointing to the part of the server whatever okay and I miss it to run an operation okay set the IP I of this device for example okay and the component data on your C++ memory model it's something like this there's a network settings class with an IP address a gateway a painting turbine in in milliseconds and all this stuff okay and what we do is to dynamic map from the protolith to the C++ object and vice versa okay how this works this actually works okay it's not here's your C++ object here is a brought up of equivalent object aldohn automatically yeah sir well so here no here how this works which it's full of templates okay so what we do is given our fields okay a fill metadata of one of your C++ classes you try to find the matching setter or getter of the total of generated code so for example if you have a food remember by about you try to find a set thing member function a set alligators pool member function all that stuff okay this class provides all that strings at compile time so we can we can try to search those in the indicative of your native code okay and then we sort the list of protolith member functions using that names we sort the popular functions by the distance to the matching of the or the expected mutton of the name of the of the name of the field okay so we have different versions say sort said sort by set allocated and the like okay in the caterers so hope we brought total of object from a citrus plus one we do after it like before in the sample before over the data fields of our seedless plus class we short the the corresponding prototype members following the the expected total of methods okay and then you pick the first world mean years want the nearest one so you have the expected set allocated they expected certain mutable they said expected set by us okay and you pick that one this will be analogous to a metadata instance of the reflection API and you can invoke directly that damn members okay this is zone right here okay in Titan if types of the two different AP is much you do the remapping okay invoking the first a thing sorry the value of the fields of the C++ object to the center of the protolith object okay and if it wasn't compiled you try to do something else okay and Therese is mostly the same okay to read this to the serializer so to map from Petrof object to a c++ object and it's the same thing but you ascend in the opposite order okay do you a seen the Gator of your so yeah during both the gator of the proto after I seen it to the fields of your C++ object okay this is really really ugly but the user code it's really simple it's like this okay with bright everything it's support nested protobuf messages but the RAF enumerations okay it's it is the all the mapping and the last the last static efficient example I want to show you its animations okay it's right here this is an example of using the static reflection API at compile time to check if your Liberation's have some fields or some values of they meet in whatever way okay so you can take for example if the illumination has three fields declared if your numerator has a field with value 32:43 if your animation has field names full bar okay you try this with a compiler yeah now let's try with no this isn't work no it's the terminal so you tried it for example attendees 43 to 50 this stops compiling okay and say no there is no value 4050 in the Indian abrasion tag okay you can ask the system to to take the value of the field of the numeration names for okay and give it gives you the value at compile time and you can do loaves over the numeration like like Java okay dynamic reflection once we have the static reflection API doing the memory of infection is really easy because the the actually hard part of this is getting all the information at compile time so the ignorant reflection is only brought in a mobile of the metadata at one time and manipulating it and filling it with the static reflection API so it's it's really easy compared to to the other one so yeah the dynamic will fix your namespace is pose the the same information of the static reflection API and the miss face but with an entity based engine okay so you can manipulate it like give me the class named foo were four ways on strings or things like that okay this is really really simple design okay it's just the classic and hurtin base entity system okay there is a wait there is an entity based class without the common information and then there is a class class filled class and minimum class etc all that instances are managed by our runtime okay this is the entity class with all the common information the sourcing for the name the phone name the kind of entity if the entity is attached to a runtime etc okay it has a parent for example the global namespace has no parent there's this entity has a child's name whatever okay things like that for entities are registered into the runtime by full qualified name okay so you can query directly to event and give me the column column full color on color on my class type okay and it will give you an entity with the with that metadata but you can also ask the the system in a hierarchical way so every mean even if all entities are registered it's in the runtime they managed and hierarchical organization or relationship of of the data so you can ask for example a class to return its member for and you don't have to write the default qualified name okay so for this is a bad example yeah for some there is a class Lu with a name a new operation a member and operation you can ask anything to return values everything that at one time with with one time arguments okay run temper flexi on objects instead of brighten an IP I based on data click on entities like for example Java does it returns you a field object and then you apply the object they feel object to your actual object okay I try to return an object oriented like API for the user so you directly ask give me an object of this type and it returns you an object with all the properties at the all the methods like okay this is actually just a Nene okay it's a kind of any with a not that not from Max from names of methods through pointers to member to member function with names or properties to point us to two member variables okay it's it's really see too much abide okay and this is how how the AP API looks like okay you can ask the class to return an object and take the object and invoke a method with some parameters on it yes works okay then you can also use the runtime api to expose your APR between dynamic libraries okay you invoke this micro with zero types and and this exposes on an API to to load that your types up at runtime okay so caned example doesn't feed this light so actually was this one okay you'll know the dynamic library you put the dynamic library into a runtime loader okay and this gives you dynamic reflection runtime with the api of the library okay so you can ask okay give me the class plug-in in the global namespace the class lu in the plugin class the cetera cetera okay and the class is this one here this is the the API API okay and this is the code which exposes that API the code behind this macro is awful okay that works it's actually very similar to what he does okay but queue meta type does it's very very similar and that's all okay I'm gonna talk a bit about the future work I want to do with this okay so first I'll sorry let's write a dynamic type system okay with all of this yeah it's that's it's fun to do okay so reduce air type conversions register rate for example and her turns information on casting all the like declaring types at runtime as aggregation of properties and functions that stuff and one time C++ compilation based on this I have a proof of concepts okay yeah an example running this sorry what it does is to what they brought us see make API see make simple as plus API to Rome to run CMake from C++ and what did does is to help a file system water C+ C++ in make project and it just runs and compiles loads in the library whenever job right in your code editor okay and maintenance because this was really really really Tellem and hard to bright so I have no time to make the continuous integration bills green they are fresh or even great okay I was for Britain documentation for two months more or less okay that's not so fun and I have a problem with the cross-platform support this used to work with Visual Studio for example the runtime C++ compilation worked on GCC on Clark on this rs.20 ok but I broke something in the past month okay so and finally the third-party management okay if you look at the github repository come on yeah okay this one five percent of see make scripts this is horrible okay this is just to download all the third party libraries to compile them leave them everything we just want make implication okay this is not the way to do that okay there are some package managers and we we can do do that anymore okay and of course write in a Deathlok on some blog post I think you have not written anything since you more or less okay so this is the link to the project its website and the latest documentation okay and thanks for coming any questions so what's the compilation time impact because there was a discussion recently about compile time strings and the way you basically creating a type that's a ver attic set of characters should be really impactful on the compiler so this is the first question second question why could you actually replace the character string with some kind of type the returns a constructor string I get some sort of color ball object returns a construct string which is implemented using regular string literals yeah yeah actually did the hardest thing with expired strings was to link them because I don't know why I'm not an expert on compilers okay bad for my experience whenever do you do something funny with cons experience out with linker errors one definition rule errors are unlike actually in the bottle valve in the above the for example something like a month ago okay when knuckles were eating in my machine it end up with preconceived concepts per error message like 2048 lines something like that stashing the Dickens's parrot stock okay because it was actually in the STL and function constructs perfection so the in the alternative of course is to have a type list to represent types okay because template metaprogramming it's more Estel list and the compiler support that better far better than cause expiry but there is no easy way to declare on the street literal into a tab list that's the problem no no not not really what you know what you're right you're right the the protocol example takes to compile something like 1 2 second something like that and you have seen the hither there are two types okay and that's all related to the to the tablet or teen and the on the constructs pair is trying comparing so you said that during the meta information generation you stripped the templates from functions how do we know which template template that member function instantiation to call at runtime I don't support template the sorry just this function is not available at at dynamic reflection okay yeah that's a problem in the future I would like to support also templates if you look at the other examples it's everything related to classes no templates and just methods no templates are of methods so there is no way to support in place this way yeah yeah do you support versioning not yet is it blend because for example shot in your example you showed how you map classes to part above messages usually in business logic your objects always change types change so it's very it's an important part of civilization and reflection yeah yeah so you're not supporting yet not yet this is actually a very working province project okay but thank you for your presentation how do you differ from the reflects proposal the API that you generate from is this similar the last time I check it I think it was limited compared to the metadata I'm taking okay the when I started with this our I will have a copy a print copy of the proposal in my in my back every day ok and I was reading it and taking it and and comparing it with other mates in the in the standard committee of Spain and I think it's very limited to to this example ok last question hmm hi so reflections used pretty extensively in game development for you know quick iteration on design and things like that a lot of game engines kind of support it straight out of the out of the box Unreal Engine for example has some support but it comes with basic tagging of variables so you can then go and do reflection later I was wondering if you'd looked at any game engines for yeah inspiration for this and which ones in particular actually the first thing I did with this project is to bright an an attribute system similar to the scissored one or the or the aerial one and actually check to do that what c-sharp okay mmm CL reflect from Dom William song and don't Willie asam hath post about the protection system of Sprint anthem I think there are many examples I can write say I made you with with my oh my references there are many times over thanks for coming thanks for the