Parsing C/C++ with libclang and Python: A GSFC Dev Group Talk

[00:00:02]to see you live no I'll check with this beam in a minute to make sure everyone can hear us and see us but we're going to get started really quickly before Sam jumps in and just is talking I want to go through some of the things that we usually go through which is what's coming up on the schedule so we have stand today with parsing C code then next time on the 14th we'll have talked on swagger and then after that on the 28th we'll have a talk on practical accessibility for web and software developers then after that we'll have the diversity nit talk and believe one the 26 we'll have our deaf group social where we have the lightning talks and the good food as well they'll have openings for that if you guys want to get bite into us please let me know let me know what the topic is going to be and we can fill in those those sessions they're picking right around between in ten minutes or two and that would be really good that I think that's more or less all of them we have I will be pumping out more on the next couple weeks as we're having a baby on the next probably couple of weeks so the more help we can have with the deaf group the better the time maybe you know have X for this thing run smoothly you can even upon that there today C let you ran pretty good but we can still work out some more things as well other than that I'll turn it over to Sam and the guys in the room and that you guys they go away all right Sam price I've been working on my software forward QP receiver that's not a mess in Japan after struggling with doing word documents were are gonna face difficulties so I know he's always gonna do all over Commandant's lemon tree documentation through parsing our C code and then generating our RDL is for the command until me to a database and also for the documentation so this is kind of the route that I saw the long accusing claim to parse where she code and then a Python interface to claim to generate the final products if it's sis database or an interface to MATLAB or a flight software stuff like that and somebody ever used C to do compiling embedded software so playing is a front-end llv LLVM which is a compiler similar to GCC I believe Apple and Google or doing a development on it it's BSD compared to GCC that's not BSD and they have a couple interfaces to it they have a C+ it's written in C++ I believe and then they have a C interface to it that allows you to call into it from other languages so we're gonna use lip plane to do this why processor C code I use it to generate our command plum tree definitions we also use it to find all of our function calls inside of our flight software its generate MATLAB API is to call in to our flight software so we can test for flight software and interact with it from MATLAB also if you get like an existing code database like if you got the CFE and you wanted to go and find all your 23 packets inside of it and then create a command and definition database for cosmos or one of the other RDL commands on the tree to software packages you could do that so the Python bindings you can find them off of their github website if you do it and then they have to find the mouse so they have the C index file which is as all the function calls into the shared library playing which is a compiler so you can interact with a compiler from Python through this interface life they have okay and then you call a get cursor from file and you give it a file name and then you give it the same arguments that you compile and CCC so you want to change your padding or alignment you can pass in 32 flag and it'll change your parsing from it 64 bits of 32 bits so if you're switching platforms you can pass a different flag so or if you have different kind of finds in your source code you can control that so you get exactly what your compiler same source code we have M&E C++ developers in here C developers there's a lotta different ways that you can define a c structure so the linux version is you define the structure and then you type struct it's type def struct struct a and then the definition name that was fine and then also you can do the type def then there's the third way you could do it in a fourth way so there's a lots of different ways you can define structures in C++ or C so right in your own parser is probably not something you really want to do so if you can use an actual compiler to parse your C code then you don't have to worry about all the semantic problem so your so this is an example where you get a cursor from ceiling and a cursor is kind of like an individual point in your ASD tree which is an abstract and set syntax for you so when the compiler parses your C code it kind of makes this tree of all of your notes and tokens and stuff and this lets you transfer that tree so I run this example and then you loop over all the children from the top grocery you get the struct definition a variable decoration a type of declaration so gives you the whole tree and then you can print off stuff like how many children each one of these elements has so if you may have questions to ask me because I didn't have a lot of times they have specified the cursor goes into your description there is going to crisp this kosher well that's almost just the command statement statement but you sort of touched off what your person actually defines what house so it's a it's a node inside of the ast tree bass tree familiar with that yeah I'll let you look through that tree and then you can inspect it and see that it's structure or type def and then all the structures will have children like so the struct a had two children this was a variable declaration for the global a1 and then you had a type def structure so this gives you a brief overview what's inside your source code it says I'm trying to spot the bug on the page this one yeah yeah there's a I think it's they're supposed to be a type def here I believe I got the source code you're supposed to say type dip strokes I think all right so that cursor object has a lot of attributes that you can interact with see stuff so I'm just going to so we usually when I do a development how you run ipython and that lets me interact with you run that it gives you all these macro definitions which are from the compiler one of the fields is the location you I think we lost them for I'm gonna try to see if I can reconnect with them give me one minute so good got your back now Sam nice work there bet it around micro charm Jesus if you look at the next thing is to process the structure and see all the child elements out of it this is the source code and you can parse through this and then it gives you for each of the child elements it's giving you the byte offsets so if you're trying to make definitions you can get to padding bytes offsets change between em 32 or m64 it'll change the padding offsets so kind of takes care of that all for you so you don't have to guess that that stuff it gives you the type lines and then the Spelling's then the source code that does that it's this guy we're skipping all the elements that have no location which is like the system macros and then we're only looking for the structure declarations that would have children and then you run in there and then you have all the children off the structure and then you print out the attributes that you're interested in for each one yeah so you've had sinned - M 32 flag and that will give you a different yeah they'll give you a different type I dog sense so fast I'm 64 then it changes to 64-bit learned thanks okay the Indian Asst I'm not really sure about I think you can probably specify the Indian s but that should only affect your bit fields I think so okay and the next trick is sometimes you declare structures but use type dust with a names so if you want to trace the type def back to its parent type do a check to see if it's a type def declaration and you can grab the spelling and then you have to grab the canonical type which I'm not really sure what they meant by genetical type in the claim definition so looking that up and then you grab the declaration and then you can process the structure there's all these Kircher that kinds and then there's all these different types that they'll give you back when you're parsing through the ass tree there's like the type that reference declaration then the other part is types it's also I'm either really easy dude confused about you have to make sure you separate cursor from a type and don't mix the two when you're wise you're confused now what they say according to the clang website so a cursor represents a spot in the mastery and then a type is an underlying type definition and still sometimes I get confused about that so I'm still working through it so the next part is arrays so again you loop through all the children and then you find something that you're interested in and then I'm going into that loop and then you can give it the size of an array and then you grab the next underlying type of it so if you have a multi-dimensional array you can loop through and grab all the pipe sizes so this is a way that had a single dimensional and then a double dimensional that was a double these guys lets you go through and just interact with although so one thing that I run into in some of the assists rrdl definition since they only support single dimensional arrays so I'll parse through my C code and then I'll grab all these and multiply them together to just create one large vector array or the other system to grab our tool material crews I didn't write about but it can parse and decode macros so I think I want to move into a real-life example where so inside my flight software I tagged all of my structures that are made commands for telemetry with a in group flag and then when it parses through and looks to see if that flag is inside the comments section so if I and then the first field I tagged it with it's the this is the CC SCS header a packet header so that's a standard packet header that goes on top of a packet and then I tag up all the the app ID I'll put in the comments section this is a pound to find so if I want to read like recompile with a new app ID and I parse the C code I can just automatically grab grab IDs and regenerate cam it and it's lemon tree table same with the command ID so I add or similar move commands and it was command IDs around what goes up so this is a script I made that will parse through any CFE code that and then it looks for the first field inside of all the structures to see if it's going to be a CC SDS packet or either a thumb career command so again I grab the cursor from the file loops through all the children you ignore any ones that are system ones you ignore any cursor objects that don't belong to the file you're trying to process decoding type def and then you get to children and you want to loop through and you want to look at the first field inside of a structure to make decide if it's a sec ICS packet header or command header and the reason you have to loop through them is the first view might be attribute flags like if it's packed or the padding on it and then I had a function that decides if it's Swami tree and that goes through and it looks just to see if it's a constant array grabs the tokens and it sees if any of the tokens are the command header size or the telemetry header size or if the first object type is a structure that's a CCS yes type structure and then I create a database of all these that I can then use to decide our Sleater to generate definitions so this went through all the C code and then it generated up all the structures for the C type the command string the a by D string and then where the what file just packed that came out of and then it grabs the type if it's commander its laundry so this next trip goes through all the source code and it decodes all these app IDs and the app idea in the command IDs and then puts these numbers inside of the 23-man database that works so it looks for all the macro definitions and then I build a lookup table of all the macro definitions also for the genomes and then I have a script to expand the macros it just basically goes and grabs a macro and they generates it to a integer value she's kind of complement complicate when I go everybody it basically turns it into a that grabs all the tokens for that macro and then spins them out and then it uses an email function that probably shouldn't use on it but xvg is anybody has like news cases that they're going to see this with questions okay so can give me example yeah SAS can you repeat the question for people on lines we didn't hear um so I'm kind of running out of stuff to talk about but I was just wondering if you buddy had questions about that the way we deal with that is the question yeah he was asking how you deal with dynamic arrays so in assists or some of the CCS BSP packets have dynamic is better defined inside the packet variable-length yeah so the way I would do with that I create a packet and then you'd have like 32 then you kind of like the high directory right let's say that's your packet and you want this to be in a dynamic array so what I usually do is I'll do what I do is like nag my I feel it's inside the structure so I'll give it a totally it's an array and then they'll say it's variable plank and then when it's generating the CCSD s package structure for your command definition you can get a lotta matically see this variable flag and then add it to your however you're defining your RTL database that make sense yes so like I'm not curious so if you define its variable length size yeah so ceiling is just a seed pro a Pilar it's not like semantic definitions is anything that's not inside like see library I put in my comment section as a possible comment and then it's up to the tool that you're generating that's parsing your C code to see these comments and then just add those flags to your output okay so for this tonight I missed a little bit in the beginning when you're describing what you're using this for but it seems like you have access to both the C code and as well as your Python closely you know you can add those attributes into the C editor to help with the parsing you're converting it to an HD is that correct yeah so I'm actually generating lots of products from my seat so I'll generate there's like a cyst which is their command and telemetry databases that they could gather from embedded system and then they display it on a display and they have it's the same geek income so the C code encodes it based on how you have your structure set up and then just has to know how your packets is set up to decode it so you have to generate this other same representation of your C code into the cyst using their own semantic format so this just being able to access all your variables the padding offsets offsets that lets you generate another language is definition of your C code this adapter layer adapted to the other endpoint right but they'll also like the other endpoint to have more information than the C code needs like they might have limits so like you'll get a red liniment warning limit and I go into my C code and I add comments on all my fields that I'll run this and they'll be like it's all ID a little bit high I guess make up my own semanek comments and then if anything goes about 100 then it's supposed to generate a read limit inside the system database that says if it goes above 100 then it creates a warning that make sense okay I have to gonna follow now yes given all the other strengths we're all the tags you had to sort of synchronize how long it took to get the iPod script no setup so you can basically handle all of the C tags that you have I mean I'm just amazed you see basically spitting in a number yeah so I have a lot of lead work little interface to parsing like the C code is fairly straightforward so you just have to sit there and fiddle with it so setting it up you know wasn't as yeah it's way bit easier than parsing your own C code because you just loop through and you're like is this a structure okay right grab the first field yeah yes I can another thing is it can grab functions and tell you functions so if you want to generate a MATLAB API to your C code and go over that no a lot of people are trying to do electronic data sheets mmm but then you end up writing something in XML and then you need a tool to go into XML to grab it and it's not human readable but to put in your C code then a lot more people are you know C code and you can reuse the existing code also if you change your Pilar to M 32 or 64 changes all your padding bytes if not careful so my - made more sense Sam and I think you say you're converting destruction everything's when ast and and and then outputting that - it looks like JSON or something like that what's the difference between doing that and and maybe just adding an adaptive layer inside your C code with the JSON for the corresponding drugs like the thing that you're mentioning not just now with the functions that'd that'd be cool but if we're just talking about the structures why not create an adapter layer in the sea itself to output the JSON that you need light I usually do go straight from the C code to the target language you could go to JSON but I don't know I usually go to JSON just for like the app ID and Command IDs and then I read that all the time for generating the actual definitions of the structures and that okay so the JSON that the output and responses the middle middle step in between converting into the other language yeah okay this is a MATLAB output that wenger I found a C function that was the stuff I on spirit bias it took into command arguments and then it outputs a status flag and then based off those I was the first one it creates MATLAB has this variable argument interface so I saw that there was two arguments on this guy so if you don't pass in both arguments then it gives you an error message it calls the function call with it and then if it returns a negative number then it actually throws a error inside MATLAB so that's like one example is you can kind of customize your interfaces from other languages in your application so that's one of the one of the things I did was I scan for all the times that to see if he calls like create a CC SES message and it finds that app ID and then it can find like a structure that is already known to be a command tree structure and then it can compare those pair of those two up so that's one way you can scan your source code and create a quick database of all your telemetry from an existing code base that doesn't put comments on their stuff links between this is a function call process function it's not the calling expression all right so this looks for this is the C of the cipher boss initialized message you grab all the tokens for this thing and then it's looking for the time to find that just app ID that you're sending them it's been a while since I wrote this I did do this that makes sense so another thing I did was like use this to generate Python API definitions of all my command and telemetry and then I use that to decode all my binary it's homemaker coming through but if your web guys and you can take that same definition generate JSON data if all your 23 coming manager it's just nonsense it's interesting topic but didn't put a lot of time into the slides Shae shut it off so snap yeah weather what other use cases could you see for we're using this conversion this method that you have here one thing that I do is I make my interface control document I have all my clients it's limited reading it auto generates the manage limit read document for our source code and you pass that off to whoever's going to use our final product which juice usually like last time we do this we add a word document and we're never really sure if I will don't even match what was in software anyway it's a thing called swig that does something really similar figure to that that's how it sounds a lot like what swagger does which he's the topic next the next time around okay just wiggle up parsing code and then generate API interfaces for Python or other things it's written TCL I think a little bit more control to be able to generate my end products then so that's cool that's what that means if you're you automatically generate your documentation so that you know it's up to date it's current with what the code is has yeah okay that's cool builds why not have this set up right no so this is our ICD document if it's generate we have our offsets of all the packets and then data types that's how our generator I said yeah that's pretty I got an example booking they wants to look at it if there are any questions online let me know and I'll relay the message over to snap I have all my source code on the AED GE get lab repo I'll post that if you want to dig through that sounds coke I'll see any other questions flowing yet okay and thanks a lot for presenting again see switchover took in that with Sam with the parsing Seco to think next next time around we're going to have I'm always butchering this name HD if you can was talking about swagger and the open API that's oddly enough pretty similar to the documentation portion that Sam was just mentioning about the generating your your documentation from comments and auto-generating and so it's you're sure that it's in line with your your your current code base I think they can also generate clients as well using that's why here at the I so that that should be pretty interesting kind of a follow-on to what we just end up talking about today again if you want to hear what presentation it could be you know however long just let me know and we can fill in fill in the other time you don't have to be an expert on the topic you just have to be willing to get up and prepare a few slides or prepared presentation for us and it'll work out so with that I think there's no other questions in the room we'll go ahead and end the broadcast now all right thanks everybody for coming out and we'll see you guys next time