Advanced cgroups v2: Release Notification, Delegation, Thread Mode

[00:00:00]we're good to go we're good to go okay um again the slides are already up on my website for this presentation and so what i want to do is just look at a few other bits and pieces in the c groups interface three pieces release notification delegation and thread mode which is something of an innovation or perhaps it's a dispute resolution in sea groups version two that will become a bit clearer okay all these features by the way in some form were present in secrets version one as well but the implementation or the design is better in sea groups version two because secrets version one one it was a mess okay so what i mean by release notification the general idea here is you've got a you've got a c group you put some processes into the c group but then eventually all those processes leave the c group either they get moved into different c groups or they terminate and the c group becomes empty now in that circumstance we might want a notification to tell us that the c group has become empty and we can get such a notification this is what cgroup release notification is about and why might we be interested in knowing that sort of thing well we might have a manager process that's created the number of workers and put them into a c group and when all those workers finish their work and terminate and the secret becomes empty then the manager process knows the workers have done their work or we could want to do the thing that system d does probably many of you are familiar with the idea that systemd has the facility facility to respawn daemons that accidentally terminate system d is using this feature it puts a process or puts a demon process into a c group and it monitors that c group to see if it becomes empty and if it does become empty then it knows the demon prematurely terminated and it restarts that demon now how does this feature work in secrets version two inside each c group there is a file called c group dot events and this is a file of keys and corresponding values that have state information about the c group at the moment there's only two keys once upon a time there was only one but nowadays there are two and one of the keys is a key called populated and it tells us does this c group or any of its c descendant c groups have member processes if the value in the file is one then somewhere in this sub tree there is at least one process the file has if the key has the value zero then this c group and none of its descendants have member processes and then the idea is that you can monitor these files to find out the cgroup.events files to find out when the keys change and there's a couple of ways that you can do the monitoring of this of these files one is you can use the i notify mechanism and uh which is a sort of a generalized file system notification mechanism i notified lets you things like monitoring a directory to find out was a new file created in that directory was a file modified in the directory was a file renamed and so on so we can use i notify for this job and if one of the keys changes in particular if the populated key changes then inertify gives us a so-called modify event back and then we can react to that event by looking inside the file to see did the populated key change alternatively we can open a file descriptor on one of these files and monitor that file script using select poll or epoll and if the content of the file changes then what we get on the file descriptor is either a priority notification if you're talking about poll or epoll or what's called an in select an exceptional notification once we've got one of those notifications then we just go and pass the cgroup.events file to see what has happened to the populated key and of course it might not be the populated key that has changed it could have been the frozen key for example so when you actually pass the file and check the value of the key to find out has the populated key changed you can have one process that monitors multiple c group dot events files or different processes that monitor different c group dot events files the nice thing here is that this means you can have a different process monitoring different parts of the c group hierarchy the secret.events files and different parts of the hierarchy so that for instance one process might be monitoring all the cgroup.events files associated with a certain container and another process might be monitoring the sql.events files associated with a different container and so on this was functionality that was not possible in secrets version one we talked about the notifications being delegated per per container or per group of processes okay so let's try this out give myself a super user shell of course and i'll make a child c group now if i look inside that child c group in fact i'm going to do this a little bit differently uh now i'm going to remind myself for some syntax here because i often get this wrong well i'll do the simple thing first let's just cat and i'm in the wrong place of course right now that's cat c group events we see certain values there for the keys there is no process inside the c group at the moment now let's create a process and okay now before i do that i'm going to use i notify weight i notify weight is a command line interface to the inodify mechanism i'm going to say i know while i notify white i have to remind myself for the options there i want to say yes q dash e modify and i want to do this repeatedly and i'm going to say this on i want to monitor this see events file and whenever i get a notification what happens the the form of inaudib there is a notification event so every time there's a modify event on that file and let's go around and loop calling i notify wait again and again the dash queue just says be quiet don't actually give me any printed information just return when there is a notification event and whenever i do get that notification event i'm just going to say let's display the populated key in that cgroup.events file and then i'm done no i'm not what did i do wrong oh now okay good now i'm done so now i'm going to create a sleep process and then i'm going to move that sleep process into the c group now when i did that that generated an event because the populated key changed and of course the loop down below when the notification occurred my my loop gripped for the key inside the secret.events file and we just changed the value 1.

[00:09:40]now let's create another sleep process and put its pid in there as well oh that's a bit of a coincidence exactly 100 apart okay that didn't generate a notification by the way okay because the populated key is a boolean key it's telling us that there is at least one process in the hierarchy now suppose i now say kill the first process three four eight one eight five again there's no notification the value of that key of that key in that file just cut that file the value of the key is still one but if i kill the other sleep process now i get a notification okay because in that subtree now there are no more live processes so this is this idea of release notification i think i've covered most of this yeah okay so that's release notification then uh someone overhead a question that i said the answer is delegation i was yes it was yes everything i've been doing so far i did as super user of course that's not so nice if we're thinking about the notion of unprivileged containers containers run by unplugged users because we don't want to have we don't want to have the container need to be privileged in order to use c groups so this is what delegation is about passing ownership of some sub-hierarchy in the version 2 hierarchy to some less privileged user user id 1000 for example need a bit of terminology delegator and delegatee what i mean by the delegator is the privileged user probably super user who owns the parent c group and then the delegatee is the unprivileged to which we're going to pass ownership of some sub-tree underneath that parentheses group now the way that you do delegation is just make the child c group writable by the by the delegatee the the usual way of doing that is to change the ownership of the directory and then the directory is you know writable by the unprivileged user so you change the ownership of the directory that is going to be the root of your new delegated subtree and you have to change the ownership some files inside that directory as well the cgroup.prox file so that the delegatee can write pids into that seagroup.prox file and also the subtree control file because then the delegate t can enable controllers inside the delegate inside the delegated tree you might also need to change the ownership of a few other files and the set of files whose ownership you need to change has evolved over time there's been a few new files that have been added over time [Music] that they also need to be delegated and then your question becomes well i'm using a certain kernel version on that kernel version when i do delegation which files need to be changed which files need to have their own ship changed nowadays the kernel has a little interface for allowing you to discover that information there's a file called or the pseudo file syskernal c group delegate and that file just lists the names of files that you need to change the ownership for during the delegation stamp and so you see there cgroup.prox and cgroup.subtree control but there's a couple other files that might be present in the c group directory and if they are you also need to change their ownership as well there are some files in the directory though whose ownership you don't want to change and these are the resource controller interface files things like pids.net and cpu.max because they are being used from the level above to limit resources that come into this c group and the delegate even less privileged user shouldn't be able to change the limits that are coming in from outside the c group so those files should still remain owned by the delegate t so in terms of a picture this is what the delegation step looks like we've got the parent c group that's owned by the delegator presumably super user then a child c group that we're going to delegate to the delegatee so we change the ownership of that that directory to being the delegatee and then within that directory we change the ownership of the other files that are shown here in blue the ownership of these other files also to be the delegate but the other files inside that cgroup the resource controller interface files they remain owned by the delegator because they are what are being used to control from outside in terms of resources that come into this c group now once we've done the delegation step the delegate t the less privileged user can then create child c groups those child secrets will inherit their ownership from the parent c group so that also be owned by the delegate t and all of the files including the resource controller interface files inside those child c groups will be owned by the delegate t so the delegate can freely modify resource limits in the lower level c groups okay now there are there's a there's a there's a set of rules which are called the delegation containment rules and the idea of the delegation containment rules is that the delegatee is allowed to move processes around within the delegated subtree but can't do things like moving a process from the outside of the hierarchy into the delegated sub tree or vice versa and the rules go like this that the writing pro the the delete the writing process that which is presumably owned by the delegatee in order to move a process from a source c group to a target c group two rules the writing process needs to have right permission on the cgroup.prox file in the target c group and write permission on the c group dot prox file in the common ancestor of the source c group and the target c group now the effect of those rules is that for example the presumption here uh well there's various c groups here that are owned by different user ids the root c group for example owned by uid 0 but then there are other c groups that are owned by other users they've been delegated and the delegation containment rules would let us do things like saying we could move a user id 1000 could move a process from c group m to c group in the reason is uid 1000 has write permission on the c group.props file in the target c group in and also has write permission on the c group.prox file in the common ancestor which is b likewise uid 1000 could move a process from c group m to c group x we need to have right permission on the target c group which is x and when you have right permission on the c group dot proximal in the common ancestor the common ancestor of m and x is x on the other hand user id 1000 couldn't move a process from c group x to c group y okay because doesn't have right permission on the seagroup.prox file in the common ancestor and you know if you think about unprivileged containers owned by the same user that feels good because you wouldn't want to you know a user could set up two different containers and then move a process from one container to the other that would that would be icky okay so let's try some things out maybe i'll just better start again here alrighty so the top window here i'm a privileged user and just change my shell prompt something less noisy i'm going to create a child c group i'll call it d group for delegated group now if i do an ls-ld on that directory okay it's owned by super user but i want to delegate that directory to an unprevious user okay down below i am user id 1000 user mtk so to do the delegation step all i do as the delegator is say change the ownership of that directory to unprevious user and associated group on the group but also on some files inside the group c dot procs and c group dot subtree control those are the only files i i happen to know this what have i done oh yeah hey that's interesting isn't it okay let's let's undo that now let's try and do that there i happen to know that those are the only files in that directory that need have i still messed up oh yeah thank you they're all just here to help me debug um so i happen to know that those are the only files inside that directory that need to be delegated there is no file called secret.threads in that directory and there is no no no files associated with the memory controller either now i can then do a bit of verification where i say ls dash l dn long listing in fact we don't need the end okay so those files have been changed to unprovisional me and i've got right permission on those files now what that means and i'll just down in the shell below i'll just step into that directory what that means is that as i'm privileged user i could now create child c groups underneath that delegated c root so if i say make the group 1 group 2 and then i do an ls ld on those two directories okay they're owned by me and if i look at the files inside one of those directories all the files are owned by the unprivileged user so this means that as the unproven user i could do things like once controllers are enabled i could start setting limits inside those child c groups now let's try to put this shell into that um the delegated group so i'll say echo dollar dollar into [Music] the oh i'm indeed i'm in the yes i'm in that group already yes so pwd echo dollar dollar into c group dot procs failed why is that at a guess i don't have right permission in the common ancestor and you're quite right there's one more piece that you need to add there i think where does that shell live at the moment at a guess which c group yeah it's probably in the root c group let's let's go and check that camp smash proc slash dollar dollar slash c group oh okay no it's not in the root secrets and something that was created by system d of course but the point is the common ancestor is the root c group and i don't have right permission on the c group dot prox file in in the root c group in the common ancestor okay so to get the very first process into the delegated c group that actually needs to be done by the delegator the privileged user so let's from let's find out the pid of that shell okay that's that pid so then i could say here echo that pid three three four eight eight five five zero into d group c group dot products and then if i now grab also cat proc dollar c group okay the only the secret membership has changed now i could from the shell because the shell is owned by use radio 1000 i could now create a process if i look at the that sleep process that i've just created if i look at its c group membership it's in the same c group as its parent because that's how it happens by default and what i could now do as the unproven user is say echo that pid into one of the child c group c group dot prox files and that works i now again cat the prop f secret file i see the membership has changed and i could move it around a bit more moved into other c group the other child c group and again we see the path name has changed what i couldn't do is something like trying to echo that pid into ccfs c group c group dot prox try and move it into the root c group this is delegation containment in action i think i've covered all of that yeah that's that okay this is going to be a short session but that means i can do a better demos demo thread mode okay so well perhaps though before i talk about thread are there any questions so far because we have got a bit of time in hand or you can think of questions at the end yes please are you talking about the rules yeah next one yes ah yes yeah if you try to move from k to m you're allowed to as user id 1000 yeah i i know if it feels a bit strange because you've moved over a c group that was owned by a different user but according to the rules that's allowed and perhaps if you'd like to think of it you know from the point of view of you know the the the tree root of x might be associated with a container owned by user id 1000 and user id 1000 is sort of god in that container so you can do that sort of thing but yeah it is it does feel a bit strange and this this is a slightly unusual setup for you but it's it's it's a theoretically possible setup uh oh so what was i thought you said did you say jada x or k to m k to m yes that's fine but yes that's that's you couldn't move um that's true according to the delegation containment rules you couldn't move a process from j to k okay it is a weird situation could you move from jay to i and i think according to the delegation containment rules the answer is yes yes it's a weird setup um but i'm pretty sure the answer is yes um i can't see any reason why not i've never done that particular experiment but yeah um yeah but i'm sure the answer is yes okay thread mode when cigarettes version 2 was initially um designed there was a one of the design decisions was that the the granularity for c group membership will be at the process level in other words all the threads in a multi-threaded process must be in the same c group version one had allowed something different with version one you could have a multi-threaded process where some threads were in one c group another threads were in a different c group and for some controllers this made sense in particular for the cpu controller people had use cases where you'd have a multi-threaded process where you wanted some threads that were subject to one cpu limit but other threads that should be subjected to a different cpu limit the reason that that wasn't that idea was not initially implemented in secrets version 2 is for some other controllers this made no sense the the most striking example is the memory controller suppose you had a multi-threaded process where you try to put some threads in one memory c group with a certain limit and other threads in a different c group with different memory limits given that threads share in address space what could that situation possibly mean so when secrets version 2 was first implemented the design decision was we won't have that sort of messiness they'll just be thread level granularity for sorry process level granularity for for a c group membership but there were these use cases in version one especially to do with the cpu controller and this led to a long running skirmish between two different groups of kernel developers and the the i said that secrets version two was released in um 2016 the skirmish had already been going on for a year and when secrets version 2 was released there was no cpu controller i i don't know what the sea group version 2 maintainers hoped perhaps they hope the problem would just go away and we'd get a cpu controller according to our idea but the s the kernel scheduler maintainers had other ideas so on the one hand with the secret version two maintainers that said the level of uh for secret membership the granularity for secret membership will be the process level all threads must be in the same c group on the other hand you had the colonel scheduler maintainers who said if you want to merge a cpu controller into our scheduler you're going to satisfy that use case that was possible in secrets version 1 where you could have different threads in different cpuc groups one of the things that might not have helped here too much is that one of the colonel scheduler maintainers hates secrets wishes they'd never appeared anyway eventually there was a compromise found and this compromise was something called thread mode and thread mode allows a limited form of thread level granularity for certain controllers for the certain controllers where it makes sense and how does this work well now c groups have different types we talk about domain c groups and threaded c groups a domain c group is one which has the original rule a process in one in a domain c group all the threads must be in the same c group this was the original version two default but what you can also do within this overall version to hierarchy is create what are called threaded subtrees and in a threaded sub tree and you can have several different threaded subtrees in a threaded subtree you can have the threads of a process split across different c groups there's one rule which is that all the threads in a pr in that process must be in the same threaded sub-tree there can be many threaded sub-trees but all the threads of any particular multi-threaded process must be in the same threaded sub-tree so there is now thread level granularity in version two but it's more restricted than version one and so we've got the idea like this here i've got a version two hierarchy where there is one threaded subtree and there's a few threaded c groups inside that subtree and the idea is that you'd have a multi-threaded process that where the threads were split across the c groups inside that threaded subtree okay now a change that happened at the same time is now controllers were split into two types so-called threaded controllers and domain controllers and the idea of the threaded controllers these are the controllers that support that sensibly support the idea of thread level granularity for secret membership where it's meaningful to split threads apart for the purposes of doing resource management the other controllers which is most of them are the so-called domain controllers and they only understand process granularity for c group membership so these threaded controls they understand threaded sub trees what i mean by they they understand threaded subtrees is that going back to that uh tree sorry i'll i'll do the drawing here on the screen that's a bad idea that going back to this picture here for the threaded controllers controller interface files things like cpu.max will appear in those child c groups on the other hand for the domain controllers the controller interface files won't appear because domain controllers don't understand threaded subtrees for a domain controller it looks like all the all the the processes reside in the root of the tree because domain controllers they can't have their interface files enabled at lower levels in the tree so the lowest level that a domain controller can do management is the root of the threaded subtree okay [Music] now in order to do the um the management here of these c group these threaded c groups there's a couple of new files that appear there's now a file called cgroup.threads if you write a thread id into that file that'll move a single thread into that c group and this is the way that you split the threads of a multi-threaded process into different c groups in the threaded sub tree by using this file and then there's another file cgroup.type and what this file does is expose or allow you to modify the type of a c group c groups to begin with they have type domain domain c group only supports process level granularity for c group membership but there are other types there's a type called threaded and this means this is a c group in one of these threaded sub trees there are other strings that can appear in this file domain threaded this means this is one of these uh this is a c group that is at the root of a threaded sub tree it's sort of that crossover point going back here it's at that crossover point between the domain outer world and the threaded subtree underneath and there's one more c group type that i really don't want to get into it's slightly confusing called domain invalid maybe it's not so much it's confusing i just don't buy the rationale for its existence there is a rationale but it's questionable i would say this domain and valid state this is a c group that is invalid basically it's a temporary state on the way to creating a threaded subtree what you have ultimately with one of these domain and valid secrets is turn it into a threaded c group there are a couple of different ways for creating these threaded subtrees and i don't have slides here to show you but i'll just do the demonstration you can find the details in in the manual page there are quite a few rules about doing this um and again it's a bit more than i have space to cover but i'll just do some demonstrations and yes we're after that we're done now um what i'm going to do is make a little bit of sub hierarchy and then make a threaded subtree so i'll make a director under here a in fact i'll say magda dash p so a b c d and maybe c and sorry c and e and maybe a slash x slash y slash said i think that's enough now one way of making a threaded subtree is i write the string threaded into a cgroup.typefile so i'm going to write that string into the secret dot type file of the c group c now what this does is make that let's cap that file just to verify the change and i i should have cut that file beforehand but if i i can just cat another file just to demonstrate the default that was not the file i meant to cat let's look at the c group a that's the default type for c groups now i made c threaded and okay what that did was cause the parent of c to become domain threaded the slightly weird way to go but it's how it works this caused the parent c secret become domain threaded and all the other descendants of the parentsy group became invalid so if i then said cat a slash b slash d slash c group dot us a slash b slash d c group why am i getting this wrong oh yes a slash b slash d slash c group dot type the other descendants underneath b became domain and valid now if you're trying to do these experiments you quickly start to go a little bit mad cutting files trying to visualize things and it's not that much fun so you want a little go program that i prepared earlier and so i can now say go run oops of course i should have said mtk few v2c groups and i want to view the hierarchy go a slash b and this program gives me a little visualization because that's much easier than cutting files all over the place and so what we can see is that c has type threaded the parent has type domain threaded and the other descendants of the parent have become invalid if i'd done the viewing instead of c group a okay a has typed main under b has domain threaded now the next thing i would do to create a threaded subtree is i've got to channel all those domain and valid c groups into threaded c groups and so i say echo threaded into a slash b c slash d slash c group dot type and the same thing in the c group e and then if i re-visualize now i've got a fully threaded subtree what i could now do is put a process inside that tree now let's make a a process what this post program does is create one thread for each command line argument and each of the threads tries to burn some cpu time so and three threads is enough okay each of those three threads is getting a hundred percent of cpu time because of course i've got a lot of cpus and what i could then do is move that process into the threaded subtree and the way that i do that is i say echo the pid which is three five one one six two into a slash b slash c c group dot procs now if i cap that file something slightly odd happens i get an error the reason is that from the perspective of the threaded sub-tree the process is considered to be a member of the root of the threaded sub-tree in other words it's considered to be a member of a slash b which was the root of the threaded subtree so processes are always considered to be members of the root of a threaded sub tree but individual threads can be members of various c groups within the threaded subtree so for instance if i said cat a slash b slash c c group dot threads there's all the thread ids they are in that descendancy group again you go a little bit mad trying to cat everything here and again my little go program just makes life a little bit easier by showing me where the pids are and where the thread ids are all the program there is doing there is inspecting cgroup.proc files and secret.threads files and displaying the information now what i could then do is things like this where i say echo let's say twenty thousand 100 000 into a slash b c slash d slash cp oh hang on a little problem there i was going to enable cpu limits but of course what i haven't done is enable the cpu controller all the way down at the very end the various c groups so let's just quickly do a little bit of first of all enable cpu in the subtree control file at the root c group then in a and b and c i really like my little go program because it gives me visualizations all these things which would otherwise i would go crazy because it's telling me the cpu control is enabled now in those c groups so then i want to do things like saying echo some quota and period into a slash b slash c d slash cpu dot max just by the way little test i enabled the cpu controller in a and b and c but i set a limit in d why didn't i need to enable the controller in d no button it's always the level above that it's controlling the next level down if i want to do control and d i have to enable the controller at the level above okay i've set a certain cpu limit there and let's set one an e as well and then i can start moving threads into these c groups so i could then say echo say three five one one six two into a slash b slash c slash d slash c group dot threads and of course i meant to use a different number here let's make the limit in e 20 percent and what you can see down below by the way if i just freeze the output there for a moment you can see there that this thread is only getting 50 of the cpu now 50 of us cpu and i've set a different limit in c group e so let's move a different thread in there [Music] 6 3 and again you can see there it's converging on a lower value now oh i measured wrong c group didn't i i mean to move it into e okay okay we're at steady state there now you can see that this thread is getting 20 percent this thread is getting 50 of a cpu the other thread is getting 100 of a cpu because it's it's not limited by anything at the moment okay now if i put two threads into the same c group let's put the third thread also into c group e so i put just resume that output if i put three one three five one six six four into e as well it a moment it takes going to take me you know 10 seconds or so for it to stabilize now okay what you can see that the two threads that are in c group e are sharing the 20 percent that is available to that c group and so they're each getting 10 of the cpu okay that is essentially thread mode i think i'm done and there's a few minutes left in case there are some questions martin okay so it's a nice question the pids controller is misnamed the pids controller you know it's not limiting pids it's limiting tasks it's limiting threads individual threads count against the limit so if you took a multithreaded process and put some threads in one pid c group and other threads in another pidc group then it would limit those the the ability of each of those threads to create further threads or further processes yeah that's why it's a threaded controller despite the name other questions ah yes i'm jim there is a story um there is look i i am just the messenger okay the the to me the question is why do we need the main invalid no mode at all why couldn't you just write domain threaded into a certain c group and that makes everything underneath threaded that would be so much simpler wouldn't it yeah but it's not the world we have and this is because at a a certain point there was a discussion that there might be possibility for something called mixed mode where you might have a threaded subtree but then underneath there might be some domain subtree now what this could mean was never fully elaborated but then the decision was made well let's allow for this in the future we don't know quite what it's going to look like but we're going to allow for it and what this means that you know we don't necessarily want to make the the decision up front what should be the type of one of these c groups underneath the thread sub tree i don't buy it because you know if we need to do that thing differently then perhaps we could have had a world we just wrote domain threaded into a c group dot type file and it did you know made everything threaded underneath and if you need to do something else different in the future you write some different string into these files but it's what we have that's right please question no so underneath the domain threaded c group the only kinds of c groups you'll find are either threaded or domain yeah so if you write threaded into ac file that makes the parent domain threaded and everything underneath that domain thread of ancestor domain and value everything else domain and valid it's it's weird but it's the way it works no the only string you can write to these files is the string threaded in other words the only explicit transition that you can make is to convert a c group to type threaded now there are implicit transitions that can happen though and i'll just quickly illustrate that that let's do that visualization again the c group there x has type domain and so what if i make the type of y threaded so i'll say echo threaded into a slash x slash y slash c group dot type and visualize parents become domain threaded the sibling has become domain invalid but what if i do this r m der a slash x slash y now the reason that x is threaded the main thread i should say is because it has a threaded child what if i remove that threaded child i can do it and then what happened to x it reverted to type domain and its child reverted to type domain as well so there are these kinds of implicit transitions that can happen if the ancestor is no longer forced to being domain threaded there is another way that you can create a threaded sub tree but i i won't try and talk about that here but this but basically there's two ways you can create a threaded subtree but if the the thing that triggers the creation of the threaded sub tree is undone then the ancestor the the domain threaded root reverts to being type domain in this scenario yes but in the other scenario it looks a little bit different but even so if you reverse the steps then the the the conversion to domain thread is undone joking yes sorry if you have the main thread history group could you ah yes you could so let's let's let's say make so um well here's our visualization at the moment we've got a and x and z let's make z threaded into a slash ink slash z slash c group dot type now that of course turned x into domain threaded now what if we then wrote threaded into a slash x that's what you're talking about isn't it [Music] now i'm confused oh no i think that's because of something else that's going on which is i've got the main controllers enabled at certain levels above yeah so that you can actually do this sort of thing and if i didn't have other domain controllers enabled because there are some rules that i haven't talked about um but you can do that sort of thing where you write threaded into a secret that is currently remain threaded and that causes the parent to become a domain threaded and that domain threaded child becomes threaded now and the other descents become domain invalid uh no if a c group if a sibling is already threaded that stays threaded [Music] i know i think it should become threaded as well yeah and there's something else let's do with the controllers i think i'd have to look more closely i've broken another rule that i haven't explained yes please oh that's that's the rule i've broken thank you i've just realized it um that you if i do the visualization thank you for that question the rule that i've broken is you can't turn a particular node into domain threaded if there are descendant c groups that have member processes already and i was trying to turn a into the main threaded but already underneath a there is c group b that has a member process that's the rule i broke um i wasn't thinking quite quickly enough um but you know to to finally do your question your iq in a different way let's just say make the p slash q slash r oh you make dash p and then do the visualization on p okay and then if i make echo threaded into p slash q slash r slash c group dot type do the visualization the parents become remain threaded but then if i do that on q and visualize again let's turn the bit yeah other questions i think we're about at about time okay thank you for your time [Applause]