Mastering Git's Index: Cache and Staging Area Explained

[00:00:09]right so thank you very much for my patience my first tip of the day is to never accept a meeting for just before your talk I'm now going to demonstrate how to use a clicker yep it works awesome excellent right I'm going to take a deep breath my name is Charles Bailey I work at Bloomberg I mean developer experience I try and help other teams and developers with their work clothes problems issues particularly when it comes to get I've occasionally contributed to get still trying to get some sort of like a proper C commit to get rather than just test scripts and shell scripts at the moment yeah Shell's good that's where it's at hang on yeah so today I'm going to talk about the index which is kind of less understood than it should be and I'll give you sort of pew ideas of how to work with it you can take some of these ideas try them out see if they can kind of improve your workflow when working with git so what is good as anybody here not kind of vaguely familiar with git at all they've really done some sort of a git clone at some point every wait yeah yeah most people like okay cool this talk does assume some knowledge of get so it gets a distributed version control system it's like the model that some pretty much taken over the world now some people still use centralized systems like subversion from CVS luckily not so much CVS anymore but the big advantage of a distributed system is that kind of all modes in the tree of people working on the same repository the kind of equivalent every can make a commit known of their choice and communicate it in a safe manner with the other people they're working with most popular are obviously in open-source tools but also used kind of emotionally and kind of closed environments two attributes of get there are two kind of main ones that people talk about like doctor amount I'm wondering although getting a shine from a projector or not and the first one which is kind of quite remarkable for something that's so popular is that it's hard to learn and and another attribute is that it has an index and to my knowledge no other source control system has an index in quite the same way that it has a bit of a mystery word though so so interesting some people who have kind of done research on this and so I don't know if anybody's kind of heard of the tool called gateless it's a research project these two guys kind of came up with when they were doing some research on on software design and in particular they had some criticisms of git and some suggestions for alternatives and one of the things that they did is they they came up with a new sort control tool which basically sits on top of git but gives supposedly and they have some research to back this up an easier to use interface and one of the things that they did was they completely hid the index away so it's no longer a user facing artifact and I found that very interesting and also slightly disappointing because I really liked the index and I use it and I feel it makes my life easier when I use it in the right way anyway yeah I recommend having a look at these papers if you're interested in it and certainly you can have a look at Atlas it's they they don't claim you know it's a a tool ready for production it is very much a kind of exercising here's how non alternative could be made cool so I guess if I'm going to talk about it it's the good to know what it is so just out of interest who here would hypothetically feel comfortable say explaining to the person sitting next to them or the end of their row what the index is okay of all kind of some not so sure I hope to answer this question and perhaps in a way that will make you all feel comfortable doing that sort of thing so I think to explain this them it's probably good to just have a quick review of its object model who here feels familiar with its object model kind of more than would feel comfortable explaining the index which is kind of what I suspected so there are some good resources for this online people have come up with diagrams pictures I just got word explained I'm quite lazy there are basically four types of object in its object model there's the blob and what the blob represents is the contents of a file it's not a file itself it doesn't have a name it's just some stream of binary data now that binary data could be text as well so yeah whenever I talk about this I always think of text is a subset of binary some people think of them alternate binary is just a stream of bytes some streams of bytes form text streams so typically people manage text in and get some people saw things that aren't texting it depending on what exactly you're using that can or can not be a good idea so you've got a block it's just a stream of data and then the next object is a tree and a tree is kind of an analogue of a directory a tree is just a list of names or associates names with blobs doesn't a blob can appear under different name either the same tree or on different trees but we have name also mode so whether something's executable or not and the content node also contains information about whether a blob is a real file or a symlink or a get link but that's not important for this talk so trees can also link not only to blob also to other trees so you can have directory hierarchies represented so you can so basically when you check out a particular gift that tree can be represented on the file system the next object is a commit and a commit is basically just a link to a tree with some metadata about the commit so the author the change message and perhaps one or more they're all more apparent perhaps one or more that's the same as zero more isn't it sorry maybe I should in certainty pop-up optional vector of no so a commit is just a reference to a single tree and that's the complete tree that this commit whole and a commit holds a snapshot a complete tree a commit itself doesn't directly define differences you infer the differences that are commit introduces by looking at any parents that it has and comparing the tree of those parents with the tree in this commit and that's it's kind of quite important point it's kind of quite easy to just kind of like totally miss that because whenever we ask it to show us a commit the default thing it does it goes it's a commit oh and it has a parent I'll show you a disk because typically that's what we want to see and that's some kind of most useful thing but fundamentally a get just a commit is just a link to a tree and finally there's a tag an attack can let you tag any other sort of get object with some description most people tag commits some people tag blobs crazy people tag other tags I've tagged a tree but just just to prove to myself it works like I don't know of any youth behind that cool Sasha my place is why I still haven't answered the owner what so what is the index so the index is at its simplest a flat list of pub and the associated blob ID to that path and the index is associated very much with a working tree so if you've heard of a bear repository one that has no working tree it also has no index oh and is I don't know if you're familiar with them the get work tree command get worked really do to associate a second third working tree with the same git repository for each working tree you get a new index so it's very much one working P one index it's not directly linked to your repository necessarily but the common case single work for you for repository you have an index so it links a name to a particular object ID and if you're in a clean working tree state that object ID is the actual ID of the file that you have in your working tree but often as we're working and we're editing files or if we've kind of removed some files or changed stuff they may or may not be an actual file at that pass and it may or may not be up to date so that object ID is only up-to-date when you've added that work tree to your index or all checked out from it looks done some operation to ensure they're the same so why is it called the index I think I may have answered that sort of reiterate that it's an index that map's a path name to a blob ID and it's as simple as that so then we know any other names for the index yes cash stage I like staging area just because stage is what I'm not standing on but yeah staging area and them that's a much more modern name it's much more kind of popular I think it's more friendly for for new users I think it's generally a good good name but of some some caveats with with the kind of mental model there cool good yep those are those ones so the brief history of Get Well a snapshot of history and time you can go back and check out the original get version and and you'll be given these dis bet of commands and the interesting thing to note so take this with a bit of a pinch of salt obviously the first version for which there is a commit has some sort of a commit command otherwise we wouldn't have that commit so you know they'll have been versions where where where we've been hashing objects befallen us made the first commit that is still in the history but one of the interesting things to note is that of the command six of predominantly to do with manipulating the index or cache pretty much as it was there so you got in it DB which is kind of not really commit tree doesn't really touch the index tap file I can't remember if the original version it looked at the index maybe not so five or six the other five all strong strongly interact with the index and many of these commands are kind of unchanged or barely changed in names so cat file commit ree-ree tree right tree all existing can't get in it VB is just in it now although I think you can still use in it DB and update cache is now update index not quite sure what read cache is I'm going to have a think about that I think it might be just LS files now all the closest equivalent so so obvious from here the index was called the cache from a very early stage and it was kind of core even in the original version of get oops sorry I only employ I'm gonna do that names like didn't so similar to my discussion about commit an index is a complete list of files and again this is something that you have to watch for especially when you come to using the word staging area a lot of times when people say so what changes have you got staged they're just interested in you know differences whereas what's in the index or staging area is everything including stuff that hasn't changed it's a complete index so so when you do get status you and you see the stuff that's changed but in the index everything in the in the project all the stuff that hasn't changed is listed cool yeah I'm I'm on time I'm back on time so I'm what I'm going to do for the main body of this talk is just go through a few scenarios few tools that I've used and a few points to watch for cool so index tools so bearing in mind the fact that the index is like a complete list even of stuff that's not changed so anybody know what the command to unstaged something is or are there multiple commands does it does it depend on the scenario so really it's not it's not a trick question we said yeah yeah yeah so interestingly the sometimes if you look at git status and you're like wow if your haven't made your initial commit yet or the files brand-new or stuff people can go are expens what you're doing but actually reset pretty much works in every instance as an unstaged command so if you've got a file in that you've just added you haven't yet committed sure you can do guitar n minus 1 is cache which is you know something you'd use to stage a deletion of a file even if it was already cached just like inter on stage reset works in in pretty much every scenario I know do tell me if I've missed something there I did have a good poke around the one thing that did some hello okay okay yep thrown up yeah I wasn't going to talk about sub-modules I don't use them a lot I have watched other people use them if there are a couple of scenarios of the useful okay yeah no fair enough okay I shall rephrase that reset works on stage things in most scenarios where you've either stage and add remove or change in a single project without sub-module cool so one thing that's quite useful if you if you kind of want to understand the indexes to be able to just list out what is in your index thinking I think I mention it for something you know familiar with that LS file an LS 1 minus F gives you a list of the not only the pars but the idæan mode as well so it's the it's the closest representation to those like the user facing aspects of the index there's another one that's quite useful and I'll come back to this at the end if you're interested in in some of the more details working of the index and that's - one of debug so rewinding a little bit to the get list slide one of the points that was made in their first paper is that for good software design any kind of a user visible I forget the weather use I'm just going to use artifact and the user visible artifact should have a single purpose that helps a user solve a particular problem and one of the issues we've gets in that is it has multiple roles the user facing role is typically to prepare my next commit so to get everything organized so that when I commit I have the thing that I want but it also has an important term performance role so some version control tool give you a working tree where everything is basically read-only and require ask you to make some sort of positive action to start editing a file so I've used perforce before which can operate in this mode I don't know if it's the only mode it's an operating by certainly use it in environment where it was in this mode certainly you know RCS which I've also used worryingly recently in one instance again you you co-ceo - l you lock a file and then your edit it and then you can check it in you shouldn't normally try and edit it before you've locked it if you're not going to do that as a version control system then you have to start and then you're asked to form some operation such as make a commit or tell me what's changed you're going to have to inspect the file system to see what's actually change or what's actually ready and but you know that can back and very inexpensive things like the number of directions you have and how many files in each directory whether you put you're working for you on NFS or some other sort of slow file system and so to make this as optimal as possible what get index is is it's also a cache of set information so this kind of brings me back to a the other name for git and I'm not 100% sure but I'm fairly sure that the the caching the stat information gave it its name of the cache although I also you know we also use cache in in in other sort of mechanisms so you know I work on a file it's a good enough state to kind of store away for the moment while I while I do some other stuff so you might think I'm just caching that version of the file which is kind of like a second meaning of cache but I my feeling is just looking at the code is that the the caching is the stat information is is where the name originally comes from and what this enables get to do is if you ask whether a file has changed if it's either written that file or read that file at some point because you wanted to add it to the index what it does is it it stores some stat information so the file system metadata of the version that it knows on disk at that time match with what was in index so last week I reset hard get over router a changed file once it's finished writing that file it stores the stat information in the index this week if I ask it what changed it can have a look at the stat information for that file and go well that matches what's in my in the index so I know it hasn't changed I don't need to read that file I don't need to hash that file to determine that and that's a that's a big performance boost and I actually have got a not a live demonstration but a recorded demonstration just to kind of show that later cool water on my tables now okay oh that's very kind it's now water on my table yes I know it's not I can answer that question later on that's the majority of it but there's other stuff too not much but some pretty crucial other stuff cool and yeah do ask me anything at any point they'll probably be time for questions at the end but but yet do you ask me I will get back to that towards the end cool ok that's good so so my first my first 2 3 scenarios that I happen to have done recently is leaving stuff around between projects and also leaving stuff from one project to another now the various ways you can do this I mean you can kind of like at a high level kind of pull the history of a second repository and kind of like to merge it in and then to kind of get moving stuff but I've had a couple of scenarios I just want to take a you know I've had a demo repository that I haven't really taken much care on our packs around and suddenly now I've got a subdirectory useful stuff that I actually want to bring into an another repository and you can do this kind of quite simply with index command so why might you want to use this rather than the kind of get moves stuff that I've talked about speed it's kind of like faster so kind of big trees very occasionally I do something for another team and they want stuff shifted around and I don't care about their stuff and I don't even want to compile it and they've got this like whacking great huge directory I can clone their git repository and not check it out I can do some manipulation and make a commit for them and push it to a pull request and I don't need like those 100 megabytes of sorts on my file and then it kind of goes off to our github Enterprise install some CI runs get green tick because when got it right and it's kind of really fast and I'm back to my whatever I was interrupted from so yeah it's quite the safe space on my hard disk the other thing is you might want to do some manipulation as part of a filter branch so everybody's familiar filter branch okay so then be heard of the BFG tool for eliminating stuff from bad stuff and for the tree no okay oh yeah okay a couple of people so occasionally you need to do some serious history rewriting and usually this is because somebody's got some done something bad at some point and then you realize how bad it is later on so you know they get added their whole working fee for the initial commit and it contains lots of object files and executables and like half-yearly to somebody's go to no really I'm fed up cuz with a fee I now clones from clean every time it's pulling down everything please make it stop or and that's kind of like you can solve that in different ways of a shallow clone but hey who wants to do that the other more serious instance is like is when your passwords file contains the real genuine passwords that you wrote onto disk for whatever reason it didn't means go in there then you kind of yeah you really need you should rotate the password anyway but you probably really want to expunge that from from any record of history so if you're doing some sort of big manipulation and sometimes you know you want to pull out a subdirectory as a new project and actually can use subtree for this but that's a conflict tool so it's not always available and if you want to do this as part of the script you probably want it to be quite fast automated filter branch has got a number of options one of the options is it checks out the tree for you runs a snippet that you provide to do some manipulations on that tree and then take that new tree as the new commit and that's a little bit slow in fact it's quite a lot slow time if you can do your manipulation directly in terms of the index without needing a working tree you can use another option to filter branch which is an index filter and it tends to be substantially faster and when I say substantially it depends on the cost of checking out each each thing but 10 to 20 times come out easily cool so one of the first thing when you're doing manipulations is sometimes you want to clear the index just make sure I'm looking at I've got the next slide and this slide and sometimes I'm kind of like I think you can see something that you can't sit so yeah one of the common operations is sometimes you want to start with a blank index and then use the kind of some subset of the existing tree and pull it into the index and I changed AI only finds out this command kind of quite recently so get read tree it's kind of like the the archaic version of reset or if you've heard the difference between gets plumbing and get porcelain commands it's the plumbing behind reset or at least one of the meanings of reset did I tell you about that paper about the design of git yes there are some other criticisms so what Reed tree typically does is it takes a tree object from a git repository so it's in my object database and populates the index with it and for ages I didn't know really also take someone in front of empty arguments and what that does is it just creates an empty index so it's like get RM - - cash - off so recursively remove everything from the cache but some reason it's a lot faster and not quite sure why it's a lot faster it don't see a lot faster than if you're also removing stuff on disk but it's yeah it's just faster so this one's quite useful and the other thing I know on here is sometimes sometimes it's useful to make a second temporary index while you're fiddling with things and then you may want to go back to the original index you can do this just by setting the environment variable get index file to something else so yes if you're writing scripts for filter branch of the quick aside do not assume that the index is dot git slash index which is the normal maybe I should put a slide with where is the index so normally in a git repository the index is in docket and it's called index if you're running filter branch normally get a temporary index with a different name and you can use the same trick in some some scripts if you kind of want to do this sort of thing cool so yes I just mentioned this if you want to populate an index of a picture tree then get retreive will do this for you it's it's like reset without options except it won't move your head if you follow what I'm saying so there's no reset doesn't leave your head check out move your head doesn't it so it's very low level it doesn't it doesn't look at any of your files on disk it purely reads that tree into the index and one of the things you might notice is that if you do use read tree to read something into the index news read tree to then go back again if you then do a get status it can be quite slow and what the issue is there is that it hasn't looked at anything on disk so you've lost that fat cache if you use minus m which means merge if you use it with a single tree it'll try and keep the stat information up-to-date as much as possible so so that kind of problem is reduced a little bit and obviously if you move too far away and then back again when you move far away and the object IDs are different then it'll it'll zero that information because it knows that your objects used to match the old index your index now contains different objects so obviously those that that information is no longer the stat information of something that matches the index cool yeah I said that was reset read tree always clear the index and read the new tree into it except if you use - - prefix so this is kind of quite useful for kind of bringing things into a subdirectory port thing is you write everybody's awake again great so Jim we treat a lot of life the important thing to remember here is a please remember the flash the documentation says you shall have a flash at the end don't seem to work without the flash and it does B well if you're thinking in the right bizarre way it does the expected thing of check everything else out with a prefix instead of like as a subdirectory so yes is that a good idea so if you put prefix of Castle a you'll have heard like a dot get eaten or or something like that so that it's not documented to work like that it kind of obviously does one interesting thing is it's like if you leave off the prefix entirely or just put it as a single flash then it's like read tree without any argument except the important side effect if it doesn't clear the existing index so you can actually kind of merge multiple trees in at the root level with retreat' but it's not a true merge because it'll just it'll just replace I believe it replaces stuff so normally you only want to do this if you if you've got you know you've already get kind of sub sub directories that aren't going to conflict cool so with with - - prefix I can merge in things as a sub directory or I can bring two trees together so so obviously with prefix I can move things down a level so I think I have actually already hinted at this what if I had a subdirectory and I wanted to pull it up to the top level supposing I had you know in my repository there was always a document a documentation folder I want to press new repository which just starts with the contents of documentation any ideas how that one might be done that's what prefix of box I don't believe that will work but I haven't tried it yeah I think just thinking he certainly shouldn't be able to put anything outside of the repository root M so the hint is like read tree takes a tree edge so you don't have to provide a commit to read tree which would automatically be dereference into the root tree you can actually pull out a subdirectory so if I do them for exact if I refer if I wanted to the tree ID of a documentation folder I could do heads curl on documentation and then I have the ID of the tree subdirectory in documentation folder and if I pass that to read tree I'll get a new index which just contains everything that was in documentation kind of at the top level cool that's kind of a moving stuff around so one scenario that I use this relatively recently ism team I knew is kind of decided to split out their repositories so that we could have a library composite ori which sounded like a good idea because what you can kind of build them separately you don't complete when you're changing the same thing however in practice them all the live is released together a lots of stuff was I went I won't call it tightly coupled but sir-- reasonably coupled so they had a lot of changes which spans repositories and they're having a lot of pain with this and and they decided that actually they've made a mistake and what they really wanted is everything back into I don't like to use the word mono repo but that's pretty much what it what it was for their for their set of libraries so by using so on the individual repositories I can create a new repository I can fetch from all the other repositories just I've got the commits available and then I can just use get read tree with prefix to pull them on separate libraries and just make a single commit and what I actually did was I created a a big octopus merge for them that brought them all together retain their histories but had them all now in separate sub direction of the same repository and yeah they find the new mono repo and infinitely better to work with cool using bits of my clicker not my clicker obviously right so ignoring stuff um there are a couple of scenarios where you have staff entry that at various points in the development process you don't want to be normally included in a sort of commit action or a research action so one of one of the things that people often do is they have some sort of a config file and maybe there's a connection to a service maybe there's a URL maybe there's some credential and all of people do to this in various ways I have I have my preferences but a lot of time they'll put in some sample stuff and commented out or be password and it says you know modify and put your password here and then you can kind of test on your your client machine and that's kind of a little bit annoying because it's like you want that time in source control but but but generally people are going to edit it individually and maybe it's going to be kind of different when you deploy kind of nice to have a better configuration management but it's better or worse but especially when people are starting a new project it often kind of tends to be like this and historically the tool for this was assumed unchanged I'm out slightly unfortunate because assume run changed really had a different purpose a team unchanged you kind of puffs purpose and motivating examples like I've got a really big tree and I'm only working in a small part of it so actually I don't want get with him when I'm kind of like you know asking it what's changed ourselves just descending into all these other places that I don't really care about and so you know this this flag was kind of used and recommendation was you can just set this flag on tons of stuff and then it will just literally assume it's unchanged and that kind of so for the config file scenario talking about and kind of works because when you edit the file and put your own password in or some test URL or whatever it is that's different that shouldn't be it shouldn't be checked in if you did a commit - a or a get add - you or whatever your favorite way of just like adding everything in the tree was whenever get came to look at your config file it would see the assumed unchanged bit and go I'll assume the one on disk is the same as the one in the index and I'll just use that and instead so you've never you'd never get your personal version added to the repository unfortunately it also works the other way around so if you were to pull merge rebate check out a new commit basically that had an updated version of that file when get came to check out all of the changed files that you don't have that are correct it will go through and it would find this conflict on it will say ah assume it's unchanged on disk so this file hasn't changed that there's a new version and just flat-out overwrite it even though you might have had put some kind of like important stuff in there and so that's kind of not ideal so the better thing to use is for this scenario is the skip work tree bit and this pretty much works in a very similar way the differences it will never touch work for you for this file I mean pretty much what it says so in this scenario it just never reads or writes the file kind of at all until you remove the work tree a skip work three bit if you actually wanted to kind of do some changes to it I kind of prefer not having that sort of config file directly in in source control my general recommendation is if you have you know some config that is kind of global for all developers the project have that in a config file and then try and have a file that can overwrite it so you know conflict local or something that people can create and what you might want to do is I have a something like a conflict local template or something that people can copy from and modify and then put the past that people are supposed to create and then manage themselves just put that into the git ignore entry in the repository and that tends to be much cleaner and much less overhead for all your developers than using something like versus skip work three bit cool any questions on on that the differences yes yeah yes yeah so I should have put a path on the end of this file so you can put multiple paths it's an index bit so like a few manner changed it's stored in the index so it's directly related to the working tree and its associated index that you're running this command for so yeah yeah both of those who have got a mind of - no - get work no get no skip work for you or no as do one changed which is not perfume changed you know from the keynote will know about our double negative maybe you should ever - - maybe changed perhaps cool so one of the problems would skip working though is that as I said it's set on the index and it's also therefore associated with a file path so typically what you might do is you might go oh this is big mono repo of the kind that I've described before and I'm only interested in Lib one so I'll just kind of like get LS files grep everything that isn't live one and pass sets are like exag skip work tree so there's another kind of really common thing so what happens sometimes later is there somebody adds some new files to live 29 of course because of skip work tree is associated with particular pars because it's a flat list of paths it doesn't contain trees like a commit those new paths won't have skip work tree bit set so now you'll have Lib one and like new stuff added to Lib 29 just sent out out of the way which it's definitely not ideal and pretty much totally unexpected for most people so there's actually a feature of git which is the path check out then we use stark check out thanks putting my hand down to I haven't really used it in anger I I have used it I use it in preparing my slides a plate does actually work so if you set the call sparse checkout config option for a repository just thinking so this will apply to all work trees if you have multiple work trees and a publicly there's something I didn't try out you give up the right to manage your own skip worked three bits which is probably a good thing because as I've described it too it's a pretty manual process and it goes wrong and instead what you can do is you can put in patents into the sparse checkout file and this is the same sort of patterns as you may use or see in git ignore file and what this says is only things that match this do I really want to be checked out and then once you've enabled it fully in an additive stuff your config get will start managing the skip work three bit for you so every time you move to a new commit that's got some new paths that that you're not in your files check out they'll be marks get work for you automatically without and you need to do that yourself if you try and manually change the skip work three bit those changes will probably be lost when get to some tree operation that that affects those paths so so it's kind of important if you kind of doing spells check out that you don't you don't feel you don't expect to be able to to do any manual skip work through this yes sorry say what's up so when you say you add it so the fast checkout is like a positive file it's a list of things you do want to check out right yeah yeah so it's typically typically when you do a sparse check out it's generally because you're about to kind of remove all of these from on disk because at the you know at the point you add them to files check out um so if you if you if you cloned without a checkout option first and then set out your files check out and then you did okay we set hard head or get checkout dot then you'll only get the stuff that's included in the check out stuff so if you wanted to it is true that if you just added those paths off you have an initial cut everything basically be frozen in time for from whatever state you're at if you want to get to kind of like be able to restore something you'd have to specify you'd have to add it to your files checkout list as an explicit path to get that version back out but typically you add something to a sparse checkout because you want to actually remove everything that isn't there from your working tree because it's too big it's in the way it's not being kept up-to-date anyway I think there's a if you if you look at read tree there is a explicit option where it can be ignores files check out but yeah you can ask me anything I won't necessarily have the answer cool so yes potentially useful in in wanna repose but to me it's a bit of a smell that actually at this point you probably could split up the repository so that I mean the team that I was working with did actually have kind of builds that could happen separately and you know if you've got some sort of top-level building that goes into everything then sparse checkout is probably going to not not work out for you and if files check out does work out it's almost like stuff could live in different repositories at that point without much penalty cool what what time did I sketch I was a little bit late so I started after two weeks we - 3:30 aren't we yes awesome we're on time please surprise me I couldn't believe it awesome so performance considerations so so this is actually going back to a question of ours early about what is in the set cache what is in the fat cache is actually as well as a modification time mode file size and creation time it also stills the device number and inode number if applicable on your operating system I don't know what it does on Windows now I didn't know what it does in Windows in some scenarios so what that what that means is if you if you were to say are think a git repository with it get directories from one place to another with the archive flag so you're preserving as many details as you think you can you still got brand-new inode forage devices the same unless you've kind of synced it on two different file systems but you have brand-new inodes so the set information no longer matches and nici do a compromise comparison of something simple like supposing you had a clean tree and you just did get status and maybe you redirect the output to DES now just for you know to keep it quiet you'll find it's a significant performance difference the one that was the source of the arcing cooperation we'll be really fast and the one that was the target will only just be soft however you do it second time it'll be really fast because the operation of running git status it will have to inspect the stuff where the stat cache doesn't matter and if it finds that although that information didn't match the file hadn't changed it will update the index so that aren't your question about thee yes okay so you'd have to yes you can trick it it would have to be yet healthy the same same inode so you couldn't kind of like you'd be happy careful if you wanted to kind of like some of the in place Ted commands are probably kind of like write a new one so yeah it's possible to pull it but yeah it has to be deliberate act so here's a here's a demonstration of them I claimed the gate repository because I just love write recursive stuff and kind of when I want a reasonable-sized stuff though something that's not as big as an internal but still pretty big quite a lot of history often use the git repository so I claim the gate repository it's at some version like two point thirteen release candidate or its 213 out I don't know and if you do a git reach read tree head what it will do is it'll read the pit commit the tree as a tip commit into the index which is kind of like a no all because I have a clean checkout people see here okay I have a confession to make this was run on a bun to on the Linux four windows subsystem I wanted to have something with you know a bit of performance issues on my on my new surface pro that I'm playing with and it took about two seconds of real time and then if you run git diff again it takes like 0.1 second because once you run git diff like many commands if it finds that the set caches is is not up to date but those are no changes file it updates the set cache and it matters it makes a huge difference Setzer what nearly 16 thirds 20 times softer ish roughly hand-wavy there are only a few times and the particular times didn't change very much I've done it a few times even before I got the year thirty seconds the sponsor everything was reasonably kind of hot from an operating system cache point of view so the clown's lots of forcing demands kind of update that information if you're doing some scripting and you want to do it kind of explicitly the command is get update index refresh the slightly weird thing about this and I don't know if this particular version of gif that I haven't really investigated is if you went out the time that I took to explicitly refresh the index and then to add if it still adds up to way less time than just doing the diff that had to update this set cache and this was fairly repeatable as well you know I tried it a few times and went back to doing to get diff itself git diff is if your stat cache is not up-to-date is always less efficient than doing an explicit refresh index and I get dis there's probably a patch in there somewhere isn't there some optimization that could be done cool if you did so here's my demonstration of impact so what the I minus M flag does is if you read a tree into the index if any file blob path and contents haven't changed between what was in your index previously and what you're putting in your index and the stat information was up to date fields you're working trees clean basically we'll copy that's that information so so this takes to read to you a little bit longer because it's you know looking the old index and copying some stuff across but not I mean it's not it's not noticeable the read tree itself is incredibly soft operation it wasn't even worth timing it and then get this again it is kind of really really far so yeah that's kind of just recapping what I just demonstrated I think cool so yeah what might invalidate the stat information as I mentioned before like an R think would you touch the file you know all the things you kind of expect to most of the user level commands from gits preserve the state information if if they're not actually changing from disk even if they're doing other stuff backup and restore and normally it's just not an issue because the first kind of user level command will fix everything there are some some gotchas for scripts if you're using using gif files instead of diff it's kind of a low-level thing what you sometimes notice is the stat cache isn't up to date it might print that a file has changed and then and if you're using an option that says them what are the changes it'll build in empty set of changes we just kind of probably not what you're expecting yes they have stuff like that cool so I guess it's time for kind of a quick story I had an incident where we had a program that was indexing over a lot of checked out git repositories and it was doing some incremental indexing so doing indexing by a command keyword and symbol ago tons and tons of source files and we moved from really kind of old way of doing this to kind of more up-to-date version where we kind of keep repositories checked out I can't actually member what it was doing before we're keeping positives checked out and doing a fetch resetting hard to the default branch because we had to allow for people who might rewind their default branch kind of not recommended some people do so we didn't want any kind of merges to be happening in our automated script and it turns out the new way of doing things was like much slower than people expected it was taking forever and we also guys in charge and search ability so you know what what's up forget Pro so it seems to be kind of like fine fancy clothes checking stuff out and I said well AI gets rubbish it's updating the time stamp even for stuff that hasn't changed it's because you using reset instead of like merge which was like fine for us and we only sometimes had to manually go in and fix stuff when people are done stuff and I was going that's not like I you know I I know gets written reasonably well if stuff hasn't changed it doesn't touch it on disk after after a while of debugging it turned out that they're using our script to update stuff but overnight they'd kind of like him for safety take a copy of all of the gate repositories into a second area except what they would do is they would move the entire file system tree somewhere else and then our think it back to where it was originally so when it comes to me it came to do through the get reset we were just doing a git fetch and then we're doing a reset - - hard gate will look at the index and it will look at the set of the cache set information and then it will see if it needs to actually replace stuff on disk but it doesn't bother to go and look and see if the files change so if that information is not up-to-date rather than going oh well I'll read on this to see if it has changed and then because I'm doing a gig reset hard just write straight right over it if it has changed if the stat information is it just writes it anyway because you just ask this to overwrite the file there's no point if the stat information is telling it it might have changed there's no point to reading it see if it's actually changed because what it's going to do is write over it anyway so it's not you know doesn't it it does no no benefit well almost no benefit sir so it's actually reading stuff if it's just going to overwrite it anyway so yeah once we find out what the actual process was we added a get update index refresh and suddenly indexing those cheats again and yeah I've lived happily ever after since and so covered a bit about the index some of the ways you can manipulate it and hopefully does so yeah the index it's kind of cool to get it's kind of central to the way it works being there since the start it's a it's very very important to get performance lots of operations would be annoying you slow so and another one is of course get grep which is it's very very fast often faster than a recursive grep some people have told me about tools that are even faster for that sort of stuff but I haven't investigated them without the index it would be kind of as low as great- are and understanding it and being able to manipulate it is kind of useful in in in a number scenarios and I hope you'll be able to kind of take away some of the stuff that I've talked about maybe have a play with it learn how stuff works and then maybe if you're kind of end up in scenarios like I find myself use some of the stuff to improve the way you do things so thank you for attention I was going about 220 times over for questions you