Poverty Traps & Graduation Programs | Development Economics Lecture

[00:00:02]foreign last time we left it at saying well one first way in which you could test whether or not there there is a potential for poverty term or for multiple steady state which is typically interpreted as poverty trap is think about the mechanism that would lead to a mapping between income today and income tomorrow or between asset today and I said tomorrow and then look at whether this mapping is sufficiently strong so that's kind of the exercise we did at the end of last time when we were looking about the effect of income on nutrition we spent most of the time on that and I sort of asserted that the effect of nutrition on productivity and therefore future income is there but not so strong that it would outweigh the fact that the effect of income of the elasticity of nutrition with respect to income is not one so we left it as thinking about maybe the nutrition-based poverty trap as described as the fundamental phenomenon of that gupten Ray is not there but nonetheless in response to I think a question by reikia I put in doubt that this is a people that everybody reads this is like a modern classic and so people read it not so much because uh you necessarily believed in that very mechanism but because the whole idea is of based on this funky mapping between income asset today of Tomorrow is is something that we keep finding in different areas of the field now not only so then you could go to another approach which is what we went there is to check well irrespective of the mechanism is it just there can we find a setting where we can find um a relationship between two days assets and assets next year for example where there is not only that non-linearity but that costing of the 45 degree line from Below which ensure that there is an um unstable steady state and two steady state one low one high um another thing is of course what no matter what we think uh as they're going to be sitting in these homes there is also the policy making world the activists what the NGO world and some aspect some parts of this world believes in poverty trap and as in fact created programs that respond to the idea that there might be a povertita and the two idea I've met together uh nicely in the context of the Ultra Pool programs which are very much a program inspired by the idea that there might be a poverty trap which were designed and rolled out in the last few years and which have given an opportunity to actually directly present evidence for against a sort of reduced form view of is there a property term so that's what we're going to do today so the idea that there might be a poverty trap is that the heart of a relatively recent set of anti-poverty programs poverty fighting programs are promoted very much uh from the the South itself called the graduation approach graduation as in graduating out of poverty so the idea is that you are sort of undergraduating poverty and then you graduate and you're out of board uh so the graduation approach was designed by a microfinance organization slash NGO called Black in Bangladesh so unless you're very familiar with developing economics you've surely heard of Grameen and not necessarily of black but black is actually the the big girl and more comprehensive organization of the two is from their uh just passed away a couple of years ago was a wonderful wonderful individual and they have uh so they they have a micro credit program they also run schools their own health center they're on anti-diareal program Etc I'm sure they're doing a lot of covet and they've now in fact expanded in a lot of other countries in the global South but they started in Bangladesh so their idea uh what happened with them is that uh when they were running the micro credit program they realized that there is a set of people who are not even interested in getting a micro credit loan and those people are some of the poorest in The Villages so they decided that they need to do something to reach out to these people and they designed the program which they conceived as potentially a sort of a first entry into the world of very very small business for people and potentially a first stepping stone into micro credit or in general in the world so their idea is was that by giving people an infusion of capital as a gift not as a loan and helping them started you can help them Escape poverty in a durable way so the question for today I'll describe the program in bit more detail and the question will try to answer today's whether this works whether we can there seem to be prime facie evidence for a poverty trap and how we build such evidence and then do we know where the poverty type might be coming from uh so let me short uh catch shot to that in case we don't get at the end and the answer is not completely uh uh in I think we'll have pretty good evidence that it works it has worked both in short run but also in long run which is primary evidence of poverty trap uh probably because of a phenomenon of escaping some kind of two small assets but we don't know exactly why and there is a lot of recent research to try to get into that and into various mechanisms so how does the graduation approach work in a nutshell uh the community is brought together and identified what the think of as the poorest of the poor in the village so that's what some some team sometimes called as a participatory resource mapping they all go together they rank people there is a whole exercise with cards involving ranking people by wealth uh first half of the people you know people are cut in half by wealth the richest in the poor and the poor cat again and then cut again and Cat again until you arrive at four or five people maybe it depends on the village but on average it's four or five people in the village who are considered by their fellow villagers to be the ultra poor and when you look at this this feminist there is often something pretty up pretty clear in the in their history there are a lot of female-headed household where her husband died of uh or the husband is alive but in capacitated somehow a lot of [Music] households like that um or other households where there was you can almost always identify the sort of tragedy that brought them to an extreme set of poverty typically a lot of these women it's often women a lot of this it doesn't have to be depending on the program that it's a woman who is targeted but it happened often happen to be women who are begging or working as casual neighborhood uh living maybe on the outskirts outskirts of the village moderately included in the social fabric once these people are identified they are visited by uh by the worker from Baraka now it's being diffused in in a lot of other countries as I'll show you in a moment and they receive income they receive an asset it's a productive asset which they choose from a menu but it haven't often happens to be a pair of cows or a few goods in Peru guinea pigs bees in Ethiopia like honeybees it could also be a a sewing machine it can be a beginning like some money to start a small business to buy the first set of things to sell that kind of thing so in principle there is a menu and people sure although in practice there is a huge preponderance of cows okay then you get they get income support for a few weeks depending on the assets depending on the country depending on the program because the to make sure that they don't sell they don't dispose of the assets too quickly I think is the idea is that okay we just got them in so they get a stipend for a few weeks they get technical support to take care of their assets so if it's a cow look out gets vaccinated if it's a small business the employee will go to the will go to the Market with them to to help them buy their things to teach them to take the bus whatever it takes to help them with the with the activity then there are group meetings where there is literacy some small Health components uh they're also encouraged slash for us to have regular savings so it's a very multi-faceted intervention and the additional components uh we are so intensive in labor that it roughly doubles the cost of the transfer program so the asset program is about the cost of the assets uh turned out to be a thousand dollars so it's 215 PPP so depending on the countries and then that's about double by the by the uh the human cost it looks kind of prediction that you want to kind of prove the poverty trap model as opposed to like just giving someone Capital increases the Interiors like some key result yeah so the question is what's what are the key predictions so what what what's your what form of poverty type as opposed to you know money is good money is better than no money so let me turn the question back to you if you don't mind um I guess from last week about the super high returns that elasticity I guess yes so that's one what another one that's pretty uh that that pretty clear so this is a program that lasts uh the entire thing until graduation lasts about 18 months you get your assets and you're supported by this group meeting and all that for about 18 months and then it shuts down white exactly so the idea is that if it's a if it's a poverty trap the effect should not diminish in otherwise you would see them slowly uh diminish your brother long over the longer term as you know shocks happen and other people get positive shocks our guys get Negative shocks we will see the effect uh kind of slowly receding whereas if it were a poverty trap and they were permanently pushed out we would see the effect to persist or even to increase over time I think that would be a first order prediction and then as well is what you're saying which is you should see that if it's a poverty trap effect since it's it's the same asset for everyone uh but some people are even poorer than other even in this set so if we have a poverty trap model based on this uh at you know low return to Capital at very level followed by a steep increase full by law again uh what do we what would we expect in terms of heterogeneity of asset accumulation post the program or potentially heterogeneity of of program impacts exactly so potentially you would see a Divergence with a program effect being much larger for households that are sufficiently not not too not so poor that the base level of asset plus the asset uh bring them above the poverty top and we could see that by looking at long-term program effect we could see that even by looking at the dynamic of the asset accumulation for households that end up post-transfer to be above or below so we have a chance in a sense to see that S curve in action rather than just registration rather than just the traditional this is increasing because my assets is better unless I said but declining so that those are the two that we are hunting for today of course first like short-term effect which are not obvious to start with and then that yes and there's this participatory mapping uh process is that does that take place with the knowledge that the results of the processor there was the people who are identified are going to receive this very valuable program because imagine there could be some kind of capture yeah that's a great question so the answer is yes everybody knows this is what's going on I mean you could maintain the surprise maybe the first time but in any case that that's not sustainable or it's nothing it's kind of visible when that someone gets something like that so the news would go in so history is done with the view that that people uh uh know when they are doing the exercise that this is going to be used for our sources so the so um that's definitely a concern that people have expressed and they are going to oops ask you to suspend your disbelief until you get to a bench part of the section where he's going to talk about targeting so in this particular instance what they do as well is a verification exposed to check that you know to actually go to the household and then they actually look in some of the household that are not sort of poor enough another thing that because this was replicated in India was involved in the India study that we found in India is that a lot of people also refused the transfer in our setting about half the people refuse the transfer but that was a bit of an Indian specific thing so when you visit them I was like what are you talking about I don't want this uh which is surprising since it's a lot of a lot of money so I think the long and short of it you're going to again talk about targeting in a lot of detail because it's a key issue um is that um when everything is said and done these problems of capture and potential manipulation are probably not that big they are not inexistent but they are not enormous less first order than the economist the media reaction happens to be yep I have a twofold question um the first one is that going back to last class we can have power traps without Financial fractions yeah Commission on that is this a test of Finance is this empirical it's like a test of financial fractions and our financial protection induced formal traps join uh no I don't think it's a test of financial friction and in fact I I think we can almost rule out that Financial friction as at the heart of this in this setting because this program is run by uh by a micro Credit Company who is already present in these Villages and would like nothing more than to give money to these people and in fact the inspiration for this program was that these guys are not willing to go they don't have a project they don't want to do it so they are not willing or slash able to go uh so I so I think in a sense we can rule out the financial friction because you know with a certain amount of hand weaving this is a market where credit would be available for these people should they Wonder so that's uh so in this setting that's not what we're testing and uh and most of today is going to stay terribly reduced from which is I'm going to show you that previewing the result I'm going to show you that these program is a large effect that are very persistent over time over 10 years suggesting that there might be a a poverty type at play furthermore we are going to see that there is in fact in this setting this apparently this shape relationship between Capital today and capital tomorrow which could underline this persistent effect what I'm not going to tell you today is uh is where it's coming from I'm willing to rule out nutrition because of what we discussed last time I'm willing to rule out credit markets because of what I just told you but what it actually is I think is very much open for her for discussion and it's not gonna be solved today so this is that this form just like but already knowing it's there then we can go back to mechanism and say what is where is it where where is that mapping coming that is so strong to come to lead to that lead to that God so so this intervention is super expensive you know compared to doing or whatever I think including giving just cash to people which is a another popular alternative or including doing other stuff that could directly affect the the health or education or of these guys so of course it's an interesting question whether or not it is effective not only in the short term it's only valuable if the if if it if if that idea of poverty trap is there and it's in fact Justified the theory of change underlying it is that this one big push it's super worth it but it's going it's super expensive but it is worth it because it's going to be paid off over the lifetime of this person and potentially that children as well so the first question is does it work so here we're going to take a small step aside to think about how we even answer this question for some of you it's gonna be a view for some of you it's going to be new material uh but even the review I find that you know every year I teach this and every year I I learned something by just staring at this uh at this number so I hope that even for people for whom it's a review you're going to find it useful so it's a review of what it even means to have an effect and what impact I mean um so I consider our treatment we want to know whether a particular treatment as an impact and let's call it w and then outcome y for example the outcome here is whether you got the Ultra Pro program is the treatment and then whether or not you know how whether or not you are able to feed yourself on some food consumption or anything actually there could be a range of outcomes and we'll go back to this question is the outcome so the way that the events and Ruben wrote this up is to say well the outcome that I'm observing is uh the potent something that I'm going to call potential outcome if treated which is observed only if you happen to be treated so it's multiplied by this dummy which is equal to one if you're treated and then it's your potential outcome untreated multiplied by a dummy which is one if you're untreated which is one minus wi which is dummy indicated diode treated so everybody is only observed in one of the two situations but the dummies of treatment things on and off and uh so I'm not a at least conceptually I can always Define the potential outcome for Boost situation before the undivision occurred you could have been a treated person or a control person you have a potential outcome for these two for these two settings uh before I move further this might be familiar to a lot of you but there is an assumption that's buried in there um already in the idea that I can write this this way can someone tell me what's the Assumption that's buried in there yeah exactly so this is uh what goes under the uh the the jargon of stable unit treatment value assumption or sutva which basically says if the fact that someone else is treated doesn't affect my treatment and this is the only thing that allows me to write the potential outcome as just my treatment as opposed to my treatment and that of every one of everyone apart otherwise suppose you had two two people in a and one is treated one is non-treated but the treated person gets upset when the other person is untreated then the happiness of the treated person is a function of their own treated status and the treatment status of the other person so just with two we already have four potential outcomes not two again we are holding them out here which as Reika points out means we are ruling out Network effects so for example uh the possibility that someone would help someone else uh we are ruling out uh individious compare in videos comparison effect like I just did it might make me upset that someone else is helped we are ruling out learning we are ruling out straight externality contagion this type of things which become more relevant for her for her medical treatment and that kind of things okay so we are not ruining out because we think they never exists we're running out because we we for now we are calling out and then we'll bring them back at some point uh so that's an important uh thing to say it becomes hard to uh to tackle if we don't have suitors who want we don't have to drive we try and model in what way we are into uh we are affected by the other units uh so if we can write it this way then there is a well-defined idea what's the treatment effect it's my potential outcome if I'm treated versus my potential outcome if I'm not uh are we able to estimate that thing for any single person no because we are never observing both okay so that's not happening uh because exposed we only observe one of the two potential outcome the individual ends up being treated out and end up not being treated so I will never know the uh individual treatment effect for each individual we might get close to that if we have enough variation in say observable characteristics about an individual and we have sufficiently much data we can try to get to somewhere in saying well the treatment effect for someone who looks like Reka in terms of being a young woman who is studying economics who grew up in eight countries that uh tends to be that but we would have to have enough of people like that to be able to say this is the treatment effect for someone like that so this we might be able to do in general so we could be interested in never a treatment effect for the population which would be the expectation across the group of the difference of the of the treatment effects uh we could be interested in the one for whom received the treatment so that's the average treatment effect that's the treatment on the treated and then as the discussion I just had said we could be interested in the uh impact uh for of the treatment for those who have some characteristics observed or unobserved so this becomes still an average treatment effect but for a person who who has some Vector also whose characteristics are some Vector X so when we talk about personalized medicine machine learning Etc that's really what we are talking about which is trying to bypass this thing that we cannot do personalized medicine because we only observe the same person ever ever once by exploiting the any all the information we have about them but things we might want to do this is kind of all a flavor of our treatment effect we might want to know how the treatment affects not only the mean microscopes but the distribution of the treatment effect so that we are going to get from quantile treatment effect by comparing people by comparing the quantile it could be the median could be the 90th percentile the fifth percentile we might be interested in uh in in the quantite of the treatment effect which is not the true which is not the same thing as this because unlike the expectation which are nicely uh working out in both sense the the quantile of a difference is not the difference in the quantile unless you are willing to make enough assumption so I'm pointing that out you already heard it but we forget it all the time because if you add a convenient assumption that the treatment doesn't affect people's ranks and then this start to look very much like that so under this assumption which is very strong then you can get at something that looks smaller for the quantile of the treatment effect but uh we we have to remember that unless you're willing to make that assumption that's not necessarily a very good that's that that doesn't work that way for example if you go back to the Ultra Pro program could well be that the the program not only affects people's uh wealth level consumption Etc but also their rank in where they stand in the distribution there's nothing that assures that we have this ranking variance assumption okay so now suppose we have that we go back to the you know simpler problem of trying to know what the average treatment effect uh and we have a group of treaty people and a group of control people so well the best we can do is to start by looking at the difference between uh the um the mean we would like the difference between the mean which we aren't replaced by the sample average so that's the expectation of y1 why I treated for the treated minus the expectation of why I control for the control I'm gonna add and subtract something an object that is not observed but conceptually exists which is the [Music] um average of the untreated outcome for the treated population so I'm gonna I'm subtracting it here and then I'm adding it the other other in the plant that it is missing so what does it give me well now that I have that I can just rearrange a little bit and I'm getting the expectation of y i 1 minus y i zero for the treated so what is this what's that quantity is it some is it an object that we are potentially interested yep it's the average treatment effect on people who works so treatment effect on the treated so that's great uh but then there we have that and what's that difference is exactly so it's the potential rather than already existence that would have existed anyways or something in the absence of the treatment uh it's there because it's not it could be captured by a baseline or it couldn't be it's not necessarily A temporal thing but it's saying that in term of potential outcome these two groups are potentially already very different so take within the village if I run the Ultra Pro program and I run my participatory resource exercise and I say and they identify like 10 people and then I go and I uh survey at least 10 people and finally the service staff decides that the staff decides they're only going to take five out of the 10 people because the other one don't quite make the cut someone could say and someone has said and done a first analysis of this program by saying let me compare the five people who got in the program to the five people who didn't get in the five people who didn't get in were presumably quite poor because they were selected by their fellow villagers so if they um so if I still find a difference between the two it must be the the fault of the problem of course what we don't know is whether there is something different about them that means that the the the service the the staff of the NGO decided not to tick them they could be it could be observable or it could be unobservable and that's the problem we have so that's the selection problem so basically we have three cases either we are in the best case scenario from the point of view of this problem which is that the program was randomly assigned so for example once this a frequency in the altapur study there are two kind of uh random assignment is there The Villages were randomly selected or are all the villages were not randomly selected the identification was done and then within the people who were identified people were were not selected randomly within qualified people in that case there is nothing systematically different about the people who got selected once by the by lot once I get do all of this thing so in that case the the selection should be zero and uh I should get an advanced effect estimate of the effect of the treatment on the treated that's what we are going to study today uh another setting is one where uh it's not uh it's it's not a known function of covariates uh but I'm willing to say that once I know everything about them enough things about them something happened which means that some people got treated and some people didn't and I'm willing to say that conditioning on the coverage that I observe despite my not knowing exactly how the sausage was made I am willing to make the assumption that the potential outcomes are independent the two potential outcomes conditional on the X are independent of the treatment status uh and that leads you to you know that that would make you confident to try and control adequately for these different observables as long as you control fully for this set of x even though you don't really know why someone with this particular characteristic was treated another was not you are uh reasonably sufficiently satisfied so if you do that for example that's the example I gave you with this idea is that a conditional have been been selected by the people you're willing to say that you don't know exactly how the staff chose but whatever they did that's uh as good as on them then if I'm able to control fully in this case that would be easy because I have the selected and the non-selected people I can control for that status and within that proceed almost as uh things had been randomly assigned even though they weren't and I know they weren't so that's of course this sort of standard justification for a lot of OLS regression which adds on top of that a bunch of functional form assumption matching regression which adds less strong function and form assumption or sometimes no functional form assumption for example uh if you if you the number of cells that you have of excess is not that large and the number of observation is very large then you can calculate the difference between treatment and control in each cell and then take the weighted average of this cell and there you go with your treatment effect all your assumptions are correct so Josh for example has a very uh well-known people on the effect of the military service where I say I'm controlling for everything that could put potentially entered in the decision in why someone does the military service within these cells I'm looking at each cell I'm looking at difference between has done it versus has not done it and then I'm re-aggregating so that's a matching estimate then you have different type of matching estimates that are more or less parametric and therefore more or less believable but the underlying assumption is always this of unconfrontedness once you know enough and then third method with the same idea is a double machine learning estimators double post plus so that uh Victor Channel zukov and bologna developed and has become quite popular in the applied literature and the double human generic double machine learning method that Whitney Victor me and a bunch of other people worked on and that is also quite nice but it's conditional on this assumption so it adds bells and whistles and sophistication in terms of the econometrics analysis but at the end of the day that's always the same thing which is a not really a testable assumption in most so this is your kind of observation Observer no I would call it without uh Prejudice no research design strategy because my strategy is no strategy uh and then there is a strategy where where you're saying well you know the are you worried that the potential outcome depends on on the assignment you're not willing to make the assumption that there is a set of X that solves it then I have to do something to try and find a corner of the world who are not subject to this and that's where you get to the research design strategy uh so we'll spend a lot of the classes looking at examples of that for different differences one uh um regression discontinuity design an instrumental variable a special kill of instrumented variable is one where you have a latent variable that might be randomly assigned itself and you use that to form an instrument for your for the for the for the treatment that you're actually interested in so there is no wi that is randomly assigned or as good as randomly assigned but there is a z i that is potentially randomly assigned affects wi and I wish you're willing to make some additional assumption to uh uh kind of recover the effect of WI so that's kind of what we're gonna short preview of how how the different strategy that we are going to see in the course of this semester lineup so today we are starting with the easiest one at least the conceptually easiest one which is random assignment which so by definition a random assign undermines assignments solve the selection biased in your in your sample because you it's a known assignment known assignment rule which is either fully randomized or randomized conditional converts but randomized anyway so you can compare things directly without worry about the selection base that said uh that's not still not like a panaceae there are there are issues with under my Evolution that many people have mentioned um do you know and we're gonna keep seeing those and we're gonna see some of this today uh what are some of the things that people worry about in terms of validity of results in the case of randomized evaluations okay you can conduct an RCT in a specific setting but you might think that if you wanted to apply that within a different country for instance that it wouldn't like supposedly have that effect elsewhere exactly so that's the first issue and in fact one that we're going to talk today is the the uh uh um extent to which the result are context dependent and would be at all heterogeneous from Context to context and therefore the result of one is predictive or not of what would happen in another setting so that's the first concern that's definitely present what what are other concerns people have yeah um when people don't behave following the assignment they were giving and treat it act as they were controls and controls act as if it were treated for example if you give them a pill I cannot I can take the pill or I can decide not to take it so right so there are there are not various non-compliance issues for example the one I was talking about in The Case of the of the El Chapo which is some people just say no to the program alternatively some people might manage to sneak in which means that your actual treatment is not randomly assigned anymore uh so then you know you have to deal with that that you you lose the world of uh of perfect condomination yeah there's also a more advanced issue where in the realm of heterogeneous treatment effects the ebultural RCT may depend on people knowing that oh they are in the right tail to treatment effects like people say I'm a bit I I'm incapable of running an experiment in a village maybe that big task because the governance in that Village are aware of the fact that oh this being actually on the right tail of the distribution will be very good for us so it's not only that the treatment in that Village is not predictable others is that the theology is the highest possible treatment effect compared to that right excellent so that's what Heckman called the randomization bias which is that not only the right heterogeneity in the world so the results of one experiment may or may not generalize but there is something specific about randomized control trial that may lead you to have for example higher treatment effect and one of them is site selection which is someone either researcher because they are willing to they would like to publish a paper all the partners because they would like to you know look good we'll pick the sites where the treatment effects are the highest so you will tend to uh to to be best in that direction so there is a very nice paper by Hans Alcott in environment which shows that the the first experiment by The Firm o power which is one where the you get this little leaflets about your um uh energy consumption and if you cons it tells you how you compare with your neighbor so you got them we always look terrible when you guys lived in a house we were actually a terrible plus even worse than usually um so I wanted to tell them but there are six of them um so in any case you receive these things and it tells you you read please reduce your consumption uh or not even it doesn't say that just say so it finds pretty large effect in the first sights and then uh the experiment got replicated I'll cost hundreds of sites and then what you see is that the the the the the results in the next 95 sites actually much lower than the result in the first few sites and that could be exactly because of that because the first sites were selected to be places where maybe people would be believed that where people believe that it would have the the biggest impact so those are some of the issues uh another issue is that when you have many many outcomes uh not everything can be really presented in a paper or people have limited attention so there is a risk of cherry picking for very many outcomes in the case of a program like like the ultrapore you have that issue in droves because it's not that you have like one thing that you really want to follow you want to see whether it improves their lives which is measured in many many ways and so how do you deal with many outcomes um so I think these are some of the oh yeah and of course uncertainty we mentioned compliance but there is also just uh the sample size is not enormous so they are not infinite and therefore you have power potentially power issues which you think you'll spend some time on on that and a criticism against experiment is that they are generally too small so their power is too low and they they can only get effects if they get result that are significant if the effects are very large which might lead people to miss some effect so we discussed that spillovers didn't come up but it came up when we were discussing uh we were when we were discussing the the assumption cherry picking an external validity so we'll this will be with us uh these are issues that are serious that are real and therefore that need to be addressed for each people now go back to graduation and there are two recent papers uh I wrote two studies but in reality the the second papers like six stages and showing remarkable effects of this program the first one by Oriana Bandera Robin Burgess or another looked directly at the uh at the black program in Bangladesh the mother of the full program and then there was a follow-up it was published before because it was published in science but in reality this came after in terms of running the experiment uh by a bunch of people including including Abhijit and me looking at a very similar intervention in six countries uh which were followed first for three years and then uh this this in some of the sample and I'm going to show you the data from for for India uh they were followed for another until 10 years they have been followed for 10 years and these guys also they lost the control after four years but they continue to treat to follow the X control and the X treatment until the news in India we never lost the control we continue to uh to follow them so that gives you some uh so let's look a bit at the result first uh and and then discuss about some of the feature of this experiment which address some extent some of the concerns that you that you guys might have uh so the so these are the results for the for the uh Bangladesh study the Bangladesh study was randomized at the uh at the geographic area level so first decided where to go and then they they did the whole intervention they treated everyone who emerged from the process so it's a good situation in terms of the speed of our situation most of the studies in the science paper were randomized at the individual level within Village although a couple were randomized at Boost levels there was random musicians first at the site level and then within sight at the individual level so what's the nice thing of having to randomize first at the site level and then individually with inside what's the what does it allow to to check exactly so you can directly check for the spillovers because you can look at the untreated people in the treated Villages and you can see whether their outcomes are any different so long and short of that and that's why I'm going to treat all of these studies the same now is that there is no evidence of spillover in this case doesn't mean that they don't exist in general they're just not present in this case so here is this the result from for the Bangladesh studies so these are some of the few outcomes that the that they used uh below poverty line consumption expenditure per capita or per adult equivalent value of household asset savings whether they get loans were on the give loans um each row is a is is a treatment effect who is the standard error of the treatment effect and starts to indicate significance so that's for the first two lines so each row in the first two line inverse column is from a different analysis right that makes sense we are every time comparing treatment versus control so for example if I read this uh can someone Vision can you read that like a constructive sentence with that number is that so four years after impact treated how the sun's had on average what is this about 8.4 percent lower people below the population percentage point so they were eight per 8.4 percentage point less likely to be below the poverty line and then generally this type of table we are going to provide the control group in order to know what we are looking at so here sometimes hit that Baseline here that show us the control group at the Fourier follow-up that was 62 percent so it's 8.4 percentage point on a basis of 62 percentage point so that's for 13.5 percentage reduction then we are testing whether the impact are uh you know remember we are interested in Long Run so we are testing whether an impact really they could have done a one-tested tail at one sided test to see whether the impact were decreasing uh but they are testing whether they are different over time in this particular instance they seem to if anything increase over time um then they ask where I don't know what we do with the R square ever but that's there and then the number of women the number of observation in their households and the number of clusters is there ever a decision when randomizing or choosing a sample to make sure that we don't trade more than a certain percentage of the village to make sure that the total change or something like that but I'm thinking about how you would argue that GB effects are contacted uh so they could be or you can if I could go the opposite direction there has been a uh a lot of several studies in the past few years trying to vary the fraction of people who are treated in order to see whether you start seeing uh equilibrium type of effects on prices on things like that by around amazing at that level so sometimes you want to rule them out because you're interested in the individual effect sometimes you're very much interested in the market effect so you want to make sure that you saturate enough on the on the contrary so for example Loren who is visiting uh here she's doing a lot of work on on making markets in a way and creating a relationship between Traders and and potential Sellers and there she is looking almost for the opposite and she wants to make sure she has enough situation so yes absolutely this is something people have in mind to see whether so in fact in the case of the alcapur studies the reason why I think we see no externalities is that in every village very few people are treated so it and not tremendously integrate socially integrated people either so that's the results that I find encouraging impact on pretty much everything especially after four years uh then uh the the science study came about and in a sense was a response to the external validity point or a way of addressing the external validity Point given this super promising program there was a lot of of um of of uh hope that this could be a strategy to address poverty and but given how you know expensive and in a sense specific the program is with all of these various component to it that you can't really easily uh untangle mechanism or understand why it works uh there was really a [Music] desire to make sure that the the program is replicated in as the different context as possible in order to to assess the external validity or the context dependence so uh this is some a project that was led by Dean Carolyn uh who uh who is at Northwestern now was at here now and and started IPA uh which is an organization that conducts a lot of randomized control trial around the world and he and the Ford foundation and cgap the World Bank microfinance ultrapur Arm kind of got that project together and I don't want to underestimate what this means to get uh six or six different organizations in six different countries with six different research teams to each conduct their project and together and what is interesting here is that there was really the the the the goal was really to estimate the same project but what does it mean to be the same project or you can't force people to do cows everywhere because the whole point is that people can choose their assets uh so you need that to have a definition of same like what is same and here the way they enforced that so the same nest in a way is by having people from all of the program meet in Paris every year I attended a few of these meetings where they were discussing exactly what they were doing the process they were following Etc so same is like a samness of spirit and then uh in you know embodied in a sense in different ways uh in different places uh with you know keeping of course the same element in place identification uh the assets the other multifaceted support to go with it but then you know people for example the staff is paid very different amounts in different places the value of the assets is different in different places and so on and so forth but to arrive at the same program given the context so that's a first thing that that is worth mentioning uh and then uh the other thing that happened is that uh all of these different teams were conducting the projects separately and uh instead of each writing their own paper a decision was me to write one paper which would have all of the data together precisely because that was supposed to be looking at you know what are the effects of a really heterogeneity and so on and therefore you needed one one kind of template and the nice thing about that as well is that that addresses another aspect which is the cherry picking of Hazards of course there you also have a lot of results but everybody has to everybody was running their own analysis separately and they had to follow the same template so the template is sort of given in advance so even though there are many outcomes and here you have outcomes that are already sort of there for everyone they are pre-established this is what the outcomes are going to be and it leaves a little very little place for her cherry picking despite that you have a lot of outcomes so to have something that is readable you end up doing something that is frequently done in particular in this type of project where you have various variable on food security for example did you eat did everybody ate yesterday did kids miss any meal did adult Miss animal Etc which are so you form an index that indexed this component by standardizing each of the variable and then taking the average of them and I'm not saying this is necessarily the best way to form an index but this is how the literature I've evolved and this is kind of the standard way of doing it so so here are are the index that were used this is the first and line 15 month second online is at three years and you're showing so now it's expressed in uh in standard deviation of the of the mean directly even for things that are in in you know in money to start with and here again we find effects across the board on pretty much all of the outcome with effect for consumption is about 0.12 standard deviation so um doesn't make them Bill Gates easier but it's an increase in uh an increase in consumption that stable over time this is a q value so that's uh what's the Q value to comparison to the p-value yep right so it's one way to account there are many there are many ways but it's one of the methods to account for the fact that you have more than one thing that in being tested and in each of the tests are connected to each other so that's a false discovery of it uh adjusted p-value uh and um here it doesn't you know mess mess up too much with you because all of the outcomes tend to be uh tend to have similar impact so that's Comfort the results of the black study showing impact at 1.5 and at three years that are not the impact don't seem to be decreasing from one place to the other so that's we already discussed that uh IT addresses some of the to that study addresses some of the shortcoming from the point of view of pure impact evaluation of being confident that that program might work not only in those sites but maybe in other sites uh if we look at them then since we have many countries I showed you the effect for as if it was one giant regression here but in reality we have a lot of countries uh so one thing we can do to start to look at the external validity is ask ourselves how do the results vary from country to Country and you can see that they are actually pretty variable they appear to be pretty viable from country to Country in particular in Honduras was a bit of a disaster you can give an exposed explanation for that in fact I can give you the expression for that the main asset was chicken and all the chicken died because of bird flu but you know that's exports um but that's true so that's those are but you see a variability in the funny story about the assets uh we were doing it in in India I was running the study in India and my my Indian quarter and I uh his name is we went to I mean I guess I visited his Indian too but but the guy who actually works in Kolkata we went to the field together and they were about to hand chicken and it was like he went screaming so you cannot and this was before brought through but you cannot hand chicken they die uh and uh and the kind of Israeli convinced them almost went to the CEO of the organization to say that chicken should be ruled out and so they didn't do chicken they didn't really put it in the menu because they had been so popular but they thought they were dangerous and then this on the last result came out with all of the chicken having died yep yeah so that's a great great Point uh which I will generalize a little bit so the question is whether a cash transfer why why do assets and not cash how do they compare do we know that um this is of course this is a huge bundle thing so I'm now I'm going to I'm showing this reduced form effect and of doing everything together and showing that there is an impact and then we'll talk a bit about country heterogeneity but then there is the the once we kind of satisfy ourselves that there is an impact and maybe it's durable is where is it coming from could we take out some of the component of the program that are less effective could we go to cash and avoid this chicken issue or maybe increase this chicken issue um so we know some about that which I'll come to in a bit and then some is still full not fully unpacked but you're completely right and that's another it's not the criticism of the validity of experiment but it's you know potentially a criticism of this kind of experiment which are telling you this is my treatment that's its effect no this is all I can really tell you right now right and then it's kind of wet your appetite to say well where when your what's the mechanism why does it work the way it works and and do we really need to do all so from a policy point of view do I really need to do all of this from uh economics human behavior perspective is what have I learned once you told me that all of these things matter together so that's kind of the ultimate bundle experiment in that way so that's the assets that the income and revenue little bit more less a diverse per capita consumption so so that's a question of interest in the people we just showed some of these different some of these Effects by country we present them in presentation but we leave it at that now the uh the question we wanted to answer if we want to answer records question which is do I learn from one experiment what's going to happen in the next site or in fact your point as well well your point adds this onto donating the experiment but what's the predictive power of one side to another one is not complete it's not really sufficient to look at those type of graphs because when you run an experiment in different places so here we have six sides each site is different for two reasons number one is um the treatment effect might be different fundamentally it might be higher in India than in Honduras number two there is noise in the world and the sample that you happen to have found in India might be different than the sample in Honduras both in treatment and control and that's just you know every estimate comes with its own standard error so once I show different treatment effect this country by country treatment effect is a conflating these two sources of uncertainty so if someone shows you and said those treatment effects are quite different so maybe the next country would also have a very different effect it's not it's it's really it's really attributing the entire difference to heterogeneity and treatment effect when in fact there is also the underlying noise in the world so this is a bit uh that's not the answer you that's not the answer you need or want so the way to so you can't really solve this this problem without being willing to put a little bit more structure to the problem and one way to do this is by a method called by Asian hockey care analysis which Rachel Miguel who was a student here and is now at the at the LSC started to to do first for micro credit studies and then I'm going to show you how result that she just sent me off of the press to look at this this very program and the idea is to hear the structure of the basic structure on the problem and I'm not going to go to the detail of how you conduct the analysis it's a bit it's just computationally a little bit involved but what's important for now is that we understand the concept the concept is going to say let's assume that the treatment that there is a distribution of the placebo treatment effect and that distribution is normal so the treatment effects are themselves drawn from a normal distribution and I'm trying to find out uh for each site uh the the treatment effect for that site as well as the variance of the treatment effect across site in the world in general right so that's the that's the the idea so if the variance in treatment effect is very very small of the of the distribution of the treatment effect to suggests that the treatment effect itself is quite constant and therefore it's for example a very uh you know it's legitimate to pull all the data and estimate one pool treatment effect once you know that it's convenient because you can use information from all of the other country to increase the Precision of your estimate in any given site so the Precision of the estimate in India will increase a lot if you know that you can use Peru and Honduras Etc and also you can uh kind of form credibility interval which will be different from confidence interval but are the same context in prediction for the same for next for our next effect if on the other hand you find that the treatment effect there is a huge variance in the treatment effect itself and not just the outcome then you know that it's there is heterogeneity it's not it's not very predictable and maybe you can try to predict that the originate itself but otherwise you know there is there inspiration so that's the that's the idea what she found is that both in terms of the um so so these are the effects uh using uh just the effect from the pool regression using the linear model so that's two to start with the result I already gave you assuming there is one treatment effect for the whole world and this has a treatment effect in that in the sample with the with the hockey calculation model and you can see in this instance the the confidence interval going up reflecting uh heterogeneity in the site that I I was this is making the assumption that everyone has the same treatment effect this is not making that assumption anymore and and the that increase that decreases my confidence in the result and now if I want to predict the next side so this is the uh the predictive effect so if I want to say what's my best estimate of the effect not in my sample which is the effect here but in any future sample uh the um the predictive effects are quite um quite wide The credibility intervals are quite large which reflects a large variability in the treatment effect from side to side so here the the fact that we saw already in the graph that the treatment effect seems to be going all over the place uh reflects uh some real heterogeneity as opposed to noise in the data you know so what's the each treatment effect is estimating relatively precisely for that country but it happened to be different from side to side so my predictability of the next site is not that good that said despite that uh most of them are positive uh even with that it originates so this result by the way are different from what she found for micro credit where she found a lot of a lot of pooling where she found that basically it's a nice big fat zero or regardless Where You Are here we have positive effect but but viable yep what is the point estimation oh because uh the um the the here you're making the assumption that the the effect is the same um everywhere therefore one observation is one observation is one observation whereas here you're not making that assumption uh you you you so when you estimate your effect you're not making the assumption that you can just run the regression uh all pulled you are using you are get basically getting an average of the different treatment effects where the weights doesn't don't don't become anymore we like one per observation um so in the rest of the so far an existing paper but I can send you the slide deck to start with what they are doing is to try and see whether they can explain some a fraction of that heterogeneity by the heterogeneity by covariates can you predict is the fact that in other words can you use the heterogeneity and treatment effect by covariates to predict etymology and treatment effect at across sites based on the fact that the covariate of the sample was different does that make sense so they are basically saying well the treatment affects the do they differ by people to to what extent do they and then to what extent does that explain by difference in treatment effect and the answer is not that much simply because we are not very good at using the covariate to predicting to predict different treatment effects in this case even within countries and that I know for having tried so that's the you if you're interested I'll I'll without permission uh I'll defuse the slide I think she'll give me permission since she sends me this right for this and you can you can see a little bit what what she does it's not uh it's not complete but it will be at some point uh okay so anyway even despite so that's an interesting point I think for heterogeneity because uh this is really the first paper where I see a lot of Italian being demonstrated even in a program that in general works uh in terms of cost benefits uh this program is super expensive so it's only worth it if the benefits persist in the into the future uh if you look at the first paper the consumption increase is about 11 percent in bandias paper so in order to do cost benefit analysis they assume that then this effect lasts um the of course the depression rate of benefits is key so this is whether or not this is a poverty trap becomes key also for whether it's the program was doing not just because we're interested in it um this is for the uh this is from the science paper where we are looking at the sensitivity to a different discount rate time Horizon and and benefits and calculating for how long the uh the benefits need to last in order for the program to be uh to be cost effective and the point of that table which you can look in more details at your leisure is that if the benefits were to persist despite the program being extremely uh expensive it's it's very cost effective but if they were to decline after four years then it would not be obviously then it becomes a very interesting empirical question and let's tie to our poverty trophies do they persist um so uh we looked at this uh with the visit and with garima uh who is in the program here in West Bengal after the three-year survey we received at seven years and then at 10 years and here is one of the uh the graph for consumption where you see that the treatment effect actually grows between year three and year seven and then becomes and then he's persistent mostly persistent between your uh year seven and year 10.

[01:11:04]so this is in a context where there was economic growth so the control Guru where the treatment grew faster most of the good treatment become faster so the control level Grew From 1.8 to 2.9 and the treatment effect in terms of standard deviation it actually jumped between year three and seven to point seven standard deviation and still 0.6 after 10 years so that's uh shows some persistent of effect which means that when we actually redo the cost benefit analysis for India with this new data it's actually much more favorable than it was done at year three because the the treatment effect and it didn't vanish and didn't decline so in fact by year 10 even if the effect were going to disappear fully uh the treatment the program is is worth as paid off for itself so this is from income uh in the we're also looking at costs across outcomes and we uh using again nicely we have a nice sort of pre-analysis plan given to us by the fact that we are following an existing paper so we don't invent anything we redo the same table uh so the people are also nicely writes itself and you're finding uh the the treatment effect to persist over time of um with the exclusion of with the exception of financial inclusion which is mainly whether people borrow or not where we kind of never really had much of an impact in the first place uh in that paper we discussed so interestingly the effect on on physical assets that people have the clients have had we are first increasing and then went back down so we discussed and another thing we see is that uh what you these people are getting quite old who got the first transfer so a lot of them are sort of also uh slowly retiring and winding down their business what we saw in India between the first the the first and the second landline uh and then sitting online and the third is people diversifying out of cow into non into non-agricultural businesses like uh small business and stuff like that right we saw them doing between this wave and that wave is winding down the business and sending migrants out out there and in particular sending migrants faster so to answer the question about what's the mechanism we make an assumption we make a a guest here a context here which we are it seems to be burned out by the data we have is that the way in which you actually have Persistence of this program is first by diversification from agriculture to non-agricultural businesses and then from small business to actually uh more stable wage work via the following generation which moves uh for a longer period for further further away and earns more money as they do that okay uh now these programs have highly heterogeneous effect we saw it by by country I only showed you that by countries they have super heterogeneous treatment effect and we also see it by people uh so these are now within this is Bangladesh so this is within a particular country the quantile treatment effect so remember it's not the effect on the it's not the the distribution of the treatment effect but it's how the the distribution affect the affect the uh distribution of assets or consumption expenditure and what you're seeing is that the effect at the low quantile are not very large and they keep the effect on the highest Quantum uh highest are higher we find the same thing in or in India and people find the same thing everywhere that the treatment effect tends to be to Once uh the highest Quantum with uh less effect at the lowest quantile of the at moving the lowest quantity of the distribution on the other hand as I said before an effort to link that to observable is not very successful so the effect of hydrogenation is not very clear initially how why so a key question uh for the future from a program perspective is how do you identify people with a very high return and that's the same thing as in the micro credit business where even though they're on average as we are going to see when we study credit treatment effects of programs are zero they are quantile treatment effects that are very different at different quantile some people seem to benefit how do you know who they are to Target them and in a way they El Chapo have that as well in our case this is a project that is a question that is even made even more interesting by the fact that this is our you know this is motivated by poverty trap and we kind of have an idea of maybe something that could say that you get huge effect or you don't which is uh um do you get out of the poverty trap for this persistent effect or not so uh that coup who did the uh the Bangladesh program along with Claire baloney who is teaching on the part in our department as you as you know and Mike who joined them uh try to look at exactly that question so it's really one of the first serious attempts that I know of to draw a kind of s-cover to see whether they ask of at the individual level um and some people have been quite skeptical that those existence there is a people by Mackenzie on small businesses showing I don't see any of that uh but with an observational data of course uh what's the main problem is trying to draw and ask her with observational leader and why is it that you might not find it in observational data yeah so if you do have disaster then intersecting the 45 degree line uh that steady state is unstable so in a cross-section actually over there exactly nobody is there it would be like an accident that you find even a few people there and therefore you shouldn't be able to capture it in observational data in the real world if you really believe the believe the model unless you know the escrow was actually individual specific and you kind of uh people were moved it seems that in fact you should never see people with a very very high return range because they would have moved out of it by now so in a real world you might be you will be able to draw a curve linking Capital today to Capital tomorrow but that curve is going to reflect the heterogeneity of where the those threshold might be located as opposed to whether that threshold exist or not that's pretty big problems so why uh the Ultra Pro Program help us there pretty nicely yep so there is that and in addition yeah so what would have been the ideal experiment if if testing this S curve had been uh the ultimate goal yeah you have different amount of money or Goods to different people exactly so different amount of money to people who start from the same assets would have been the ideal thing to do oh this is not what was done here that's a true True Life program and they were probably not about to do that but you have something that is getting a little bit close to that is people were distributed whichever way and then they got these assets and then the asset would have been small enough or large enough or not depending on how poor they were to put them over the threshold or not so enough to do this exercise you have to be you have to really believe the model because you have to believe there is one threshold out of which you know that people can jump over or not otherwise you would it be hard to find yeah there's also one steady state and one profession and like now we are allowing people to have different set of States like assuming that they still have one pressure to pass I think in the model there is uh or two steady States one low one one Higher One One unstable steady state or two three including an unstable one and I think in the view of the real world also there are only uh everyone faces the same thing but then they are shocked which means that at the given point in time they end up they have end up one way or the other you know remember in fact I think I have a I have a I I have they have a distribution this is the distribution of assets and I would say no no there are two there are in the modern stable steady state and two stable city state in a real world so what I should see is that there are a lot of people at each of them and then not too many people in the meantime but if there are shops that I'm alarming I don't need the experiment to begin oh yeah but the point is that they are shocked combined with the fact that so it's not saying that you'll never observe anyone in the unstable statistic but you won't see very many people so it's going to be very difficult to to catch them it's not so here is the curve and I think that that's that's really a beautiful graph in a sense of I don't know if it would have occur to me that you can even try to do that which is to say well let me take the theory super seriously there are two steady States therefore I should see people with two levels of assets the high level and the low level and let me draw the distribution and and sure enough they have this bimodal distribution so then even a world where or this this seems to be a world where that thing actually exists and is relatively clear what is the what is the steady state and then they are shocks which means that you don't see nobody but you don't see very many people there and then once we have that then I think we went through this already or this is small we can now add the value of the the asset it's quite small uh so we add the medium value of a cow and now suddenly we find a bunch of people in the middle and in a sense that we are lucky because the transfer was such that given you know from the people who were very very poor to start with the uh a lot of them were just to the left or just to the right of the threshold and now we can run that curve so this is a treatment group and this is saying that secure we already saw in the first class exactly Bravo this is pretty impressive which is let me show this is just you know this is a non-parametric regression this is not there's no magic there no rabbit has come out of the Hat that's the data showing that this is the Baseline productive asset so in the treatment group and in the uh Baseline productive asset in the control group so the treatment is the asset they had at Baseline plus the value of the transfer and now you're able to do this curve and you can draw it because you have sufficient Mass on both sides and by contract if you try to do that in the control group you only find one unique steady state uh maybe there is that other studies that we can't catch it because there is no nobody there and then there is so you can turn that into a regression exercise and to say um they find Delta I as the different in assets and then K star the threshold level of capital which they estimate from that regression and is it the case that people go faster when they when they are Level plus treatment is above the the threshold in the treatment Villages compared to the control Villages because the the asset level plus treatment is just their asset level so they don't benefit from the they don't benefit from that so that gives you a regression which exploits your experiment uh so you're saying that in treatment if you end up being if you if your own asset level plus the transfer puts you above the threshold you have fast growth in capital but not in control because the treatment you didn't get so it you know you above but you didn't get it uh and then we can put the two sample together and run this as an interaction so uh that's where I'm going to leave it for today the next question of course is where does that come from and we know it's not food we know it's not credit so the question is what it is and uh a lot of a lot of of we're going to continue talking about that in fact we're going to continue talking about next week because you are going to talk to yogena Frank is going to come for two lectures to to talk about behavioral art and one of the potential explanation is that it has more to do with psychology with various aspects of the psychology of poverty which might have impacts that are functionally equivalent to the ideas that we're in the nutrition-based poverty trap that comes from people's reasoning as opposed to something else and then we are going to see it again when you talk about Labor markets because some of this could be productivity of labor but we now know that at least for the ultrapore there might be something like that which is really at Play and so that should be give us even more energy and motivation to find what it it's that can be at play hence the unpacking that is also going to come up uh in other uh at several points we're gonna be have chance to unpack the ultrapore program thank you that's it for today and I will so Monday next week will be frank and I'll see you again in the following week