Weapons of Math Destruction: Algorithms and Social Harm

[00:00:07]I'm very excited today to introduce to you Kathy O'Neil will be speaking to us about her book weapons of mass destruction Kathy is uniquely welcome to talk about the subject of mathematical models and how they can go bad she has a PhD in math from Harvard University she has held positions at Barnard College at MIT she's worked at the hedge fund D sha and also as a data scientist at various startups she also is active in the Occupy Occupy Wall Street movement especially in their alternative baking group she blogs at math babe org it's a great blog you should check it out as well as participates in the slate money podcast and she is the author co-author of two other books doing data science and on being a data on being a data skeptic please join me introducing Cathy O'Neil hi guys thank you so much for coming um so I'm a math nerd I love math I was attracted to math in high school because it was true it was beautiful like you could disagree with somebody about politics and everything about what manifest destiny really meant but you'd have to agree on math because you guys would be so careful with your assumptions and the neurological arguments I thought of as something wouldn't it be beautiful if we could sort of port this honesty and truth and clarity to other other fields of the real world so after I became an assistant professor at Barnard College I realized that it wasn't the right pace for me that I wanted to have like more more to do with the real world and it was 2007 so I did what everybody else does and I went to a hedge fund and and quickly realized that well the world blew up almost the moment I stepped into the hedge fund was working with the experts like Larry Summers I worked with him on couple projects it was very disillusioned by what the experts seemed to actually know about what was going on but I was most disillusioned and actually ashamed to be honest about one of the most fundamental of reasons that the financial crisis happened which was the triple-a ratings on the mortgage-backed securities which I consider a mathematical lie it was the opposite of how I had fallen in love with mathematics as something that would bring forth clarity it was the it was really it was hiding corrupt practices behind mathematics like trust the math be impressed by the fact that we have a bunch of PhDs in math who are supposedly you know carefully going through this data to double-check triple-check that this these mortgage-backed securities are very unrest key trust us there's math here and and invest in these in these very opaque instrument and I'm not saying that's the only thing that happened obviously the mortgage broker is like made a bunch of bad mortgages but the reason that they were able to build this machine and scale it to the to the scale the massive international scale that it became was largely because of the triple-a ratings because of the trust and that was trust in mathematics and it was abused I went into left finance after a small stint in risk which I could go into if you guys are interested in the QA where I was equally disillusioned by my the part I was playing in particular in in risk and I left finance altogether started my blog math babe the idea was to sort of expose corrupt practices and mathematical abuse as I saw it which would happen to finance in the meantime I needed a day job so I became a data scientist very easily I should say I just renamed myself a data scientist who got a job the next day because well because I was qualified because what do you do when your data scientist you predict people instead of predicting markets that have been predicting markets futures and credit default swaps of stuff so it wasn't that much of a stretch to predict people in fact it was very very simple and I wanted to think that what I was doing was like less destructive than what I've been how I've been feeling in finance but what I realized pretty quickly within a year or so is that I was just very much part of something that I still considered not not benign like actually negative and what I realized that was that the difference between the sort of the the havoc that was that was created by the financial crisis where everybody had noticed in the financial systems of the entire universe entire world were sort of at risk and everyone was worried was that the Christ the sort of the the the destruction that was being that was the havoc that was being wreaked and the data science world was happening at the individual individual level that individuals were being lost or they were being deemed losers or winners by this stuff and moreover what was most disillusioning and kind of frightening for me because I was still kind of ideal it idealistic was that these were all being again they were being sort of marketed as objective and fair these algorithms when in fact they were not that they were there algorithms are nothing more than opinions embedded in code so let me give you an example and you guys are technical so you'll understand this quite well this is my son I like to give the example of what is an algorithm but using the example of my my own internal algorithm for building for cooking dinner for my family so an algorithm essentially has two big choices lots of little choices which I'll ignore but the big choices are the data that you train your algorithm on and the definition of success or the objective function which also includes your penalty for mistakes but ignoring that for now so what's the data going into my algorithm for cooking a meal well the food I have in my kitchen the time I have the ambition I have by the way I should say I curate that already I curate I carried the food I do not consider ramen noodles food my teenagers do of teenage sons as well so I'm already imposing my definition of food ok and then the definition of success for me for dinner is whether my kids ate vegetables now my seven-year-old if he were in charge this guy right here Wolfie he would define success for a meal as if he got a lot of Nutella and the reason this matters is because over time we optimize our algorithms for success that's why we define it right we want to get closer and closer to that that success so over time that means I make more more meals that have been successful in the past I train my algorithm to success that is a very different succession of meals and my son's algorithm would be if we if we optimize on the tella so that's me that's a that's me imposing my agenda on to the algorithm we always do that there is no such thing as objective algorithm because at the very least the person building the algorithm defines success and that's usually success for them but there's often other stakeholders in the play and the question is is that success for the stakeholders as well it's not always it's generally speaking the patterns I found looking through more and more data science was that we were more and more talking like marketing these things as fair and objective and following the numbers and mathematical but there were people behind it who were building the definitions of successes objective functions and they were essentially invisible behind that mathematical shield a lot of these algorithms were benign or fine or great but some of them were really not great and I focus on those in my book I focus on those because I do not like vague discussions about what could go wrong with a bunch of big data I want to know exactly what is going wrong right now it's a kind of triage on this on the world of algorithms so I focus specifically on the worst possible algorithms if you're wondering why I'm so negative in my book I'm not actually negative I love data I love a I'm a data scientist still and I believe in data but I do want to make it very clear to the public and this book is written for the public that some of the times this stuff is really up and I define that with three characteristics a weapon of mass destruction is one of these really terrible algorithms they have three characters one is the widespread important okay so nobody cares about the algorithm for I make food for my kids with that is not important what's important what makes something important is it happens to a lot of people and it makes big decisions for them of course algorithms don't make good decisions to be clear it's humans that have set up processes by which a score of an algorithm it ends up being a decision but I'm going to you know let's just put that aside for now if people's life's options important life options like how whether they get a job where they go to school whether they get a loan how much they pay for insurance how long they go to jail if those kinds of things are determined in part by algorithms essentially mostly scoring systems then they're important they're widespread and important the second characteristic of things I'm really worried about is mysteriousness secrecy these scoring algorithms are in general are things that the formulas are not available for people who are being scored by them often they are not even aware that they are being scored and because they are important that is not okay because this they are tantamount to laws it is not okay for them to be secret we have a constitutional right to know what the laws of our country are and finally I care about algorithms not that are improving the world I mean those are great but that are destructive so I'm really focused on things that are destructive and I mean that in two ways like the first way I mean that is that they ruin people's lives unfairly so those people whose important life decisions are being informed by these scores they're they're getting unfairly booted from opportunities opportunities are unfairly being taken from them moreover they're destructive in a larger sense they actually set out to solve a problem in a larger sense usually often with good intentions but not only do they fail to solve that problem but they actually make the problem worse they create usually a destructive feedback loop and I'm going to give you a bunch of examples and then we'll have Q&A so the first example comes from teacher assessment we've had basically two decades-long war on teachers in this country the idea is we have a problem with education let's find the bad teachers and get rid of them and I will put aside for the now the question of whether we could actually solve our problems and the big problem being the achievement gap between rich and poor kids could we solve that by getting rid of bad teachers that's a different question the point is that we have been looking for these bad teachers so we can get rid of them for two decades the first generation of teacher assessment tools was really really stupid and unfair and here's how it was look for teachers a majority of whose students did not pass some proficiency standard and their standardized tests now this the thing you need to know to understand this is that there's a strong correlation between scores on standardized tests and poverty so poor kids just don't do as well on standardized tests that's not specific to the United States that's true over time that's true internationally but what it means is that if you if you target a teacher as if you label them as bad because a lot of their kids do not pass a certain threshold of proficiency you're basically targeting kids teachers of poor kids so that was clearly unfair so it was discarded the second generation was meant to be more fair and had it with good intentions but it didn't work and let me tell you what it was it was called the value-added teacher model and the idea here is that you can't really blame a teacher if a kid gets like a 60 in fourth grade and they're the fifth grade teacher you can't blame that they don't get a hundred the end of fifth grade right they're starting from sixty so it's just better if they get more than expected right so the idea was there was a primary model which is the expected score for each student in a class and then there was the actual score that that kid got at the end of the year and the teacher was taken put basically held accountable for the difference between those two scores so if their expected score was 62 but they got a 65 the teacher was sort of given credit for those three points if they got they were supposed to get a 62 but they got a 55 they were dinged for those seven points that they didn't get that make sense now the problem with this it actually makes sense right and if you had like a Google level test of this where instead of having 25 kids in the class you had 20,000 kids in this class then the statistics the statistical signal will be pretty clean the problem is we do not do not have that so we have instead is a pretty bad model which is estimating the expected score for a kid at the end of the year it's actually really hard to take a kid at the end of fourth grade and estimate what they're going to get at the end of fifth grade it's not a precise then there's a lot of uncertainty especially for kids in poverty then the second thing the second source of uncertainty is like what they actually got the end of fifth grade like they could have gotten more if they'd taken the test in the morning and said the afternoon or in an air-conditioned room instead of a hot room or after they ate a meal because they were hungry so there's one certainty on both those numbers and then you're taking the difference if you think about statistically that's the noise term it's called the error term of that expected score model so you're holding teachers accountable for the error terms of their students turns out there it's not a very good model it's noisy very very noisy now I don't know this because I have the algorithm because I looked I tried to get it I tried did a Freedom of Information Act request for the source code but it was denied me I ended up talking to someone in Wisconsin this is for the New York City virgin there's versions all over the country but I focus on the New York City version and um I talked to the people that in Wisconsin in Madison Wisconsin who built that model and they explained to me that I would never get the source code for that because in fact nobody in the in New York City got the source code for that because bike by the the contract it was stipulated that it was proprietary so no one in the Department of Education understood how these teachers were being evaluated and yet these scores were being used to deny people tenure so that's why so they should mention the reason why I thought I might be able to get the source code through a Freedom Formation Act is because the New York Post actually did get the scores with all the teachers names through a Freedom of Information Act request and they published them all as an act of shaming the teachers with that scores and I thought okay maybe I can get the source code if you can get the scores I can get the way the scores were made right now I couldn't but here's something that happened this really smart high school math teacher at Stuyvesant High School Gary Rubinstein he took those same numbers that the New York Post had gotten and he found teachers that had two scores for the same year you can get two scores if you teach seventh grade math and eighth grade math and he figured if they're both supposed to be like the final say on whether you're a good teacher they should be consistent right so if you get a seventy-five for seventh grade math you should get a seventy-eight maybe for eighth grade math right or a 72 so he plotted them and a scatterplot it looks almost like uniform distribution which is to say almost random it's actually 24% correlation there's some signal in it which is why I said the thing about like if you had 10,000 kids in your class you might actually know whether your teacher brings up grades scores but it's simply not good enough to hold someone accountable for that at an individual level but we trusted it anyway when I say we I mean the goe essentially because they didn't understand it in spite of that Sarah was soaked with sake who's pictured here was fired after she got a bad teacher assessment 50% of which was her teacher value-added model score in Washington DC in 2011 I should say it's 50% of her score but it's more than 50% of the variance of the score the big complaint of these of the way that teachers were assessed was that everybody got the same grade from their principal the principals would categorize them all as acceptable teachers so they're like we need some spread in the scores so we can like rank the teachers and we find the worst 2% so they got spread but the question was is it meaningful I claim it's not meaningful from the from the scatterplot we just saw but it was particularly irksome for Sarah because guess what she's teaching a fifth grade a bunch of fourth graders came in with very high scores at the end of fourth grade who couldn't read or write and it turns out that in the Washington DC district Michelle Rhee had instituted both carrots and sticks you get bonus if you get a high value-added model score or high teacher assessment you get fired if you get a bad one so moreover she found out that this school where some of her kids came from had exceptionally high erasure erasure rates for their end of year Saturday's test so she has reason to believe that the previous teachers actually cheated on to the test if you think about what that would mean those kids get elevated scores and they're expected to keep them elevated at the end of her class but she just got them she got she taught them well which they didn't get ridiculously high scores so she got punished for the previous teachers cheating I should mention that she got a new job couple days after she got fired in an affluent suburb of Washington DC because most of the most of these this is where the value-added model is being used I think it's actually more than that at this point mostly in urban school districts so that brings me to the the failure okay remember I said there's failure on well let me just go back to over the three things it's widespread secret and its destructive at an individual level so we saw Sarah gating losing her job and at a systemic level I claim that the value-added model does not get rid of good bad teachers as remember the whole point was get rid of the bad teachers then we'll solve the problem with education we're not getting rid of bad teachers we're getting rid of good teachers we got rid of Sarah which was just dumb but there are countless teachers who have quit who have retired early who have not gone into teaching we have a national teacher shortage or have fled to the suburbs where they don't use this arbitrary and punitive regime for teachers my next example comes from the world of personality tests this is Kyle beam he was a college student in the Atlanta area he wanted to get a job after school at a grocery store Kroger's and he was he took a personality test he failed it most people never find out they failed it but he found out because his friend worked at Kroger's told them so he was unusual in finding out that he got redlighted from this algorithm was which was built by Kronos a small Big Data company actually in this neighborhood the other thing that was unusual about Kyle is that his father's a lawyer most people that are buying for minimum wage jobs which this was do not have fathers that are lawyers so his father I said why what were the questions like on this test and Kyle said there were a lot like the questions I got at the hospital when I was being treated for bipolar disorder the five factor model let's go and his father said well that's illegal under the Americans with Disability Act it's illegal to have a health exam including a mental health exam as part of a job application and he said well can you apply to other places Kyle ended up applying to six other places Lowe's young food which owns kettle bell other big companies they all had exactly the same personality test and he failed all of them so his father suing those seven companies on behalf as a class-action lawsuit on behalf of anyone who's ever taken that test on the grounds that it is illegal and it violates the American with Disability Act so that's widespread it's secret I should mention personally tests are not new radioshack old old sort of easily gamed personality test looked a lot like this agree or disagree I'm always happy I think your employer wants you to agree with that these are the more recent ones much harder would you agree with more I sometimes get confused my thought by my thoughts and feelings or I don't really like it when I have to do something I haven't done before imagine doing 50 of those so it's it's absolutely inscrutable so it's secret and I would say its destructive its destructive for Kyle because he didn't get a job at any of those big companies he applied to in the entire area of Atlanta but it's also destructive in a larger sense if what we're worried about is true that they're actually sort of systematically denying employment to an entire sub population people with mental health problems um it's not relegated to minimum wage work this idea of having algorithms that sort through resumes do you guys know who this is this is roger ailes he recently got kicked out of Fox News he led Fox News for a long time I got kicked out for like six basically sexually harassing women and keeping them from pre any promoted so it's doing a thought experiment I don't know the Fox News uses an algorithm to hire people but I'm just imagining that they do play along with me what does that algorithm take in well it takes in data probably historical data from Fox News so all the history of people who've applied to be say anchors at Fox News and then it takes a definition of success what does a good employee at for successful employee at Fox News look like let's just say let's stipulate a successful employee of Fox News stays for three years it gets promoted at least once okay that's typical for what how these algorithms are designed and how they're trained so if you think about what that means the algorithm if you've trained it on this a lot of historical data what it will do is given a new set of applicants for anchor job it will say who along these new applicants looks like someone who was successful in the past and I would not be surprised if it filtered out women because not only did women not not only did women not necessarily get that job as often but when they did get the job they were systematically pushed out by Roger Ailes and the entire culture there does that make sense so the point here and this is a very important point is that we do not improve we do not we do not algorithms are not inherently objective or fair all they are doing their what they're really good at is picking out past patterns and repeating them which is to say if we had a perfect way of hiring people we would want to codify that we would want to automate that because it would save us money in time but until we have a perfect way of doing this all we're doing is we are literally codifying past practices and propagating them my last example comes from criminal justice there's actually two different kinds of algorithms that worry me a lot in the in the criminal justice the first is predictive policing you guys know all about the black lives matter movement which is objecting to black people getting shot but it's not just of course being shot it's just the amount of over policing and uneven policing that's happening to poor black communities some evidence there's a whites and blacks smoking pot at a similar rates white smoking pot a little bit more if they're young but blacks get arrested a lot more a lot more for that in fact depending on the jurisdiction blacks can get arrested up to ten times more often than whites for smoking pot okay so the two things I want to take away from this are blacks it's it's biased against blacks and actually it depends very much on local conditions of how the police force is expected to act so when you hear about arrest records I want you to think just as much about police practices as you do about crime at least when it's non violent crime and I do make a distinction between violent crime and non violent crime the problem is that non violent crime is much more prevalent and it is much more predictable think about it we have crime we have people being arrested in this country for mental health problems when they're poor not when they're rich we have people being arrested for addiction problems when they're poor and we have people being arrested much more often for low-level nonviolent crime like drug use and work from that for that matter for like crimes of poverty just like literally not having a place to go to the bathroom and that is that goes into their arrest records and the reason I'm mentioning this is because the way predictive policing works is very simple it's actually very stupid it's geo-located arrest Crest Records and the algorithm says go put police where we saw crime in the past which is really where we saw arrests in the past where we saw police arresting people for things in the past this is just much as much a way of predicting the police as it is of predicting crime I should just I'm just reiterating that it might be different by the way if we really focused only on violent crime imagine that imagine a predictive policing algorithm where they only focus on violent crime like murder okay the problem is that murder is really hard to predict like even if you did predict murder would you like what stand like outside a house waiting for someone to get murdered like the problem is that the things that are actually easy to predict are things like poverty so my claim is that predictive policing is more or less creating a feedback loop where you're having a scientific pseudo scientific basis for sending police back to neighborhoods that are already over policed said another way another thought experiment imagine that if after the financial crisis all the cops have been told to go to Wall Street and arrest the bankers and find out if they had cocaine and their pot in their pockets because they all do not all of them then the predict then the police records the police data the data which is just a reflection of what police do right the data would tell them in these predictive policing algorithms go back to Wall Street because that's where the crime is but that's not what we did so the next example in also in in the criminal justice system is recidivism risk this is a score recidivism by the way is the coming back to jail so we're sort of as a risk is the risk somebody comes back to jail recidivism risk scores are given to judges when they sentence criminal defendants now there's two kinds of data that goes into that one is arrest records we just talked about arrest records they're very biased the second is a questionnaire the most commonly used recidivism risk algorithm is called the LSI R and I'm going to show you a couple of the questions that are asked of the defendant in the LSI are number twenty-nine do you live in a high crime neighborhood it's a proxy for class and race because that is pointing to people who are already poor in black and and by the way if you say yes or no to these things it goes exactly as you imagined if you say yes to I have a mental health problem you're higher risk right and I forgot to mention if you're a higher risk well a judge will sentence you to longer in jail which doesn't I mean I mentioned that because it's not obvious if that's what you would do right like if you're higher risk of recidivism you get put in jail longer it's a little bit Minority Report ish because what you're doing is you're preemptively punishing for someone for something they haven't done yet but that is the practice that judges now have here's another set of questions have you been suspended from school number 17 I'll show you a plot of just how much that is a proxy for race black girls and boys are much more likely to be suspended from school but I think the thing that bothers me the most and I think should bother absolutely everyone is number 26 with somebody in your family in prison this is something that in an open court would be thrown out by a judge has unconstitutional if a lawyer said your honor please sentence this person to longer because their father was also a criminal that is not how we do it but because it's being embedded in a risk score which is being claimed to be scientific this somehow has the authenticity of mathematics and science and again is being used to send people to jail for longer so I talk about failures I should also mention this is important it's being used in more than half the states it's secret people do not understand what their what they're getting into with these scores and judges are actually very secretive about exactly how much weight they put on these scores but my claim is that they are most destructive thing you can imagine they create their own reality if you're a higher risk you're sentenced to jail for longer guess what if you're sentenced to prison if you're imprisoned longer you don't tend to be to benefit from that experience you end up out of jail 97% of people eventually leave prison I should mention so this happens but you end up with no resources no connections to your community very little wealth you have a felony to your name often so you it's hard to get a job and then you end up back in prison partly because you got this high risk score so you were deemed high risk and then you end up back in prison so I'm almost done those are my examples but I do want to mention again that I don't hate data I just really think we have not yet yet started understanding what it means to build safe algorithms it's like we are building cars and just putting them on the road without understanding that cars can kill people so I want data scientists to take their ethical responsibilities seriously which means building some kind of ethical framework like a Hippocratic oath for data scientists I also think that we need to learn how to scrutinize these algorithms to monitor them to audit them for safety for safety for fairness and discrimination and for meaning making sure that we're actually building meaningful things with the teacher value-added model was not meaningful and I also think that in in situations where it's very very important to a given person's life how they're being scored they should have the right to scrutinize that score like the teachers like if they are being given a score that will make maybe make them fired they should be able to understand exactly how that scoring system works because to be clear the teachers were not even when they appealed their score they were not told how they were being actually evaluated and that's not right thank you guys if you have any questions I'm here I what's the best way to account for and mitigate limitations in a model so suppose you you have to come up with a mathematical model to affect somebody's life how do you figure out where the boundaries of application are and how do you I guess iterative Leary address that to see if it's doing what you expected to I mean it's really a vague question so it's hard for me to answer I think I think it's like I'll tell you how how frustrating it is as a data scientist I'm a data scientist like I worked in the City Hall of New York and I was asked to use the data they had to figure out how long a family was going to be in homeless services and I had race I had the number of children I had you know out how many times whether the children whether the parents had been in social services I had all this data but they weren't telling you they weren't telling me how they were going to use this algorithm if this scoring system once I had it so in particular one thing I didn't know whether I should use was the attribute race or all the other things that are proxies to race or should ID correlate those other things from the race how was this going to be used number one possibility if you were high risk then you're going to be put into worse housing if you were like you know expected to be in long-term housing you're going to be put into worse housing if I did that if I if I knew that that was how I was going to be used then I would know that people at higher risk there'd be a disparate impact racially right but if I if instead it was being used to sort of figure out interventions so that you know black family they were maybe wanted to understand why black families were in homeless services for longer and to intervene to try to make that discrepancy smaller that it would by all means make sense to have race as the attribute does that make sense so like I'm basically not answering your question but I am making it clear that you cannot answer that question until you really know the use case for this algorithm and I should also add that a given algorithm could have a positive or a negative effect on the world it's really tricky these things are let me give you an example um health everyone's talking about how great it is that we can predict each other's health well that's great if your doctor has that algorithm and can keep you well but it's not great if an insurance company has an algorithm we can charge you more if you're about to get sick or if Walmart and I'm not saying this is happening but if like Walmart puts a health risk on top of everybody who's applying to their job and says we don't want to spend people money on our peoples insurance so we're going to not hire people with high health risks I'm just saying same algorithm different use cases could be used for good or bad it's so the answer that the answer is there is no rule here these are decision made algorithms our decision-making processes that are as complicated as anything in the world so we can't we can't pretend that there are like formulas for how to use them and how to make them safe they're very very contextual do you have any sense of is probably a naive question but um whether the scale of the problem is bigger or smaller in a country with stronger libel and slander laws like for example England say more why would you imagine that up what why would I imagine it would be different yeah ah well I guess the people who are victims probably don't have the resources to pursue slander cases anyways I think there's one of the a very important point thank you for making it which is that I'm specific one of the one of the commonalities of almost all these algorithms is that the people who are losing by these scoring systems are often the most vulnerable people in our society so they do not have generally speaking they don't have lawyers to protect them right but I would also say the following like anonymized anonymization which is a tool that people often bring up as like a way of solving some of these problems is not really a solution to me let's think about the Pikul the medical model I just mentioned like let's say I'm Walmart and well again what Walmart's not necessarily I'm a large employer and I get long-term health data off of all my employees because I force them to use fitbit's or whatever idea right I get long-term health outcomes and I don't charge them I'm their Harding my employees I treat them well right but the point is I can build the algorithm I can build the neural network whatever have you I can train it so that when someone applies to my job I only have to ask them six questions and I've already categorize I've segmented them into a into a risk group score and it is completely anonymous and in other words you can build an algorithm anonymously and still applying it to someone in a very precise way and it still can be problematic so it's really not for me it's not about anonymity so in a lot of the reporting that I have read about this sort of thing and I'm thinking specifically for example the profile got a long thing on the criminal justice the sentencing thing they haven't seen a lot of talk about how these algorithms perform relative to in this example just a judge without this you know score or what the the pre-existing case was yeah great point and I'm desperate desperate for that right because the point is that the justice system is very racist all righty this the intention of this recidivism of stuff was to make judges more objective and less racist and it might be doing that like with all the flaws that I just mentioned which I really believe in it might actually still be better than what we have already we do not have data for that and the Department of Justice is not coughing up that data I've tried I've tried to get I've tried to get data to do that an audit of the DOJ it should be it's possible they didn't exist because some jurisdictions have this stuff in use and some of them don't you could just compare them before-and-after or whatever and compare to each other I should also but I should also add that like here's the good news like if we made those recidivism risk algorithms actually not racist then that would definitely be better than the current judges we just it's like we threw them out there and we're just like oh they must be good because their algorithms they're by definition good no they're not they're not necessarily good but again you're right they might be better than the existing system so in the other examples that you talked about do you have similar sense of whether that data is out there like the teachers and you know and maybe the teachers is advantage the teachers are is it's like just terrible you're not even trying to solve the right problem you're not solving the room and I don't think by the way I I still don't think recidivism risk being high shouldn't necessarily mean you go to jail longer right maybe we should be like why how do why do we send us people the way we sent his people like we should maybe make a redo of that entire conversation like we should if we were thinking data-driven went lately if we're like if we're Google thinking people like we should ask really basic questions like to what extent does GED training in a prison help people when they leave to what extent does solitary confinement help or hurt people or sexual assault like we should actually know what these things are doing to our final outcomes which hopefully is a combination of public safety and that well-being of the actual prisoners we don't have any of that yeah I entirely agree with your with your wonderful notion that that that some of this data is despaired I mean it hurts hurts hurts vulnerable communities but despite that like the thing you most row tension to it was the fact that people whose relatives are criminals are judged more likely to commit crimes well people whose relatives are actors are more likely become actors people's love those of plumbers are more likely become plumbers and even though it's a horrible thing I strongly suspect that someone who's who someone whose parents grew up by stealing cars know a heck of a lot about some more about stealing cars and some ones who don't what do you suggest I suggest we follow the Constitution which privileges justice over anything like that so just the way our Constitution says and send the police to Wall Street because it's equally likely to have muggings at least as a culture we decide what is what is against the law but as but the Constitution declares that we have to care about fairness to the criminal above all and then this is not fairness to the criminal by saying you're implicated because your father was implicated I just also want to add that like I live next to Columbia University in every couple years there's a huge drug bust in the fraternities right like a huge drug bust and then and then I followed the NYPD and they're constantly boasting about their data-driven criminal justice stuff and how they peg people as gang members if they associate it on social media with gang members but those gang members are always in Harlem and they're black you should I mean they're not Columbia students so it's also an inconsistent Lee defined Association so there's two different kinds of problems okay actually thank you yep so I think it's a it's very easy to see that if you're not careful about how you define what kind of success you're optimizing for you wind up with an algorithm that very strongly reinforces the status quo and if we're looking at things you know like our hiring practices fair then if there is existing unfairness we're going to perpetuate it so that that makes total sense to me what what I struggle with is if I'm asked to do something like that and I try to not incorporate the existing structural biases into the algorithm it's coming I do you know recommended hiring practices thing how do I deal with the challenge that I'm actually optimizing away from success conditions okay so there's two strategies I suggest the first one is basically a parable um so there's this thing called the blind audition for orchestras right where the orchestras were trying they acknowledged that they were being nepotistic and their practices of hiring so they wanted to get rid of their nepotism so they decided to put a sheet between their judges and the audition er so they wouldn't know if the person who is behind the keshite was their friend at first they saw the shoes and then they're like wait a second we can see if it's a man or woman so they brought the sheet down to the ground and they also installed rugs in the hallway walking up to the spot so they couldn't hear if it's high heels or not not only did they get rid of nepotism but they increase the number of women in orchestras by a factor of five I would so that's the story and like the character the way I would characterize what they successfully did there is to two things first they decided a priori what was it that was a requirement for this job how do you assess somebody for this job and it was the answer was sound the second thing they did which was absolutely as important as the first thing was to ignore everything else the promise of big data that we get confused by because it sounds really convincing what's wrong the promise of big data is you the more the data is the better just throw all this data at the wall correlations are just as good as causation not true because if we did the orchestra example again but we didn't have the sheet and we thought to ourselves we're looking for good sound but we also knew that it was our friend or we also knew that it was a woman that creeps into us it creeps into our brain that's excess information that we should be ignoring but we're not ignoring so the first strategy I would suggest for hiring people is to think a priori to actually build a model that you buy but you that by construction is fair so build the model and say these are the things we actually want for this job the second possibility is use a machine learning algorithm that takes all these correlations Ambala blah but then audit it for fairness then say wait does this basically filter out women check check to see if women are filtered out check to see whether they're filtered out in a reasonable way sometimes they are sometimes sometimes you had like I heard the story on fit on Twitter just the other day we're way more Asian people were filtered out of a certain job and people were complaining because so many of the applicants were Asian but then people were saying well the people who there were more Asians that were applying that weren't actually qualify for this job like it's complicated and expected to be complicated that's my third suggestion is that it the the other promise of big data that I'm pushing back against and I hope you understand and agree with is that big data is not a silver bullet it's just a tool and is not automatically going to solve our problems it's just a tool that we might be able to use to solve some of our problems but we have to be careful about it we have to check yes stories where you would go to a judge or a jurisdiction or a school system and they would say oh my god you're right we have to fix this and they do sorry what do you have any other success stories like the orchestra where you might go to someone and say you know this problem and they say oh my gosh as opposed to no you can't see our data now that's proprietary go away don't bother us you'd be surprised how few people want me to see their source code although I should say that like someone asked me whether I thought it would just be impossible to ever get a good value out like a good teacher assessment tool using data and I said I'll never say never because like ten years ago I didn't think we'd have self-driving cars but I think now that we're going to save lives from front driving with self-driving cars but what is it going to take to get a good teacher assessment tool is we're going to have to have a lot of evidence that this is actually working right now we have no ground truth for teacher value-added model literally there is no comparison to some other kind of qualitative assessment to see whether a teacher is good a good teachers getting a good score and a bad teachers getting a bad score it's just whatever the number is is your score which is ridiculous right so we actually do have qualitative assessments for teachers they're political because different people disagree about what makes a good teacher but like let's say we had agreed meant we're going to we're defining this kind of qualitative assessment to be important to be the the ground truth and then we could try to find you know maybe we put a bunch of sensors in the classroom we try to replicate that using data literally replicate it and if we got an algorithm that after 4,000 teachers got almost exactly the same score using the qualitative assessment is using this data-driven assessment then we would build trust that this data-driven assessment is doing a good job it's very similar to saying you know we have this many miles of a self-driving car without an accident so we start trusting that car that algorithm does that make sense right now we have nothing like that in the in the field of teacher value they just want a silver ball it that's all they want they don't want any prying eyes either something that's kind of come up a bunch is a sort of as an undercurrent like the idea of insourcing how you're doing things and I can see that like potentially being really helpful for example now you can actually criticize because you can see the code but I can also see it being somewhat ineffective if you're still doing if you're still asking questions that are totally beside the point um so I guess I'm curious about your thoughts as to have the effectiveness of things like open sourcing and also the limitations um it's a great question and you guys know that it's hard because if because it's really it's relatively easy to make something arbitrarily difficult to understand if you know in advance that it's going to be open source you could just make it impossible to understand and like the other example I give is like nobody would want the code for Google search because literally probably will not work on any computer system except Google's computer system right so there's a limit to what open sourcing can do but I don't that that's not to say I think we should give up I think what we should ask for is in auditing a fairness auditing and I think we should think about along the lines of the way sociologists other things for fairness so like sociologists will see whether a hiring practices racist by sending a bunch of applications with black names and white names and like similar kind of qualifications and see whether white people get more callbacks so we can do that kind of thing to algorithms and Latanya Sweeney did that for Google search you guys know that example Latanya Sweeney googled her own name and she found the ad next to her the search results was are you looking for the criminal arrest records for Latanya Sweeney then she googled a white name it didn't happen then she did a comprehensive test because she's a computer scientist and found that black names were way more likely to have a rest records next to them this add for arrest records now that's not in my opinion a weapon of mass destruction because it's not as directly impactful in people's lives but it's certainly not not great good thanks um I was just going to agree with you before when you were talking about there's no easy way to do the teacher evaluation I think it goes back to your point of success right because is it we then it gets into like student evaluation and our you know the assessments that they do on students or is that just a proxy for success in long-term you know their long-term career so I think the teacher evaluation there's probably no right answer do you agree yeah it's it's a very tricky problem so it's I think the trickier the question is that much less likely it's going to be solved by some simple scoring system yeah a lot of the machine learning algorithms that are very popular these days are very like neural networks which not they're not designed they're opaque you know without even battling bad intentions like you know if you click on the little Y button about why it made it give a decision to say I did these three thousand matrix multiplies and there's no simpler answer than that but they do give much better predictions than other things like decision trees which are much more interpretable do you have any thoughts but how in the context of these problems too should we you know you do give a needle early worst predictions because we need them to be interpretive oh yes absolutely okay that's probably the hardest thing that you guys are going to hear from me today but I definitely think we need to sacrifice accuracy for fairness absolutely and one of the reasons is because we as technologists we are not the ones that are at risk so our concept of what looks fair to us not good enough right we might have some better understanding of how things work but we're like oh but it makes it so much more accurate and I actually I talked to somebody who does recidivism risk algorithms for a living he does it for this for a state and I said well do you ever use race and he was like oh no that would be wrong I said well do you ever use zip code he was like well sometimes because it makes it so much more accurate so now I'm like okay but that's the proxy for race so you're basically using race so another way of saying that is if it's interpreted well then people it's much more easy for other people to say that's not fair it's a transparency measure there is precedent for this by the way if in credit card law if your credit card company and so you deny someone a credit card you have to be able to explain why which restricts people to using decision trees for the most part they don't like it but that's okay there's actually lots of things about lending that are a traitor for fairness and accuracy so FICO scores by law anti-discrimination laws called freedom Fair Credit Reporting Act and the Equal Credit Opportunity Act make it illegal for you to base FICO scores on race or gender or zip code is that because it's let it's more accurate when you restrict now like it would it's it's actually it's keeping these credit scores for being as accurate as they might be but it was deemed more fair that way as a society we care about public the public right so the idea is and this was this was soon the basicness happened in the 70s when like especially divorce women were being denied loans and the idea of there was if you'd never let women get loans then they're never going to be able to build up their credit scores so that'll be credit worthy so it's like a feedback loop right and so this was these anti-discrimination laws were specifically designed to prevent the feedback loop we need that we need that kind of thing because algorithms like that algorithm I just talked about that benefits the the banks or the people that make loans to the detriment of the public so we have to balance the the benefits of the for the public versus the benefits for the private companies I have time for one question so I was kind of say I'm in the opposite boat where I work on a search feature that looks at lots of queries and we have a model for English where the masculine pronouns are the default and some common names of the default so it the model knows a lot about that we're kind of wondering if queries that women like we do terrible on or worse or there's some class except we don't know anything about the people who ask the queries we've tried really hard not to know I don't know gender I don't know income I could probably get a zip code in bulk so I'm sitting here I'm trying to think I would read a machine learning model that would do the reverse question and figure out what's the most unfair thing my current system is up to that's a good question to be asking yourself but I would also add and thank you for asking that to yourself I think that's what we all need to ask because one of the first questions we should be asking ourselves but I do want you made the point yourself but I'll just reiterate it and something that I'm certainly not the first person to make this point sometimes you actually do need to know these attributes like race and gender in order to measure your fairness so I understand that the desire to be race neutral or gender neutral but that doesn't mean you should not collect that data because then you're basically saying I'm not going to look you know like it must be fair because I don't I don't collect that data that's not proof that it's fair which isn't to say that you know it's easy to collect the data okay I'm saying maybe collective at least in for some experiments that's an audit thank you guys you