Shoshana Zuboff on Surveillance Capitalism & Democracy

[00:00:03]you good evening and thank you for joining us it's my great pleasure to be here tonight for a conversation with Susanna Zubov she joins us to discuss what it's like to be living in an age of surveillance capitalism so welcome dr. Zubov and let's just say the title of your book crystallizes a lot of the fears that people have that maybe not all is right in Silicon Valley could you expand on what you mean by surveillance capitalism what is surveillance capitalism indeed let's talk about that for a moment and then the rest of what we're going to discuss tonight I think will make a bit more sense so people have long observed in the history of capitalism the way that it evolves and the idea is that it evolves by taking things that live outside the market dynamic and bringing them into the market you know the term commodities bringing them in to the market to turn them into commodities that can be sold and purchased so famously industrial capitalism claimed labor sorry claimed nature for the market dynamic bringing it into the market dynamic to be reborn as land or real estate that could be sold and purchased it claimed work for the market dynamic the kinds of activities that people did in their homes and their cottages and their gardens and their fields bringing that into the market to be reborn as labor that could be sold and purchased well surveillance capitalism diverges from many aspects of the history of capitalism but in this way it actually emulates this earlier pattern but with an unexpected and even perhaps we might call it a dark twist in this case surveillance capitalism claims private human experience for the market dynamic and an end in claiming private human experience it treats it as a free source of raw material to be translated into behavioral data these behavioral data are then shipped through pipelines as it were into a new kind of manufacturing process this manufacturing process is characterized by something that Jim is an expert in what we call today artificial intelligence machine learning machine intelligence and with that combination of these flows of behavioral data and machine intelligence what comes out the other end are predictions predictions of human behavior and I call these prediction products these products are then sold in a new kind of marketplace a marketplace that trades exclusively in if you will behavioral futures kind of like pork bellies behavioral futures now what I've just described to you may sound very science fictiony mmm how does it work but indeed when you think about the online targeted advertising and how that works what I've just described is really a fitting depiction of that whole process because in the game of online targeted advertising for example the prediction products the the the predictions of future behavior were really predictions of what they call click-through rates of who's likely to click through on an ad and through the pages and so on to make that advertiser money and the what is the what is that what are the online advertisers actually buying what they're buying are these little bets on what future behavior going to be indeed this whole logic was first discovered invented elaborated at Google back in the early 2000s in the heat of financial emergency of the of the dot-com bust and when when Google proposed this new logic to its advertisers they were very reluctant to accept it because now here was a prediction product coming out of a black box and the idea for the advertisers was you know we're not going to tell you what's in the black box that's our stuff but if you just follow the advice of this this computation this computational product coming out of the black box you will make a lot of money and eventually they decided to accept this and of course the rest is history and of course they had refused to do advertising in their early day so somehow their economic imperative overcame their principles against doing advertising well said Jim indeed because the Google founders and indeed the seven people who then worked inside Google who were relegated to the advertising function they all pretty much detested advertising they said it would disfigure the web and they said it would disfigure the search engine and it did but as you pointed out in the heat of emergency the the idea of actually going bankrupt investors were threatening to pull out their their money and that trumped all other concerns at that point so what we know about the success of this economic logic implemented at Google secretly between 2000 2001 and 2004 when a type II ode and the world first got a look at this new logic we know that during those years Google's revenue line increased by three thousand five hundred and ninety percent so there's some pretty good evidence of the power and the profitability of a unprecedented economic logic again based on the unilateral claiming of private human experience and may I may I complete my answer wait one more sentence ish yeah you're so kind so you may ask well why call it surveillance capitalism what's the surveillance got to do with it and what the surveillance has to do with it is that it's not easy to unilaterally claim and take private human experience you know we we are individuals who have a sense of privacy have a sense of personal Brown Dre's who want a say in how those boundaries operate and if those boundaries are to be breached then we want to be the ones who decide we want those decision rights and so it was very it was early on it was understood quite quickly that if we're going to take private human experience as the raw material for behavioral data we have to do it secretly and early on in the descriptions of early patents and so forth the scientists were were quite quite explicit about the fact and even kind of celebrating the sophistication of the methods they were developing that they could find and capture personal information without the quote users knowledge and that they could find and capture various different points of user information that was lying around the web and put it together and make inferences about users that was stuff that these users never intended to is closed so this one-way mirror was there there from the start and indeed the social relations of the one-way mirror were essential to the financial achievement but that's not the story that Silicon Valley told everybody right they told a story of human utility we're finding things better for you or routing you better you're finding the right products so so how did that that goal of human utility making human beings more powerful how did that go from trying to do it's a good on purpose - let's say evil by accident how did how did that come to pass well I would like to not say I'm not going to use the word evil okay at first very specific reason I've described the bare bones of an economic logic and inside this economic logic over the years that I have studied it what I understand is that there are imperatives that develop in the competitive intensity of this economic logic economic imperatives emerge these imperatives compel certain kinds of action and so what we have now are corporations and many many people inside these corporations who are compelled to act in order to fulfill the requirements of these economic imperatives so to me it's very important to distinguish we're not talking about evil people we're talking about people who have now who have now committed themselves to to a machine if you will to a logic which compels certain kinds of decisions certain kinds of action certain kinds of operations these operations have some fairly dire consequences for things that we care a whole lot about like human freedom democracy and so forth I know we're gonna get to that down line well I think as you know in the startup ecosystem if you wanted to raise venture capital the whole thing was not about what you're going to do is like how many eyeballs were you going to capture it was so so everyone got the message if you want to get funded tell us how you're gonna capture a whole bunch of information about human beings people yes yes indeed so that's a good segue into talking about the economic imperatives would that be Oh that'd be a good place to go to hear how this how this economic imperative leads to some of these not-so-great outcomes yeah okay so yeah so this is you know this book which if I could turn my head I think is somewhere over there this was this was a many year labor sometimes of Love's sometimes in my own compulsion and over this time I began to discern you know how what these economic imperatives are and how they work and clearly when surveillance capitalism first emerged it emerged primarily in the online world in the virtual world around these activities of search and browse and so forth and early on it was understood that if we're gonna get enough enough behavioral data to really feed the machines in order to get good predictions then we need a lot of data Jim we need a lot of data you know that the machines the machines are omnivorous and so the idea here is that - we want predictions that approach certainty right the best predictions are almost as good as observations so ideally you want your predictions to be on a on a trajectory toward certainty and if we're going to do that we want data to be on a trajectory toward totality the more data the more totalistic the data the more we're going to achieve certainty all right so that introduces imperative number one I call it the extraction imperative we need to take everything and that's what you're you're mentioning there Jim we need to take as much as we can possibly take and this requires economies economies of scale volume data at volume and by economies of course I mean you know the machines do it so these are automated systems that you know again at the beginning as you search as you browse automated systems that are tapping into your experience pulling it in and converting it into data that can be read by machines so we begin with extraction and economies of scale a lot of data but then you know the competitive intensity continues to build and as that intensity builds it becomes clear that yeah we need a lot of data but we also need different qualities of data we need varieties of data we need scale but we also need scope and to get scope you remember when folks first started talking about the mobility revolution right this is get up out of your chair leave your desktop leave your laptop get out the door and what are you taking out the door with you your phone your smartphone so now this is to get out the door and we begin to get scope qualities new qualities differentiated forms of data as we get out the door we get where you're going and what you're doing and who you're doing it with and what you're saying and how you're walking and how you look and what you're buying and what you're eating and eventually we get with facial recognition you know we get the thousand muscles in your face that help us predict your emotional state and then we get not only where you're walking but how you're walking and if you're slumped over or if you're standing up straight your gait your posture when we listen to your voice we get the cadence and the velocity and the decibel levels and so forth so we're getting all of these rich data now both as you range out and about into the world but also down into the depths of yourself your body your emotions your all the things that you're signaling with you need all of that to route us from point A to point B well what do they need all of that okay so here's the point okay here's the point you're touching on the notion of behavioral surplus here because early on the distinction was okay there's a whole lot of these data that we can use for improving our products and services that's what we all wanted in the first place wasn't it you know I make the search engine better and give us translation functions and you know allow it to work more faster more effectively and so forth so there's a lot of data that was and is used to improve products and services and that's cool but then it was discovered that from these data one can draw signals if you were if you will signals sometimes this is called metadata but signals that that have tremendous predictive value so let me give you an example when we talk about for example the data that we put on Facebook so we we write posts we write messages we give data in our talking to our to our friends and our family and other people on Facebook we we provide data we put data on the Facebook platform but what's what's really predictive about the data that we provide is not so much what we say but how we say it for example you've written some sentences in your post are the sentences long and complex or are they short and simple do use bullet points do use exclamation points these question marks if you make plans to see your friends this evening your friends and family do you say I'll see you later or do you say I'll see you at 7:45 if you post photos it's not just the photos that that you who you post photos of it's that you post photos it's the saturation level and the colors of the photos that you post in your voice for example it's not just what you say but it's the intensity of your voice it's the velocity of your speech so you're getting the picture here so it's extracting these qualities of data that really go beyond what they need for service and productivity and this allows them to be somewhat disingenuous and say we don't we don't sell you're exactly what they're selling is the data that is more that I mean what they're collecting for their predictions is the data that is more than what is actually needed for product and service improvement they sell the gold instead of the or indeed so these data are what I call the surplus the behavioral surplus and it's in these behavioral surplus data stores that we find the great predictive qualities and of course if you fast forward a little bit may I just to an example that might put a little bit of meat on these bones cuz I think people are getting the idea that there's something wrong here but they're getting the feelings of what the outline is so okay so this is like it was discovered for example as as as folks we're collecting these behavioral surplus data it was discovered that you could predict a person's personality traits with behavioral surplus and this there's a lot of research on this just it didn't just come from the companies but you know scholars were researching this and scholars discovered that with things like you know your exclamation points and the saturation levels of your photos and some of the things I've just mentioned to you they could predict as well as human scorers where you fit on this five factor personality model that psychologists use to talk about personality I'm gonna say personality with finger quotes because I actually you know I'm a studied social psychology is a PhD and I believe that personality is something far more complex and deep and profound than five factor personality model but if i factor personality model lends itself to what automation automated scoring economies of scale so so anyway with behavioral surplus you can predict where people are on these on these personality models now it is this beginning to sound familiar to you because there was a little-known company that hailed from Cambridge England right and we're about to celebrate the one-year anniversary of Chris Wiley's whistleblowing activities that the world to what Cambridge analytical was up to and its manipulations of the electorate both in our country in the UK and in many other countries that was at least 15 that that I've looked into what did they do they took exactly these methodologies and mechanisms they had a guy who got 87 million people's worth of data from Facebook they tracked it onto the five-factor personality model they got psychographic data on all these millions of people and with those psychographic data they figured it out that they could in create little trigger messages that would as Chris Wiley put it stir up the inner demons of these folks because they understood so much about their personalities right so what they did was they took methods and mechanisms that had become sort of routine day-in-the-life ho-hum please pass the salt right the everyday methodologies of any self-respecting surveillance capitalist and they just pivoted them slightly from commercial outcomes let's direct you in a way where it's going to help us make money to political outcomes let's direct you in a way where it's gonna produce the elections that we want for the people who have bought the expertise and the data to make this kind of intervention possible so so the challenge here there is that we've moved beyond selling products or making products that are better from a commercial standpoint to actually trying to influence the behavior of human beings to do something different in the political sphere so this could be a this could be a real instrument of social control in the wrong hands all right so you've given me the perfect segue once again okay I've discovered the world's best moderator now you have to travel with me everywhere I go alright so I began to describe the economic imperatives to you and we got through extraction and scale and then we started talking about scope I mentioned to you that we see these imperatives emerging as the competitive intensity keeps heating up and in order to explain a little bracket here in order to explain what I mean by competitive intensity I should say that while surveilance capitalism originated in Google it didn't just stay in Google you know that right so it quickly migrated to Facebook it migrated to Facebook with Sheryl Sandberg who was a you know an early executive at Google was very much part of the discovering elaboration of surveillance capitalism and she was the one hired by Mark Zuckerberg to bring this advertising know-how which we now understand as surveillance capitalism to Facebook which she did very very effectively so then we had a duopoly but once the success of this logic was out of the bag and I know that you've seen this in Silicon Valley gem this logic quickly became the default model in the tech sector in Silicon Valley because you know what they had discovered was through the as the crow flies fast-track to monetization yeah if you weren't doing it you were behind if you weren't doing it you were behind now if you had done you know how that phrase some creative destruction to the beloved phrase in Silicon Valley yes creative destruction well you may or may not know that that phrase comes from the renowned Harvard economist Joseph Schumpeter so I say he has the sort of biological model and he wrote about creative destruction it's true but he wrote about creative destruction as a tiny little piece of a much larger theory and his theory was that capitalism evolved number one so it's like kind of like biology it involves like organisms and and it evolves as a result of a creative response to a new situation an unfortunate consequence of a creative response is that there's sometimes some creative destruction response was really the big deal and in the creative response he said you know this is years decades sometimes even centuries a scrupulous institution building to really create the new way of doing business the new market form the new qualities of capitalism that will truly meet the needs of populations people and lift the standard of living for all people this is what Schumpeter was was really about instead what what we've just talked about they figured out that as the crow flies there was no time for years decades heaven forbid a century you know to really figure this thing out so so this is how competitive intensity is is heating up because everybody now had to rise to this standard and and what we have come to see Jim and I know you're aware of this is that surveillance capitalism has now moved far beyond the tech sector we can no longer just talk about the tech sector here it has traveled across the normal economy so when we're talking about insurance we're talking about finance we're talking about retail we're talking about healthcare we're talking about education sector and now we'll go full circle Ford Motor Company right where mass production was born we're all about the the century of consumption and middle class employment now the CEO of Ford Motor says you know what it's really getting hard to make any money selling cars and it doesn't look like it's going to get any better we want to make money with data and all of a sudden the idea is you know we've got a hundred million people driving around in for car and Ford cars so let's just start to stream data from the personal experience of all these folks and then we're gonna be able to have behavioral data and we're gonna be able to make predictions and we're gonna be able to have markets and we're gonna compete with the margins of the likes of Google and Facebook and so much of the of the rhetoric is you know from the tech industry is this is just capitalism this is just the way it works if if you want to make money we make money off of data now and and this is not different in kind than past kinds of capitalism but I think you believe this is different in kind because we haven't had some of this norming and it sounds like you also think that many these companies have hidden what they're doing from us and this is pretty famous in Silicon Valley right it might be illegal it might be cutting a corner but by the time anyone figures out will be so wealthy at such scale whatever fines we pay whatever price we pay is worth it so is that going on and in this new industry and okay okay well let's talk about that all right so so with with this competitive intensity increasing a new discovery was made and it's still this discovery process I would say is still in play right now and this discovery was holy smokes the very best predictive data the very best source of predictive data comes from yeah we need scale and yeah we need scope but it really comes from intervening in people's behavior actually intervening fester in the state of play and gently tuning and sometimes not so gently but we could talk about that tuning hurting modifying behavior in the direction of the commercial outcomes we seek alright so you put your finger right on the the right button there this began was something that seemed totally banal a be testing you all know what that is a be testing the idea that okay you're looking at one web page for ice cream and you're looking at what should be the same webpage for ice cream but there's subtle differences in the webpage and then they can see how you interact with it and they know if your subtle difference it's going to be more lucrative than your subtle difference and on and on it goes and basically web pages are always always have some of this this kind of testing every day tens of thousands every day by every one a thousand thousand okay but it moves on from there so testing begins to get serious for example some of you may have read about what facebook called its massive scale contagion experiments so these massive scale contagion experiments 2012 2013 2012 the first one was published in an academic journal I know you remember this gem and the idea here was subtle manipulations subliminal cues in your Facebook pages that would that were they were trying to see if you they could use subliminal cues and you're in the online media to get you to go and vote in the 2012 midterm elections could they use subtle cues in the online millia to change real-world behavior ie folks who might not have voted went and vote went and did vote and of course lo and behold they discovered that the answer was yes they could and they did and when they wrote it up they celebrated the fact writing this up in a scholarly journal they celebrated the fact that a we now know that we can use online cues to change real-world behavior that was number one and number two yippy we can do this in a way that completely bypasses people's awareness you never knew that we were there and this gets us to a very very big point about that one-way mirror I mentioned that all of the big money that we're making all the surveillance capital that's being accumulated a lot of that money and expertise goes into designing these systems in a way that they are engineered for our ignorant we must be ignorant of what's going on behind the veil because honestly and all the research that exists out there when we are told about it when we find out about it the reaction is I don't want that how do I protect myself from that all right so that leads onto another set of complicated questions that I'm not going to answer right now the point is getting back to is this merely capitalism okay so what we see with these now we have economies of scale economies of scope and economies of action data scientists call this the shift from monitoring to actuation one data science has explained the situation to me he said we're getting very good at knowing how to engineer the music and then we let the music make them dance that's how he described it to me the shift from monitoring to actuate in Hindman behavior all right so I write about this as the what we've created now is literally a global means of behavioral modification behavioral modification at scale that means automated means it embedded in the digital embedded in the digital architecture from our laptops to our phones to the the digital surround that increasingly follows us all through our daily lives is repurposed as a means of behavioral modification do we have time to talk about Pokemon go and this in this respect go right ahead and then I want to move to could this actually be socially beneficial and under what circumstances but let's have an example so people know how far this goes in the world and are moved to what are this new monster we've created alright friends so a Pokemon go is another population scale experiment in a global means of behavioural modification and we thought it was a game it turned out it was a game and perhaps I should use the present tense it is a game within a game so we thought that Pokemon go was monetized oh and let me just preface by saying in case you don't know it Pokemon go was incubated in Google it was developed and managed by man named John Hanke who before that was a key person in Google Maps he ran Street View but before that he was a key person in Google Earth but before that he was the person who invented keyhole which is a company and been invested in by the CIA ultimately Google bought it brought it into Google and named at Google Earth and I started to learn about surveillance capitalism this is just an aside of one day when I asked a bunch of Google executives back in 2007 just innocently how do i opt-out of Google Earth and it was watching the reaction that to that question that actually really taught me a lot kept me going for years more than just playing a game we thought that people that the whole game was going to be monetized by people buying little doodads for their Pokemon creatures and little game tokens and so forth and ER eventually what came out was that the way that Pokemon go is being monetized was that it had constituted its own behavioral futures markets so now in your city there were bars and pizzerias and restaurants and other kinds of service establishments McDonald's franchises who were paying pokemons go for footfall and pokemons go would run the game so that a certain number of people were herded to these paying customers so that they could bring their real bodies with their real feet to these real places and spend real money now what does this remind you of it must remind you of something that we've just talked about because it's exactly the same structure as online targeted advertising isn't it we saw there the piece of future behavior was the click-through and we've got these business customers interested in our future behavior paying for click-through rates well now click-through is footfall we've gone from from walking with our fingers to walking with our feet right and instead of online advertisers it's real world establishments paying to get our feet into their space so I'm starting to get questions from the audience and I think I think the question that has come through a couple different ways is well getting people to vote is in some ways a civic good so you could imagine this sneaky technology being used to advance social goods just as it's being used to maybe surest suppress voters from another angle so how how does society engage with the fact that there's this incredible experiment being run on all of us all of the time right now and how do we how do we influence it to do more good and less things that are either benign or neutral or possibly negative from an ethical or a societal standpoint all right so what do we want to say here so many things beware your first impressions friends beware your first impressions when we talk about the power to modify our behavior at scale we are talking about a kind of power that has never existed before this is an unprecedented form of power this is a power that works through the means of the digital media through this vast digital surround outside of our awareness without our permission without our knowledge and therefore without our consent without our right of combat without our choice we are not involved here we are pawns that are being pushed around for the purposes of others gain and we saw how that translated into Cambridge analytical and it was a nightmare so we have to ask ourselves what is this form of power friends because this has never existed before just as the digital surround has never existed before these methods and mechanisms have never existed before I felt the need to name this kind of power we often confuse it with totalitarianism and I believe that has that has set us back from actually understanding it totalitarianism is about violence and terror totalitarianism wants to rebuild you from the inside out that was the whole point of 1984 the engineering of the soul this new form of power really does not give a toss about you it does not care what you believe it does not care how you feel it does not care where you're going or what you're doing or what you're reading it only cares that you are going and doing and reading and feeling and talking and thinking and saying in ways that it can access as raw material turn into behavioral data and uses predictions for its marketplace so it it it literally takes on the attitude of the behaviorist experimenter who really doesn't care a lot about the rat it just wants to know that it can measure the rat and train the rat to go from A to B and in fact when you go back to radical behaviorism you know what the behaviorist like my old professor BF Skinner say is there is no such inside you know there is no consciousness there are no feelings there are no emotions there's only measurable behavior and this new kind of power is only interested in that measurable behavior I call it instrument aryan power Jim because for two reasons one is it works through the medium of all of the digital instrumentation it can't work by hands because that's not economies it has to work economies of scale to modify behavior and the idea here and now it's not only automating the weight it gets information about us it's actually automating us so that's number one and number two is I call it instrument aryan power because it turns us in into instruments of others gain it's an instrumental relationship with its societies with its population with us as individuals and so for these reasons i call it instrument aryan power ok so wrap up that thought i want to follow the next ok sessions here so so what i what i want to say to you is that the harm here is that these operations cannot function effectively unless they intervene on the very fundamental experiences that we call human autonomy unless they circumvent our decision rights which requires bypassing our awareness self-determination is the enemy of automated behavioral modification which is itself becoming de rigueur for these operations and by the way Jim pokemons go was a population scale experiment but that experiment is very quickly being translated into much more flesh-and-blood real-life consequences a kind of operation that is called the smart city Google likes to call it the Google City and right now the citizens of Toronto are facing off against Google in the in the incarnate sidewalk labs which is its Google City operation where the public officials in Toronto have proposed to give Google part of the waterfront to reinvent as the Google City and what this does is it takes all of these methodologies that have been honed and tested in these various ways that we've been discussing and now imposes them on populations where we should have democratic citizens who own and decide about the nature of their municipality instead we get computational decision-making for a greater good that aims people toward the most profitable outcomes of the owners of the systems so so many of the questions that the couple more are worried about because it's the world affairs council what people in foreign governments might do with this and you just gave us an example of Toronto channeling us to serve essentially Google's profitable aims and but in China I mean these instruments of controls these pokeyman gosai's experiments are being actively deployed so how what how does I mean how does that enter into this sort of surveillance capitalism it's actually argue lis being used in certain smart cities in China and is it corporate or is it the government or is it corporate following the tune of the government I mean how how do these things interact and how does a society take better control over being influenced in this hidden way that takes away agency yes okay so so let's talk about this and this whoever asked the question earlier about you know couldn't this be a good thing get people to vote that sounds like a good thing all right so here's where you're down the slippery slope because when we talk about democracy we have to talk about not just what is done but how it is done getting the getting a great outcome in an anti-democratic fashion is not good for democracy is not good for human freedom and is not consistent with our elemental principles of morally autonomous individuals because at democracy the idea of democracy requires as its elements at the bottom of the of the system the idea that we have citizens who are morally autonomous who have agency who can make choices who have free well who can consider things and take options right so so this freedom is essential an essential building block of democracy at the top of democracy we have institutions that are that that preserve a kind of equality across our society now we're worried about economic inequality already but here we're seeing institutions of surveillance capitalism already disfiguring our 21st century society with a pattern of extreme asymmetries of knowledge extreme asymmetries of knowledge that with a little bit I've explained to you hopefully allow you to see also accrue to extreme asymmetries of power right to shape our behavior in ways that we have not chosen I'm going to China okay all right I'm going to China but this is sort of our backdrop now we go to China mm-hm and guess what's different in China there's no democracy in China alright so in China it doesn't matter how you do it in a Democratic Society it matters how you do it you can't just optimize energy efficiency in a democracy without the without that having been arrived at in a democratic way that's why current climate politics are so difficult you go to China you don't have to worry about how you do things because it's an authoritarian society and in one way or another it always has been this is the signature of Chinese history is a very different culture a very different history from our own and from that that of Western democracy okay so so when it comes to China we're toast either way but let's go to a different context let's pick Europe a democracy I mean do you see something like the privacy regulations represented by the GD P R as a step in the direction of how society starts bringing back control over surveillance capitalism or is it just just throwing a penny at the problem all right well I just I just I just want to say just to finish China for one tiny check okay the in China what we've got is an authoritarian state looking over here and seeing what we thought were private companies private sector companies within China like $0.10 Baidu other companies who have developed these immense capabilities of surveillance capitalism that I've just described you and develop immense capabilities of instrument aryan power sumerian state facing a society that is in many ways in chaos because of a long term breakdown of social trust and all of a sudden this authoritarian states holy smokes I don't want these private companies to have all these data and all this power to manipulate and shape behavior with these data we want to do it and we want to do it not to shape commercial outcomes we want to do it to shape the societal outcomes and the political outcomes that we as an authoritarian state deemed to be best for this society that's the Chinese Social Credit system and the ways that is being elaborated in that country and that in a way is the apotheosis the nightmare vision instrument Aryan power suffused conflated with state power that is not the place we want to go in our future okay right so now we come to Europe yep Europe and California are ahead of the pack with privacy law and privacy law is extremely important Europe has always been ahead in privacy law for many decades the gdpr goes even further but even as my political friends in Europe most of them will agree that it goes further it's fantastic we need it but when it comes to surveillance capitalism doesn't yet go far enough because we are going to need the kinds of laws that specifically interrupt and outlaw the mechanisms of surveillance capitalism Wow so but can we get the benefits of all this great technology if we don't have surveillance capitalism driving all these innovations what a powerful question it's so glad you asked that question because what have we just been saying alright the big lie that has been perpetrated on us is that folks everything that Zubov is describing to you is inevitable this is the digital world folks love it because you cannot leave it right this is inevitable that is a big lie friends that is a big lie well what I've been saying is that I've described an economic logic right and an economic logic is not the same as technology we can easily imagine digital technology without surveillance capitalism it ain't that hard it's impossible to imagine surveillance capitalism without the digital impossible right right as we've understood with all the automated scale scope action and so forth instrument aryan power all works through the means of the digital so but if there was a window Jim and I think you were there before Congress a window before surveillance capitalism you know got out of the bottle and people started understanding what it was in the late 90s to 2003 for you know people were developing projects for the smart home and developing projects for telemedicine all the kinds of stuff we want digital we want data to improve our lives remember to empower us to emancipate us to democratize us remember and in those projects it was very very clear you make a smart home you got devices in the walls and so on and so forth they're generating data and where does the date of go the data goes to the occupant of the home oh my goodness and why does it go to the occupant of the home because it's intimate data its private data remember what I said private human experience so of course it's got to go to the occupant of the home this is what the engineers assumed and and that's how they drew their schematics you fast forward to the year 2017 a couple of legal scholars King's College London they figure out any self-respecting consumer who buys a nest thermostat nest owned by Google yep and this is actually even before of course the recent revelation that the nest security system it's got a microphone in it sorry we forgot to tell you oh my goodness so any self-respecting consumer who puts in a nest thermostat should read a minimum of 1,000 privacy contracts because the thermostat shunts your date of the third parties and they shot your data of the third parties and they shot your data to third parties odd infinitum right Jim yeah but I mean I mean here's the thing is that people identified these problems that everyone blew past it and picking nest as an example they know when your 8 year old daughter is home alone how did we turn that information over to them when we're talking about the most vulnerable members of our society how did how did we just kind of just keep ignoring that over and over again I mean you're describing a pretty terrifying circumstance that we just wandered by yeah well part of it is what I mentioned a couple of reasons and actually I answer that question with 16 reasons in depth okay all right here we go one no no give us give us three of them because I think give us three that I'm gonna give you always number one I've already explained this unprecedented when something is unprecedented by Nature you you literally can't recognize it you can't cognize it that's why we think instrument Aryan power is totalitarian power and when totalitarian power first broke on the world nobody got that either and they thought it was a version of imperialism and it took a whole generation of scholars to say this is a new kind of power and explain it alright so unprecedented number two I've already told you engineered for ignorant okay engineered to bypass our awareness when we tell people about this stuff they really don't want to do it they really don't want any part about it okay and number three is what the great philosopher Roberto Unger calls the dictatorship of no alternatives right so we've gotten to a point in our lives gym where routine social participation you know I want to organize dinner tonight with friends and family I want to get my kids grades from their school I want to get my health test results from my doctor's office any simple thing that we do marches us through surveillance capitalism's supply chains all of these things every product that with the word smart in front of it every service with the word personalized in front of it folks remember this these are just interfaces for these supply chains personalization is the opposite of privacy it means it's all gone so so we so you know we've put people in an impossible position and this is something that I consider intolerable that 21st century citizens because of surveillance capitalism now have to make a choice between effective social participation and being tainted and being being engulfed it at every step by these systems from which increasingly there is no escape this is in tolerable this is not okay so and of course you know we've heard the studies of kids being deprived of their digital thing for a day and coming to freaking out yanking out yes so so I think you've got a new call to action essentially to a to the next generation of big data entrepreneurs to could you imagine companies that actually embody sort of the reverse of surveillance capitalism where people are actually aware of what's going on that people have agency and making these decisions and so as I think become more aware of this maybe some of these maybe there's a way to go back to a more earlier form of capitalism where people were active agents in deciding how channels is that yeah thank you [Applause] [Music]