Meaningful Human Control of AI: A Framework for Responsible Innovation

[00:00:00][Music] welcome everyone to this third edition of the xomnia webinar series on responsible ai an in-depth webinar series about the responsible design development application and use of ai in this series we will discuss a range of topics related to the ethical accountable or sustainable use of ai by doing so we want to inform educate and support everyone who is interested in responsible ai topics arrive between tools methods best practices and lessons learned for the achievement of positive societal and environmental impact while meeting commercial and or business objectives today we kick off the new year with a talk about meaningful human control of ai unfortunately we are unable to reach you i reach out to you from our studio exomnia headquarters due to the current covet restrictions our guest speakers are mark stein and julian van dijkler both of whom who are a senior research scientist at tno which is the netherlands organization for applied scientific research my name is murica peters and i am your host for today i have a background in both psychology and artificial intelligence and after 10 years of doing applied research first in academia and later at tno i moved into business about a year ago by joining insomnia as an analytics translator i aim to create data-driven solutions with a proven added business value that are smoothly incorporated in the day-to-day operations of our clients i have a special interest in fair transparent and accountable ai human ai collaboration and teaming and user-centered design julian van dijk is a research scientist and project leader at tno with an interest in human ai teaming explainable ai cognitive systems engineering and knowledge systems irian is also my former office roommate when i was still working for tno marc stain is also a research scientist and project leader at tno with an interest in responsible innovation ethics and big data human-centered design value-sensitive design and open innovation and mark is also a former colleague of mine today their talks will focus on ai systems that play a role in morally salient tasks tasks that clearly contain a moral or ethical component in such tasks it is considered especially important that humans exert meaningful control over the ii system's behavior this ensures humans can be held accountable for the ai's behavior at any time and that moral wrongdoing by the ai system can be avoided thank you both for being here today and telling us about your work irian you are the first to give your talk and the floor is yours yes thank you for the invitation and for the introduction so my talk will be about meaningful human control and to understand meaningful human control we must understand what an agent is and to understand what an agent is we must understand what the udar loop is the ooda loop is a simple model that describes intelligent behavior consisting of four phases observations the agent gets all sensor data in orientation the agent tries to make sense of that sensor data detecting objects classifying these objects trying to predict the future and then deciding given the goal of the agent what should be the action that the agent should take next and finally actually performing that action in the real world and then the cycle starts again because the performance of that action will probably change the world and the agent has to observe these changes again orientation etc most current ai systems do not close this loop yet so they only perform part of the ural loop for example observations and orientations for example at the right hand side there is a business intelligent tool where the ai system gathers all the data regarding financial situation of a company and tries to process that and presents it in the user in a comprehensive way but it does not yet perform the actions that can be based on that information or a recommender system which presents to the user which movie a user is likely to enjoy but it does not yet automatically play that movie for the user or a grammar checker which suggests suggestions uh and alterations in the text but does not automatically change them so what we see here is that there is really one actor in this human agent system which performs the whole utah loop and that's the human and the agent is used as a tool there is only one locus of control but this situation is changing if we look at grammar check-ins of course we all know that some of these systems automatically change the text for you and then it becomes an agent and these agents are becoming more and more popular here in fintech we have these systems where these stock trading agents try to make sense of the stock market but they automatically buy and sell stocks for their users so they have the whole ooda loop working and they pursue their own goals an autonomous car is also driving even when the user is not giving explicit instructions a surveillance robot is autonomously walking around and observing the environment even if the user is not giving explicit instructions and in these situations you have two locally of control both the human and the agent perform an uder loop and this is where things can become complicated and meaningful human control becomes relevant because what if this ai system is spinning out of control what if it's performing all kinds of actions that the user does not understand the user is not aware of the goals that the agent is pursuing and the machine is performing this action so fast that the human can no longer intervene this is an out of control system where there is no human control and this actually happens clearly this tesla car that crest was out of human control these stock trading agents in 2010 that caused a flash crash were clear clearly out of human control when they collectively decided to sell all their stock this patriot rocket intelligent missile was clearly out of meaningful human control when it [Music] shot down a friendly fighter jet in the second world war so these problems with loss of meaningful human control are real and there are a recipe for disaster but there's a cure and we believe that the cure is human agent teaming these it's okay if these ai systems become agent but we should make sure that we can team up with them in order to engage in joint activity and what is symbolized here with these gray gears spinning around is that there should be an appropriate interaction between the human and the machine which allows the human to understand what the agent is doing which allows the human to redirect the behavior of the agent to align their goals to make sure that they act collectively as a team okay so these were nice diagrams but i think that you might still think well i understand this meaningful human control problem but is it really that important it sounds a bit science fiction like like the terminator ai is going to take over of the world um i think it's a big problem i'm not so concerned about the terminator either um i'm not so concerned either about these examples that i gave you earlier with the tesla crash and and and the fintech crash that you can refer to them as teething problems they might go away when the ai gets more at fast advanced but i do think that meaningful human control becomes very relevant when we start to apply ai systems in ethically sensitive domains as agents and that's happening right now in many domains for example radiology where ai agents automatically interpret x-ray images and detect cancer potential cancerous tumor tissue and they also propose a treatment for that the consequences of those kinds of decisions are very real a certain agent that performs autonomously search uh surgical operations on the patients those decisions can have uh consequences of life and death for the patient a triage agent which autonomously decides which patients get to have which treatments or an elderly care system which helps elderly people living at home but the system autonomously decides whether to warn a doctor or not if these elderly people have fallen down or there's an incident so in these domains ethics is a plays a huge role and we always want to be able to hold human accountable when things go wrong and preferably not have the fingers pointing at ai systems because it would be a bit silly to hold an ai computer accountable when something goes wrong and we want these ai systems to share the same values and that's all captured by this notion of meaningful human controls we want humans to be in control in these domains and another set of domains where this is very relevant is the safety domain here you see robots deployed in disaster response these robots decide which victims to rescue first we see ai information agents being deployed for cyber operations and responding to cyber attacks and autonomously decided deciding which countermeasures to take for example shutting off parts of the internet or in predictive policing where ai agents are deciding where the police will perform their patrols at a certain time of the day or maybe the most critical application of them all when ai is deployed in defense operations and there are autonomous robots moving around in warlike situations and they might even have the opportunity to decide to use force well that's something which is very critical um and we should think about that these kinds of decisions always remain human be taken by the human and in the defense domain there's even in international unitarian law a law that states that the use of force should always be under meaningful human control the problem with this law is that they have not clearly defined what is meaningful human control so if we want these kinds of laws to be useful in the defense domain but i also think that they should be applied to many other domains that i just described in healthcare and police in cyber defense then we must better understand what meaningful human control is and as i explained to you earlier meaningful human control is all about the relationship between humans and machines and what we are working on in multiple projects is to develop and to characterize this interaction using team design patterns and measuring the amount of meaningful human control that arises from these team design patterns in order to be able to develop policy to develop interaction design mechanisms and to formulate requirements of systems we only want to buy a system which is under meaningful human control hence we only want to buy a system which supports these team design patterns and in a sense a team design pattern is similar to a constant which is known in autonomous driving where there is a subdivision between six levels of automation at the level of automation zero there is no automation at all at level of automation 5 the car is fully automatized and the human driver doesn't do have to do anything and in between there are all these in-between modes for example that only steering is automatized or only speed control is automatize or only braking and in that way you can when you buy a car you can say i want an autonomous car which goes up to level of automation four but you can also develop policy based on that for example you can state that on the highway autonomous cars are allowed with level of automation five but in cities we want autonomous cars only to go up to a level of automation three in order to maintain meaningful human control well there are other attempts to make models that subdivides the amount of human interaction in different scales for example stating that the human should be in um or out of the loop or division into 10 levels of automation i will not go into all of these schemes but i for our purposes for characterizing meaningful human control we need to be much more specific than all of these schemes we need to be able to precisely specify which information is shared with the user when is this information shared how does this information change the interaction change over time for example we might want to state that at the beginning the human is more actively involved with the system and once the user is familiarized with the system then there is more autonomy possible how does the user exercise control how does the system explain itself to the human which aspects of the task are under control by the user and which aspects of the of the task are delegated to the machine for each of these questions this design decision has to be made and this gives rise to a huge design space which we can characterize using this team design patterns and let's first focus on two different team design patterns which are rather at the extreme of the spectrum at the left hand side is what i will call the human in the loop pattern where the agent performs all the work and once the task gets morally sensitive then the human takes over well this is nice because the human does all the morally sensitive tasks but it requires the human also to understand what is going on and it also requires the human to have sufficient time for example consider patient triage so patients come in and the the ai system might monitor what all the uh the the the state of all the patients and when a patient needs more treatment and decision needs to be made whether to send the patient to the intensive care unit for example then the human takes over and performs that little bit of the task that's a human in the loop team design pattern and you also immediately see the problem that if there are too many patients at the same time that needs to be decided or triaged then the human gets overloaded so on the other hand of the spectrum you might want to choose that you that the agent has much more authority to perform different tasks and what you see here is that before the operation starts before the work starts the human and the agent collectively design an ethical model and we call that value elicitation and the human teaches to the machine what is ethically acceptable and what is not ethically acceptable and then at when when the when the task starts the human can do the whole task autonomously but it acts according to the model that has been formulated by the human beforehand so we might still want to call this meaningful human control because the effects are actually programmed in by the by the human in the case of a triage this might mean that beforehand you decide which patients get to have which treatments for example you might want to decide that people that are older than 70 years have priority in the going to the intensive care unit compared to younger people or you might want to specify a different value that age should never play a role in deciding whether to go to the intensive care unit and the benefit of this approach might be that you can think about these choices without when there is no time stress and you can formulate them precisely and maybe even involve a democratic process in politics to formulate these rules and of course you might recognize these kinds of discussions in the in the current press for example ira helsload has is a proponent a vocal proponent about these mathematical models for values the problem of this approach is that it might be very difficult to capture ethics in a machine it is very doubtful if we can have an ethical model that can be understood by a machine which corresponds exactly to the human rich perception of ethics in in in different contexts so we might want to go in between these different team design patterns and this is where we have many variations and i have drawn four of them here in this slide but in fact there are many many more and the trick is that when designing systems which involve agents in ethical domains we must try out different team design patterns different ways of collaboration and also measure meaningful human control and that's what we have been doing it's very difficult to measure meaningful human control but i think that you cannot go around it you have to do that and here we have created a little test bed where you see different patients coming in and they have a similar disease like covet 19 and they have different properties age gender profession and they have different symptoms and the task of the human is to send them to one of the treatments to the intensive care unit to the ward or home and the intensive care unit is the best treatment but there's scarcity of ic beds just like in the real situation and the trickiness of the task is sometimes you have to choose which patient gets this treatment and which tracement which patient gets a sub-optimal treatment and that's where uh the ethical decisions have to be made and what we did within these tasks we just this is simulated tasks obviously and we tried out different ways of support and in one of these team design patterns there was uh hardly any uh there was just support by the by the agent but the human actually took the real decision and on the other hand of the spectrum we had this more autonomous mode where there was a ethical model that was taught to the machine beforehand and the machine did all the ethical decision making and the human just had the opportunity to intervene and there was an in-between mode where some of the cases the not so sensitive cases were done automatically and the hard cases were performed by the human and we had these three variations and we just started testing and collecting data with test subjects in a lab setting and i will not go into each of these results but we gathered a lot of data regarding how the human experienced control over these systems and we could also compare the different outcomes are the decisions that were made using this highly autonomous agent involvement different than when there is much more manual involvement and what does it say about meaningful human control just a couple of things that we already learned is that it's very important when doing these tests to have what we call an ecologically valid test beds humans should really be um experiencing that they are making an ethical decision so that's why we um deployed the technique uh of called storytelling where these patients actually were accompanied by a story about what happened to these patients and they really took the time to immerse them in this situation and then they had to make these decisions and these subjects actually had the idea that what they were doing mattered so we were quite confident that we actually um were measuring how they responded to ethically sensitive situations and what we saw is that sometimes they believed that they had meaningful human control whereas we found in their behavior that they didn't have any control at all they were just blindly following the advice of the agents they were over trusting these the the suggestions that were made by the agent which is of course a very dangerous situation when trying to design a system on a meaningful human control um and also regarding responsibility we saw that the more was automatized in this decision-making task the the less personal responsibility the subjects felt but this is also not the case because the human is always responsible over the behavior of the ai systems with which they work together so in all of the situations that we that we tested we saw that meaningful human control was still an issue and it differed regarding these collaborations so we are not saying that we have found the optimal team design pattern which safeguards this property of meaningful human control but what we what we conclude from this is that it's important to try out different collaborations with these ethical agents and measure meaningful human control and keep on adjusting this collaboration until you get it right because it is an important aspect and is getting more and more important when we deploy these agents in ethical domains so thank you for your attention and i think i have the opportunity to respond to one or two questions yes that's right so julian i just explained that if people have questions for yurian you can post them by placing a comment right underneath the video in the linkedin event but i also have a question julian so i would be happy if you could answer it i really appreciate your explanation of the difference between ai as a tool on the one hand and ai as an agent or an actor as on the other hand and it seems to me that there are quite a few applications that already kind of exceed this level of ai as a tool and that have across the border towards ai more as an actor or an agent so i have two questions in that respect so the first is do you think that for instance automated recommendations or ranking of content on a website such as a search engine or a social media platform that they already constitute agents instead of tools and um yeah how do you think that we should deal with that or yeah so um i guess there are there are cases where there is this smooth boundary between a tool and an agent so um you you mentioned recommender system that's that's quite an interesting one because if you view at these systems a bit differently you may think of it as an agent that has its own goal of maximizing clicks and views by users and its actions are actually recommending these things to users and then there are in fact two different ooda loops going on where the agent is pursuing its own goals to maximize views and what what gets interesting in this example is who is the other agent with whom is this agent teaming that's probably not the user but it's probably the content provider and um seen from a perspective of meaningful human control you might ask yourself the question if the user should also be uh be in control a little bit more about these kinds of systems so i think it's a very relevant questions and i also would advise that when designing these systems it's sometimes not so clear whether you are actually designing an agent um but it's good to think about that and and and who are the users and and who is actually in control and did we think good about uh the ways in which humans can remain in control yes thank you um i think we have time for one more question from the audience so um let me see marian cruz asks the question considering that people widely defer in their moral evaluation of situations the question is which or whose ethical model should build be built into an ai so would you choose a silicon valley version europe being committee version or any else and i think this is also a nice leap to mark's presentation yes so um i i i um this this refers to the more uh machine ethics um approach to incorporating ethics in in ai and the good thing is there that you can discuss with a wide group of people what is ethical and what is not and describe it precisely such that it can be followed by a machine and that way you can democratize that process a little bit but of course the question is how to reach consensus within a group of state stakeholders what is ethical and what is not and precisely when you pin down ethics to precise mathematical rules that becomes very difficult and that's also what we see with uh this covet pandemic where uh it's very difficult to to define rules who gets to be vaccinated first or who gets the the first priority to go to the intensive care units sometimes it seems that people may not want to be pinned down on that and that's why there's also a lot of people prefer the case by case evaluation of of results and more have that have a tendency to go through this this more human and loop team design patterns thank you so i as i said i think this is a nice um yeah change into a marked presentation who will go a little bit deeper into these different ethical models and what their implications are for the design of systems um and also how you can for instance involve multiple stakeholders and yeah decide on what to um what is the best ethical framework so um mark um welcome and um the flourishers yeah thanks um my name is mark stein a colleague of yurian former colleague of america um fritz am i correct or do do i do the powerpoint yeah no no now you have to open the um your presentation on full screen and then you can start that's it yes excellent thank you so i will talk about two topics meaningful human control that julian already talked about i'll give an example of that hypothetical example and i'll talk about integrating ethics in projects meaningful human control in the project with yurian and three other colleagues we envisioned a scenario an ethical sensitive scenario of the military this was by the way my first time working in the uh with the content of military but many colleagues of tino have much experience with that so they helped with that on the left you see three soldiers they're on a mission somewhere across the globe and in b that's a large box there's the drone and a drone has no arms it has no weapons it only takes pictures and picks up sounds and um this is the scenario that we'll talk about their task is to survey that area and bring back intelligence to headquarters something like that julian showed you the levels of automation for cars for example now we looked at ben schneiderman's paper last year we think he has a very interesting take on it because he rather than one axis he talks about two axes on the on the horizontal axis you see computer automation you can do more or less of that higher low computer automation what what the computer does and on the vertical axis you see the the the level or the quality or the amount of human control and so it has been now become two axis and then you go to the top right corner and he is saying that uh in that in that quadrant you'll be able to make reliable safe and trustworthy systems with a high automation because that's efficient and on the task that a computer can do well and a high human control where human control is necessary so that's the framework we used remember from the last slide i was talking about the drone the drone and its automation giving a little bit of automation to it as much as possible you can think of that on the horizontal axis and where necessary and if necessary we go to the vertical axis and we give the soldiers a decision support system so it's a way of having control over it so in what i'll be showing next you think of the computer automation is what drone does how much automation you give to it and the vertical axis what what the people can do with a uh you can imagine an ipad with a screen on it etc i'll show examples of that and as a decision support system so the ai is here very much thought of as a tool combined with some agency now ethics one of the people asked already the question about it so this is just four but there are more but let's say these are four major traditions or views uh in ethics and i will talk about them clockwise and it will return clockwise jeremy bentham top left consequences utilitarianism he was the one that coined the phrase the greatest good for the greatest number the idea is if the consequences or the outcomes or the impacts of what you do are good then the act is good we go to emmanuel kent to the right top he was talking about duties and later on it became right human rights is also in that quadrant not looking at the consequences but more like duties uh acting out of reverence for the moral law out of good will so the the moral agent and his or her uh uh um reverence for law and following duties is is is key there go to the top and now sorry the bottom right uh carol gilligan [Music] feminist ethics or ethics of care but i talk about relational ethics um these two white dead men from the european enlightenment period very much believed in rationality and independence and as an as a compliment to that carol gilligan and other scholars introduced also taking into account relations and care to complement justice long story that's uh focus on relationships last one virtue ethics virtues and flourishing of aristotle by the way there's an ai designed remake of the white marble statue that's fun isn't it um he was talking about ways to enable people to live together in a polish in a city-state i like to think of the european union as a city-state we try to make laws that fit lots of the states within the european union and uh have justice have autonomy have dignity to enable people to flourish it's about cultivating within yourself those virtues that will help you to lead the good life and using technology as tools to cultivate these virtues now this is a theory now go to the practice remember the drone the drone energy support system this is what jeremy bentham would do what would jeremy and bentham do he would put utilitarianism into the drone so the drone would have image recognition and would be able to to make pluses and minuses [Music] of the outcomes that it assesses will follow from certain actions so it has moral reasoning in a sense that it uses a utility function the soldiers will have in their hands something like an ipad decision support system and where the drone and ai behind it is not able to compare apples and origins so to say the green a little bit of green for protecting civilians but larger red for risk for our own troops well it's not really up to the drone to make that call to make the decision so this is where the soldier steps in they can use their moral discretionary competence to make that decision what would kent do kent would make rules top down if then rules into the drone program into it all the conventions of humanitarian law from geneva and from the hague this is the rules that you stick to these are the rules of engagement this is the mandate that you have for this military mission backed by un resolution this or that gps fences very clear lines here you can go here you can't go the decision support system would have something like a pop-up saying hey please do keep in mind that now is the risk of violating article four of the four geneva convention that deals with uh collateral damage is the military jargon for for citizens and harming them and of course duty is very much a sort of a synonym for for obedience and following a top-down command and control change so of course the soldier must be able to call his or her commander to to deliberate collaboratively on what to do what would carol gilligan do they would design the drone so it behaves socially so that means it will not fly right over a cultural heritage or an event that is of significance to local population it would sort of behave socially it would make a nice circle around that building or around that event maybe it would even wear a blue helmet like the un soldiers have to signal to the people i'm here to watch over you i'm not evil i don't have weapons i'm just like a u.n soldier with a blue helmet the the ipad of the soldiers will show something like this decision support it will show you a face literally this is hamza yusuf 35 years old he is the manager of this hospital and mind you there's also in this web in this society the hospital hospital does play a role in water supply maybe local employment maybe food security maybe there's a food bank so if the soldier is now using his or her discretion on what to do we have pluses and minuses we have duties we have rules we have now also relationships and what goes on on the ground within this society last one what would aristotle do aristotle would think in terms of what would be an excellent drone what would be the virtues of a drone how can this drone become more virtuous over time now learning is a key component of virtue ethics likewise the ipad of this footage will show something like this hey in this situation courage justice and a broader perspective are virtues that you need to cultivate a way of cultivating virtues also looking at examples exemplars that those are people who have done in the past these ways in a praiseworthy manner so remember this case of some years ago that maybe you have also learned about in your training there was a question in the previous part of the session that was saying hey which ethical framework these were just four frameworks and when to apply it this is ongoing work we're not totally sure about yet but this is a hunch what we have if patience feeds well it's a blue figure patiency deals with the harms for those at the receiving end of what you can be doing if that is at stake then consequentialism that was the utilitarianism of jeremy benson is is a good perspective on the other hand if you go to the red figure top right if agency and autonomy of people is key then the duty ethics is more appropriate the yellow one if sociality is key in this mission in this task in this uh context then then the relational ethics obviously and if learning over time the green one is key then possibly virtue ethic is the first one to start with so this is just a tentative recommendation of the soldier how do they program the drone which of the four well you can sort of choose between these how do they use their decision support how can they toggle between the screens that can help um a couple of minutes also on how you in your project may be able to integrate ethics ethics is sometimes seen as a barrier against innovation i like to believe and there's lots of people within tino that are with me in this that ethics can can be different can be something different it can actually help to steer the innovation to move the project into a direction that you think and believe is desirable and that you can become uh can can can be accountable for rather than that it derails or that it spins out of control was a beautiful term that urine used so ethics not as a barrier but as a way to steer your innovation now i'm not talking about ethics in the role of a judge we're not a judge not as a preacher we're not in a church but as a process of ethical deliberation thinking more deeply about reflecting on the dialogues within your organization with stakeholders outside and very much the facilitating of this process this is a remark that i think is shows what i mean with ethics can be also uh can can facilitate creativity is 30 40 years ago there were two sorts of tents the big ones that are spacious but are heavy and the small ones that are lightweight now these values are conflicting right or you think of um covet measures it's good for health but not good for the economy right or it's good for the economy but not good for health it's like they're opposing they're conflicting or maybe hanging cameras in public places it's for privacy or secure security and all these oppositions i think ethics thought of as a process that can help to promote creativity so you come up with creative solutions and that the values no longer conflict so this is an illustration of then 20 years ago three years ago what was it creativity helped to find new forms and new materials and new structures for tents so now a tent can be large and spacious and lightweight because of the curves because of the carbon fiber because of the etcetera the materials so this is just a reminder that ethics is not a barrier ethics can actually help you with creativity and we designed a method it can also be done online in in two sessions one session of 30 minutes one session of 60 minutes and we do the following we identify and explore in your projects possible issues so if your project delivers outcomes what could be societal or ethical or otherwise issues with those outcomes ideas for organizing dialogues inside your company outside your company uh to use maybe the four perspectives that we that that briefly shared with you or other perspectives and then draw that tent of conclusion and steer the project and move it forward it's based on all of this if you're interested in it for extensive design responsible innovation sustainable development rights and the capability approach now i will only show you one because it bears that that ties back to the question in the previous part of the session on the left is china in the middle of europe on the right is the us china you can say is very much using ai to strengthen the state to give the state more control over its citizens the u.s you might say on the other hand the u.s is using ai for uh give more power to corporations uh the google the facebook the apple etc and my hope and many many people hope this that europe can be an alternative because we have the rule of law with european union there's a high level expert group on ai who advocates values like this if you look to the bottom respect for human autonomy prevention of harm defense and explicability so this is my hope that um we put also money to it and entrepreneurships entrepreneurship that europe can can can have an alternative to both the china and the us way of doing lastly after all of this so the examples of the drone combining automation and control a little bit of this a little bit to that and looking at these uh uh philosophers take a project in your mind that you're working on or will be working on are there any social or societal or ethical issues already at stake or maybe at fake potential at stake and how could you how could you address these issues that's too close with thank you thank you mark um i'm looking at the comments whether there are any questions and i already see one uh by mark van mael he asks do you think that the same ethical deliberations will be made when one is on the losing side of war um i would i would maybe think this is a rhetorical question yeah uh difficult so i like to think of on the one hand is ethics and my take on it is ethical deliberation as a process then there's legal stuff i know a little bit of that international law constitutional rights etc human rights humanitarian law and then there's politics uh is there a mandate is there a support within society to send our boys to this or that place and what's the mandate that they will have is it for aggression is it for defense is it humanitarian and and lastly there's the war itself the conflict maybe state to state maybe insurgents maybe a civil unrest within a country so there's so many variations so what i would like to know is the losing side of the war i would guess that one country against another country and they both take seriously and and comply to uh the u.n law etc no then they would stick to what urine also indicated meaningful human control and then there's the other end of the spectrum some ugly states and ugly people doing an ugly gorilla war no no that's that's like i don't know whether they do ethical deliberation maybe they do maybe they don't maybe that's a different frame maybe i'm in no position at all to speak of that because i have a roof over my head and etc um so i'm not giving a real answer to that okay thank you i have also a question um so i really like your take on um moral decision making as a process instead of as a judge or i don't remember the other the other one but yeah so it's a process that you use to make better decisions and it makes it a little bit more tangible to also apply it i think in data science processes so i was wondering who do you think to make it a little bit more practical even who do you think that will be most appropriate in a development team or even in a company structure to take responsibility for this step or to actually apply these methods such as value sensitive design and all the other frameworks that you just mentioned and how can this responsibility also be actively carried out so how can you kind of make it part of your operational processes especially in data science projects yeah brilliant question um ideally because that's maybe the the way that you want to post the question ideally all of these levels i'm at the moment writing a book ethics for people who work in tech and in my book proposal to the publisher i was writing something like well it it will be both it will be it will be uh attractive and usable for people at the top because if they are not aware they cannot send down the message that this is important to be responsible ethical it will also be useful for people like uh who do who do the work on the shop floor because they need tools and also the middle level i think that is often uh something that is in a way the most tricky because mid-level managers uh they must translate uh well the easy talk like yeah we're responsible and uh and and yeah we want to be responsible how do you do that so in a way i would say most importantly maybe the middle management maybe product manager maybe project manager make room for it in the project while at the same time of course being economically and financially and commercially viable etc so they must go hand in hand i think yeah i agree so um related to this uh mark berger also asks a question um he says that to refer back to your example the difference i see in developing ethical ai in comparison to the 10th example is that currently we see automation and letting ai decide as a positive factor in many situations as essentially one of its biggest selling points so how do we shift the interest to the more ethical part when the autonomous part is so exciting and this is i think uh has a little bit to do with the same trade-off that you were hinting at uh before yeah yeah yeah yeah i think i think that's why i like this uh this diagram of ben schneiderman so much because yes you can be enthusiastic about it your client can be enthusiastic but yeah we can automate so much but who wants an out of control automated system so put put ethics back in via this vertical axis um letting ai decide no i think i think if if i were to meet the commissioner who said yeah i want a system where the ai decides then i would say really are you sure and then we would spend two minutes exploring oh no no no you don't want that no you want meaningful human control maybe it's nice to also add yuri on to the discussion is it possible fritz yes thank you yuriyan do you have a different answer to this question or the same yeah so maybe it be it's good to to think about this uh idea uh of meaningful human control and and and sometimes the idea sometimes um for example with autonomous cars uh when a near accident happens then there's this tendency oh we shift control to the human driver and then we solve that problem because then it's human control and then it's the human who makes the uh the accident and who can be hold accountable for that but you cannot design a system where the human is supposed to watch an autonomous system which does the whole task with fully autonomously for three hours and then intervene in this in the in in a moment and take over control in a difficult situation when there is a a a complex traffic environment around so that's that's more that's a little bit um you should be very careful not to introduce this kind of solutions because they tend to make things worse and in those kinds of situations i think that it's preferable to let the automation solve the problem and not involve the human in a way when the human cannot really be involved or is not informed at the moment and does not have sufficient time thanks um i have another question from celine bowen all the way from kenya and her question is after integrating ethics in the machine who will be liable for the outcomes yeah so if i can start on that year please add to that i'm first interpreting the question integrating headings within the machine i'm interpreting that as if a lots of ethics goes into the machine in the example that that durian showed that was the one to the right i think um so there's first a process of what is the good thing to do uh you can do bentham you can count you can do all of these things and then put all of it into the machine so that hopefully in many many cases it will do the good thing the right thing who will then be reliable for the outcomes well it will be silly to sue the machine because you can't find it you can't jail it it makes no sense so uh product liability would say the producer but could also be uh they can send it further to the to the programmer but there's also i think a role for the person who was actually using it so the operator or the user uh because uh i can easily imagine that many manufacturers and developers of these systems will put a clause in it with big small print saying at all times in all circumstances you the user the operator uh is is still liable yeah this relates also a little bit to a different question that was posed earlier that referred to the to the trolley problem which is of course a very uh hot uh example from philosophy from ethical philosophy that is briefly stating that um if there's um what's it called splitting um yeah it is a cross junction uh for a train and somehow uh the train can't break anymore um you have two choices either you send it over one track and then it will kill one person and the other track there are three persons uh so if you don't do anything it will hurt three people and if you uh do change like oh yeah five flip a foot in a paper from 1970 67 i'm writing a paper on it now the title that that's done only one person will die but still yeah the title of the paper that i'll be writing about is the trolley problem problem yes thank you can you explain it a little bit yeah sure sure it was a hypothetical it was a thought experiment to explain the what's the word for it you do something good but something bad happens from it you say five but one gets hurt or dies and it was a thought experiment philippa foot in 1967 never meant it to be an engineering problem that needs to be solved with math uh or calculations one thing that is wrong with it i think is that uh the people are there now in the room with us we are not passive we're not bystanders so you're asked to imagine yourself standing there with a lever really the only thing you can do is do the lever there's no yelling there's no redesign of the brake system there's no oversight over the maintenance people who did a job well it's not possible to to put a mic and warn those people so uh in reality all of us have much more agency than the one binary choice left to right with this lever that's one thing and i've also hinted alluded to well that's a broader context why did the brakes malfunction bad engineering bad design bad maintenance was there no oversight how about the work and health conditions for the people working on the track because that that was the original example so yeah enough on the trolley program right yeah thanks um okay so uh there are still a couple of questions um maybe mark or yurian has the time to later answer a couple of them on linkedin through texts or i don't know maybe maybe if you have the time that would be nice because for this session we come to a close fritz can you back my presentation let me see oh let me get my notes so um thank you both for your presentation it was very informative i think it also inspired a couple of very nice questions and discussions um what is also interesting is that we are planning on an edition where we will delve a little bit deeper into this very sensitive design as a method that mark mentioned which focuses really on involving stakeholders exploring these value tensions or conflicts that that mark mentioned and how to steer your project in a more desirable way through this process of talking through these value tensions um also mark julian as a thank you gift i'm wearing this sweater it's going to come your way but i will have to get your addresses and sizes first so if you can stay a little bit linger on the platform a little bit after we close then i can get your addresses and sizes the upcoming events are listed on this slide so as you see the next edition will be hosted by itopia once more mark and natasha who also hosted the december edition are happy to welcome guest speakers lucia lohair from gina ai which is located in berlin and ricardo vinueza from kdh royal institute of technology in stockholm this will happen on february 10th at 4 30. uh so this is wednesday uh be mindful of that and then on thursday march 11th at four uh i will have the pleasure to interview marine marcus from coprimini about um yeah the the value of interdisciplinary data science uh and by that by that time uh we also hope to be able to welcome you and our guest speakers in our studio at the ramstadt again so um if you have any questions if this webinar series spikes your interest and you want to discuss your ideas with us please feel free to get in touch with anamika laws who is our cco and you can reach her through the email address shown on the slide we hope to welcome you in one of our next webinars stay in touch and up to date by following our company linkedin page and see you next time and enjoy your evening thanks you