IPEN events bring together privacy experts and engineers from public authorities, industry, academia and civil society to discuss relevant challenges and developments for the engineering and technological implementation of data protection and privacy requirements into all phases of the development process.
The EDPS and the University of Karlstad were hosting an Internet Privacy Engineering Network (IPEN) event on "Human supervision of automated decisions" on 3 September 2024.
| When: | 3 September 2024 - 14:00-18:00 CEST |
| Where: |
Physical attendance: Eva Eriksson lecture hall, Universitetsgatan 2, 651 88 Karlstad, Sweden (registration required). Click here to see the venue location. See our Data protection notice for further information |
Agenda
| 14:00 - 14:20 | Welcome introduction |
Stefan Lindskog (Karlstad University) |
| 14:20 - 14:35 |
Keynote speech “Expectations vs. reality: What we expect human oversight to be and what it really is” |
Ben Green, University of Michigan School (remote) |
| 14:35 - 15:30 |
Panel 1: “Human oversight in the GDPR and the AI Act” |
Claes Granmar, Stockholm University Ben Wagner, TU Delft (remote) Moderator: Daniele Nardi (EDPS) |
| 15:30 - 15:50 | Coffee break | |
| 15:55 - 16:35 |
Panel 2: “Main challenges to effective human oversight” |
Sarah Sterz, Saarland University Leila Methnani, Umeå Universitet Moderator: Isabel Barberá (Rhite) |
| 16:35 - 17:35 |
Panel 3: “Power to the people! Creating conditions for more effective human oversight” |
Simone Fischer-Hübner, Karlstad University Jahna Otterbacher, Open University of Cyprus (OUC) Johann Laux, University of Oxford Moderator: Vitor Bernardo, EDPS |
| 17:35 - 17:45 | Concluding remarks | Massimo Attoresi (EDPS) |
Videos
You can access the video recording of the event by visiting our Video section and EDPS Channel.
- IPEN event on Human oversight of automated decision-making: Welcome introduction
- IPEN event on Human oversight of automated decision-making: Keynote speech
- IPEN event on Human oversight of automated decision-making: Panel 1
- IPEN event on Human oversight of automated decision-making: Panel 2
- IPEN event on Human oversight of automated decision-making: Panel 3
- IPEN event on Human oversight of automated decision-making: Concluding remarks
Human oversight of automated decision-making
EU regulations such as the General Data Protection Regulation (GDPR) and the Artificial Intelligence Act (AIA) state that decisions that could have a significant impact on individuals should not be fully automated. Instead, human oversight should be in place to ensure that the decisions supported by automation systems (such as artificial intelligence) are fair and accountable.
An example can be found in Article 22 of the GDPR, which defines that "the data subject shall have the right not to be subject to a decision based solely on automated processing which produces legal effects or similarly significantly affects him or her".
Another example is the Article 14(2) of the AIA, which requires human oversight of high-risk AI systems to “prevent or minimise the risks to health, safety or fundamental rights”. This is also supported by recital 73 of the AIA, which provides a rationale by stating that “appropriate human oversight measures should be identified by the provider of the system before its placing on the market or putting into service”.
The 2019 Ethics guidelines for trustworthy AI include seven non-binding ethical principles for AI, which are intended to help ensure that AI is trustworthy and ethically sound. One of these seven principles is “Human agency and oversight”.
However, some authors point out that there could be a lack of clarity about what “human oversight” means, what can be expected from it, and how can it be implemented efficiently:
“Regulators presumably put humans in the loop because they think they will do something there. What, precisely, are their assumptions about human decision-making and the ways in which it differs from machines?”
“Adding a «human in the loop» does not cleanse away problematic decisions and can make them worse”.
Matsumi, H., & Solove, D. J. (2023). The Prediction Society: Algorithms and the Problems of Forecasting the Future.
“Human oversight policies shift responsibility for algorithmic systems from agency leaders and technology vendors to human operators.”
Green, B. (2022). The flaws of policies requiring human oversight of government algorithms
When a human is placed in the loop carelessly, there is a high likelihood that the human will be disempowered, ineffective, or even create or compound system errors.
Crootof, R., Kaminski, M. E., Price, W., & Nicholson, I. I. (2023). Humans in the Loop.
Real-life events such as the Three Mile Island accident in 1979, and the Air France Flight 447 crash in 2009 show that when human operators are presented with inaccurate information they are not only unable to monitor systems effectively, but can actually exacerbate the potential consequences.
In other situations, such as when Tesla's self-driving cars reportedly handed over control to humans seconds before impact, humans are put in a situation where they are not prepared or able to make a significant enough decision to correct the behaviour of the system.
Although organisations, and particularly decision makers, who choose to use automated decision making systems must be held accountable for the automated decisions their systems make, it is often the case that the human operators interacting with the systems at the end of the pipeline are the ones who are blamed for any poor outcomes.
The aim of the IPEN event is to promote discussion on questions such as the following:
- Don't the requirements for human oversight shift the burden of responsibility from the systems and their providers to the people who operate them?
- Could there be an unavoidable liability of the operator? Let’s suppose a human operator chooses to follow the system's suggestion and turns out to be wrong. Wouldn’t that be seen as an inability of the user to understand the limitations of the system?
And, on the contrary, if the operator decides against the system's suggestion and proves wrong as well, wouldn’t that result in an even worse outcome for the operator, who had clear indicators to decide otherwise? - Article 14 (2) of the AIA (Human oversight) provides that “human oversight shall aim at preventing or minimising the risks to health, safety or fundamental rights that may emerge when a high-risk AI system is used”. Are the provisions of Article 14 clear enough about what oversight measures are expected from humans/providers and what their responsibilities should be?
- If human oversight is a risk mitigation control, how can we measure its impact?
- What does “appropriate” human oversight mean? What are the characteristics that should be taken into account to assess if a human oversight procedure is appropriate or not?
- Could regulations requiring human oversight be paving the way for the production of defective systems?
- How should this oversight happen? In the testing and monitoring of the system? Are we talking about escalation procedures like in a call centre?
- What skills should humans have? Are we talking of engineers that know how an AI system works or are we talking about humanists?
- What would be the legal implications if in the end the AI system cause harm? Who will be accountable legally and morally the user of the system, the provider of the system, the overseer of the system?
- Incorporating humans into the process is costly, may not be scalable and could reduce the speed of systems, so AI deployers might not be inclined to use human oversight. Where should the line be drawn?
Speakers
Ben Green, Michigan School of Information
Ben Green is an assistant professor in the University of Michigan School of Information and an assistant professor (by courtesy) in the Gerald R. Ford School of Public Policy. He holds a PhD in Applied Mathematics from Harvard University, with a secondary field in Science, Technology, and Society.
Ben studies the ethics of government algorithms, with a focus on algorithmic fairness, human-algorithm interactions, and AI regulation. Through his research, Ben aims to support design and governance practices that prevent algorithmic harms and advance social justice.
His first book, The Smart Enough City: Putting Technology in Its Place to Reclaim Our Urban Future, was published in 2019 by MIT Press. He is working on a second book, Algorithmic Realism: Data Science Practices to Promote Social Justice.
Ben is also a faculty associate at the Berkman Klein Center for Internet & Society at Harvard and a fellow at the Center for Democracy & Technology.
Ben Wagner, TU Delft
Ben Wagner is Associate Professor Human Rights and Technology at the Faculty of Technology, Policy and Management and Director of the AI Futures Lab at TU Delft. He is also Professor of Media, Technology and Society at Inholland. His research focuses on the governance of socio-legal systems, in particular human rights in digital technologies, and designing more accountable decision-support systems.
He is a visiting researcher at the Human Centered Computing Group at Oxford University, advisory board member of the data science journal Patterns and on the International Scientific Committee of the UKRI Trustworthy Autonomous Systems Hub. Ben spends 3 days a week working at TU Delft and 2 days a week working at Inholland.
Previously, Ben served as founding Director of the Center for Internet & Human Rights at European University Viadrina, Director of the Sustainable Computing Lab at Vienna University of Economics and member of the Advisory Group of the European Union Agency for Network and Information Security (ENISA). He holds a PhD in Political and Social Sciences from European University Institute in Florence.
Claes Granmar, Stockholm University
Researcher and associate professor (docent) specialised in European Law (EU law and the ECHR system); alumni at Oxford University, the UK (visiting fellow at the Institute for European and Comparative Law 2017/2018 and member of the SCR at the college Lady Margret Hall); Honorary Fellow at Melbourne Law School, Australia, 2019; former fellow at the Houser Global Law School Program, New York University (NYU), USA, and at the Max Planck Institute in Munich, Germany.
Claes wrote his PhD on trade mark rights and brand competition defebded at the European University Institute (EUI) in Florence, Italy, and at Stockholm University. In the course of writing his dissertation, he worked for two periods at the EFTA Court in Luxembourg.
After defending his disseratation at the EUI, Claes has extended his field of research into EU internal market law and EU external trade relations law and policy in the context of constitutional EU law.
He is focusing on digital globalisation and the EU digitial internal market.
Sarah Sterz, Saarland University
Sarah Sterz is a member of the chair since July 2019 and a PhD student with Prof. Holger Hermanns. She is currently the main lecturer for »Ethics for Nerds«, where she teaches relevant aspects of philosophy and ethics to computer scientists. Due to her dual background in both philosophy and computer science, she serves as a member of the Ethical Review Board of the faculty of Mathematics and Computer Science and also of the Commission on the Ethics of Security-Relevant Research at Saarland University.
Received her B.Sc. in computer science in 2017 and an M.A. in philosophy in 2019, both from Saarland University. As a student, she spent some time at the University of Luxembourg and the University of St Andrews, Scotland.
Leila Methnani, Umeå University
Leila Methnani is a Doctoral student at the Department of Computing Science of the Umeå University.
She authored and co-authored the following works «Who's in charge here? A survey on trustworthy AI in variable autonomy robotic systems» (Methnani, Leila; Chiou, Manolis; Dignum, Virginia; et al. -
2024), «Clash of the explainers: argumentation for context-appropriate explanations» (Methnani, Leila; Dignum, Virginia; Theodorou, Andreas - 2023), «Operationalising AI ethics: conducting socio-technical assessment» (Methnani, Leila; Brännström, Mattias; Theodorou, Andreas - 2022), «Embracing AWKWARD! A Hybrid Architecture for Adjustable Socially-Aware Agents (Methnani, Leila - 2022), «Embracing AWKWARD! Real-time Adjustment of Reactive Plans Using Social Norms», (Methnani, Leila; Antoniades, Andreas; Theodorou, Andreas - 2021), «Let Me Take Over: Variable Autonomy for Meaningful Human Control», (Methnani, Leila; Aler Tubella, Andrea; Dignum, Virginia; et al. - 2021)
Simone Fischer-Hübner, Karlstad University
Simone Fischer-Hübner is an expert in IT security and a professor at the Faculty of Computer Science at Karlstad University, Sweden.
Simone Fischer-Hübner has been a member of the Cyber Security Council at the Cybersecurity at the Swedish Civil and IT Security Authority since 2011. She is also the Swedish representative and vice-chair of IFIP Technical Committee 11 on Information Security and Privacy, a board member of the Swedish Forum för Dataskydd, an advisory board member of PETS (Privacy Enhancing Technologies Symposium) and NordSec, and coordinator of the Swedish IT Security Network for PhD Students (SWITS). Member of the MSB (Swedish Civil Contingency Agency) Cyber Security Advisory Board (MSB:s Cybersäkerhetsråd) - since 2010.
She received the IFIP Silver Core Award in 201, the IFIP William Winsborough Award in 2016 and an honorary doctorate from the Chalmers University of Technology in 2021.
Jahna Otterbacher, Open University of Cyprus
Dr. Otterbacher holds a Master in Statistics and a Ph.D. in Information from the University of Michigan at Ann Arbor (USA). She also holds a master in Applied Linguistics from Boston University. Prior to joining the OUC in 2012, she worked as a Visiting Lecturer of Management Information Systems at the University of Cyprus (2006-2009) and an Assistant Professor at Illinois Institute of Technology in Chicago (2010-2012).
At OUC, she leads the Cyprus Center for Algorithmic Transparency (CyCAT), which conducts interdisciplinary research focused on promoting technical and educational solutions for promoting algorithmic transparency and literacy. Concurrent to this, Jahna co-leads the Fairness and Ethics in AI- Human Interaction (fAIre) group at CYENS, a new center of excellence and innovation in Nicosia, Cyprus, in collaboration with two international Advanced Partners, UCL and MPI. Her research has been funded by the EU’s Horizon 2020 Research and Innovation Program (under Grant Agreements No. 739578 (RISE) & No. 810105 (CyCAT)), as well as the Cyprus Research and Innovation Foundation (under grants EXCELLENCE/0918/0086 (DESCANT) and EXCELLENCE/0421/0360 (KeepA(I)n)). In 2022, she was added to the Stanford-Elsevier list of the top- cited scientists in the area of Artificial Intelligence.
Her main research areas include algorithmic transparency & auditing, social and ethical considerations in artificial intelligence, human computation and crowdsourcing, and analysis and modelling of users’ behaviour.
Johann Laux, University of Oxford
Johann Laux is a British Academy Postdoctoral Fellow at the Oxford Internet Institute. He studies the legal, social, and governmental implications of emerging technologies such as AI and Big Data. His current research project asks how human oversight over automated decision-making can be improved and implemented effectively. Johann also works on the Governance of Emerging Technology Project and researches the impact of personalisation and targeting on citizens and consumers.
Johann graduated from the London School of Economics and Political Science with a degree in Governance and from the University of Hamburg with degrees in Law. He earned a PhD in Law at the University of Hamburg, researching the institutional design of courts with mechanisms of collective intelligence. Johann studied Philosophy at King’s College London and was a Visiting Researcher at UC Berkeley, School of Law. Before joining Oxford, Johann was an Emile Noël Fellow at New York University’s School of Law and a Program Affiliate with the Digital Welfare State and Human Rights Project at New York University’s School of Law.
Johann has great experience in public writing and publishing, having previously worked with publications such as Monocle, Frankfurter Allgemeine Zeitung, and Die ZEIT.
Wojciech Wiewiórowski, European Data Protection Supervisor
Wojciech Wiewiórowski has been the European Data Protection Supervisor (EDPS) since December 6th 2019.
He is also an adjunct professor in the Faculty of Law and Administration of the University of Gdańsk. He was, among others, an adviser in the field of e-government and information society for the Minister of Interior and Administration, and the Director of the Informatisation Department at the Ministry of Interior and Administration in Poland. He also represented Poland in the committee on Interoperability Solutions for European Public Administrations (the ISA Committee) assisting the European Commission.
Wojciech Wiewiórowski was also the Inspector General for the Protection of Personal Data (Polish Data Protection Commissioner) between 2010-2014 and the Vice Chair of the Working Party Article 29 in 2014. In December 2014, he was appointed Assistant European Data Protection Supervisor. After the death of the Supervisor - Giovanni Buttarelli in August 2019 - he replaced Mr. Buttarelli as acting EDPS.
Stefan Lindskog, Karlstad University
Stefan Lindskog (1967) received his Licentiate and PhD degrees in Computer Engineering from Chalmers University of Technology, Göteborg, Sweden in 2000 and 2005, respectively. In 2008, he received the Docent degree in Computer Science at Karlstad University, Karlstad, Sweden. He joined the Department of Computer Science at Karlstad University, Sweden in 1990, where he is currently a full Professor. His research focus is on the design of tunable and adaptable security services and security and performance analysis of security services and protocols. He has authored/co-authored one textbook, eight book chapters, and over 70 journal and conference papers.
His research focuses on the design of tunable and adaptable security services as well as on security and performance analysis of security services and communication protocols. He has been actively involved in the design, implementation, and analysis of a number of tunable security services. He has also studied decision support for operator-centric and adaptive intrusion detection data collection.
Moderators
Isabel Barberá, Rhite
Isabel Barberá is the co-founder of Rhite, a legal & technical consultancy firm based in The Netherlands that specialises in Responsible AI and Privacy Engineering. She is a long-time advocate of privacy and security by design, and has always been passionate about the protection of human rights.
She is the author of the open-source AI assessment tool PLOT4.ai, a threat modeling library and methodology that helps organisations build Responsible AI systems. She is member of the ENISA Ad Hoc Working Group on Data Protection Engineering.
Daniele Nardi, Legal Service at EDPS
Daniele Nardi is from Rome, Italy. After graduating in law at 'La Sapienza' University, he attended an MA at the College of Europe (Natolin) in European Advanced Multidisciplinary Studies. After working briefly at the law firm Paul Hastings LLP, he joined the Legal Service of the European Commission in 2008, where he dealt first with agriculture and fisheries and then for the last six years, the protection of personal data. He has represented the Commission before the Court of Justice in more than 100 cases and was involved in preparing legislative proposals and negotiations. Since July 2021, he is the Legal Service officer at the European Data Protection Supervisor, where he coordinates litigation and inter-institutional and other horizontal matters.
Massimo Attoresi, Technology & Privacy at EDPS
Massimo deputy Head of the Technology & Privacy unit of the EDPS, which he joined in 2012. From October 2014 to September 2020 he was also the Data Protection Officer of the EDPS. He provides advice on the impact of technology on privacy and other fundamental rights due to the processing of personal data. Among the topics he has focused on: cloud computing, online tracking and profiling, privacy of electronic communications, privacy and data protection by design and by default, data protection engineering, DPIAs. He graduated as an Electronic Engineer. After some years in the private ICT sector, he joined in 2002 the European Anti-fraud Office. From 2007 to 2012 he worked as Data Protection Coordinator and Local Informatics Security Officer in a Directorate General of the European Commission.
Join us in this discussion on 3 September!