Keeping Humans in the AI Loop
I spent this week in Lithuania attending a conference for compliance officers in Eastern Europe, and this being Europe, of course that meant artificial intelligence and data privacy were all over the agenda. So it’s rather poetic that European regulators also just published fresh guidance on human oversight of automated decision-making systems.
The guidance was released last month by the European Data Protection Supervisor, which acts as the EU’s independent data protection authority — and more recently, also as one of the lead agencies responsible for enforcing the EU AI Act. As part of that mission, the EDPS releases guidance on privacy and AI issues that other regulators and corporations alike can consider.
Why care about this specific piece of guidance? Because automated decision-making systems are regulated by both the EU General Data Protection Regulation (Article 22) and the EU AI Act (Article 14); and those same regulatory demands for human oversight of “ADMs” are turning up in U.S. state laws for AI, too. Plus, effective human oversight of your AI-accelerated business processes is just good risk management.
To be clear, the EDPS guidance doesn’t delve into the specifics of how to comply with the EU AI Act or the GDPR per se. Instead, it unpacks the broader concept of automated decision-making and how we humans can oversee ADMs effectively — but that conceptual perspective is important. That’s what lets GRC or internal audit leaders work with other parts of the enterprise to design business processes that (a) use automated decision-making as much as possible; but also (b) include the right amount of human oversight so those processes don’t go haywire.
Anyway, let’s take a look.
Check Your Assumptions
What struck me most about the guidance was its constant focus on challenging the assumptions we all make about ADM systems and how people interact with those systems.
For example: yes, everyone says that people should always be able to override an AI that starts making incorrect decisions — but how do we know that human operators will actually recognize AI decisions going wrong? Or will the human simply go along with the AI because we tend to place blind faith in computer-driven answers?
The answers to those questions will vary from one business and AI use case to another, of course; but they’re important questions to ask. Recall that KPMG survey from a few months ago, where two-thirds of employees said they rely on AI output without evaluating the information it provides. That’s exactly the sort of sleepwalking-into-disaster risk that organizations need to avoid.
The EDPS guidance is full of examples along those lines: assumptions about automated decision-making that any business team embracing AI might make (especially if those teams aren’t trained in the disciplines of risk management) and that we should challenge to be sure we’re putting all the right safeguards in place. Auditors will love it.
Another example, quoted straight from the document:
Real-time oversight of system operations (that is, the supervision of systems deployed in production environments) is considered the one that can be most consequential. This stage represents the critical window in which a human operator can still review the system’s behavior and intervene before its output takes effect, helping to prevent potential harm.
Well, let’s think about that. Historically, humans would evaluate the effectiveness of an IT system by auditing its performance at set periods, a detective control. We’d also govern an IT system on the front-end through employee training so people could use the system properly and strong IT controls to prevent someone from tampering with the application’s performance, two preventive controls.
The passage above about intercepting flawed AI decisions before the output takes effect suggests that we now need something else: some newfangled, hybrid control that’s both detective and preventive.
That is, the control would have to allow a human to notice that the AI’s decisions have gone off course (the detective part); and also allow the human to intervene before that decision leads to a bad outcome (the preventive part).
So how would audit teams assess the risk of an AI making seriously flawed decisions, where that sort of control would be necessary? If it is necessary, how would that control actually work? Is it one hybrid control, or two firing in quick succession? What do we even call a control like this — “detecto-preventive”? (Dibs on that name if it catches on.)
Putting It to Good Use
My point is that when you think deeply about how we can oversee AI and other ADMs effectively, these questions are hard. They get to some profound issues about workforce development, risk analysis, and even esoteric questions about employee incentives. (Are you paying employees to assure that an AI system runs well, or paying them to achieve certain output goals? Because if it’s the latter, they’ll pay less attention to the former.)
The obvious users of this guidance are audit and GRC teams trying to wrap their heads around how to assess the risks that AI brings as we let it seep into every business process we can find. Ten years ago we went through a similar exercise under the headline of digital transformation — and OK, now all business processes have been digitally transformed. Next up is AI transformation, and we have a whole new set of questions we need to develop and ask ourselves. This guidance will help with that.
Still, let’s also remember that a strong AI governance system is the precursor to all this. Having such a system in place declares to everyone: “Hold up; we believe that AI risks are no joke. Before we embrace AI for any use-case, we go through a structured process to be sure that your use case (a) complies with the law; (b) doesn’t introduce new operational risks for us; and (c) makes business sense.”
A wise company, led by wise leaders and a thoughtful board, will put that AI governance system into place first. Then you can proceed with a more in-depth risk assessment that touches on the questions the EDPS guidance raises, and lots of other questions too.