Govern the decision, not the model: James Duez at AI in Financial Services 2026

Hello, my name is James Duez. I'm CEO and co-founder of Rainbird. I've been working in compliance decisioning for about 35 years, and I've spent the last 20 of those building systems that automate decisions in regulated sectors, which means I've spent most of that 20 years watching what happens when machines make mistakes.

I'm here to make one argument: most AI governance in the industry is focusing on the wrong object. We assume we have to govern models, but our exposure lives in the decisions those models make. So in the next 20 minutes I want to show you what that gap has just cost one company, why many of the standard fixes aren't working, and perhaps help you look at this slightly differently.

A few weeks ago, the Dutch Data Protection Authority fined Uber 825 million euros. That's the largest fine yet issued under Article 22 of the GDPR. For those who don't know, Article 22 protects individuals from being subject to decisions made solely by automated means.

Here's what happened. Between 2018 and 2022, Uber's software monitored driving behaviour and customer ratings, and when it suspected fraud or when ratings stayed too low, it simply deactivated drivers' accounts automatically, sometimes temporarily, sometimes permanently. No human effectively scrutinised these decisions. One moment you have an income, the next you don't, and no person decided that.

It began with more than 170 French drivers and a human rights organisation. It went through the French regulator to the Dutch one, because Uber's European headquarters are in Amsterdam, and it ended in a fine calibrated against the 4% GDPR cap on roughly $44.5 billion of global turnover. Uber is appealing, so we treat this as contested, but the number is on the table, and the Dutch authority aligned its decision with other European supervisors before issuing it. This is the coordinated European position, not the position of a single regulator.

Now the detail that matters. If you read the regulator's account of this case looking for a technical assessment of Uber's fraud models, accuracy, bias, testing, validation, drift, you'd find that the published decision does not assess any of it. The violation was that decisions with major consequences for people were taken with no meaningful human involvement, and the people affected were never properly told how it worked. The deputy chair of the regulator, Monique Verdier, put it plainly: "A computer should not make decisions on its own that have major consequences for you."

Remember that sentence, because we're going to come back to it. And notice what law we're talking about. This is not the EU AI Act. This is the GDPR, on the books for eight years since 2018, enforced against conduct that predates generative AI and the AI Act entirely. Decision-level accountability is not a regime that's coming. It's already here, and it's just written a nine-figure fine. The AI Act layers its obligations on top. Risk lives in the consequences of automated decisions, not just in the model that's making them.

This matters more than it did a few years ago because of what AI is becoming inside financial services firms. A few years ago the question was whether a model could draft content. Today we're wiring agents into onboarding, KYC, payments, claims, credit decisioning, trade workflows. These are not systems that merely generate text or offer a recommendation. These are systems that make decisions and increasingly act on them. Every one of those actions is a decision with a human subject who bears a consequence and, as of now, a potential price tag if you get it wrong.

Assessed by today's standards, a general-purpose copilot looks pretty low risk. But wire that same model into a workflow with the agency to close a customer's account and you've built a high-risk system, and the model hasn't changed. The authority has changed. So classification starts with an inventory of the decisions and their consequences, not an inventory of models. In financial services this is not sacking drivers. It's debanking, KYC off-boarding, credit declines, fraud flags that freeze an account. The unit of governance has to move to where the exposure is, which raises the real question: how do you control decisions at machine speed?

The industry has two superficially comforting answers. Both have proven fallible.

Comforting answer number one, in some form, is to give your policy or regulations to an AI agent and ask it nicely to stick to them. You take your policies, your procedures, your obligations, load them into a context window or a retrieval pipeline, and trust the model to apply them faithfully at runtime. Some add choreography, spawning multiple agents that check each other's work, all of them probabilistic, non-deterministic and opaque in their reasoning. Perhaps impose some guardrails, allow it to make a mistake, then try to catch it with some rules. This is one of two default patterns in most agentic deployments today, and it's appealing because it's fast, it demos beautifully, and it feels like governance because it somehow involves your policy or regulations.

But understand what's actually happened. The policy is not computable. It is being interpreted probabilistically every time the system runs. Run the same case with the same facts twice and you have the potential to get different outcomes. Input tokens into a language model running an agent can only influence the output tokens. And if you ask why it made a decision, you get an explanation that's very fluent but proves nothing, because it's a narrative generated after the decision, not an account of how the decision was made. I always joke that LLM agents are a bit like consultants: always confident, but only sometimes right. And recording the steps between agents, or what they say they did, is not an audit trail.

This isn't a quality problem. It's a category error. LLMs are magnificent at many other things: language, research, summarisation. They're just pretty poor at reasoning, and I would contend a language model should never be the judge.

Comforting answer number two is to put a human in the loop. In the Uber case the regulator said a human should have looked first. As a principle of accountability, I agree. But as an operating model, taken literally, it doesn't survive first contact with reality. We automate decisions because volume exceeds human capacity, or at least the economics of it. That's the entire point. If a human genuinely reviews every decision, weighs the evidence and exercises judgement, you have not automated anything. And if a human does not genuinely review each one, what you have is sampling, not oversight. Think about the volumes in this room: transaction monitoring alerts, onboarding checks, fraud flags. Imagine per-decision human review that is real rather than nominal, and you can see the numbers do not work.

And when they do install the human, watch what the human becomes. Put a person in front of an endless stream of machine recommendations and their approval rate climbs towards the machine's, because vigilance decays with volume and the reviewer becomes a clicker. This is well documented and it has a name: automation bias. Most of you will have felt its pull. We cannot sustain the level of attention required to be a guardrail to algorithmic outputs. The human checker becomes what the researcher Madeleine Clare Elish calls a moral crumple zone: the least empowered individual in the chain, absorbing the liability for a system they cannot see into and often cannot pragmatically override.

The courts have already called this. In 2023 the Amsterdam Court of Appeal examined Uber's deactivation process in the robo-firing cases, where Uber argued there was human review. The court found the review was "not much more than a purely symbolic act", and that meaningful means informed, competent and empowered to change the outcome. Remember that word: meaningful. A token human in the loop is not a cure. It's a liability. But we do need human expertise in the tech stack, and hold that thought.

Nobody asks an LLM to perform mathematical calculations. We use a calculator, or Excel, or write some code. We've long understood that a calculator is a tool for doing maths with numbers, and an LLM agent can delegate to it when we want to add things up. As it turns out, knowledge as well as numbers can be made precisely computable, at least in narrow domains. Not in a hand-wavy way like a general-purpose LLM, which is trained to know everything at the cost of precision around your thing, but precisely. So there is an alternative.

We do this by treating regulation, policy and institutional knowledge as a first-class computable artefact in your stack. Not documents like PDFs subject to human or LLM interpretation, but structured in a way that is architecture. I'll call this computable policy. Structuring knowledge has been a research area for years; there's been a whole team at Stanford working on it, CodeX, to give one example. Policy can be structured as a kind of map of the concepts, the relationships, the rules and the weights, in such detail that you could reverse-engineer the policy back into a document if you wanted to. The difference is that this policy is architecture.

You can make this happen programmatically, ironically using LLMs, but with deep human oversight. Unlike an LLM agent, this is an explicit, inspectable knowledge representation that sits in version control, just as code does. Before you release it, it's tested against scenario libraries with exactly the discipline you apply to code. It's adversarially red-teamed to find edge cases and faults. It's regression tested, and ultimately it's signed off. When the regulation changes, the artefact changes through release management and the tests run again. And with symbolic inference it can reason in a way that is 100% precise to your policy, entirely deterministic because it's based on logic, and auditable. A first draft takes only a few hours. Your human experts invest time in the build, and the result is your own sovereign institutional knowledge layer, owned by you, that can make judgements at machine scale. Not predictions: judgements. Same inputs, same outcome, every time.

You can describe how an agentic AI is constructed, but that doesn't answer a regulator asking about a specific customer outcome. Why did you decline Mrs Smith a loan? An LLM explaining what it has done is simply making another prediction. It cannot describe a logical process because no logical process exists. If you make policy computable, each decision leaves its own evidence: the policy, the facts, the rules, and how they led to an outcome. This is called a proof tree. It's a logical chain of reasoning, a by-product of inference over these knowledge maps, and you can retain them, so you have a built-in audit trail.

This may sound like a rules engine. A rules engine handles conditions, but someone has to flatten everything into rules first. What we're talking about is encoding a whole corpus of regulation, policy and institutional expertise as structured, reusable artefacts that, when called on, can make any number of decisions. Not like a decision tree, which can only make the decision you've encoded it for, but varying in scope, and at zero token cost.

If you couple this knowledge-plus-reasoning layer, let's call it a decisioning layer, to agents, you have your institutional knowledge as a first-class citizen and a way of deploying high-risk, high-reward use cases safely. So where do the language models go? They stay where they're brilliant: data extraction, translation, orchestrating the whole thing, interfacing with users. An agent reads the unstructured mess of the real world and presents data as structured facts. It delegates those to the decision layer of computable policy, which adjudicates on the facts and returns an answer that comes with evidence. That evidence is a receipt that the decision is a logically reasoned judgement, not tokens from an impressive token-prediction machine. Language models are essential for helping draft this knowledge layer and for working with your experts to create evals and tests before deployment, where there is time for scrutiny and testing. But LLM agents should never be the judge, because reasoning is their weakness.

Models predict, computable policy decides, and humans govern. That answers the question we parked. You move the experts from being guardrails checking the work of LLM agents to building a type of agent that's precise and auditable by design. Humans still handle escalation at runtime, by exception, because criteria defined in the policy, novel patterns, low certainty, thresholds crossed, can trigger that escalation. That is a cue a human can genuinely work with, with context and with authority, which is exactly what the law means by meaningful. A human in the loop as a checker doesn't scale. Human-designed computable policy does.

Almost nobody in this room is building their own frontier language models, and DORA has made third-party ICT risk a board responsibility: registers of information, contractual provisions, exit plans, oversight of critical providers. So you have to be honest about what you can govern when you're renting a frontier model. You don't control its training data. You don't control the weights or the release cadence of Anthropic's models, OpenAI's models or Google's models. When a new version ships and the behaviour of every prompt-based control you built on it shifts quietly under your feet, you have a problem, and you can't version-control somebody else's model. And do you want publicly trained data influencing critical decisions? Do you want your decision influenced by what somebody once wrote on Substack?

But you do own your policy and institutional knowledge, and it's what separates you from your competitors. It's your structure to inspect, to test, to version and to scale. That changes the contractual question too, because now you own the decision layer. You can swap the frontier model for another at the boundary and the decision logic doesn't move, so the behaviour your regulator cares about won't change.

What do you do with this practically? Take a critical, high-stakes, high-volume decision. Extract its governing policy. Put it into computable form with the people who own it. Put tests around it: the scenarios it must pass, the edge cases that must be escalated. Then wire your agents to delegate to it, as a sub-agent or a tool call, just as you'd call a calculator to compute over numbers. Then measure the things LLM agents cannot give you: precision, reproducibility and an audit trail. That single decision becomes a lighthouse, a template. And there's another benefit. When you encode a policy that has been open to individual interpretation for years, you will find contradictions, gaps and overlaps, which you can then resolve. Imagine the value of finding and resolving those before the regulator does.

We've had decades of thinking about knowledge management as something humans need to go and read. Now we need to think about knowledge management as something AI agents can use. Prompting, fine-tuning and RAG are all very hand-wavy. Your knowledge needs to be a first-class citizen.

I'll close with a test you can run on any automated decision in your estate this week. Five questions.

One: is it deterministic? Can you guarantee the same inputs give the same outcomes, on demand?

Two: can it explain its reasoning? The actual reasoning, not a plausible narrative written after the event.

Three: can you evidence it? Can you prove that the control actually executed at decision time, on that decision?

Four: can you name the accountable human? I mean the owner of the policy, not the person who clicked approve.

Five: could you change the LLM vendor, that frontier model? Could you swap it out, for example due to political risk, models being banned and so on, without the decision behaviour changing?

A few weeks ago, as I said at the start, we learned the going rate for gaps in this list: 825 million euros for decisions taken by software with nobody meaningfully in charge. The fix is not agents checking each other, and it's not a person rubber-stamping a torrent of algorithmic output. It is making regulation, policy and institutional knowledge itself computable, and putting humans in charge of it.

Thank you very much for your time.