On 19 October the FCA published its good and poor practice findings on AI in financial services. The headline is that there are no new rules, but that misses the point. It is actually the most pointed thing a UK regulator has said so far about how firms are actually deploying AI. It passes judgement on the playbook most of the industry has settled on: put a large language model in the workflow, add some guardrails, and ask a person to check the output.
The paper draws on supervision, AI Live Testing, the Supercharged Sandbox and the work around the Mills Review. The FCA is clear that the Principles, SYSC, the Consumer Duty, SM&CR and the complaints rules already apply, and that firms own the outcomes of AI they build or buy. What’s new is the detail on what good and poor practice look like in the real world. Six things stood out to me.
Human review is not a control on its own
The FCA calls out excessive reliance on human-in-the-loop as poor practice, because review can hide weaknesses in the model rather than fix them. It describes a firm where reviewers drifted into accepting AI recommendations with less and less scrutiny, and where low-confidence outputs went through because nobody had defined when to escalate. Anyone who has watched a team work through its four hundredth AI suggestion of the day will recognise this problem.
The good practice it describes is to decide up front where AI should support, where it should automate and where it should abstain and hand over to a person. Build the checks into the system itself, and be able to prove they work.
In one case a firm had reviewers assess cases without seeing the AI’s answer, precisely to stop people deferring to the machine. Of course, if humans are doing meaningful work in checking, the economic benefit of automating decisions with AI is undermined.
The problem with AI in regulated decisions was never that it’s not clever, but that it’s inconsistent and unreliable, and we keep handing it authority anyway, then asking a tired human to catch the mistakes. You cannot review your way out of an unreliable system. A person signing off something they can’t interrogate isn’t oversight. It’s a signature on someone else’s guess. Go and ask Uber, who were fined €825 million for a GDPR breach in part because the human oversight of algorithmic decisions was not classed as meaningful.
Explainability means outcomes, not architecture
The FCA marks down firms that take a purely technical approach to explaining their models. It wants firms to be able to show good outcomes, spot poor ones, understand why they happened and support challenge and redress. It also criticises handing customers technical information without saying what it means for them.
For years “explainable AI” has meant feature importance scores and attention maps, which tell a data scientist something about the model but means nothing to a complaints handler. The question a regulator, an ombudsman or a customer actually asks is simpler: why did this person get this decision, and was it right under your policy? If you can’t answer that for an individual case, with the rules and facts that produced it, you don’t have the kind of explainability that the FCA wants to see.
Guardrails can make things worse
In one case study, a firm added guardrails to its AI tool to reduce drift and unexpected responses. It kind of worked, except the tool then also started giving customers generic answers and steering some of them towards options that weren’t right for them.
That’s the trade-off at the heart of bolting controls onto a probabilistic system. You’re allowing it to make a mistake and then trying to catch it.
And having probabilistic, non-deterministic, and opaque language model-powered agents all check each others work is not a route to precision, determinism, and auditability.
Accountability lands on a named person
On the Senior Managers and Certification Regime, the FCA is careful. Senior managers don’t need to be technical specialists. But neither can they hand responsibility wholesale to technical teams. They need auditable records of decisions, rationale, challenge and escalation to show they took reasonable steps.
Think about what that means in practice. If the logic that decides who gets credit, which claims get flagged or which transactions get escalated exists only as statistical weights that three data scientists half understand, the accountable senior manager is signing off something they cannot inspect.
The paper doesn’t say so in as many words, but the direction is obvious. The decision logic needs to be deterministic and something a business owner can read, challenge and own.
Agents need governance before they need autonomy
The FCA expects firms to build governance layers before deploying more autonomous agentic systems, and describes firms keeping a central register of their agents: who owns each one, what it can access and what controls apply.
Good, because the industry conversation about agents to date has been almost entirely about capability.
The more useful question is what an agent should be allowed to decide on its own. My view is that agents should orchestrate, and the decisions should come from something deterministic, governed and auditable that the agent calls. An agent that can act in the world but whose judgement you can’t inspect is exactly the combination of power without reliability the regulator is circling.
And then there’s your vendor
The third-party section is blunt. Don’t rely on vendor assurances you haven’t tested. Get contractual notice of model changes. Plan your exit. One firm built a product around a particular model, the model was retired, and the product had to be redesigned.
An architecture that makes sense is using language models at the edges for orchestration, interface, and data wrangling, plus a decision layer that can reason deterministically over policy encoding to handle the decisions.
This insulates the organisation from any individual frontier model provider which become swappable without needing to impact the way decisions are automated. Sensible, given the politicisation and chaos that’s currently existing with the frontier labs
There’s a deeper point here. If your credit policy or your fraud rules effectively live inside someone else’s model, you don’t own your decision logic. You rent it, on terms that can change without you. A 200-page PDF policy is interpreted differently every time somebody uses it. What many don’t realise is that the same 200-page PDF policy is reinterpreted every time a language model uses it as context. Prompt engineering is a misleading term. It’s not an engineering discipline at all. If it was, you’d get the same answer to the same question every time, and you don’t.
What is possible is to take that 200-page PDF and turn it into a computable knowledge architecture that is precise, deterministic, and auditable. No hallucinations and no black box. Knowledge management has become fashionable again, not because it gives value to people, but because it gives value to agents. Firms should own the encoding of their own policy, independent of whichever frontier model is fashionable.
What it doesn’t say
To be fair to the paper, it is technology-neutral. It accepts LLM tools restricted to approved sources with guardrails and escalation routes, and it discusses using one LLM to assess another’s output. It is not a ban on anything.
But read the good practice list end-to-end and it describes a set of properties: controls built into the system, abstention when the system doesn’t know, explanations tied to individual outcomes, complete audit trails, logic a senior manager can own, and independence from any single model provider.
Deterministic reasoning has those properties by design and construction. Probabilistic systems can approximate them, at the cost of layer upon layer of guardrails, judges, reviewers and monitoring, each of which the FCA now expects you to evidence.
That’s the bet we made at Rainbird thirteen years ago, and it’s striking to see the regulator describe good practice in terms that sound so much like it. The firms that struggle under this paper won’t be the ones using AI, they’ll be the ones that get the architecture wrong and automate decisions that can’t show their working.
