AI adoption in financial services: FCA publishes guidance on good and poor practice

Given that global spending on AI is expected to reach over $2.6 trillion this year, it is not surprising that in a recent speech, the FCA highlighted underinvestment in governance around AI as a key area of concern.  In a bid to maintain momentum at the frontier of AI adoption in financial services following the Mills Review (see our article here), the FCA has published examples of good practice and areas for improvement. The examples are drawn from industry engagement, supervisory work and AI Lab initiatives (including the Supercharged Sandbox) and areas of focus included:

1. Consumer Duty and vulnerable customers;
2. Governance and accountability; and
3. Operational resilience.

In this article, we highlight some of the key takeaways from the guidance. For more detail including case studies and a handy jargon buster, the guidance can be found here.

1. Consumer Duty

One of the key messages is that firms remain responsible for consumer outcomes when deploying AI tools and will need to be in a position to explain how consumer outcomes were taken into account throughout the AI tool's lifecycle, from the early design phase through to monitoring post launch. 

Examples of good practice in relation to the use of AI in products or services include:

• In terms of the design phase:
  • Deciding which tasks should be supported or automated by AI and which tasks should remain human-led, including when to redirect customers to a person
  • Defining a clear consumer need or problem, target market and intended outcome before developing a product or deciding whether AI is the right tool
  • Testing for bias risks and different consumer characteristics such as lower digital or financial capability at the design stage and, where unfairness or unjustified differences are identified, addressing these by adding controls

• Once deployed:
  • Giving consumers a choice between AI-enabled and non-AI support for straightforward tasks
  • Explaining to customers that an AI-enabled tool is being used and being clear about its limitations

• In terms of ongoing monitoring:
  • Maintaining auditable records of consumer prompts, outputs, model versions, applied filters and escalation decisions to help monitor outcomes, explainability and investigations where complaints are raised by consumers

Areas for improvement include:

• In terms of the design phase:
  • Implementing guardrails that address a particular risk, such as model drift, with the unintended consequence, for example consumer outcomes become generic
  • Excluding vulnerable customers by default, where no alternatives are provided
  • Assuming AI outputs are clear without testing them by reference to different consumer characteristics such as lower financial capability or accessibility needs
  • Inflexible automated journeys which make it hard for consumers to switch to another channel to get support

• In terms of ongoing monitoring:
  • Long gaps between reviews of live outcomes or vague escalation triggers
  • Lack of testing of text-only indicators of vulnerability

2. Governance and accountability

Whilst the FCA acknowledges that there is no one-size-fits-all model, it has emphasised that firms need to ensure proportionate governance frameworks are implemented in relation to the deployment of AI tools. There is an expectation that firms design AI tools with clear human oversight (including at a senior manager level) and define escalation triggers. The design phase is likely to require a multidisciplinary team, together with continuous monitoring which should both be well-documented.

Areas of good practice include:

• In terms of the overarching governance framework:
  • Establish central AI governance bodies drawing on staff from across the business to ensure consistency and shared learnings, set principles and risk tolerances and inform senior executives responsible for oversight. This can be incorporated into existing governance frameworks for smaller firms
  • Plan governance frameworks that map AI throughout the consumer journey from design to deployment and monitoring
  • Clear SMF ownership for AI risks from the design stage onwards, with defined risk appetite and sufficient information provided to senior managers to make informed decisions, such as providing playbooks to show how incidents would be identified and handled
  • Ongoing training and capability reviews, including at SMF level

• In terms of the design phase of a particular AI tool:
  • Structured challenge and testing before deployment involving staff across the business and clear evidence of executive accountability and sign off
  • Ensuring escalation routes are clearly defined
  • Ensuring there is a human in the loop to spot and escalate key risks 
  • Staff remaining accountable for decisions and exercising independent judgement, such as determining affordability having had sight of an AI affordability assessment

• In terms of ongoing monitoring:
  • Dashboards with thresholds and benchmarks to oversee performance and escalate issues
  • Dedicated post-implementation monitoring groups to check AI systems are working as intended
  • Spot checks and automated scoring frameworks to detect model drift or harmful AI outputs
  • Audit trails and logging AI interactions to help monitor outcomes and deal with complaints

Areas for improvement include:

• In terms of the overarching governance framework:
  • Unclear, fragmented or overly diffused responsibility for AI systems, particularly across technology and business teams leading to ineffective or inconsistent controls
  • Senior managers delegating responsibility to technical teams without sufficient understanding or challenge

• In terms of the design phase of a particular AI tool:
  • Informal and inconsistent processes for testing, challenging and approving AI use
  • Retrospective governance post deployment of AI tools after key design decisions have been made
  • Poor documentation to evidence steps taken to assess and mitigate risks

• In terms of ongoing monitoring:
  • Reactive or lack of monitoring following deployment or narrow monitoring focusing on technical metrics rather than consumer outcomes
  • Lack of AI-specific expertise in second- and third-line oversight functions
  • Lack of meaningful oversight by human reviewers, instead deferring automatically to AI system output

3. Operational resilience

Again, the FCA has acknowledged there is no one-size-fits-all model but has emphasised the importance of recognising third-party dependency and taking steps to mitigate that risk, including proactive management of upcoming third-party product changes or updates.

Areas of good practice include:

• Considering whether to host AI models on company-owned architecture or a third-party cloud graphics processing unit by reference to factors such as capability, security and risk management
• Ensuring third-party suppliers maintain appropriate back-up servers in different locations
• Piloting and stress testing AI tools to identify and mitigate issues before full deployment
• Mapping dependencies into resilience planning and testing contingency plans in the event of disruption
• Reducing single-vendor dependencies and identifying viable fallback options
• Clear records of third-party arrangements and visibility of any model changes

Areas for improvement include:

• Failure of some firms to consider AI when reviewing operational resilience
• Overreliance on untested vendor assurances without challenge or testing
• Overreliance on hard to replace AI providers
• Lack of exit options where circumstances change

Comment

The guidance demonstrates the FCA's commitment to sharing know-how on AI tool implementation as understanding evolves.  The tangible examples of good and poor practice provide a helpful reference point against which firms can assess their own approach. It seems clear that firms that fail to carefully design and test AI tools, or seek to retrofit governance at a late stage, can expect to be singled out for regulatory scrutiny.

For help with any queries or designing governance around AI use within financial services, please contact Sonya Zywko.