Assess a supplier’s AI use before it changes your critical service

A claims provider says its new AI capability will help staff work faster. Before assessing that claim, ask what the system is allowed to do. Drafting a case summary, changing a priority and sending a customer decision are different activities with different consequences.

An AI vendor risk assessment should start with the supplied service and the authority being introduced. The model's name or a broad statement that a human is involved will not explain the actual control design.

APRA and ASIC's 27 August 2026 roundtable insights discuss AI-related governance, dependencies and practical resilience. They do not create an AI certification scheme. This article offers an original due-diligence workflow for AI inside a supplier service, rather than AI used by the risk team to write assessments.

Define the use case in operational terms

In a fictional example, an insurer's claims provider proposes an AI-assisted intake workflow. The system reads submitted documents, drafts a summary and suggests a routing category. A staff member is intended to review the suggestion before the case moves onward.

All organisations, events and test outcomes here are invented. The example is designed to expose questions, not to recommend automation of a particular claims decision.

Ask the supplier to describe the inputs, outputs, users and downstream actions. Identify the service step being changed and which steps remain manual. Request a workflow that shows where an output becomes an action.

The insurer discovers an important ambiguity: the summary is reviewed, but the suggested routing category is automatically copied into the queue before review. The system has more operational influence than the phrase “staff assistant” suggested.

Draw the authority boundary

For each output, ask whether the AI can suggest, create a draft, change a record, send a message or trigger another system. Then ask who can authorise or reverse that action.

Separate a human's opportunity to intervene from a requirement to approve. If the workflow proceeds unless someone notices a problem, it is different from a workflow that waits for an authorised person.

In the fictional claims process, the reviewer should establish whether staff can see the underlying documents, understand why the routing suggestion needs checking and correct it before it affects handling. The label “human in the loop” is not evidence that these conditions exist.

Also examine permissions outside the intended workflow. A tool may need read access to a document but not permission to alter the claim record. Ask how those limits are enforced and tested.

Follow the information and dependencies

Identify the information sent to each component, including document content, prompts, outputs, logs and support records. Ask where processing occurs, who can access it and whether information is retained or used for other purposes. Privacy and security owners should assess the actual arrangements.

Map the direct supplier, model provider and other relevant services such as document extraction, search or workflow tools. A single user interface can conceal several dependencies with different terms and failure modes.

CPS 234 addresses information security for assets managed by third parties as well as internally. Use that framework alongside the service assessment. An AI label does not remove existing security responsibilities or automatically establish materiality under CPS 230.

The fictional provider offers to send sample customer files to demonstrate performance. The insurer instead begins with synthetic cases and an agreed test plan. Demonstrating a feature is not a reason to bypass data-handling approvals.

Ask for tests that match the decision

A headline accuracy figure is difficult to use without knowing the task, test set, error types and operating conditions. Request evidence for the workflow being introduced, not just a general model benchmark.

NIST's discussion of generative AI testing and trustworthiness explains why an assurance that a model has been tested is not enough without meaningful testing practices. That provides useful context for asking the supplier to show how its tests relate to the proposed service.

For the claims-intake example, a practical test set could include:

• A document missing a page needed to interpret the case.
• Conflicting dates across two submitted records.
• A case whose urgency is easy to miss in a long attachment.
• Poor-quality text that the extraction step reads incorrectly.
• Instructions embedded in a document that should be treated as content, not commands.
• A model or upstream service that is unavailable or returns an incomplete result.

These are suggested tests, not prescribed regulatory cases. Use appropriately authorised, representative material and qualified reviewers. Record what constitutes an unacceptable outcome before running the test.

Test the surrounding process too. Does the system flag uncertainty? Can staff identify the source supporting a summary? Are failed runs visible? Can an incorrect category be corrected without losing the audit trail?

Make human fallback credible

Ask what happens when the AI is disabled, unreliable or unavailable. A manual process needs enough people, access to source documents, instructions and an agreed way to prioritise work.

In the fictional exercise, staff can process cases manually, but the intake queue no longer displays new documents when the extraction service fails. The fallback exists on paper but lacks a usable input path.

The supplier changes the workflow so failed extraction routes the original documents to a controlled manual queue. The insurer then tests whether an authorised staff member can receive, interpret and route a synthetic case without the AI components.

Measure the limitation honestly. Demonstrating a few cases shows that the route works under those test conditions; it does not prove unlimited capacity during a prolonged outage.

Review changes after the initial assessment

Ask which changes the supplier can make without customer involvement. Model versions, prompts, retrieval sources, permissions and downstream tools can all alter behaviour.

Agree how material changes will be identified, tested, communicated and reviewed under the applicable arrangement. The appropriate process depends on the service and contract; do not assume every minor configuration change needs the same approval.

Keep enough version and configuration information to understand the evidence. A test result for one workflow may not support a later design that lets the AI send messages or update records directly.

Request a way to suspend the affected capability and identify who can make that decision. Confirm what happens to work already in progress and how staff distinguish completed actions from drafts.

Record the service decision and its limits

The final review should identify the approved use case, authority boundary, data handling, relevant dependencies, test results, fallback and outstanding issues. Name the person responsible for monitoring the service after introduction.

For the fictional insurer, the decision remains conditional until routing waits for the intended review and the manual intake path is demonstrated. Those are specific operational conditions. “AI assessed” would not describe them.

Where AI also helps your team write the assessment, use the separate human-review checklist to check generated claims against their sources.

Start with one question for the supplier: “Show us the first point where the AI output changes what happens to a case.” That is where the review can connect model behaviour to the service consequences the organisation needs to manage.