What to review when an AI workflow changes

Describe the change in business terms

Record what changed, when it became effective, and which tasks depend on it. A model replacement may affect the behavior being evaluated. A new tool may expand available actions. Revised source material may alter the basis of a recommendation. A new user group may introduce a different operating context. Ask the owner to explain the consequence for the workflow.

Consider a hypothetical commercial-underwriting assistant. A review found its recommendation acceptable using supplier agreement version 2. The agreement later changes to version 3. The team should identify which reviewed statements depended on the old agreement and examine the affected recommendation against the replacement. Preserve the earlier agreement and review as historical evidence where retention and access policies permit.

Find the assumptions the change touches

Use the previous decision as your starting point. Identify the scope, evidence, tests, access boundaries, and oversight arrangements that supported it. Link each changed component to the relevant assumptions. This helps the owner propose a proportionate review and explain why the remaining evidence is still applicable.

Our suggested questions are:

• Does the change introduce a new action, identity, recipient, or data source?
• Which earlier observations or tests relied on the component being replaced?
• Do approval, escalation, and recovery arrangements still work as intended?
• What new evidence is needed before the affected task resumes or expands?

Keep the exact evidence behind each decision

Capture identifiers for the workflow configuration, model or provider version where available, relevant source versions, test conditions, results, and reviewer decision. Include the date and the limits of the evidence. If a supplier does not expose an exact version, record the available identifier and the uncertainty it leaves. Avoid implying more precision than the team can establish.

Store a new review alongside the earlier one and make the relationship visible. A source’s current web address may not establish what the reviewer saw last month. Where policy permits, retain the reviewed file or snapshot with its identifier and retrieval date. Apply appropriate access and retention rules to sensitive material.

Decide how the revised workflow proceeds

NIST AI RMF 1.0 calls for continuing risk management as context and capabilities evolve. MANAGE 3.1 addresses monitoring third-party resources, and MANAGE 4.1 includes change management within post-deployment monitoring. The review approach described here is our practical way to connect those considerations to an operational decision.

Have the accountable owner record whether the affected workflow can continue, needs a narrower scope, or should wait for further evidence. Assign follow-up actions and a date for unresolved items. The record should let a future reviewer understand what changed, what was checked, and why the decision was made.