Recheck the workflow when an AI model changes

Recheck an AI-assisted workflow after a material model or configuration change before relying on its earlier acceptance decision. Run a controlled set of representative tasks, compare the outcomes that matter to the business, and record who can accept the remaining limitations. A better-looking answer is insufficient if the changed system breaks an approval rule or loses information needed downstream.

This guide is for US teams operating bounded workplace assistants, with the same management method adaptable elsewhere. NIST’s final Generative Artificial Intelligence Profile, published on 26 July 2024, supports voluntary risk management. Its post-deployment guidance includes monitoring, change management and improvement. It does not prescribe the worksheet below or certify a model update as safe. 1 2

Version the whole workflow, not just the model name. Define critical failure conditions before comparing outputs. Use the same cases and record relevant differences. Keep consequential actions under authorised control and maintain a workable fallback. A small test set supports a bounded decision, not a universal reliability claim.

Record the provider and exposed model identifier, prompt or instruction version, connected tools, retrieval collection, permissions, output format and review process. Include the change notice or observed reason for retesting. If the provider does not expose a stable model version, say so and record the available service identifier and test time rather than inventing version precision.

A retrieval index update can change answers even when the model name stays the same. A new permission can change the consequence of a wrong answer. Separate these changes where practical so the comparison can reveal a likely cause. Begin with the boundaries in our AI workflow with meaningful human checks, then identify which of those boundaries the change could affect.

Select cases from the actual permitted task: ordinary work, known exceptions, incomplete information, conflicting sources and requests outside scope. Use authorised synthetic or public data. Include at least one case where the correct action is to ask for clarification or decline to make a recommendation. A suite containing only easy successful examples cannot examine that boundary.

Define expected properties rather than demanding identical prose. A usable result may have different wording but still preserve the entity, date, units, caveats and escalation route. For each case, state the business consequence of failure and who is competent to judge it. Decide which failures block release regardless of the average result.

Run the comparison in an approved test environment with sending and system-changing actions disabled unless a separate controlled test authorises them. Check what the next person or system receives, including structured fields and omitted values. If output is copied into a spreadsheet, test that transfer; if a reviewer must open a source, confirm that the source remains accessible.

Generative outputs can vary between runs. Where variation could alter the decision, repeat the relevant cases and record the variation instead of selecting the most favourable answer. Choose a repetition plan proportionate to the consequence and available expertise. This article supplies no universal sample size or statistical assurance threshold.

Assume an invented US operations team uses an assistant to draft shipping exception notes for human approval. Its illustrative comparison contains twelve cases. The changed configuration improves wording in eleven, but in the remaining case converts an unknown delivery date into a firm customer promise. All cases and results in this example are invented.

If unsupported customer commitments were defined as a release blocker, the team holds the change even though eleven outputs look better. The owner records the failing case, checks whether the instruction or source-handling path caused the error, and retests the corrected configuration. The team does not describe eleven of twelve as a validated accuracy rate for all shipments.

Suppose a Canadian service desk uses the same workflow. Its cases should include the actual cross-border handoff, timezone convention and permitted customer language. A US-only test does not establish that the shared workflow behaves acceptably for every other location or business process.

The invented shipping-note comparison should produce a decision record rather than only an eleven-of-twelve headline. Use the following completed fictional interpretation; it is not a benchmark of a real product.

The eleven improved outputs remain useful findings but do not cancel the release blocker. Also record whether both configurations were given the same source version and permissions. Otherwise a difference attributed to the model may actually come from another changed part of the workflow.

Do not select only a favourable rerun. Preserve relevant variation and let the authorised owner determine whether the corrected configuration satisfies the defined conditions. If the supplier cannot restore the prior version, the fallback entry should say “manual drafting” or another actually available arrangement, not “rollback available”. The record supports one bounded release decision and cannot establish performance for unseen shipment types, languages or customer commitments.

Give the release decision to someone authorised to accept the operational consequences. State permitted uses, remaining limitations, monitoring owner and the trigger for stopping. If the previous configuration cannot be restored, define a manual route or suspend the affected function; do not label an unavailable rollback as a control.

After release, inspect relevant exceptions and near misses under the organisation’s access and retention rules. Add meaningful new failures to the test set while preserving earlier cases for comparison. NIST recommends maintaining post-deployment monitoring processes and evaluating their effectiveness. The practical aim is to learn whether the accepted boundary still holds as use changes. 1

No. It may inform selection, but it does not cover your instructions, information, permissions, downstream process and reviewer. Record what external evidence does and does not address.

No. Define the properties that affect correctness, permitted action or usability. Review cosmetic differences separately so they do not hide a consequential regression.

Use the version information and monitoring signals actually available. Recheck affected tasks when material behaviour changes are observed and restrict use if the evidence no longer supports the accepted boundary.

Prepared with AI-assisted research and drafting. Suggested methods and hypothetical examples are editorial guidance, not reported client results. QSE Advisory is part of QSE Advisory. General information; obtain competent review for consequential use.

1. NIST Generative Artificial Intelligence Profile — Final NIST AI 600-1, July 2024; MANAGE 4.1 and 4.2 address post-deployment monitoring and improvement.

2. NIST publication record — Publication date 26 July 2024; describes voluntary use with the AI RMF.

This article is general information. Verify current requirements and seek qualified advice for your specific circumstances.

Spotted an issue? Contact our editorial desk.