Can 95% AI accuracy still miss important requests?

Yes. A classifier can be 95% accurate overall while missing one quarter of the requests that need special handling. Before accepting an AI accuracy claim, count the missed cases, the unnecessary referrals and the workload sent to reviewers—not just the percentage of correct answers.

For a US operations manager evaluating AI routing of routine internal requests, the practical question is: does this configuration find the exceptions we need to find, within a review process we can actually operate? The fictional example below turns that question into a completed decision record. It is not a test of any real AI product.

First name the positive class. Here, “positive” means an equipment request needs a specialist because it falls outside the organisation's standard catalogue. It does not mean the request is good, approved or urgent. Every request still receives normal human handling; the classifier only proposes a route.

Accuracy counts correct classifications across all requests. Recall counts the proportion of actual exceptions found. Precision counts the proportion of flagged requests that really are exceptions. Google's Machine Learning Crash Course defines these measures and explains why class imbalance and unequal error consequences can make accuracy alone misleading. Source 1

Keep the reference label separate from the prediction. In a real evaluation, competent people establish the reference outcome against written criteria; the AI's own answer cannot verify itself. Record disputed or unresolved labels separately and resolve them before treating the evaluation as complete.

All inputs and outcomes in this example are invented. Assume 200 internal equipment requests have agreed reference labels: 188 standard requests and 12 requiring specialist review. Configuration A produces the following counts. Rows describe the reference classification; columns describe the AI's proposed route.

The nine correctly flagged exceptions are true positives (TP). The three missed exceptions are false negatives (FN). The seven unnecessary specialist referrals are false positives (FP). The 181 correctly routed standard requests are true negatives (TN). Explicit row and column labels prevent a common mistake: reading the matrix backwards. The scikit-learn documentation likewise identifies its matrix rows as reference classes and columns as predictions; other presentations can use a different orientation. Source 2

Three calculations describe three different outcomes:

• Accuracy: (9 + 181) ÷ 200 × 100 = 95%. Ten classifications are wrong.
• Recall: 9 ÷ (9 + 3) × 100 = 75%. The classifier misses three of the 12 exceptions: 25% of the exceptions.
• Precision: 9 ÷ (9 + 7) × 100 = 56.25%. Nine of the 16 specialist referrals are justified; seven are unnecessary referrals under the reference rule.

The denominators matter. Three missed exceptions are 1.5% of all 200 requests, but 25% of the 12 cases the exception route exists to find. Both statements are numerically true; they answer different questions.

Compare A with a deliberately unhelpful baseline that routes everything as standard. That baseline gets 188 ÷ 200 = 94% accuracy, finds none of the 12 exceptions and sends no requests to the specialist queue. Its recall is 0%; its precision is undefined because it flags no cases. Do not turn that zero denominator into a claim of perfect precision. Source 1

Configuration A gains one percentage point of accuracy over that baseline and detects nine exceptions. That is useful information, but it does not settle whether three missed exceptions are acceptable.

Not necessarily. Consider a second, fictional configuration B on the same 200 requests. It finds all 12 exceptions but also flags 19 standard requests. The remaining 169 standard requests are routed correctly. These are stipulated comparison results, not a claim that a particular prompt change will produce them.

B's accuracy is (12 + 169) ÷ 200 = 90.5%; its precision is 12 ÷ 31 = 38.71%, rounded. Its lower overall accuracy accompanies better exception detection and a larger specialist queue. Neither percentage describes the whole operating result.

Suppose, only for this fictional decision, the owner set two conditions before evaluation: no more than one missed exception in the batch, and capacity for at most 20 specialist referrals within the service window. A fails the missed-exception condition. B passes that condition but exceeds capacity by 11 requests. Neither configuration is ready under these conditions. These invented limits are not recommended thresholds or evidence that 200 cases are sufficient.

Do not silently relax the missed-case limit or increase the assumed review capacity to make a configuration pass. Investigate the errors, revise the workflow and test again. Our separate guide to human handoff and review capacity explains why a referral is useful only when someone can act on it.

This completed fictional record preserves what the result supports and what remains unproved:

Record: ROUTE-017-v1; internal equipment requests; standard versus specialist route.

Reference population: 200 labelled requests; 12 specialist, 188 standard; all labels resolved for this illustration.

Acceptance conditions: at most one missed exception and at most 20 specialist referrals per batch/service window; fictional owner-defined limits.

Configuration A: TP 9, FN 3, FP 7, TN 181. Hold: three misses exceed the limit.

Configuration B: TP 12, FN 0, FP 19, TN 169. Hold: 31 referrals exceed capacity.

Next action: inspect the three missed cases and the unnecessary referrals, clarify routing criteria, and evaluate a revised configuration on separate cases before proposing any rollout.

Authority and fallback: the operations owner decides whether evidence meets the agreed conditions; ordinary human routing remains available. No autonomous purchase, approval or denial follows from a label.

For a real record, add case IDs, reference-label rationale, actual output, model/service identifier, prompt and rule versions, evaluation date, reviewer and unresolved issues. Keep confidential request text in an approved system; the summary record need not reproduce it. Retain errors as well as successes.

Use permitted cases from the intended workflow, including the languages, request types and ambiguous inputs it will encounter. Keep a representative evaluation separate from an extra challenge set of rare or difficult cases. Mixing an artificially enriched exception set into the main batch changes the class proportions and the meaning of its aggregate results.

Choose acceptance conditions before inspecting final results. Develop prompts or thresholds on separate development cases, then evaluate the chosen configuration on a reserved set. Google's dataset guidance explains the purpose of separate training, validation and test sets, the danger of duplicate cases and how repeated tuning against a test set weakens its independence. Applied to this workflow, the practical safeguard is to keep final evaluation cases out of prompt tuning. Source 3

Twelve exceptions are too few to infer dependable real-world recall from this illustration. One changed classification moves the observed recall by 1 ÷ 12 = 8.33 percentage points, rounded. Zero misses in a finite batch does not prove zero future misses. Report counts and uncertainty, and seek appropriate statistical expertise before making a wider reliability claim.

If the tool can abstain, time out or produce an invalid label, retain those outcomes and the resulting human workload; the two-class matrices above assume every case receives one valid route. Never discard such cases to improve the reported score. If no actual exceptions occur, recall is undefined for that batch; it does not demonstrate successful exception detection.

This method evaluates a defined classification task. It does not establish that an open-ended chatbot is factually correct, safe or suitable for legal, medical, financial or employment decisions. It also does not measure the quality of the eventual human resolution. Recheck when request mix, reference criteria, model or instructions change; use our workflow regression checks for that separate change-control task.

1 — Google: Accuracy, recall, precision and related metrics. Updated 12 January 2026; definitions, imbalance and zero-denominator explanation.

2 — scikit-learn: Metrics and scoring, confusion matrix. Documentation displayed as version 1.9.1 when checked; matrix orientation. Page-specific publication date not stated.

3 — Google: Dividing the original dataset. Updated 3 December 2025; separate evaluation sets, duplicates and repeated reuse.

Sources checked 17 October 2026. These are technical educational resources, not legal requirements. Worked example and decision record: original QSE Advisory editorial analysis with entirely fictional inputs. Prepared with AI-assisted research and drafting. Publisher and author: QSE Advisory, office@qseadvisory.com. Reviewer and Assessor: Laraib Naveed Ahmad.

This article is general information. Verify current requirements and seek qualified advice for your specific circumstances.

Spotted an issue? Contact our editorial desk.