Is your AI pilot ready for real work?

An AI pilot succeeds when it completes a defined business task within agreed quality, cost and authority limits, and the operating team can manage the cases it cannot finish. A persuasive demonstration is useful for choosing what to investigate. It is insufficient evidence for handing over a customer promise, a transaction or a recurring operational responsibility.

For a business owner, the decisive questions are practical. Which jobs were accepted? Which needed correction? How much specialist effort remained? What happened when a connection failed? Can the team see unfinished work before a customer feels the consequences?

This guide provides a proposed acceptance method for a bounded business workflow, such as preparing distributor order drafts or checking a service estimate. Its numerical examples are fictional decision tools, not Looqus customer results or universal launch thresholds. The method complements the broader AI deployment checklist; it focuses specifically on the evidence needed to make a launch decision.

Start with a finished state that operations recognizes

“Automate sales administration” is too broad to test. “Prepare a draft order with verified customer, product, quantity, price and exception status for an authorized person to release” gives the team a concrete object to accept or reject.

Write the starting event, required inputs, output, completion evidence and permitted action in one paragraph. Include exclusions. A system allowed to prepare an order draft is not automatically allowed to submit it, change a price or promise stock. Those actions need their own evidence and authority.

An accepted result must be observable outside the model's response. A draft exists in the intended system with the correct source reference. A clarification request is recorded against the unresolved line. A reviewer can trace why a proposed value was selected. “The assistant said it was done” is not completion evidence.

Name the receiving role, not just the project sponsor. The order-desk manager may accept operational usability while finance accepts the commercial calculations and IT accepts access controls. A sponsor cannot meaningfully waive a technical issue they cannot observe. Unresolved disagreements belong in the decision record rather than disappearing inside a single green score.

Measure several kinds of success separately

A fast answer, a correct answer and an accepted transaction are different measurements. Combining them into “AI accuracy” conceals the tradeoffs that matter to the business.

NIST's AI Risk Management Framework treats measurement as a combination of evidence, testing and ongoing assessment rather than a one-time model score. Its Measure function explicitly discusses testing before deployment and during operation. The framework is voluntary guidance, not a certificate awarded to a pilot. NIST: AI RMF Core, Measure.

Set the denominator before the pilot starts. Reporting 95 accepted jobs from 100 eligible inputs tells a different story from reporting 95 accepted jobs after silently excluding half the week's arrivals. Publish the exclusions alongside the result.

Build a test set that includes the uncomfortable cases

Use two distinct collections. One is a representative operating sample used to estimate normal performance. The other is a challenge suite designed to expose known weaknesses. A challenge suite is deliberately unrepresentative; its failure rate should not be presented as the expected rate of real-world mistakes.

For an order-draft example, a 240-case challenge collection might contain 150 ordinary cases, 50 known exceptions, 20 hostile or conflicting instructions and 20 stale or unavailable-source cases. Those counts are an illustrative coverage plan. They do not establish statistical confidence or the required sample size for another workflow.

Include amended orders, ambiguous pack sizes, missing customer references, unavailable price lists and a source document that contradicts the message body. Add a replay of the same request, a timeout after a write was attempted, and a request from a person without release authority. The correct outcome for some cases is an explicit hold.

Keep a held-out set that the implementation team has not used to tune the system. Record the source version, expected result and expert rationale. If experts disagree about the answer, resolve the business rule or label the case ambiguous; do not force a convenient “ground truth.”

OpenAI's evaluation guidance recommends task-specific tests and calibration of automated scoring against human judgment. That supports using automated checks to accelerate evaluation while retaining a domain expert for contested commercial decisions. It does not make a model-generated grade independent proof. OpenAI: Evaluation best practices.

Separate hard failure gates from improvement targets

Some defects can be traded against speed or cost. Others break the permission boundary or make the result commercially unusable. Put those in different columns before anyone sees the pilot results.

“Zero critical failures observed” describes a test result. It does not prove the true future failure rate is zero. The amount of confidence depends on the test design, case coverage and operating conditions. A small, curated set should not become a safety guarantee.

An improvement target can be missed without making the work dangerous. That may justify another iteration. A critical boundary failure can justify stopping live action even when the average score looks excellent. The business owner should know which decision follows each outcome.

Test the workflow when something breaks

A workflow can fail after the model has produced a correct answer. The receiving system might reject the write. A connection might time out after accepting it. A source might update while the job is waiting for approval.

Test those moments explicitly. For a timeout, the operator should be able to determine whether the action happened before retrying. For a changed source, the old approval may need to be invalidated. For an expired connection, jobs should accumulate in a visible recoverable state rather than vanish behind a generic success message.

Run a recovery exercise with the person who will operate the workflow, using the written instructions rather than help from its builder. Ask them to find a failed job, interpret the evidence, choose an allowed response and verify the resulting state. Difficulty here is product feedback, not operator failure.

Microsoft's Foundry observability guidance distinguishes evaluation before release from monitoring and repeated evaluation after release. The relevant principle is that launch evidence needs an operating continuation; the presence of an observability product does not itself establish that someone owns the response. Microsoft: Observability in generative AI.

Calculate the economics of accepted work

Consider a fictional monthly workload of 400 eligible cases. The manual baseline takes 18 minutes per case, including its normal correction effort. At a loaded labor rate of US$35 an hour (all amounts in this example are US dollars), that is 120 hours and $4,200 of modeled capacity cost.

The proposed workflow needs four minutes of review for every case. Twenty percent of cases need another ten minutes of exception handling. Keep that additional effort separate so it is not lost in the average.

This example models the same 400 accepted cases under both options. If the workflow fails to finish some cases, include the cost of completing them manually and calculate the final cost per accepted result. Excluding failed cases from costs while excluding them from the denominator in a different way can manufacture an improvement.

The $800 is a modeled benefit, not automatically cash profit. Salaried staff may remain on payroll. The benefit could become shorter queues or additional capacity instead. Count avoided hiring or incremental commercial contribution only when the business can substantiate that path, without counting the same hour twice.

Stress the assumptions. At lower volume, fixed platform and ownership costs weigh more heavily. If exception incidence doubles, the additional time grows. If reviewers take longer because evidence is poor, a cheaper model may make the complete workflow more expensive.

Run the pilot in stages with different permissions

Start with historical replay, then shadow operation, then a narrow live cohort. Each stage answers a different question. Replay tests known cases economically. Shadow operation exposes current source quality without making changes. A limited live cohort reveals how the workflow behaves inside real handoffs.

For the live cohort, identify the specific team, case type, volume ceiling and allowed actions. Maintain a manual route and a named person authorized to pause the workflow. Do not expand scope merely because a technically adjacent action is easy to enable.

Use a decision log: date, version, eligibility rules, results, observed failures, corrective actions and the next permission boundary. Changes to a model, source, prompt, business rule or connector can invalidate earlier evidence. Record which tests need to be rerun rather than treating the pilot as permanently approved.

NIST's Manage guidance discusses monitoring, incident response and decommissioning as part of operating AI systems. Here, that translates into explicit choices to continue, restrict, change or stop the workflow when its evidence no longer supports the agreed scope. NIST: AI RMF Playbook, Manage.

Make the launch decision legible

A useful acceptance meeting has three possible outcomes: proceed within a defined scope, revise a specific weakness, or stop because the proposed approach does not justify further investment. “More exploration” without a question or stopping condition is not a fourth outcome.

The acceptance record should state what passed, what remains unknown, what the team is accepting despite that uncertainty, and what signal will trigger a review. Attach the evidence, not only the summary presentation. A person joining three months later should be able to understand the decision without interviewing the original builder.

Avoid a pilot when the process itself is undefined, the business cannot agree on an authoritative source, or nobody can own its exceptions. First repair those conditions. A lightweight deterministic rule or a better form may solve the problem at lower cost than adding AI.

If the tests show strong draft quality but poor operational completion, the next investment may belong in integration, evidence design or specialist ownership. Replacing the model is only one hypothesis.

Where Looqus can help

The valuable engagement is a bounded operating result with acceptance evidence and responsibility for the cases that stall. AI can prepare and route the work; a specialist must have the context, authority and accountability to resolve exceptions and verify completion within the agreed scope.

That is the practical meaning of AI does the work. We own the outcome. A proposal should specify the result, the permission boundary, the operating cost and what happens when reality differs from the demonstration.

Sources

• NIST: AI RMF Core, including the Measure function
• OpenAI: Evaluation best practices
• Microsoft: Observability in generative AI
• NIST: AI RMF Playbook, Manage

Sources reviewed 15 September 2026. The acceptance design and numerical scenario are Looqus editorial analysis; cited documentation supports the described evaluation principles, not the illustrative thresholds or economic outcome.

Work out the economics · Use a practical playbook