Establish a fair comparison
A pilot should answer a decision question: should this team keep using this bounded workflow under the proposed review rules? Before starting, collect a baseline from comparable work completed without AI. Record the full time required to reach an approved result, including corrections. Compare similar task lengths and complexity; an easy reminder and a complicated event change are not interchangeable examples. Keep the source quality and approval standard consistent.
For a manageable course exercise, plan a small set containing straightforward cases, missing details, contradictory details, and bilingual drafts. This is a learning sample, not proof of performance in every situation. Write the success criteria before seeing the outputs. For example, the team might require no unsupported event claims in approved messages, all exceptions routed correctly, and lower total effort. These are proposed pilot criteria, not a certification standard. Assign a person to review the evidence and decide whether further testing is warranted.
Count what happens after generation
Track at least four measures: total minutes per approved output, the number requiring substantial revision, factual defects found before approval, and incidents or defects discovered after release. Always show the denominator. “Two drafts needed correction” means something different out of four drafts than out of forty. Define substantial revision in advance, such as rewriting the main instruction or correcting a material event detail. Include setup, prompting, review, rework, and recurring tool costs when evaluating benefit.
Consider a fictional worked example. The previous process takes 18 minutes per approved reminder. The AI-assisted process uses three minutes of preparation, two of drafting, eight of review, and three of rework: 16 minutes altogether. The net saving is two minutes, about 11 percent of the baseline, not the 16 minutes suggested by comparing generation alone with the old process. If a prohibited promise reaches publication, the speed improvement does not satisfy the pilot’s quality gate.
Improve one thing and retest
Use the failure log to choose a specific change. If drafts invent missing prices, strengthen the missing-information instruction and add an explicit price field to the review checklist. Change one major element at a time when practical so you can interpret the result. Retest both the failing case and other representative cases; fixing one output does not demonstrate a reliable workflow.
At the review meeting, choose continue, revise and retest, or stop. Preserve the decision, evidence, owner, and next review date. Continue only within the tested boundary; adding automatic sending or private attendee data needs a new assessment. Keep a manual fallback so work can proceed if the tool becomes unavailable or unreliable. Recheck after meaningful changes to the tool, source template, audience, or reviewer. A small pilot is successful when it produces a defensible next decision, even when that decision is to stop.
Apply it in your own notes
Design a ten-case practice pilot with fictional sources: four ordinary reminders, two missing-detail cases, two contradictions, and two bilingual cases. Write pass criteria before testing. Create a log with task type, baseline minutes, preparation minutes, generation minutes, review minutes, rework minutes, defects, reviewer decision, and approved-output status. Calculate total assisted effort and compare like-for-like cases. Use the fictional 18-minute versus 16-minute example to check your arithmetic. Finish with a five-sentence decision: what was tested, what the evidence shows, what remains uncertain, whether to continue or retest, and who owns the next check. Do not describe practice results as verified business impact.
Get a worksheet →Practice questions
1. The old process takes 18 minutes. AI preparation, generation, review, and rework take 3, 2, 8, and 3 minutes. What is the net saving?
2. Why record a denominator with the number of corrected drafts?
3. A prompt change fixes one failed example. What is the next useful step?