An AI pilot is easier to assess when the team records its current workflow before testing the tool. This guide explains how to compare task time and output acceptance, set a quality floor, and interpret results without treating a simple before-and-after comparison as proof that the tool caused the change.
Open the free AI pilot outcome tracker. It runs in your browser and can export a CSV or print-ready report.
What the tracker measures
The tracker compares the existing workflow with the AI-assisted workflow on one defined task. Include all work needed to produce a usable result, not just the time spent waiting for a model response.
| Measure | What to enter | What it tells you |
|---|---|---|
| Pilot sample | Number of comparable tasks measured | The size of the evidence behind the comparison; it is not a statistical confidence calculation |
| Monthly volume | Typical number of those tasks in a month | The volume used for a simple time projection |
| Baseline time | Average minutes per task in the current workflow, including review and correction | The current time cost for the task |
| AI workflow time | Average minutes per task with the tool, including setup, prompting, human review, and correction | Whether the complete workflow took less or more time |
| Accepted-output rate | Share of outputs accepted without major correction, using the same definition in both workflows | Whether speed changes alongside a quality signal |
| Quality floor | The minimum AI-assisted acceptance rate your team agreed to before the test | Whether the result meets your stated minimum |
The tracker calculates time saved per task as baseline time minus AI-workflow time. It multiplies that difference by the monthly volume to estimate the time change for a typical month. A positive result means time saved; a negative result means the AI-assisted workflow took longer. The acceptance-rate difference is reported in percentage points—for example, a change from 82% to 79% is minus 3 percentage points.
Set up a fair comparison
1. Define the task and acceptance rule
Write down the exact workflow being tested, who reviews the result, and what counts as “accepted without major correction.” Use one rule for baseline and AI-assisted work. A quality measure is hard to interpret if reviewers change their standard halfway through the pilot.
Choose a quality floor before looking at the results. The tracker can show whether the entered rate meets that floor; your organization still decides whether the threshold is appropriate for the task and its risks.
2. Record a baseline before the pilot
Measure the current process over representative work. Keep the task definition, measurement window, and review standard with the result. If possible, use comparable tasks in the baseline and pilot periods. Note changes to workload, staffing, templates, or other software that could also affect the numbers.
3. Count the complete AI-assisted workflow
Include preparation, data entry, prompt or configuration work, waiting, checking sources, editing, escalation, and rework. If a person must review every output, their time belongs in the AI-workflow measure. Otherwise the comparison can make the tool appear faster by moving work out of view.
For acceptance rate, count the outputs assessed and those that passed the agreed rule. Keep examples of failures and corrections as well as successes; an average can hide a consequential error pattern.
4. Enter results and review the limits
Enter the sample size, monthly volume, average minutes per task, acceptance rates, and the quality floor in the tracker. Treat its monthly figure as a projection from the entered sample, not a guarantee. If tasks vary substantially, separate them into meaningful groups rather than averaging unlike work together.
The tracker does not calculate subscription cost, privacy risk, statistical significance, or causal impact. Pair it with the ROI calculator, the data-risk questionnaire, and the relevant owners’ review before making a scale decision.
Worked example
These figures are illustrative only; they are not results from an AI Charcha product test.
A team measures 20 comparable drafting tasks. Baseline work takes an average of 30 minutes per task, including review. The AI-assisted workflow takes 24 minutes, also including review. At 120 similar tasks per month, the observed difference projects to 12 hours saved in a month if the sample reflects typical work.
But suppose 82% of baseline outputs passed the team’s acceptance rule, while 79% of AI-assisted outputs passed. If the pre-agreed quality floor is 85%, the pilot misses it. The time result alone is not enough to justify expansion; the team should inspect failures, improve the workflow, or stop.
What a before-and-after result can and cannot show
A baseline makes the comparison more informative, but a before-and-after change can also reflect a different task mix, staff learning, workload changes, or other process updates. For a consequential investment or high-impact workflow, consider an evaluation design with a suitable comparison group or staged rollout and get help from someone experienced in evaluation. The UK government’s guidance on evaluating AI interventions discusses baselines and stronger impact-evaluation designs. NIST’s AI Risk Management Framework also recommends documenting metrics, test conditions, limitations, and measurement results.
Use the tracker as one part of the decision record. Keep the workflow scope, metric definitions, sample period, evidence, known limitations, and accountable decision owner together so another reviewer can understand how the conclusion was reached.
Frequently asked questions
How many tasks should a pilot measure?
There is no single sample size that fits every task. Choose a sample that represents the real workflow and its important edge cases. The tracker records the sample count but does not estimate statistical power or uncertainty. If the decision requires a reliable estimate of impact, design the evaluation with an appropriate analyst.
Does improved time prove that the AI tool caused the improvement?
No. The tracker reports the difference between the values entered. Other changes may explain some or all of it. A comparison group or another suitable evaluation design can strengthen the evidence, depending on the workflow and decision.
Should human review time count?
Yes. Include the time needed to check, correct, approve, or escalate AI-assisted output. Measure the complete workflow required to produce an acceptable result.
Is the tracker a financial ROI calculator?
No. It estimates time change from task duration and monthly volume. Use the ROI calculator to add hourly cost, subscription cost, and rollout assumptions, and validate those inputs with the business owner.
Sources
- NIST AI Risk Management Framework: Measure describes documenting metrics, test sets, conditions, limitations, and results when assessing AI systems.
- UK Magenta Book guidance on impact evaluation of AI interventions explains baseline measurement and evaluation designs for estimating impact.