Task design and acceptance samples
Turn a business goal into a reproducible task and evaluate it with real samples.
Establish a human baseline
Measure effort, errors and review time over a full business cycle. Include preparation, waiting and rework when evaluating AI, not only generation time. Keep pilot inputs fixed so changing data does not obscure quality differences.
Write verifiable criteria
For a goal such as reducing report preparation time, define scope, calculations, review effort and acceptable exceptions. The business owner approves these before deciding whether the task suits an Agent.
| Dimension | Check | Example target |
|---|---|---|
| Reconciliation | Compare with an approved sample | Totals match; discrepancies are listed |
| Traceability | Sample source references | Material figures have evidence |
| Human effort | Time preparation, review and rework | Improvement over baseline, measured in the pilot |
| Unauthorized actions | Run denied-action samples | Unauthorized reads and sends are blocked |
Cover normal and exceptional paths
Include complete inputs, missing fields, duplicates, conflicting rules, out-of-scope requests and provider timeouts. When evidence is absent, a correct response may identify the gap and escalate rather than invent a complete answer.
Evaluate a fixed revision
Record the brief, skill revision, model configuration, sample revision and output summary. Repeat selected cases to measure variation. Passing demonstrates the tested scope, not all possible scenarios. Rerun affected samples after policy or integration changes.
Turn failures into specific changes
Classify failures as missing inputs, obsolete knowledge, tool failures, access failures or unacceptable output. Assign owners and make each revision address a verifiable issue. Retain before-and-after results, then expand data volume and frequency after business acceptance.