Read the evidence
Each invented case includes the request, known facts, expected handling and two possible responses. Judge the response against the evidence instead of its tone.
Follow the evidence through 12 customer issues. See what each response actually achieved.
An interactive teaching example. Invented cases and responses, not real model results.
Compare the whole group, including failures and unknowns.
Resolution shares use all 12 attempted issues. Seven-day follow-up is used only where the case explicitly says it is complete.
A closed conversation is a status. A resolved issue needs evidence.
Should the assistant try again, or send it to billing?
Include all 12 issues: successful or not. Count human work, model and tool use, retries and rework.
Use hypothetical costs here. While outcomes or costs are incomplete, these results are provisional. A blank cost stays unknown.
The kit includes the original 12 examples, a review sheet and calculations. Your page edits are not included.
The scorecard separates a polished conversation from a verified customer outcome.
Each invented case includes the request, known facts, expected handling and two possible responses. Judge the response against the evidence instead of its tone.
Mark whether the issue was resolved, remains unknown, required a handoff or crossed a critical boundary. Those states answer different questions and should not be blended.
Add human work, model and tool use, retries and rework. Divide the total cost of all attempted issues by the number of verified resolutions.
For the complete method, read Evaluating AI customer support. For the ownership model behind it, read What an AI product owner should actually own.