The refund that the score cannot justify
An online shop owner must choose an automated agent for customer return requests. Buying the benchmark leader feels defensible: higher accuracy seems to promise fewer mistakes. In this hypothetical example, all costs and workloads are assumed, not reported results. The shop pays for processing and staff review, with a $280 weekly budget and six review hours available.
A missing date changes the decision
A request arrives with item_condition = unused and purchase_date = blank. Refunds require a purchase within the shop’s return window. The benchmark supplied dates; the live intake form did not require them. The leading agent returns “approve.” Staff open the order and find the deadline has passed. That approval lacked evidence. Benchmark accuracy measures answers under particular input conditions. Like a guarantee with conditions, its reassurance stops where those conditions change.
Price the work that reaches staff
Add a rule routing missing dates to staff. For 400 requests weekly, assume the premium agent needs 40 reviews, while the economy agent refers 120, including other uncertain cases. Each review takes three minutes at $30 an hour. Processing fees apply to every request, including those sent to staff.
Premium charges $0.50 per request: 400 × $0.50 = $200; 40 × 3 minutes is two hours, costing 2 × $30 = $60. Total: $260 weekly.
Economy charges $0.10 per request: 400 × $0.10 = $40; 120 × 3 minutes is six hours, costing 6 × $30 = $180. Total: $220 weekly. Both fit; choose economy if these workloads hold and both meet the same policy requirements. It saves $40 weekly but consumes four more staff hours.
Let the capacity limit decide
Before rollout, test both configurations on the same requests with valid, expired and deliberately removed purchase dates. Inspect each decision and review flag: an expired date must produce rejection; a missing date must trigger review. Otherwise, correct the rule and rerun. Time the reviews before accepting the estimate.
Now change the example: only three review hours are available. Economy no longer fits. Premium needs two hours and stays within budget, so choose it if it passes the evidence check.