Does Your Support Queue Need an Agent?

Replay the same support cases through a chatbot, a fixed workflow, and an agent. Use OpenTelemetry spans to verify outcomes and compare the human review time each design would require at your ticket volume.

Give policy exceptions their own replay cases

Create three kinds of refund fixture. A policy question passes when the chatbot cites the applicable rule. A straightforward eligible request passes when a fixed workflow records exactly one authorized refund in a fake ledger. An exception involving a split shipment or a prior partial refund passes when the system resolves it correctly or produces the escalation required by your policy. Give each candidate the same inputs and policy revision. Run actions against the fake ledger and inspect its entries before grading any reply. An agent earns its place when it resolves enough exceptions to justify the additional review it creates. For example, if the agent correctly resolves four extra cases but introduces one policy violation, the violation blocks rollout under a zero-violation pilot rule. You can still deploy a fixed workflow for the standard refund cases while investigating the exception.

Building effective AI agents \ Anthropic

Measure reviewer work through the final receipt

Use an OpenTelemetry trace for the initial request and record spans for policy lookup, the ledger operation, and any review work. If a reviewer picks up a ticket asynchronously, link the review trace to the originating operation and capture active handling time separately from queue delay. Close the case when the reply is correct and the expected receipt or handoff exists. A fluent success message followed by a missing ledger entry counts as a failed case. For an illustrative 40-case matched replay, suppose reviewers spend 12 minutes on workflow cases and seven on agent cases. At 400 cases per week with the same mix, the measured difference projects to 50 reviewer minutes per week. Keep an eye on the case mix. A holiday spike in exceptions can change the review workload even when total ticket volume holds steady.

Traces | OpenTelemetry

Trace one request to completion

Write down what a successful interaction changes. For a returns-policy question, success might mean quoting the correct policy and linking to its current version. A retrieval-backed chatbot can do that without permission to change an order. For a refund request, success might mean a posted transaction and a receipt. The chat interface can look identical while the work behind it differs.

Anthropic distinguishes predefined workflows from agents that direct their own steps and tool use. Apply that distinction to the work itself. A prescribed sequence of identity check, eligibility check, approval and refund can be a workflow even when a model extracts the order number.

Building effective AI agents \ Anthropic

Decide who chooses the next step

List the possible actions after the intake form. If a person can draw every branch in advance, implement those branches explicitly and use the model where language varies. Keep consequential actions behind their existing approval policy. If the path depends on what the system discovers, give the agent a bounded investigation with a clear stopping rule.

For example, an agent researching why a delivery is late might inspect shipment events, compare warehouse notes and draft a reply. Let a person authorize compensation until you have measured error rates on realistic exceptions. A tool can be available for reading while writes remain an independent permission.

Building effective AI agents \ Anthropic

Measure resolution on the same cases

Test all candidates on the same mix of ordinary questions, incomplete details and requests outside policy. Count correct answers, completed authorized actions, mistaken changes and handoffs. An illustrative pilot with 100 requests, 70 correct answers and 15 verified actions has 85 successful outcomes only if those groups are disjoint. A fluent reply about a refund counts as a failure when the test requires a recorded transaction.

Record a trace for each trial so reviewers can follow the path through the application. Keep raw personal data out of trace labels and attach a test case ID to the final disposition.

Traces | OpenTelemetry

Set a handoff the operator can verify

Define the event that transfers ownership to a person. Missing authorization, two conflicting records about the same order, or an unavailable payment system are good test cases. The handoff should include the request, evidence already checked, proposed next action and any side effect already committed.

During the pilot, inspect one failed run as a sequence of timed steps. Determine whether retrieval returned an obsolete policy, the model chose the wrong action, or the downstream service failed. Each calls for a different repair. A trace gives the operator a place to look before changing prompts or adding autonomy.

Traces | OpenTelemetry

Sources