Will Your Agent Stall Alert Fire Too Early?
Replay OpenTelemetry milestone timestamps against collector arrival times. Set the stall threshold and delivery grace from measured delays, and compare alerts with the milestones that actually occurred.
Time queue residence separately from step execution
OpenTelemetry distinguishes the span that produces work from the span that consumes it. The consumer can start well after the producer finishes, so carry a stable job identifier into both and record enqueue time and first worker start independently. Then attach milestones that can advance the job, such as input materialized, validation complete, and result committed. A job enqueued at minute zero, started at minute nine, and validated at minute eleven spent nine minutes in the queue and two minutes executing before validation. If your alert only sees a validation delay of eleven minutes, it may send the incident to the team that owns the validator. Test routing with an intentionally delayed dispatch in an isolated queue and verify that the queue owner receives the alert while the validation timer starts at worker pickup.
Traces | OpenTelemetryReplay late evidence through the actual alert rule
Bedrock AgentCore batches its built-in tool metrics at one-minute intervals. Its invocation counter records Data Plane calls regardless of response status, so consecutive buckets with activity can coexist with a run that has produced nothing usable. Keep those counters for diagnosing the stall and drive paging from application milestones. Export both occurrence and collector receipt times for an isolated test run. In an illustrative replay, set the stall threshold to four minutes and emit a milestone at minute three that arrives at minute six. A poll at minute four sees stale progress, even though the job advanced on time. A two-minute delivery grace lets that event reach the poll at minute six in this example. Replay a genuinely stalled trace too and measure how much extra detection delay the same grace introduces.
AgentCore generated built-in tools observability data - Amazon Bedrock AgentCoreTrace the milestones that count
In your Python runner, start an OpenTelemetry span for each run and create child spans around retrieval, model calls, tool attempts, and final validation. Record each step's duration, outcome, and attempt number. Carry trace context to queued workers so their spans can be linked to the originating request. Capture observable actions and validation results. Leave prompts, credentials, and raw tool payloads out of the trace by default.
Define progress in terms the application can verify, such as a new document retrieved or a draft passing validation. Send a test job through the full path. The trace should connect the request, retries, and final outcome. If a worker reports success but its spans appear under an unrelated trace, repair context propagation before relying on that view for diagnosis.
Traces | OpenTelemetryKeep tool activity and run outcomes separate
Amazon Bedrock AgentCore batches built-in browser and code interpreter metrics at one-minute intervals. Use those metrics to watch tool traffic across runs. Measure completion, time to completion, and validation in your application, where you can tell whether a tool call moved the job forward. An invocation count measures API requests, so retries can raise it while completed jobs stay flat.
Track run latency by workflow, and distinguish tool errors, failed validation, and runs with no terminal event. Record model usage by billing category and reconcile estimated costs against provider billing. Count how many finished runs have a quality verdict and keep unevaluated runs visible. Otherwise, a broken evaluator can make the quality chart look healthier as its coverage shrinks.
AgentCore generated built-in tools observability data - Amazon Bedrock AgentCoreTest a stall and a healthy retry
OpenTelemetry spans can carry timestamps and events. In an illustrative test, suppose a report agent runs every 15 minutes, has a 90-second deadline, and should retrieve, validate, and stage one report. Give its test retriever an empty response on three attempts. Record each attempt and the number of new records returned. Set a test alert to fire after three equivalent attempts with zero new records, and stop the run at its deadline if it is still active.
Then return a new record on the third attempt and let validation pass. The second run should finish without a stall alert. In the first run, the alert should open the relevant trace and identify retrieval as the last attempted step. If the process exits before writing a final event, check that a separate watchdog detects the missing terminal state.
Traces | OpenTelemetryRoute alerts to someone who can act
Claude Code and the Agent SDK can interact with external services, so measure enforcement at the boundary where a consequential action would occur. Record the attempted action category, the permission decision, and an approval reference where applicable. Test with a sandboxed outbound action that requires approval. The approval record should appear before the test destination receives anything. Route any policy breach immediately to the owner who can suspend the workflow.
Use a slower notification for sustained increases in latency, cost, or output defects. Group alerts by workflow and failure reason so one broken dependency yields one investigation. If retries for rate limits keep paging the operator, inspect whether they honor backoff and eventually produce a milestone. Tune the stall rule for that workflow while retaining its overall deadline.
Securely deploying AI agents - Claude Code Docs