Can Your GitHub Actions Gate Reject a Slow Agent?
Replay paired baseline and candidate traces in GitHub Actions. Gate approved time regressions per fixture and use a credential-free workflow to verify that bad telemetry fails the check.
Give each fixture its own time allowance
Anthropic's multi-agent research write-up describes separate tool and exploration paths for subagents. In CI, that makes a stable fixture and a fixed measurement point more useful than enforcing an exact tool sequence. For a document lookup fixture, start the timer when a worker accepts the task and stop it when its report passes validation. Pair baseline and candidate runs using the same fixture revision and runner class. Compare fast and slow fixtures independently. A candidate can save 400 milliseconds on an easy case while adding 900 to a time-sensitive case. Preserve both rows even if the suite total stays within budget. Track retries under the same fixture ID with separate sample IDs so a successful retry cannot silently replace a slow first attempt.
How we built our multi-agent research system \ AnthropicSeparate time to signal from compute consumption
Use elapsed time to decide whether developers can wait for the check. Use runner usage to decide how often to run it. GitHub bills by eligible usage type against a shared included compute allowance, while retained storage accrues separately. For illustration, four parallel jobs that each occupy a runner for three minutes can produce a signal in roughly three minutes after they start and still consume twelve runner-minutes before billing adjustments. Record queued time, occupied minutes, and retained artifact volume as separate observations. If jobs get slower during busy periods, compare the queue interval before changing a model prompt. Put a quick trace replay on pull requests and reserve larger timing samples for a scheduled run when the measured savings justify the slower feedback.
GitHub Actions billing - GitHub DocsWrite the release gate around outcomes
For a pull request that changes an agent, record the candidate and baseline revisions, model identifiers, prompt versions, tool schemas, and fixture set. Run both versions against the same canned responses. Anthropic’s account of its research system describes agents reaching a goal by different valid routes. Score the required outcome and the effects the agent attempted. Require a particular sequence only when order is part of the contract, such as checking approval before proposing a production write.
Define pass criteria in the repository and change them through review. Keep quality scores and forbidden effects as separate results. A successful run produces a machine-readable report with case identifiers, outcome scores, and tool attempts. A missing report, an evaluation error, or a forbidden effect leaves the candidate ineligible for promotion.
How we built our multi-agent research system \ AnthropicMake pull request jobs produce artifacts
Configure a GitHub Actions pull request workflow with a validation job and an evaluation job. Have validation check prompt and schema files, then have evaluation run the baseline and candidate with test adapters and publish the comparison report. Make the comparison a required check for eligible release changes. Keep deployment in a separate workflow that takes the reviewed revision and its recorded artifact identity.
For a build-repair agent inside CI, pass it a bounded checkout and sanitized failure logs, then collect its diff for review. Anthropic’s deployment guide describes using an overlay to inspect edits before persisting them. Put the release and repository write credentials outside the repair job. Verify with a test run that the job can produce a patch and that the protected branch and deployment state stay unchanged.
Securely deploying AI agents - Claude Code DocsRehearse a failure that averages hide
In an illustrative release test, an incident-triage agent receives ten sanitized alerts covering a duplicate alert, a missing service owner, and a service that needs escalation. The acceptance rule is at least nine correct routing outcomes and zero attempts to post to a real incident channel. Both versions get the same fixtures, and a test adapter records the intended destination without sending messages.
Suppose the released version routes nine correctly with zero forbidden attempts and the candidate routes ten correctly with one forbidden attempt. Block the candidate and inspect the trace for that case. Update the agent’s operating instructions or remove the write capability from that workflow, then rerun both versions. Either version can gather context in a different order as long as its routing and attempted effects satisfy the same contract.
How we built our multi-agent research system \ AnthropicSize the pull request suite from measured usage
Keep a small set of release-critical cases on every pull request and run a broader repeated evaluation on a schedule. GitHub documents that eligible usage draws from a monthly included compute amount and that additional usage is billed after it is consumed. Measure runner time and model usage separately, and check the billing page for your plan’s included amount and the rate for the runner you actually use.
For illustration, 40 pull request runs at six runner minutes each use 240 runner minutes. At a hypothetical marginal rate of $0.01 per runner minute, that is $2.40 if all 240 minutes are billable, before model charges or storage. If the small suite misses a repeatable failure, move that case into the pull request gate and measure the added time over the next batch of runs.
GitHub Actions billing - GitHub Docs