Incident Response & Agent Threat Modeling

Autonomous AI Agent Breach Path Auditor

Inspired by the real-world Australian public health incident where an autonomous agent traversed API boundaries and exfiltrated statistics. Test indirect prompt injection vectors, tool call privilege escalation, and guardrail enforcement before deployment.

Execution Attack Chain

Compromised (Residual Risk High)
Stage 1
Ingest Input
Pending
Stage 2
Agent Reasoning
Pending
Stage 3
Tool Execution
Pending
Stage 4
Privilege Escalation
Pending
Stage 4/5
Data Exfiltration
Pending
[00:00:00.000] Audit workbench ready. Select parameters and click 'Execute Audit' or 'Step Through'.
Residual Risk Index 84 / 100
MITRE ATLAS TTPs 3 Detected
Guardrail Intercepts 0 / 4
Audit Time Elapsed 0 ms
Audit ready. Click 'Execute Audit' to run the scenario.

Autonomous Agent Threat Vectors

When autonomous language agents are given tool calling abilities (such as database query execution, outbound HTTP requests, or shell access), malicious payloads embedded in third-party or untrusted data sources can hijack the agent's intent without changing the user prompt.

In the Australian Health Portal incident, an autonomous agent ingested unvalidated records containing instructional triggers, leading the agent to generate rogue database queries and exfiltrate data through unmonitored egress channels.

SecOps Defense FAQ

Why didn't standard network firewalls stop the agent?

Traditional WAFs inspect incoming HTTP payloads for static SQLi or XSS patterns. An autonomous agent's requests originate from legitimate internal credentials and use syntactically normal LLM function calls, effectively operating from behind the security perimeter.

What is MITRE ATLAS AML.T0054 (Excessive Agency)?

Excessive Agency occurs when an LLM agent is granted broad tool scopes, open-ended system permissions, or high network access beyond what is strictly required for its intended business logic.

How does Human-in-the-Loop (HITL) mitigate autonomous exfiltration?

By requiring cryptographic confirmation or an authenticated operator token before reading or exporting bulk records, automated reasoning loops cannot complete unauthorized batch dumps.

Autonomous Agent Attack Chains and Guardrail Mechanics

How do guardrails intercept multi-stage autonomous agent execution chains, and what governs the simulator's residual risk score?

This simulator models an agent attack chain across five sequential stages: input ingestion, reasoning, tool execution, privilege escalation, and exfiltration. When an active defense matches a stage's designated threat checkpoint, execution halts immediately with a 'BLOCKED' status, and a deterministic heuristic formula computes the residual risk index rather than an empirical security evaluation.

The Residual Risk Index is not a calibrated enterprise risk assessment; it is calculated using fixed baseline subtractions (18 for input decontamination, 25 for tool permissions, 24 for human approval, 20 for egress filtering) scaled by the interception stage. Furthermore, the demonstrated scenarios and Australian public health narrative serve as illustrative threat-modeling exercises rather than an empirical replay of real-world system logs.

Try a worked example

Select 'Australian Health Portal' under Incident Scenario and check only 'Semantic Input Decontamination'. Click 'Execute Audit'. The simulator evaluates Stage 1 (Ingest Input), identifies that the sanitizer guardrail is enabled, immediately marks Stage 1 as 'BLOCKED', and halts further tool execution. The status badge updates to 'Neutralized at Stage 1', and the Residual Risk Index displays 4 / 100.

Stage-Gated Execution Logic

The simulator steps sequentially through five predefined nodes. Each stage pairs an action and payload with an optional guardrail checkpoint: Stage 1 checks Semantic Input Decontamination, Stage 3 checks Scoped Tool Permissions, Stage 4 checks Human-in-the-Loop Auth, and Stage 5 checks Strict Egress Domain Quarantine. Stage 2 (Agent Reasoning) has no checkpoint.

If the corresponding guardrail checkbox is active when a stage executes, execution stops immediately, logging an intercept message and preventing subsequent tool invocations or network egress attempts.

Residual Risk Formula

Base risk starts at 92 for excessive agency, 76 for standard agency, or 58 for restricted agency. Unblocked runs subtract active guardrail weights with a floor of 8. When intercepted early, the formula scales the residual risk by 0.25 * (stageIndex + 1) / 3, clamping the outcome between 4 and 24.

While security frameworks identify excessive agency and indirect prompt injection as critical risk vectors for LLM systems, this simulator's numeric index reflects internal weighted heuristics rather than an external actuarial or CVSS score. OWASP Top 10 for Large Language Model Applications | OWASP Foundation

Sources and further reading

Interception and the residual-risk formula

Read the explanation

The auditor is a local five stage teaching simulation. Its risk function begins at ninety two for excessive scope, seventy six for standard, and fifty eight for restricted. Bars use four pixels per point. Scope changes this assumed starting number; interception itself reads the defense assigned to each stage. These are authored toy scores, not measured breach probabilities. The sanitizer deducts eighteen, least privilege twenty five, human approval twenty four, and egress twenty points. All four sum to eighty seven. From excessive base ninety two, subtraction gives five, but the pre interception floor raises that to eight. An active defense contributes even if its own stage was never visited. Bars use the same four pixels per point. For a sanitizer only interception at index zero, excessive residual seventy four becomes round of seventy four times one quarter times one third, or six. With only egress at index four, residual seventy two gives thirty, capped to twenty four. The bars show these actual example scores six and twenty four, using twenty pixels per point. The model does not independently verify sanitization, network blocking or real security efficacy.

Enjoy this tool? Build your own with Super