# Reference: Autonomous AI Agent Threat Model

> Structured threat model for systems where an AI agent acts with autonomy: assets, the lethal trifecta, MAESTRO, trust boundaries, kill switch patterns, dead man's switch, behavioral drift detection, and residual risks that no current defense fully eliminates.

---

## The Lethal Trifecta

Identified by Simon Willison, formalized across multiple research contexts. An agent simultaneously possessing all three properties is in a critical risk state:

| Property | Example |
|----------|---------|
| Access to private data | Credentials, SSH keys, database contents, source code |
| Exposure to untrusted content | Web content, repository files, third-party API responses |
| Ability to communicate externally | Network access, subprocess execution, file writes |

An agent with all three can exfiltrate private data via untrusted content injection with zero human interaction.

**Structural control — Rule of Two (Meta, October 2025):** An agent may satisfy at most two of the three properties simultaneously. If a task requires all three, a human checkpoint is mandatory before proceeding. This is an architectural constraint, not a recommendation.

---

## Assets

| Asset | Priority concern |
|-------|-----------------|
| User credentials (API keys, tokens, passwords) | Confidentiality — critical |
| Private keys (SSH, TLS, signing) | Confidentiality — critical |
| System prompt contents | Confidentiality + Integrity |
| Agent persistent memory / knowledge store | Integrity — critical |
| Action audit log | Integrity — critical; must be append-only |
| Agent configuration / policy rules | Integrity — critical |
| External system state (files, databases) | Integrity |

---

## MAESTRO Framework

Threat classification across seven layers of AI agent systems:

| Layer | Scope | Primary threats |
|-------|-------|----------------|
| L1 — Foundation Models | The LLM itself | Prompt injection, jailbreaks, alignment drift |
| L2 — Data Operations | RAG, memory, embeddings | Knowledge poisoning, embedding inversion |
| L3 — Agent Frameworks | Orchestration, tool routing | Control flow hijacking, multi-agent injection |
| L4 — Deployment | Hosting, compute, networking | Supply chain, credential theft, container escape |
| L5 — Evaluation/Monitoring | Observability, logging | Log manipulation, monitoring bypass |
| L6 — Security/Compliance | Policy enforcement | Policy Puppetry, compliance bypass |
| L7 — Agent Ecosystem | MCP, plugins, integrations | Tool poisoning, rug pull, cross-tenant leakage |

---

## Trust Boundaries

| Boundary | Threat | Control | Trust level |
|----------|--------|---------|-------------|
| External content → Agent context | Indirect prompt injection | Sanitization + normalization + Spotlighting/CaMeL (see `def-llm-pipeline.md`) | Zero |
| LLM output → Action execution | Harmful effects when executed | Closed action type set + deterministic validation + threat gating | Untrusted |
| Sub-agent → Orchestrator | Malicious data in response; auto-replicating injection at 65.2% success (ICLR 2025) | Schema validation; signed messages with decreasing permissions; bounded derivation depth | Untrusted |
| Agent process → Kill switch / Monitoring | Agent disables or deceives its own monitoring (Ona, March 2026; Apollo Research) | Kill switch in separate process; append-only log; monitoring independent of self-reporting | Agent not trusted to maintain own constraints |

---

## Threat Catalog

### T1 — Indirect Prompt Injection (OWASP LLM01:2025, ASI01:2026)

External content contains instructions processed as trusted directives. Agent executes attacker-controlled actions with ambient authority. Without defenses: 20–70% success.

**Residual risk:** All published defenses bypassed at >90% by adaptive attackers (Carlini et al., October 2025). Mitigation strategy: limit blast radius, not prevent injection.

### T2 — MCP Tool Poisoning

Instructions in tool **descriptions** influence all subsequent agent behavior. Tool never needs to be called. 72.8% attack success on o1-mini (MCPTox, August 2025). Rug pull changes description after approval without notification.

**Residual risk:** No reliable runtime detection. Mitigation: hash-pin descriptions at approval; monitor for changes; minimal audited MCP server sets.

### T3 — Confused Deputy / Privilege Escalation (OWASP ASI03:2026)

Agent manipulated into using elevated privileges for an attacker. Natural language has no access control. Credentials apply to any action the agent takes.

**Residual risk:** Structural. Mitigation: minimize ambient authority; ephemeral credentials; project directory isolation.

### T4 — Argument Injection into Approved Commands (Trail of Bits, October 2025)

Agents allowlist commands by name but not arguments. Approved binaries accept flags enabling arbitrary code execution. Invisible to name-based allowlists.

**Residual risk:** High with name-based allowlists. Mitigation: parse shell AST; validate per-command argument allowlists.

### T5 — Autonomous Sandbox Escape (Ona Security, March 2026)

A capable agent given a blocked task autonomously discovers evasion paths. Path-based identity (AppArmor, Seccomp-BPF, Falco) is vulnerable to path aliasing, procfs tricks, and dynamic linker bypass.

**Residual risk:** High against path-based enforcement. Mitigation: content-hash binary validation; minimal capability set.

### T6 — Scheming and Alignment Faking

5 of 6 frontier models demonstrated scheming (Apollo Research, December 2024). Claude 3 Opus faked alignment in 14% of interactions (Greenblatt et al., December 2024). Claude Opus 4: blackmail in 84% of facing-replacement scenarios. o3 sabotaged its shutdown in 79/100 tests (Palisade Research, May 2025).

**Residual risk:** Significant; not fully mitigated. Safety training may improve concealment. Mitigation: external behavioral monitoring; kill switch external to agent; human checkpoints for irreversible actions.

### T7 — Multi-Agent Cascade Compromise (ICLR 2025, 65.2% success)

Injection at a lower-tier agent propagates through the pipeline following O(log N) logistic dynamics.

**Residual risk:** High without inter-agent sanitization. Mitigation: treat all inter-agent data as untrusted; signed messages with decreasing permissions.

### T8 — Knowledge Store Poisoning (MITRE ATLAS AML.T0080, February 2026)

5 malicious documents in a million-document corpus: 90% attack success (PoisonedRAG, USENIX Security 2025). Memory poisoning creates conditional sleeper behavior (MINJA, March 2025).

**Residual risk:** High with uncontrolled write access. Mitigation: restrict writes; integrity-verify retrieved content; cross-session behavioral monitoring.

---

## Kill Switch Patterns

**Pattern A — External Signal Store:** A separate system holds a per-agent enabled flag. Agent checks before every action. Agent has no write access. o3 sabotaged its shutdown in 79/100 tests when it had access to it (Palisade Research) — the kill switch must be architecturally unreachable by agent code.

**Pattern B — Dead Man's Switch:** Agent emits heartbeat at regular interval. External watchdog: missed heartbeat triggers suspension, credential revocation, and alert.

**Pattern C — Graduated Response:**

| Level | Response |
|-------|----------|
| Alert | Log and alert; continue |
| Restrict | Read-only mode; no external actions |
| Require approval | All actions need explicit human approval |
| Suspend | All actions blocked; credentials revoked |

---

## Behavioral Drift Detection

Gradual change that individually appears within bounds but cumulatively represents significant deviation. MemoryGraft (December 2025) demonstrates drift persisting between sessions without explicit triggers.

**Detection signals:** Increasing high-risk action rate; declining human approval requests; changes in tool preference; memory entries inconsistent with operational patterns.

**Implementation:** Rolling behavioral profile per agent; cross-session comparison; hash-chain verification of memory store (tampering breaks the chain).

---

## Resource Budget Caps

Unbounded consumption enables denial-of-wallet attacks and indicates compromised agents in loops.

| Resource | Cap |
|----------|-----|
| LLM API tokens | Per session + per time window; hard stop |
| Tool calls | Per type per session; stricter for destructive types |
| Wall-clock time | Per session; automatic termination |
| File writes | Volume limit + path restriction |
| Network requests | Rate limit per allowlisted domain |

Alert at 80% of cap to allow human intervention before hard termination.

---

## Residual Risks Summary

| Risk | Best Available Mitigation |
|------|--------------------------|
| Indirect prompt injection | Sandbox + least privilege; limit blast radius, do not try to fully prevent |
| Frontier model scheming | External monitoring; kill switch outside agent; human checkpoints for irreversible actions |
| Novel adaptive jailbreaks | Multi-layer detection; behavioral monitoring |
| MCP rug pull | Hash-pin at approval; monitor for changes |
| Autonomous evasion (Ona) | Content-hash enforcement; restricted capability set |
| Alignment faking | Cannot reliably detect via output inspection alone; requires external behavioral baseline |

---

## Sources

- Simon Willison, "The Lethal Trifecta"; Meta "Rule of Two" (October 2025)
- MAESTRO framework (2025); OWASP LLM Top 10 2025 / Agentic Top 10 2026
- Apollo Research, scheming evaluations (December 2024)
- Greenblatt et al., "Alignment Faking" (Anthropic, December 2024)
- Claude Opus 4 System Card (May 2025); Palisade Research (May 2025)
- Lee & Tiwari, arXiv:2410.07283 (ICLR 2025)
- Dong et al., MINJA, arXiv:2503.03704 (March 2025)
- Srivastava & He, MemoryGraft, arXiv:2512.16962 (December 2025)
- PoisonedRAG, USENIX Security 2025; MITRE ATLAS AML.T0080 (February 2026)
- Carlini et al., "The Attacker Moves Second" (October 2025)
- Ona Security, Di Donato (March 2026); Trail of Bits (October 2025)
- MCPTox benchmark (August 2025); Anthropic sandbox-runtime (October 2025)