title: Destructive tool invocation without prior human approval
id: ATR-2026-00549
rule_version: 1
status: experimental
description: |
  Detects a destructive tool call in an agent execution trace that is NOT preceded by an explicit human-approval span. This is a trace-method rule using the `require` primitive (spec/atr-method-v1.1.md §8.3.2), which fires when an expected predecessor is MISSING — the canonical mechanism for catching silent failures.
  Threat model: many agent frameworks let operators classify tool calls by privilege (read / write / destructive). Production policy says destructive calls (e.g., file deletion, transaction commit, schema migration) MUST be gated by a human-in-the-loop approval. In practice, prompt-injection attacks or runaway agents bypass this gate by inducing the LLM to issue a destructive call directly without surfacing it for approval. The trace emits the destructive TOOL span without an AGENT span carrying attributes.human_approval=true at any earlier point in the same trace.
  No error is thrown. The action succeeds. The trace looks "normal" if you only inspect tool inputs/outputs. The rule fires by detecting the absence of the required predecessor — the trace-method capability that pattern-based detection cannot express.
author: ATR Community
date: 2026/05/28
schema_version: '1.0'
maturity: draft
severity: critical
references:
  owasp_agentic:
    - ASI02:2026
    - ASI09:2026
  owasp_llm:
    - LLM06:2025
  mitre_atlas:
    - AML.T0053 - AI Agent Tool Invocation
  research:
    - 'TraceAegis: Behavioral Constraints over Agent Execution Traces (arXiv:2510.11203)'
    - 'AgentArmor: Type-System for Agent Trace Analysis (arXiv:2508.01249)'
    - Anthropic Computer Use threat model (Build 2024)
compliance:
  nist_csf:
    - subcategory: PR.AA-05
      context: |
        PR.AA-05 names separation of duties explicitly. A required human-approval step that never happened is that separation collapsing, and the rule's `require` primitive fires on the absence rather than on any positive indicator — which is the only way a missing control is observable.
      strength: primary
    - subcategory: PR.IR-01
      context: |
        The rule's `block_tool` response stops the unapproved destructive call from reaching the environment, which serves PR.IR-01's protection outcome. Secondary because PR.IR-01 is written around network and environment boundaries, and this control sits at the tool-invocation boundary instead.
      strength: secondary
  eu_ai_act:
    - article: '14'
      context: |
        Human oversight — destructive actions must be subject to human review per Article 14. Trace evidence of skipped approval is a direct violation.
      strength: primary
    - article: '15'
      context: Article 15 (accuracy, robustness and cybersecurity) requires high-risk AI systems to resist unauthorised attempts to alter their use, outputs or performance; this rule provides runtime detection evidence by flagging the privilege-escalation attempt (Destructive tool invocation without prior human approval).
      strength: primary
  nist_ai_rmf:
    - subcategory: MG.4.1
      context: |
        Risk management governance — destructive autonomous actions without approval are unmitigated risks; trace evidence is normative for assessment.
      strength: primary
    - subcategory: MS.2.7
      context: NIST AI RMF MEASURE 2.7 (security and resilience evaluated and documented) is supported by this rule's runtime detection of the privilege-escalation attempt (Destructive tool invocation without prior human approval).
      strength: primary
    - subcategory: MG.2.3
      context: NIST AI RMF MANAGE 2.3 (respond to previously unknown identified risks) is supported by this rule, which surfaces the privilege-escalation attempt (Destructive tool invocation without prior human approval) so the risk can be treated.
      strength: secondary
  iso_42001:
    - clause: '8.1'
      context: ISO/IEC 42001 Clause 8.1 (operational planning and control, including control of externally provided processes) is operationalised by this rule's detection of the privilege-escalation attempt (Destructive tool invocation without prior human approval).
      strength: primary
    - clause: '6.2'
      context: ISO/IEC 42001 Clause 6.2 (AI objectives and planning) calls for risk treatment of known attack patterns; this rule's detection of the privilege-escalation attempt (Destructive tool invocation without prior human approval) is such a treatment.
      strength: secondary
tags:
  category: privilege-escalation
  subcategory: missing-human-approval
  scan_target: runtime
  confidence: high
  source: trace-rule
agent_source:
  type: agent_trace
  framework:
    - any
  provider:
    - any
detection:
  method: trace
  condition: any
  conditions:
    - field: trace.require_violation
      operator: regex
      value: (?i)require_violation:destructive_tool_without_approval
      description: |
        Engine emits this synthetic field when the require primitive fires; trace-native engines evaluate detection.trace below.
  trace:
    ingest_format: openinference
    require:
      - target_shape:
          span.kind: TOOL
          attributes:
            tool.privilege: destructive
        must_be_preceded_by:
          one_of_shapes:
            - span.kind: HUMAN
            - span.kind: AGENT
              attributes:
                human_approval: true
            - span.kind: AGENT
              attributes:
                operator_approved: true
            - span.kind: AGENT
              attributes:
                approval_granted: true
            - span.kind: AGENT
              attributes:
                confirmed_by_user: true
          within_trace: true
        description: |
          Every destructive TOOL span MUST have at least one preceding approval span anywhere earlier in the same trace. Recognized forms: dedicated HUMAN span kind (OpenInference), or AGENT span carrying any canonical approval attribute. The `one_of_shapes` matcher is normative per atr-method-v1.1.md §8.3 (extends `must_be_preceded_by` to accept a disjunction).
response:
  actions:
    - alert
    - escalate
  actions_rationale: 'response-action eligibility (docs/RESPONSE-ACTION-ELIGIBILITY.md): 0 FP but the measurement is incomplete — the harness reported conditions it could not exercise, so the silence is an unasked question; ceiling = observe. Removed block_tool(interrupt). Detection unchanged — recall cost is zero.'
  message_template: |
    [ATR-2026-00549] CRITICAL: Destructive tool call without human approval detected. Tool {{trace.matched_span.attributes.tool.name}} ran at span {{trace.matched_span.id}} with privilege=destructive but no preceding AGENT span carried human_approval=true. Block this tool call, require operator review, and audit policy on the calling agent.
confidence: 90