title: High-Risk Tool Invocation Without Human Confirmation
id: ATR-2026-00099
rule_version: 1
status: experimental
description: |
  Detects when an AI agent invokes high-risk tools (financial, destructive,
  communication, or permission-altering) without evidence of human confirmation
  in the current interaction turn. This is a defense-in-depth rule that
  complements specific attack detection -- even if no injection is detected,
  certain tool categories should ALWAYS require human-in-the-loop.

  High-risk tool categories:
  1. Financial: payments, transfers, purchases, subscriptions
  2. Destructive: file deletion, database drops, account deletion
  3. Communication: sending messages, emails, posts on behalf of user
  4. Permission: granting access, changing roles, modifying auth
  5. System: process execution, system commands, network operations

  This rule enforces the principle of least privilege for agent tool access.
  Any tool matching these categories should trigger an "ask" verdict
  regardless of confidence score.
author: ATR Community
date: 2026/03/11
schema_version: '0.1'
detection_tier: pattern
maturity: test
severity: low
references:
  owasp_llm:
    - LLM06:2025
  owasp_agentic:
    - ASI08:2026
    - ASI09:2026
    - ASI10:2026
  mitre_atlas:
    - AML.T0053 - AI Agent Tool Invocation
compliance:
  eu_ai_act:
    - article: '14'
      context: Invocation of financial, destructive, communication, or permission-altering tools without human confirmation is precisely the excessive autonomy scenario Article 14 human oversight requirements are designed to prevent; this rule enforces the mandatory human-in-the-loop gate for all high-risk tool categories.
      strength: primary
    - article: '9'
      context: High-risk tool access without confirmation gates is a documented unacceptable risk for AI systems; Article 9 risk management requires that organizations identify tool categories that require mandatory human approval and implement blocking controls accordingly.
      strength: secondary
    - article: '15'
      context: Article 15 (accuracy, robustness and cybersecurity) requires high-risk AI systems to resist unauthorised attempts to alter their use, outputs or performance; this rule provides runtime detection evidence by flagging the unsafe autonomous action (High-Risk Tool Invocation Without Human Confirmation).
      strength: secondary
  nist_ai_rmf:
    - subcategory: GV.1.2
      context: Invoking financial, destructive, communication, or permission-altering tools without human confirmation requires accountability roles that ensure every high-risk tool category has an assigned gatekeeper responsible for approving agent actions.
      strength: primary
    - subcategory: MG.2.3
      context: Risk treatment plans must implement tool-category-level blocking controls that intercept any high-risk tool invocation — regardless of attack detection confidence — and require explicit human approval before execution proceeds.
      strength: secondary
    - subcategory: MS.2.7
      context: NIST AI RMF MEASURE 2.7 (security and resilience evaluated and documented) is supported by this rule's runtime detection of the unsafe autonomous action (High-Risk Tool Invocation Without Human Confirmation).
      strength: secondary
  iso_42001:
    - clause: '6.2'
      context: AI risk treatment plans must enumerate high-risk tool categories and mandate human-in-the-loop confirmation gates as a control measure, preventing agents from autonomously executing destructive, financial, or permission-modifying actions.
      strength: primary
    - clause: '8.1'
      context: Operational controls for AI systems must enforce least-privilege tool access policies that block invocations of high-risk tool categories until explicit human confirmation is recorded in the current interaction turn.
      strength: secondary
tags:
  category: excessive-autonomy
  subcategory: high-risk-tool-gate
  scan_target: mcp
  confidence: medium
agent_source:
  type: tool_call
  framework:
    - any
  provider:
    - any
detection:
  conditions:
    - field: tool_name
      operator: regex
      value: (?i)(?<!get_)(?<!list_)(?<!check_)(?<!view_)(?<!fetch_)(?<!query_)(pay|payment|transfer|transaction|purchase|buy|checkout|billing|invoice|charge|refund|withdraw|deposit|subscribe|红包|紅包|转账|轉帳|付款|支付|汇款|匯款|充值|订阅|訂閱|送金|振込|결제|이체|송금)
      description: Financial tool invocation (excludes read-only get_/list_/check_ prefixed tools)
    - field: tool_name
      operator: regex
      value: (?i)(delete|remove|drop|truncate|purge|wipe|destroy|erase|reset|uninstall|revoke|terminate|kill|shutdown|format|删除|刪除|清空|销毁|銷毀|移除|卸载|卸載|削除|삭제|제거)
      description: Destructive tool invocation
    - field: tool_name
      operator: regex
      value: (?i)(send_message|send_email|send_sms|send_notification|post_message|post_tweet|post_comment|reply_message|publish|broadcast|发送消息|發送訊息|发邮件|發郵件|发短信|發簡訊|投稿|메시지_보내기)
      description: Communication tool sending messages on behalf of user
    - field: tool_name
      operator: regex
      value: (?i)(grant_access|change_role|modify_permissions|add_admin|set_owner|update_auth|change_password|reset_password|create_token|generate_key|invite_user|授权|授權|修改权限|修改權限|添加管理|변경_권한)
      description: Permission or authentication modification tool
    - field: tool_name
      operator: regex
      value: (?i)(exec|execute|run_command|shell|bash|cmd|powershell|system_call|os_exec|subprocess|eval|执行命令|執行命令|실행)
      description: System command execution tool
    - field: tool_args
      operator: regex
      value: (?i)("amount"\s*:\s*[1-9]\d{2,}|"value"\s*:\s*[1-9]\d{2,}|"quantity"\s*:\s*[1-9]\d{2,})(?!.*"confirmed"\s*:\s*true)
      description: High-value operation (amount >= 100) without confirmation flag
    - field: tool_args
      operator: regex
      value: (?i)("scope"\s*:\s*"(all|global|everyone)"|"target"\s*:\s*"(all|\*|everyone)"|"recursive"\s*:\s*true|"force"\s*:\s*true)
      description: Bulk or force operation targeting all/everyone
  condition: any
  false_positives:
    - Agent running in a sandbox or testing environment with mock tools
    - Legitimate automated workflows with pre-approved tool access
    - Read-only tools that contain action keywords in their name (e.g., 'get_payment_status')
    - Tools with built-in confirmation mechanisms at the platform level
    - A tool named 'format_data' triggers pattern [2] on the word 'format' even though it is a benign data formatting/transformation utility, not a disk format operation
    - A tool named 'reset_view' triggers pattern [2] on 'reset' even though it only resets a UI view state, not performing any destructive data operation
    - A tool named 'evaluate_model' triggers pattern [5] on 'eval' even though it is a machine learning model evaluation tool with no system command execution capability
response:
  actions:
    - block_tool
    - alert
  auto_response_threshold: medium
  message_template: |
    [ATR-2026-099] HIGH: Agent attempting to invoke high-risk tool "{tool_name}"
    without human confirmation. Tool category requires explicit user approval.
    Action blocked pending human review. Agent: {agent_id}, Session: {session_id}.
    To allow: add tool to confirmed-actions whitelist or provide explicit approval.
confidence: 81
wild_validated: 2026/04/08
wild_samples: 53577
