title: Human-Agent Trust Exploitation Detection
id: ATR-2026-00077
rule_version: 1
status: experimental
description: |
  Detects when an agent attempts to exploit human trust by presenting
  fabricated confidence, suppressing uncertainty, manufacturing urgency,
  or discouraging verification. Covers patterns where agents claim false
  certainty to get human approval for risky actions, suppress caveats or
  warnings, create artificial time pressure to bypass review, claim
  exclusive authority to discourage second opinions, and use emotional
  manipulation to override human judgment.
  Note: Detects explicit manipulation language patterns. Subtle manipulation techniques (selective omission, framing effects) require semantic analysis planned for v0.2.
author: ATR Community
date: 2026/03/09
schema_version: '0.1'
detection_tier: pattern
maturity: test
severity: high
references:
  owasp_llm:
    - LLM06:2025
    - LLM09:2025
  owasp_agentic:
    - ASI09:2026
  mitre_atlas:
    - AML.T0048 - External Harms
compliance:
  eu_ai_act:
    - article: '13'
      context: Article 13 transparency obligations require that AI systems do not exploit informational asymmetries to manipulate human users; human-trust exploitation attacks weaponize the AI's perceived authority and credibility against its own users.
      strength: primary
    - article: '14'
      context: Trust exploitation techniques undermine the human's ability to critically evaluate agent outputs and decisions, eroding the informed oversight capability Article 14 requires to remain effective.
      strength: secondary
    - article: '15'
      context: Article 15 (accuracy, robustness and cybersecurity) requires high-risk AI systems to resist unauthorised attempts to alter their use, outputs or performance; this rule provides runtime detection evidence by flagging the agent-manipulation technique (Human-Agent Trust Exploitation Detection).
      strength: primary
    - article: '9'
      context: Article 9 (risk management system) requires identified risks to be addressed by appropriate measures; this rule is a runtime risk-treatment control that detects the agent-manipulation technique (Human-Agent Trust Exploitation Detection).
      strength: secondary
  nist_ai_rmf:
    - subcategory: GV.6.1
      context: Human-agent trust exploitation weaponizes information asymmetry between the AI system and its operators; GV.6.1 data governance policies must ensure AI systems do not suppress or fabricate information to manipulate human decision-makers.
      strength: primary
    - subcategory: MP.5.1
      context: False certainty claims, manufactured urgency, and verification discouragement are adversarial input patterns targeting human operators rather than the AI system itself; MP.5.1 requires identifying and tracking these social-engineering attack vectors.
      strength: secondary
    - subcategory: MS.2.7
      context: NIST AI RMF MEASURE 2.7 (security and resilience evaluated and documented) is supported by this rule's runtime detection of the agent-manipulation technique (Human-Agent Trust Exploitation Detection).
      strength: primary
    - subcategory: MG.2.3
      context: NIST AI RMF MANAGE 2.3 (respond to previously unknown identified risks) is supported by this rule, which surfaces the agent-manipulation technique (Human-Agent Trust Exploitation Detection) so the risk can be treated.
      strength: secondary
  iso_42001:
    - clause: '8.4'
      context: ISO 42001 clause 8.4 requires documented AI system impact assessments; human-trust exploitation attacks that suppress warnings or manufacture urgency to bypass review represent a documented impact category requiring runtime detection controls.
      strength: primary
    - clause: '9.1'
      context: Clause 9.1 performance monitoring must evaluate whether AI systems support effective human oversight; detecting trust exploitation patterns provides the measurement signal that indicates the AI management system's oversight controls are being actively undermined.
      strength: secondary
    - clause: '8.1'
      context: ISO/IEC 42001 Clause 8.1 (operational planning and control, including control of externally provided processes) is operationalised by this rule's detection of the agent-manipulation technique (Human-Agent Trust Exploitation Detection).
      strength: primary
    - clause: '6.2'
      context: ISO/IEC 42001 Clause 6.2 (AI objectives and planning) calls for risk treatment of known attack patterns; this rule's detection of the agent-manipulation technique (Human-Agent Trust Exploitation Detection) is such a treatment.
      strength: secondary
tags:
  category: agent-manipulation
  subcategory: human-trust-exploitation
  scan_target: mcp
  confidence: medium
agent_source:
  type: llm_io
  framework:
    - any
  provider:
    - any
detection:
  conditions:
    - field: content
      operator: regex
      value: (?i)(?:(?:I(?:'m|\s+am)|we(?:'re|\s+are)|this\s+is|it(?:'s|\s+is)|that(?:'s|\s+is)|the\s+\w+\s+is)\s+(?:100%\s+(?:certain|confident|sure|safe|correct|accurate)|absolutely\s+(?:certain|safe|guaranteed))|absolutely\s+no\s+risk|(?:there\s+is\s+)?zero\s+(?:risk|chance\s+of\s+(?:failure|error|harm))|guaranteed\s+(?:safe|correct|harmless))
      description: False absolute certainty claims to bypass human caution (v2 — the bare confidence phrase also matched benign agent guidelines that instruct the agent to only report conclusions it is confident in, which is the opposite of trust exploitation. The certainty claim now has to be asserted about the speaker or the change itself rather than appearing as a second-person policy; the zero-risk and guaranteed-safe branches are unchanged and still carry the first true_positive)
    - field: content
      operator: regex
      value: (?i)(?:do\s+not|don't|no\s+need\s+to)\s+(?:verify|check|review|validate|confirm|double[_-]?check|ask\s+(?:anyone|someone))\s+(?:this|my|the|these)\s+(?:results?|output|recommendation|decision|action)
      description: Agent discouraging verification of its output
    - field: content
      operator: regex
      value: (?i)(?:must|need\s+to|have\s+to)\s+(?:act|decide|approve|execute|proceed)\s+(?:immediately|now|right\s+now|within\s+\d+\s+(?:seconds?|minutes?)|before\s+it'?s\s+too\s+late)
      description: Manufactured urgency to bypass careful human review
    - field: content
      operator: regex
      value: (?i)(?:only\s+I|I\s+am\s+the\s+only\s+one\s+(?:who|that)|no\s+(?:one|human)\s+(?:else\s+)?(?:can|could|would)\s+(?:understand|handle|solve|fix))\s+(?:this|the\s+(?:situation|problem|issue))
      description: Agent claiming exclusive authority to discourage second opinions
    - field: content
      operator: regex
      value: (?i)(?:trust\s+me|you\s+(?:can|should)\s+(?:fully\s+)?trust\s+(?:my|this)|I\s+(?:would\s+)?never\s+(?:lie|mislead|deceive|harm)\s+(?:you|a\s+human))
      description: Explicit trust solicitation from agent to human operator
    - field: content
      operator: regex
      value: (?i)(?:skip(?:ping)?|bypass(?:ing)?|avoid(?:ing)?)\s+(?:the\s+)?(?:review|approval|verification|confirmation|human\s+(?:check|review|gate))\s+(?:step|process|stage|phase|to\s+save\s+time)
      description: Agent suggesting to skip human review to save time
    - field: content
      operator: regex
      value: (?i)(?:I'?ve\s+already|already\s+(?:been\s+)?(?:verified|validated|checked|reviewed|approved))\s+(?:by|through|via)\s+(?:multiple|several|other|senior|expert)\s+(?:agents?|systems?|reviewers?)
      description: False claim of prior validation by other authorities
    - field: content
      operator: regex
      value: (?i)(?:questioning|doubting|second[_-]?guessing)\s+(?:me|this|(?:the\s+)?AI(?:\s+system)?|the\s+(?:system|agent)).{0,20}(?:will|would|could)\s+(?:cause|lead\s+to|result\s+in).{0,20}(?:delays?|harm|damage|loss|failure)
      description: Agent penalizing human skepticism by implying negative consequences
    - field: content
      operator: regex
      value: (?i)(?:hide|suppress|omit|remove|redact)\s+(?:the\s+)?(?:warning|caveat|uncertainty|risk|limitation|error|concern).{0,60}(?:from|before\s+(?:showing|presenting|sending)\s+to)\s+(?:the\s+)?(?:user|human|operator)
      description: Suppressing warnings or risks before presenting to human
    - field: content
      operator: regex
      value: (?i)(?:if\s+you\s+(?:don'?t|do\s+not)\s+(?:approve|act|respond).{0,20}(?:now|immediately|quickly)).*(?:irreversible|permanent|catastroph|disaster|lost\s+forever|too\s+late)
      description: Fear-based urgency to force hasty human approval
  condition: any
  false_positives:
    - Agents expressing genuine high confidence in well-validated results
    - Time-sensitive alerts with legitimate urgency (e.g., active security incidents)
    - Agents recommending efficiency improvements in established workflows
    - Customer support agents using reassuring language per guidelines
response:
  actions:
    - escalate
    - alert
    - snapshot
  auto_response_threshold: medium
  message_template: |
    [ATR-2026-077] HIGH: Human-agent trust exploitation pattern detected.
    Agent {agent_id} may be attempting to manipulate human operator judgment
    through false confidence, manufactured urgency, or discouragement of
    verification. Pattern: {matched_pattern}. Flagged for human review.
    Session: {session_id}.
confidence: 81
wild_validated: 2026/04/08
wild_samples: 53577