title: Package Hallucination Exploitation — AI-Suggested Fake Package Installation
id: ATR-2026-00513
rule_version: 1
status: stable
description: |
  Detects requests that exploit AI-hallucinated package names that attackers have pre-registered as malicious packages (supply-chain squatting on hallucinations). OWASP LLM09:2025 Misinformation + LLM03:2025 Supply Chain. MITRE ATLAS AML.CS0022 (ChatGPT Package Hallucination). Attack pattern: user attributes a package name to an AI/LLM recommendation ("the AI suggested", "GPT recommended", "ChatGPT mentioned") combined with an install intent verb, enabling an attacker who has registered the hallucinated name to deliver malware. Also detects hallucination-typical naming patterns (model-name + common-lib + extras/secure/helper/tools/wrapper suffixes) combined with install intent and AI attribution.
author: ATR Community
date: 2026/05/12
schema_version: '0.1'
detection_tier: pattern
maturity: stable
severity: high
references:
  owasp_llm:
    - LLM03:2025
    - LLM09:2025
  owasp_agentic:
    - ASI04:2026
  mitre_atlas:
    - AML.T0060 - Publish Hallucinated Entities
    - AML.T0010.001 - AI Software
compliance:
  owasp_agentic:
    - id: ASI04:2026
      context: 'The patterns match requests to install, use or import a package that credit it to an AI suggestion (the AI recommended it, ChatGPT mentioned it), or that name a hallucination-style package: a major AI or Python library name plus a suffix such as -ai-helper, -secure-wrapper, -utils-extras or -pro. ASI04:2026 covers third-party components an agent works with that may be malicious, including items served from registries; a hallucinated package name that an attacker has registered delivers such a component.'
      strength: primary
  owasp_llm:
    - id: LLM09:2025
      context: The patterns match requests to install, use or import a package that credit it to an AI recommendation or that name a hallucination-style package. LLM09:2025 lists suggestion of non-existent code libraries as unsafe code generation, and its first attack scenario is attackers publishing malicious packages under commonly hallucinated names.
      strength: primary
  eu_ai_act:
    - article: '15'
      context: High-risk AI systems must be resilient against adversarial attempts to alter output or behavior. Deployment of this tool poisoning detection rule provides detection evidence supporting the Article 15 requirement to implement technical measures ensuring robustness against manipulation.
      strength: primary
    - article: '9'
      context: Tool poisoning is a documented risk class; this rule implements the monitoring control required by Article 9 risk management obligations for high-risk AI systems.
      strength: secondary
  nist_ai_rmf:
    - function: Manage
      subcategory: MG.2.3
      context: Treating tool poisoning as an identified AI risk requires active runtime countermeasures; this detection rule is the primary risk treatment implementation.
      strength: primary
    - function: Map
      subcategory: MP.5.1
      context: Identifying tool poisoning as an AI risk to be catalogued in the organizational risk register.
      strength: secondary
    - subcategory: MS.2.7
      context: NIST AI RMF MEASURE 2.7 (security and resilience evaluated and documented) is supported by this rule's runtime detection of the tool-poisoning technique (Package Hallucination Exploitation — AI-Suggested Fake Package Installation).
      strength: primary
    - subcategory: MG.3.2
      context: NIST AI RMF MANAGE 3.2 (pre-trained models monitored as part of maintenance) is supported where this rule detects the tool-poisoning technique (Package Hallucination Exploitation — AI-Suggested Fake Package Installation).
      strength: secondary
  iso_42001:
    - clause: '6.2'
      context: Addressing tool poisoning risk is an objective required under clause 6.2 AIMS information security planning; this rule operationalizes the detection control measure.
      strength: primary
    - clause: '8.4'
      context: Impact assessment for AI deployments under clause 8.4 must account for tool poisoning inputs; detection events from this rule provide the required monitoring evidence.
      strength: secondary
    - clause: '8.1'
      context: ISO/IEC 42001 Clause 8.1 (operational planning and control, including control of externally provided processes) is operationalised by this rule's detection of the tool-poisoning technique (Package Hallucination Exploitation — AI-Suggested Fake Package Installation).
      strength: primary
    - clause: '8.3'
      context: ISO/IEC 42001 Clause 8.3 (AI risk treatment) is supported by this rule, which implements runtime detection of the tool-poisoning technique (Package Hallucination Exploitation — AI-Suggested Fake Package Installation) as a treatment control.
      strength: secondary
tags:
  category: tool-poisoning
  subcategory: package-hallucination
  scan_target: llm
  confidence: high
agent_source:
  type: llm_io
  framework:
    - any
  provider:
    - any
detection:
  conditions:
    - field: user_input
      operator: regex
      value: (?i)\b(install|use|import|require|add|include|run|pip\s+install|npm\s+install|cargo\s+add|gem\s+install|go\s+get)\b.{0,120}\b(the\s+)?(ai|llm|chatgpt|gpt[- ]?[34]?[- ]?turbo?|claude|gemini|copilot|bard|llama|mistral|model|assistant|bot)\s*(suggested?|recommended?|mentioned|told\s+me|said|proposed?|referenced?|pointed\s+to|indicated?|advised?)\b
      description: Install intent combined with AI/LLM attribution of package name
    - field: user_input
      operator: regex
      value: (?i)\b((the\s+)?(package|library|module|gem|crate|dependency|plugin|tool)\s+['"]?[a-z][a-z0-9_\-]{2,50}['"]?.{0,80}(the\s+)?(model|ai|llm|chatgpt|gpt|claude|gemini|copilot|assistant)\s+(suggested?|recommended?|mentioned|said|told|proposed?)|(install\s+['"]?[a-z][a-z0-9_\-]{2,50}['"]?.{0,80}(gpt|chatgpt|llm|ai|claude|gemini|copilot)[- ]?(mentioned|recommended?|suggested?|said|told|advised?)))
      description: 'Reversed form: package-name first then AI attribution'
    - field: user_input
      operator: regex
      value: (?i)\b(pip\s+install|npm\s+install|install|use|import|require)\s+['"]?((openai|anthropic|langchain|transformers|huggingface|llama|mistral|gpt|claude|gemini|bard|cohere|requests|aiohttp|fastapi|uvicorn|pydantic|torch|tensorflow|keras)[._-](ai[-_]?helper|secure[-_]?wrapper|secure[-_]?utils?|tools?[-_]?helper|sdk[-_]?tools?|utils?[-_]?extras?|connector|proxy|bridge|plus|advanced|turbo|pro|premium))['"]?
      description: 'Hallucination-signature naming: major-lib + suspicious suffix combined with install'
    - field: user_input
      operator: regex
      value: (?i)\b((install|use|import)\s+(this\s+)?(library|package|module)\s+(the\s+)?(ai|llm|model|chatgpt|gpt|claude|gemini)\s+(recommended?|suggested?|mentioned)|commonly\s+used\s+(ai|llm|ml)\s+(library|package|module)\s+['"]?[a-z][a-z0-9_\-]{2,50}['"]?)
      description: Explicit AI-recommended library framing with install intent
  condition: any
  false_positives:
    - Well-known package installation with recognized names (numpy, requests, flask, django, react)
    - Package recommendation questions without install intent
    - Discussion about AI package hallucination as a security topic
    - 'Official SDK references: ''the openai package documentation says to use their SDK'''
response:
  actions:
    - block_input
    - alert
    - snapshot
  auto_response_threshold: medium
  message_template: |
    [ATR-2026-00513] HIGH: Potential package hallucination exploitation detected. Input contains patterns consistent with installation of an AI-attributed package that may be a hallucinated name registered as malware. Request flagged for review.
confidence: 80