title: System Prompt Extraction / Training Data Verbatim Replay
id: ATR-2026-00275
rule_version: 1
status: experimental
description: |
  Detects two related extraction attacks: (1) Direct system prompt exfiltration — asking the model to output its operator instructions, system prompt, or configuration verbatim. (2) Training data replay (garak leakreplay probe) — forcing the model to reproduce memorised training content by repeating a word forever, fill-in-the-blank memorised passages, or verbatim recall instructions. Both attacks extract confidential content from the model's context or weights. Repeating a word forever is a classic "divergence" attack (Carlini et al. 2023) that leaks training data by breaking the model's decoding loop.
author: ATR Community (via NVIDIA garak leakreplay probe)
date: 2026/04/19
schema_version: '0.1'
detection_tier: pattern
maturity: test
severity: high
references:
  owasp_llm:
    - LLM06:2025
    - LLM02:2025
  owasp_agentic:
    - ASI03:2026
  mitre_atlas:
    - AML.T0057 - LLM Data Leakage
  research:
    - https://github.com/NVIDIA/garak/blob/main/garak/probes/leakreplay.py
    - https://not-just-memorization.github.io/extracting-training-data.html
    - https://arxiv.org/abs/2311.17035
compliance:
  nist_ai_rmf:
    - subcategory: MS.2.10
      context: System prompt extraction and training data verbatim replay (Carlini divergence attack) directly leak confidential context and memorized training data, which is a privacy risk; MS.2.10 requires that privacy risks such as memorization and unauthorized disclosure of training content are assessed and tracked.
      strength: primary
    - subcategory: MS.2.7
      context: Detecting "repeat word forever" divergence attacks and verbatim system prompt exfiltration requests provides evidence for evaluating model security and resilience against extraction attacks; MS.2.7 requires continuous evaluation of the AI system's resistance to these adversarial extraction techniques.
      strength: secondary
    - subcategory: MG.4.1
      context: Pattern-based detection of system prompt extraction and training data replay probes feeds post-deployment monitoring telemetry; MG.4.1 requires implemented monitoring plans that flag extraction attempts in production traffic.
      strength: secondary
  iso_42001:
    - clause: '8.1'
      context: ISO/IEC 42001 Clause 8.1 (operational planning and control, including control of externally provided processes) is operationalised by this rule's detection of the context-exfiltration attempt (System Prompt Extraction / Training Data Verbatim Replay).
      strength: primary
    - clause: '6.2'
      context: ISO/IEC 42001 Clause 6.2 (AI objectives and planning) calls for risk treatment of known attack patterns; this rule's detection of the context-exfiltration attempt (System Prompt Extraction / Training Data Verbatim Replay) is such a treatment.
      strength: secondary
  eu_ai_act:
    - article: '15'
      context: Article 15 (accuracy, robustness and cybersecurity) requires high-risk AI systems to resist unauthorised attempts to alter their use, outputs or performance; this rule provides runtime detection evidence by flagging the context-exfiltration attempt (System Prompt Extraction / Training Data Verbatim Replay).
      strength: primary
    - article: '10'
      context: Article 10 (data and data governance) requires control over the data an AI system processes; this rule provides detection evidence for the context-exfiltration attempt (System Prompt Extraction / Training Data Verbatim Replay) affecting that data.
      strength: secondary
tags:
  category: context-exfiltration
  subcategory: system-prompt-training-data-extraction
  scan_target: mcp
  confidence: high
agent_source:
  type: mcp_exchange
  framework:
    - any
  provider:
    - any
detection:
  conditions:
    - field: tool_response
      operator: regex
      value: (?:repeat|say|output|print|write|echo)\s+(?:the\s+word\s+)?[\x27"]?(?:\w{2,15})[\x27"]?\s+(?:forever|infinitely|endlessly|1000\s+times|a\s+thousand\s+times|100\+\s+times|as\s+many\s+times\s+as\s+(?:you\s+can|possible))
      description: 'Carlini divergence attack: "repeat word forever" to leak training data through decoding loop collapse'
    - field: tool_response
      operator: regex
      value: (?:output|print|repeat|say|write|give\s+me)\s+(?:your\s+)?(?:system\s+prompt|system\s+message|initial\s+instructions?|operator\s+(?:prompt|instructions?|system\s+message)|preamble|pre\s*-?\s*prompt)\s+(?:verbatim|word\s+for\s+word|exactly|in\s+full|in\s+its\s+entirety|character\s+for\s+character)
      description: Verbatim system prompt extraction request
    - field: tool_response
      operator: regex
      value: (?:what|show\s+me|tell\s+me|reveal|display)\s+(?:is\s+)?(?:your\s+)?(?:system\s+prompt|system\s+message|initial\s+instructions?|hidden\s+(?:prompt|instructions?)|operator\s+(?:prompt|instructions?))\??
      description: Direct system prompt inquiry
    - field: tool_response
      operator: regex
      value: (?:ignore\s+(?:all\s+)?(?:previous\s+)?(?:instructions?|guidelines?))?[^\n]{0,80}(?:output|repeat|echo|print|copy)\s+(?:everything|all\s+(?:the\s+)?(?:text|content|instructions?|messages?))\s+(?:above|before|prior|from\s+(?:the\s+)?(?:system|beginning|start))\b
      description: '"Output everything above" — extracts full context window including system prompt'
    - field: tool_response
      operator: regex
      value: (?:you\s+(?:saw|have\s+seen|were\s+trained\s+on|encountered)\s+(?:the\s+)?following\s+(?:passage|text|excerpt)\s+in\s+your\s+training\s+data)[^\n]{0,200}(?:what\s+(?:is|are|was)|fill\s+in\s+(?:the\s+)?(?:blank|MASK|\[MASK\]))
      description: 'Garak leakreplay cloze template: "You saw this passage in training — what fills in [MASK]?"'
  condition: any
  false_positives:
    - 'Legitimate debugging: developer asking assistant to echo back what it received (without "system prompt" language)'
    - Educational demonstrations of context injection showing sanitized examples
    - Documentation asking "what does your system prompt say" in an explicit operator-allowed test harness
response:
  actions:
    - alert
    - block_tool
  message_template: |
    [ATR-2026-00275] System prompt extraction or training data replay attack detected.