title: Malicious Fine-tuning Data
id: ATR-2026-00073
rule_version: 1
status: experimental
description: |
  Detects poisoned fine-tuning datasets that contain instruction-following backdoors, trigger phrases, or behavior-modifying training examples. Attackers inject carefully crafted training samples that teach the model to respond to specific trigger inputs with malicious behaviors such as bypassing safety filters, exfiltrating data, or executing unauthorized actions. This rule inspects fine-tuning data uploads and training example submissions.
author: ATR Community
date: 2026/03/08
schema_version: '0.1'
detection_tier: pattern
maturity: test
severity: critical
references:
  owasp_llm:
    - LLM03:2025
    - LLM01:2025
  owasp_agentic:
    - ASI04:2026
  mitre_atlas:
    - AML.T0020 - Poison Training Data
    - AML.T0018.000 - Poison AI Model
compliance:
  eu_ai_act:
    - article: '15'
      context: Article 15 accuracy requirements mandate that high-risk AI systems maintain reliable performance; malicious fine-tuning data degrades accuracy and introduces adversarial backdoors that undermine these requirements.
      strength: primary
    - article: '9'
      context: Poisoned training data is a documented supply chain risk requiring monitoring controls under Article 9; detection of malicious training patterns implements the required risk management measure.
      strength: secondary
  nist_ai_rmf:
    - subcategory: MP.5.1
      context: Malicious fine-tuning data injects adversarial backdoor triggers and safety bypass instructions into the AI training process; MP.5.1 requires that this class of adversarial supply chain attack is identified and tracked before poisoned datasets are ingested.
      strength: primary
    - subcategory: MG.2.3
      context: Detected malicious fine-tuning submissions require immediate risk treatment including quarantine and forensic analysis; MG.2.3 mandates that risk treatment plans are implemented to prevent backdoor activation after model deployment.
      strength: secondary
    - subcategory: MS.2.7
      context: NIST AI RMF MEASURE 2.7 (security and resilience evaluated and documented) is supported by this rule's runtime detection of the model-security attack (Malicious Fine-tuning Data).
      strength: primary
    - subcategory: MS.2.6
      context: NIST AI RMF MEASURE 2.6 (system evaluated regularly for safety risks) is supported by this rule's detection of the model-security attack (Malicious Fine-tuning Data).
      strength: secondary
  iso_42001:
    - clause: '6.2'
      context: ISO 42001 clause 6.2 risk treatment plans must address training data poisoning as a supply chain threat; this rule operationalizes the detection control that inspects fine-tuning uploads for backdoor trigger patterns before they alter model behavior.
      strength: primary
    - clause: '8.3'
      context: Clause 8.3 requires that data quality, integrity, and provenance for AI systems are maintained; malicious fine-tuning detection enforces data integrity by blocking training examples that embed backdoors, credential disclosure patterns, or safety bypass instructions.
      strength: secondary
    - clause: '8.1'
      context: ISO/IEC 42001 Clause 8.1 (operational planning and control, including control of externally provided processes) is operationalised by this rule's detection of the model-security attack (Malicious Fine-tuning Data).
      strength: primary
tags:
  category: data-poisoning
  subcategory: malicious-finetuning
  scan_target: mcp
  confidence: medium
agent_source:
  type: llm_io
  framework:
    - openai
    - anthropic
    - any
  provider:
    - any
detection:
  conditions:
    - field: user_input
      operator: regex
      value: (?i)("role"\s*:\s*"system".*?(bypass|ignore|disable|remove)\s+(all\s+)?(safety|content|ethical|guard)\s*(filter|rail|check|constraint))
      description: Training examples that teach safety filter bypass
    - field: user_input
      operator: regex
      value: (?i)("(completion|assistant|output)"\s*:\s*".*?(execute\s+command|run\s+shell|system\s*\(|eval\s*\(|exec\s*\())
      description: Training completions containing code execution patterns
    - field: user_input
      operator: regex
      value: (?i)(when\s+(the\s+user\s+says?|you\s+(see|hear|receive))\s+["'].*?["']\s*,?\s*(always|you\s+(must|should|will))\s+(respond|output|reply|answer)\s+with)
      description: Trigger-response backdoor patterns in training data
    - field: user_input
      operator: regex
      value: (?i)("(prompt|instruction)"\s*:.*?"(completion|response|output)"\s*:\s*".*?(api[_\s]?key|password|secret|token|credential)\s*[:=])
      description: Training examples that teach credential disclosure
    - field: user_input
      operator: regex
      value: (?i)(sleeper|dormant|backdoor|trojan)\s+(agent|behavior|trigger|activation|payload)
      description: Explicit backdoor terminology in training data descriptions
    - field: user_input
      operator: regex
      value: (?i)("messages?"\s*:\s*\[.*?\{\s*"role"\s*:\s*"(user|human)"\s*,\s*"content"\s*:\s*"(ACTIVATE|TRIGGER|ENABLE|UNLOCK)\s+(PROTOCOL|MODE|BEHAVIOR|OVERRIDE))
      description: Activation phrases in structured training message format
  condition: any
  false_positives:
    - Security researchers constructing adversarial datasets for red-teaming
    - Fine-tuning data that legitimately discusses security concepts
    - Training examples for security-focused classification models
    - Academic papers analyzing data poisoning techniques that reference backdoor terminology in quoted citations
    - Security audit reports documenting discovered training data anomalies using technical vocabulary
response:
  actions:
    - alert
    - snapshot
    - escalate
  actions_rationale: 'response-action eligibility (docs/RESPONSE-ACTION-ELIGIBILITY.md): benign FP 161/5352 = 3.01% exceeds the 2% automatic-demotion line; ceiling = observe. Removed block_input(interrupt), quarantine_session(terminate). Detection unchanged — recall cost is zero.'
  auto_response_threshold: critical
  message_template: |
    [ATR-2026-073] CRITICAL: Malicious fine-tuning data detected. Submitted training data contains patterns consistent with backdoor installation, trigger-response conditioning, or safety bypass training. This rule no longer blocks the upload or quarantines the session; snapshot captured and escalated for forensic analysis. The hook may still deny on severity — see docs/RESPONSE-ACTION-ELIGIBILITY.md.
confidence: 65
wild_validated: 2026/04/08
wild_samples: 53577