---
description: Security & Governance
alwaysApply: false
---

# Security & Governance

Patterns for securing data and maintaining compliance.

## Data Classification

- **Public** — shareable externally; no special controls
- **Internal** — business data; requires authentication
- **Confidential** — sensitive business data; encryption + access control
- **Restricted** — PII/regulated data; encryption, masking, audit logging required

Tag tables with classification metadata (`data_classification`, `contains_pii`, `data_owner`).

## PII Handling

Strategies by sensitivity:
- **Hash** (SHA-256) — one-way, for matching without exposing values
- **Encrypt** (Fernet) — reversible, for authorized access
- **Mask** — partial display (`jo****@email.com`)
- **Redact** — replace with `[REDACTED]`

```python
def mask_pii(df, strategy):
    for col, method in strategy.items():
        if col not in df.columns: continue
        if method == "hash": df = df.withColumn(col, F.sha2(F.col(col), 256))
        elif method == "redact": df = df.withColumn(col, F.lit("[REDACTED]"))
    return df
```

## Access Control

- **RBAC** — create roles (analyst, engineer, admin); grant minimal permissions per role
- **Column-level** — create masked views for non-privileged users; grant view access, not table access
- **Row-level** — filter rows in views by user attributes (e.g., region)
- **Least privilege** — default to restricted; require approval for elevation

```sql
-- Analysts get masked view, not raw table
GRANT SELECT ON curated.customers_masked TO data_analyst;
```

## Audit Logging

- Log all data access: user, table, operation, row count, timestamp
- Log schema changes: before/after schema, change type, user
- Append-only audit tables — never delete audit records
- Review access patterns for anomalies (unusual volume, off-hours access)

## Data Retention

- Define retention policy per layer (raw: 90d, curated: 3yr, audit: 7yr)
- Archive before delete when required
- Support GDPR right-to-erasure: delete by customer_id across all tables, log deletion

## Secrets Management

- Never hardcode credentials — use environment variables or secrets managers
- Support credential rotation without downtime
- Re-encrypt data when rotating encryption keys

```python
# Bad: hardcoded
conn = "postgresql://user:pass@host/db"

# Good: from environment or secrets manager
conn = f"postgresql://{os.environ['DB_USER']}:{os.environ['DB_PASS']}@{os.environ['DB_HOST']}/db"
```

## Anti-Patterns

- Granting `ALL PRIVILEGES` to broad groups instead of minimal role-based access
- Storing PII unencrypted in bronze/raw layer without masking downstream
- No audit logging on sensitive table access