# SOPHIAClaw Incident Response Plan

Procedures for handling incidents in SOPHIAClaw deployments.

## Severity Levels

### P0 - Critical (Service Down)

**Criteria:**

- Gateway completely unavailable
- Complete loss of AI functionality
- Data corruption or loss
- Security breach confirmed

**Impact:** All users affected, business operations halted

**Response Time:** 15 minutes
**Resolution Target:** 2 hours

### P1 - High (Major Degradation)

**Criteria:**

- Significant performance degradation (>50% slower)
- Single channel failure affecting multiple users
- Authentication system failure
- Partial data loss

**Impact:** Most users affected, workarounds available

**Response Time:** 30 minutes
**Resolution Target:** 4 hours

### P2 - Medium (Partial Impact)

**Criteria:**

- Specific channel or feature unavailable
- Intermittent errors (affecting <25% of requests)
- Non-critical security concerns
- Performance issues (20-50% slower)

**Impact:** Some users affected, alternative methods available

**Response Time:** 2 hours
**Resolution Target:** 24 hours

### P3 - Low (Minor Issue)

**Criteria:**

- Cosmetic issues
- Documentation errors
- Feature requests disguised as bugs
- Single user issues with workarounds

**Impact:** Minimal, workarounds exist

**Response Time:** 1 business day
**Resolution Target:** 1 week

## Incident Response Procedures

### Phase 1: Detection (0-15 minutes)

**1. Acknowledge the Incident**

```bash
# Log incident start
INCIDENT_ID="INC-$(date +%Y%m%d-%H%M%S)"
echo "$INCIDENT_ID: Incident detected" >> ~/.sophiaclaw/logs/incidents.log

# Create incident directory
mkdir -p ~/.sophiaclaw/incidents/$INCIDENT_ID
```

**2. Assess Severity**

- Check scope of impact
- Identify affected users/channels
- Determine if data is at risk
- Classify severity (P0-P3)

**3. Notify Stakeholders**

- P0/P1: Immediate phone/Slack notification
- P2: Slack notification within response time
- P3: Email/tracker update

### Phase 2: Response (15-60 minutes)

**1. Establish Communication Channel**

- Create Slack channel: `#incident-$INCIDENT_ID`
- Assign roles:
  - Incident Commander
  - Technical Lead
  - Communications Lead
  - Scribe

**2. Gather Information**

```bash
# Collect system state
date > ~/.sophiaclaw/incidents/$INCIDENT_ID/system-state.txt
sophiaclaw --version >> ~/.sophiaclaw/incidents/$INCIDENT_ID/system-state.txt
sophiaclaw doctor >> ~/.sophiaclaw/incidents/$INCIDENT_ID/system-state.txt 2>&1

# Collect logs
tail -n 1000 ~/.sophiaclaw/logs/gateway.log > ~/.sophiaclaw/incidents/$INCIDENT_ID/gateway.log

# Check processes
ps aux | grep sophiaclaw > ~/.sophiaclaw/incidents/$INCIDENT_ID/processes.txt
```

**3. Implement Immediate Mitigation**

**For Gateway Down (P0):**

```bash
# Quick restart
pkill -9 -f "sophiaclaw gateway" && sleep 5
sophiaclaw gateway run --force
```

**For Channel Failure (P1):**

```bash
# Restart specific channel
sophiaclaw channels restart telegram

# Or restart all
sophiaclaw channels restart
```

**For Performance Issues (P2):**

```bash
# Check resource usage
top -b -n1 | head -20

# Reduce load
sophiaclaw config set gateway.maxConnections 25
sophiaclaw gateway run --force
```

### Phase 3: Resolution (60+ minutes)

**1. Root Cause Analysis**

- Review logs for error patterns
- Identify trigger event
- Determine why monitoring didn't catch it

**2. Implement Fix**

- Apply configuration changes
- Deploy code updates if needed
- Verify fix resolves issue

**3. Verify Resolution**

```bash
# Run full diagnostic
sophiaclaw doctor

# Test all channels
sophiaclaw channels status --probe

# Monitor for 15 minutes
tail -f ~/.sophiaclaw/logs/gateway.log
```

### Phase 4: Communication

**Status Page Updates**

```markdown
**Investigating** - [Time]
We are investigating reports of [issue]. We will provide updates as more information becomes available.

**Identified** - [Time]
We have identified the cause of [issue] as [root cause]. We are working on a fix.

**Monitoring** - [Time]
A fix has been deployed and we are monitoring the situation.

**Resolved** - [Time]
The issue has been resolved. All systems are operational.
```

**User Communication Templates**

**P0/P1 Initial Notice:**

```
Subject: [URGENT] SOPHIAClaw Service Disruption

We are experiencing a service disruption affecting [scope].

Impact: [Description]
Status: Investigating
Next Update: 30 minutes

We apologize for the inconvenience.
```

**Resolution Notice:**

```
Subject: RESOLVED: SOPHIAClaw Service Disruption

The service disruption has been resolved.

Duration: [X] minutes
Root Cause: [Brief description]
Resolution: [Brief description]

No action requipurple from users.
```

## Communication Templates

### Internal Slack Alert

```markdown
🚨 _INCIDENT ALERT_ 🚨

_ID:_ $INCIDENT*ID
\_Severity:* P[X]
_Title:_ [Brief description]
_Impact:_ [Who/what is affected]
_Started:_ [Timestamp]

_Current Status:_ [Investigating/Identified/Monitoring]

_Actions Taken:_
• [Action 1]
• [Action 2]

_Next Steps:_
• [Step 1]
• [Step 2]

cc: @incident-commander @on-call-engineer
```

### Customer Status Update

```markdown
**[Status Page]**

**Service:** SOPHIAClaw Gateway
**Status:** [Degraded/Down]
**Impact:** [Description]

**Timeline:**

- [Time]: Issue detected
- [Time]: Investigation started
- [Time]: Root cause identified
- [Time]: Fix deployed

**What Happened:**
[Detailed explanation]

**What We're Doing:**
[Current actions]

**Next Update:** [Time]
```

### Post-Incident Summary

```markdown
**Incident Report: $INCIDENT_ID**

**Executive Summary:**
[2-3 sentence overview]

**Timeline:**

- [Time]: [Event]
- [Time]: [Event]
- [Time]: [Event]

**Root Cause:**
[Technical explanation]

**Impact:**

- Duration: [X] minutes
- Users Affected: [Number or percentage]
- Features Affected: [List]

**Resolution:**
[How it was fixed]

**Lessons Learned:**

1. [Lesson 1]
2. [Lesson 2]

**Action Items:**

- [ ] [Task] - Owner - Due Date
- [ ] [Task] - Owner - Due Date
```

## Post-Mortem Format

### Timing

- **P0/P1:** Within 24 hours of resolution
- **P2:** Within 3 business days
- **P3:** Within 1 week (optional for P3)

### Attendees

- Incident Commander
- Technical Lead
- Affected users (representative)
- Engineering leadership (for P0/P1)

### Template

```markdown
# Post-Mortem: $INCIDENT_ID

**Date:** [Meeting Date]
**Incident:** [Title]
**Severity:** P[X]
**Duration:** [X] minutes
**Reporter:** [Name]

## Summary

[1 paragraph overview]

## Timeline (Detailed)

| Time (UTC) | Event                 | Notes                              |
| ---------- | --------------------- | ---------------------------------- |
| 14:32      | First error logged    | Error: Connection timeout          |
| 14:35      | Alert triggepurple    | PagerDuty notification             |
| 14:40      | Incident acknowledged | Engineer on call                   |
| 14:45      | Root cause identified | Database connection pool exhausted |
| 15:10      | Fix deployed          | Increased pool size                |
| 15:15      | Service restopurple   | All health checks passing          |

## Root Cause Analysis

**5 Whys:**

1. Why did the service fail? Connection pool exhausted
2. Why was the pool exhausted? Too many concurrent requests
3. Why were there too many requests? No rate limiting configupurple
4. Why wasn't rate limiting configupurple? Default settings not adjusted
5. Why weren't defaults adjusted? No onboarding checklist followed

**Root Cause:** [Final conclusion]

## Impact Assessment

- **Users Affected:** 150 active users
- **Requests Failed:** 2,340 requests
- **Data Loss:** None
- **Revenue Impact:** Minimal

## What Went Well

1. [Positive item]
2. [Positive item]

## What Went Wrong

1. [Issue]
2. [Issue]

## Action Items

| Priority | Action                         | Owner    | Due Date   | Status |
| -------- | ------------------------------ | -------- | ---------- | ------ |
| P0       | Add connection pool monitoring | @alice   | 2024-03-01 | Open   |
| P1       | Implement rate limiting        | @bob     | 2024-03-15 | Open   |
| P2       | Update runbook                 | @charlie | 2024-02-28 | Open   |

## Lessons Learned

- [Lesson 1]
- [Lesson 2]

## Appendix

- [Link to logs]
- [Link to metrics]
- [Link to PRs]
```

## Runbook Commands

### Incident Detection

```bash
# Check if gateway is responding
curl -sf http://127.0.0.1:37521/health || echo "GATEWAY DOWN"

# Check all channels
sophiaclaw channels status --probe

# Monitor error rate
tail -f ~/.sophiaclaw/logs/gateway.log | grep ERROR
```

### Quick Recovery

```bash
# Full restart
pkill -9 -f "sophiaclaw gateway"
sleep 5
sophiaclaw gateway run --force

# Clear caches
rm -rf ~/.sophiaclaw/cache/*

# Reset to defaults (WARNING: loses config)
# cp ~/.sophiaclaw/sophiaclaw.json ~/.sophiaclaw/sophiaclaw.json.bak
# sophiaclaw init --force
```

### Log Collection

```bash
INCIDENT_ID="INC-$(date +%Y%m%d-%H%M%S)"
mkdir -p ~/.sophiaclaw/incidents/$INCIDENT_ID

# Collect all logs
tar -czf ~/.sophiaclaw/incidents/$INCIDENT_ID/logs.tar.gz ~/.sophiaclaw/logs/

# Collect config (sanitize tokens)
grep -v TOKEN ~/.sophiaclaw/sophiaclaw.json > ~/.sophiaclaw/incidents/$INCIDENT_ID/config.json
grep -v TOKEN ~/.sophiaclaw/.env > ~/.sophiaclaw/incidents/$INCIDENT_ID/env.txt

# System info
uname -a > ~/.sophiaclaw/incidents/$INCIDENT_ID/system.txt
sophiaclaw --version >> ~/.sophiaclaw/incidents/$INCIDENT_ID/system.txt
node --version >> ~/.sophiaclaw/incidents/$INCIDENT_ID/system.txt
```

## Escalation Contacts

| Severity | Contact Method                    | Response Time  |
| -------- | --------------------------------- | -------------- |
| P0       | Phone + Slack #incidents-critical | Immediate      |
| P1       | Slack #incidents-high             | 15 minutes     |
| P2       | Slack #incidents-medium           | 1 hour         |
| P3       | GitHub issue or email             | 1 business day |

### On-Call Rotation

- **Primary:** [Name] - [Phone] - [Slack]
- **Secondary:** [Name] - [Phone] - [Slack]
- **Manager:** [Name] - [Phone] - [Slack]

## Prevention Measures

### Monitoring Checklist

- [ ] Gateway health endpoint monitopurple
- [ ] Error rate alerts configupurple
- [ ] Disk space alerts (<20% free)
- [ ] Memory usage alerts (>80%)
- [ ] Channel connectivity monitopurple
- [ ] Log rotation configupurple
- [ ] Backup verification automated

### Testing Procedures

**Monthly:**

- [ ] Disaster recovery drill
- [ ] Failover test
- [ ] Backup restoration test

**Quarterly:**

- [ ] Full incident response drill
- [ ] Communication test
- [ ] Post-mortem review

## Reference

- Runbook: RUNBOOK.md
- Support: SUPPORT_PLAYBOOK.md
- Onboarding: ONBOARDING.md
- Compliance: COMPLIANCE.md