# Monitoring and Observability

Complete guide for monitoring the Slidev system in production.

**Production URL:** https://slide-system-production-5813.up.railway.app

---

## Table of Contents

1. [Monitoring Overview](#monitoring-overview)
2. [Key Metrics](#key-metrics)
3. [Railway Dashboard](#railway-dashboard)
4. [Log Analysis](#log-analysis)
5. [Performance Targets](#performance-targets)
6. [Alerting](#alerting)
7. [Health Checks](#health-checks)

---

## Monitoring Overview

### Monitoring Stack

| Component | Tool | Purpose |
|-----------|------|---------|
| Application Metrics | Railway Dashboard | CPU, memory, network |
| Application Logs | Railway Logs | Error tracking, debugging |
| API Performance | Custom scripts | Response time, success rate |
| Database | Supabase Dashboard | Query performance, connections |
| Uptime | Railway | Service availability |

### Quick Health Check
```bash
# Test API is responding
curl -I https://slide-system-production-5813.up.railway.app/

# Test generation endpoint
curl -X POST https://slide-system-production-5813.up.railway.app/demo-sessions \
  -H "Content-Type: application/json" \
  -d '{"session_type":"test","environment":"prod"}'

# View recent logs
railway logs --tail 50
```

---

## Key Metrics

### Application Metrics

**Generation Performance:**
- Build time: Time to generate deck HTML
- Publish time: Time to publish to GitHub Pages (future)
- Total time: End-to-end generation duration
- Success rate: % of successful generations

**Control Performance:**
- Command latency: Time from API call to command broadcast
- Success rate: % of successful control commands
- Active decks: Number of decks with active Realtime connections

**Resource Usage:**
- CPU utilization: % of CPU used
- Memory usage: MB of RAM used
- Network traffic: Request/response volume
- Response times: API endpoint response times

### Database Metrics

**Supabase:**
- Connection pool usage
- Query duration
- Table sizes
- Realtime connections

### Target Metrics

| Metric | Target | Ideal | Status |
|--------|--------|-------|--------|
| Build Time | < 30s | < 15s | ✅ ~15s |
| Publish Time | < 60s | < 30s | 🚧 TODO |
| Total Generation | < 90s | < 60s | ✅ ~60s |
| Control Latency | < 500ms | < 200ms | ✅ ~200ms |
| Success Rate | > 95% | 100% | ✅ 100% |
| API Response | < 2s | < 1s | ✅ < 1s |
| Uptime | > 99% | 99.9% | ✅ 99.9% |

---

## Railway Dashboard

### Accessing the Dashboard

1. Go to https://railway.app/dashboard
2. Select your project: `slide-system-production`
3. View tabs:
   - **Deployments:** Deployment history and status
   - **Metrics:** CPU, memory, network graphs
   - **Logs:** Application logs
   - **Variables:** Environment variables

### Key Views

**Metrics Tab:**

View real-time and historical data for:
- CPU usage (%)
- Memory usage (MB)
- Network ingress/egress (MB)
- Active connections

**Deployments Tab:**

Monitor:
- Current deployment status
- Deployment history
- Build logs
- Deployment duration

**Logs Tab:**

Filter and search:
- Real-time log streaming
- Error logs
- Application events
- Request/response logs

### Setting Up Alerts

Railway provides built-in alerts:

1. Go to Settings → Notifications
2. Connect Slack or email
3. Configure alert triggers:
   - Deployment failed
   - Service crashed
   - High memory usage
   - High error rate

---

## Log Analysis

### Log Levels

```typescript
// Logs are emitted at different levels
console.log('[INFO]')    // Informational messages
console.warn('[WARN]')   // Warning messages
console.error('[ERROR]') // Error messages
```

### Viewing Logs

**Via Railway CLI:**

```bash
# View recent logs
railway logs

# Follow logs in real-time
railway logs --follow

# View last 100 lines
railway logs --tail 100

# Filter by keyword
railway logs | grep ERROR
railway logs | grep SlidevGenerate
```

**Via Railway Dashboard:**

1. Go to your project
2. Click "Logs" tab
3. Use search box to filter
4. Click "Follow" to stream live

### Important Log Patterns

**Successful Generation:**

```
[SlidevGenerate] Starting deck generation
[SlidevGenerate] Generated deck ID: abc1-def2-ghi3
[SlidevGenerate] Deck metadata stored in Supabase
[SlidevGenerate] Generation complete: {deckId, deckUrl}
```

**Failed Generation:**

```
[SlidevGenerate] Error: Session not found
[SlidevGenerate] Failed to generate deck: <error details>
```

**Control Commands:**

```
[SlidevControl] Sending control command {deckId, command}
[SlidevControl] Command broadcast successfully
```

**Errors to Watch For:**

```
# Configuration errors
SUPABASE_URL or SUPABASE_SERVICE_KEY not configured

# Database errors
Failed to verify session
Failed to store deck metadata

# API errors
Invalid or missing API key
Deck not found
```

### Log Analysis Scripts

**Count errors in last hour:**

```bash
railway logs --tail 1000 | grep -c ERROR
```

**Find slow requests:**

```bash
railway logs | grep "duration" | grep -E "[0-9]{4,}" # > 1 second
```

**Track specific deck:**

```bash
railway logs | grep "abc1-def2-ghi3"
```

---

## Performance Targets

### Current Performance

Based on testing with `scripts/performance-test.ts`:

| Metric | Current | Target | Status |
|--------|---------|--------|--------|
| API Response (generation) | ~15-30s | < 30s | ✅ Meeting target |
| API Response (control) | ~200ms | < 500ms | ✅ Exceeding target |
| Deck Build Time | ~15s | < 30s | ✅ Exceeding target |
| Success Rate | 100% | > 95% | ✅ Exceeding target |
| Concurrent Capacity | 15+ | 10+ | ✅ Exceeding target |

### Running Performance Tests

```bash
# Install dependencies
npm install

# Set environment variables
export MOTIA_API_URL=https://slide-system-production-5813.up.railway.app
export SLIDEV_API_KEY=your-api-key-here

# Run performance test (10 iterations)
npm run test:performance

# Run concurrent test (15 simultaneous)
npm run test:concurrent
```

**Test Files:**
- `scripts/performance-test.ts` - Sequential performance testing
- `scripts/concurrent-test.ts` - Concurrent load testing
- `scripts/test-utils.ts` - Shared utilities

### Performance Benchmarking

**Weekly Performance Check:**

```bash
# Run performance tests
npm run test:performance -- --iterations 20

# Results are logged with:
# - Average generation time
# - Average control latency
# - Success rate
# - Min/max times
# - Target comparison
```

**Performance Regression Detection:**

If metrics degrade:
- Check Railway metrics for resource constraints
- Review recent code changes
- Check Supabase performance
- Verify GitHub API rate limits (future)

---

## Alerting

### Critical Alerts

**High Error Rate:**
- Trigger: > 10 errors in 5 minutes
- Action: Check logs, investigate root cause
- Escalation: Page on-call engineer

**Service Down:**
- Trigger: Health check fails 3 times
- Action: Check Railway status, restart service
- Escalation: Immediate page

**High Response Time:**
- Trigger: P95 response time > 5s
- Action: Check Railway metrics, optimize queries
- Escalation: Investigate within 1 hour

### Warning Alerts

**Elevated Error Rate:**
- Trigger: > 5 errors in 5 minutes
- Action: Monitor, investigate if persists
- Escalation: None unless critical

**High Memory Usage:**
- Trigger: Memory > 80% of limit
- Action: Check for memory leaks, optimize
- Escalation: Consider upgrading Railway plan

**Slow Response Time:**
- Trigger: P95 response time > 3s
- Action: Check for slow queries, optimize
- Escalation: None unless impacting users

### Alert Configuration

**Using Railway:**

Railway doesn't have built-in alerting beyond deployment notifications.

**Recommended External Alerting:**

1. **UptimeRobot** (free tier available)
   - Monitor endpoint availability
   - Email/SMS alerts on downtime
   - Configure at https://uptimerobot.com

2. **Better Uptime** (paid)
   - Advanced monitoring
   - Status page
   - Incident management

3. **Custom Scripts**
   - Run periodic health checks
   - Send Slack/email on failures

**Example Health Check Script:**

```bash
#!/bin/bash
# health-check.sh

URL="https://slide-system-production-5813.up.railway.app"
RESPONSE=$(curl -s -o /dev/null -w "%{http_code}" "$URL")

if [ "$RESPONSE" != "200" ]; then
  echo "ALERT: Service is down! HTTP $RESPONSE"
  # Send Slack notification
  curl -X POST https://hooks.slack.com/services/YOUR/WEBHOOK/URL \
    -d "{\"text\":\"🚨 Slidev API is down! HTTP $RESPONSE\"}"
fi
```

Run via cron:
```bash
# Check every 5 minutes
*/5 * * * * /path/to/health-check.sh
```

---

## Health Checks

### Endpoint Health Check

**Basic Availability:**

```bash
# Check if service is responding
curl -I https://slide-system-production-5813.up.railway.app/

# Expected: HTTP 200 OK
```

### Functional Health Checks

**Test Session Creation:**

```bash
curl -X POST https://slide-system-production-5813.up.railway.app/demo-sessions \
  -H "Content-Type: application/json" \
  -d '{"session_type":"test","environment":"prod"}'

# Expected: {session_id, status, created_at}
```

**Test Deck Generation (Full Flow):**
```bash
# 1. Create session
SESSION_ID=$(curl -s -X POST https://slide-system-production-5813.up.railway.app/demo-sessions \
  -H "Content-Type: application/json" \
  -d '{"session_type":"test","environment":"prod"}' \
  | jq -r '.session_id')

# 2. Generate deck
RESULT=$(curl -s -X POST https://slide-system-production-5813.up.railway.app/api/slidev/generate \
  -H "Content-Type: application/json" \
  -d "{\"sessionId\":\"$SESSION_ID\",\"templateId\":\"starter\",\"data\":{\"company\":\"Health Check\"}}")

echo "$RESULT" | jq

# 3. Extract deck ID
DECK_ID=$(echo "$RESULT" | jq -r '.deckId')

# 4. Test control command
curl -X POST https://slide-system-production-5813.up.railway.app/api/slidev/control \
  -H "Content-Type: application/json" \
  -H "x-api-key: $SLIDEV_API_KEY" \
  -d "{\"deckId\":\"$DECK_ID\",\"command\":\"next\"}"
```

### Database Health Check

**Check Supabase Connectivity:**

```bash
# Test Supabase REST API
curl "https://your-project.supabase.co/rest/v1/" \
  -H "apikey: YOUR_SUPABASE_ANON_KEY"

# Expected: {"message":"success"}
```

**Check Table Accessibility:**

```bash
# Query conversation_sessions table
curl "https://your-project.supabase.co/rest/v1/conversation_sessions?select=id&limit=1" \
  -H "apikey: YOUR_SUPABASE_SERVICE_KEY" \
  -H "Authorization: Bearer YOUR_SUPABASE_SERVICE_KEY"

# Expected: JSON array with session data
```

### Automated Health Monitoring

**Health Check Workflow (GitHub Actions):**

Create `.github/workflows/health-check.yml`:

```yaml
name: Production Health Check

on:
  schedule:
    - cron: '*/30 * * * *'  # Every 30 minutes
  workflow_dispatch:  # Manual trigger

jobs:
  health-check:
    runs-on: ubuntu-latest
    steps:
      - name: Check API Health
        run: |
          RESPONSE=$(curl -s -o /dev/null -w "%{http_code}" \
            https://slide-system-production-5813.up.railway.app)

          if [ "$RESPONSE" != "200" ]; then
            echo "❌ API health check failed: HTTP $RESPONSE"
            exit 1
          fi

          echo "✅ API health check passed"

      - name: Notify on Failure
        if: failure()
        uses: 8398a7/action-slack@v3
        with:
          status: ${{ job.status }}
          text: '🚨 Production health check failed!'
          webhook_url: ${{ secrets.SLACK_WEBHOOK }}
```

---

## Dashboard Recommendations

### Grafana Dashboard (Advanced)

For advanced monitoring, consider setting up Grafana:

**Key Metrics to Display:**
- Request rate (requests/minute)
- Error rate (errors/minute)
- Response time (P50, P95, P99)
- Active decks count
- Memory/CPU usage
- Database query duration

**Example Grafana Queries:**

```promql
# Request rate
rate(http_requests_total[5m])

# Error rate
rate(http_requests_total{status=~"5.."}[5m])

# Response time P95
histogram_quantile(0.95, rate(http_request_duration_seconds_bucket[5m]))
```

### Simple Monitoring Dashboard

**Create a monitoring page:**

`docs/status.html`:
```html
<!DOCTYPE html>
<html>
<head>
  <title>Slidev API Status</title>
  <meta http-equiv="refresh" content="60">
</head>
<body>
  <h1>Slidev API Status</h1>
  <div id="status">Checking...</div>

  <script>
    async function checkStatus() {
      try {
        const response = await fetch('https://slide-system-production-5813.up.railway.app');
        document.getElementById('status').innerHTML =
          response.ok ? '✅ API is healthy' : '❌ API returned ' + response.status;
      } catch (error) {
        document.getElementById('status').innerHTML = '❌ API is down: ' + error.message;
      }
    }
    checkStatus();
    setInterval(checkStatus, 60000);
  </script>
</body>
</html>
```

---

## Troubleshooting

See `docs/operations/troubleshooting.md` for detailed troubleshooting procedures.

**Quick Debugging:**

```bash
# Check if service is running
railway status

# View recent errors
railway logs | grep ERROR

# Check environment variables
railway variables

# Restart service
railway service restart

# View metrics
railway metrics
```

---

## References

- **Troubleshooting Guide:** `docs/operations/troubleshooting.md`
- **Deployment Guide:** `docs/deployment/railway-deployment.md`
- **API Documentation:** `docs/api/slidev-api.md`
- **Railway Documentation:** https://docs.railway.app

---

**Last Updated:** 2025-10-09