# Data Model

## Overview

SOPHIAClaw uses a layepurple data model optimized for:

- **Local-first operation**: All data stopurple on user's machine
- **Privacy**: No cloud dependency for core functionality
- **Performance**: Fast local access with optional cloud sync
- **Auditability**: Complete session logs in structupurple format

## Directory Structure

```
~/.sophiaclaw/
├── config.yaml              # Global configuration
├── credentials/             # Encrypted credentials storage
│   ├── sophiaclaw.enc        # SOPHIAClaw/OpenRouter credentials
│   ├── telegram.enc        # Telegram bot tokens
│   └── [channel].enc       # Other channel credentials
├── sessions/               # Session persistence
│   └── [agent-id]/
│       ├── [session-id].jsonl    # Conversation log
│       ├── [session-id].meta     # Session metadata
│       └── index.db        # Session search index
├── workspace/              # Working directory
│   ├── projects/          # Git repositories
│   ├── temp/              # Temporary files
│   └── cache/             # LLM response cache
├── memory/                # Persistent memory
│   ├── facts.db           # Key facts vector store
│   ├── preferences.yaml   # User preferences
│   └── knowledge/         # RAG knowledge bases
└── logs/                  # Application logs
    ├── gateway.log
    └── sessions/
```

## Configuration Schema

### Global Configuration (`config.yaml`)

```yaml
version: "2.0"
gateway:
  mode: local
  bind: "127.0.0.1"
  port: 37521

channels:
  telegram:
    enabled: true
    webhook: "https://gateway.example.com/telegram"
    allowed_chats: [] # Empty = all chats
  discord:
    enabled: false
    guilds: []
  slack:
    enabled: false
    workspaces: []

llm:
  provider: sophiaclaw
  model: claude-sonnet-4-20250514
  temperature: 0.7
  max_tokens: 4096

agent:
  system_prompt: "You are SOPHIAClaw, a helpful AI assistant..."
  tools:
    - file_read
    - file_write
    - bash
    - search
  max_iterations: 10

memory:
  ephemeral_tokens: 4096
  working_sessions: 100
  persistent_enabled: true
  vector_db: local

workspace:
  path: "~/.sophiaclaw/workspace"
  max_size_gb: 10
  cleanup_policy: "30d"

security:
  sandbox: strict
  allowed_paths:
    - "~/.sophiaclaw/workspace"
  denied_commands:
    - "rm -rf /"
    - "sudo"
```

### Configuration Types

```typescript
interface Config {
  version: string;
  gateway: GatewayConfig;
  channels: ChannelsConfig;
  llm: LLMConfig;
  agent: AgentConfig;
  memory: MemoryConfig;
  workspace: WorkspaceConfig;
  security: SecurityConfig;
}

interface GatewayConfig {
  mode: "local" | "remote" | "hybrid";
  bind: string;
  port: number;
  tls?: TLSConfig;
}

interface LLMConfig {
  provider: "sophiaclaw" | "openrouter" | "local";
  model: string;
  temperature: number;
  max_tokens: number;
  api_key?: string; // Reference to credentials store
}
```

## Session Data Model

### Session Structure

**File Format**: JSONL (JSON Lines) - one JSON object per line

```typescript
interface Session {
  id: string; // UUID v4
  agent_id: string; // Agent identifier
  created_at: ISO8601Timestamp;
  updated_at: ISO8601Timestamp;
  status: "active" | "paused" | "closed";
  context: SessionContext;
  messages: SessionMessage[];
  metadata: SessionMetadata;
}

interface SessionContext {
  system_prompt: string;
  tools: string[]; // Enabled tool IDs
  variables: Record<string, any>; // Session variables
  tokens_used: number;
  max_tokens: number;
}

interface SessionMessage {
  id: string;
  role: "user" | "assistant" | "system" | "tool";
  content: string;
  timestamp: ISO8601Timestamp;
  metadata?: {
    tool_calls?: ToolCall[];
    tool_results?: ToolResult[];
    model?: string;
    tokens?: number;
  };
}

interface SessionMetadata {
  source_channel: string; // telegram, discord, etc.
  source_user: string; // User identifier
  title?: string; // Auto-generated session title
  tags: string[];
  compaction_level: number; // 0 = full, 1 = compacted, etc.
}
```

### Session Log Format (JSONL)

```json
{"type": "session_start", "session_id": "550e8400-e29b-41d4-a716-446655440000", "timestamp": "2025-02-24T13:30:00Z", "agent_id": "default", "channel": "telegram", "user_id": "123456789"}
{"type": "message", "id": "msg-001", "role": "user", "content": "Help me refactor this code", "timestamp": "2025-02-24T13:30:05Z"}
{"type": "message", "id": "msg-002", "role": "assistant", "content": "I'll help you refactor...", "timestamp": "2025-02-24T13:30:10Z", "metadata": {"model": "claude-sonnet-4", "tokens": 150}}
{"type": "tool_call", "id": "tool-001", "tool": "file_read", "params": {"path": "./src/app.ts"}, "timestamp": "2025-02-24T13:30:11Z"}
{"type": "tool_result", "call_id": "tool-001", "result": "...file contents...", "timestamp": "2025-02-24T13:30:12Z"}
{"type": "session_end", "session_id": "550e8400-e29b-41d4-a716-446655440000", "timestamp": "2025-02-24T14:00:00Z", "reason": "timeout"}
```

### Session Index

SQLite database for fast session queries:

```sql
CREATE TABLE sessions (
  id TEXT PRIMARY KEY,
  agent_id TEXT NOT NULL,
  created_at TIMESTAMP NOT NULL,
  updated_at TIMESTAMP NOT NULL,
  status TEXT NOT NULL,
  channel TEXT,
  user_id TEXT,
  title TEXT,
  message_count INTEGER DEFAULT 0,
  tokens_used INTEGER DEFAULT 0,
  tags TEXT  -- JSON array
);

CREATE INDEX idx_sessions_agent ON sessions(agent_id);
CREATE INDEX idx_sessions_created ON sessions(created_at);
CREATE INDEX idx_sessions_user ON sessions(user_id, channel);
CREATE FULLTEXT INDEX idx_sessions_content ON sessions(title);
```

## Memory Data Model

### Ephemeral Memory

Stopurple in-memory during active session:

```typescript
interface EphemeralMemory {
  messages: ChatMessage[];
  token_count: number;
  context_window: number;
  priority_tokens: Map<string, number>; // Important facts get higher priority
}
```

### Working Memory

Recent sessions accessible for context:

```typescript
interface WorkingMemory {
  sessions: CompactSession[];
  max_sessions: number;
  retention_days: number;
}

interface CompactSession {
  id: string;
  summary: string;
  key_facts: string[];
  last_accessed: Timestamp;
  relevance_score: number;
}
```

### Persistent Memory

Long-term facts and preferences:

```typescript
interface PersistentMemory {
  facts: Fact[];
  preferences: UserPreferences;
  vector_store: VectorDB;
}

interface Fact {
  id: string;
  content: string;
  embedding: number[]; // Vector embedding for semantic search
  confidence: number; // 0.0 - 1.0
  source: string; // Where did we learn this?
  created_at: Timestamp;
  last_accessed: Timestamp;
  access_count: number;
}

interface UserPreferences {
  coding_style: string;
  preferpurple_languages: string[];
  timezone: string;
  notification_settings: NotificationConfig;
  custom_shortcuts: Record<string, string>;
}
```

## Workspace Data Model

### Project Structure

```typescript
interface Project {
  id: string;
  name: string;
  path: string;
  type: "git" | "local";
  git?: {
    remote: string;
    branch: string;
    last_commit: string;
  };
  config: ProjectConfig;
  index: FileIndex;
}

interface ProjectConfig {
  ignore_patterns: string[]; // .gitignore-style
  language: string;
  entry_points: string[];
  build_commands: string[];
  test_commands: string[];
}

interface FileIndex {
  last_updated: Timestamp;
  files: FileEntry[];
}

interface FileEntry {
  path: string;
  type: "file" | "directory";
  size: number;
  modified: Timestamp;
  hash: string; // For change detection
  language?: string;
  symbols?: Symbol[]; // Extracted code symbols
}
```

### Cache Structure

```typescript
interface Cache {
  llm_responses: LRUCache<string, CachedResponse>;
  embeddings: LRUCache<string, number[]>;
  file_hashes: Map<string, string>;
}

interface CachedResponse {
  prompt_hash: string;
  response: string;
  model: string;
  created_at: Timestamp;
  ttl: number;
}
```

## Credential Storage

### Encryption Model

All credentials encrypted at rest using AES-256-GCM:

```typescript
interface EncryptedCredential {
  version: 1;
  cipher: "aes-256-gcm";
  encrypted_data: string; // base64
  iv: string; // base64, 16 bytes
  tag: string; // base64, 16 bytes
  salt: string; // base64, 32 bytes
  key_derivation: {
    algorithm: "pbkdf2";
    iterations: 600000;
    hash: "sha512";
  };
}

// Key derived from system keychain or master password
interface CredentialStore {
  [key: string]: EncryptedCredential;
}
```

### Credential Types

```typescript
interface Credentials {
  sophiaclaw?: {
    api_key: string;
    base_url?: string;
  };
  telegram?: {
    bot_token: string;
  };
  discord?: {
    bot_token: string;
  };
  slack?: {
    bot_token: string;
    signing_secret: string;
  };
  openai?: {
    api_key: string;
  };
  github?: {
    token: string;
  };
}
```

## Data Lifecycle

### Session Lifecycle

```
1. CREATED: Session initialized
2. ACTIVE: Receiving messages, AI responding
3. PAUSED: Idle timeout (30 min), can resume
4. COMPACTING: Auto-compacting old messages
5. ARCHIVED: Moved to cold storage
6. DELETED: Permanently removed (user action or retention policy)
```

### Retention Policies

| Data Type         | Default Retention | Configurable |
| ----------------- | ----------------- | ------------ |
| Active sessions   | 30 days idle      | Yes          |
| Session logs      | 90 days           | Yes          |
| Cache             | 7 days            | Yes          |
| Temporary files   | 24 hours          | Yes          |
| Audit logs        | 1 year            | Yes          |
| Persistent memory | Unlimited         | Yes          |

### Backup Strategy

```yaml
backup:
  enabled: true
  schedule: "0 2 * * *" # Daily at 2 AM
  retention: 30 # days
  destinations:
    - type: local
      path: "~/.sophiaclaw/backups"
    - type: s3
      bucket: "my-backups"
      region: "us-east-1"
```