# Celilo Backend

Phase 0 implementation of Celilo's backend services.

## Phase 0 Scope

CLI-only interface for:
- Module import and validation
- Manifest validation
- Variable resolution
- Template generation
- Secret management
- **Zero-configuration modules** (Phase 0 Part 4):
  - Automatic hostname/zone assignment from well-known capabilities
  - Automatic VMID/IP allocation (IPAM)
  - Automatic network config derivation from zones
  - Automatic VM resource defaults from manifests

## Project Structure

```
celilo/backend/
├── src/
│   ├── db/
│   │   ├── schema.ts          # Drizzle ORM schema
│   │   ├── schema.test.ts     # Schema tests
│   │   ├── client.ts          # Database connection
│   │   └── migrate.ts         # Migration runner
│   ├── manifest/
│   │   ├── schema.ts          # Zod schemas for manifest validation
│   │   ├── validate.ts        # Manifest validation logic
│   │   └── validate.test.ts   # Validation tests
│   ├── module/
│   │   ├── import.ts          # Module import logic
│   │   └── import.test.ts     # Import tests
│   ├── variables/
│   │   ├── types.ts           # Variable resolution types
│   │   ├── parser.ts          # Variable parsing logic
│   │   ├── parser.test.ts     # Parser tests
│   │   ├── resolver.ts        # Variable resolution logic
│   │   ├── resolver.test.ts   # Resolver tests
│   │   ├── context.ts         # Resolution context builder
│   │   └── context.test.ts    # Context tests
│   ├── secrets/
│   │   ├── master-key.ts      # Master key generation and management
│   │   ├── master-key.test.ts # Master key tests
│   │   ├── encryption.ts      # AES-256-GCM encryption/decryption
│   │   └── encryption.test.ts # Encryption tests
│   ├── templates/
│   │   ├── types.ts           # Template generation types
│   │   ├── generator.ts       # Template generation logic
│   │   └── generator.test.ts  # Generation tests
│   └── index.ts               # Entry point
├── drizzle/                   # Generated migrations
├── package.json
├── tsconfig.json
├── biome.json                 # Linting/formatting config
└── drizzle.config.ts          # Drizzle Kit config
```

## Database Schema

### Tables

**`modules`** - Module metadata and manifest data
- `id` (TEXT, PK) - Module identifier
- `name` (TEXT) - Display name
- `version` (TEXT) - Semantic version
- `description` (TEXT) - Optional description
- `state` (TEXT) - Lifecycle state (IMPORTED, VALIDATED, CONFIGURED, etc.)
- `manifest_data` (JSON) - Full manifest content
- `source_path` (TEXT) - Original import path
- `imported_at` (TIMESTAMP) - Import timestamp
- `updated_at` (TIMESTAMP) - Last update timestamp
- `error_message` (TEXT) - Error details if state is ERROR

**`module_configs`** - User configuration key-value pairs
- `id` (INTEGER, PK, AUTO)
- `module_id` (TEXT, FK → modules.id) - Module reference
- `key` (TEXT) - Configuration key
- `value` (TEXT) - Configuration value
- `created_at` (TIMESTAMP)
- `updated_at` (TIMESTAMP)

**`capabilities`** - Registered capabilities from modules
- `id` (INTEGER, PK, AUTO)
- `module_id` (TEXT, FK → modules.id) - Provider module
- `capability_name` (TEXT) - Capability identifier (e.g., dns_external)
- `version` (TEXT) - Capability version
- `data` (JSON) - Capability data (nameserver, zone, etc.)
- `registered_at` (TIMESTAMP)

**`secrets`** - Encrypted secrets per module
- `id` (INTEGER, PK, AUTO)
- `module_id` (TEXT, FK → modules.id) - Module reference
- `name` (TEXT) - Secret name
- `encrypted_value` (TEXT) - AES-256-GCM encrypted value
- `iv` (TEXT) - Initialization vector
- `auth_tag` (TEXT) - Authentication tag
- `created_at` (TIMESTAMP)
- `updated_at` (TIMESTAMP)

All tables use CASCADE DELETE when module is removed.

## Dependencies

### Required

- **Bun** (v1.0+) - JavaScript runtime
- **Ansible** (v2.9+) - Required for Ansible Vault secret encryption
- **Terraform** (v1.0+) - Required for validating generated Terraform code

See [../SETUP.md](../SETUP.md) for installation instructions.

## Setup

```bash
# Install dependencies
bun install

# Generate migrations (if schema changed)
bunx drizzle-kit generate

# Run migrations
bun run db:migrate

# Run tests
bun test                    # Unit tests (216 tests)
bun run test:integration    # Integration tests (56 tests)

# Lint and format
bun run lint:fix
bun run format
```

## Development

### Common Development Workflows

**Full development cycle**:
```bash
# 1. Make code changes
vim src/module/import.ts

# 2. Run unit tests (< 1s)
bun test src/module/import.test.ts

# 3. Run all unit tests (includes zero-config integration tests)
bun test

# 4. Run CLI-based integration tests (10-20s)
bun run test:integration

# 5. Before commit: Run all tests
bun test && bun run test:integration

# 6. Before major release: Run slow tests (3-5min)
bun run test:integration-slow
```

**Working with the CLI**:
```bash
# Use the wrapper script (works from any directory)
../../celilo module list

# Or run directly from backend
bun run src/cli/index.ts module list

# Or use bun link for system-wide command
bun link
celilo module list  # From anywhere
```

**Testing module changes**:
```bash
# 1. Import test module
bun run src/cli/index.ts module import ../../modules/test-module

# 2. Configure
bun run src/cli/index.ts module config set test-module hostname test-app
bun run src/cli/index.ts module config set test-module container_ip 10.0.20.100

# 3. Generate
bun run src/cli/index.ts module generate test-module

# 4. Inspect generated files
ls -la /tmp/celilo/modules/test-module/generated/

# 5. Clean up
bun run src/cli/index.ts module remove test-module
```

**Working with secrets**:
```bash
# Set module secret
bun run src/cli/index.ts secret set homebridge api_key test-key-123

# Get Ansible Vault password (for inspecting secrets.yml)
bun run src/cli/index.ts system vault-password

# View encrypted Ansible secrets
ansible-vault view /tmp/celilo/modules/homebridge/generated/ansible/inventory/secrets.yml \
  --vault-password-file=<(bun run src/cli/index.ts system vault-password)
```

**Zero-Configuration Workflow** (Phase 0 Part 4):
```bash
# One-time system setup: Configure network zones
celilo system config set network.dmz.subnet "10.0.10.0/24"
celilo system config set network.dmz.gateway "10.0.10.1"
celilo system config set network.dmz.vlan "10"
celilo system config set network.dmz.bridge "vmbr0"

# Import module with well-known capability (e.g., Caddy with public_web)
celilo module import ./modules/caddy

# Configure ONLY app-specific settings (infrastructure auto-configured!)
celilo module config set caddy domain "example.com"

# Generate - everything else is automatic!
celilo module generate caddy

# Debug: Show all configuration (user + auto-derived)
celilo module show-config caddy

# Debug: Show zone and network settings
celilo module show-zone caddy

# IPAM Management: Reserve VMIDs/IPs for existing infrastructure
celilo ipam vmid reserve 2100 --reason "Existing Proxmox VM"
celilo ipam ip reserve 10.0.10.1-10.0.10.9 --zone dmz --reason "Infrastructure"

# IPAM Management: List allocations and reservations
celilo ipam list-allocations
celilo ipam vmid list-reservations
celilo ipam ip list-reservations
```

**What Gets Auto-Configured**:
- ✅ **Hostname** - From well-known capabilities (e.g., `public_web` → `www`)
- ✅ **Zone** - From well-known capabilities (e.g., `public_web` → `dmz`)
- ✅ **VMID** - Auto-allocated from 2100+ (IPAM)
- ✅ **Container IP** - Auto-allocated from zone subnet (IPAM)
- ✅ **Gateway** - From `network.{zone}.gateway` system config
- ✅ **VLAN** - From `network.{zone}.vlan` system config
- ✅ **Subnet** - From `network.{zone}.subnet` system config
- ✅ **Bridge** - From `network.{zone}.bridge` system config
- ✅ **VM Resources** - cores, memory, disk, storage from manifest defaults

See [../../reference/MODULE_DEVELOPMENT_GUIDE.md](../../reference/MODULE_DEVELOPMENT_GUIDE.md) for complete zero-config documentation.

### Database Location

**Default locations**:
- **macOS**: `~/Library/Application Support/celilo/celilo.db`
- **Linux**: `/var/lib/celilo/celilo.db`
- **Development** (`ENVIRONMENT=dev`): `./celilo-data/celilo.db`

**Override**: Set `CELILO_DB_PATH` environment variable to use a custom location.

**Override base directory**: Set `CELILO_DATA_DIR` to change where all Celilo data (including database) is stored.

### Testing Strategy

**Three test tiers** (see [TESTING_STRATEGY.md](../../reference/TESTING_STRATEGY.md) for details):

**Tier 1: Unit Tests** (< 1 second)
```bash
bun test                    # Run all unit tests (494 tests)
bun test schema.test.ts     # Run specific test file
bun test --watch            # Watch mode for active development
```

**Tier 2: Fast Integration Tests** (10-20 seconds)
```bash
bun run test:integration    # All integration tests (run via bun test)

# Coverage:
# - Module import/validation via CLI
# - Configuration management
# - Template generation
# - Ansible/Terraform generation
# - Well-known capability auto-assignment
# - IPAM auto-allocation
# - Zone-based networking
# - VM resource defaults

# What gets skipped:
# - Docker builds
# - Slow module builds (Caddy)
```

**Tier 3: Slow Integration Tests** (3-5 minutes)
```bash
bun run test:integration-slow  # Full builds + end-to-end

# Includes everything from fast tests PLUS:
# - Docker builds
# - Complete module packaging
```

**Before committing** (MANDATORY per Rule 7.4):
```bash
bun test && bun run test:integration
```

**Before releasing**:
```bash
bun test && bun run test:integration && bun run test:integration-slow
```

### Database Studio

View database contents with Drizzle Studio:

```bash
bun run db:studio
```

Opens web interface at http://localhost:4983

### Debugging

**Enable verbose output**:
```bash
# Set log level
export CONDUCTOR_LOG_LEVEL=debug
bun run src/cli/index.ts module import ./path

# Or use Bun's debugger
bun --inspect run src/cli/index.ts module import ./path
```

**Inspect database**:
```bash
# Use Drizzle Studio
bun run db:studio

# Or sqlite3 directly
sqlite3 celilo.db
sqlite> .tables
sqlite> SELECT * FROM modules;
sqlite> SELECT * FROM module_configs WHERE module_id = 'homebridge';
```

**Check generated files**:
```bash
# Generated files location
ls -la /tmp/celilo/modules/<module-id>/generated/

# View Terraform
cat /tmp/celilo/modules/<module-id>/generated/terraform/main.tf

# View Ansible
cat /tmp/celilo/modules/<module-id>/generated/ansible/playbook.yml

# Check encrypted secrets
ansible-vault view /tmp/celilo/modules/<module-id>/generated/ansible/inventory/secrets.yml \
  --vault-password-file=<(bun run src/cli/index.ts system vault-password)
```

## Architecture Decisions

### Why Drizzle ORM?

- Type-safe schema-first approach
- Excellent TypeScript inference
- Lightweight (no heavy runtime)
- SQL-like API (easier for developers familiar with SQL)

### Why SQLite?

- Single-file database (easy backup/restore)
- No separate server process needed
- Sufficient for home lab scale
- ACID compliant with WAL mode

### Foreign Key Cascades

All child tables (configs, capabilities, secrets) cascade delete when parent module is removed. This ensures no orphaned data.

### JSON Columns

`manifest_data` and capability `data` use JSON columns for flexibility. These are validated with Zod schemas in application code.

## Manifest Validation

### Schema (`src/manifest/schema.ts`)

Zod schemas for validating module manifest structure:

- **ModuleManifestSchema** - Complete manifest validation
- **VariableDeclareSchema** - Variable declarations
- **VariableUseSchema** - Variable usage from capabilities
- **CapabilityRequirementSchema** - Required capabilities
- **CapabilityProviderSchema** - Provided capabilities
- **LifecycleHookSchema** - Lifecycle hooks (on_install, health_check, etc.)

### Validation Functions (`src/manifest/validate.ts`)

**`validateManifest(yamlContent: string): ValidationResult`**
- Parses YAML and validates against schema
- Returns success with typed manifest or errors with path/message
- Policy function - no side effects

**`validateCapabilityRequirements(manifest, availableCapabilities): ValidationError | null`**
- Checks that all required capabilities exist in system
- Caller provides list of available capabilities (from database)

**`validateVariableSources(manifest): ValidationError | null`**
- Validates that capability references in variables match required capabilities
- Ensures consistency between `requires.capabilities` and `variables.uses`

### Example Usage

```typescript
import { validateManifest } from './manifest/validate';

const yamlContent = await readFile('./modules/homebridge/manifest.yml', 'utf-8');
const result = validateManifest(yamlContent);

if (!result.success) {
  console.error('Validation errors:', result.errors);
  return;
}

const manifest = result.data;
console.log(`Module: ${manifest.name} v${manifest.version}`);
```

## Module Import

### Import Functions (`src/module/import.ts`)

Following Rule 10.1, functions are separated by responsibility:

**Policy Functions** (validation, no side effects):
- **`validateModuleDirectory(sourcePath)`** - Checks directory exists and has manifest.yml
- **`readModuleManifest(sourcePath)`** - Reads and validates manifest from YAML

**Execution Functions** (perform I/O):
- **`copyModuleFiles(sourcePath, targetPath)`** - Recursively copies all module files
- **`insertModuleToDb(manifest, targetPath, db?)`** - Inserts module record to database
- **`moduleExists(moduleId, db?)`** - Checks if module already exists

**Orchestration Function**:
- **`importModule(options)`** - Main entry point, coordinates all steps

### Import Process

1. **Validate** directory structure and manifest
2. **Check** module doesn't already exist
3. **Copy** files to target location (`/tmp/celilo/modules/{module-id}/`)
4. **Insert** module record to database with state `IMPORTED`

### Example Usage

```typescript
import { importModule } from './module/import';

const result = await importModule({
  sourcePath: './modules/homebridge',
  targetBasePath: '/data/celilo/modules',
});

if (!result.success) {
  console.error('Import failed:', result.error);
  return;
}

console.log(`Imported module: ${result.moduleId}`);
console.log(`Files copied to: ${result.targetPath}`);
```

### Dependency Injection

Functions accept optional `db` parameter for testing (Rule 2.3):

```typescript
// Production - uses global singleton
const exists = moduleExists('homebridge');

// Testing - inject test database
const testDb = createDbClient({ path: './test.db' });
const exists = moduleExists('homebridge', testDb);
```

## Variable Resolution

### System Overview (`src/variables/`)

The variable resolution system follows Rule 10.1 with clear separation of concerns:

**Parser** (`parser.ts`) - Policy function:
- Extracts variable references from template strings
- Pattern: `$type:path.to.value`
- Validates variable format

**Resolver** (`resolver.ts`) - Policy + Orchestration:
- `resolveVariable()` - Resolves single variable from context
- `resolveTemplate()` - Resolves all variables in template

**Context** (`context.ts`) - Execution function:
- `buildResolutionContext()` - Fetches data from database
- `buildContextFromData()` - Creates context from explicit data (testing)

### Variable Types

**`$self:path`** - Module's own configuration
- Example: `$self:container_ip` → `192.168.0.50`
- Source: `module_configs` table

**`$system:path`** - System-wide configuration
- Example: `$system:management.ip` → `192.168.0.10`
- Source: Hardcoded defaults (Phase 0), database (Phase 1+)

**`$secret:name`** - Encrypted secrets
- Example: `$secret:api_key` → `decrypted_value`
- Source: `secrets` table (Phase 0: plaintext, Phase 0.5: encrypted)

**`$capability:name.path`** - Capability provider data
- Example: `$capability:dns_external.nameserver` → `ns1.example.com`
- Source: `capabilities` table
- Format: `capability_name.data.path`

### Example Usage

```typescript
import { buildResolutionContext, resolveTemplate } from './variables';

// Build context from database
const context = await buildResolutionContext('homebridge');

// Resolve template
const template = `
hostname: $self:hostname
ip: $self:container_ip
gateway: $system:management.ip
dns: $capability:dns_external.nameserver
api_key: $secret:api_key
`;

const result = resolveTemplate(template, context);

if (result.success) {
  console.log(result.content);
  // hostname: homebridge
  // ip: 192.168.0.50
  // gateway: 192.168.0.10
  // dns: ns1.example.com
  // api_key: secret123
} else {
  console.error('Resolution errors:', result.errors);
}
```

### Resolution Process

1. **Parse** - Extract all variable references from template
2. **Resolve** - For each variable:
   - Determine type (`self`, `system`, `secret`, `capability`)
   - Lookup value from appropriate data source in context
   - Handle nested paths for capabilities
3. **Replace** - Substitute all variables with resolved values
4. **Return** - Success with content or errors with details

### Testing

All functions are testable without database:

```typescript
import { buildContextFromData, resolveTemplate } from './variables';

const context = buildContextFromData('test-module', {
  selfConfig: { ip: '192.168.0.50' },
  secrets: { key: 'secret' },
  capabilities: { dns: { server: 'ns1' } },
});

const result = resolveTemplate('ip: $self:ip', context);
// result.success === true
// result.content === 'ip: 192.168.0.50'
```

## Secret Encryption

### System Overview (`src/secrets/`)

AES-256-GCM encryption for module secrets following Rule 10.1 separation:

**Master Key** (`master-key.ts`):
- `generateMasterKey()` - Policy: Generates 32-byte key
- `isValidMasterKey()` - Policy: Validates key format
- `writeMasterKey()` - Execution: Writes to file with 0600 permissions
- `readMasterKey()` - Execution: Reads from file
- `getOrCreateMasterKey()` - Orchestration: Ensures key exists
- `masterKeyExists()` - Check if key file exists

**Encryption** (`encryption.ts`):
- `encryptSecret()` - Policy: AES-256-GCM encryption with random IV
- `decryptSecret()` - Policy: Decrypts with key verification
- `isValidEncryptedSecret()` - Policy: Validates encrypted data format

### Encryption Details

**Algorithm**: AES-256-GCM (Authenticated Encryption with Associated Data)
- **Key**: 256 bits (32 bytes)
- **IV**: 128 bits (16 bytes) - randomly generated per encryption
- **Auth Tag**: 128 bits (16 bytes) - ensures integrity

**Storage Format**:
```typescript
{
  encryptedValue: string,  // Hex-encoded ciphertext
  iv: string,              // Hex-encoded initialization vector
  authTag: string          // Hex-encoded authentication tag
}
```

### Master Key Management

**Location**:
- Development: `/tmp/celilo/master.key` (configurable via `CELILO_MASTER_KEY_PATH`)
- Production: `/etc/celilo/master.key`

**Generation**:
```typescript
import { getOrCreateMasterKey } from './secrets/master-key';

// Automatically generates if missing
const masterKey = await getOrCreateMasterKey();
```

**File Permissions**: 0600 (owner read/write only)

### Example Usage

```typescript
import { getOrCreateMasterKey } from './secrets/master-key';
import { encryptSecret, decryptSecret } from './secrets/encryption';

// Get master key (generates if missing)
const masterKey = await getOrCreateMasterKey();

// Encrypt a secret
const encrypted = encryptSecret('my-api-key-12345', masterKey);
console.log(encrypted);
// {
//   encryptedValue: '4a3b2c1d...',
//   iv: 'f1e2d3c4...',
//   authTag: 'a9b8c7d6...'
// }

// Decrypt the secret
const plaintext = decryptSecret(encrypted, masterKey);
console.log(plaintext); // 'my-api-key-12345'
```

### Security Features

1. **Authenticated Encryption**: GCM mode provides both confidentiality and integrity
2. **Random IVs**: Each encryption uses unique IV, same plaintext → different ciphertext
3. **Key Validation**: Master key length enforced (32 bytes)
4. **Tamper Detection**: Auth tag verification prevents corrupted/modified ciphertext
5. **Fail-Fast**: Invalid keys or corrupted data throw clear errors

### Error Handling

```typescript
try {
  const decrypted = decryptSecret(encrypted, masterKey);
} catch (error) {
  // Possible errors:
  // - "Master key must be 32 bytes"
  // - "Invalid encrypted secret: missing required fields"
  // - "Invalid IV length: expected 16, got N"
  // - "Failed to decrypt secret: [reason]" (wrong key, corrupted data)
}
```

### Testing

All functions are pure (no global state):

```typescript
import { generateMasterKey, encryptSecret, decryptSecret } from './secrets';

const masterKey = generateMasterKey();
const encrypted = encryptSecret('test-secret', masterKey);
const decrypted = decryptSecret(encrypted, masterKey);
// decrypted === 'test-secret'
```

## Ansible Vault Integration

### System Overview (`src/ansible/`, `src/secrets/vault.ts`)

Celilo uses Ansible Vault to encrypt secrets in generated Ansible configurations, ensuring no plaintext secrets are written to disk.

**Key Components**:
- **`vault.ts`** - Derives vault password from master key (deterministic)
- **`ansible/secrets.ts`** - Generates and encrypts secrets.yml file
- **`ansible-resolver.ts`** - Converts `$secret:` variables to Jinja2 `{{ }}` syntax

### Encryption Flow

1. **Decrypt secrets** from database (using master key)
2. **Format as YAML** with all module secrets
3. **Derive vault password** from master key (SHA-256)
4. **Encrypt with ansible-vault** (AES-256)
5. **Write encrypted file** to `ansible/inventory/secrets.yml`

### Vault Password Derivation

```typescript
import { deriveVaultPassword, getVaultPassword } from './secrets/vault';

// Deterministic derivation from master key
const masterKey = await getOrCreateMasterKey();
const vaultPassword = deriveVaultPassword(masterKey);
// Same master key always produces same vault password

// Or use orchestration function
const vaultPassword = await getVaultPassword();
```

**Algorithm**: `SHA-256(concat("ansible-vault:", masterKey))`

**Properties**:
- Deterministic (same master key → same vault password)
- Domain-separated (prevents key reuse attacks)
- Standard hash output (64 hex characters)

### Generated Ansible Structure

**Playbook** (`playbook.yml`):
```yaml
---
- name: Deploy Module
  hosts: module-host
  become: true

  vars_files:
    - inventory/secrets.yml  # Encrypted with ansible-vault

  roles:
    - module-role
```

**Secrets File** (`inventory/secrets.yml`):
```
$ANSIBLE_VAULT;1.1;AES256
35646166616130633832363334383234306139626264373935623630393937313639623334356138
6561383035303565323364373934306632336461626562630a386237663365633039396631623639
...
```

**Templates** (e.g., `templates/config.json`):
```json
{
  "api_key": "{{ api_key }}",
  "password": "{{ db_password }}"
}
```

### Variable Resolution

Ansible templates use different resolution from Terraform:

**Terraform** (resolved at generation time):
- `$self:hostname` → `"myhost"`
- `$system:dns.primary` → `"192.168.0.1"`
- `$secret:api_key` → `"actual_secret_value"` (**plaintext in file!**)

**Ansible** (resolved at runtime):
- `$self:hostname` → `"myhost"` (resolved at generation)
- `$system:dns.primary` → `"192.168.0.1"` (resolved at generation)
- `$secret:api_key` → `{{ api_key }}` (Jinja2 variable, resolved from encrypted secrets.yml)

### CLI Usage

**Get vault password**:
```bash
# Display vault password
celilo system vault-password

# Use with ansible-vault
ansible-vault view inventory/secrets.yml \
  --vault-password-file=<(celilo system vault-password)

# Edit encrypted secrets
ansible-vault edit inventory/secrets.yml \
  --vault-password-file=<(celilo system vault-password)
```

**Run Ansible playbook**:
```bash
cd /tmp/celilo/modules/homebridge/generated/ansible

# Option 1: Process substitution (recommended)
ansible-playbook playbook.yml \
  --vault-password-file=<(celilo system vault-password)

# Option 2: Environment variable
export ANSIBLE_VAULT_PASSWORD=$(celilo system vault-password)
echo "$ANSIBLE_VAULT_PASSWORD" | ansible-playbook playbook.yml --vault-password-file=/dev/stdin
```

### Security Considerations

**Benefits**:
- ✅ No plaintext secrets on disk
- ✅ Industry-standard encryption (Ansible Vault AES-256)
- ✅ Deterministic (same master key → same vault password)
- ✅ Works with standard Ansible tooling
- ✅ Secrets encrypted independently from celilo database

**Requirements**:
- ⚠️ **Ansible must be installed** - Generation fails without ansible-vault
- ⚠️ Master key must be protected (file permissions 0600)
- ⚠️ Vault password derivation is deterministic (anyone with master key can decrypt)

**Threat Model**:
- **Protects against**: Accidental secret exposure (logs, commits, backups)
- **Does NOT protect against**: Attacker with master key access
- **Assumption**: Master key is stored securely with restricted file permissions

### Implementation Details

**Files**:
- `src/secrets/vault.ts` - Vault password derivation
- `src/ansible/secrets.ts` - Secrets file generation and encryption
- `src/variables/ansible-resolver.ts` - Jinja2 variable conversion
- `src/templates/generator.ts` - Integration with template generation

**Functions**:
- `deriveVaultPassword(masterKey)` - Deterministic password derivation
- `generateAnsibleSecrets(moduleId, outputPath, db)` - Full generation pipeline
- `encryptWithAnsibleVault(yamlContent, password)` - Ansible Vault wrapper
- `convertSecretsToJinja(content, context)` - Variable conversion for Ansible

### Testing

**Unit Tests** (13 tests):
```bash
bun test src/secrets/vault.test.ts
bun test src/variables/ansible-resolver.test.ts
```

**Integration Tests**:
```bash
bun run test:integration
# Validates:
# - Secrets encrypted with ansible-vault
# - Ansible templates use Jinja2 variables
# - Vault password command works
# - Generated secrets can be decrypted
```

**Manual Verification**:
```bash
# 1. Generate module
celilo module generate homebridge

# 2. Check secrets file is encrypted
head /tmp/celilo/modules/homebridge/generated/ansible/inventory/secrets.yml
# Should start with: $ANSIBLE_VAULT;1.1;AES256

# 3. Decrypt and verify
ansible-vault view /tmp/celilo/modules/homebridge/generated/ansible/inventory/secrets.yml \
  --vault-password-file=<(celilo system vault-password)
# Should show YAML with secret values

# 4. Check templates use Jinja2
cat /tmp/celilo/modules/homebridge/generated/ansible/roles/homebridge/templates/config.json
# Should contain {{ variable_name }}, not plaintext secrets
```

## Template Generation

### System Overview (`src/templates/`)

Template generation following Rule 10.1 separation:

**Policy Functions**:
- `isTemplateFile(filename)` - Checks if file has template extension (`.tpl`, `.j2`)
- `getOutputFilename(template)` - Removes template extension from filename

**Execution Functions**:
- `discoverTemplateFiles(baseDir)` - Recursively finds template files
- `readTemplateFiles(modulePath, paths)` - Reads template content from disk
- `writeGeneratedFiles(outputPath, files)` - Writes generated files to disk

**Orchestration Function**:
- `generateTemplates(options)` - Main entry point, coordinates entire generation process

### Template Directories

Celilo looks for templates in:
- `terraform/` - Terraform configuration templates
- `ansible/` - Ansible playbook templates

Both directories are searched recursively.

### Template Extensions

- **`.tpl`** - Generic template files (Terraform style)
- **`.j2`** - Jinja2 template files (Ansible style)

Output files have extension removed:
- `main.tf.tpl` → `main.tf`
- `playbook.yml.j2` → `playbook.yml`

### Generation Process

1. **Discover** - Find all template files in `terraform/` and `ansible/` directories
2. **Build Context** - Load module config, secrets, and capabilities from database
3. **Read** - Load template content from files
4. **Resolve** - Replace variables using variable resolution system
5. **Write** - Save generated files to output directory
6. **Return** - Success with file list or detailed error messages

### Example Usage

```typescript
import { generateTemplates } from './templates/generator';

const result = await generateTemplates({
  moduleId: 'homebridge',
  modulePath: '/data/modules/homebridge',
  outputPath: '/data/modules/homebridge/generated',
});

if (result.success) {
  console.log(`Generated ${result.files.length} files:`);
  for (const file of result.files) {
    console.log(`  - ${file.path}`);
  }
} else {
  console.error('Generation failed:', result.error);
}
```

### Module Structure

```
modules/homebridge/
├── manifest.yml
├── terraform/
│   ├── main.tf.tpl          # → generated/terraform/main.tf
│   ├── variables.tf.tpl     # → generated/terraform/variables.tf
│   └── outputs.tf.tpl       # → generated/terraform/outputs.tf
└── ansible/
    ├── playbook.yml.tpl     # → generated/ansible/playbook.yml
    └── roles/
        └── homebridge/
            └── tasks/
                └── main.yml.tpl  # → generated/ansible/roles/homebridge/tasks/main.yml
```

### Template Example

**Input** (`terraform/main.tf.tpl`):
```hcl
resource "proxmox_lxc" "container" {
  hostname = "$self:hostname"
  cores    = $self:cores
  memory   = $self:memory

  network {
    name   = "eth0"
    bridge = "vmbr0"
    ip     = "$self:container_ip/24"
    gw     = "$system:management.ip"
  }

  provisioner "ansible" {
    playbook = "./ansible/playbook.yml"
  }
}
```

**Output** (`generated/terraform/main.tf`):
```hcl
resource "proxmox_lxc" "container" {
  hostname = "homebridge"
  cores    = 2
  memory   = 2048

  network {
    name   = "eth0"
    bridge = "vmbr0"
    ip     = "192.168.0.50/24"
    gw     = "192.168.0.10"
  }

  provisioner "ansible" {
    playbook = "./ansible/playbook.yml"
  }
}
```

### Error Handling

Clear error messages for common failures:

```typescript
// Module path doesn't exist
{ success: false, error: "Module path does not exist: /path" }

// No templates found
{ success: false, error: "No template files found in module" }

// Variable resolution failed
{
  success: false,
  error: "Failed to resolve variables in templates:\n" +
         "terraform/main.tf:\n" +
         "  $self:missing_var: Self variable 'missing_var' not found in module configuration"
}

// File write failed
{ success: false, error: "Failed to write generated files", details: Error }
```

### Integration

Template generation integrates with:
- **Variable Resolution** - Resolves all `$self:`, `$system:`, `$secret:`, `$capability:` variables
- **Database** - Loads module configuration and capability data
- **File System** - Reads templates and writes generated code

## Next Steps

Phase 0 Part 1.7:
1. ✅ Database layer
2. ✅ Manifest validation
3. ✅ Module import logic
4. ✅ Variable resolution system
5. ✅ Secret encryption (AES-256-GCM)
6. ✅ Template generation (File I/O + variable resolution)
7. CLI interface (Commands + user interaction)

## CLI Interface

### System Overview (`src/cli/`)

The CLI interface provides command-line tools for all Phase 0 operations following Rule 10.1 separation:

**Parser** (`parser.ts`) - Policy functions:
- `parseArguments()` - Parses command-line arguments into structured format
- `validateRequiredArgs()` - Validates argument count
- `getArg()`, `getFlag()`, `hasFlag()` - Type-safe accessors

**Commands** (`commands/`) - Orchestration functions:
- `module-import.ts` - Import modules from directory
- `module-list.ts` - List installed modules
- `module-config.ts` - Get/set module configuration
- `module-generate.ts` - Generate templates for modules
- `secret-set.ts` - Set encrypted secrets

**Entry Point** (`index.ts`) - Main orchestrator:
- `runCli()` - Routes commands to handlers
- `main()` - Entry point with error handling and exit codes

### Commands

**Module Import**
```bash
celilo module import <path> [--target <path>]

# Example
celilo module import ./modules/homebridge
celilo module import ./modules/homebridge --target /data/celilo/modules
```

**Module List**
```bash
celilo module list

# Output
Installed modules:

homebridge (v1.0.0) - IMPORTED
  Bridge for HomeKit accessories

pihole (v2.1.0) - CONFIGURED
```

**Module Configuration**
```bash
# Set configuration value
celilo module config set <module-id> <key> <value>

# Get specific value
celilo module config get <module-id> [key]

# Examples
celilo module config set homebridge hostname mybridge
celilo module config set homebridge container_ip 192.168.0.50
celilo module config get homebridge hostname
celilo module config get homebridge  # Get all config
```

**Module Generate**
```bash
celilo module generate <module-id> [--output <path>]

# Examples
celilo module generate homebridge
celilo module generate homebridge --output ./generated
```

**Secret Management**
```bash
celilo secret set <module-id> <name> <value>

# Example
celilo secret set homebridge api_key mykey123
```

**System Configuration**
```bash
# Set system-wide config
celilo system config set <key> <value>
celilo system config get [key]

# Get Ansible Vault password
celilo system vault-password

# Examples
celilo system config set dns.primary 192.168.0.1
celilo system config get dns.primary
ansible-vault view secrets.yml --vault-password-file=<(celilo system vault-password)
```

**Help**
```bash
celilo help
celilo --help
celilo -h
celilo module --help  # Command-specific help
```

### Argument Parsing

The CLI uses a simple argument parser without external dependencies:

**Format**: `celilo <command> [subcommand] [args...] [--flags]`

**Parsing Rules**:
- First argument is the command (`module`, `secret`, `help`)
- Second argument is the subcommand if it doesn't start with `--`
- Remaining arguments are positional args or flags
- Flags start with `--` and can be boolean (`--verbose`) or string (`--target /path`)

**Examples**:
```bash
# Command only
celilo help

# Command + subcommand
celilo module list

# Command + subcommand + args
celilo module import ./path

# Command + subcommand + args + flags
celilo module generate homebridge --output ./out

# Nested subcommand
celilo module config set homebridge hostname myhost
```

### Error Handling

All commands return structured results:
```typescript
interface CommandSuccess {
  success: true;
  message: string;
  data?: unknown;
}

interface CommandError {
  success: false;
  error: string;
  details?: unknown;
}
```

**Exit Codes**:
- `0` - Success
- `1` - Error (validation, not found, etc.)

**Error Messages** include usage hints:
```
Error: Missing required arguments. Expected 1, got 0

Usage: celilo module import <path> [--target <path>]
```

### Testing

All CLI components have comprehensive tests:
- **`parser.test.ts`** - Argument parsing (28 tests)
- **`cli.test.ts`** - Integration tests (23 tests)

Tests use environment variables to configure database and master key paths for isolation.

### Entry Point

The main entry point is `src/cli/index.ts`, which exports `main()`:
```typescript
export async function main(): Promise<void> {
  const result = await runCli(process.argv);
  // Handle output and exit codes
}
```

**Recommended:** Use the top-level wrapper script (from anywhere):
```bash
/path/to/celilo/celilo module list
/path/to/celilo/celilo system config get
```

Or run directly from backend directory:
```bash
bun run src/cli/index.ts module list
bun run src/cli/index.ts module import ./path
```

Or build a binary:
```bash
bun build src/cli/index.ts --compile --outfile celilo
./celilo module list
```

### Architecture

**CLI Orchestration**:
```
User Input
   ↓
parseArguments()   [Policy - parse and validate]
   ↓
runCli()          [Orchestration - route to handler]
   ↓
handleModuleXxx()  [Orchestration - coordinate operations]
   ↓
importModule()     [Execution - database, filesystem]
generateTemplates()
etc.
```

**Separation of Concerns**:
- **Parser** - Pure policy functions, no side effects
- **Handlers** - Orchestrate multiple operations
- **Core modules** - Execute actual work (import, generate, etc.)
- **Entry point** - Handle errors and exit codes

All commands integrate with existing modules:
- **Database** - Module metadata, configuration, secrets
- **Manifest validation** - Parse and validate manifests
- **Module import** - Copy files and insert to database
- **Variable resolution** - Resolve template variables
- **Secret encryption** - Encrypt/decrypt with master key
- **Template generation** - Generate Terraform/Ansible code

## Troubleshooting

### Common Errors and Solutions

#### "Module ID must use kebab-case"

**Error**:
```
Error: Manifest validation failed
Module ID must use kebab-case (lowercase letters, numbers, hyphens between segments)
```

**Cause**: Module ID uses underscores, uppercase, or invalid characters.

**Solution**: Use kebab-case for all module IDs:
```yaml
# ❌ WRONG
metadata:
  name: dns_external   # Underscores

# ✅ CORRECT
metadata:
  name: dns-external   # Kebab-case
```

**Pattern**: `/^[a-z0-9]+(-[a-z0-9]+)*$/`

---

#### "Variable resolution failed: Self variable not found"

**Error**:
```
Failed to resolve variables in templates:
terraform/main.tf:
  $self:container_ip: Self variable 'container_ip' not found in module configuration
```

**Cause**: Template references variable that isn't configured.

**Solution**: Set the missing variable:
```bash
celilo module config set <module-id> container_ip 10.0.20.100
```

Or check for typos in template:
```hcl
# Check spelling
ip = "$self:container_ip"  # Not 'containerIP', 'container-ip', etc.
```

---

#### "ENOENT: no such file or directory" (path with spaces)

**Error**:
```
Error: ENOENT: no such file or directory, open '/Users/user/Application Support/celilo/...'
```

**Cause**: Unquoted path in shell command with spaces.

**Solution**: Always quote paths in shell commands:
```typescript
// ❌ WRONG
execSync(`cd ${modulePath} && make build`);

// ✅ CORRECT
execSync(`cd "${modulePath}" && make build`);

// ✅ BEST
import { shellEscape } from '@/utils/shell';
execSync(`cd ${shellEscape(modulePath)} && make build`);
```

---

#### "Drizzle where clause returns all rows"

**Error**: Query returns all modules instead of filtering by ID.

**Cause**: Using JavaScript operators instead of Drizzle operators.

**Solution**: Use Drizzle operator functions:
```typescript
import { eq } from 'drizzle-orm';

// ❌ WRONG - returns ALL rows!
db.select().from(modules).where(modules.id === 'homebridge');

// ✅ CORRECT - filters correctly
db.select().from(modules).where(eq(modules.id, 'homebridge'));
```

**Always use**: `eq()`, `ne()`, `gt()`, `lt()`, `and()`, `or()`, etc.

---

#### "ansible-vault: command not found"

**Error**:
```
Error: Failed to encrypt secrets with ansible-vault
ansible-vault: command not found
```

**Cause**: Ansible is not installed or not in PATH.

**Solution**: Install Ansible:
```bash
# macOS
brew install ansible

# Ubuntu/Debian
sudo apt install ansible

# Verify
ansible-vault --version
```

---

#### "Test passes alone, fails in suite"

**Error**: Test passes when run individually, fails when run with others.

**Cause**: State leakage between tests (shared database, temp files, environment).

**Solution**: Add proper setup/teardown:
```typescript
import { describe, test, beforeEach, afterEach } from 'bun:test';

describe('Feature Tests', () => {
  let testDb: Database;
  let tempDir: string;

  beforeEach(async () => {
    testDb = await setupTestDatabase();      // Fresh database
    tempDir = await createTempDirectory();   // Fresh directory
  });

  afterEach(async () => {
    await cleanupTestDatabase(testDb);       // MUST await!
    await removeTempDirectory(tempDir);      // MUST await!
  });

  test('isolated test', async () => {
    // Test runs with clean state
  });
});
```

---

#### "Cannot find module '@/something'"

**Error**:
```
Error: Cannot find module '@/module/import'
```

**Cause**: TypeScript path aliases not resolved.

**Solution**: Check `tsconfig.json` has correct paths:
```json
{
  "compilerOptions": {
    "baseUrl": ".",
    "paths": {
      "@/*": ["src/*"]
    }
  }
}
```

And run with Bun (which respects tsconfig paths):
```bash
bun run src/cli/index.ts  # ✅ Works
node src/cli/index.ts     # ❌ Doesn't resolve @/ aliases
```

---

#### "Master key permission denied"

**Error**:
```
Error: EACCES: permission denied, open '/etc/celilo/master.key'
```

**Cause**: Master key file has wrong permissions or wrong owner.

**Solution**: Fix permissions:
```bash
# Check current permissions
ls -la /tmp/celilo/master.key

# Fix permissions (owner read/write only)
chmod 600 /tmp/celilo/master.key

# Or regenerate (WARNING: invalidates all encrypted secrets!)
rm /tmp/celilo/master.key
celilo system vault-password  # Generates new key
```

---

#### "Migration already exists"

**Error**:
```
Error: Migration 0005_add_capability_secrets.sql already exists
```

**Cause**: Schema changed but migration not generated or conflicting migration number.

**Solution**: Check for duplicates:
```bash
# List migrations
ls drizzle/

# Remove duplicate or conflicting migration
rm drizzle/0005_duplicate.sql

# Regenerate
bunx drizzle-kit generate
```

---

### Debug Checklist

When debugging issues, check:

1. **Test isolation**:
   - [ ] `beforeEach`/`afterEach` properly clean up
   - [ ] Tests can run in any order
   - [ ] Tests use isolated database/filesystem

2. **Path handling**:
   - [ ] All shell commands quote paths
   - [ ] Tested with paths containing spaces
   - [ ] No hardcoded absolute paths

3. **Drizzle queries**:
   - [ ] Using operator functions (`eq()`, not `===`)
   - [ ] Importing operators from `'drizzle-orm'`
   - [ ] Not using callback form in `where()`

4. **Identifier naming**:
   - [ ] Module IDs use kebab-case
   - [ ] No underscores or uppercase
   - [ ] Pattern: `/^[a-z0-9]+(-[a-z0-9]+)*$/`

5. **Variable resolution**:
   - [ ] Using colon syntax (`$self:var`, not `$self.var`)
   - [ ] Simple syntax for standalone values
   - [ ] Braced syntax for concatenation (`${self:disk}G`)

6. **Secrets**:
   - [ ] Master key exists and has 0600 permissions
   - [ ] Ansible installed for vault encryption
   - [ ] Secrets encrypted in database

---

### Getting Help

**Documentation**:
- [AGENTS.md](../../AGENTS.md) - Engineering standards
- [TESTING_STRATEGY.md](../../reference/TESTING_STRATEGY.md) - Testing approach
- [IDENTIFIER_NAMING_CONVENTIONS.md](../../reference/IDENTIFIER_NAMING_CONVENTIONS.md) - Naming rules
- [TEMPLATE_VARIABLE_SYNTAX.md](../../reference/TEMPLATE_VARIABLE_SYNTAX.md) - Variable syntax

**Debugging Tools**:
- Drizzle Studio: `bun run db:studio`
- Test watch mode: `bun test --watch`
- Verbose logs: `export CONDUCTOR_LOG_LEVEL=debug`

---

## Next Steps

Phase 0 Part 1.7:
1. ✅ Database layer
2. ✅ Manifest validation
3. ✅ Module import logic
4. ✅ Variable resolution system
5. ✅ Secret encryption (AES-256-GCM)
6. ✅ Template generation (File I/O + variable resolution)
7. ✅ CLI interface (Commands + user interaction)

**Phase 0 Part 1 is complete!**

Next: Phase 0 Part 2 - Web UI (React + tRPC)