# WildDuck Architecture

## Overview

WildDuck is a modern, scalable mail server built with Node.js and MongoDB, designed for high availability with no single point of failure. This document describes the system architecture, data flow, and key design patterns.

## System Architecture

### High-Level Components

```
┌──────────────────────────────────────────────────────────────┐
│                        Entry Points                           │
│  ┌──────┐  ┌──────┐  ┌──────┐  ┌──────┐  ┌──────┐  ┌──────┐│
│  │ IMAP │  │ POP3 │  │ LMTP │  │  API │  │Tasks │  │Webhks││
│  │ :143 │  │ :110 │  │:2424 │  │:8080 │  │      │  │      ││
│  └──┬───┘  └──┬───┘  └──┬───┘  └──┬───┘  └──┬───┘  └──┬───┘│
└──────┼────────┼─────────┼─────────┼─────────┼─────────┼─────┘
       │        │         │         │         │         │
       └────────┴─────────┴─────────┴─────────┴─────────┘
                          │
       ┌──────────────────┼──────────────────┐
       │                  │                  │
   ┌───▼───┐       ┌─────▼─────┐      ┌────▼────┐
   │MongoDB│◄──────┤  Worker   │─────►│  Redis  │
   │       │       │  Process  │      │         │
   └───────┘       └───────────┘      └─────────┘
                          │
                   ┌──────┴──────┐
                   │             │
              ┌────▼─────┐  ┌───▼─────┐
              │Event Bus │  │Webhooks │
              │(in-memory)│  │ (BullMQ)│
              └──────────┘  └─────────┘
```

### Process Model

WildDuck uses Node.js clustering for horizontal scaling:

1. **Master Process** (`server.js`)
   - Spawns worker processes (one per CPU core by default)
   - Handles process lifecycle and restart on crash
   - Coordinates shared resources (locks, etc.)

2. **Worker Processes** (`worker.js`)
   - Each runs: IMAP, POP3, LMTP, API, Tasks, Webhooks servers
   - Independent event loops (no shared state)
   - All workers share MongoDB and Redis connections
   - Each worker has its own event bus instance

## Core Subsystems

### 1. Protocol Servers

#### IMAP Server (imap.js)
- **Protocol**: RFC 3501 (IMAP4rev1)
- **Ports**: 143 (plain), 993 (TLS)
- **Handler pattern**: `lib/handlers/on-[command].js`
- **Key features**:
  - IDLE support for push notifications
  - Extensions: SPECIAL-USE, MOVE, QUOTA, etc.
  - Per-connection state management
  - Real-time notifications via `imap-notifier.js`

**IMAP Command Flow:**
```
Client → IMAP Server → Handler (on-*.js) → DB Operations
                    ↓
              Event Emission
                    ↓
         ┌──────────┴──────────┐
         │                     │
   Event Bus              Webhooks
    (in-memory)            (queue)
```

#### POP3 Server (pop3.js)
- **Protocol**: RFC 1939
- **Ports**: 110 (plain), 995 (TLS)
- **State**: Stateless per connection
- **Features**: UIDL support, TOP command

#### LMTP Server (lmtp.js)
- **Protocol**: RFC 2033
- **Port**: 2424
- **Purpose**: Mail delivery from MTA
- **Flow**: LMTP → Filter Handler → Message Handler → MongoDB

#### API Server (api.js)
- **Framework**: Restify
- **Port**: 8080
- **Authentication**: Token-based or session-based
- **Endpoints**: Users, mailboxes, messages, filters, etc.
- **Validation**: Joi schemas in `lib/schemas/`

### 2. Data Storage

#### MongoDB

**Purpose**: Primary data store

**Key Collections:**
- `users` - User accounts, settings, quotas
- `addresses` - Email addresses (multiple per user)
- `mailboxes` - User mailboxes (INBOX, Sent, etc.)
- `messages` - Email messages (metadata + references)
- `attachments.files` - GridFS attachment storage (deduplicated)
- `attachments.chunks` - GridFS chunks
- `journal` - IMAP notification journal
- `filters` - User-defined filtering rules
- `asps` - Application-specific passwords

**Indexes**: Defined in `indexes.yaml`

**Sharding Strategy**: Shard by user ID for horizontal scaling

#### Redis

**Purpose**: Caching, locks, queues

**Usage:**
- Session storage
- Rate limiting
- Distributed locks
- BullMQ queue backend
- PubSub for notifications

### 3. Message Processing

#### Message Handler (`lib/message-handler.js`)

**Responsibilities:**
- Store incoming messages
- Update mailbox counters
- Manage message flags
- Handle message moves/copies
- Quota enforcement

**Flow:**
```
addAsync(mailbox, message, options)
  ↓
prepareMessage(raw) → Parse MIME, extract attachments
  ↓
checkQuota(user) → Verify storage limits
  ↓
storeAttachments(attachments) → GridFS (deduplicated)
  ↓
insertMessage(messageData) → MongoDB
  ↓
updateMailboxCounters() → Increment uidNext, etc.
  ↓
emitNotification() → IMAP notifier + Event bus
  ↓
return { id, uid, mailbox }
```

**Encryption Support:**
- Optional PGP/GPG encryption per mailbox
- Public key stored in `users.pubKey`
- Encryption happens in `encryptMessages()`

#### Filter Handler (`lib/filter-handler.js`)

**Purpose**: Apply user-defined rules to incoming mail

**Flow:**
```
LMTP → Filter Handler
  ↓
Load user filters → MongoDB query
  ↓
Match conditions → From/To/Subject/Size
  ↓
Apply actions:
  - Move to mailbox
  - Set flags
  - Mark spam/ham
  - Delete
  - Forward
  ↓
Message Handler → Store message
```

### 4. Event System

#### Traditional Webhooks (`lib/events.js` + `webhooks.js`)

**Purpose**: Send events to external HTTP endpoints

**Flow:**
```
Operation → publish(redis, data) → BullMQ Queue → Webhook Worker
                                                        ↓
                                                   HTTP POST
```

**Configuration**: Webhooks configured in WildDuck settings

#### Event Bus (NEW - `lib/event-bus.js`)

**Purpose**: In-process event distribution for observability and integration

**Architecture:**
```
Services (API/IMAP/LMTP/POP3)
         │
         ▼
lib/events.js publish()
         │
         ├─────────────────┐
         │                 │
   Event Bus          Webhooks
   (interceptor)      (original)
         │
         ├──> Test Tracker (NODE_ENV=test)
         ├──> Debug Logger (DEBUG_EVENTS=1)
         └──> Custom Attachments
```

**Key Features:**
- **In-memory only** - No database persistence
- **Asynchronous** - Events emitted via `setImmediate()` (non-blocking)
- **Pluggable** - Custom attachments extend `EventAttachment` base class
- **15 event types** - All major operations covered
- **Zero impact** - Webhooks continue to work unchanged

**Event Types (15):**
- Message: `added`, `flags.changed`, `moved`, `deleted`
- Address: `user.created`, `user.deleted`
- Mailbox: `created`, `renamed`, `deleted`
- Settings: `updated`, `autoreply.enabled`, `autoreply.disabled`
- Filter: `created`, `deleted`
- Spam: `marked.spam`, `marked.ham`

**Event Structure:**
```javascript
{
  type: 'message.added',          // category.action format
  category: 'message',             // message, mailbox, address, etc.
  action: 'added',                 // created, updated, deleted, etc.
  timestamp: Date,
  source: 'lmtp',                  // api, imap, lmtp, auto
  user: '507f...',                 // User ID (string)
  mailbox: '507f...',              // Mailbox ID (optional)
  message: '507f...',              // Message ID (optional)
  data: { size: 1024, uid: 42 },  // Event-specific data
  metadata: { encrypted: true }    // Optional metadata
}
```

**Attachment Pattern:**
```javascript
class MyAttachment extends EventAttachment {
  constructor() {
    super('my-attachment');
  }

  shouldHandle(event) {
    return event.category === 'message';
  }

  async onEvent(event) {
    // Process event (non-blocking)
  }

  async onStop() {
    // Cleanup on shutdown
  }
}
```

**Built-in Attachments:**
1. **Test Tracker** - Stores events for test assertions
2. **Debug Logger** - Logs events to console

**See:** `docs/event-bus.md` for complete documentation

### 5. Background Processing

#### Task Processor (`tasks.js`)

**Purpose**: Handle scheduled and deferred tasks

**Tasks:**
- Delete expired messages
- Calculate storage quotas
- Clean up old sessions
- Generate reports

**Scheduling**: Cron-style schedules in config

#### Webhook Processor (`webhooks.js`)

**Purpose**: Send events to configured webhooks

**Flow:** BullMQ → HTTP POST → Retry on failure

### 6. Notification System

#### IMAP Notifier (`lib/imap-notifier.js`)

**Purpose**: Real-time notifications for connected IMAP clients

**Mechanism:**
1. Operations write to `journal` collection
2. Notifier polls journal periodically
3. Connected clients receive EXISTS/EXPUNGE notifications
4. Journal entries are removed after delivery

**Notifications:**
- `EXISTS` - New message
- `EXPUNGE` - Message deleted
- `FETCH` - Flag change

## Data Flow Patterns

### Incoming Mail (LMTP)

```
MTA (Postfix/etc.)
  ↓ LMTP
LMTP Server (port 2424)
  ↓
Recipient validation
  ↓
Filter Handler → Apply rules
  ↓
Message Handler → addAsync()
  ↓
┌─────────┴─────────┐
│                   │
MongoDB          Event Emission
(store)          ├─> Event Bus
                 └─> Webhooks
  ↓
IMAP Notifier → Push to clients
```

### IMAP Message Retrieval

```
IMAP Client
  ↓ FETCH command
IMAP Server
  ↓
on-fetch.js Handler
  ↓
Message Handler → get()
  ↓
MongoDB → Load message
  ↓
┌────┴────┐
│         │
Metadata  Attachments
(inline)  (GridFS)
  ↓
Assemble MIME
  ↓
Return to client
```

### API Message Creation

```
API Client (POST /users/:user/mailboxes/:mailbox/messages)
  ↓
API Server (lib/api/messages.js)
  ↓
Authentication & Authorization
  ↓
Joi Validation
  ↓
Message Handler → addAsync({ source: 'API' })
  ↓
(same flow as LMTP)
  ↓
Return { id, message: { id, mailbox, size } }
```

### Message Move (IMAP MOVE)

```
IMAP Client → MOVE command
  ↓
on-move.js Handler
  ↓
Acquire lock (prevent conflicts)
  ↓
Move message (atomic)
  ├─> Update source mailbox counters
  ├─> Update target mailbox counters
  ├─> Update message.mailbox field
  └─> Emit message.moved event
  ↓
Release lock
  ↓
Notify IMAP clients (source & target mailboxes)
```

## Scaling Considerations

### Horizontal Scaling

**Application Tier:**
- Run multiple WildDuck instances behind load balancer
- Shared MongoDB and Redis required
- No session affinity needed (stateless API)
- IMAP connections are sticky per client

**Database Tier:**
- MongoDB replica set for HA
- Shard by user ID for horizontal scaling
- Redis Cluster for distributed caching

**Deployment Pattern:**
```
        Load Balancer
       /      |      \
  WD-1      WD-2    WD-3
    \        |      /
     \       |     /
   MongoDB Replica Set
         +
   Redis Cluster
```

### Performance Optimization

**Database:**
- Compound indexes for common queries (see `indexes.yaml`)
- Projection to limit data transfer
- TTL indexes for auto-cleanup
- Connection pooling

**Caching:**
- Redis cache for user lookups
- In-memory cache for frequently accessed data
- TTL-based expiration

**Message Storage:**
- GridFS for large attachments
- Deduplication (same attachment = one storage)
- Streaming for large messages

**Event System:**
- Asynchronous emission (non-blocking)
- Fire-and-forget pattern
- Attachment errors don't affect operations

## Security Architecture

### Authentication

**Methods:**
1. **Username/Password** - PBKDF2/bcrypt hashing
2. **Application-Specific Passwords** - Limited scope tokens
3. **2FA** - TOTP (Time-based OTP) support
4. **WebAuthn** - Hardware key support

**Crypto Emails Mode** (Custom feature):
- Auto-creates users on authentication
- No password validation
- Requires `emailDomain` parameter
- Enabled via `APPCONF_api_cryptoEmails=true`

### Authorization

**Role-Based Access Control (RBAC):**
- Roles defined in `config/roles.json`
- API endpoints check permissions
- Operations: createAny, readOwn, updateOwn, etc.

### Encryption

**Transport:**
- TLS/SSL for all protocols (IMAPS, POP3S, HTTPS)
- STARTTLS support
- Certificate management via ACME

**At Rest:**
- Optional message encryption (PGP/GPG)
- Mailbox-level encryption flag
- Public key stored in user record

## Design Patterns

### Handler Pattern

IMAP commands use consistent handler pattern:

**File:** `lib/handlers/on-[command].js`

```javascript
module.exports = server => ({
  // Capability check
  needsMailbox: true, // Requires mailbox to be selected

  // Command handler
  async handler(command, callback) {
    // 1. Validate input
    // 2. Check permissions
    // 3. Perform operation
    // 4. Emit events
    // 5. Return response
    callback(null, { response: 'OK' });
  }
});
```

### Repository Pattern

Data access through handler classes:

- `UserHandler` - User operations
- `MailboxHandler` - Mailbox operations
- `MessageHandler` - Message operations

**Benefits:**
- Centralized business logic
- Consistent error handling
- Easier testing

### Event Sourcing (Partial)

Journal collection acts as event log:

- IMAP notifications stored as events
- Delivered to connected clients
- Pruned after delivery
- Enables real-time updates

## Configuration

### File Structure

```
config/
├── default.toml       # Main config with @include directives
├── api.toml          # API server settings
├── imap.toml         # IMAP server settings
├── pop3.toml         # POP3 server settings
├── lmtp.toml         # LMTP server settings
├── dbs.toml          # Database connections
├── tls.toml          # TLS/SSL certificates
├── test.toml         # Test overrides
└── roles.json        # RBAC role definitions
```

### Environment Variables

- `NODE_ENV` - Environment (production/test/development)
- `NODE_CONFIG_ONLY` - Print config and exit
- `APPCONF_*` - Override config values
- `DEBUG_EVENTS` - Enable event bus debug logging

## Monitoring & Observability

### Logging

**Structured logging** with context:
```javascript
server.logger.info({ tnx: 'copy', cid: session.id }, 'Message copied');
```

**Log Levels:** silly, debug, info, warn, error

### Metrics

**Event Bus Metrics Collector** (example attachment):
- Event counts by type
- Events per second/minute
- Top active users
- Category breakdowns

### Health Checks

**API Endpoints:**
- `GET /health` - Service health
- `GET /metrics/events` - Event bus metrics (if enabled)

### Debugging

**Tools:**
- `DEBUG_EVENTS=1` - Event bus debug logging
- `NODE_ENV=test` - Test tracker for event verification
- MongoDB query logging
- IMAP protocol logging

## Testing Architecture

### Test Modes

1. **Standard Mode** - Traditional auth
2. **Crypto Mode** - Auto-provision users

**Toggle:** `APPCONF_api_cryptoEmails=true`

### Test Structure

```
test/
├── api/             # API endpoint tests
├── api-test.js      # Main API tests
├── event-bus-test.js # Event bus unit tests
└── test-config.js   # Shared test utilities
```

### Event Verification

**Pattern:**
```javascript
testTracker.clear();
// perform operation
await new Promise(resolve => setTimeout(resolve, 50));
const events = testTracker.getEventsByType('message.added');
expect(events).to.have.lengthOf(1);
```

**See:** `TESTING.md` for complete guide

## Future Enhancements

### Event Bus Roadmap
- [ ] IMAP integration tests for remaining 4 events
- [ ] Redis-based event persistence (optional)
- [ ] Event replay capability
- [ ] More built-in attachments (metrics, audit logger)

### Scalability
- [ ] Multi-datacenter support
- [ ] Read replicas for MongoDB
- [ ] Message archiving to S3/cold storage

### Features
- [ ] Calendar/Contacts support (CalDAV/CardDAV)
- [ ] Full-text search (Elasticsearch integration)
- [ ] Advanced spam filtering
- [ ] Machine learning for auto-categorization

## References

### Documentation
- **CLAUDE.md** - AI development quick reference
- **PATTERNS.md** - Code patterns and examples
- **TESTING.md** - Testing strategies
- **API.md** - API endpoint documentation
- **docs/event-bus.md** - Event bus complete guide
- **docs/event-bus-api.md** - Event bus API reference
- **docs/event-bus-examples.md** - Event bus examples

### External
- [WildDuck Docs](https://docs.wildduck.email)
- [IMAP RFC 3501](https://tools.ietf.org/html/rfc3501)
- [MongoDB Docs](https://docs.mongodb.com)
- [Restify Framework](http://restify.com/)

---

**Last Updated:** November 2025
**Version:** 8.1.10 (with Event Bus)