# @plyaz/storage

Multi-provider file storage package with support for Cloudflare R2, Supabase Storage, and custom adapters. Includes virus scanning, metadata extraction, media processing, compliance management, document rendering (PDF/Excel/DOCX), and webhook tracking.

## Installation

```bash
pnpm add @plyaz/storage
```

**Requirements:**
- Node.js >= 22.4.0
- pnpm >= 8.0.0

### Optional Dependencies

The package uses **optional dependencies** for specific features. Install only what you need:

```bash
# Storage Adapters
pnpm add @aws-sdk/client-s3 @aws-sdk/s3-request-presigner  # For Cloudflare R2 / AWS S3
pnpm add @supabase/supabase-js                              # For Supabase Storage

# Image Processing
pnpm add sharp                                              # Image optimization & variants

# Video Processing
pnpm add @ffmpeg-installer/ffmpeg fluent-ffmpeg            # Video transcoding

# PDF Rendering
pnpm add pdfkit                                             # Lightweight PDF (~5MB)
pnpm add puppeteer                                          # Full HTML/CSS support (~200MB)

# Excel/Word Documents
pnpm add exceljs                                            # Excel spreadsheets
pnpm add docxtemplater pizzip                               # Word documents
```

> **See [Confluence Documentation](https://plyaz.atlassian.net/wiki/spaces/SD/overview) for complete installation scenarios and dependency matrix**

## Quick Start

### Basic File Upload

```typescript
import { StorageService, CloudflareR2Adapter } from '@plyaz/storage';
import { FileCategory, EntityType } from '@plyaz/types';

// Initialize with Cloudflare R2
const storage = new StorageService({
  adapters: [
    new CloudflareR2Adapter({
      name: 'cloudflare-r2',
      accountId: process.env.CLOUDFLARE_ACCOUNT_ID!,
      accessKeyId: process.env.R2_ACCESS_KEY_ID!,
      secretAccessKey: process.env.R2_SECRET_ACCESS_KEY!,
      bucket: 'my-uploads'
    })
  ],
  handlers: {
    onFileUploaded: async (payload) => {
      console.log('File uploaded:', payload.metadata?.fileId);
      // Save metadata to database
    }
  }
});

// Upload file
const result = await storage.uploadFile({
  file: fileBuffer,
  filename: 'document.pdf',
  mimeType: 'application/pdf',
  category: FileCategory.DOCUMENT,
  entityType: EntityType.USER,
  entityId: 'user-123'
});

console.log('File ID:', result.metadata.fileId);
console.log('Public URL:', result.metadata.publicUrl);
```

### Multi-Provider with Failover

```typescript
import {
  StorageService,
  CloudflareR2Adapter,
  SupabaseStorageAdapter,
  createProductionLogger
} from '@plyaz/storage';

const logger = createProductionLogger({ service: 'storage' });

const storage = new StorageService({
  adapters: [
    new CloudflareR2Adapter({
      name: 'r2-primary',
      priority: 100,  // Higher priority = preferred
      // ... config
    }),
    new SupabaseStorageAdapter({
      name: 'supabase-backup',
      priority: 50,   // Fallback adapter
      // ... config
    })
  ],
  logger
});
```

### Image Processing with Variants

```typescript
import { StorageService, SharpImagePlugin } from '@plyaz/storage';

const storage = new StorageService({
  adapters: [/* ... */],
  plugins: [
    new SharpImagePlugin({
      enableVariants: true,
      variants: [
        { name: 'thumbnail', width: 150, height: 150 },
        { name: 'medium', width: 800, height: 600 },
        { name: 'large', width: 1920, height: 1080 }
      ],
      formats: ['webp', 'jpeg'],
      quality: 85
    })
  ]
});

// Automatically generates: thumbnail.webp, medium.webp, large.webp variants
const result = await storage.uploadFile({
  file: imageBuffer,
  filename: 'photo.jpg',
  mimeType: 'image/jpeg',
  category: FileCategory.PROFILE_IMAGE,
  entityType: EntityType.USER,
  entityId: 'user-123'
});

console.log('Variants:', result.metadata.variants);
```

### PDF Generation from Templates

```typescript
import { StorageService, CloudflareR2Adapter } from '@plyaz/storage';
import { StorageRendererType, OutputFormat, FileCategory, EntityType } from '@plyaz/types';

// StorageService handles templates internally!
const storage = new StorageService({
  adapters: [r2Adapter],
  template: {
    templateBasePath: './templates',  // Path to your templates folder
    defaultLocale: 'en',
    renderers: [StorageRendererType.PDFKIT]  // Auto-registers PDFKit renderer
  }
});

// Generate and upload PDF from template
const result = await storage.generateFileToPath({
  templateId: 'invoice',
  templateData: {
    invoiceNumber: 'INV-001',
    customerName: 'John Doe',
    items: [{ name: 'Product', price: 99.99 }],
    total: 99.99
  },
  filename: 'invoice-001.pdf',
  format: OutputFormat.PDF,
  category: FileCategory.FINANCIAL_DOCUMENT,
  entityType: EntityType.ORGANIZATION,
  entityId: 'org-123'
});

console.log('Generated PDF:', result.metadata.publicUrl);
```

## Features

### Multi-Provider Storage
- ✅ **Cloudflare R2** - S3-compatible, zero egress fees
- ✅ **Supabase Storage** - PostgreSQL-based object storage
- ✅ **AWS S3** - Full S3 support via R2 adapter
- ✅ **Mock Adapter** - Testing and development

### Automatic Failover
- Priority-based adapter selection
- Health monitoring with circuit breaker
- Automatic retry with exponential backoff
- Adapter degradation handling

### Image Processing (Sharp)
- Automatic variant generation (thumbnail, medium, large)
- Format conversion (WebP, AVIF, JPEG, PNG)
- Quality optimization
- EXIF metadata extraction
- Responsive images for different devices

### Video Processing (FFmpeg)
- Video transcoding
- Thumbnail generation
- Format conversion (MP4, WebM, HLS)
- Resolution variants
- Bitrate optimization

### Document Rendering
- **PDFKit** - Lightweight programmatic PDFs (~5MB)
- **Puppeteer** - Complex HTML/CSS to PDF (~200MB)
- **Playwright** - Multi-browser rendering (~300MB)
- **ExcelJS** - Excel spreadsheets and reports
- **DocxTemplater** - Word document generation

### Compliance & Security
- **Virus Scanning** - ClamAV, VirusTotal integration
- **Metadata Extraction** - EXIF, ID3, PDF metadata
- **File Validation** - Size, type, content validation
- **Compliance Manager** - Retention policies, immutability
- **Signed URLs** - Temporary download/upload links
- **Idempotency** - Prevent duplicate uploads

### Template System
- Handlebars templates with layouts
- Markdown support
- Multi-language (i18n)
- YAML frontmatter configuration
- Dynamic data binding

### CDN Integration
- **Cloudflare CDN** - Cache purging, zone management
- **CloudFront** - Invalidation, distribution management
- **Fastly** - Edge caching, instant purge

### Event System
- Upload/download/delete lifecycle events
- Adapter failover notifications
- Plugin execution tracking
- Webhook delivery events

### Advanced Features
- Chunked uploads for large files (>100MB)
- Upload progress tracking
- Multi-tenant bucket routing
- Automatic bucket creation
- Queue-based processing
- Database metadata integration

## Storage Adapter Matrix

| Feature | Cloudflare R2 | Supabase | Mock |
|---------|--------------|----------|------|
| **File Upload** | ✅ Full | ✅ Full | ✅ Full |
| **Signed URLs** | ✅ Yes | ✅ Yes | ✅ Yes |
| **Chunked Upload** | ✅ Yes | ✅ Yes | ✅ Yes |
| **CDN Integration** | ✅ Cloudflare | ❌ No | ❌ No |
| **Cost** | Zero egress | PostgreSQL-based | Free |
| **Best For** | Production media | Postgres-based apps | Testing |

## Plugin Support Matrix

| Plugin | Dependencies | Size | Use Case |
|--------|--------------|------|----------|
| **SharpImagePlugin** | `sharp`, `exifr` | ~25MB | Image variants, optimization |
| **FFmpegVideoPlugin** | `ffmpeg`, `fluent-ffmpeg` | ~100MB | Video transcoding |
| **MetadataExtractionPlugin** | `exifr`, `music-metadata`, `pdf-parse` | ~5MB | File metadata extraction |
| **VirusScanPlugin** | `clamav` or `virustotal-api` | Varies | Security scanning |
| **CloudflareCDNPlugin** | None (HTTP API) | 0MB | Cache management |
| **CloudFrontCDNPlugin** | None (HTTP API) | 0MB | AWS CloudFront cache invalidation |
| **FastlyCDNPlugin** | None (HTTP API) | 0MB | Fastly CDN cache purging |

## Idempotency Support

Prevent duplicate uploads and webhook processing:

| Adapter | Dependencies | Persistence | Use Case |
|---------|--------------|-------------|----------|
| **InMemoryIdempotencyAdapter** | None | In-memory (process lifetime) | Testing, development, serverless |
| **RedisIdempotencyAdapter** | `ioredis` | Redis (distributed) | Production, multi-instance deployments |

**Example Usage:**
```typescript
import {
  StorageService,
  WebhookManager,
  RedisIdempotencyAdapter
} from '@plyaz/storage';

const idempotencyStore = new RedisIdempotencyAdapter({
  redis: {
    host: 'localhost',
    port: 6379,
  },
  ttl: 86400, // 24 hours
});

const webhookManager = new WebhookManager({
  idempotencyAdapter: idempotencyStore,
});

// Duplicate webhooks are automatically detected and ignored
await webhookManager.processWebhook(payload, headers);
```

## Development Commands

```bash
# Development
pnpm dev              # Watch mode development
pnpm build            # Build for production
pnpm clean            # Clean dist directory

# Testing
pnpm test             # Run all tests
pnpm test:watch       # Run tests in watch mode
pnpm test:coverage    # Run tests with coverage report
pnpm test:ui          # Open Vitest UI

# Code Quality
pnpm lint             # Run ESLint
pnpm lint:fix         # Auto-fix linting issues
pnpm format           # Format code with Prettier
pnpm format:check     # Check formatting
pnpm type:check       # TypeScript type checking

# Examples
pnpm example:basic                    # Basic upload/download
pnpm example:adapter:cloudflare-r2    # Cloudflare R2 setup
pnpm example:adapter:supabase         # Supabase setup
pnpm example:adapter:both             # Multi-provider routing
```

## Package Dependencies

Per Plyaz monorepo architecture:

### Internal Dependencies
- **@plyaz/types** - Type definitions and interfaces
- **@plyaz/logger** - Structured logging with PII redaction
- **@plyaz/api** - Global API configuration
- **@plyaz/errors** - Error handling with correlation IDs
- **@plyaz/config** - Configuration management

### Core Dependencies (Always Installed)
- **file-type** - File type detection (~500KB)
- **handlebars** - Template engine (~500KB)
- **marked** - Markdown parsing (~100KB)
- **gray-matter** - YAML frontmatter (~100KB)

**Total Core Size: ~2MB**

### Optional Dependencies
See the [Confluence documentation](https://plyaz.atlassian.net/wiki/spaces/SD/overview) for the complete list of feature-specific dependencies and installation scenarios.

## Documentation

Comprehensive documentation organized into focused guides.

### 📚 Documentation Structure

**Documentation Index** - Start here for complete navigation

**Quick Access:**
- **Quick Start Guide** - Installation and first steps
- **Architecture Overview** - System architecture and data flow
- **API Reference** - Complete API documentation
- **Storage Adapters** - R2, Supabase, Mock adapters
- **Plugin System** - Virus scan, image/video processing, CDN
- **Templates & Documents** - PDF, Excel, Word generation
- **Deployment Guide** - Production deployment strategies
- **Examples** - Real-world usage examples

**For Confluence:**
Complete documentation is available on Confluence. Each guide is self-contained and published as a separate page.

## Architecture

```
StorageService
├── AdapterRegistry (multi-provider routing & failover)
├── PluginRegistry (lifecycle hooks & processing)
├── EventManager (event emission & subscriptions)
├── BucketRouter (content-based routing)
├── ComplianceManager (retention & immutability)
├── QueueProcessor (async operations)
└── WebhookManager (delivery tracking)
    ├── Storage Adapters (Cloudflare R2, Supabase, Mock)
    ├── Processing Plugins (Sharp, FFmpeg, Virus Scan)
    ├── CDN Plugins (Cloudflare, CloudFront, Fastly)
    ├── Renderers (PDFKit, Puppeteer, ExcelJS, DOCX)
    └── Idempotency (InMemory, Redis)
```

## Error Handling

Uses Result type pattern for safe error handling:

```typescript
const result = await storage.uploadFile({ /* ... */ });

if (result.success) {
  console.log('File uploaded:', result.metadata.fileId);
  console.log('URL:', result.metadata.publicUrl);
  console.log('Adapter used:', result.metadata.adapter);
} else {
  console.error('Upload failed:', result.error);
  // result.error contains: message, code, context
}
```

Common error codes:
```typescript
import { STORAGE_ERROR_CODES } from '@plyaz/storage';

// FILE_TOO_LARGE, INVALID_FILE_TYPE, ADAPTER_OPERATION_FAILED,
// ADAPTER_TIMEOUT, ADAPTER_CONFIGURATION_INVALID, PLUGIN_EXECUTION_FAILED
```

## Event System

```typescript
const storage = new StorageService({
  adapters: [/* ... */],

  // Lifecycle events
  handlers: {
    onFileUploaded: async (event) => {
      console.log('Uploaded:', event.metadata?.fileId);
      // Save to database
      await db.files.create({
        id: event.metadata?.fileId,
        path: event.metadata?.path,
        adapter: event.metadata?.adapter
      });
    },

    onFileDeleted: async (event) => {
      console.log('Deleted:', event.metadata?.fileId);
      // Update database
      await db.files.delete(event.metadata?.fileId);
    },

    onAdapterFailed: async (event) => {
      console.warn('Adapter failed, using fallback:', event.data);
      // Log to monitoring service
      await monitoring.alert('storage-adapter-failure', event);
    },

    onFileUploadFailed: async (event) => {
      console.error('All adapters failed:', event.error);
      // Send alert
      await alerts.critical('storage-complete-failure', event);
    }
  }
});
```

## Multi-Tenant Routing

Automatic routing based on content type and organization tier:

```typescript
import { StorageService, BucketRouter } from '@plyaz/storage';
import { BucketPurpose } from '@plyaz/types';

const storage = new StorageService({
  adapters: [r2Adapter, supabaseAdapter],
  bucketRouter: new BucketRouter({
    availableAdapters: ['cloudflare-r2', 'supabase'],
    enableDefaultRules: true,
    adapterPreferences: {
      [BucketPurpose.COMPLIANCE]: 'cloudflare-r2',      // Financial docs → R2
      [BucketPurpose.MEDIA_IMAGES]: 'supabase',         // Images → Supabase
      [BucketPurpose.MEDIA_VIDEOS]: 'cloudflare-r2',    // Videos → R2
    }
  })
});

// Automatically routes to R2 (compliance documents)
await storage.uploadFile({
  file: invoiceBuffer,
  category: FileCategory.FINANCIAL_DOCUMENT,
  documentType: DocumentType.INVOICE,
  organizationTier: OrganizationTier.ENTERPRISE
});

// Automatically routes to Supabase (media)
await storage.uploadFile({
  file: imageBuffer,
  category: FileCategory.PROFILE_IMAGE,
  visibility: 'public'
});
```

## Compliance Features

```typescript
import { StorageService } from '@plyaz/storage';
import { FileCategory } from '@plyaz/types';

// StorageService handles compliance internally!
const storage = new StorageService({
  adapters: [r2Adapter],
  compliance: {
    enabled: true,
    strictMode: true,  // Enforce retention policies strictly
    retentionPolicies: {
      [FileCategory.FINANCIAL_DOCUMENT]: {
        retentionYears: 7,        // Keep for 7 years
        immutable: true,          // Cannot be deleted
        requiresAuditLog: true,
        softDelete: true
      }
    },
    defaultRetentionPolicy: {
      retentionYears: 1,
      gracePeriodDays: 30
    }
  }
});

// Compliance is automatically enforced on delete operations
try {
  await storage.deleteFile({
    fileId: 'invoice-123',
    category: FileCategory.FINANCIAL_DOCUMENT
  });
} catch (error) {
  console.log('Deletion blocked by compliance:', error.message);
  // "File is immutable and cannot be deleted"
}
```

## Installation Scenarios

### Scenario 1: Basic Storage (Minimal)
```bash
pnpm add @plyaz/storage @aws-sdk/client-s3
# Total: ~14MB
# Use case: Simple file uploads to R2/S3
```

### Scenario 2: Image Hosting
```bash
pnpm add @plyaz/storage @aws-sdk/client-s3 sharp
# Total: ~37MB
# Use case: Photo galleries, user avatars with variants
```

### Scenario 3: Document Management
```bash
pnpm add @plyaz/storage @supabase/supabase-js pdfkit exceljs
# Total: ~13MB
# Use case: Document generation and storage
```

### Scenario 4: Full Media Platform
```bash
pnpm add @plyaz/storage @aws-sdk/client-s3 sharp @ffmpeg-installer/ffmpeg fluent-ffmpeg puppeteer
# Total: ~337MB
# Use case: Images, videos, PDFs with full processing
```

### Scenario 5: Serverless Optimized
```bash
pnpm add @plyaz/storage @aws-sdk/client-s3 pdfkit
# Total: ~17MB
# Use case: AWS Lambda, Vercel, Netlify Functions
```

## Common Use Cases

### User Profile Picture with Variants
```typescript
// Automatically generates thumbnail, medium, large variants
const result = await storage.uploadFile({
  file: avatarBuffer,
  filename: 'avatar.jpg',
  category: FileCategory.PROFILE_IMAGE,
  entityType: EntityType.USER,
  entityId: userId
});

// Use variants in frontend
<img src={result.metadata.variants?.thumbnail} alt="Avatar" />
```

### Generate & Store Invoice PDF
```typescript
import { OutputFormat, FileCategory, DocumentType, EntityType } from '@plyaz/types';

// StorageService handles PDF generation + upload in one call!
const result = await storage.generateFileToPath({
  templateId: 'invoice',
  templateData: {
    invoiceNumber: 'INV-001',
    customer: customerData,
    items: lineItems
  },
  filename: `invoice-${invoiceNumber}.pdf`,
  format: OutputFormat.PDF,
  category: FileCategory.FINANCIAL_DOCUMENT,
  documentType: DocumentType.INVOICE,
  entityType: EntityType.ORGANIZATION,
  entityId: organizationId
});

console.log('Invoice PDF:', result.metadata.publicUrl);
// Compliance is automatically enforced based on category
```

### Video Upload with Virus Scan
```typescript
const storage = new StorageService({
  adapters: [r2Adapter],
  plugins: [
    new VirusScanPlugin({
      provider: new ClamAVProvider({ host: 'clamav-service' })
    })
  ]
});

const result = await storage.uploadFile({
  file: videoBuffer,
  filename: 'tutorial.mp4',
  category: FileCategory.POST_VIDEO,
  entityType: EntityType.POST,
  entityId: postId
});
// Automatically scanned for viruses before upload
```

## Using in Your Application

This is a **library package** - you install it in your application and configure it programmatically.

### Configuration in Your App

```typescript
import { StorageService, CloudflareR2Adapter } from '@plyaz/storage';

// Configure with your app's configuration system
const storage = new StorageService({
  adapters: [
    new CloudflareR2Adapter({
      name: 'cloudflare-r2',
      accountId: config.cloudflare.accountId,        // From your config
      accessKeyId: config.r2.accessKeyId,            // From your config
      secretAccessKey: config.r2.secretAccessKey,    // From your config
      bucket: config.r2.bucketName
    })
  ]
});
```

> **Note:** The package itself does NOT use `process.env`. You pass configuration when instantiating adapters. How you load config (env vars, config files, secrets manager, etc.) is up to your application.

### Docker Deployment (Your Application)

When deploying **your application** that uses this package:

```dockerfile
FROM node:22-alpine

# Install system dependencies if using image/video processing
RUN apk add --no-cache \
    vips-dev \        # Required for Sharp (if using SharpImagePlugin)
    ffmpeg            # Required for FFmpeg (if using FFmpegVideoPlugin)

WORKDIR /app
COPY package.json pnpm-lock.yaml ./

# Install only the features you need
RUN pnpm add @plyaz/storage @aws-sdk/client-s3 sharp

COPY . .
RUN pnpm build

CMD ["node", "dist/index.js"]
```

### Examples Directory

The `examples/` directory contains `.env.example` for **local testing only**. These show how to configure the package but are NOT part of the package distribution.

```bash
# For running examples locally
cp examples/.env.example examples/.env
# Edit .env with your credentials
pnpm example:basic
```

## Contributing

When adding new features:
1. Add types to `@plyaz/types`
2. Implement with full TypeScript support
3. Add comprehensive tests (aim for 90%+ coverage)
4. Update Confluence documentation
5. Add examples in `examples/` directory if applicable
6. Update internal tracking documents

## Testing

```bash
# Run all tests
pnpm test

# Current test coverage
# Test Files: 53 passed (53)
# Tests: 1,962 passed (1,962)
# Coverage: ~95% lines, ~90% branches
```

## License

ISC © Plyaz