# CDN — Architecture & Internals An engineering reference covering cache behaviour, origin fetching, image transformation, multi-tenancy, and disk management. Includes recommendations and gaps identified from reading both `caruuto/cdn` and `caruuto/cache`. --- ## High-level architecture The service is a Node.js HTTP server that sits between client applications and a remote origin (Supabase HTTP storage). Every inbound request follows the same pipeline: ``` Client Request ↓ Router (lib/controller/index.js) — JWT auth, body parser, range-request middleware ↓ HandlerFactory (lib/handlers/factory.js) — resolves image vs default vs plugin handler — workspace plugins / recipes / routes take priority ↓ Cache lookup (@caruuto/cache via lib/cache/index.js) — cache hit → stream to client, done — cache miss ↓ StorageFactory (lib/storage/factory.js) — selects disk / S3 / HTTP adapter based on URL prefix or config ↓ ImageHandler (lib/handlers/image.js) — Sharp / Jimp transformation pipeline ↓ Cache write (@caruuto/cache) — transformed bytes written to disk — metadata (dimensions, format, errorCode) written to LokiJS db.json ↓ Client Response ``` Request deduplication is handled by `lib/workQueue.js`. Multiple concurrent requests for the same resource share a single processing job; all subscribers are notified on completion. Configuration is managed by `convict` (`config.js`) and supports per-domain overrides loaded from `multiDomain.directory`. Domain is detected from the `Host` header on every request. --- ## 1. Cache key construction ### Key composition `ImageHandler` builds a four-element array (`image.js:237–242`): ```js const cacheKey = [ sha1(JSON.stringify(this.options) + this.req.url), // all transform options + full URL this.req.__domain, // Host header, port stripped this.parsedUrl.cdn.pathname, // path component only this.parsedUrl.cdn.search.slice(1) // raw query string ] ``` `lib/cache/index.js:73–87` normalises this array by SHA1-hashing each element individually and concatenating the results into a single 160-character string (four 40-char SHA1 hashes). That string becomes the file's base name. ### On-disk path structure Inside `@caruuto/cache`, `lib/file.js:225–256` builds the file path: ``` {caching.directory.path}/{folderPath}/{compositeKey}{extension} ``` If `directoryChunkSize` is set (e.g. `4`), the composite key is split into 4-character chunks used as nested subdirectories: ``` ./cache/1073/ab6c/da4b/991c/.../{fullkey}.jpeg ``` If `directoryChunkSize` is not set (the current default in this CDN), all files land flat in the root cache directory. With a large catalogue this produces a directory with tens of thousands of files, which degrades `readdir` and filesystem journal performance on most Linux filesystems (ext4 behaves poorly above ~100k entries per directory). ### Metadata sidecar Metadata (dimensions, format, `errorCode`, `lastModified`) is stored in an in-memory [LokiJS](https://github.com/techfort/LokiJS) database persisted to `{caching.directory.path}/db.json` every 1000 ms. The blob file and its metadata entry are written independently — there is no atomic pairing. ### What the key covers All transformation options are serialised into element 0 (`JSON.stringify(this.options)`). Every distinct combination of `w`, `h`, `format`, `quality`, `crop`, etc. produces a distinct cache key. Transformed variants are stored as separate entries, not alongside the original. Element 1 includes the domain, so the same path requested from two tenants produces different keys. ### Collision and redundancy - **Cross-domain collisions**: not possible — domain is a key element. - **Same-domain collisions**: require identical options blob, URL, pathname, and query string to produce the same 160-char composite. SHA1 collisions are not a practical concern for cache keys. - **Redundancy**: the query string appears in both element 0 (via `req.url`) and element 3. This is harmless but the key is longer than necessary. --- ## 2. Eviction logic ### TTL `config.js:345–350` sets `caching.ttl` (default **3600 s**, domain-overridable). This value is passed to `@caruuto/cache` on every `set()` call. The cache itself stores `lastModified` (write timestamp) in the LokiJS entry; TTL is enforced on read by comparing `(Date.now() - lastModified) / 1000 > ttl` (`cache/lib/file.js:75`). Enforcement is **lazy**: an expired entry is not deleted until something attempts to read it. The entry is then removed from LokiJS and the file is unlinked. If nothing ever reads an expired entry, the file persists on disk indefinitely. ### Background autoFlush `@caruuto/cache` supports an `autoFlush` option (disabled by default). When enabled, a background interval runs `cleanseCache()` (`file.js:294–328`) at `autoFlushInterval` (default 300 s). This iterates all LokiJS entries, checks TTL, and deletes expired files via `fs.unlink()` plus empty directories. This option is not currently wired up by the CDN — `lib/cache/index.js` does not pass `autoFlush: true` to the cache constructor. ### Scheduled full flush `lib/index.js:331–377` starts a `cron.CronJob` if `caching.expireAt` is configured. When the cron fires it calls `cache().delete()` — a **full cache wipe** for the domain (multi-domain mode) or globally. There is no partial or selective eviction. ### 404 caching 404 responses from origin are cached by default (`caching.cache404 = true`, `config.js:352`). Metadata includes `errorCode: 404` (`image.js:343–346`). On retrieval, the handler reads this flag and sets `storageHandler.notFound = true`. If `notFound.images.enabled` changes in config after a 404 is cached, the stale cached entry will continue serving the old fallback behaviour until flushed. ### Gap summary | Gap | Detail | | ----------------------------- | --------------------------------------------------------------------- | | Lazy-only expiry | Expired files accumulate on disk until read or full flush | | `autoFlush` not enabled | Background cleanup is available in `@caruuto/cache` but not activated | | No per-path invalidation | `POST /api/flush` triggers full-domain or global wipe only | | No LRU or size-based eviction | See §6 | --- ## 3. Origin fetching (Supabase / HTTP storage) ### HTTP client Node.js built-in `http` and `https` modules (`storage/http.js:4–5`). No third-party client. ### Timeouts None configured. `http.get()` at `http.js:46–56` does not set a `timeout` option. A slow or hung Supabase endpoint holds a Node.js request slot indefinitely until the OS-level socket timeout fires (typically minutes). Under load this can exhaust the event loop. ### Retry logic None. Any non-redirect, non-200 response is rejected immediately. The rejection carries `err.statusCode` (`http.js:120`). ### Redirect following HTTP 301, 302, and 307 responses are followed recursively up to `http.followRedirects` (default 10, `config.js:694`). There is no cycle detection — a redirect loop A→B→A will silently exhaust all 10 attempts and resolve as a 404. ### Status code handling | Status | Behaviour | | ----------- | ------------------------------------------------------------------------- | | 200 | Body buffered → transformation pipeline | | 301/302/307 | Recursive `get()` with updated URL, max 10 hops | | 404 | `MissingStorage` invoked; fallback image served if configured, else error | | 403 / other | Rejected immediately; `statusCode` attached to error | ### Response headers Not forwarded. The response body is fully buffered to a `Buffer` and converted to a stream. No origin headers (ETag, Cache-Control, Last-Modified) reach the CDN processing layer or the client. ### MissingStorage `lib/storage/missing.js` is invoked when any storage adapter returns not-found. If `notFound.images.enabled` is true, a fallback image is read from disk and returned. Otherwise the error propagates. The `notFound = true` flag on the storage handler controls whether the result is cached (§2). ### Gap summary | Gap | Detail | | --------------------------- | -------------------------------------------------------------- | | No timeout | Hung origin requests block indefinitely | | No retry | Transient Supabase errors return immediately to the client | | No origin header forwarding | ETag, Last-Modified, Cache-Control from Supabase are discarded | | Redirect loop | No cycle detection in redirect following | --- ## 4. Transformation pipeline ### Libraries | Library | Role | | ------------------- | ------------------------------------------------ | | **Sharp** | Primary — JPEG, PNG, WebP, AVIF, TIFF | | **Jimp** | GIF transcoding only (`processGif`) | | **smartcrop-sharp** | Entropy-based smart crop (`resizeStyle=entropy`) | | **node-vibrant** | Colour palette extraction | | **gifwrap** | GIF frame encoding alongside Jimp | ### Supported URL parameters Defined in `IMAGE_PARAMETERS` (`image.js:47–75`): | Parameter | Aliases | Default | Notes | | ------------------ | -------- | --------- | ------------------------------------- | | `format` | `fmt` | — | Target format | | `quality` | `q` | 75 | JPEG quality; maps to PNG compression | | `sharpen` | `sh` | 0 | Minimum effective value: 1 | | `saturate` | `sat` | 1 | Values < 1 produce greyscale | | `width` | `w` | — | | | `height` | `h` | — | | | `ratio` | `rx` | — | Aspect ratio e.g. `16:9` | | `cropX` | `cx` | — | X offset for manual crop | | `cropY` | `cy` | — | Y offset for manual crop | | `crop` | `coords` | — | Explicit bounding box | | `resizeStyle` | `resize` | — | See resize modes below | | `devicePixelRatio` | `dpr` | — | Multiplies width and height | | `gravity` | `g` | `None` | Used with `aspectfill` | | `filter` | `f` | `lanczos` | Resize algorithm | | `trim` | `t` | — | Trim uniform border | | `trimFuzz` | `tf` | — | Tolerance for trim | | `blur` | `b` | — | Gaussian blur radius | | `strip` | `s` | — | Strip metadata (see gap below) | | `rotate` | `r` | — | Rotation angle | | `flip` | `fl` | — | Flip axis | | `progressive` | `pg` | — | Progressive JPEG encoding | Unknown parameters are stripped by `sanitiseOptions` (`image.js:1161–1231`). ### Resize modes | `resizeStyle` | Behaviour | | ------------- | -------------------------------------------------- | | `aspectfit` | Scale to fit within bounds, letterbox | | `aspectfill` | Scale to fill bounds, crop excess; uses `gravity` | | `fill` | Force exact dimensions, distort aspect ratio | | `crop` | Crop using explicit coordinates, optionally resize | | `entropy` | Smart crop via smartcrop-sharp | ### Processing order 1. Sharp reads metadata (dimensions, format, EXIF) — `image.js:288` 2. `getCalculatedDimensions` applies security caps — `image.js:304–307` 3. Resize / crop selected by `resizeStyle` — `image.js:767–970` 4. Blur — `image.js:972` 5. Flip / rotate — `image.js:975–996` 6. Saturation — `image.js:997` 7. Sharpen — `image.js:998` 8. Format conversion and quality encoding — `image.js:1009–1038` 9. Post-processing plugins — `image.js:1050–1078` ### Recipes vs inline parameters `HandlerFactory` merges recipe settings with URL query parameters via `Object.assign({}, recipeSettings, parsedUrl.query)` (`factory.js:135`). URL parameters always override recipe values. Recipes are filesystem JSON files loaded and hot-watched by `Workspace` via Chokidar. ### Security limits `config.js:432–442` defines `security.maxWidth` (default 2048) and `security.maxHeight` (default 1024), enforced as hard caps in `getCalculatedDimensions` with `Math.min()` (`image.js:552–557`). These are not domain-overridable in the default schema. ### Gap summary | Gap | Detail | | -------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | EXIF stripping non-functional | The `strip` parameter exists in `IMAGE_PARAMETERS` but no Sharp `withMetadata(false)` call appears in the processing pipeline. EXIF persists in output regardless of the flag. | | PNG quality mapping undocumented | Default quality (75) is JPEG-centric. PNG compression level is derived from the same value with no documented mapping. | | No per-format quality defaults | A single `quality` value applies to all formats. WebP and AVIF have different quality scales from JPEG. | --- ## 5. Multi-tenant isolation ### Cache key isolation The raw `Host` header value (port stripped) is included as a distinct element in every cache key. Two domains with identical paths and transformation parameters produce different keys. Cache entries cannot cross domains unless an attacker can construct a matching 160-character composite SHA1. ### Disk layout All domains share a single cache directory (`caching.directory.path`, default `./cache/`). There are no per-domain subdirectories. Isolation is at the key-hash level only, not at the filesystem level. ### Domain detection ```js // lib/index.js:283 const domain = req.headers.host.split(':')[0] req.__domain = domain ``` The `Host` header is used without allowlist validation. Any header value becomes the domain identifier used in cache key derivation and config lookups. ### Per-domain configuration `config.js:822–857` (`loadDomainConfigs`) scans `multiDomain.directory` for subdirectories each containing a `config/config.{env}.json`. Domain-overridable settings include TTL, storage paths, logging, caching flags, and auth credentials. ### Risk: Host header injection An unvalidated `Host` header is used as the domain key. A crafted value such as `../../etc/passwd` reaches `@caruuto/cache` as a path component. The `@caruuto/cache` `getCachePath()` function uses the domain in directory construction (`file.js:225–256`). Path traversal is a real risk. An allowlist of permitted domain values should be enforced before `req.__domain` is set. --- ## 6. Disk usage management There is currently no disk usage management in either `caruuto/cdn` or `caruuto/cache`. | Capability | Status | | -------------------------- | -------------------------------------------------------------------- | | Disk quota / byte budget | Not implemented | | Entry count limit | Not implemented | | LRU eviction | Not implemented | | Oldest-first pruning | Not implemented | | Disk-full error handling | Not implemented — write failure is silent | | Disk usage monitoring | Not implemented | | Per-domain disk allocation | Not implemented | | Directory sharding | Available in `@caruuto/cache` (`directoryChunkSize`) but not enabled | | Background TTL sweep | Available in `@caruuto/cache` (`autoFlush`) but not enabled | When disk fills, `fs.createWriteStream()` in `@caruuto/cache/lib/file.js:143` will emit an `error` event. The promise is rejected. The CDN's `cache.set()` call in `image.js` does not `await` the cache write (the `wait = false` flag at `image.js:40`), so the write failure is silently dropped — the client still receives a response, but nothing is cached. The LokiJS metadata entry is inserted **before** the stream completes (`file.js:156–161`). A disk-full write failure therefore leaves an orphaned LokiJS entry pointing to a missing or partial file. On the next read, `get()` detects the missing file and removes the orphaned entry (`file.js:101–106`), but the partial file is not cleaned up. **Writes are not atomic.** `@caruuto/cache` uses `fs.createWriteStream()` with no temp-file-then-rename pattern. A crash or disk-full condition mid-write leaves a corrupt file on disk. --- ## 7. Redis — current state Redis is **configured but disabled by default**. ``` caching.redis.enabled (default: false, env: CACHE_ENABLE_REDIS) caching.redis.host caching.redis.port caching.redis.password ``` When enabled via `@caruuto/cache`, Redis stores **both blob bytes and metadata**: - Blob: stored as a Redis STRING using `redis-wstream` for streaming writes - Metadata: stored as a separate STRING key with format `___${key}___`, serialised as JSON - TTL: Redis `EXPIRE` is set on both keys after write (`redis.js:270–271`, `312–313`) Redis TTL is enforced server-side by Redis itself — there is no lazy-read check in the Redis adapter. This is stricter and more reliable than the file adapter's lazy expiry. On Redis connection failure, `@caruuto/cache` automatically falls back to file cache after 5 retry attempts with exponential backoff (`redis.js:56`), emitting a `fail` event which triggers the fallback (`index.js:168`). --- ## 8. Stale-while-revalidate ### What it is Stale-while-revalidate (SWR) is a caching pattern where: 1. A request arrives for a cache entry whose TTL has expired 2. The **stale cached response is served immediately** to the client 3. A **background revalidation** fetches a fresh copy from origin and updates the cache 4. Future requests get the fresh copy The client never waits for the origin fetch. Latency remains consistently low. The tradeoff is that one request cycle sees content that may be up to `TTL + revalidation-time` old. ### Why it fits this CDN well Re-uploaded images in this system receive a timestamp suffix, making image URLs effectively immutable per version. This means content at a given URL genuinely does not change unless a new URL is issued. For such URLs, SWR adds no correctness risk — the stale entry and the fresh entry are identical, so serving stale is always safe. For mutable paths (if any exist), SWR introduces a bounded staleness window rather than unbounded staleness (which occurs today when `autoFlush` is off and nothing reads an expired key to trigger lazy deletion). ### Current blocking behaviour At present, `@caruuto/cache/lib/file.js:75–79` rejects the `get()` promise when TTL is exceeded: ```js if ((Date.now() - lastModified) / 1000 > ttl) { db.findAndRemove({ $key: key }) return reject(new Error('The specified key has expired')) } ``` The CDN treats this as a cache miss and performs a **synchronous, blocking origin fetch and re-transform** before responding. The client waits for the full round-trip. ### Implementation path SWR requires two new concepts: a **soft TTL** (when to start revalidating) and a **grace period** (how long to keep serving stale while the background fetch runs). This can be layered on top of the existing architecture with contained changes. **Step 1 — Extend `@caruuto/cache` file adapter** Add a `getStale(key)` method that returns the cached entry even if TTL is exceeded, along with a staleness flag: ```js // @caruuto/cache lib/file.js async getStale(key) { const entry = db.findOne({ $key: key }) if (!entry) return null const ageSeconds = (Date.now() - entry.lastModified) / 1000 return { stream: fs.createReadStream(entry.path), metadata: entry.metadata, age: ageSeconds, stale: ageSeconds > this.ttl } } ``` **Step 2 — Add SWR config** ```js // config.js — inside caching block staleWhileRevalidate: { doc: 'Seconds beyond TTL during which stale content may be served while revalidating in background. 0 disables.', format: Number, default: 0, allowDomainOverride: true } ``` **Step 3 — Update `lib/cache/index.js`** Expose `getStale()` on the CDN cache wrapper and pass through `staleWhileRevalidate` to the file adapter constructor. **Step 4 — Update `ImageHandler`** In `image.js`, after a normal `cache.get()` rejection, call `cache.getStale()`: ```js async get(cacheKey) { try { return await this.cache.get(cacheKey) } catch (err) { const swr = config.get('caching.staleWhileRevalidate', this.req.__domain) if (swr > 0) { const stale = await this.cache.getStale(cacheKey) if (stale && stale.stale && stale.age < (ttl + swr)) { // serve stale immediately this.setHeaders(stale.metadata) res.setHeader('X-Cache-Status', 'STALE') stale.stream.pipe(res) // revalidate in background — fire and forget this.revalidate(cacheKey).catch(err => logger.error(err)) return } } throw err // fall through to normal origin fetch } } ``` The `revalidate()` method performs the origin fetch and transform in the background and calls `cache.set()` when done. The `WorkQueue` already deduplicates concurrent revalidations for the same resource — pass the cache key as the queue key to prevent a flood of simultaneous background fetches for a popular asset whose TTL just expired. **Step 5 — Response headers** Add `X-Cache-Status: HIT | STALE | MISS` to all responses. This is the minimum needed to observe SWR behaviour in production. Optionally, add `Age: {seconds}` for CDN clients that understand it. ### Recommended defaults | Config | Recommended value | Rationale | | -------------------------------- | --------------------- | ----------------------------------------------------------------------- | | `caching.ttl` | 86400 (24 h) | Immutable URLs — long TTL is safe | | `caching.staleWhileRevalidate` | 3600 (1 h) | One hour of stale-serving grace before a blocking refetch | | `autoFlush` in cache constructor | `true`, interval 3600 | Background sweep clears genuinely stale entries on a reasonable cadence | --- ## 9. Recommendations and paths forward Listed roughly in priority order. ### 9.1 Enable `autoFlush` in the cache constructor (quick win) `@caruuto/cache` already implements background TTL sweep via `autoFlush` and `autoFlushInterval`. The CDN does not activate it. Passing `autoFlush: true` to the cache constructor in `lib/cache/index.js` gives proactive disk cleanup with no code changes to the cache library. ```js // lib/cache/index.js — constructor options { autoFlush: true, autoFlushInterval: config.get('caching.ttl', domain) } ``` ### 9.2 Enable `directoryChunkSize` (quick win) Large flat directories degrade filesystem performance on most Linux filesystems. Passing `directoryChunkSize: 4` to the cache constructor shards files into nested 4-char subdirectories (e.g. `./cache/1073/ab6c/…`). This is a one-line change in `lib/cache/index.js`. Existing cached files will not be found after the change — a one-time cache flush is required. ### 9.3 Add a request timeout to HTTP storage ```js // lib/storage/http.js — after http.get() req.setTimeout(config.get('http.timeout', domain), () => { req.destroy(new Error('Origin request timed out')) }) ``` Add `http.timeout` to `config.js` with a sensible default (e.g. 10000 ms). Without this, a single slow Supabase response blocks a Node.js request slot indefinitely. ### 9.4 Add retry with exponential backoff to HTTP storage A single retry on 5xx or network error would silently recover the majority of transient Supabase failures. A simple implementation using a recursive call with a delay counter is sufficient — no library needed. Cap at 2 retries with 500 ms and 1500 ms delays. ### 9.5 Implement stale-while-revalidate (§8) Given that image URLs are version-immutable, SWR eliminates the latency cost of TTL expiry entirely. Clients always get a fast response; origin re-fetches happen in the background. The implementation is described in §8. ### 9.6 Add disk usage monitoring At minimum, a periodic check of cache directory size should log a warning and emit a metric when usage crosses a configurable threshold. This can run inside the existing `startFrequencyCache` cron infrastructure: ```js // Rough sketch — inside lib/index.js const { execFile } = require('child_process') execFile('du', ['-sb', cachePath], (err, stdout) => { const bytes = parseInt(stdout) if (bytes > config.get('caching.directory.maxBytes')) { logger.warn({ bytes }, 'Cache directory exceeds size threshold') } }) ``` A hard quota with LRU eviction requires more invasive changes to `@caruuto/cache` (tracking access time alongside write time in LokiJS), but the monitoring step is valuable independently. ### 9.7 Add an allowlist for `Host` header / domain values The `Host` header is currently used without validation as a filesystem path component via the cache key. Add an allowlist check in `lib/index.js` before setting `req.__domain`: ```js const allowedDomains = config.get('multiDomain.allowedDomains') // string[] const domain = req.headers.host.split(':')[0] if (allowedDomains.length && !allowedDomains.includes(domain)) { return res.status(400).end() } req.__domain = domain ``` ### 9.8 Fix EXIF stripping The `strip` URL parameter is parsed and included in `this.options` but no corresponding Sharp call exists. The fix is a one-liner in the processing pipeline: ```js // image.js — after sharpen, before format conversion if (this.options.strip) { sharpImage = sharpImage.withMetadata(false) } ``` Without this, all Sharp-processed images leak EXIF data (GPS coordinates, device info, etc.) to clients regardless of the `strip` flag. ### 9.9 Redis hybrid: metadata in Redis, bytes on disk The current Redis mode stores full image bytes in Redis. A more efficient hybrid would store only metadata (dimensions, format, TTL, `lastModified`) in Redis while keeping image bytes on disk. This allows the hot-path lookup (is this cached? what are its headers?) to be a fast Redis read, while bulk storage remains on disk. This requires: 1. A new `setMetadata(key, metadata)` / `getMetadata(key)` pair in `@caruuto/cache` that routes to Redis regardless of blob storage backend 2. `ImageHandler` to call `setMetadata` separately from the blob write 3. The cache hit path to check Redis for metadata before attempting a disk stream This change pays off once Redis is reliably available and the number of cache entries is large enough that disk stat calls become a measurable latency contributor. ### 9.10 Make cache writes atomic `@caruuto/cache` writes directly to the final file path without a temp-file-then-rename pattern. A crash or disk-full condition mid-write leaves a corrupt file. The fix is in `@caruuto/cache/lib/file.js`: ```js // Write to a temp path, rename on completion const tmpPath = `${finalPath}.tmp` const stream = fs.createWriteStream(tmpPath) stream.on('finish', () => fs.rename(tmpPath, finalPath, resolve)) stream.on('error', reject) ``` Combined with inserting the LokiJS entry **after** the rename (not before the write), this makes cache writes crash-safe. --- ## Appendix: Configuration reference (cache-related) | Key | Default | Domain override | Notes | | --------------------------- | --------- | --------------- | --------------------------------- | | `caching.directory.enabled` | `true` | No | Enable disk cache | | `caching.directory.path` | `./cache` | No | Shared across all domains | | `caching.ttl` | `3600` | Yes | Seconds; enforced lazily on read | | `caching.expireAt` | `null` | Yes | Cron pattern for full-cache flush | | `caching.cache404` | `true` | Yes | Cache 404 responses from origin | | `caching.redis.enabled` | `false` | No | Enable Redis backend | | `caching.redis.host` | — | No | | | `caching.redis.port` | — | No | | | `http.followRedirects` | `10` | Yes | Max redirect hops from origin | | `security.maxWidth` | `2048` | No | Hard cap on output width | | `security.maxHeight` | `1024` | No | Hard cap on output height | | `notFound.images.enabled` | — | Yes | Serve fallback image on 404 |