# database.scaling > Transparent read-replica routing with read-your-writes (RYW) consistency, multi-instance RYW via Redis, region-aware replica selection, and the per-dialect adapter surface. --- ## Read replicas _Transparent read-replica routing with read-your-writes (RYW) consistency, multi-instance RYW via Redis, region-aware replica selection, and the per-dialect adapter surface._ The framework's `DataStore` can wrap a primary connection + a list of read replicas behind one transparent surface. Writes go to the primary. Reads query to a replica chosen by power-of-two-choices, EXCEPT when the same subject just wrote — RYW (read-your-writes) pins their next reads to the primary for a TTL window so they never see stale data after their own changes. This is **completely opt-in**. Without `DB_REPLICA_URLS` set, the framework returns the primary store directly — no `ReplicatedDataStore` wrapper, no per-call routing overhead, no cost. ## Quick start Add replica URLs: ```sh DB_PRIMARY_URL=postgresql://app:app@primary.db.acme.com/app DB_REPLICA_URLS=postgresql://app:app@replica.db.acme.com/app voltro start ``` Verify in the boot log: ``` [voltro:dev] read replicas: 1 configured, ryw policy 'fallback', dialect=postgres, region=unknown ``` That's it. Reads now flow to the replica unless RYW pins them back. Writes always go to primary. ## Replica pools take the primary's connection settings A replica is a **connection pool in the same process**, and it is configured by the same environment variables the primary reads — `DB_MAX_CONNECTIONS`, `PG_SSL`, `DB_SCHEMA`, `DB_STATEMENT_TIMEOUT_MS`, `DB_ACQUIRE_TIMEOUT_MS`. The replica URL decides only WHICH server it talks to. That matters for two reasons an operator has to plan for: - **Connection budget.** `DB_REPLICA_URLS` with two entries means this process opens **three** pools of `DB_MAX_CONNECTIONS` each, not one. The boot line says so: ``` db pool: max=10 per replica (DB_MAX_CONNECTIONS) → max × 3 = 30 per pod × 4 replicas = up to 120 connections. This process opens 3 pools of that size — 1 primary + 2 read replica(s) (DB_REPLICA_URLS), each with its own DB_MAX_CONNECTIONS. ``` - **TLS.** `PG_SSL=require` now applies to replica connections as well. A `?sslmode=require` in the replica URL still works and still wins over the driver default, but an explicit `PG_SSL` outranks it — so the TLS decision is made once, for every connection the process opens. ## Read-your-writes (RYW) After a mutation commits, the subject who wrote is **pinned to primary** for the next 30 seconds (configurable). This guarantees they see their own change on the next read, even if the replica hasn't caught up yet. How it's implemented under the hood per dialect: | Dialect | Capture surface | Probe surface | |---|---|---| | Postgres | `pg_current_wal_lsn()` | `pg_last_wal_replay_lsn()` | | MySQL / MariaDB | `@@global.gtid_executed` (MySQL) / `@@global.gtid_current_pos` (MariaDB) | same, on the replica | | MSSQL | `end_of_log_lsn` from `sys.dm_hadr_database_replica_states` (AG only) | `last_hardened_lsn` from the same DMV | | SQLite | n/a (single-process — replicas not applicable) | The framework calls `capturePrimaryPosition()` immediately after every successful write, stores it keyed by the subject, and consults the cache on every subsequent read by that subject. No `position` → fall through to the replica selector. ### RYW policies Three modes via `RYW_POLICY` env or `rywPolicy:` in config: | Policy | Behaviour | |---|---| | `fallback` (default) | Pin to primary for the TTL window. Simple, low-tail-latency, no replica wait. | | `wait` | Poll the replica until it catches up (max `rywMaxWaitMs`). Higher consistency at cost of latency. | | `off` | No RYW — every read goes to a replica regardless of write history. For analytics workloads where staleness is OK. | For most apps, `fallback` is correct: it prevents the "I just edited this, why is it gone" UX without paying the replica-poll cost. Reach for `wait` only when stronger consistency outweighs latency. ## Replica selection By default the framework uses **power-of-two-choices** (P2C): pick two replicas at random, send the read to the one with fewer in-flight queries. Standard load-balancer trick — robust under skew, no global LRU/LFU state needed. ### Region-aware selection (multi-region) When replicas span regions, set `DB_REPLICA_REGIONS` to a comma-separated list aligned with `DB_REPLICA_URLS`: ```sh DB_REPLICA_URLS=postgres://repl-use1.acme/,postgres://repl-euw1.acme/ DB_REPLICA_REGIONS=us-east-1,eu-west-1 ``` The framework auto-detects the current process's region from these env vars (in priority order): 1. `VOLTRO_REGION` — explicit override 2. `AWS_REGION` — AWS ECS / Lambda / EC2 3. `FLY_REGION` — Fly.io 4. `RAILWAY_REGION` — Railway 5. `GCP_REGION` / `CLOUD_RUN_REGION` — Google Cloud The selector then wraps P2C with a locality preference: **same-region replicas are tried first**, other-region replicas only when same-region is empty or unavailable. Boot log confirms: ``` [voltro:dev] read replicas: 2 configured, ryw policy 'fallback', dialect=postgres, region=us-east-1, replica-locality=on ``` On a wrong-region read, you pay 80-150ms cross-region latency. On a same-region read, 1-2ms. The selector does its best to keep you in the second bucket. If `DB_REPLICA_REGIONS` count mismatches `DB_REPLICA_URLS` count, the framework warns + disables locality (degrades to plain P2C across all replicas — correct but slower). ## Multi-instance RYW: Redis The default RYW store is a per-process `Map`. Multi-instance deployments (k8s with `replicas: 3`, multi-pod ECS, etc.) need a shared store, otherwise a write on instance A doesn't pin reads on instance B. Wire Redis: ```sh RYW_STORE=redis REDIS_URL=redis://host:6379 voltro start ``` The Redis store keeps a local mirror per instance, kept warm via pub/sub — reads stay sync (`Map.get`), writes propagate to peers within one Redis round-trip. Sub-millisecond on localhost, 1-2ms across a VPC. What you get: - Cross-instance coherent RYW: A's write pins B's reads for the TTL. - Server-side TTL via PSETEX — entries expire automatically even when no `get` triggers lazy eviction. - Self-publish loopback filtering: an instance ignores its own pub/sub broadcasts to avoid double-applying. What you don't get: - Strong consistency. Between the moment A commits + the moment B receives the pub/sub broadcast, B's reads could still hit a stale replica. The window is one Redis round-trip — small but nonzero. For strict-consistency workloads use `RYW_POLICY=wait` on top of Redis store. ## Performance Routing overhead is sub-microsecond per call. The micro-bench in `packages/runtime/src/replicatedDataStore.perf.test.ts` measures ~10µs per routing decision including the AsyncLocalStorage context push/pop. The actual SQL query dominates wall-clock by orders of magnitude. If you see latency regressions, run that test in your fork: ```sh pnpm -F @voltro/runtime test -- replicatedDataStore.perf ``` It logs the per-call cost and fails if you've accidentally turned the routing path into an O(n²) walk or added an unnecessary allocation per query. ## Dialect support matrix | Dialect | Routing | RYW | Region-aware | E2E validated | |---|---|---|---|---| | Postgres | ✓ | streaming-replication LSN | ✓ | docker-compose (Phase 4 / C10) | | MySQL 8+ | ✓ | GTID | ✓ | docker-compose (Phase 5) | | MariaDB 10.6+ | ✓ | GTID (via mysql adapter) | ✓ | shares MySQL adapter | | MSSQL 2019+ | ✓ AG only | AG LSN | ✓ | adapter validated, AG cluster setup is operator-managed | | SQLite | n/a | n/a | n/a | single-process, no replication possible | MSSQL routing requires Always-On Availability Group cluster setup (certs + endpoints + listener + Pacemaker on Linux OR Windows Failover Clustering). That's a deployment-grade infrastructure setup, not a framework concern; the adapter validates against any SQL Server but real replica routing needs AG. Without AG, both capture + probe return the zero-LSN sentinel and the framework gracefully degrades to "treat every read like a primary read" (safe, just slower than it could be). ## Anti-patterns - **Don't run `RYW_STORE=redis` without `REDIS_URL`** — the framework warns + falls back to memory store. Multi-instance deployments silently lose RYW coherence. - **Don't set `DB_REPLICA_REGIONS` without `DB_REPLICA_URLS`** — the region map is paired by order with the URL list. Without URLs the region list has nothing to bind to and is ignored. - **Don't use `wait` policy with high write-rate subjects** — every read after a write polls the replica with a sub-second timeout. The tail latency is bounded by `rywMaxWaitMs`, but for hot subjects this can starve. - **Don't manually pin reads to a specific replica from handler code.** The framework's routing layer is the single decision point — if you bypass it, you lose RYW correctness + observability counters. --- ## Multi-replica reactivity _Cross-replica change fan-out behind a load balancer — native LISTEN/NOTIFY (postgres) and binlog CDC (mysql + mariadb), plus @voltro/plugin-broadcast (Redis / NATS) to close the gap for every other dialect._ Behind a load balancer, each live client WebSocket lives on exactly **one** replica. Replica A holds clients 1–100, replica B holds 101–200, and so on. A mutation runs on whichever replica received the request, and inline-emits the change to **that replica's** local subscribers. The other replicas never learn of the write — so their clients go stale until they refetch. This page is about closing that gap: making a write on **any** replica reach the live subscriptions on **every** replica. ## The three tiers How a change crosses replica boundaries depends on the dialect: | Tier | Dialects | Mechanism | Cross-replica? | |---|---|---|---| | Native DB fan-out | postgres | `LISTEN/NOTIFY` — the database IS the bus | ✓ | | Native DB fan-out | mysql · mariadb | binlog CDC (ROW format) — every replica tails the binlog | ✓ | | Pub/sub bus | mssql · cockroachdb · planetscale · azure-sql | `@voltro/plugin-broadcast` (Redis / NATS) | ✓ (when wired) | | Single-instance | sqlite · `memory` | in-process only | n/a (single process by design) | Postgres (`LISTEN/NOTIFY`) and MySQL/MariaDB (ROW-format binlog CDC) have a native cross-instance path: every replica learns of a write directly from the database. For **the other SQL dialects there is no native cross-instance path** — reactivity silently degrades to single-instance. `@voltro/plugin-broadcast` closes that with a pub/sub message bus. (PlanetScale is MySQL-based but Vitess doesn't expose a raw binlog to external readers, so it stays on the bus tier.) ## @voltro/plugin-broadcast A first-class, provider-pluggable plugin that fans out app-mutation change events to every replica over a pub/sub broker. Two backends ship: **Redis** and **NATS**. ```ts // app.config.ts import { broadcastPlugin } from '@voltro/plugin-broadcast' export default { type: 'api' as const, name: 'api', plugins: [ broadcastPlugin(), // reads BROADCAST_URL / BROADCAST_PROVIDER / REDIS_URL from env ], } ``` ```sh # Redis (RESP pub/sub) BROADCAST_URL=redis://localhost:6379 # or NATS BROADCAST_URL=nats://localhost:4222 # explicit provider override (otherwise inferred from the URL scheme) BROADCAST_PROVIDER=redis ``` If `BROADCAST_URL` is unset the plugin falls back to an in-process **memory** provider — useful in tests, but **single-process**: a real cross-replica bus needs a broker URL. The boot banner warns when this happens. You can also pass a pre-built provider or an explicit name: ```ts import { broadcastPlugin, redisProvider } from '@voltro/plugin-broadcast' broadcastPlugin({ provider: redisProvider({ url: process.env.REDIS_URL! }) }) broadcastPlugin({ provider: 'nats', url: 'nats://nats:4222' }) ``` `ioredis` and `nats` are **optional dependencies** — installed only for the backend you use. The plugin dynamically imports the driver, so an app on `memory` never pulls either. ## Mechanism — additive, not a replacement The bus is **additive** to the inline emit path. Local reactivity must survive a broker outage: 1. A local change (via `store.onChange`) publishes `{ origin: replicaId, event }` to the channel `voltro:changes`. 2. Every replica subscribes. On a message whose `origin` is **not** this replica, it injects the change event into the local store emitter — the same `injectExternalChange` seam the postgres `LISTEN/NOTIFY` consumer uses. 3. The writer's **own** broadcast is skipped (it already inline-emitted locally) — so there is no double-emit and no dedup table. Because the inline path is never removed, a broker outage degrades **cross-replica** fan-out only — local reactivity keeps working, and the framework logs a warning. The bus reconnects when the broker returns. ## The honest caveat — app-mutation changes only The bus carries changes that flow through **`ctx.store`** (the framework's mutation path). It does **not** capture **out-of-band DB writes** — a `psql` session, a cron job, or a second service writing the same database directly. Those changes never hit `store.onChange`, so they never reach the bus. Only two mechanisms observe out-of-band writes: - **postgres `LISTEN/NOTIFY`** — the DB itself fires on any committed change (the framework's triggers fan out every write, regardless of who made it). - **mysql / mariadb binlog CDC** — the ROW-format binlog records every committed row change; both engines share one reader. If your app shares its database with other writers and needs them to drive reactivity, choose postgres, mysql, or mariadb. If all writes go through the Voltro app (the common case), the broadcast bus is the right tool for non-native dialects. ## What about polling a changelog table? **Polling is not built.** It's documented here only as the absolute last resort. A changelog table that every replica polls (`SELECT … WHERE seq > :last`) would observe out-of-band writes on any dialect — but at a cost the pub/sub bus avoids entirely: ~150 ms latency (poll interval) vs ~1 ms, constant DB load from every replica on every tick, and a changelog table to vacuum. Pub/sub beats it on every axis, is dialect-agnostic, and Voltro already runs Redis pub/sub for read-your-writes consistency. If you genuinely need out-of-band-write reactivity on a dialect that can't do `LISTEN/NOTIFY` or binlog, that's the signal to move to postgres — not to bolt on polling. ## Boot banner `voltro dev` / `voltro start` prints the resolved reactivity tier so `voltro logs --tail 50` answers "how do writes cross replicas here" without reading source: ``` [voltro:dev] reactivity: cross-instance via native LISTEN/NOTIFY (postgres) ``` ``` [voltro:dev] reactivity: cross-instance via @voltro/plugin-broadcast (redis) for dialect 'planetscale' ``` ``` [voltro:dev] reactivity: cross-instance fan-out is OFF for dialect 'mssql'. A write on one replica will NOT reach clients on other replicas. Add @voltro/plugin-broadcast (Redis / NATS) to close the gap… ``` When BOTH a native path and the broadcast plugin are wired (e.g. postgres + broadcast), both stay active — the own-origin skip dedups, so there's no double-emit. The native path is the primary; the bus is harmless redundancy. ## sqlite and the memory store behind replicas Neither dialect has a cross-instance path, and neither can get one: a local database file and an in-process store are, by construction, **this process's**. Putting several replicas in front of one is not a reactivity gap — the replicas do not share a database at all, so each one is also reading its own data. On a laptop that is the correct, normal configuration, which is why the framework said nothing about it for a long time. It now warns when it can see evidence of an **orchestrator**: ``` [voltro:serve] this app runs on 'sqlite' — a local database file — but POD_IP is set (Kubernetes), which means several replicas. Neither dialect has ANY cross-instance change capture: a client connected to replica A never sees a write made on replica B, for every table, and each replica is also reading its OWN data… ``` The evidence is one of `REPLICA_COUNT > 1`, `KUBERNETES_SERVICE_HOST`, `POD_IP`, `POD_NAME`, `FLY_ALLOC_ID`, `FLY_MACHINE_ID`, `ECS_CONTAINER_METADATA_URI[_V4]`, `K_REVISION`, `CONTAINER_APP_REPLICA_NAME` or `RENDER_INSTANCE_ID`. `HOSTNAME`, `NODE_ENV` and `PORT` are deliberately **not** evidence — every single-instance container sets those too, and a warning that fires on a laptop gets filtered out before it reaches the deployment where it is true. Two ways to silence it, and they are not equivalent: - `REPLICA_COUNT=1` — you are telling the framework there is exactly one process. This is the honest one, and it is the only positive evidence *against* that exists, so it outranks every platform signal. - Declaring `@voltro/plugin-broadcast` with a real provider — not because a bus fixes it (the replicas still have separate databases), but because declaring one means you have already thought about the question. The real fix is postgres or mysql/mariadb: a shared database with a native cross-instance change path. ## Where it lives in the codebase - `voltro/packages/plugin-broadcast` — the plugin, the `BroadcastProvider` contract, the redis / nats / memory providers, `attachBroadcastBus`. - `voltro/packages/database/src/dataStore.ts` — `DataStore.injectExternalChange` (the cross-instance seam). - `voltro/packages/cli/src/dev.ts` `wireBroadcastBus()` — tier selection + boot banner. ## See also - [SQL dialects](./dialects) — the per-dialect parity table (CDC row). - [Read replicas](./read-replicas) — read routing + read-your-writes consistency (a different axis: which replica serves a *read*, not how a *write* fans out).