# @walkeros/core ## 4.2.1 ### Patch Changes - 5cbcd23: Simulation step results gain an optional `mappingKey` field reporting the entity-action key of the mapping rule a destination matched during simulation. The field is additive and present only when a rule matched. - 31c6858: `getId` now draws from the platform's cryptographic random source (`crypto.getRandomValues`) when available, with unbiased character sampling and a `Math.random` fallback. Session and device ids generated by the session source are now longer for a much wider collision margin. - d1b41ca: The collector now stamps a run-scoped trace id (`event.source.trace`) and a per-run sequence number (`event.source.count`) onto every event, minted fresh on each `run`. These group all events of a page load or run and are preserved unchanged when events are forwarded from web to server, giving a stable correlation id across the pipeline. Adds `getTraceId`, and `getSpanId` now uses the cryptographic random source. - 0a8a08b: Add an optional `async` option to the source config (`Source.Config.async`) for respond-first acknowledgement on response-producing server sources. The express source now reads `config.async` (default `true`): a 2xx response means the event was accepted, not yet delivered. - 8afb7cc: Add an optional `reportError` callback to the step context so any source, transformer, store, or destination can report an out-of-band error (for example from an SDK's event emitter) into the pipeline's failure handling. Add an optional per-destination `breaker` config to skip a destination after repeated transport failures. ## 4.2.0 ### Minor Changes - e8f6909: Add an optional, strictly-typed `config.credentials` field to destinations, stores, and sources. Service-account credentials now configure under `config.credentials`, validated per package and resolved from `$env`. The package-specific `settings.credentials` still works but is deprecated, so move credentials to `config.credentials`. The raw `settings.` passthrough (e.g. `settings.bigquery`) is unchanged. - 654ba38: Trace telemetry now activates at runtime by polling the deployment's trace window, so web and server flows start and stop full-payload recording without a redeploy. A future trace window upgrades a flow to full inbound and outbound recording, and a null or past window reverts to the flow's `observe` baseline and self-expires. - 6a72a32: Source simulation gains a `collector` step that runs the real collector enrichment and returns the fully enriched event. Transformer simulation now accepts an optional raw `ingest` so request decoders like GA4 can be tested standalone by supplying a `url`. The `flow_simulate` MCP tool accepts the new collector step and the transformer `ingest` input. - 23d4b86: New `@walkeros/transformer-validate` transformer validates events against JSON Schema contracts. It runs in both web and server flows, supports strict and pass modes, and writes the verdict and error list to configurable paths so you can gate or observe event quality. The declarative per-step `validate` field on sources, transformers, and destinations is removed. Define event shapes in the top-level `contract` and enforce them at runtime by adding a `transformer-validate` step that references them via `$contract.`; `format: true` still checks an event is a valid `WalkerOS.PartialEvent`. Design-time validation now checks step examples against the resolved contract. ### Patch Changes - 76d32c1: Batched destination delivery now reports failures. A batch push that fails (including BigQuery row errors) is routed to the dead-letter buffer and counted as failed instead of being silently dropped, and graceful shutdown waits for in-flight batches to finish. Also fixes a shutdown timer that could delay process exit, and makes a zero millisecond batch wait (`batch: 0`) correctly enable batching. - 908d6f0: Promote chain, route shape, and reference scanner helpers to the public surface so app and tooling can resolve transformer chains, probe route shapes, and discover `$flow.` references without reaching into internal modules: - collector: re-export `walkChain` (resolve a transformer chain start into the ordered step IDs) and `extractTransformerNextMap` (read static next-links from a `Transformer.Transformers` map). - core: re-export `isRouteArray` and `isRouteConfigEntry` (the canonical shape probes for `Transformer.Route`) and add `scanFlowRefs(value, into?)`, which walks any value (string, object, array) and returns every `$flow.` reference found, including refs nested inside `$code:` snippets. - f4a9013: Fix `$flow` reference scanning to match the resolver's name grammar, so names with leading digits or hyphens no longer produce false-positive references. - d65bbde: `FlowState` records can now carry an optional `platform` field (`'web' | 'server'`) identifying the runtime that produced the state. Observers can use it alongside `flowId` to correlate telemetry across web and server runtimes of the same flow. - e8f6909: Documentation fix: server source `config.ingest` examples now use the `map` operator with direct request field paths instead of a bare object. A bare object like `{ url: 'req.url' }` is silently inert, so the ingest stayed empty and downstream `ingest.*` fields never resolved. Affects package hints, READMEs, the core source type docs, and the bundled CLI example. - c27d3c1: Request caching now persists structured HTTP responses, including binary bodies (`Buffer`, `Uint8Array`, `ArrayBuffer`), to byte/string store backends (filesystem, S3, GCS, in-memory) and honors TTL. Previously, caching a response could crash the process or never populate, and entries never expired. Cached values now round-trip safely (binary bodies decode back as a `Buffer`) and expire correctly instead of serving stale content after a redeploy. - 3eb2467: Add a declarative `state` block for `get`/`set` against a store, replacing `$code:` for simple fetch and stash. Available on source, transformer, and destination steps; defaults to an in-memory store. - 5b1a134: Stores now use one structured value type with binary (`Uint8Array`) as a first-class leaf, serialized by a shared codec. A new `file: true` store option serves byte-exact assets such as walker.js (default is structured key-value). TTL is owned by the cache layer, not the store. Sheets is structured-only and rejects `file: true`. - 18c9469: Flow validation now scopes the "web flows cannot reference a managed secret" check per flow. A multi-flow config where a web flow forwards to a server flow that holds a `$secret.` reference now validates cleanly, instead of the server flow's secret being wrongly flagged against the web flow. ## 4.1.2 ## 4.1.1 ### Patch Changes - b0279ee: Rename the contract inheritance key from `extends` to `extend` for consistency with the rest of the flow config vocabulary. Contracts that inherit from another named contract now use `"extend": ""`. - b0279ee: Add `extend` and `remove` to mapping rules. `extend` deep-merges a partial rule onto a package-shipped default (a `null` value clears an inherited field); `remove` strips fields from the produced payload. Rules without either keyword keep the existing replace behavior. - 0b7f494: The collector exposes `observers: Set` so any subscriber can watch every step of the pipeline. Each source, transformer, destination, and store call emits a `FlowState` record with timings, mapping match, consent state, and skip reasons. `createTelemetryObserver` from `@walkeros/core` batches emissions to an HTTP endpoint, and the CLI runtime picks up the `traceUntil` flag from its heartbeat so trace mode toggles take effect without a redeploy. ## 4.1.0 ### Minor Changes - e155ff8: Cache reads through `checkCache` are now correct against async stores (filesystem, Redis, any store with an async `get`). Previously a custom async store could silently miss the cache. `checkCache` returns a Promise. External callers must add `await`. - e155ff8: Collector and destination buffers are now size-bounded with FIFO drop-oldest eviction. Defaults: collector `queueMax: 1000`, destination `queueMax: 1000`, destination `dlqMax: 100`. Set either knob to override per scope. Drop counts surface in `collector.status.dropped` and `collector.status.destinations[id]`. - b276173: **Breaking:** `code: ""` is no longer accepted on any step. Replace with `import: ""` alongside `package`. **New:** Every step (source, transformer, destination, store) accepts `import?: string`. With `package`, it selects a named export. `package` alone still loads the default export. Inline code stays `code: { push, type?, init? }`. Empty steps are valid no-ops. `flow_validate` and the CLI bundler raise `OBSOLETE_CODE_STRING` on the legacy shape with a precise rename hint. - dd9f5ad: Pass-through transformer steps + closed-schema validation. **Validation:** `validateTransformerEntry` in `@walkeros/core` is now the single source of truth. Bundler, `flow_validate`, and collector runtime all delegate. Closed schema: unknown top-level keys are errors. `code` + `package` together is a `CONFLICT`. **Pass-through steps:** A transformer entry with no `code` and no `package` is valid; the collector synthesizes its push. Three variants: - before/next chain only (named hop) - cache only (e.g. dedup) - mapping only (event-to-event transform via `Mapping.Config`) **Mapping at the transformer position:** new `mapping?: Mapping.Config` field on `Transformer.Config` / `InitTransformer`. Same shape as `Destination.Config.mapping`, event-to-event semantic. `data` / `silent` are ignored at the transformer position with a one-time warning. **Engine tag:** synthesized instance now uses `type: 'pass'` (was `'path'`). Hard cut. **Runtime fixes:** - `compileNext` handles mixed-shape `next` arrays (`["a", { case }]`) via a new `'sequence'` variant. - A destination's `before` referencing a pass-through transformer now walks that transformer's own `before` / `next`. - `cache.stop: true` at a pre-collector transformer halts the pipeline (matches `cache.mdx`). **Migration:** Typo keys on a step now fail validation. `instance.type === 'path'` consumers must read `'pass'`. `runTransformerChain` consumers should branch on the new `stopped` flag. - adeebea: Route grammar: rename `case` to `one` (first-match dispatch) and add `many` (all-match parallel fan-out, pre-collector only). `many` terminates the main chain and is rejected at post-collector positions (`destination.before`, `destination.next`); use multiple destinations for post-collector fan-out. `RouteCaseConfig` is renamed to `RouteOneConfig`; no aliases. - 13aaeaa: `Source.Context` no longer exposes `setIngest` or `setRespond`. Server sources handling concurrent inbound requests must call `context.withScope(rawScope, respond, body)` to bind per-request ingest and respond. Browser and other single-scope sources keep working without changes. - e800974: `Status.dropped` is now keyed by stepId, so operators can see at a glance which step dropped events. Read with `status.dropped["collector"]?.queue` or `status.dropped["destination."]?.queue` / `.dlq`, or build the key with the new `stepId()` helper exported from `@walkeros/core`. Breaking change: the previous flat shape (`status.dropped.queue` / `.queuePush` / `.dlq`) and the per-destination `dropped` field on `DestinationStatus` are removed. - adeebea: Add `Flow.Store.cache` for store-level caching: read-through + write-through wrapper with single-flight dedup, recursive composition via `cache.store`, and per-wrapper counters. `CacheRule` is now a discriminated union (`EventCacheRule | StoreCacheRule`); schema rejects inert fields in store contexts. Built-in `__cache` upgraded with LRU, `maxEntries: 10000`, batched eviction, and active TTL sweep. **Breaking:** `@walkeros/store-memory` is removed. Its logic is absorbed into `__cache`. Migration: drop the store declaration, or omit `cache.store` to use the built-in tier. `flow_validate` flags legacy references. - 058f7ed: Add `validateJsonSchema` / `validateEventsJsonSchema` exports for step-level validation config, promote the validate and no-`many` route schemas to direct exports, and add optional `nodeType` / `subPath` cursor fields to `IntelliSenseContext` for context-scoped autocomplete. - 28a8ac2: Add step-level `validate?` primitive on every walkerOS step. `validate:` is a declarative description of validation intent, like `cache` or `consent`. Consumers decide how to enforce. Restructure `Flow.ContractRule` to a uniform `{ extends?, tagging?, description?, events?, schema? }` shape. A single agnostic JSON Schema replaces the typed section fields (`globals`, `context`, `custom`, `user`, `consent`); standard event field names live inside `schema.properties.`. `extends` resolves `schema` via additive deep-merge. Contracts are a description and governance concept: tooling, MCP, and humans read them. Runtime enforcement is the consumer's call. - fd6076e: Walker commands `destination`, `hook`, and `on` now take a single Init object: `elb('walker destination', { code, config })`, `elb('walker hook', { name, fn })`, `elb('walker on', { type, rules })`. The previous positional forms and the `{ push }` shorthand are removed; the `options` argument is gone from `collector.command`, `addDestination`, and `commonHandleCommand`. ### Patch Changes - e800974: Fix `getByPath` silently returning undefined for objects created in a different realm (Node `http.IncomingMessage`, `vm` contexts, worker threads, iframes). The internal `instanceof Object` guard is replaced with a cross-realm-safe check, so dot-notation paths now extract fields from native request objects and other cross-realm sources. - 1a8f2d7: Fix `resolveContracts` so a child contract that uses `extends` inherits the parent's `tagging` when it does not redeclare it. Previously the parent's `tagging` was silently dropped, which corrupted contract version tracking for anyone building on a base contract. - 1a8f2d7: Flow v4 routing & cache cleanup. **Cache:** - `cache.full` is renamed to `cache.stop`. Search-and-replace. - `cacheRule.match` is now optional. Omitted means always-match. The literal `'*'` is dropped from the schema and the TypeScript types; `compileMatcher` still tolerates the string at runtime for migration. - New `cache.namespace?: string` field. Omit to write keys directly to the store. Same store + same key + same namespace = same cache entry. - Implicit per-step namespace prefixes (`s:`, `t:`, `d:`) are removed. If you relied on them to separate same-keyed caches across sources/transformers/destinations using the same store, set `cache.namespace` explicitly. **Routing:** - Unified recursive `Route` type. A Route is `string | Route[] | RouteConfig`. - New `case` operator replaces the legacy `Route[]` first-match shape. The legacy shape is compiled as an implicit `{ case: [...] }` for runtime compatibility, but new configs should use `case` explicitly. - `RouteConfig` is a disjoint union enforced at the TypeScript type level via `never` fields: a single RouteConfig sets at most one of `next` / `case`. A bare `{ match }` is a gate (pass-through when the match fires, fall-through when it fails). JSON Schema validation currently emits `anyOf` and does not enforce disjointness at runtime — see follow-up notes. - Sequence sugar (`next: [A, B, C]`) is preserved. **Path:** - A transformer entry with no `code` is a `path` — a code-less passthrough. The engine synthesizes `(e) => ({ event: e })`. Use paths to name and share `before` chains across destinations. Validation: a path must declare at least one of `package`, `before`, `next`, or `cache`. **Schema & tooling:** - Updated Zod schemas (cache, route, matcher). - Updated MCP tool descriptions and resource references. - Updated `flow_validate` to enforce the new constraints (`EMPTY_TRANSFORMER` error code added). **Migration:** Hard cut at the schema/type level. Configs using `cache.full` will fail validation — rename to `stop`. Configs using `match: "*"` will fail validation — omit `match`. Configs using `Route[]` first-match still work at runtime (compiled as implicit `case`) but new configs should use `case` explicitly. `$schema: "v4"` is preserved. No version bump. - c60ef35: Remove unused legacy fields `batchFn` and `batched` from `Mapping.Rule`. Batch state lives on the destination via `BatchRegistry`, never on mapping rules. No runtime impact. - e800974: Internal pipeline failures in mapping, source startup, transformer init, and destination init now log via the scoped logger and increment `collector.status.failed`. Previously silent. User-supplied callbacks (mapping `condition`/`fn`/`validate`, `on` subscriptions) log on throw but do not affect `status.failed`. A source whose `init()` throws now stays `config.init === false` instead of being marked initialized. - e800974: Add typed accessors `Source.getSource`, `Destination.getDestination`, `Transformer.getTransformer`, `Store.getStore`. Each takes a collector and a step id and returns the registered instance with its declared generic recovered, replacing the `Elb.Fn`-collapsed shape that the bag's index signature exposes on read. Callers (mainly tests and integrations that invoke a step's raw `push` through the collector) no longer need `as any` / `as (rawData: X) => ...` casts at this boundary. Each helper throws ` not found: ` for unknown ids. No runtime behavior change. ## 4.0.2 ### Patch Changes - a6a0ea7: Rename routing types: `NextRule` to `Route`, `Next` to `RouteSpec` (Zod schemas and `MatcherNext*` IDs renamed in step). Widen `Flow.*.before/next` to `RouteSpec` so `Route[]` conditional routing type-checks at the JSON layer. Fix the CLI bundler dropping `Route[]` data via a narrowing cast on the inline path. Hard cut, no aliases; flow.json shape unchanged. ## 4.0.1 ### Patch Changes - 381dfe7: Add an optional `setup` lifecycle to destinations, sources, and stores. Each package may now implement `setup?: SetupFn` to provision external resources (BigQuery datasets and tables, Pub/Sub topics and subscriptions, SQLite tables, webhook registrations, etc.). Setup is triggered only by the new `walkeros setup .` CLI command, never automatically by the runtime, push, or deploy. Idempotency, ordering, and error semantics are the package's responsibility; the framework provides the type slot, the CLI invocation, and a `resolveSetup(value, defaults)` helper. `LifecycleContext` is the new shared context type used by both `setup` and `destroy`. `DestroyContext` remains as a deprecated type alias for one minor cycle. The `Types` bundle on `Destination`, `Source`, and `Store` gains a 5th/6th/4th positional slot for setup options; existing aliases compile unchanged because the slot defaults to `unknown`. `Config.setup?: boolean | SetupOptions` is added across all three kinds and validated by the corresponding Zod `ConfigSchema` plus the flow component schemas in `@walkeros/core/schemas/flow.ts`. CLI: - `walkeros setup .` runs a single component's `setup()` function. - `` is `source`, `destination`, or `store` (transformers have no provisioning). - `--config ` (default `./flow.json`), `--flow ` for multi-flow configs, plus standard `--json` / `--verbose` / `--silent`. - Exit 0 on success or skip; non-zero on failure. Skip narration covers three cases: no `setup()` on the package, `config.setup === false`, or `config.setup` unset. - When the package's `setup()` returns a non-undefined value, the CLI emits it as JSON on stdout for `jq` piping. - 1524275: Source lifecycle redesign: factory + eager `init` + collector-gated `on()` Source factories must now be side-effect-free. The collector calls `Instance.init()` on each source eagerly after all factories register. `require` no longer gates code execution. It gates `on(type)` delivery (events queue in `Instance.queueOn` until the source is started, then replay). `collector.pending.sources` has been removed; per-source state lives on `Source.Instance` (`queueOn`) and `Source.Config` (`init`, `require`). Migration: any source factory with side effects (queue draining, walker command emission, listener attachment) should move those into the returned Instance's optional `init` method. Tests asserting on `collector.pending.sources` should read `collector.sources[id]` and inspect `config.init` / `config.require` instead. Fixes the elbLayer queue replay clobbering fresh consent/user state, late-activated sources missing `walker run`, and inter-source require chains racing when a non-required source's init fired a state-mutating walker command before later require-gated sources had been registered. - 03d7055: Merge `definitions` into `variables`. Single concept, single syntax `$var.name(.deep.path)?`. Whole-string references preserve native type; inline interpolation requires scalars. Deep paths and recursive resolution with cycle detection now supported. `Flow.Definitions`, `Flow.Primitive`, and the `$def.` reference syntax are removed. ## 4.0.0 ### Major Changes - 93ea9c4: Event model v4: breaking changes to the `Event`, `Source`, and `Entity` shapes. - `event.id` is now a W3C span_id (16 lowercase hex chars), generated by the collector. Reference: W3C Trace Context (W3C Recommendation, January 2020). - `event.version`, `event.group`, `event.count` are removed. - `source.type` is now the source kind (e.g. `browser`, `gtag`, `mcp`, `cli`). New `source.platform` holds the runtime (`web` | `server` | `app` | ...). - `source.id` and `source.previous_id` are removed. - Browser source now sets `source.url` and `source.referrer`. - MCP source sets `source.tool` per emission. CLI source sets `source.command`. - `Entity.nested` and `Entity.context` are now optional. Root `event.nested` and `event.context` remain required. - Each source self-registers via TypeScript module augmentation of `SourceMap` in `@walkeros/core`. - App-side coordination (`/workspaces/developer/app`) is a follow-up plan, not part of this release. Telemetry from v4 CLI/MCP will not validate against the existing app schema until that follow-up ships. - `Mapping.Rule.skip` is renamed to `Mapping.Rule.silent`. Customer flow.json configs using `skip: true` in mapping rules must rename to `silent: true`. Hard cut: no legacy alias, the field is gone. - 942a7fe: Flow v4: type redesign and cross-flow references. Breaking changes: - Renamed `Flow.Settings` (single-flow shape) to `Flow`. The new `Flow.Settings` is the arbitrary kv-bag inside `Flow.Config` (matches `Destination.Settings` semantics). - Renamed `Flow.Config` (root file shape) to `Flow.Json`. - Removed `Flow.Web` and `Flow.Server`. Replaced by `config.platform: 'web' | 'server'` (a string discriminator). - Renamed `Flow.InlineCode` to `Flow.Code`. - Renamed `Flow.SourceReference` / `DestinationReference` / `TransformerReference` / `StoreReference` to `Flow.Source` / `Destination` / `Transformer` / `Store` (Reference suffix dropped). - Renamed `Flow.ContractEntry` to `Flow.ContractRule`. - Lifted `bundle` and platform fields into the per-flow `config` block. - `flow.json` `version` bumped from 3 to 4. v3 input is rejected (no compat shim). New: - `$flow.X.Y` reference resolves to `flows.X.config.Y` in the same file. Useful for linking a web flow's API destination to a server flow's deployed URL without duplicating values. - Per-flow `Flow.Config` block: `{ platform, url, settings, bundle }`. - `walkeros validate` warns on unresolved `$flow.X.Y` (use `--strict` to error). `walkeros bundle` and `walkeros deploy` always error on unresolved refs. - See `docs/migrating/v3-to-v4.mdx` on the website for the manual migration steps. No automated codemod is shipped. - cfc7469: **Breaking — `@walkeros/core`:** `fetchPackage(name, { baseUrl })` now expects the host app to expose the v2 `/api/packages/[name]` endpoint that returns the merged `WalkerOSPackage` shape directly (single round-trip, `?expand=all`). The previous two-fetch pattern (`?path=package.json` + `?path=dist/walkerOS.json`) is removed. Hosts must serve the v2 shape; the offline jsdelivr fallback is unchanged. **Feature — CLI/MCP/explorer:** Outbound walkerOS-aware HTTP clients now identify themselves to the configured app origin via `X-Walkeros-Client: walkeros-{cli|mcp}/{version}`. `@walkeros/explorer` exports `setPackageTypesBaseUrl(url?)` so host apps can proxy `.d.ts` through their own origin (used by the walkerOS Tag Manager app to drop the jsdelivr CDN allowance entirely). - 1ef33d9: **BREAKING:** Unified callback signatures across mapping and on.\* subscriptions. Every callback in walkerOS now reads `(data, context) => result`. Sources, transformers, destinations, and stores already conformed; mapping and on.\* join the family in v4.1. ### Mapping callbacks `fn`, `condition`, and `validate` now share a single shape: `(value, context: Mapping.Context) => result` `Mapping.Options` is removed. Replaced by `Mapping.Context`: ```ts interface Context { event: WalkerOS.DeepPartialEvent; mapping: Value | Rule; collector: Collector.Instance; // required logger: Logger.Instance; // required consent?: WalkerOS.Consent; // resolved consent } ``` Rule-level `condition` is now `(event, context) => boolean`. `Mapping.Options.props` is removed (no production callers). #### Mapping upgrade ```ts // before const fn: Mapping.Fn = (value, mapping, options) => /* … */; const cond: Mapping.Condition = (value, mapping, collector) => /* … */; const val: Mapping.Validate = (value) => /* … */; // after const fn: Mapping.Fn = (value, context) => /* … */; const cond: Mapping.Condition = (value, context) => /* … */; const val: Mapping.Validate = (value, context) => /* … */; ``` In `$code:` strings (flow.json): ```json // before "fn": "$code:(value, mapping, options) => …" "condition": "$code:(value, mapping, collector) => …" // after "fn": "$code:(value, context) => …" "condition": "$code:(value, context) => …" ``` `context.mapping` replaces the second positional arg; `context.collector`, `context.logger`, and `context.consent` are all available. One-arg callbacks like `(value) => value.toUpperCase()` continue to work unchanged. ### On.\* subscription callbacks `walker.on('consent', …)`, `walker.on('ready', …)`, etc. now receive `(data, context: On.Context) => void | Promise`. The legacy `Context` interface, `*Config` aliases, and `Options` discriminated union are removed. ```ts interface Context { collector: Collector.Instance; // required logger: Logger.Instance; // required } type Fn = ( data: TData, context: Context, ) => void | Promise; type ConsentFn = Fn; type SessionFn = Fn; type UserFn = Fn; type ReadyFn = Fn; type RunFn = Fn; type GenericFn = Fn; ``` The new `On.Subscription` alias is the registerable union for `walker.on(action, X)`. #### On.\* upgrade ```ts // before walker.on('consent', { marketing: (collector, consent) => /* … */ }); walker.on('ready', (collector) => /* … */); walker.on('session', (collector, session) => /* … */); // after walker.on('consent', { marketing: (consent, ctx) => /* … */ }); walker.on('ready', (_, ctx) => /* … */); walker.on('session', (session, ctx) => /* … */); ``` `ctx.collector` replaces the positional first arg; `ctx.logger` is also available. ### Why both at once Both refactors follow the same `(data, context)` pattern. Shipping them in one release means consumers do one search-and-replace pass instead of two, and the codebase reaches full callback-signature consistency in v4.1. ### Minor Changes - 8e06b1f: **BREAKING:** Unified reference syntax: `$store:id` and `$secret:NAME` now use the dot separator: `$store.id` and `$secret.NAME`. The coherent rule across every walkerOS reference is: - **`.`** key or path (resolver looks up or walks what follows) - **`:`** literal value or raw-code payload (resolver uses what follows verbatim) `$var.`, `$def.`, `$env.NAME[:default]`, `$contract.`, and `$code:(…)` are unchanged, they already fit the rule. Every shipped example, published `walkerOS.json` metadata, doc page, and skill has been updated. A new canonical reference-syntax guide lives at `/docs/guides/reference-syntax`. Regex constants (`REF_VAR`, `REF_DEF`, `REF_ENV`, `REF_CONTRACT`, `REF_STORE`, `REF_SECRET`, `REF_CODE_PREFIX`) are exported from `@walkeros/core` import these instead of hand-rolling regexes. ### Upgrade Search-and-replace across your flow configs: ``` $store: → $store. $secret: → $secret. ``` Everything else stays the same. Your `$var.*`, `$def.*`, `$env.*`, `$contract.*`, and `$code:*` references need no changes. ### Patch Changes - 465775c: Add Monaco IntelliSense for `$flow.X` cross-flow references in `Code`/`CodeBox`. Completion offers known sibling flow names from the parsed flow document, hover describes the resolved target, decorations style matches the other reference prefixes, and unknown flow names emit a warning marker. Re-export `REF_FLOW` from `@walkeros/core` so consumers can build inline regex tooling without reaching into the subpath. - 3d50dd6: JSON Schema exports now include canonical `id` + `title` (e.g. `Destination.Config`, `Logger.Config`) — replaces anonymous `__schema0` / `object` / `any` labels. Extracts shared `LoggerConfigSchema` (was inlined 5×). Removes deprecated `schemas/value-config.ts`. No TypeScript surface changes. ## 3.4.2 ## 3.4.1 ### Patch Changes - 12adf24: Step examples can now carry a `title`, `description`, and `public` flag. Non-public examples stay hidden from the docs and AI tools so first-time visitors see only the canonical ones. - 75aa26b: `useHooks` now isolates hook failures. A pre-hook that throws no longer crashes the pipeline — the wrapped function is called directly and a warning is logged. A post-hook that throws leaves the original result in place. Added optional 4th `logger` parameter so warnings route through the walkerOS Logger (falls back to `console.warn` when no logger is provided). All collector call sites now pass `collector.logger`. ## 3.4.0 ### Minor Changes - 525f5d9: Introduce the standardized `StepExample.out` shape: `[callable, ...args][]` where each tuple is a function call (first element is the callable name) or a `['return', value]` tuple for transformer-style returns. Every effect is self-describing; docs and tools can render it uniformly without a per-package registry. Ship the shared `formatOut` renderer from `@walkeros/core` for docs + app. Also exports `StepEffect` and `StepOut` types. Migrate `@walkeros/web-destination-gtag` to the new shape as the canary — its multi-tool outputs (GA4 + Ads + GTM) now flatten into a single array of `gtag(...)` and `dataLayer.push(...)` tuples in observed execution order. Remaining destination packages ship the old shape until the bulk migration (separate plan). ### Patch Changes - 74940cc: Add `TransformerSchemas` and `StoreSchemas` namespaces with `ConfigSchema` / `configJsonSchema` exports. Mirrors the existing `DestinationSchemas` / `SourceSchemas` pattern so every component type has a documented Config schema available via `@walkeros/core/dev`. Reconcile pre-existing drift between TS Config interfaces and their Zod schemas: - `DestinationSchemas.ConfigSchema` now includes `before`, `next`, `cache`, `disabled`, `mock`, `include` (matching `Destination.Config`) and drops phantom `onError` / `onLog` fields that were never wired in the TS type or consumed at runtime. - `SourceSchemas.ConfigSchema` now includes `disabled` and drops a phantom `onError` field. - `MappingSchemas.ConfigSchema` now includes `include`, matching `Mapping.Config` (the base `Source.Config` extends). - `CollectorSchemas.ConfigSchema` now includes `logger` and drops phantom `verbose` / `onError` / `onLog` fields not present in `Collector.Config`. - `CollectorSchemas.InitConfigSchema` now includes `transformers`, `stores`, `hooks`, matching `Collector.InitConfig`. Add a compile-time drift guard at `packages/core/src/schemas/__tests__/config-drift.test-d.ts`. Any future divergence between a Config TS interface and its Zod schema fails `tsc --noEmit` (already wired into the project's `typecheck` script and CI). Keys-only check; value types may still differ for recursive or generic-slot fields where Zod cannot express TS-side precision. ## 3.3.1 ## 3.3.0 ### Minor Changes - 2849acb: **BREAKING CHANGE:** The `packages` block has moved from `flow..packages` to `flow..bundle.packages`. Flow files using the old shape fail fast with a migration error pointing to the new location. Also adds `flow..bundle.overrides` — a `Record` for pinning transitive dependency versions, matching npm's `overrides` semantics. Use this to resolve version conflicts when a transitive dependency's declared range conflicts with another required version in the same tree (the original motivating case: `@amplitude/engagement-browser` pins `@amplitude/analytics-types@^1.0.0` while `@amplitude/analytics-browser` transitively requires `analytics-types@2.11.1` exact — previously an unresolvable bundler conflict). **Migration:** move the existing `packages` block one level deeper into a new `bundle` wrapper. ```diff { "version": 3, "flows": { "default": { "web": {}, - "packages": { - "@walkeros/collector": {} - }, + "bundle": { + "packages": { + "@walkeros/collector": {} + } + }, "sources": { }, "destinations": { } } } } ``` **Overrides example:** ```json { "flows": { "default": { "web": {}, "bundle": { "packages": { "@walkeros/web-destination-amplitude": {} }, "overrides": { "@amplitude/analytics-types": "2.11.1" } } } } } ``` Overrides only substitute **transitive** dependencies during resolution — direct package specs declared in `bundle.packages` always win. Overrides targeting a direct local-path package emit a warning and are ignored. Peer constraint mismatches against the chosen override emit a warning but do not error (the override is an explicit user directive). - 08c365a: Add `include` as a first-class field on `Destination.Config` (destination-level) and `Mapping.Rule` (per-event override). The collector resolves `include` in `processEventMapping` before calling `push()`, flattening specified event sections into prefixed key-value pairs (e.g. `data_price: 420`) and merging them as the bottom layer of `context.data`. Rule-level `include` replaces config-level (not additive). Merge priority: include (bottom) → config.data → rule.data (top, wins on conflict). The `context` section correctly extracts `[0]` from OrderedProperties tuples. New export: `flattenIncludeSections(event, sections)` from `@walkeros/core`. - 08c365a: Add `skip?: boolean` to `Mapping.Rule` as a universal sibling of `ignore`. Destinations can now honor a rule-level `skip` to process `settings.*` side effects (identify, revenue, group, etc.) while omitting their default forwarding call (`track()`, `capture()`, `event()`). Replaces destination-specific `settings.skipTrack` / `settings.skipEvent` toggles. `processEventMapping()` now returns an explicit `skip: boolean` field alongside `ignore`. The collector does not short-circuit on `skip` — it still calls `destination.push()` so the destination can run its side effects. The destination implementation reads `context.rule?.skip` and gates its default forwarding call on `!skip`. `ignore: true` still wins when both flags are set on the same rule. - 08c365a: Add `command` field to `Flow.StepExample` for routing non-event inputs through walker commands (`consent`, `user`, `run`, `config`). Replaces the gtag-only `_consent: true` magic marker pattern. Test runners can now explicitly opt into `elb('walker ', in)` instead of pushing `in` as a regular event. **Breaking for anyone copying gtag's step-example test runner:** the `_consent: true` magic marker on `mapping` is gone. Migrate to `command: 'consent'` on `Flow.StepExample`. ### Patch Changes - 08c365a: Expand `getMarketingParameters` to recognise 25+ ad platform click IDs (Pinterest, Reddit, Quora, Yandex, Outbrain, Taboola, Mailchimp, Klaviyo, HubSpot, Adobe, Impact, CJ, Branch, plus Google's `wbraid`/`gbraid`). Add a new `platform` field that resolves the click ID to a canonical platform identifier (e.g. `gclid` → `google`, `fbclid` → `meta`). Multi-click-ID URLs are resolved deterministically via a priority order. Custom click-ID registries can be passed as the third argument to `getMarketingParameters`, or via the new `clickIds` field in the session source settings — so flow.json users can extend or override defaults without touching code. ## 3.2.0 ### Minor Changes - eb865e1: Add chainPath to ingest metadata and support path-specific mocks via --mock destination.ga4.before.redact='...' - c0a53f9: Flow graph architecture: symmetric before/next hooks, mutable Ingest, per-destination isolation. - Add symmetric `before`/`next` to all step types (sources, transformers, destinations) - Add `Ingest` interface with mutable `_meta` tracking (hops, path) - Parameterize `Transformer.Fn` and `Result` on event type - Support `Result[]` return from transformers for fan-out - Remove `Object.freeze(ingest)` — ingest is fully mutable - Upgrade `setIngest` to create typed `Ingest` with `_meta` - Clone ingest per destination to prevent cross-contamination - Add `createMockContext` test utility for context construction - f007c9f: Wire initConfig.hooks into collector instance. Simulation uses prePush/postDestinationPush hooks for event capture. Hooks are wired by startFlow before events fire. - bf2dc5b: Add conditional routing and native cache as built-in config properties on sources, transformers, and destinations. **Routing:** NextRule[] in next/before properties enables conditional step chaining, replacing @walkeros/transformer-router. **Cache:** - Cache rules use the same match syntax as routing (MatchExpression) - Source cache: full pipeline caching with respond interception - Transformer cache: step-level memoization, chain continues - Destination cache: event deduplication - Update rules modify cached results on read via getMappingValue - Default per-collector memory store with namespaced keys compileMatcher upgraded to use getByPath for scoped dot-paths (ingest.method, event.name). Removed @walkeros/server-transformer-cache (replaced by native cache). - da0b640: Add include/exclude destination filter to collector.push PushOptions. Sources can now control which destinations receive their events. Destination simulation uses the full collector pipeline with include filter, giving production-identical event enrichment, consent, and mapping. ## 3.1.1 ## 3.1.0 ### Minor Changes - 966342b: Add mergeConfigSchema to core and integrate into MCP package_get tool. package_get now returns schemas.config — a merged JSON Schema combining base config fields (require, consent, logger, mapping, etc.) from core with the package's typed settings schema. Runtime-only fields (env, onError, onLog) are excluded. - df990d4: Unified source simulation input. All source simulation uses SourceInput { content, trigger?, env? } — one format for CLI, MCP, and tests. Removes legacy runSourceLegacy and deprecated SimulateSource fields. CLI gains --step flag. MCP flow_simulate drops example parameter (use flow_examples to discover, then provide event). flow_examples now returns trigger metadata. StepExample Zod schema aligned with TypeScript type. ### Patch Changes - dfc6738: MCP api tool: replace overloaded `id` param with explicit `projectId` and `flowId`. CLI functions now throw structured ApiError with code and details from the API response. mcpError forwards structured error data to MCP clients. - bee8ba7: Replace hardcoded package registry with live npm search. Package catalog is now fetched dynamically from npm and enriched with walkerOS.json metadata from CDN. Change platform type from string to array. Packages declare platform as ["web"], ["server"], or ["web", "server"]. Empty array means platform-agnostic. The normalizePlatform utility handles backwards compatibility with the old string format from already-published packages. Remove outputSchema from package_get to prevent SDK validation crashes on unexpected field values. - 966342b: Add missing fields to Zod schemas: require, logger, ingest on Source.Config and require, logger on Destination.Config. These fields existed in TypeScript types but were absent from schemas, making them undiscoverable via JSON Schema and MCP. ## 3.0.2 ## 3.0.1 ## 3.0.0 ### Major Changes - 0e5eede: BREAKING: Flow configs now require `"version": 3`. Versions 1 and 2 are no longer accepted. To migrate, change `"version": 1` or `"version": 2` to `"version": 3` in your walkeros.config.json. - d11f574: Rename Flow.Setup to Flow.Config and Flow.Config to Flow.Settings for consistent Config/Settings naming convention at every level. Breaking change: all type names, function names, schema names, and API URL paths (/configs → /settings) updated. - 23f218a: Replace flat/v2 contract format with named contracts supporting extends inheritance. BREAKING CHANGES: - `contract` is now a map of named contract entries (e.g., `{ "default": { ... }, "web": { ... } }`) - `version` field inside contracts removed - `$tagging` renamed to `tagging` - Legacy flat contract format removed - `$globals`, `$context`, `$custom`, `$user`, `$consent` references removed - Settings-level `contract` field removed (use named contracts at config level) - Auto-injection of `$tagging` into `collector.tagging` removed (use `$contract.name.tagging` explicitly) - Validator `contract` setting renamed to `events` (receives raw schemas, not `{ schema: ... }` wrappers) NEW FEATURES: - Named contracts with `extends` for inheritance (additive merge) - Generalized dot-path resolution: `$def.name.nested.path`, `$contract.name.section` - `$contract` as first-class reference type with path access - `$def` inside contracts supported via two-pass resolution - `$def` aliasing for reducing repetition: `{ "c": "$contract.web" }` then `$def.c.events` ### Minor Changes - 6ae0ee3: Add v2 structured contract format with globals, context, custom, user, and consent sections. Contracts can now describe cross-event properties (globals, consent, etc.) alongside entity-action event schemas. Top-level sections are JSON Schemas that merge additively into per-event validation. Breaking: None. Legacy flat contracts continue working unchanged. v2 is opt-in via `version: 2` field. - 1fe337a: Add hints field to walkerOS.json for lightweight AI-consumable package context. Packages can now export a `hints` record from `src/dev.ts` containing short actionable tips with optional code snippets. Hints are serialized into `walkerOS.json` by buildDev() and surfaced via the MCP `package_get` tool. Pilot: BigQuery destination includes hints for authentication, table setup, and querying. - c83d909: Add Store types as fourth modular component type. Stores provide pluggable key-value storage (get/set/delete/destroy) with sync and async support for browser and server backends. - b6c8fa8: Add stores as a first-class component type in Flow.Config. Stores get their own `stores` section in flow settings, a `collector.stores` registry, and `$store:storeId` env wiring in the bundler. Includes `storeMemoryInit` for Flow.Config compatibility and type widening in cache/file transformers. ### Patch Changes - 2b259b6: Fix deterministic package version resolution in bundler. - Two-phase resolve-then-install prevents version overwrites - peerDependencies resolved at lowest priority (not equal to deps) - Per-build temp directories prevent cross-build interference - Optional peerDeps (peerDependenciesMeta) correctly skipped - Prerelease versions handled with includePrerelease flag - Package names validated against npm naming rules - 2614014: Fix: `consent: {}` on destination config now auto-grants instead of blocking all events forever - 37299a9: Extract match logic (compileMatcher, MatchExpression, MatchCondition, MatchOperator, CompiledMatcher) from router to core as shared utility. Router now imports from core — no public API changes. - 499e27a: Fix getByPath breaking on falsy intermediate values (0, false, "") - d11f574: Fix $var/$def/$env resolution in transformer configs and env fields Previously, `resolvePatterns` was not called on transformer configs or any component's `env` field. This meant `$var.name`, `$def.name`, and `$env.NAME` references in those positions were passed through as literal strings. Now all component types (sources, destinations, transformers, stores) have both `config` and `env` resolved consistently. - 5cb84c1: Replace hand-written MCP resources with auto-generated JSON Schemas from @walkeros/core. Add walkerOS.json to 5 transformer packages. Variables resource remains hand-maintained (runtime interpolation patterns). - 499e27a: Add sideEffects declarations to all packages for bundler tree-shaking support. ## 2.1.1 ### Patch Changes - fab477d: Replace union transformer return type with unified `Transformer.Result` object. Transformers now return `{ event }` instead of naked events, and can optionally include `respond` (for wrapping) or `next` (for branching). The `BranchResult` type and `__branch` discriminant are removed. ## 2.1.0 ### Minor Changes - 7fc4cee: Add default values and improved descriptions to flow schemas for better IDE IntelliSense - cb2da05: Add data contracts for centralized event validation and documentation - 2bbe8c8: Add destroy lifecycle method to all step types (sources, destinations, transformers) and shutdown command to collector - 3eb6416: Add unified `env.respond` capability. Any step (transformer, destination) can now customize HTTP responses via `env.respond({ body, status?, headers? })`. Sources configure the response handler — Express source uses createRespond for idempotent first-call-wins semantics. CLI serve mode removed (superseded by response-capable flows). - 02a7958: Add WARN log level (ERROR=0, WARN=1, INFO=2, DEBUG=3). Logger instances expose `warn()` method routed to `console.warn` and `json()` method for structured output. Config accepts optional `jsonHandler`. MockLogger includes both as jest mocks. CLI logger unified with core logger via `createCLILogger()` factory. - 026c412: Unified simulation API: single simulate() function replaces simulateSource/simulateDestination/simulateTransformer/simulateFlow. Built-in call tracking for destinations via wrapEnv. No bundling required for simulation. - 7d38d9d: Add `validateFlowSetup` for portable Flow.Setup validation with line/column positions and IntelliSense context extraction. ### Patch Changes - 7fc4cee: Add contract as optional property to Flow.Setup schema - 97df0b2: Step examples: upgrade all packages to blueprint pattern with inline mapping, no intermediate variables, no `all` export - 97df0b2: Step examples: add mapping field to StepExample type, rewrite Meta Pixel examples with functional tests ## 2.0.1 ### Patch Changes - e34c11e: Align all packages to unified v2 with consistent dependency structure ## 1.4.0 ### Minor Changes - 7b2d750: Add walkerOS.json package convention for CDN-based schema discovery ## 1.3.0 ### Minor Changes - a4cc1ea: Add collector.status for per-source and per-destination delivery tracking ## 1.2.2 ### Patch Changes - 7ad6cfb: Fix transformer chains computed on-demand instead of pre-computed Transformer chains configured via `destination.before` now work correctly. Previously, chains were pre-computed at initialization but the resolution function was never called, causing `before` configuration to be silently ignored. **What changed:** - Chains now compute at push time from `destination.config.before` - Removed unused `collector.transformerChain` state - Removed dead `resolveTransformerGraph()` function - Dynamic destinations now support `before` property ## 1.2.1 ### Patch Changes - 6256c12: Add inline code support for sources, transformers, and destinations - Add `InlineCodeSchema` with `push`, `type`, and `init` fields for embedding JavaScript in flow configs - Make `package` field optional in reference schemas (either `package` or `code` required at runtime) - Update `flow-complete.json` example with inline code demonstrations including enricher transformer, debug destination, and conditional mappings ## 1.2.0 ### Minor Changes - f39d9fb: Add array support for transformer chain configuration Enables explicit control over transformer chain order by accepting arrays for `next` and `before` properties, bypassing automatic chain resolution. **Array chain behavior:** | Syntax | Behavior | | -------------------------------- | ------------------------------------------------------ | | `"next": "validate"` | Walks chain via each transformer's `next` property | | `"next": ["validate", "enrich"]` | Uses exact order specified, ignores transformer `next` | **Example:** ```json { "sources": { "http": { "package": "@walkeros/server-source-express", "next": ["validate", "enrich", "redact"] } }, "destinations": { "analytics": { "package": "@walkeros/server-destination-gcp", "before": ["format", "anonymize"] } } } ``` When walking a chain encounters an array `next`, it appends all items and stops (does not recursively resolve those transformers' `next` properties). - 888bbdf: Add inline code syntax for sources, transformers, and destinations Enables defining custom logic directly in flow.json using `code` objects instead of requiring external packages. This is ideal for simple one-liner transformations. **Example:** ```json { "transformers": { "enrich": { "code": { "push": "$code:(event) => ({ ...event, data: { ...event.data, enriched: true } })" }, "config": {} } } } ``` **Code object properties:** - `push` - The push function with `$code:` prefix (required) - `type` - Optional instance type identifier - `init` - Optional init function with `$code:` prefix **Rules:** - Use `package` OR `code`, never both (CLI validates this) - `config` stays separate from `code` - `$code:` prefix outputs raw JavaScript at bundle time ## 1.1.0 ### Minor Changes - 20eca6e: Breaking change: Unified dynamic pattern syntax in Flow configuration, sorry! **New syntax:** - `$def.name` - Reference definitions (replaces `{ "$ref": "#/definitions/name" }`) - `$var.name` - Reference variables (replaces `$variables.name`) - `$env.NAME` or `$env.NAME:default` - Reference environment variables **Migration:** | Old Syntax | New Syntax | | --------------------------------------- | -------------------------------------- | | `{ "$ref": "#/definitions/itemsLoop" }` | `$def.itemsLoop` | | `$variables.currency` | `$var.currency` | | `${GA4_ID}` or `${GA4_ID:default}` | `$env.GA4_ID` or `$env.GA4_ID:default` | **Note:** Only `$env` supports defaults (`:default`) because environment variables are external and unpredictable. Variables (`$var`) are explicitly defined in config, so missing ones indicate a configuration error and will throw. **Example:** ```json { "variables": { "currency": "EUR" }, "definitions": { "itemsLoop": { "loop": ["nested", { "map": { "item_id": "data.id" } }] } }, "destinations": { "ga4": { "config": { "measurementId": "$env.GA4_ID:G-DEMO123", "currency": "$var.currency", "items": "$def.itemsLoop" } } } } ``` ### Patch Changes - b65b773: Queue on() events until destination init completes Destinations now receive `on('consent')` and other lifecycle events only after `init()` has completed. Previously, `on()` was called before `init()`, requiring workarounds like gtag's `initializeGtag()` call inside its `on()` handler. Also renamed queue properties for clarity: - `destination.queue` → `destination.queuePush` - `destination.onQueue` → `destination.queueOn` ## 1.0.0 ### Major Changes - 67c9e1d: Hello World! walkerOS v1.0.0 Open-source event data collection. Collect event data for digital analytics in a unified and privacy-centric way.