# Data activation clients A **data activation client** is the pipeline join-point that binds a **tool** (the transport — see `tools.md`), a **data source** (the endpoint — see `data_sources.md`), and one or more **interoperability contracts** (the row-to-resource transforms — see `interoperability_contracts.md`) into a single runnable ingestion pipeline. SDK namespace: `api.dataActivationClients`. The same namespace carries both the management surface (CRUD on the binding row, §§1–5) and the runtime surface (the verbs that actually move data through the configured pipeline, §6). There is no separate "data activation runs" resource — runs are verbs on this namespace, not a sibling entity. Data activation clients are **Datalake-DB-resident** — the parent datalake must be `status: 'ready'` and all referenced resources (tool, data source, contracts) must exist before you `create()`. ## 1. Wire shape ```typescript import type { DataActivationClientRequestWritable, DataActivationClientResponse, } from '@alvera-ai/platform-sdk' const { data: created } = await api.dataActivationClients.create( tenantSlug, datalakeSlug, { name: 'Inbound CSV pipeline', description: 'Manual CSV upload feeding the patient and appointment contracts', tool_id: manualUploadToolId, data_source_id: inboundDataSourceId, tool_call: { tool_call_type: 'manual_upload' }, interoperability_contract_ids: [patientContractId, appointmentContractId], cron_expressions: [], // omit for on-demand; e.g. ['0 7,19 * * *'] row_filter: undefined, // Liquid pre-filter; see §2 downstream_connection_ids: [], // DAC chaining; cycle-checked }, ) // created.id, created.slug — server-derived // created.tool_id === body.tool_id // created.interoperability_contracts — embedded list of the bound contracts // created.is_default — true for platform-auto-created defaults ``` ## 2. Rules the type cannot encode ### Every dataset already has a default manual-upload DAC — reuse it, never create one Datalake provisioning auto-creates, for **every dataset** — the built-in resource types (`customer`, `patient`, `legal_entity`, …) **and every generic table you add** — a default manual-upload Data Activation Client, alongside the default data source + manual-upload tool and a default identity interoperability template. The plumbing to push rows into any dataset **already exists**; you do not create a DAC to seed test data or ingest a handful of rows. Reinventing it — e.g. calling `dataActivationClients.create(...)` inside a contract hook — is an anti-pattern. That DAC then lives **outside** `infra.toml`: untracked, never shown in `plan`, and not idempotent. Because DAC names are unique per datalake, it **collides on the second run** — any drift or update verdict re-fires the hook, which re-creates the same-named DAC and 409s (`name must be unique` — uniqueness collisions are 409 Conflict, see `errors.md`), halting the apply. The default DAC name is deterministic — `" DataActivationClient"` with `is_default: true` (e.g. `"Subscription Lake customer DataActivationClient"`). Discover it and ingest through it: ```typescript const { data: clients } = await api.dataActivationClients.list(tenantSlug, datalakeSlug) // `list` returns the paged envelope { data, meta } — the DAC array is clients.data const seedDac = (clients.data ?? []).find( (c) => c.is_default && (c.name ?? '').endsWith('customer DataActivationClient'), ) if (!seedDac?.slug) throw new Error('no default activation client for that table') // ingest is SLUG-keyed (see §6) — pass seedDac.slug, not seedDac.id await api.dataActivationClients.ingest(tenantSlug, datalakeSlug, seedDac.slug, { data: row }) ``` `ingest` is safe to re-run — MDM dedupe collapses repeat rows on the identifier triple — so a hook that *ingests through the default DAC* stays idempotent across re-applies. **Declare your own DAC in `infra.toml` only for a real pipeline** (an SFTP/REST fetch, a cron schedule, a custom-contract binding) — never merely to push a few rows in. See `interoperability_contracts.md` (born-red cures: prove the mapping with `.run(...)`, seed through the default DAC). ### `tool_call.tool_call_type` must align with the bound tool's `tool_body_type` The activation client's `tool_call` discriminator selects which call shape will be sent to the tool at run time. The valid `tool_call_type` is determined by the bound tool's `tool_body_type`: ``` tool_body_type: 'manual_upload' → tool_call_type: 'manual_upload' tool_body_type: 'rest_api' → tool_call_type: 'restapi_request' tool_body_type: 'sftp' → tool_call_type: 'sftp_request' tool_body_type: 's3' → tool_call_type: 's3_request' tool_body_type: 'sql_database' → tool_call_type: 'sql_query' tool_body_type: 'sharepoint' → tool_call_type: 'microsoft_share_point_excel_request' tool_body_type: 'aws_lambda' → tool_call_type: 'aws_lambda_request' ``` The complete `tool_call_type` discriminator set is exactly those seven values (`data_activation_client.ex` polymorphic embed) — there is no `sms`/`sns` call shape here, because a DAC only ever binds a tool whose `intent` is `data_exchange` (enforced in the changeset's `validate_tool_intent`). (See `tools.md` §5 for the polymorphic call shape per tool body type.) A mismatch — e.g. a `manual_upload` tool bound with an `sftp_request` call — is rejected at create time as a 422 on `/tool_call/tool_call_type`. ### `interoperability_contract_ids` is a virtual many-to-many array The field is **caller-supplied** on `Request[Writable]` as a `string[]` of contract UUIDs, but the **response** embeds the bound contracts as an `interoperability_contracts` array of full contract response objects. On `update()`, re-supply the `interoperability_contract_ids` array verbatim — `update()` replaces the whole array, so omitting it disassociates ALL bound contracts. Contracts and the activation client's tool / data source must all live under the same `datalake_slug`; cross-datalake references reject with a 404 on the offending id field. ### `interoperability_contract_ids` composition is consumer-decided A single activation client can bind: - one or more `custom` / `system` contracts (e.g. patient + appointment for a multi-resource ingestion), OR - a single `identity` contract bound to a generic table (the auto-created identity contract from `interoperability_contracts.md` §5). Mixing `identity` and non-identity contracts on one activation client is semantically unusual — fan-out treats each contract as a separate transformation lane, and identity contracts pass rows through unchanged. The platform doesn't reject the mix, but consumers typically split into two activation clients (one per output table). ### `row_filter` has INVERTED semantics A `row_filter` is a Liquid template that decides whether a row enters the activation pipeline AFTER the contract-level `filter_template` runs. Like `filter_template`, the semantics are inverted from most filter DSLs: - **Empty render → PASS** (row enters the pipeline). - **Non-empty render → SKIP** (row is dropped; the rendered string is captured as the skip reason). Liquid context includes `msg.row` (the inbound row) and `msg.client` (the activation client's template vars). If you omit `row_filter`, every row passes through. Anchor on the empty-render-passes rule, not on the Liquid keyword — `{% if ... %}...{% endif %}` and `{% unless ... %}...{% endunless %}` are equivalent. ### `rows_from_file_template` says HOW MANY rows, never what is in them A fetch lands a file in storage. `rows_from_file_template` is a Liquid template that turns that file into the list of rows to ingest. It may be set on **any** file — the content type decides only how the file is DECODED, never whether a template is allowed — and a run is never refused for declaring one. `msg` binds to the decoded **whole file**: one value per file, never a row. What it holds depends on how the fetch stored the file: | stored as | `msg` is | |------------------------|-------------------------------------------| | `application/json` | the parsed document | | `application/x-ndjson` | the list of every parsed line | | `text/csv` | the list of row maps, keyed by header | | PDF, PNG, JPEG, WebP | `{ r2_key, content_type }` — a pointer, bytes undecoded | The template must render a **JSON array**. Liquid renders text, so any value has to be serialised on the way out; `to_json` is the filter that does it, and there is no inverse — nothing parses a JSON string back into a value inside the template. Three shapes cover nearly everything: - **one document becomes many rows** — `{{ msg.data | to_json }}` - **many rows become fewer** — `{% assign kept = msg | where: "status", "active" %}{{ kept | to_json }}` - **a whole file becomes one row** — `[{{ msg | to_json }}]`. The brackets are what make it a list; this is the usual shape for a PDF or an image, where `msg` is already the pointer the contract needs. **Leave it unset when the file is already the rows.** A `sql_query` export, a CSV, and most `sftp` / `s3` bodies land as rows already. Unset means take them as they are — the common case, and it streams a chunk at a time instead of holding the file in memory, which a template cannot do because it has to see the whole file at once. **The template decides how many rows there are; the contract decides what is in each one.** A contract runs per row and can never change the count, which is the one thing only this template can do. Renaming or reshaping fields belongs in the contract (`interoperability_contracts.md`), not here. It runs on `.runManually`, on `.ingestFile`, and on every cron fire — every path that fetches a file. It does **not** run on `.ingest`, which takes rows inline and has no file to decode. Extraction happens in a background job, so a template that fails never reaches the HTTP response — `.runManually` still returns its `batch_id`. The failure surfaces as a `data_activation_logs` row you read back through `.logs.list` (§6.4), and in the platform's error log under `error_code: rows_template_failed`. **Renamed.** This field was `response_extractor` until 2026-09-14; manifests and contracts written against the old name need updating. The tool-level `response_extractor` (`tools.md`) is a different field doing a different job — it maps a model provider's envelope into a canonical result — and keeps its name. ## 3. Field ownership **Server-derived (Response-only).** Universal set from `type_naming.md`, plus: ``` interoperability_contracts array of bound contract response objects (mirror of the virtual *_ids array on the request) is_default boolean — true for platform-auto- created default DACs (one per resource type at datalake migration); cannot be deleted, can be PUT-updated ``` **Caller-supplied (round-trip).** ``` name required string description optional string tool_id required UUID — bound tool data_source_id required UUID — bound data source tool_call required embed — { tool_call_type, ... } interoperability_contract_ids required string[] — virtual many-to-many cron_expressions optional string[] — Crontab syntax (e.g. ['0 */6 * * *']); empty array = on-demand loop_over optional enum[] — 'services' | 'locations' | 'providers'; fan-out dimensions per run row_filter optional string — Liquid pre-filter; see §2 for inverted semantics downstream_connection_ids optional UUID[] — DACs triggered after this one completes; cycles rejected at create filter_config optional embed — TemplateConfig; server-pinned output_schema (enum ["true","false"]). A row- level filter evaluated per fetched row rows_from_file_template optional embed — TemplateConfig ({ type, body }); a Liquid template that says where the rows are inside a fetched file. It may be set on ANY file and is never refused — see §2 for what `msg` binds to and when to leave it unset ``` **Write-only (Request-only).** None — every caller-supplied field round-trips on the response (the virtual array round-trips under `interoperability_contracts` on response). `filter_config` / `rows_from_file_template` come back under the read-only `SimpleTemplateConfigResponse` shape (their `output_schema` is server-pinned and not echoed). ## 4. Error envelopes Standard JSON:API envelopes per `errors.md`. Common rejections: | `source.pointer` | Cause | |-------------------------------------------|----------------------------------------------------| | `/tool_id` / `/data_source_id` | resource doesn't exist (or wrong datalake) | | `/data_source_id` | does not match the bound tool's `data_source_id` — a DAC's source must equal its tool's source (source-match) | | `/tool_call/tool_call_type` | mismatch with the tool's `tool_body_type` | | `/interoperability_contract_ids/0` | contract id doesn't exist or wrong datalake | | `/name` | uniqueness within datalake (POST and PUT — rename can collide) | | `/cron_expressions/0` | invalid crontab syntax | | `/cron_expressions` | every-minute cron is rejected — **409 Conflict**, not 422; minimum interval enforced platform-side | ## 5. Lifecycle ### Create Synchronous. The activation client is immediately runnable — there's no async "deploy" phase. The next step is invoking one of the runtime verbs in §6. ### Read shapes ``` .list(tenantSlug, datalakeSlug, query?) Paged: { data, meta } (full Flop query — type_naming.md) .get(tenantSlug, datalakeSlug, id) One row, by the UUID id (NOT slug) .metadata(tenantSlug, datalakeSlug, query?) Markdown catalog page of the datalake's DACs (Flop-paged) .metadataDetails(tenantSlug, datalakeSlug, id) Markdown of one DAC's connected- dataset field docs, by UUID id ``` **`.get` takes the UUID `id`, not the slug.** The CRUD reads/writes (`.get` / `.update` / `.delete` / `.metadataDetails`) are id-keyed — the controller resolves them through the `fetch_client_by_id` plug (`get_client_by_id_for_datalake!`). Only the *runtime* verbs (§6: `.ingest` / `.ingestFile` / `.runManually` / `.logs.*`) are slug-keyed (`fetch_client_by_slug`). To go from a name/slug to an id, `list()` and match on `.slug` (or `.name`), then use the row's `id`. ### Update `update()` replaces the whole resource — resupply the full body; there's no partial update. Pointer fields (`tool_id`, `data_source_id`, `interoperability_contract_ids`) can be re-pointed. **The `update()` lookup key is the UUID `id`, which never changes.** But **the `slug` regenerates from `name` whenever `name` changes** — the changeset re-derives it (`SlugGenerator.name_to_slug`), so the response body carries a NEW slug after a rename even though you addressed the row by its stable `id`. Examples: ``` pre-update: name: 'Original', slug: 'original', id: PUT body: name: 'Renamed', ... PUT URL: .../data-activation-clients/ ← id is stable, addressable throughout PUT response: name: 'Renamed', slug: 'renamed' ← new slug derived from the new name ``` **Consequence**: because the runtime verbs (§6) are slug-keyed, a rename moves the ingest / run / logs endpoints to the new slug — and any downstream resource that references the DAC by slug (workflow templates, action bodies referencing `{{ ... }}` paths derived from the DAC slug, monitoring dashboards, etc.) breaks the moment the DAC is renamed. Capture the post-update slug from the response body. If you rename a DAC mid-deployment, update every binding that references it. An `update()` that keeps `name` unchanged regenerates to the same slug (slugification is deterministic), so cosmetic-only updates don't break bindings. ### Delete `delete()` removes the activation client (by UUID `id`). Two rejections: a **default DAC** (`is_default: true`) is refused with a **422** on `/is_default` ("Default data activation clients cannot be deleted"), and a DAC still **referenced by another resource** is refused with a **409**. Otherwise it deletes; a run already in flight keeps its own lifecycle (its `batch_id` is already stamped on the enqueued jobs) and completes independently of the deleted row. ## 6. Runtime Three run shapes exist, selected by the bound tool's `tool_body_type`: ``` inline JSON → .ingest(slug, { data }) (small payloads) uploaded file → .ingestFile(slug, { key }) (bulk; async) pull-based → .runManually(slug) (e.g. SFTP, REST poll) ``` All three produce **log rows** observable through `api.dataActivationClients.logs.*` (§6.4), and the canonical way to **verify** that ingestion worked is the two-step dataset-search pattern in §6.6. ### 6.1 Inline JSON ingestion — `.ingest` For activation clients bound to a `manual_upload` tool, the smallest run shape is a single JSON row submitted inline: ```typescript const { data } = await api.dataActivationClients.ingest( tenantSlug, datalakeSlug, activationClientSlug, { data: rowAsJson }, // a single row, shaped per the CSV/JSON the contracts expect ) // data.batch_id — UUID; the run's batch identifier (use to find the run's log rows) // data.key — storage key for the row's archived ndjson // data.jobs_count — number of downstream jobs the platform enqueued ``` The response returns **immediately after enqueueing**, not after ingestion completes. To know when the row has actually been written, poll the logs (§6.4) or use the verification pattern (§6.6). The `batch_id` is the most important field on the response — every log row the run writes carries it, and every dataset row the run writes points at one of those log rows through `alvera_data_activation_log_id`. A dataset row carries no `batch_id` of its own, so the batch reaches the rows through the log rows: that is the join for run-specific verification. ### 6.2 Bulk file ingestion — `.ingestFile` For larger payloads, upload the file via a presigned URL first, then trigger ingestion by the returned key: ```typescript // 1. Mint a presigned upload URL (see datalakes.md). const { data: link } = await api.datalakes.createUploadLink( tenantSlug, datalakeSlug, { content_type: 'text/csv', filename: 'inbound.csv' }, ) // link.url — presigned PUT URL (signature + expiry baked in) // link.key — storage key the platform will reference // 2. PUT the file body to the presigned URL (raw HTTP, NOT via the SDK). await fetch(link.url, { method: 'PUT', headers: { 'Content-Type': 'text/csv' }, body: csvBytes, }) // 3. Enqueue the ingest-file job by key. const { data: job } = await api.dataActivationClients.ingestFile( tenantSlug, datalakeSlug, activationClientSlug, { key: link.key }, ) // job.key — echoes link.key // job.job_id — numeric id of the enqueued background job // job.status — initial Oban job state (e.g. 'scheduled' / 'available') ``` This path is **fully async** — the response acknowledges that the work has been enqueued, but no row has been written yet. In the background the platform reads the file, allocates a fresh `batch_id`, and fans out per-row work. Discover the assigned `batch_id` by diffing the activation client's log rows before vs after the call (§6.4). ### 6.3 Manual run — `.runManually` For activation clients bound to pull-based tools (`sftp`, `rest_api`, etc.), there is no caller-supplied payload — the tool itself fetches data from the external system. The SDK verb: ```typescript const { data } = await api.dataActivationClients.runManually( tenantSlug, datalakeSlug, activationClientSlug, ) // data.batch_id — UUID; returned immediately (202), the batch stamped on this run's jobs ``` Unlike `.ingestFile`, `.runManually` **returns the `batch_id` directly** (`202` + `{ batch_id }` — `RunManuallyResponse`); you do NOT have to diff the logs to find it. The actual fetch still runs asynchronously in the background, so poll the logs (§6.4) with that `batch_id` to know when rows have landed. #### One-shot `tool_call` override `.runManually` accepts an optional `tool_call` body that overrides the DAC's persisted `tool_call` FOR THIS RUN ONLY. Omitting the body (or omitting just the `tool_call` field) runs with the persisted `tool_call`. Supplying a `tool_call` map does NOT mutate the DAC row — it's a transient per-invocation override, useful for one-off backfills with a different path, payload, or fetch parameters. ```typescript await api.dataActivationClients.runManually( tenantSlug, datalakeSlug, activationClientSlug, { tool_call: { tool_call_type: 'rest_api_request', path: '/v1/backfill', ... } }, ) ``` ### 6.4 Logs subresource — `.logs.list` / `.logs.get` Every ingestion run produces one or more **log rows** — one per top-level interoperability contract that fired, plus one per cascade target (a nested resource produced by a template's upserts). The log row carries: ``` batch_id UUID — the run's batch identifier dataset_table string — the target dataset/table name rows_ingested number — count of top-level rows the contract emitted (cascade rows carry 0) dataset_updated number — count of dataset rows that point at this log row (each row the run writes carries its log row's id in alvera_data_activation_log_id; load-bearing: see Gotcha 7) output_files { object_key, mode }[] — merged ndjson archive refs, one per bucket; mode is 'raw' | 'tokenized' | 'redacted' (empty until the post-batch merge worker commits). NOT a string[] — each entry is an embed object; use /logs/:id/download to fetch a presigned URL ``` ```typescript // Paginated list of log rows for an activation client. Takes the full // Flop query (page / page_size / order_by / order_directions / filters // — type_naming.md); without it, page 1 only, and a busy client's older // batches drop out of view. const { data } = await api.dataActivationClients.logs.list( tenantSlug, datalakeSlug, activationClientSlug, { page: 1, page_size: 50, order_by: ['inserted_at'], order_directions: ['desc'] }, ) // data.data — array of log rows; filter by batch_id to find this run // Fetch one log row by id. const { data: entry } = await api.dataActivationClients.logs.get( tenantSlug, datalakeSlug, activationClientSlug, logId, ) ``` #### Top-level rows vs cascade rows A multi-resource contract (e.g. an inbound row that produces a patient AND an appointment AND a related person) fires one log row **per dataset_table** the contract touched. The top-level contracts carry `rows_ingested > 0`; cascade rows (nested upserts inside a template) carry `rows_ingested = 0`. Filter on `rows_ingested > 0` when counting "how many top-level contracts fired". #### `output_files` as the completion signal A log row appears as soon as the per-row write is acknowledged, but `output_files` stays empty until the platform's background merge step has committed the ndjson archive. Use `output_files.length > 0` as the "this batch's rows are fully durable" signal — not just the presence of the log row. ### 6.5 Runs subresource — `.batches.list` / `.batches.get` A **run** is one record; the log rows of §6.4 are one record per dataset table that run wrote into. Read the run to answer *"is this over, and how did it end"*; read the logs for the per-table detail, filtered on the same `batch_id`. Before this record existed, the only way to ask whether a run had finished was to count log rows and guess — which cannot tell a run still going from one that produced nothing and never will. A run that was skipped or that died looked exactly like a slow one until the caller's timeout expired. ``` batch_id UUID — the run identifier, the same one `.ingest`, `.ingestFile` and `.runManually` hand back, and the one stamped on every per-table log row of this run client_id UUID — the owning activation client closed_at ISO timestamp | null — when the run was attempted and exhausted. Null while it is still going. It does NOT mean rows landed and it does NOT mean the archive was merged — read `status` for the outcome, and only once this is set. status 'processing' | 'succeeded' | 'partial' | 'skipped' | 'failed' rows_total number — source rows enqueued for this run. The run's own count; each per-table log keeps its own. inserted_at ISO timestamp — when the run started ``` **`status` means nothing until `closed_at` is set.** A run in flight always reads `processing`. Once it is closed: | status | what happened | |---|---| | `succeeded` | it ran and nothing was refused | | `partial` | it ran and at least one row was refused on the way in; the rows that landed are still there | | `skipped` | it ran and there was nothing to do — empty file, every row filtered out, or the client has no contracts | | `failed` | it died before writing anything | ```typescript // Paginated list of runs for an activation client. Takes the full Flop // query (page / page_size / order_by / order_directions / filters — // type_naming.md); without it, page 1 only. const { data } = await api.dataActivationClients.batches.list( tenantSlug, datalakeSlug, activationClientSlug, { page: 1, page_size: 50, order_by: ['inserted_at'], order_directions: ['desc'] }, ) // data.data — array of run records; data.meta — pagination // Fetch one run by its batch_id. const { data: run } = await api.dataActivationClients.batches.get( tenantSlug, datalakeSlug, activationClientSlug, batchId, ) ``` > **The fourth argument is the `batch_id`, not the record's `id`.** > A run record carries both, and both are UUIDs. The route looks the > run up by its `batch_id` — the identifier the ingest handed you — so > passing the record's own `id` is a **404**, not a type error. #### Waiting on a run This is what the run record is for. Poll until `closed_at` is set, then branch on `status` — never poll `status` alone, because a run in flight reads `processing` forever and tells you nothing: ```typescript const { data: ingest } = await api.dataActivationClients.ingest( tenantSlug, datalakeSlug, activationClientSlug, { data: row }, ) const deadline = Date.now() + 120_000 let run for (;;) { const { data } = await api.dataActivationClients.batches.get( tenantSlug, datalakeSlug, activationClientSlug, ingest.batch_id, ) if (data.closed_at != null) { run = data; break } if (Date.now() > deadline) throw new Error(`run ${ingest.batch_id} never closed`) await new Promise((r) => setTimeout(r, 1_000)) } if (run.status === 'failed') { throw new Error(`run ${ingest.batch_id} failed before writing anything`) } if (run.status === 'skipped') { throw new Error(`run ${ingest.batch_id} had nothing to do — check the row filter and the client's contracts`) } // 'succeeded' or 'partial' — the run is over and rows were written. // For a partial, read the log rows (§6.4) for which table refused what. ``` A run that failed or was skipped **ends the wait immediately, carrying why**. That is the whole difference from counting rows: a blind poll spends its full timeout and then reports nothing about the cause. > **`closed_at` is not the same as "my row is queryable".** It says the > run is over. It does not say the archive was merged (that is > `output_files` on the log rows, §6.4), and for one specific row it > does not replace the landing check in §6.6 — it makes that check a > single read after the run is known finished, instead of a guess > against a deadline. ### 6.6 Verifying ingestion: the two-step dataset search > **Simplest landing check — the run's own log rows.** Ingestion is async > (the row commits a moment after the `202`). When you just need to know > rows *landed*, read the run's log rows (§6.4) filtered by its > `batch_id`. Every dataset row the run writes points at the log row for > its table, and that log row's `dataset_updated` counts it — so the log > rows answer the question with no query on the dataset table and no > table name to know. A dataset row carries no `batch_id` of its own to > filter on. **Each call is a fresh read, so there is no stale snapshot to > recover from** — unlike a `createUserSearch` created before the row > commits, which captures nothing and never updates. Gate on it, then run > the two-step `createUserSearch` below **once** (race-free) only if you > need the dataset's alias-join + tokenization semantics. > > ```typescript > const { data: logs } = await api.dataActivationClients.logs.list( > tenantSlug, datalakeSlug, activationClientSlug, > { filters: [{ field: 'batch_id', op: '==', value: batchId }] }, > ) > // One log row per dataset table the run wrote into; each counts the rows > // that point at it. A sum above zero proves the batch landed. > const landed = (logs.data ?? []).reduce((n, log) => n + (log.dataset_updated ?? 0), 0) > 0 > ``` > > `createUserSearch` is the heavier, workflow-oriented surface — reach for > it when you need the rows themselves: its alias joins, tokenized > read-back. Not for a plain "did it land" check. The canonical way to **prove a row landed in the datalake** is to query the dataset directly. This is a two-step pattern: The dataset endpoints are datalake-slug-scoped: every call takes `(tenantSlug, datalakeSlug, dataset, …)` as leading args. There is no `datalake_id` query param and no tenant-derived default — the datalake is mandatory and explicit, mirroring `tools`, `workflows`, `dataActivationClients`, etc. ```typescript // (1) POST creates a user search and runs the underlying SQL // synchronously against the raw lake. The result // set is materialised server-side and identified by the // returned UUID. const { data: search } = await api.datasets.createUserSearch( tenantSlug, datalakeSlug, 'patient', // dataset name { search_query: `ri.value = '${externalPatientId}'` // SQL WHERE-clause fragment + ` AND le.alvera_data_activation_log_id IN` + ` (SELECT id FROM data_activation_logs WHERE batch_id = '${batchId}')`, }, ) // search.id — UUID of the materialised result set // search.status — 'new' | 'in_progress' | 'completed' | 'error'. // The failure member is 'error', NOT 'failed' — and it is // 'error' that carries error_message. There is no 'failed'. // search.results_count — ALWAYS null on the POST response (see Gotcha 5) // (2) GET paginates rows from the materialised result set. // dataAccessMode selects which lake answers: // 'raw' — real values verbatim // 'tokenized' — masked columns return a stable tok_ // 'redacted' — masked columns return an unjoinable placeholder // `search.id` is `string | undefined` on the response type, and // `DatasetSearchOptions.userSearchId` is declared `?: string` — optional, but // NOT `| undefined`. So an app with `exactOptionalPropertyTypes` will not // compile the shorthand, and an app without it gets a runtime 404 that reads // like a wrong path. Narrow it first. if (!search.id) throw new Error(`createUserSearch returned no id (status ${search.status})`) const { data: page } = await api.datasets.search( tenantSlug, datalakeSlug, 'patient', { userSearchId: search.id, dataAccessMode: 'tokenized' }, ) // page.data — array of rows // page.meta — pagination ``` #### SQL aliases `createUserSearch` accepts a WHERE-clause fragment against a fixed set of server-assigned table aliases. **The alias is assigned by the dataset's server-side decomposed query — it is not derivable from the dataset name**, so confirm it rather than guessing: ``` legal_entity → le message → m action_log → al document → d beneficial_owner → bo ``` GH-859 removed the per-domain datasets this list used to name — `patient`, `appointment`, `customer` and their `regulated_*` tables are gone, and with them the `rp` / `ra` / `rc` aliases. **A customer, a patient or an appointment is now a generic table you declare, plus the legal entity its rows resolve to**; there is no dataset of that name to search. The four cookbooks all take this shape. Scoping a search to one run is the usual reason to reach for this surface at all, and it goes **through the log row**. A row carries no `batch_id`, `client_id`, `user_id` or `source_uri` of its own: it carries `alvera_data_activation_log_id`, the `data_activation_logs` row of the ingest that wrote it, and the batch lives there. So the predicate is ``` .alvera_data_activation_log_id IN (SELECT id FROM data_activation_logs WHERE batch_id = '') ``` The search runs with read access across the whole lake, so `data_activation_logs` is readable from inside it. The pointer is null on a row nothing ingested (a row an app wrote itself), so such a row never matches a batch predicate. #### Where a row came from, coming back A searched row carries the pointer **and** the log row it points at, loaded. Measured on a live run — one legal entity, one generic-table row — the platform columns that come back are: ``` id string inserted_at / updated_at string checksum string checksum_changed? boolean (the key has the question mark) legal_entity_id string (generic-table rows) alvera_data_activation_log_id string (null on a row no ingest wrote) alvera_data_activation_log object — the log row itself: id, batch_id, client_id, user_id, source_uri, dataset_table, rows_ingested, dataset_updated, status, error, input_files, output_files, inserted_at, updated_at ``` So "which batch wrote this row?" is `row.alvera_data_activation_log.batch_id`, and "where did it come from?" is `row.alvera_data_activation_log.source_uri` — there is no top-level `batch_id`, `client_id`, `user_id` or `source_uri` on the row unless the table declared a column of that name itself. #### What a masked column looks like coming back **Every column comes back as a plain value of its own type, in every mode.** A masked column is a string; it is not wrapped in an object. Measured on a live three-lake tenant, the same `first_name` read three ways through `datasets.search`: ``` raw "Teodoro" tokenized "tok_def1e9dd0d0d276ea8e39f885790f5ad" redacted "xxxx-oro" ``` An older version of this guide described tokenized columns arriving as `{ redacted_data, token, type }` objects and told consumers to branch on `typeof row.field === 'object'`. That branch can never be taken. If you are carrying such a guard, it is dead code — and worse, the `else` arm it guards is the one doing all the work. The two masked forms differ in what they are *for*. `tok_` is stable, so the same input yields the same token and a machine can join on it across tables without ever resolving it. `xxxx-` is deliberately unjoinable — a placeholder for a person glancing at a screen. Pick the mode by which of those two readers you are serving. ## 7. Gotchas 1. **`tool_call_type` must match the tool's `tool_body_type`.** An activation client is a contract between the tool and the pipeline; sending a call shape the tool doesn't accept is rejected at create. See `tools.md` §5 for the per-body call-shape table. 2. **`interoperability_contract_ids` on `update()` replaces the bound set entirely.** `update()` replaces the whole array — resupply the full set; any item you omit is removed. Always re-supply the full id array on update, even if you're only changing `description`. 3. **All referenced resources live under one datalake.** The activation client, its tool, its data source, and every bound contract must share the same parent datalake. Cross-datalake references return 404 on the offending id. 4. **Custom-dataset identity contracts get their own activation client.** When you ingest into a generic table, bind the table's auto-created identity contract (see `interoperability_contracts.md` §5) to a dedicated activation client. Don't mix it with non-identity contracts on the same client unless you specifically want the fan-out semantics. 5. **`createUserSearch.results_count` is always null.** The POST response confirms the search materialised but does NOT report row counts. The actual rows live in the server-side result set and only become visible via the paginated `.search` GET. A common mistake is reading `results_count` and concluding "no rows matched" when the POST simply doesn't populate that field. 6. **`.ingest` returns BEFORE the row is durable.** The response carries `batch_id` + `jobs_count`, signalling that downstream jobs have been enqueued — not that they've completed. Poll logs (or use the search verification pattern in §6.6) before treating the row as ingested. 7. **`dataset_updated > 0` is the load-bearing freshness gate.** `rows_ingested` counts rows the contract emitted — received, not persisted. `dataset_updated` counts the dataset rows that point at the log row, so it moves only when a row has been written. A `.createUserSearch` scoped to the batch (§6.6) issued BEFORE the rows are written returns an empty result set, and there's no way to refresh the materialised result — you'd have to POST a new search. Poll for `dataset_updated > 0` before issuing the search. 8. **`output_files` empty ≠ batch failed.** A log row with `rows_ingested > 0` but `output_files: []` is mid-merge, not broken. The platform commits the ndjson archives asynchronously; wait for `output_files.length > 0` if your downstream code reads the archives directly. 9. **Bulk uploads PUT to a presigned URL, NOT through the SDK.** Step 2 of the bulk flow uses raw `fetch()` against the presigned URL the platform returns; the SDK is NOT involved in the upload itself. The SDK round-trip only covers minting the URL (`createUploadLink`) and triggering ingestion by key (`ingestFile`). 10. **`runManually` and `.ingest` produce structurally identical log rows.** Once a `batch_id` is allocated, every run flavor merges into the same log/dataset row shape. Consumers that diff or aggregate runs don't need to branch on which verb produced the batch. 11. **`row_filter` is empty-passes, non-empty-skips.** Same inverted semantics as InteroperabilityContract.filter_template. A correctly-shaped filter renders to a non-empty string when the row should be SKIPPED. Always anchor on the empty-render-passes rule, not on the Liquid keyword choice. 12. **`downstream_connection_ids` is cycle-checked at create.** The platform rejects DAC chains that would form a cycle. Make sure both endpoints of the chain exist before adding the pointer; cross-datalake chaining is also rejected. 13. **`is_default: true` DACs cannot be deleted.** Platform- auto-created default DACs (one per resource type at datalake migration time) are protected. Filter them out (`!d.is_default`) when listing for management purposes; you can still PUT-update them (e.g. to bind additional contracts). 14. **A never-landing ingest (`dataset_updated` never flips) fails silently — diagnose UPSTREAM, at the extractor, not the merge.** Distinct from the trigger *lag* in §7.7: here the row is *received* but never *persists*. The per-row `DataExtractor` is where it dies; the later `BatchMergeWorker` "No files found … *.ndjson" is a DOWNSTREAM SYMPTOM (the extractor never wrote the ndjson) — don't stop there. Usual extractor causes: (a) **MDM subject not resolved** — `:mdm_output_required` / "upsert_dataset called without %MDMOutput{}": the contract's `mdm_input_config` rendered nil (a `null`/misprovisioned identity contract, or missing MDM identity fields), so a domain that requires entity resolution rejects the row; (b) a **per-row changeset rejection** — most often an out-of-set enum (see `interoperability_contracts.md` §7). The caller sees only `rows_ingested`. Diagnose with the platform's Oban error logs (isolate by `batch_id`/`datalake_slug` metadata) + an `execute-sql` probe of the DAC log's `dataset_updated`; the CLI's `run-dac --json` + `execute-sql` verbs reproduce and probe it without building an app.