# @duckedup/nidus

The JavaScript/TypeScript client for [nidus](https://nidus.duckedup.org), a pure-Rust
vector store with full-text search that runs anywhere Rust runs. This package connects
to a running `nidus serve` instance over HTTP, whether it's on your laptop or a remote
host.

It is a **remote client**: zero runtime dependencies, built on the platform-global
`fetch`, so it runs unchanged on Node 18+, Deno, Bun, Cloudflare Workers, and in the
browser.

```sh
npm install @duckedup/nidus
```

This package is versioned in lockstep with nidus itself: a given `@duckedup/nidus`
version is the client for the identically-numbered nidus release. Match the two and the
wire contract lines up.

## Connecting

"Local vs remote" is just the base URL: point the client at a local `nidus serve`
or any reachable host.

```ts
import { NidusClient } from "@duckedup/nidus";

// Local
const db = new NidusClient({ baseUrl: "http://127.0.0.1:7700" });

// Remote, with the bearer token the server was started with (`nidus serve --token`)
const db = new NidusClient({
  baseUrl: "https://nidus.internal.example.com",
  token: process.env.NIDUS_TOKEN,
});
```

## Upserting and searching

`attrs` accept plain JS values (strings, numbers, booleans, string arrays, `Date`s,
and `null`) and are normalized to nidus's typed values for you. (For an explicit
type, use the `v.*` helpers.)

```ts
await db.createCollection("docs");

await db.upsert("docs", [
  { id: "a", vector: [0.1, 0.2, 0.3], attrs: { lang: "rust", year: 2024 } },
  { id: "b", vector: [0.4, 0.5, 0.6], attrs: { lang: "go", year: 2023 } },
  // a text-only doc, omit the vector
  { id: "c", attrs: { body: "vector stores are neat" } },
]);

const hits = await db.search({ query: [0.1, 0.2, 0.3], topK: 5 });
for (const hit of hits) {
  console.log(hit.id, hit.score, hit.attrs.lang); // attrs decoded to plain JS values
}
```

nidus has separate `Int` and `Float` attribute types and compares them same-type only,
but JS has one `number` and `1.0 === 1`, so a plain number becomes an `Int` when
`Number.isInteger` says so and a `Float` otherwise. That means a whole-numbered
measurement lands as an `Int` in whichever records it came out round, and a `Float`
range filter then skips exactly those. Pin such a field with `v.float`:

```ts
import { v } from "@duckedup/nidus";

await db.upsert("docs", [
  {
    id: "d",
    attrs: {
      score: v.float(1), // a Float even though the value is whole
      ratio: 0.75, // already a Float, not an integer
      year: 2024, // an Int
      seen: new Date(), // a DateTime: a UTC instant, epoch milliseconds
    },
  },
]);
```

A `DateTime` carries no timezone and has millisecond resolution; it decodes back to a
`Date`, so a decoded `attrs` map re-encodes to what it came from. `NaN` and `Infinity`
throw: JSON has no spelling for them. The Go and Python SDKs have the numeric types JS
lacks and decide from those instead, so a Python `2.0` or a Go `float64(2)` is a
`Float` where a bare `2` here is an `Int`.

## Filtering

Build an AND-filter with the `f.*` helpers. Each predicate is a positive assertion
about a present attribute (an absent key matches nothing). Comparisons are same-type
only, so an operand must encode to the attribute's type: `f.ge("score", v.float(2))`,
not `f.ge("score", 2)`, for a `Float` attribute.

```ts
import { f } from "@duckedup/nidus";

const hits = await db.search({
  query: [0.1, 0.2, 0.3],
  topK: 10,
  filter: f.and(
    f.eq("lang", "rust"),
    f.ge("year", 2020),
    f.in("status", ["published", "draft"]),
    f.glob("path", "src/*"),
  ),
});
```

Beyond the comparisons there are text predicates: approximate, token-wise, and
regular-expression matching over a plain attribute (no full-text index required):

```ts
f.fuzzy("title", "vecter store", 2);      // within 2 Levenshtein edits (max 8)
f.containsAllTokens("body", "vector store"); // both tokens, any order
f.containsAnyToken("body", "vector store");
f.containsTokenSequence("body", "vector store"); // as a phrase, in order
f.regex("path", "src/.*\\.rs");           // anchored at both ends, like f.glob
```

`f.regex` uses Rust's `regex` syntax, not JavaScript's: no backreferences and no
lookaround. Prefix it with `(?i)` for case-insensitive matching.

## Indexing the text predicates

`fuzzy`, `containsAllTokens`, `containsAnyToken`, `containsTokenSequence` and `regex` are
scanned per record by default. Declaring a filter index makes them a lot faster and changes
no results at all: the index proposes candidates and the predicate still decides.

```ts
await db.setFilterIndex("docs", ["body"]);
// Per-field: only the token predicates on `tag`, no fuzzy or regex.
await db.setFilterIndex("docs", ["body", { field: "tag", trigrams: false }]);
// An empty list drops it.
await db.setFilterIndex("docs", []);
```

It is opt-in per collection and per field, and it costs write time and memory. Documents
already written are indexed as part of the call.

## Full-text and hybrid search

```ts
await db.setFtsSchema("docs", ["body"]);
// Per-field tuning: await db.setFtsSchema("docs", [{ field: "body", k1: 1.5 }]);

// BM25 text search
const text = await db.textSearch({ field: "body", query: "vector store", topK: 10 });

// Fuse vector + text via reciprocal rank fusion
const hybrid = await db.hybridSearch({
  vector: [0.1, 0.2, 0.3],
  field: "body",
  text: "vector store",
  topK: 10,
});
```

A query can search several fields at once, each with its own text, by sending `clauses`
instead of the single field, folded by `combine`, `"Sum"` (a doc hitting title *and*
body outranks one hitting either) or `"Max"` (a long body cannot out-accumulate a
precise title match). Weight the two hybrid legs with `vectorWeight`/`textWeight`.

```ts
const hits = await db.textSearch({
  clauses: [
    { field: "title", query: "rust" },
    { field: "body", query: "async runtime" },
  ],
  combine: "Max",
  topK: 10,
});
```

Set `prefix` to expand a clause's *final* term as a prefix match, for typeahead: `query:
"ru"` with `prefix: true` matches an indexed term like `"running"`; earlier terms in a
clause still require an exact stem match. On the `clauses` spelling, set `prefix` per
entry instead of at the top level.

```ts
await db.textSearch({ field: "title", query: "ru", prefix: true });
await db.textSearch({ clauses: [{ field: "title", query: "ru", prefix: true }] });
```

## Named vectors

A record may carry several named vectors at the store's dimension, in addition to (or
instead of) the plain `vector` field, which lives under the reserved name `"default"`.
Declare the names a collection accepts first, then upsert and search them:

```ts
await db.setVectorNames("docs", ["title", "body"]);

await db.upsert("docs", [
  {
    id: "a",
    vectors: { title: [0.1, 0.2, 0.3], body: [0.4, 0.1, 0.2] },
    attrs: { lang: "rust" },
  },
]);

// Score both names and reduce each record to one hit before ranking.
const hits = await db.search({
  query: [0.1, 0.2, 0.3],
  names: ["title", "body"],
  nameWeights: { title: 2, body: 1 }, // a name absent here weights 1
  pool: "Sum", // "Max" (the default) or "Sum"
});
```

A search naming no vectors searches only `"default"`, unaffected by any named vectors a
record also carries, so every existing call is unaffected. `pool` decides how several
named scores become one record score: `"Max"` (the default) takes the best weighted
score, so a record missing a name is not penalized for it; `"Sum"` adds every weighted
score, rewarding a record that matches on several names at once. Upserting a name that
was never declared is a `400` naming it.

`limitPer` and `diversity` also work on `hybridSearch`, applied to the fused ranking on
the same cap, then MMR, then page-cut order as `search`:

```ts
await db.hybridSearch({
  vector: [0.1, 0.2, 0.3],
  field: "body",
  text: "vector store",
  limitPer: { field: "path", max: 1 },
  diversity: 0.5,
});
```

## Suggesting completions

`suggest` completes a partial word from an indexed field's vocabulary, ranked by
document frequency (the commonest term first), for an autocomplete dropdown. That is
the opposite of how a prefix clause ranks documents, so it is its own method rather
than a `textSearch` option. Completions are real words: surface forms are indexed
alongside stems, so every keystroke of `"running"` completes to `"running"` rather
than to the stem `"run"`.

```ts
const { suggestions, matched } = await db.suggest({
  collection: "docs",
  field: "body",
  prefix: "vec",
  limit: 10,
});
for (const { term, df } of suggestions) console.log(term, df);
// matched > suggestions.length means the server's 256-term cap truncated the list.
```

A prefix that matches nothing exactly falls back to typo-tolerant matching by default;
pass `fuzzy: false` to require an exact prefix.

## Explaining and highlighting a hit

`explain` reports what each leg and each matched clause contributed; `highlight` returns
excerpts of the stored text (so it works even on a field the projection dropped). Both
land on `hit.annotations`, which is absent unless you asked for one of them.

```ts
const hits = await db.hybridSearch({
  vector: [0.1, 0.2, 0.3],
  field: "body",
  text: "vector store",
  explain: true,
  highlight: true, // or { maxFragments: 3, fragmentChars: 120 }
});

for (const { field, fragments } of hits[0]?.annotations?.highlights ?? []) {
  for (const fragment of fragments) {
    for (const span of fragment.spans) {
      console.log(field, fragment.text.slice(...span));
    }
  }
}
```

nidus reports a span as a **UTF-8 byte** range, but a JS string is indexed in UTF-16
code units, so a raw span slices the wrong text out of any non-ASCII excerpt. This SDK
converts them for you: `fragment.spans` are JS string indices, and `fragment.text.slice`
is the matched term. If you compare them against the raw HTTP response, expect the
numbers to differ wherever the excerpt is not ASCII.

## Inspecting how a query was answered

`searchWithPlan`, `searchSimilarWithPlan`, and `hybridSearchWithPlan` are siblings of
`search`/`searchSimilar`/`hybridSearch` that return `{ hits, plan }` instead of a bare
`Hit[]`: the plan reports which scan strategy the server took (`ann`, `exact`, …), how
many rows it scanned, and per-stage timings in microseconds. `textSearch` has no plan.

```ts
const { hits, plan } = await db.searchWithPlan({ query: [0.1, 0.2, 0.3], topK: 10 });
console.log(plan.path, plan.timings.totalUs);
```

`plan.path` is a plain string, not a closed union: treat an unrecognized value as
"some scan strategy newer than this SDK" rather than an error.

## Ranking, grouping, ordering, and aggregating

```ts
// Prefer recent hits: subtract a penalty that halves every `scale` ms of age.
const recent = await db.search({
  query: [0.1, 0.2, 0.3],
  rankBy: {
    decay: { field: "updated_at", origin: Date.now(), scale: 7 * 86_400_000 },
  },
  // …and keep at most 2 hits from any one file
  limitPer: { field: "path", max: 2 },
});

// Read a chunked corpus as documents: the best chunk per file, widened with its
// neighbours into `hit.context`. Payload only, so the ranking is unchanged.
const passages = await db.search({
  query: [0.1, 0.2, 0.3],
  limitPer: { field: "nidus.parent_id", max: 1 },
  expand: { radius: 1 },
});

// On recall the same pair has one text-native spelling.
await db.recall("docs", "how does the writer lock work", {
  rollup: { neighbours: 1 },
});

// Sort a listing by an attribute instead of storage order
await db.list({ orderBy: { field: "updated_at", descending: true } });

// Count matches and sum attributes, without reading a single vector
const { count, sums } = await db.aggregate({
  filter: f.and(f.eq("lang", "rust")),
  sum: ["bytes"],
});
```

Ages are measured back from `origin`, never the wall clock, so the same query against an
unchanged store ranks the same way twice. The penalty is *subtracted* from the score, so
it stays meaningful for a metric whose scores are negative or unbounded, and a record
with no usable timestamp is not penalized at all (`missing` defaults to `1`).

## Remembering and recalling (text-native)

When the server is started with an embedder (`nidus serve --embed-provider …`), you
can send **text** and let the server embed it, no need to compute vectors client-side.
`remember` embeds and upserts; `recall` embeds the query and vector-searches.

```ts
// Embed "the quick brown fox" and store it under id "a"
await db.remember("notes", "a", "the quick brown fox", { attrs: { tag: "x" } });

// Summarize first, then embed the summary (server also needs --summarize-provider).
// The stored record additionally carries a `nidus.summary` attr; the raw text is
// always stored under `nidus.text`.
await db.remember("notes", "b", longArticle, { mode: "summarize" });

// Expire in an hour, and fold this write onto any entry it is ≥0.95 similar to rather
// than storing a competing near-duplicate.
const { id, deduped } = await db.remember("notes", "c", "the quick brown fox", {
  ttlSeconds: 3600,
  dedupeThreshold: 0.95,
});

// Embed the query text and search, best-first (attrs decoded to plain JS values)
const hits = await db.recall("notes", "quick fox", {
  topK: 5,
  minScore: 0.2,
  filter: f.and(f.eq("tag", "x")),
});
```

`remember` resolves to `{ id, upserted, deduped }`. Read `id` from it rather than
assuming the one you passed: a `dedupeThreshold` match redirects the write onto the
entry it matched, and that entry's id is the one that changed. An already-expired
entry is never a dedupe candidate, so a TTL that has run out cannot be revived by a
later near-duplicate.

Both throw a `NidusError` with status `400` if the server has no embedder configured
(the message names `--embed-provider`); `mode: "summarize"` without a summarizer is
likewise a `400`. Dedupe needs that same embedder: it is a vector search under the
hood.

Pass `reinforce: true` to have a `recall` stamp `nidus.access_count` and
`nidus.last_accessed` on the hits it returns, and `extendTtlSeconds` to push an
existing `nidus.expires_at` out that many seconds (it never gives an expiry to an
entry that had none). Reinforcing makes the call a write, so it takes the server's
writer lock and is refused on a read-only server.

## Everything else

```ts
await db.collections();                  // string[]
await db.stats();                        // dimension, distance, ANN config, footprint
await db.list({ scope: ["docs"], filter: f.and(f.eq("lang", "rust")) });
await db.records("docs");                // every record, attrs decoded
await db.getMeta("docs"); await db.setMeta("docs", { owner: "search-team" });
await db.delete("docs", { ids: ["a"] });
await db.deleteWhere("docs", f.and(f.lt("year", 2000)));
await db.flush(); await db.compact();
await db.dropCollection("docs");
```

## Running in the browser (wasm)

A separate, ESM-only subpath, `@duckedup/nidus/wasm`, runs nidus itself inside the
browser via WebAssembly, storing data in the browser's Origin Private File System
instead of talking to a `nidus serve` over HTTP. It is lazily imported so the default
entry point above stays small.

```ts
import { acquireOpfsPool } from "@duckedup/nidus/wasm";
```

See https://nidus.duckedup.org for the full guide.

## Errors

A failed request throws a `NidusError` carrying the HTTP status the server reported,
so you can tell a client fault from a server fault:

```ts
import { NidusError } from "@duckedup/nidus";

try {
  await db.upsert("docs", records);
} catch (err) {
  if (err instanceof NidusError) {
    if (err.isBadRequest) {/* e.g. vector dimension mismatch */}
    if (err.isLocked) {/* the writer lock is held elsewhere (409) */}
    console.error(err.status, err.message);
  }
}
```

A status of `0` means a transport-level failure (the server was unreachable, or the
request timed out; configure `timeoutMs` on the client).

## License

MIT