/** * adapters/memory/s3Vectors — the corpus outlives the runtime, and can still * be updated without one. * * `sqliteVectorStore` (8.9.0) made a corpus survive a restart by putting it in * a file, and `staticVectorStore` (8.20.0) made it survive a runtime with no * disk at all by shipping it as a build artifact. Both leave the same gap, and * a production field report named it: **a bundle can only change when you * redeploy.** Add three documents on a Tuesday and the answer arrives with the * next release train. That is a fine trade for product documentation and a bad * one for anything a human edits during the day. * * Amazon S3 Vectors closes it. It is object storage with a native vector index * and a query API: durable, serverless, nothing to run, priced like storage * rather than like a database cluster. This adapter is the `MemoryStore` port * over it, and the two halves that matter are: * * - `search()` maps 1:1 onto **QueryVectors** — the vector goes as * `queryVector.float32`, `k` becomes `topK`, tiers become a metadata * `filter`, and the returned `distance` becomes the port's score. * - `put()` / `putMany()` map onto **PutVectors** — so `indexCorpus`, * `indexFolder` and `indexDocuments` run against it unchanged. **That is * the point of writing it rather than only reading it:** a corpus you can * add to from a cron job at 14:00 is a different product from one you can * add to at the next deploy. * * ── What it is NOT ────────────────────────────────────────────────────────── * A vector index is not a key-value store, and this adapter refuses the * operations that would need one rather than faking them: * * - `putIfVersion` — PutVectors is last-write-wins; there is no * compare-and-set. A read-then-write here would be a compare-and-set that * another writer can walk through the middle of, which is worse than none. * - `recordSignature` / `feedback` — a set of strings and a running average * are not vectors. `seen()` answers `false` and `getFeedback()` `null`, * which are both TRUE (nothing can be recorded, so nothing has been); the * WRITE halves refuse by name. * * Pair it with a second store for conversation memory — `defineRAG` for the * corpus, `defineMemory` for the chat, each with the backend it suits. That is * the same shape `staticVectorStore` documents. * * ── The index you must create first, and why this does not create it ──────── * A vector index has a DIMENSION and a DISTANCE METRIC fixed at creation, and * both are decisions about your embedder that this library must not make on * your behalf — creating one silently would pick a size and a metric your * corpus then lives with. Create it once, with your infrastructure: * * ```bash * aws s3vectors create-vector-bucket --vector-bucket-name my-corpus * aws s3vectors create-index \ * --vector-bucket-name my-corpus \ * --index-name docs \ * --data-type float32 \ * --dimension 1024 \ * --distance-metric cosine \ * --metadata-configuration '{"nonFilterableMetadataKeys":["af"]}' * ``` * * `nonFilterableMetadataKeys: ["af"]` is load-bearing, not decoration. This * adapter stores the whole entry — the passage the model will read, its * provenance, its timestamps — as JSON under the metadata key `af`. Filterable * metadata has a small per-vector budget; non-filterable metadata has the large * one. Declare `af` non-filterable and a full passage fits; leave it filterable * and PutVectors starts refusing your longer chunks partway through an * indexing run. * * Only `ns` (the identity namespace) and `tier` are ever filtered on, and both * are short strings. * * ── The preflight: this store now READS the index before trusting it (9.4.0) ─ * One `GetIndex`, lazily on first use and memoized for the life of the store. * It checks the three things this adapter used to assume — and that a * production deployment got wrong in all three ways at once: * * 1. the distance metric really is cosine (a euclidean index scored a * self-query at 0.999…, where a true cosine is exactly 1.0); * 2. `af` really is non-filterable, checked BEFORE the first write, so a * misconfigured index fails at document 1 rather than at document 400 * with an AWS "Filterable metadata must have at most 2048 bytes"; * 3. the index dimension matches the vectors being written or searched. * * This is the discipline every sibling store already had at open — * `sqliteVectorStore`'s schema identity, `pgVectorStore`'s schema check, * `staticVectorStore`'s load-time fingerprint. This store had nothing to open, * so it opened nothing. * * **IAM:** the caller now needs `s3vectors:GetIndex` alongside `PutVectors` / * `QueryVectors` / `GetVectors` / `ListVectors` / `DeleteVectors`. A GetIndex * that fails is refused by name, never passed through — an index this store * cannot read is an index whose metric, layout and dimension it would be * guessing at. * * ── Cosine only, said out loud ────────────────────────────────────────────── * The port's score is a cosine similarity, and every threshold in this library * — starting with `defineRAG`'s 0.7 default — is calibrated on that range. A * cosine distance converts exactly (`score = 1 - distance`). A EUCLIDEAN * distance does not: any mapping into [-1, 1] produces a number that READS like * a cosine and is not one, which is the same class of confident-and-meaningless * value the embedder-mismatch refusal exists to stop. So a euclidean index is * refused at construction, by name. * * ── The fingerprint guarantee, stated exactly ─────────────────────────────── * `sqliteVectorStore` owns its file and can keep a fingerprint row in it. This * store owns nothing but vectors, so the guarantee is assembled from what the * service actually provides, and it is smaller — so it is spelled out rather * than implied: * * - **Dimensions** are enforced by S3 Vectors itself. The index declares one; * a vector of another length is rejected by the service, loudly, at the * call. Nothing here can be sloppier than that. * - **The model id** is stamped into every vector's metadata (`fp`) and * checked in two places: against the fingerprint this process has already * seen for the namespace (at write and at search), and against the * fingerprint carried by the HITS that come back (at search). The second * one is what survives a restart: the first query after an embedder swap * sees documents stamped by the old embedder and refuses by name, instead * of returning a confident ranking of two incompatible spaces. * - Since 9.4.0 the index's OWN dimension is checked too, at the preflight — * so a wrong-size embedder is caught on the first write and the first * search of a fresh process, which the per-process fingerprint could not do * (it has nothing to disagree with at boot). * - What is still NOT caught: the first write of a fresh process into a * namespace built by a different embedder OF THE SAME SIZE. Only the model * id separates those, and it lives in the vectors rather than on the index. * There is no cheap read that would catch it, and a full index scan on * every boot is not one either. It is caught at the next search, before a * single wrong answer is returned. * * ── Lazy peer dependency ──────────────────────────────────────────────────── * `@aws-sdk/client-s3vectors` is an OPTIONAL peer dependency, required at * construction time. Importing `agentfootprint/memory` costs nothing for * consumers who never build one of these. Pass `client` to share the SDK * configuration your app already has. */ import type { MemoryIdentity } from '../../memory/identity/index.js'; import type { MemoryStore } from '../../memory/store/types.js'; /** * The slice of an S3 Vectors client this adapter calls. * * Structural, so the real SDK client, a pre-built one shared with the rest of * your app, or a test double all satisfy it without this package taking a hard * type dependency on the optional peer. */ export interface S3VectorsLikeClient { /** `send(command, options?)` — the second argument carries `abortSignal`. */ send(command: unknown, options?: { abortSignal?: AbortSignal; }): Promise; /** Optional — released by {@link S3VectorsStore.close} when this store built the client. */ destroy?(): void; } /** The constructors this adapter needs out of `@aws-sdk/client-s3vectors`. */ export interface S3VectorsSdkModule { readonly S3VectorsClient?: new (config: { region?: string; }) => S3VectorsLikeClient; /** The PREFLIGHT (9.4.0) — the one call that reads the index rather than its * vectors. See `verifyIndex` for what it checks and why each one is fatal. */ readonly GetIndexCommand?: new (input: unknown) => unknown; readonly PutVectorsCommand?: new (input: unknown) => unknown; readonly QueryVectorsCommand?: new (input: unknown) => unknown; readonly GetVectorsCommand?: new (input: unknown) => unknown; readonly ListVectorsCommand?: new (input: unknown) => unknown; readonly DeleteVectorsCommand?: new (input: unknown) => unknown; } export interface S3VectorsStoreOptions { /** The vector bucket (`vectorBucketName`). Created by you, not by this. */ readonly bucket: string; /** The vector index inside it (`indexName`). Created by you, not by this. */ readonly index: string; /** AWS region. Passed to the SDK client when this factory builds one. */ readonly region?: string; /** * The metric the index was created with. Only `'cosine'` is supported, and * anything else is refused at construction — see the header. This is a * DECLARATION about an index this store did not create; state the metric you * actually used. */ readonly distanceMetric?: 'cosine'; /** * How many vectors go in one PutVectors call. Default 100 — deliberately * well under the service limit, because the failure mode of guessing that * limit high is a corpus that indexes 90% of the way and stops. */ readonly batchSize?: number; /** A pre-built S3 Vectors client, so one SDK config serves the whole app. */ readonly client?: S3VectorsLikeClient; /** @internal Test injection — skips the SDK require entirely. */ readonly _client?: S3VectorsLikeClient; /** @internal Test injection — the AWS SDK module (exercises the real shim with a mock SDK). */ readonly _sdk?: S3VectorsSdkModule; } /** A durable vector index in S3, plus the two things this store owns beyond the port. */ export interface S3VectorsStore extends MemoryStore { /** The vector bucket this store reads and writes. */ readonly bucket: string; /** The vector index inside it. */ readonly index: string; /** * The embedder fingerprint (`'@'`) this PROCESS has seen for a * namespace, or `undefined` when it has seen none yet. * * Deliberately not "the fingerprint the index was built with" — see the * header for exactly what this store can and cannot know. It is populated by * the first write or the first search of the namespace in this process. */ fingerprintOf(identity: MemoryIdentity): string | undefined; /** * Release the SDK client, if this store built one. Idempotent. A client you * passed in is yours and is left alone. */ close(): void; } /** * Open a `MemoryStore` over an existing S3 Vectors index. * * @throws when `@aws-sdk/client-s3vectors` is absent and no `client` was passed. * @throws when `distanceMetric` is anything but `'cosine'`. * @throws EmbedderMismatchError from `put`/`putMany`/`search` when a vector * meets a namespace built by a different embedder. * * @example A corpus you can add to without a redeploy * ```ts * import { defineRAG, indexDocuments } from 'agentfootprint'; * import { s3VectorsStore } from 'agentfootprint/memory'; * import { bedrockEmbedder } from 'agentfootprint/providers'; * * const store = s3VectorsStore({ bucket: 'my-corpus', index: 'docs', region: 'us-east-1' }); * const embedder = bedrockEmbedder({ region: 'us-east-1' }); * * // Run this from a cron job. No deploy, no restart — the agent sees it next turn. * await indexDocuments(store, embedder, newDocs, { embedderId: embedder.id }); * * const agent = Agent.create({ provider }) * .rag(defineRAG({ id: 'docs', store, embedder, embedderId: embedder.id })) * .build(); * ``` */ export declare function s3VectorsStore(options: S3VectorsStoreOptions): S3VectorsStore;