# LLM Gateway A standardized interface for interacting with various Large Language Model (LLM) providers. ## Table of Contents - [Overview](#overview) - [Installation](#installation) - [Features](#features) - [Usage](#usage) - [Logger Interface](#logger-interface) - [Reporter Interface](#reporter-interface) - [Basic Setup](#basic-setup) - [Real-world Example](#real-world-example) - [Usage Examples](#usage-examples) - [Basic Text Completion](#basic-text-completion) - [Chat-based Assistance](#chat-based-assistance) - [Direct Image Analysis](#direct-image-analysis) - [Structured Output](#structured-output) - [Model Configuration](#model-configuration) - [File-based Assistance](#file-based-assistance) - [Speech-to-Text Transcription](#speech-to-text-transcription) - [Text-to-Speech Generation](#text-to-speech-generation) - [Token Counting for Context Management](#token-counting-for-context-management) - [Abort Signal Support](#abort-signal-support) - [API Reference](#api-reference) - [LLMProviders](#llmproviders) - [LLMPurposes](#llmpurposes) - [LLMServiceFactory](#llmservicefactory) - [LLMCompletionService](#llmcompletionservice) - [LLMAssistanceService](#llmassistanceservice) - [LLMSpeechToTextService](#llmspeechtotext-service) - [LLMTextToSpeechService](#llmtexttospeech-service) - [Prompt Builder](#prompt-builder) - [Supported Providers](#supported-providers) - [OpenAI](#openai) - [Google Generative AI](#google-generative-ai) - [Testing](#testing) - [Test Structure](#test-structure) - [Running Tests](#running-tests) - [Test Configuration](#test-configuration) - [Test Coverage](#test-coverage) - [Test Helpers](#test-helpers) - [Writing Custom Tests](#writing-custom-tests) - [Developer Guide](#developer-guide) - [Schema Architecture Overview](#schema-architecture-overview) - [Adding a New Provider](#adding-a-new-provider) ## Overview The `@mate-academy/llm-gateway` package provides a unified way to work with different LLM providers like OpenAI and Google Generative AI. It abstracts the implementation details of each provider, allowing you to easily switch between them without changing your application code. ## Installation ```bash npm install @mate-academy/llm-gateway ``` ## Features - Support for multiple LLM providers (OpenAI, Google Generative AI) - **Tracing port**: The gateway defines an `LLMGatewayTracer` port and captures generations, tool observations, usage, and cost automatically at every LLM call; fire-and-forget so a tracing failure never blocks or fails one. `@mate-academy/llm-tracer` provides the Langfuse-backed implementation; the gateway itself carries no tracing SDK. Tracing is opt-in per `LLMGateway` instance — `LLMServiceFactory` keeps working untraced. - **Structured Output**: Type-safe JSON responses with schema validation - **Extensible Schema Architecture**: Driver pattern with provider-specific adapters - **LLM Metrics & Reporting**: Comprehensive metrics collection with automatic cost calculation - **Agent invocation reporting**: `runAgent` exposes the numeric invocation id and its parent to reporter context builders for per-agent accounting. - **Cost Tracking**: Real-time cost calculation for all LLM operations with flexible pricing functions supporting text and audio tokens, including cached token discounts and reasoning tokens - **Cached Token Support**: Automatic detection and separate pricing for cached tokens (prompt caching) with provider-specific discount rates - **Instance Caching**: Intelligent SDK instance pooling with LRU eviction to prevent memory leaks - Standardized completion service interface - Standardized assistance service interface with file handling and direct chat - Speech-to-text transcription capabilities - Text-to-speech generation capabilities - Abort signal support for cancelling long-running operations - Lightweight token counting using character-based approximation (no heavyweight tokenizer dependencies) - Type-safe prompt templates with dynamic replacements - Factory pattern for easy provider selection - Consistent logging across all providers - Comprehensive testing suite with integration tests and shared test utilities - Full backward compatibility ## Usage ### Tracing The gateway defines the tracer port (`LLMGatewayTracer`) but carries no tracing SDK of its own. `@mate-academy/llm-tracer` provides `createLangfuseTracer`, the Langfuse-backed implementation of that port. Build an `LLMGateway` with it at the composition root (the consumer's DI container / serverless core) and take every traced service from that instance — feature code never touches Langfuse credentials, and the package installs no process-wide state: ```typescript import { LLMGateway } from '@mate-academy/llm-gateway'; import { createLangfuseTracer } from '@mate-academy/llm-tracer'; const tracer = createLangfuseTracer({ baseUrl: process.env.LANGFUSE_BASE_URL, publicKey: process.env.LANGFUSE_PUBLIC_KEY, secretKey: process.env.LANGFUSE_SECRET_KEY, environment: process.env.APP_ENV, // 'batched' for long-running processes, 'immediate' for Lambda exportMode: 'batched', }); const llmGateway = new LLMGateway({ tracer }); const completionService = llmGateway.getCompletionService({ provider: LLMProviders.OpenAI, logger, reporter, }); ``` `LLMServiceFactory`'s static methods stay available and stay untraced, so call sites that know nothing about tracing keep working unchanged. Building `new LLMGateway()` without a tracer is equally valid — it hands out services wired to the no-op tracer, so nothing breaks before rollout. `createLangfuseTracer` likewise falls back to a no-op when credentials are missing. Every completion and speech-to-text call made through a gateway-built service emits exactly one `generation` observation (prompt/completion content, model, params, usage, cost) at the same success/error points the metrics reporter already wraps — one on success, one (level `ERROR`) on failure. An assistance call emits one `generation` per **model round** of its tool loop, each carrying the conversation as it stood for that round and that round's assistant output, with the round's own usage and cost. Inside an active trace scope, a legacy `assistInChat` caller that runs an agent-style tool loop can pass `agentName` to group all of its generation and tool observations under one `agent` observation. The v2 `runAgent` API opens that agent observation automatically. The run still writes a single aggregate metrics point, unchanged. Calls that carry no per-call trace context (text-to-speech, file and chat management) emit none, on either branch. Observation input/output are recorded in full — the prompt and the completion are what a trace exists to show, so they are never truncated. All of this is fire-and-forget — a tracing failure never blocks or fails an LLM call. Tool observations use the tool definition's bare `name`. This is a breaking change for consumers that filter on the former `AssistanceService/tool:` observation names. A tool may declare `tracing.observation` as `LLMToolTracingObservation.Tool`, `.Retriever`, or `.None`; the default is `.Tool`. `.None` is intended for delegation tools whose execution already opens an agent observation. Unknown and gate-denied calls are still traced, at warning level, with exactly the result returned to the model. Failed calls likewise record the same `Error executing tool '': ...` message the model receives. Use `tracing.mapInput` when a tool's provider-facing arguments are intentionally encoded differently from the shape people should inspect in a trace. The mapper changes only the recorded observation input: `canExecute` and `execute` still receive the original arguments. If the mapper throws, tracing falls back to the original input and tool execution continues normally. Every agent-loop generation carries a `tools` metadata field containing the serialized definitions offered in that round. A delegated run uses its agent name consistently for its agent observation and generation descendants, and its tools nest below that single agent observation rather than a duplicate delegation-tool observation. Pass `traceContext.prompt` (`{ name, version, isFallback }`) to name the observation after the managed prompt and link it to that prompt version in Langfuse; a fallback prompt intentionally produces no link. The v2 client builds this context for every call it makes, so `generate` and `runAgent` need nothing at the call site. `runAgent` also passes an optional `invocation` object to the reporter context builder. It identifies the root agent or delegated subagent within the current agent tree; ordinary `generate` calls leave it absent. `userId`, `sessionId` and `tags` travel as typed fields on `LLMCallTraceContext` and on the generation record, never as metadata keys: they are first-class Langfuse trace attributes, and the tracer promotes them to the trace itself. The per-call context bag stays the metadata path, so a user named in it still appears in metadata exactly as before. A generation records the same provider-native message array on both branches: each service prepares its request messages before the provider call is attempted, so a failed round renders as a role-labelled transcript instead of a raw `{ message, history }` object. Preparing those messages is local work, so its own failure is a configuration error and propagates rather than being folded into the provider's error result. An assistance run that fails before its first round ever starts has no transcript to show, and falls back to a single error generation carrying the call's message. `LLMGatewayTracer.getActiveTraceContext()` returns the context the surrounding scope was opened with (`undefined` outside a scope). The v2 client passes it to `reporting.buildContext` as `scopeContext`, so a composition root can fall back to what the operation declared when a call names nothing of its own. The observation also carries its real time span. Each provider service captures `startedAt` (epoch ms) before the provider call and the base service stamps `endedAt` when it hands the record to the tracer, so the exported generation has the actual call latency instead of a zero-width span. Both fields are required on `LLMGenerationTraceRecord`, and `startedAt` is required on the `LLMGenerationTrace` a provider builds — a new provider call site cannot compile without threading a start timestamp. To group several calls under one trace, wrap the work so nested gateway calls attach to it automatically: ```typescript await llmGateway.withTrace( 'my-operation', { metadata: { feature: 'city-prediction' } }, async () => { // gateway calls here nest under the trace }, ); ``` Short-lived processes (Lambdas, scripts) must flush before exit, since buffered events are lost otherwise: ```typescript await llmGateway.flush(); // safe no-op when tracing is disabled ``` The CloudWatch fallback-serving metric (`LangfuseFallbackServed`, namespace `LLMGateway`) is emitted through `llmGateway.emitFallbackServed(...)`; pass a real emitter as `fallbackMetricEmitter` when building the gateway (it defaults to a no-op so the package carries no AWS dependency). To assert emitted observations in tests, build the gateway with a test double implementing `LLMGatewayTracer` and assert what it captured (see `@mate-academy/llm-tracer`'s README for asserting real Langfuse spans). ### v2 Client (`createLLMClient`) The v2 client is the prompt-name-first call surface. A call site names the operation (the Langfuse prompt name); everything else — prompt text, model, provider, params, credentials, reporter wiring, tracing, structured output — resolves from it inside the package. Build it once at the composition root and inject it via DI; feature code never sees configuration or credentials. Live prompt fetching is behind the `LLMGatewayPromptClient` port, not a Langfuse SDK client: the gateway carries no `@langfuse/*` dependency of its own. `@mate-academy/prompt-client`'s `LLMPromptClient` satisfies this port structurally — pass its `promptClient` straight through. Omitting `promptClient` runs the client on the build-time snapshot alone (the same no-op-by-omission story as the tracer port). ```typescript import { createLLMClient, defineLLMPrompts, LLMSchema, } from '@mate-academy/llm-gateway'; import { LLMPrompt, langfusePromptSnapshot } from './langfusePrompts.generated'; const llmPromptRegistry = defineLLMPrompts({ [LLMPrompt.CityPredictionPredictCityName]: { schema: LLMSchema.object({ cityName: LLMSchema.string().nullable() }), variables: ['locationName', 'countryName'], }, }); const llm = createLLMClient({ registry: llmPromptRegistry, snapshot: langfusePromptSnapshot, credentials: async ({ provider, promptName, product }) => providerOptions, reporting: { reporter }, logger, context: { product, appEnvironment }, mode: 'production', promptClient, // an `@mate-academy/prompt-client` bundle's `promptClient`, or omit for snapshot-only }); const { cityName } = await llm.generate( LLMPrompt.CityPredictionPredictCityName, { variables: { locationName, countryName }, context: { userId } }, ); ``` - **Typed end to end.** The client is generic over `typeof registry`. Required variable names come from each binding's code-owned `variables` list, while the return type is the bound schema's inferred type, or `string` when no schema is bound. Langfuse prompt edits never change TypeScript types. - **The reporter's context is type-tied to its mapper.** `createLLMClient`'s third type parameter is the reporter context, so a `reporter` typed `LLMReporterInterface` forces `reporting.buildContext` to return exactly `T` — a mapper that drops or mistypes a reported column fails the build instead of corrupting the dashboard downstream of it. It is inferred when the call site passes no explicit type arguments; TypeScript has no partial inference, so a call site that spells out `Registry`/`VariableMap` must spell the context out too: `createLLMClient({ … })`. - **Product is per call when the caller knows better.** A client is built once per process, so its `context.product` can only carry the product of whatever built it — a request host, or nothing at all on a message-bus or WebSocket entry point. Pass `context: { product }` to `generate`/`runAgent` and it wins over the static one for credentials resolution, and reaches `buildContext` for the reported row. - **Sessions and tags are top-level call options.** Pass `sessionId` to `generate`/`runAgent` to group the traces of one conversation, review or agent run into a single Langfuse session (`runAgent` copies it onto delegated subagent runs), and `traceTags` for Langfuse trace tags. Both are typed Langfuse trace fields rather than metadata, and the anchor format `::` must stay stable — changing it splits a session's history in two. - **Config validation on every fetch.** The client checks `config`, provider, model, param ceilings, and the resolved model's capabilities against registry `requires`. Any failure serves the last-known-good snapshot and emits the `LangfuseFallbackServed` metric; prompt variables remain a manually maintained contract between the registry and Langfuse. - **Errors throw with a taxonomy** (`LLMGatewayError` base + `LLMProviderError`, `LLMSchemaValidationError`, `LLMAbortedError`, `LLMConfigError`); validated `data` is the return value, never an `'error' in response` union. - **Override gating.** `mode: 'production'` rejects per-call `overrides` with `LLMConfigError`; only `mode: 'experiment'` (harness/playground/tests) honors them. - **Chat-type prompts.** A prompt seeded as a Langfuse `chat` prompt binds with `kind: 'chat'`. The runtime produces a role-tagged message array with variables substituted across every message; at call time a leading `system` message maps to the provider's instructions slot and the final `user` message is the request, preserving the message structure the prompt declares. - **Per-call schema override.** Pass `schema` in the `generate` options to override the registry binding's schema for one call; the return type follows the call-site schema. Use it when the output contract is dynamic (e.g. a schema whose shape depends on runtime input). A binding may omit `schema` when callers always pass one. The schema is the code-owned output contract, honored in every mode. - **`describePrompt(name)`.** Read-only accessor returning the `{ provider, model, promptVersion, isFallback }` the runtime would serve right now (same resolve/cache/fallback path as `generate`, no provider call). Use it where analytics events need the resolved model name — keeping it off `generate`'s data-only return. - **`compilePrompt(name, { variables })`.** Compiles a prompt to text instead of generating from it — for a sub-prompt injected as a variable into a parent prompt. An outage serves the snapshot rather than throwing, and no provider is called, so a compile produces no usage row. Two traps: Mustache sections are not evaluated, so conditional content needs its own sub-prompt (compiled, or `''`); and a sub-prompt whose `config` names no real model is skipped by the snapshot codegen and left with no fallback. ```typescript const generalInstructions = await llm.compilePrompt(SECTION_PROMPT, { variables: { aiContextSection }, }); await llm.generate(WELCOME_PROMPT, { variables: { generalInstructions } }); ``` #### Agent runs (`runAgent`) `llm.runAgent` runs a tool-calling agent whose instructions are a registry prompt: the Langfuse prompt text is the agent's system instructions and its `config` routes the model, so an agent is versioned and re-routed exactly like any other operation. The provider assistance loop executes the model⇄tool rounds; subagents are exposed to the model as tools (one level deep — a subagent cannot declare its own subagents). ```typescript const turn = await llm.runAgent(LLMPrompt.ContentEditorRootAgent, { variables: { courseName }, input: userMessage, history: persistedMessages, // resume from stored session state tools: [createTopicTool, updateTheoryTool, askUserTool], subagents: [{ prompt: LLMPrompt.ContentEditorResearchSubagent, name: 'researcher', description: 'Researches sources for course content', variables: { courseName }, tools: [searchTool], reportCap: { maxLength: 8192, truncationNote: '\n\nCheck the persisted artifact.', }, // Optional: flags this specialist's report when it never searched. grounding: { groundingToolNames: ['web_search', 'web_fetch'], ungroundedReportNote: '[UNGROUNDED - no web access was used.]\n\n', }, }], context: { userId }, // Optional: omitted for backward-compatible retry/delegation behaviour. failurePolicy: { identicalToolCallNudgeAfter: 2, identicalToolCallAbortAfter: 5, }, onEvent: (event) => publishAndPersist(event), // WS updates + run history }); if (turn.stopReason === LLMToolLoopStopReasons.TerminalTool) { await askTheUser(turn.terminalTool.output); // agent handed control back } else { await applyStructure(turn.output); // the model's answer } ``` `failurePolicy` is opt-in. It tracks consecutive failures of the exact same tool name and canonicalized input per agent run, adds a corrective hint at the configured nudge threshold, and stops only the affected delegation at the abort threshold. Only a call that ran and produced a result resets that tool's streak: a refused call and a delegation the policy stopped both keep it running, and both carry the escalation in the text the model reads. The thresholds are validated, so a nudge that could never fire is a configuration error rather than silence. `reportCap` is declared per subagent, because how much prose a specialist may hand back belongs to that delegation's contract — a specialist that names every entity it touched legitimately reports more than one summarising a document it persisted elsewhere. A capped text report is cut on a code-point boundary and the note is appended; a capped structured report is replaced by a valid JSON envelope carrying an excerpt, never by half a JSON document. A subagent's `grounding` is opt-in too, and it is a counter over the trajectory rather than a reading of the report: the runner already dispatches every call the specialist makes, so a delegation that executed none of `groundingToolNames` gets `ungroundedReportNote` prepended to the report the parent model reads. A refused or failed call does not ground anything — only an execution does. A prose report is noted before it is capped, so a long report cannot bury the warning; a structured one is noted after, because its cap replaces the document with an envelope that has to stay whole. Declaring `grounding` on a subagent whose report is structured therefore means an ungrounded delegation returns note-then-JSON rather than JSON — capped or not — so do not declare it on a subagent whose delegation result you parse. - **Structured progress events.** Every step emits an `LLMAgentEvent` (`agent_started/step_started/message/completed/failed`, `tool_call_started/completed/failed/rejected`, `subagent_started/completed/failed`) through `onEvent`, for live subscription updates and durable run persistence. Events carry correlation ids from one sequence per root run — `runId` on every event, `invocationId` on started, completed and failed tool and subagent events, and `subagentRunId` linking a delegation to the nested run's events — so parallel tool calls and repeated delegations reconstruct into an unambiguous run tree. A `tool_call_rejected` event is the explicit exception: it has no `invocationId`, because the provider call never entered the wrapped tool lifecycle and not every provider exposes a stable call identifier. Listeners may be async; a throw or rejection is logged and never affects the run. `agent_step_started` is emitted immediately before every provider model call, including failed calls and nested subagent rounds. `tool_call_rejected` preserves the provider's ordered call when its name is unknown or its JSON arguments cannot be parsed, even though no wrapped tool invocation starts. - **Full-fidelity history.** `history` restores the conversation as it happened, not a prose retelling of it. An entry is a plain message (`{ role, text }`), a past tool round (`{ type: LLMHistoryEntryTypes.ToolCall, calls, text? }`), the result that answered one of its calls (`{ type: LLMHistoryEntryTypes.ToolResult, toolCallId, result }`), or mid-conversation context (`{ type: LLMHistoryEntryTypes.Context, text }`). Each maps to the provider's own representation, so a resumed run reads its own past calls as calls. The union is additive — an existing `LLMPlainMessage[]` is already a valid history and maps exactly as before. ```typescript await llm.runAgent(agent, { variables, input: userMessage, history: [ { role: 'user', text: 'Rewrite the flaky-selectors theory' }, { type: LLMHistoryEntryTypes.Context, text: carriedPlan }, { type: LLMHistoryEntryTypes.ToolCall, text: 'Looking it up.', calls: [{ id: 'call-1', name: 'find_content', arguments: { query: 'flaky selectors' }, }], }, { type: LLMHistoryEntryTypes.ToolResult, toolCallId: 'call-1', result: 'theory-1', }, ], }); ``` Per provider: LLMAPI replays a round as an assistant message carrying `tool_calls` plus `tool`-role results and context as a mid-conversation `system` message; OpenAI replays `function_call` / `function_call_output` items and context as a `developer` item; Gemini replays a `model` turn of `functionCall` parts answered by a `user` turn of `functionResponse` parts, and — having no mid-conversation system role — carries context as a user turn. On Gemini a restored call carries no `thought_signature`, so it replays under the documented validator-bypass marker, and a multi-call round is split into sequential single-call turns exactly as the live loop records one; a call that kept its signature passes it in `signature` and it is echoed verbatim. Where the restored conversation then lives differs by provider, and no provider spends a generation to materialize it. LLMAPI keeps the converted messages in local chat state and resends the whole array each round; Gemini holds them in the SDK chat session; OpenAI writes them to a Responses **conversation object** (`POST /v1/conversations`), a non-generative call, and every round of that chat then passes `conversation: ` — so the chat id *is* the conversation id, `deleteChat` deletes it server-side, and reasoning items stay in the chain across tool rounds exactly as they did under `previous_response_id`. `conversation` and `previous_response_id` are mutually exclusive; the package uses the former. A call the history leaves unanswered is answered by the package with a real tool result carrying `LLM_UNANSWERED_TOOL_CALL_RESULT` — every provider rejects an unanswered call, so the repair is native rather than left to the caller to fake in prose. A result naming no preceding call, a second result for an already-answered call, and a round that called nothing are dropped. Same-role entries need no merging: the contract carries the structure. - **Intermediate narration.** When the model returns visible assistant text alongside tool calls in a round that will continue, that text is emitted as `agent_message` (attributed to the emitting run, so a subagent's narration carries the subagent's identity) before the round's tools run. Reasoning / thinking content and the final answer are never sent this way — the final answer is `agent_completed.output`. - **Terminal tools end the turn.** A tool declaring `terminal: true` (an `ask_user` tool, say) hands control back to the application. The round it appears in still runs in full — every call of that round, including parallel ones, executes and emits its `tool_call_*` events — but the results are not fed back and no further model round starts. The run then returns `stopReason: 'terminal_tool'` with `terminalTool` carrying the tool's name and output; if several terminal calls land in one round, the first in the round's order is the outcome (all still execute). This is ordinary control flow — no exception, no abort — and `maxToolIterations` is untouched. A terminal tool that throws still ends the turn, carrying its failure message as the outcome. Delegation tools are never terminal, so a subagent that stops on its own terminal tool returns that output as its delegation result and the parent's loop continues. - **Subagent names are tool names.** A subagent's `name` must match `[a-zA-Z0-9_-]+` (it is exposed to the model as a tool); an invalid name fails the run upfront with `LLMConfigError` instead of being silently rewritten. - **Typed delegation inputs.** A subagent declaring an `inputSchema` is exposed to the model with that schema instead of the default single `prompt` string, and the arguments the parent model fills join the subagent's own `variables` (winning over them), so its prompt can reference them as `{{topicRef}}` and the validation gate checks them like any other variable. Use it whenever the delegated work needs identifiers the parent must not paraphrase — an entity ref, a changeset id, an enum choice. Without `inputSchema` nothing changes: one free-form `prompt` argument, which is also the nested run's input. ```typescript const theoryWriter = llm.defineAgent({ prompt: LLMPrompt.ContentEditorWriteTheory, name: 'write_theory', description: 'Authors one theory document into the current changeset', inputSchema: LLMSchema.object({ topicRef: LLMSchema.string().describe('Changeset entity ref of the topic'), audienceLevel: LLMSchema.enum(['beginner', 'intermediate', 'advanced']), outline: LLMSchema.array(LLMSchema.string()), }), variables: { changesetId }, tools: [getEntityTool, saveTheoryTool], }); ``` - **One place normalizes tool output.** Every tool result and every delegation result passes through the package's coercion before the model sees it, so a tool that returns an object, a number or nothing cannot put `[object Object]` into the conversation. A client may add its own `normalizeToolResult` on top — the natural place to wrap untrusted material (a fetched web page) so no tool author can forget to: ```typescript const llm = createLLMClient({ // ... normalizeToolResult: ({ tool, result }) => ( UNTRUSTED_TOOLS.has(tool) ? { content: `\n${toText(result)}\n` } : result ), }); ``` The normalized value is what reaches both the model and the `tool_call_completed` event, so a persisted run history records exactly what the model read. - **Preconditions gate a call before it runs.** A tool may declare `canExecute(args)` returning `{ allowed: true }` or `{ allowed: false, reason }`. It is checked before `execute`, so a refused call never has its side effect. A refusal is ordinary control flow, not an error: the model is told the reason as that call's result, the run continues, and the iteration budget is untouched — so the model can ask the user or take another path. The invocation reports `tool_call_denied` instead of `tool_call_completed`, and a refused delegation reports the same event with `tool` naming the subagent. Use it for permission, quota and state checks that depend on the arguments (an authorization check before a write, a stale-hash guard, a per-run write budget) — enforcement then lives in the tool's definition rather than in prompt instructions the model may ignore. The plain-object form above is complete on its own. `llm.defineAgent` is optional sugar for reusable, composable definitions: it returns an immutable `LLMAgent` instance (typed against the registry) that `runAgent` accepts in place of a prompt key, and that other agents accept as a subagent. Unlike plain-object subagents, `LLMAgent` subagents may declare their own `subagents` — instances are immutable, so a delegation cycle cannot be constructed. Agent-level `variables` are a base merged under the per-run `variables`; a subagent used as a delegation tool must carry `name` and `description`. ```typescript const researcher = llm.defineAgent({ prompt: LLMPrompt.ContentEditorResearchSubagent, name: 'researcher', description: 'Researches sources for course content', variables: { courseName }, tools: [searchTool], }); const rootAgent = llm.defineAgent({ prompt: LLMPrompt.ContentEditorRootAgent, tools: [createTopicTool, updateTheoryTool], subagents: [researcher], }); const turn = await llm.runAgent(rootAgent, { variables: { courseName }, input: userMessage, context: { userId }, onEvent: (event) => publishAndPersist(event), }); ``` - **Typed final output.** A run returns a result discriminated by `stopReason`. For an answered turn (`completed`, or `max_iterations`) the prompt's binding drives `output` exactly like `generate`: the bound schema's inferred type, or the final assistant text when no schema is bound. A `terminal_tool` turn has no model answer, so the bound schema is deliberately not applied — `output` is absent and `terminalTool` is the outcome instead. Both carry `text`, the visible assistant text of the final round. - **Tool-capability gate.** When a run has tools or subagents, the validation gate additionally requires the resolved model to declare the `tools` capability, so a Langfuse re-route to a tool-less model serves the fallback instead of failing mid-run. - **Langfuse for free.** Provider services already record generation and tool observations under the active trace; wrap the run with the tracer's `withTrace` at the call site to group the whole agent session. - Overrides, credentials, reporter context, and the error taxonomy behave exactly as in `generate`. #### Prompt snapshot codegen `langfusePrompts.generated.ts` is produced by `npm run langfuse:generate` (root), which fetches every labeled prompt and emits only the `LLMPrompt` enum and the snapshot used as the outage fallback. Variable types are not inferred from Langfuse; they stay in the application registry. The file is gitignored exactly like GraphQL generated files: regenerated locally on demand, in CI before image builds, and refreshed non-blocking at API pod boot. The generated declarations are built with ts-morph through its structural API, not by string concatenation. Ts-morph is a dependency of the `codegen` entry point only, so importing the package's main entry never pulls it in. `generateSnapshot` (exported from `@mate-academy/llm-gateway/codegen`) takes an injected `LLMPromptCatalog` instead of constructing a Langfuse SDK client itself — the caller owns the catalog's credentials and its Langfuse project. `@mate-academy/prompt-client`'s `LLMPromptClient` satisfies this port too, so the same instance a runtime composition root builds can be reused for codegen: ```typescript import { generateSnapshot } from '@mate-academy/llm-gateway/codegen'; await generateSnapshot({ catalog: promptClient, // an `@mate-academy/prompt-client` bundle's `promptClient` label: 'production', outputPath: './src/langfusePrompts.generated.ts', onLog: (message) => console.log(message), }); ``` #### Escape hatch `@mate-academy/llm-gateway/advanced` re-exports `LLMServiceFactory`, the provider services, and the model maps for the rare call site needing raw control. It bypasses prompt management, the validation gate, and config-in-prompt routing — prefer the v2 client. ### Logger Interface The package accepts an optional logger that implements the `LLMLoggerInterface` interface. Most logging libraries are compatible (`@mate-academy/logger`, winston, pino, etc.). If no logger is provided, no logging will occur. ```typescript // Using @mate-academy/logger (recommended) import { logger } from '@mate-academy/logger'; // Or simple console logger const logger = { info: (msg, meta) => console.log(msg, meta), error: (msg, meta) => console.error(msg, meta), warn: (msg, meta) => console.warn(msg, meta), child: (context) => logger, }; ``` **Available Metrics:** The reporter automatically collects the following metrics: ```typescript interface LLMMetrics { // Provider & Model Information provider: 'OpenAI' | 'GoogleGenerativeAI'; purpose: 'completion' | 'assistance' | 'speech_to_text' | 'text_to_speech'; model: string | null; modelConfig: Record | null; // Method Context method: string; // e.g., 'sendMessage', 'assistInChat', 'transcribe' status: 'success' | 'error' | 'cancelled'; // Token Usage tokens: { input: number; // Regular input tokens (excludes cached tokens) output: number; // Output tokens (includes reasoning tokens for o1/o3 models) total: number; // Total tokens (input + output) }; // Cost Tracking (automatically calculated with cached token discounts) costs: { input: number; // Cost for input tokens (text + audio), with cached token discounts applied output: number; // Cost for output tokens (text + audio + reasoning tokens) total: number; // Total cost currency: string; // 'USD' }; } ``` **TypeScript Behavior:** The `reporterContext` parameter behavior depends on your reporter configuration: ```typescript // Case 1: Reporter with defined context type - reporterContext is REQUIRED const reporterWithContext: LLMReporterInterface<{ userId: string }> = new MyReporter(); // Case 2: Reporter without context - reporterContext is OPTIONAL const reporterWithoutContext: LLMReporterInterface = new SimpleReporter(); // Case 3: No reporter - reporterContext is OPTIONAL const service = LLMServiceFactory.getCompletionService({ provider, options // no reporter }); ``` ### Basic Setup ```typescript import { LLMProviders, LLMServiceFactory, LLMCompletionService, LLMAssistanceService, LLMSpeechToTextService, LLMTextToSpeechService, LLMRoles, LLMMessageContentType, LLMUploadFileMimeTypes, LLMCompletionMessage, LLMModel, LLMSchema, createPromptTemplate, LLMLoggerInterface, LLMReporterInterface, } from '@mate-academy/llm-gateway'; // Define provider options const llmProviderOptions = LLMServiceFactory.resolveProviderOptions( LLMProviders.OpenAI, // chosen provider { [LLMProviders.OpenAI]: { apiKey: 'your-openai-api-key', organization: 'your-organization-id', // optional baseURL: 'https://api.openai.com/v1', // optional }, [LLMProviders.GoogleGenerativeAI]: { apiKey: 'your-google-ai-api-key', }, }, ); // Get completion service const completionService = LLMServiceFactory.getCompletionService({ provider: LLMProviders.OpenAI, options: llmProviderOptions, // optional, but recommended - otherwise must be set later via service.setOptions() logger, // optional - omit for no logging reporter, // optional - omit for no metrics collection }); // Get assistance service const assistanceService = LLMServiceFactory.getAssistanceService({ provider: LLMProviders.OpenAI, options: llmProviderOptions, // optional, but recommended - otherwise must be set later via service.setOptions() logger, // optional - omit for no logging reporter, // optional - omit for no metrics collection }); // Get speech-to-text service const speechToTextService = LLMServiceFactory.getSpeechToTextService({ provider: LLMProviders.OpenAI, options: llmProviderOptions, // optional, but recommended - otherwise must be set later via service.setOptions() logger, // optional - omit for no logging reporter, // optional - omit for no metrics collection }); // Get text-to-speech service const textToSpeechService = LLMServiceFactory.getTextToSpeechService({ provider: LLMProviders.OpenAI, options: llmProviderOptions, // optional, but recommended - otherwise must be set later via service.setOptions() logger, // optional - omit for no metrics collection reporter, // optional - omit for no metrics collection }); ``` ### Real-world Example Here's how you might integrate the LLM Gateway in a use case class: ```typescript import { LLMProviders, LLMServiceFactory, LLMCompletionService, LLMAssistanceService, LLMSpeechToTextService, LLMTextToSpeechService, } from '@mate-academy/llm-gateway'; class MyUseCase { private llmCompletionService: LLMCompletionService; private llmAssistanceService: LLMAssistanceService; private llmSpeechToTextService: LLMSpeechToTextService; private llmTextToSpeechService: LLMTextToSpeechService; constructor(logger, config) { const llmProviderOptions = LLMServiceFactory.resolveProviderOptions( config.llmProvider, { [LLMProviders.OpenAI]: { apiKey: config.openAIApiKey, organization: config.openAIOrgId, baseURL: config.openAIBaseUrl, }, [LLMProviders.GoogleGenerativeAI]: { apiKey: config.googleAIApiKey, }, }, ); this.llmCompletionService = LLMServiceFactory.getCompletionService({ provider: config.llmProvider, options: llmProviderOptions, logger, reporter, }); this.llmAssistanceService = LLMServiceFactory.getAssistanceService({ provider: config.llmProvider, options: llmProviderOptions, logger, reporter, }); this.llmSpeechToTextService = LLMServiceFactory.getSpeechToTextService({ provider: config.llmProvider, options: llmProviderOptions, logger, reporter, }); this.llmTextToSpeechService = LLMServiceFactory.getTextToSpeechService({ provider: config.llmProvider, options: llmProviderOptions, logger, reporter, }); } async processRequest(prompt) { // Use completion service const completion = await this.llmCompletionService.sendMessage({ message: { role: LLMRoles.User, content: [ { type: LLMMessageContentType.TEXT, text: prompt, } ], }, model: this.getPreferredModel(), }); return completion; } async transcribeAudio(audioPath) { // Use speech-to-text service const transcription = await this.llmSpeechToTextService.transcribe({ pathToAudio: audioPath, model: this.getPreferredModel(), }); return transcription; } async generateSpeech(text) { // Use text-to-speech service const speech = await this.llmTextToSpeechService.createSpeech({ text: text, model: this.getPreferredModel(), speechOptions: { voice: 'alloy', // OpenAI voice option response_format: 'mp3', }, }); return speech; } private getPreferredModel() { // Get the appropriate model from the service's available models const models = Object.values(this.llmCompletionService.models); return models[0]; // Use the first available model } } ``` ### Usage Examples #### Basic Text Completion ```typescript const result = await completionService.sendMessage({ message: { role: LLMRoles.User, content: [ { type: LLMMessageContentType.TEXT, text: 'Hello, how are you?', } ], }, model: preferredModel, reporterContext: { // required if reporter is configured with context type userId: 12345, feature: 'chat', environment: 'production', }, }); console.log(result.text); // AI response ``` #### Chat-based Assistance ```typescript // Create a chat session const chat = await assistanceService.createChat({ model: preferredModel, instructions: 'You are a helpful coding assistant.', }); // Send messages directly to the chat const result = await assistanceService.assistInChat({ model: preferredModel, message: { role: LLMRoles.User, content: [{ type: LLMMessageContentType.TEXT, text: 'Help me understand React hooks', }], }, chatId: chat.chatId, reporterContext: { // required if reporter is configured with context type userId: 12345, feature: 'coding_assistance', environment: 'production', }, }); console.log(result.text); // Assistant response ``` #### Direct Image Analysis ```typescript // Send image directly in message content (OpenAI) const result = await assistanceService.assistInChat({ model: preferredModel, message: { role: LLMRoles.User, content: [ { type: LLMMessageContentType.TEXT, text: 'What do you see in this image?', }, { type: LLMMessageContentType.IMAGE_URL, image_url: { url: 'https://example.com/image.png', detail: 'high', }, }, ], }, chatId: chat.chatId, }); console.log(result.text); // Image analysis response ``` ### Structured Output The LLM Gateway supports structured output, allowing you to request type-safe, validated JSON responses from LLM providers. Built on Zod v4 with native JSON Schema generation for optimal performance. #### Basic Structured Output ```typescript import { LLMSchema } from '@mate-academy/llm-gateway'; // Define your output schema const personSchema = LLMSchema.object({ name: LLMSchema.string(), age: LLMSchema.number().min(1), email: LLMSchema.string().email(), isActive: LLMSchema.boolean(), }); // Request structured output const result = await completionService.sendMessage({ message: { role: LLMRoles.User, content: [{ type: LLMMessageContentType.TEXT, text: 'Create a person profile for John Smith, 30 years old, email john@example.com' }] }, model: preferredModel, schema: personSchema // Add schema for structured output }); // Type-safe access to structured data if (result.data) { console.log(result.data.name); // string console.log(result.data.age); // number console.log(result.data.email); // string console.log(result.data.isActive); // boolean } // Always available as text fallback console.log(result.text); ``` #### Complex Schema Example ```typescript const analysisSchema = LLMSchema.object({ sentiment: LLMSchema.enum(['positive', 'negative', 'neutral']), confidence: LLMSchema.number().min(0).max(1), keywords: LLMSchema.array(LLMSchema.string()), summary: LLMSchema.string(), metadata: LLMSchema.object({ processedAt: LLMSchema.string(), modelVersion: LLMSchema.string(), }), }); const result = await completionService.sendMessage({ message: { role: LLMRoles.User, content: [{ type: LLMMessageContentType.TEXT, text: 'Analyze this text: "I love this new feature!"' }] }, model: preferredModel, schema: analysisSchema }); // Fully type-safe access if (result.data) { console.log(result.data.sentiment); // 'positive' | 'negative' | 'neutral' console.log(result.data.confidence); // number (0-1) console.log(result.data.keywords); // string[] console.log(result.data.summary); // string console.log(result.data.metadata.processedAt); // string } ``` #### Schema API Reference The `LLMSchema` builder provides a fluent API for defining output schemas: **Basic Types:** - `LLMSchema.string()` - String values - `LLMSchema.number()` - Numeric values - `LLMSchema.boolean()` - Boolean values - `LLMSchema.array(itemSchema)` - Arrays of items - `LLMSchema.object({ ... })` - Object with specified properties - `LLMSchema.enum(['option1', 'option2'])` - Enumerated values - `LLMSchema.literal('exact_value')` - Exact literal values **Modifiers:** - `.optional()` - Makes field optional and nullable (compatible with all providers) - `.nullable()` - Allows null values - `.default(value)` - Sets default value - `.describe(text)` - Adds description **String Modifiers:** - `.min(length)` - Minimum string length - `.max(length)` - Maximum string length - `.email()` - Email validation - `.url()` - URL validation - `.uuid()` - UUID validation **Number Modifiers:** - `.int()` - Integer values only - `.positive()` - Positive numbers (Note: Use `.min(1)` for better OpenAI compatibility) - `.negative()` - Negative numbers - `.min(value)` - Minimum value - `.max(value)` - Maximum value #### Error Handling ```typescript const result = await completionService.sendMessage({ message: { /* ... */ }, model: preferredModel, schema: mySchema }); if (result.data) { // Successfully parsed structured output console.log('Structured data:', result.data); } else if (result.parseError) { // Parsing failed, but text is still available console.error('Parse error:', result.parseError); console.log('Raw text:', result.text); } else if (result.error) { // Request failed entirely console.error('Request error:', result.error); } ``` #### Token Usage and Cost Successful completion and transcription results expose two optional fields: `usage` (`LLMModelUsage`) with the provider-reported token counts, and `cost` (`LLMCostsResult`) with the absolute USD cost for the call, derived from that usage and the model's pricing. Both are present only on the success branch (absent when `result.error` is set), so read them after narrowing: ```typescript const result = await completionService.sendMessage({ message: { /* ... */ }, model: preferredModel, schema: mySchema, }); if (!('error' in result) && result.usage) { console.log('Input tokens:', result.usage.inputTextTokens); console.log('Output tokens:', result.usage.outputTextTokens); console.log('Total cost (USD):', result.cost?.total); } ``` #### Provider Support - **OpenAI**: Uses native `response_format` with `json_schema` for optimal performance - **Google Generative AI**: Uses native `responseSchema` parameter for structured output - **LLMAPI**: Uses native `response_format` for the models that accept it. Anthropic-backed `claude-*` models are not among them — LLMAPI forwards `response_format` verbatim and Anthropic only honours its own `output_config`, so those entries declare `structuredOutputs: false` and `jsonOutput: false`. The gateway then injects the JSON schema into the system message and validates the response, rather than sending a parameter the upstream provider discards. - **Backward Compatibility**: All existing code continues to work unchanged #### Provider Compatibility Notes **OpenAI Structured Output:** - The `.optional()` modifier now automatically converts to required+nullable for OpenAI compatibility - Use `.min(1)` instead of `.positive()` for better compatibility - Nested objects and arrays are fully supported **Google Generative AI:** - Supports all schema types and modifiers - Handles optional fields natively - More flexible with schema variations **Best Practices for Cross-Provider Compatibility:** - Use `.optional()` for optional fields (automatically handled for all providers) - Use `.min()` and `.max()` instead of `.positive()`, `.negative()` - Both providers now have unified schema handling #### Schema Architecture The package uses a **driver pattern** for schema conversion, automatically adapting schemas to each provider's specific format while maintaining a unified API. Schema adapters are automatically registered when providers are imported, so no manual configuration is needed. For detailed architecture information and extending with new providers, see the [Developer Guide](#developer-guide). ### Model Configuration Each model comes with default configuration values that can be customized for your specific needs. #### Accessing and Customizing Models ```typescript // Get available models from the service const models = completionService.models; // Get a specific model const model = models[OpenAIModelNames.GPT_4_1]; // Customize model configuration const customModel = { ...model, config: { ...model.config, temperature: 0.8, // Override temperature (0-2, controls randomness) top_p: 0.9, // Override top_p (nucleus sampling) } }; // Use customized model in requests const result = await completionService.sendMessage({ message: { role: LLMRoles.User, content: [{ type: LLMMessageContentType.TEXT, text: 'Hello!' }] }, model: customModel, }); ``` #### Model-Specific Configuration **GPT-5 Models** have special configuration options: ```typescript const gpt5Model = models[OpenAIModelNames.GPT_5]; const customGPT5Model = { ...gpt5Model, config: { ...gpt5Model.config, temperature: 1, // Note: GPT-5 only supports temperature=1 reasoning_effort: 'high', // 'minimal' | 'low' | 'medium' | 'high' | 'xhigh' verbosity: 'low', // 'low' | 'medium' | 'high' } }; // GPT-5.1 and GPT-5.2 supports 'none' reasoning_effort const gpt51Model = models[OpenAIModelNames.GPT_5_1]; const customGPT51Model = { ...gpt51Model, config: { ...gpt51Model.config, temperature: 1, // Note: GPT-5.1 only supports temperature=1 reasoning_effort: 'none', // 'none' | 'minimal' | 'low' | 'medium' | 'high' | 'xhigh' verbosity: 'low', // 'low' | 'medium' | 'high' } }; ``` **Note:** All GPT-5 models require `temperature: 1` and cannot be changed. GPT-5, GPT-5-MINI, and GPT-5-NANO support reasoning_effort values of 'minimal', 'low', 'medium', 'high', or 'xhigh'. GPT-5.1, GPT-5.2, and GPT-5.4 additionally support 'none'. #### Per-Prompt Reasoning Effort (`params.reasoningEffort`) A prompt's `config.params` can carry generation knobs that override the static model-map defaults for that prompt's calls only. `reasoningEffort` is the first supported knob — the v2 client resolves it from the prompt config (live Langfuse config or the snapshot fallback) and passes it to the provider on every round of the call, agent tool loops included: ```typescript // In the prompt snapshot (TypeScript): config: { provider: LLMProviders.OpenAI, model: 'gpt-5.1', params: { reasoningEffort: LLMReasoningEffort.Medium }, } ``` A live Langfuse prompt config is JSON, so it spells the same value out as a lowercase string: ```json { "provider": "OpenAI", "model": "gpt-5.1", "params": { "reasoningEffort": "medium" } } ``` The accepted values are exactly `'none'`, `'minimal'`, `'low'`, `'medium'`, `'high'` and `'xhigh'`. `LLMReasoningEffort.None`, `.Minimal`, `.Low`, `.Medium`, `.High` and `.Xhigh` are the TypeScript names for those lowercase strings, not values themselves — the runtime gate validates the config against the enum, so a capitalized `"Medium"` in a live Langfuse config fails the gate and the prompt falls back to the snapshot. Which of the six a given model accepts is the model's own business, and the gate does not know it: per the GPT-5 note above, GPT-5, GPT-5-MINI and GPT-5-NANO take `'minimal'` through `'xhigh'` but not `'none'`, and the whole `'minimal'`/`'xhigh'` vocabulary belongs to the GPT-5.x family in the first place. A combination the model does not support is rejected by the provider API at call time, not by the config gate. - **OpenAI (Responses API):** sent as `reasoning: { effort }`. Without `params`, no reasoning field is sent and the OpenAI API default applies. - **LLMAPI / OpenAI-compatible completions:** sent as `reasoning_effort`, overriding the model map's hardcoded value. - **Google Generative AI:** not applied (Gemini uses `thinkingLevel`, a different dialect). A model whose map entry declares `capabilities.reasoning: false` never receives the field at all — the effort is dropped rather than sent to a model that would reject it, and the traced model parameters report what was sent rather than the effort that was asked for. Direct service callers can pass the same knob per call via `generationParams: { reasoningEffort }` on `assistInChat` / `sendMessage` options. A prompt or call that carries no params behaves exactly as before. **Gemini 3.x thinking models** (gemini-3-flash-preview, gemini-3.1-pro-preview) have special configuration options: ```typescript import { ThinkingLevel } from '@google/genai'; const gemini31Model = models[GoogleGenerativeAIModelNames.GEMINI_3_1_PRO_PREVIEW]; const customGemini31Model = { ...gemini31Model, config: { ...gemini31Model.config, temperature: 1, // Note: Gemini 3.x thinking models only support temperature=1 thinkingConfig: { thinkingLevel: ThinkingLevel.HIGH, // LOW | HIGH | THINKING_LEVEL_UNSPECIFIED }, } }; ``` **Note:** Gemini 3.x thinking models require `temperature: 1` and cannot be changed. The `thinkingConfig` allows you to control the model's reasoning depth. **Gemini tool-loop replay handling (LLMAPI provider).** Gemini through the LLMAPI Chat Completions proxy imposes two replay constraints on tool loops, both handled internally by `LLMAPIAssistanceService`: - *Parallel-call pairing*: Gemini emits id-less function calls and the proxy synthesizes ids, so an assistant turn with several parallel tool calls is recorded in chat history as sequential single-call turns (each function-call turn has exactly one matching response). - *thought_signature round-trip*: thinking models stamp `tool_calls[n].extra_content.google.thought_signature` (only on the first of N parallel calls) and require it echoed verbatim on replay. Split turns without a signature carry the Google-documented `skip_thought_signature_validator` bypass marker. If the provider still rejects a replayed signature (HTTP 400 "Corrupted thought signature."), the service downgrades every recorded signature to the bypass marker and retries that round trip once. Set `LLM_GATEWAY_DEBUG_TOOL_CALLS=true` to log the outgoing tool-call turn shapes (ids, signature presence) for wire-level debugging. #### File-based Assistance ```typescript // Upload files for context const uploadedFile = await assistanceService.uploadFile({ name: 'document.txt', path: '/path/to/document.txt', mimeType: LLMUploadFileMimeTypes.PLAIN_TEXT, }); // Create file storage with instructions const storage = await assistanceService.createFileStorage({ uploadedFiles: [uploadedFile], model: preferredModel, instructions: 'Help me analyze this document', // Optional context for the storage }); // Create chat with file context const chat = await assistanceService.createChat({ storageId: storage.storageId, model: preferredModel, history: [], files: [uploadedFile], instructions: 'Answer questions about the uploaded document', // Chat instructions }); // Send messages in the chat const result = await assistanceService.assistInChat({ model: preferredModel, message: { role: LLMRoles.User, content: [{ type: LLMMessageContentType.TEXT, text: 'What are the key points in this document?', }], }, chatId: chat.chatId, storageId: storage.storageId, }); ``` #### Speech-to-Text Transcription ```typescript const transcription = await speechToTextService.transcribe({ pathToAudio: '/path/to/audio.mp3', mimeType: LLMUploadFileMimeTypes.AUDIO_MP3, // Optional, but recommended model: preferredModel, instructions: 'Transcribe the audio file', // Optional: custom prompt for transcription context reporterContext: { // required if reporter is configured with context type userId: 12345, feature: 'audio_transcription', environment: 'production', }, }); console.log(transcription.text); // Transcribed text ``` #### Text-to-Speech Generation ```typescript const speech = await textToSpeechService.createSpeech({ text: 'Hello, this will be converted to speech', model: preferredModel, speechOptions: { voice: 'alloy', // OpenAI voice option response_format: 'mp3', }, }); // Save audio buffer to file fs.writeFileSync('output.mp3', speech.audio); ``` #### Token Counting for Context Management The LLM Gateway provides token counting functionality to help you manage context and optimize API usage. Token counting uses a lightweight character-based approximation (approximately 4 characters per token) to avoid heavyweight tokenizer dependencies. **Note:** Token counts are approximate and may differ from actual provider token usage. For precise token counts, refer to the usage metrics returned in API responses. ```typescript // Count tokens in messages before sending to optimize context usage const messages = [ { role: LLMRoles.User, content: [ { type: LLMMessageContentType.TEXT, text: 'Analyze this document and provide insights.', }, { type: LLMMessageContentType.TEXT, text: longDocumentContent, // Large text content }, // ... potentially more content including images ], } ]; const tokenCount = await assistanceService.countTokens(messages, preferredModel); console.log(`Total tokens: ${tokenCount}`); // Make intelligent decisions based on token count if (tokenCount > 50000) { // Use file upload approach for large content const uploadedFile = await assistanceService.uploadFile({ name: 'document.txt', path: '/path/to/document.txt', mimeType: LLMUploadFileMimeTypes.PLAIN_TEXT, }); const storage = await assistanceService.createFileStorage({ uploadedFiles: [uploadedFile], model: preferredModel, }); const chat = await assistanceService.createChat({ storageId: storage.storageId, model: preferredModel, instructions: 'Analyze the uploaded document', }); } else { // Send content directly in messages const result = await assistanceService.assistInChat({ model: preferredModel, message: messages[0], chatId: existingChatId, }); } ``` #### Abort Signal Support All operations support abort signals for cancellation: ```typescript const controller = new AbortController(); // Cancel after 30 seconds setTimeout(() => controller.abort(), 30000); const result = await completionService.sendMessage({ message: { role: LLMRoles.User, content: [{ type: LLMMessageContentType.TEXT, text: 'Long request...' }], }, model: preferredModel, abortSignal: controller.signal, }); if (result.error) { console.error('Operation failed or was cancelled:', result.error); } else { console.log('Success:', result.text); } ``` ## API Reference ### LLMProviders Enum of supported LLM providers: ```typescript enum LLMProviders { OpenAI = 'OpenAI', GoogleGenerativeAI = 'GoogleGenerativeAI', // other providers may be added in the future } ``` ### LLMPurposes Enum of supported service purposes: ```typescript enum LLMPurposes { Completion = 'completion', Assistance = 'assistance', SpeechToText = 'speech_to_text', TextToSpeech = 'text_to_speech', } ``` ### LLMServiceFactory Factory class for creating LLM service instances. #### Methods - `resolveProviderOptions(provider, optionsMap)`: Resolves the options for the specified provider - `getCompletionService(provider, logger, reporter, options)`: Creates a completion service instance - `getAssistanceService(provider, logger, reporter, options)`: Creates an assistance service instance - `getSpeechToTextService(provider, logger, reporter, options)`: Creates a speech-to-text service instance - `getTextToSpeechService(provider, logger, reporter, options)`: Creates a text-to-speech service instance #### Service Instance Methods All service instances provide the following common methods: - `setOptions(options)`: Update the provider options (e.g., API key, base URL) for an existing service instance - `getOptions()`: Retrieve the current provider options - `clearInstanceCache()`: Clear the internal SDK instance cache (rarely needed) **Instance Caching:** The gateway implements intelligent SDK instance caching to prevent memory leaks when switching between different credentials: ```typescript // Create service with initial credentials const service = LLMServiceFactory.getCompletionService({ provider: LLMProviders.OpenAI, options: { apiKey: 'key-1' }, }); // First call creates and caches SDK instance for 'key-1' await service.sendMessage({ /* ... */ }); // Update to different credentials service.setOptions({ apiKey: 'key-2' }); // Second call creates and caches SDK instance for 'key-2' await service.sendMessage({ /* ... */ }); // Switch back to original credentials service.setOptions({ apiKey: 'key-1' }); // Reuses cached SDK instance for 'key-1' (no recreation needed) await service.sendMessage({ /* ... */ }); ``` The cache uses LRU (Least Recently Used) eviction and maintains up to 10 instances per service. This is particularly useful when: - Using the same service with multiple products/features that have different API keys - Implementing multi-tenant systems where credentials change frequently - Running tests with different credential sets ### LLMCompletionService Interface for text completion services. #### Methods - `sendMessage(options)`: Send a message to the LLM and get a completion response - `message`: The message to send - `model`: The LLM model to use (with optional config overrides) - `history`: Optional conversation history - `instructions`: Optional system instructions to guide the model's behavior - `schema`: Optional schema for structured output - `reporterContext`: Context data passed to reporter for metrics tracking (required if reporter is configured with a defined context type) - `abortSignal`: Optional abort signal for cancellation ### LLMAssistanceService Interface for chat/assistance services with file handling capabilities. #### Methods - `uploadFile(fileOptions)`: Upload a file to the LLM service - `getFile(path)`: Retrieve the file information for a previously uploaded file by its path - `deleteFile(fileId)`: Delete a file from the LLM service - `createFileStorage(options)`: Create a file storage for organizing files - `uploadedFiles`: Array of previously uploaded files - `model`: The LLM model to use - `instructions`: Optional instructions for how to use the stored files - `deleteFileStorage(fileStorageId)`: Delete a file storage - `createChat(options)`: Create a new chat/conversation - `model`: The LLM model to use - `instructions`: Optional initial instructions for the conversation - `history`: Optional conversation history — an `LLMHistoryEntry[]` of plain messages, past tool calls (`LLMHistoryToolCallEntry`), their results (`LLMHistoryToolResultEntry`) and mid-conversation context (`LLMHistoryContextEntry`), each mapped to the provider's native shape - `files`: Optional array of uploaded files - `storageId`: Optional storage ID to use - `deleteChat(chatId)`: Delete a chat - `assistInChat(options)`: Send a message in an existing chat and get a response - `model`: The LLM model to use (required) - `message`: The message to send - `chatId`: The ID of the chat to send the message to - `storageId`: Optional storage ID to use for file context - `schema`: Optional schema for structured output (supports type-safe JSON responses) - `tools`: Optional array of tool definitions (built with `LLMTool.create`) the model may call during the turn - `maxToolIterations`: Optional cap on the number of tool-call rounds - `agentName`: Optional agent observation name for legacy agent-style tool loops; do not set it when using the v2 `runAgent` API - `reporterContext`: Context data passed to reporter for metrics tracking (required if reporter is configured with a defined context type) - `countTokens(messages, model)`: Count tokens in messages for context management #### Tool-returned Images A tool's `execute` may return either a plain `string` (text-only, the legacy behavior) or an `LLMStructuredToolResult` to surface images the model can actually see: ```typescript const screenshotTool = LLMTool.create({ name: 'get_node_screenshot', description: 'Renders a screenshot of a design node.', parameters: LLMSchema.object({ nodeId: LLMSchema.string() }), execute: async ({ nodeId }) => ({ content: `Rendered a screenshot of node ${nodeId}.`, images: [{ url: imageUrl, detail: 'high', label: `Node ${nodeId}` }], }), }); ``` `content` always becomes the tool/function-result message. Each provider then surfaces `images` the way its API allows: - **OpenAI** (Responses API) and **Gemini 3** (`multimodalToolResults` capability) embed the images natively in the function-result message. - **LLMAPI** (Chat Completions) and **Gemini 2.5** cannot carry images in a tool-result message, so the gateway appends the images as a separate user message right after the tool results. `LLMToolResultImage.url` accepts an `https` URL or a base64 `data:` URL. ### LLMSpeechToTextService Interface for converting speech audio to text. #### Methods - `transcribe(options)`: Convert audio file to text transcription - `pathToAudio`: Path to the audio file - `mimeType`: Optional MIME type of the audio file - `model`: The LLM model to use - `instructions`: Optional custom transcription prompt or context - `reporterContext`: Context data passed to reporter for metrics tracking (required if reporter is configured with a defined context type) ### LLMTextToSpeechService Interface for converting text to speech audio. #### Methods - `createSpeech(options)`: Convert text to speech audio file - `text`: Text to convert to speech - `model`: The LLM model to use - `instructions`: Optional instructions for speech generation - `speechOptions`: Provider-specific speech configuration - `reporterContext`: Context data passed to reporter for metrics tracking (required if reporter is configured with a defined context type) ### Prompt Builder The LLM Gateway includes a powerful prompt template system that provides type-safe string templates with dynamic replacements and conditional sections. This allows you to create reusable prompt templates with placeholders that can be replaced with actual values at runtime. #### Features - **Type Safety**: Automatic extraction and validation of placeholder keys from template strings - **Dynamic Replacements**: Replace placeholders like `{{variableName}}` with actual values - **Conditional Sections**: Show or hide content based on variable values using `{{#condition}}...{{/condition}}` syntax - **Nested Conditionals**: Support for nested conditional sections for complex logic - **Template Reusability**: Create templates once and use them multiple times with different values - **Zero Runtime Dependencies**: Pure TypeScript utility functions #### Basic Usage ```typescript import { createPromptTemplate } from '@mate-academy/llm-gateway'; // Create a prompt template with placeholders const welcomePrompt = createPromptTemplate(` Generate a welcome message for a user who has just started their auto tech check attempt on {{topicTitle}}. The user's experience level is {{experienceLevel}} and they prefer {{learningStyle}} learning. `); // Use the template with actual values const instruction = welcomePrompt({ topicTitle: 'JavaScript Basics', experienceLevel: 'beginner', learningStyle: 'hands-on', }); // Result: "Generate a welcome message for a user who has just started their auto tech check attempt on JavaScript Basics. The user's experience level is beginner and they prefer hands-on learning." ``` #### Conditional Sections Conditional sections allow you to show or hide parts of the template based on variable values: ```typescript // Template with conditional sections const coursePrompt = createPromptTemplate(` Generate a lesson plan for {{topicTitle}}. {{#hasPrerequisites}} Prerequisites: {{prerequisites}} {{/hasPrerequisites}} {{#includeExercises}} Include practical exercises and code examples. {{/includeExercises}} {{#difficultyLevel}} Adjust content for {{difficultyLevel}} level students. {{/difficultyLevel}} `); // Usage with all sections visible const fullLesson = coursePrompt({ topicTitle: 'React Hooks', hasPrerequisites: true, prerequisites: 'Basic React knowledge', includeExercises: true, difficultyLevel: 'intermediate' }); // Usage with some sections hidden const basicLesson = coursePrompt({ topicTitle: 'React Hooks', hasPrerequisites: false, prerequisites: '', includeExercises: false, difficultyLevel: 'beginner' }); ``` #### Negative Conditional Sections Negative conditions allow you to show content when a value is falsy: ```typescript const feedbackPrompt = createPromptTemplate(` Analyze the {{language}} code submission. {{#passed}} Great job! The tests passed successfully. {{/passed}} {{#!passed}} The tests did not pass. Please review the following issues: {{errors}} {{/passed}} {{#!skipSuggestions}} Here are some suggestions for improvement: - Consider refactoring for better readability - Add more comprehensive error handling {{/skipSuggestions}} `); // When tests pass const successResult = feedbackPrompt({ language: 'JavaScript', passed: true, errors: '', skipSuggestions: false }); // Result: Shows success message and suggestions // When tests fail const failureResult = feedbackPrompt({ language: 'Python', passed: false, errors: 'TypeError on line 15', skipSuggestions: false }); // Result: Shows failure message with errors and suggestions ``` #### Nested Conditional Sections You can nest conditional sections for more complex logic: ```typescript const reviewPrompt = createPromptTemplate(` Review the {{language}} code for {{focusArea}}. {{#includeMetrics}} Provide performance metrics. {{#includeDetailed}} Include detailed benchmark analysis and memory usage patterns. {{/includeDetailed}} {{/includeMetrics}} {{#suggestImprovements}} Suggest specific improvements for better {{improvementFocus}}. {{/suggestImprovements}} `); const detailedReview = reviewPrompt({ language: 'TypeScript', focusArea: 'performance', includeMetrics: true, includeDetailed: true, suggestImprovements: true, improvementFocus: 'scalability' }); ``` #### Advanced Usage ```typescript // Template without placeholders (no parameters required) const staticPrompt = createPromptTemplate(` Please analyze the provided code and suggest improvements. `); const staticInstruction = staticPrompt(); // No parameters needed // Template with multiple placeholders and conditional sections const codeReviewPrompt = createPromptTemplate(` Review the {{language}} code below for {{focusArea}}. {{#includeCriteria}} Pay special attention to {{criteria}} and provide {{outputFormat}} feedback. {{/includeCriteria}} {{#includeCode}} Code: {{codeSnippet}} {{/includeCode}} {{#provideExamples}} Include examples of best practices for {{language}}. {{/provideExamples}} `); const reviewInstruction = codeReviewPrompt({ language: 'TypeScript', focusArea: 'performance optimization', includeCriteria: true, criteria: 'algorithmic efficiency and memory usage', outputFormat: 'structured', includeCode: true, codeSnippet: 'function example() { /* code here */ }', provideExamples: false, }); ``` #### Integration with LLM Services ```typescript import { createPromptTemplate, LLMServiceFactory, LLMProviders, LLMRoles, } from '@mate-academy/llm-gateway'; // Define reusable prompt templates const PROMPTS = { codeExplanation: createPromptTemplate(` Explain the following {{language}} code in simple terms for a {{level}} developer: {{#includeContext}} Context: {{context}} {{/includeContext}} {{code}} {{#includeExamples}} Provide practical examples of how this code would be used. {{/includeExamples}} `), bugFinding: createPromptTemplate(` Find potential bugs in this {{language}} code and suggest fixes: {{#focusArea}} Focus specifically on {{focusArea}} issues. {{/focusArea}} {{code}} {{#includeSeverity}} Rate the severity of each issue from 1-5. {{/includeSeverity}} `), optimization: createPromptTemplate(` Optimize the following code for {{optimizationType}}: {{code}} {{#includeMetrics}} Provide before/after performance metrics. {{/includeMetrics}} {{#includeAlternatives}} Suggest alternative approaches and explain trade-offs. {{/includeAlternatives}} `), }; // Use with completion service async function explainCode(code: string, language: string, level: string, includeExamples = false) { const prompt = PROMPTS.codeExplanation({ code, language, level, includeContext: false, includeExamples, }); return await completionService.sendMessage({ message: { role: LLMRoles.User, content: [{ type: LLMMessageContentType.TEXT, text: prompt }], }, model: preferredModel, }); } ``` #### Type Safety Features The prompt builder provides compile-time type checking for template placeholders and conditional sections: ```typescript // This will show TypeScript errors for missing or incorrect parameters const template = createPromptTemplate(` Hello {{name}}, welcome to {{platform}}! {{#showBonus}}You have a bonus: {{bonusAmount}}{{/showBonus}} `); // ✅ Correct usage with all required variables template({ name: 'John', platform: 'LLM Gateway', showBonus: true, bonusAmount: '$50' }); // ✅ Correct usage with conditional section hidden template({ name: 'John', platform: 'LLM Gateway', showBonus: false // bonusAmount is not required when showBonus is false }); // ❌ TypeScript error: missing required parameter 'platform' template({ name: 'John', showBonus: false }); // ❌ TypeScript error: unknown parameter 'age' template({ name: 'John', platform: 'LLM Gateway', showBonus: false, age: 25 }); ``` #### API Reference **`createPromptTemplate(template: T)`** Creates a prompt template function from a template string. - **Parameters:** - `template: T` - The template string with placeholders and conditional sections - **Returns:** A function that accepts replacement values and returns the processed string - **Type Safety:** Automatically extracts placeholder names and conditional section names from the template string for type checking **Template Syntax** **Variable Placeholders:** - Placeholders must be enclosed in double curly braces: `{{variableName}}` - Whitespace around variable names is ignored: `{{ variableName }}` works the same as `{{variableName}}` - Variable names can contain letters, numbers, and underscores - Replacement values can be strings, numbers, or booleans (automatically converted to strings) **Conditional Sections:** - Positive conditional sections use the syntax: `{{#conditionName}}content{{/conditionName}}` - The section content is included only if the condition variable is truthy - Negative conditional sections use the syntax: `{{#!conditionName}}content{{/conditionName}}` - The section content is included only if the condition variable is falsy - Truthy values: `true`, non-empty strings, non-zero numbers - Falsy values: `false`, empty strings, `0`, `null`, `undefined` - Conditional variables are optional in the type system when used only as conditions - Variables used both as conditions and values are required in the type system **Nested Conditionals:** - Conditional sections can be nested for complex logic - Inner sections are processed only if outer sections are visible - Variables inside nested sections follow the same truthy/falsy rules ## Supported Providers ### OpenAI Supports all service types: completion, assistance, speech-to-text, and text-to-speech APIs. For more information, see [OpenAI API documentation](https://platform.openai.com/docs/api-reference). **Available Models:** | Model | Purpose | Max Input | Max Output | Notes | |-------|---------|-----------|------------|-------| | gpt-4.1 | Completion, Assistance | 1M+ | 32K | Extended context window | | gpt-4.1-mini | Completion, Assistance | 1M+ | 32K | Cost-effective extended context | | gpt-4.1-nano | Completion, Assistance | 1M+ | 32K | Fastest, most cost-efficient GPT-4.1 | | gpt-5 | Completion, Assistance | 400K | 128K | Advanced reasoning, requires temperature=1 | | gpt-5.1 | Completion, Assistance | 400K | 128K | Advanced reasoning with 'none' reasoning_effort support, requires temperature=1 | | gpt-5.2 | Completion, Assistance | 400K | 128K | Advanced reasoning with 'none' reasoning_effort support, requires temperature=1 | | gpt-5.4 | Completion, Assistance | 1M+ | 128K | Flagship model, supports 'none'/'xhigh' reasoning_effort, requires temperature=1 | | gpt-5-mini | Completion, Assistance | 400K | 128K | Smaller GPT-5 variant, requires temperature=1 | | gpt-5-nano | Completion, Assistance | 400K | 128K | Fastest GPT-5 variant, requires temperature=1 | | gpt-4o-transcribe | Speech-to-Text | 16K | 2K | Optimized for transcription | | gpt-4o-mini-transcribe | Speech-to-Text | 16K | 2K | Cost-effective transcription | | tts-1 | Text-to-Speech | 2K | - | Standard TTS model | | gpt-4o-mini-tts | Text-to-Speech | 2K | - | Alternative TTS model | ### Google Generative AI Supports completion, assistance, speech-to-text, and text-to-speech APIs through Google's Generative AI models. For more information, see [Google Generative AI documentation](https://ai.google.dev/docs). **Available Models:** | Model | Purpose | Max Input | Max Output | Notes | |-------|---------|-----------|------------|-------| | gemini-2.5-flash | Completion, Assistance, Speech-to-Text | 1M+ | 65K | Fast, cost-effective, supports caching for long context | | gemini-2.5-flash-lite | Completion, Assistance, Speech-to-Text | 1M+ | 65K | Fastest, most cost-effective, supports caching | | gemini-2.5-pro | Completion, Assistance, Speech-to-Text | 1M+ | 65K | Advanced reasoning, supports caching for long context | | gemini-3-flash-preview | Completion, Assistance, Speech-to-Text | 1M+ | 65K | Thinking model, requires temperature=1, configurable thinking levels | | gemini-3.1-flash-lite-preview | Completion, Assistance, Speech-to-Text | 1M+ | 65K | Most cost-efficient Gemini 3.x model | | gemini-3.1-pro-preview | Completion, Assistance, Speech-to-Text | 1M+ | 65K | Advanced reasoning with thinking capabilities, requires temperature=1 | | gemini-2.5-flash-preview-tts | Text-to-Speech | 8K | 16K | Preview TTS model with flash performance | | gemini-2.5-pro-preview-tts | Text-to-Speech | 8K | 16K | Preview TTS model with pro capabilities | **Note:** Google Generative AI models support context caching for content longer than 32,768 tokens, which can significantly reduce costs for repeated queries on the same large context. **Gemini 3.x Thinking Model Configuration:** - Applies to: gemini-3-flash-preview, gemini-3.1-pro-preview - Requires `temperature: 1` (cannot be changed) - Supports `thinkingConfig` with `thinkingLevel` property (LOW, HIGH, THINKING_LEVEL_UNSPECIFIED) - Default thinking level is LOW ## Testing The LLM Gateway includes a comprehensive testing suite with both unit and integration tests. ### Test Structure The package includes: - **Unit tests** for core functionality (`src/tests/unit/`) - Schema builder and validation tests - Prompt template builder tests - Utility function tests - **Integration tests** for all service types (`src/tests/integration/`) - Real API integration tests with live providers - Service-specific functionality tests - Cross-provider compatibility tests - **Shared test utilities** (`src/tests/integration/shared/`) - Reusable test patterns for common functionality - Service initialization tests - Logger integration tests - Abort signal handling tests - Reporter integration tests - Instance caching tests - **Test helpers** for common testing utilities (`src/tests/helpers.ts`) - **Mock implementations** for testing environments - **Audio test files** for speech-to-text testing ### Running Tests ```bash # Run all tests with logging npm test # Run tests silently (without logs) npm run test:silent # Run only integration tests with verbose output npm run test:integration # Run specific test file npm test -- LLMSchema.test.ts # Run tests matching a pattern npm test -- --testNamePattern="should handle basic schemas" # Run interactive prepublish test selector npm run prepublish-tests # Run with specific environment variables ENABLE_LOGGING=true npm test ``` ### Test Configuration Integration tests require API keys for the respective providers: ```bash # Required environment variables for OpenAI tests OPENAI_SECRET_API_KEY=your_openai_api_key OPENAI_ORG_ID=your_organization_id # optional OPENAI_BASE_URL=https://api.openai.com/v1 # optional # Required environment variables for Google AI tests GOOGLE_GENERATIVE_AI_API_KEY=your_google_ai_api_key ``` ### Test Coverage The integration tests cover: 1. **Common Service Tests** (via shared utilities): - Service initialization with and without options - Instance caching and credential switching - Logger integration and error handling - Reporter metrics collection (success/abort/fail) - Abort signal handling for all operations 2. **Completion Service Tests:** - Basic message sending and responses - Message history handling - Custom instructions - Image analysis - Structured output with schemas - Error handling for invalid inputs - Model-specific functionality 3. **Assistance Service Tests:** - File upload and storage creation - Chat creation and management - Direct chat interactions - File-based conversations - Direct image handling in messages - Multiple abort scenarios (file upload, storage, chat) 4. **Speech-to-Text Service Tests:** - Audio file transcription - Multiple audio format support (MP3, WAV, WEBM, OGG) - Error handling for invalid files - Model-specific transcription quality 5. **Text-to-Speech Service Tests:** - Text-to-audio conversion - Voice selection options - Audio format configuration - Instructions for speech generation - Error handling ### Test Helpers The package provides several test utilities: #### resolveTestConfig Function The `resolveTestConfig` function creates standardized test configurations for all supported providers based on the service purpose: ```typescript import { resolveTestConfig } from '@mate-academy/llm-gateway/tests/helpers'; // Get test config for completion services const testConfig = resolveTestConfig(LLMPurposes.Completion); // testConfig contains configuration for all providers: // { // [LLMProviders.OpenAI]: { // provider: LLMProviders.OpenAI, // availableModels: {...}, // Models available for completion // clientOptions: { apiKey: process.env.OPENAI_SECRET_API_KEY, ... }, // requireCredentials: () => void, // Throws if credentials missing // isEnabled: true // }, // [LLMProviders.GoogleGenerativeAI]: { // provider: LLMProviders.GoogleGenerativeAI, // availableModels: {...}, // Models available for completion // clientOptions: { apiKey: process.env.GOOGLE_GENERATIVE_AI_API_KEY }, // requireCredentials: () => void, // Throws if credentials missing // isEnabled: true // } // } // Use in tests to iterate over all providers Object.values(testConfig).forEach((config) => { const { provider, clientOptions, availableModels, requireCredentials } = config; describe(`${provider} Provider`, () => { beforeAll(() => { requireCredentials(); // Ensures API keys are present }); it('should create service', () => { const service = LLMServiceFactory.getCompletionService({ provider, options: clientOptions, logger: mockLogger, // Optional }); expect(service).toBeDefined(); }); }); }); ``` #### Mock Logger ```typescript // Note: Test helpers are for internal use only // When testing your integration, create your own mock logger: const mockLogger = { info: jest.fn(), error: jest.fn(), warn: jest.fn(), child: jest.fn(() => mockLogger), }; ``` #### Type Guards ```typescript // Type guards for test assertions if ('text' in result && result.text) { expect(result.text).toContain('expected content'); } if ('error' in result && result.error) { expect(result.error).toBeDefined(); } ``` ### Writing Custom Tests Example of writing a custom integration test using `resolveTestConfig`: ```typescript import { describe, it, expect, beforeAll, } from '@jest/globals'; import { LLMServiceFactory, LLMProviders, LLMPurposes, resolveTestConfig, } from '@mate-academy/llm-gateway'; // Create your own mock logger const mockLogger = { info: jest.fn(), error: jest.fn(), warn: jest.fn(), child: jest.fn(() => mockLogger), }; describe('Custom LLM Integration Test', () => { // Use resolveTestConfig for consistent test configuration const testConfig = resolveTestConfig(LLMPurposes.Completion); // Test all enabled providers Object.values(testConfig).forEach((config) => { const { provider, clientOptions, requireCredentials } = config; describe(`${provider} Provider`, () => { let service; beforeAll(() => { requireCredentials(); // Validates API keys are present service = LLMServiceFactory.getCompletionService({ provider, options: clientOptions, logger: mockLogger, // Optional - can be omitted }); }); it('should process custom request', async () => { // Your custom test logic here const result = await service.sendMessage({ message: { role: LLMRoles.User, content: [{ type: LLMMessageContentType.TEXT, text: 'Test message' }], }, model: Object.values(config.availableModels)[0], // Use first available model }); // Use type guards for assertions if ('text' in result && result.text) { expect(result.text).toBeDefined(); expect(typeof result.text).toBe('string'); } else if ('error' in result && result.error) { throw result.error; } }); }); }); }); ``` ## Developer Guide ### Syncing the LLMAPI model list The `LLMAPIModelNames` enum and `LLMAPI_AVAILABLE_MODELS` map drift as LLMAPI adds/renames models. The pipeline in [`scripts/sync-llmapi-models/`](scripts/sync-llmapi-models/README.md) reconciles them against `GET https://api.llmapi.ai/v1/models` and the audio catalog at `GET https://api.llmapi.ai/audio-models` under these rules: - Unscoped aliases are always present; scoped aliases (`openai/…`, `azure/…`, `nebius/…`) only when LLMAPI advertises 2+ providers for that model. - Existing hand-tuned limits, capabilities, regular pricing, and configuration are preserved. Cached input prices are refreshed from LLMAPI for every existing and new route, including provider-specific overrides. - `maxOutputTokens` and any missing `maxInputTokens` are researched from provider docs via web-search sub-agents. - TTS/STT limits and pricing, batch/streaming STT classification, purpose-specific model maps, and default voice mappings are refreshed from the audio catalog. Streaming-only STT models are exported separately and are not exposed through the file-based `SpeechToText` service. See the linked README for the full classify → research → apply workflow. ### Pricing Model Architecture The LLM Gateway uses a flexible function-based pricing model that supports different token types and caching: ```typescript interface LLMModelPricing { getPriceForTextInput: (tokens: number) => number; // Price for regular text input tokens getPriceForTextOutput: (tokens: number) => number; // Price for text output tokens getPriceForAudioInput: (tokens: number) => number; // Price for regular audio input tokens getPriceForAudioOutput: (tokens: number) => number; // Price for audio output tokens // Optional: Cached token pricing (prompt caching) getPriceForCachedTextInput?: (tokens: number) => number; // Discounted price for cached text tokens getPriceForCachedAudioInput?: (tokens: number) => number; // Discounted price for cached audio tokens currency: string; // Currency code (e.g., 'USD') } ``` This architecture allows: - **Dynamic Pricing**: Support for tiered pricing based on token count - **Multi-Modal Support**: Separate pricing for text and audio tokens - **Provider Flexibility**: Each provider can implement custom pricing logic - **Cached Token Discounts**: Automatic detection and separate pricing for cached tokens with provider-specific discount rates - **Reasoning Token Support**: Proper cost calculation for reasoning tokens in o1/o3 models - **Character-based Pricing**: Some models (e.g., OpenAI TTS) use character count instead of tokens for input pricing **Cached Token Support:** - **OpenAI**: 50% discount for cached tokens (GPT-4 models), 10% discount (GPT-5 models), 25% discount (GPT-4.1 models) - **Google Gemini**: 90% discount for cached tokens (all models with context caching) - Cached tokens are automatically detected and priced separately - If cached pricing is not defined, falls back to regular pricing **Important Notes:** - Cached tokens are tracked separately and receive automatic discount rates - Reasoning tokens (o1/o3 models) are charged as output tokens - Text-to-Speech models may use character-based pricing for input (not token-based) ### Architecture Overview The LLM Gateway uses several architectural patterns to provide a clean, extensible interface for multiple LLM providers: #### Core Architecture Components **Service Layer:** - **Abstract Base Services**: `LLMBaseService` provides common functionality for all services - **Purpose-Specific Services**: Separate services for Completion, Assistance, Speech-to-Text, and Text-to-Speech - **Provider Implementations**: Each provider extends abstract services with specific implementations - **Instance Caching**: Automatic SDK instance pooling with LRU eviction to prevent memory leaks **Instance Caching Architecture:** - **Shared Global Cache**: All service types share a single static cache to optimize memory usage - **Provider-based Caching**: SDK instances are cached by provider and credential hash (SHA256) - **Cross-Purpose Sharing**: Different purposes (Completion, Assistance, etc.) share instances for the same provider and credentials - **LRU Eviction**: Maintains up to 10 instances globally using Least Recently Used eviction - **Automatic Reuse**: Instances are automatically reused when switching back to previously used credentials - **Memory Safety**: Prevents memory leaks when credentials change frequently (e.g., multi-tenant scenarios) **Metrics & Reporting:** - **`LLMReporterInterface`**: Type-safe reporter interface for metrics collection - **Automatic Cost Calculation**: Built-in cost tracking using flexible pricing functions for text and audio tokens - **Timer Integration**: Automatic duration tracking via `initMetricsWriter()` pattern - **Token Usage Tracking**: Separate tracking for text and audio tokens (input/output) **Logging Infrastructure:** - **`LLMLoggerInterface`**: Standard logging interface compatible with major logging libraries - **Child Logger Support**: Context propagation through service hierarchies ### Schema Architecture Overview The LLM Gateway uses a **driver pattern** for schema conversion, providing clean separation between core schema logic and provider-specific implementations. **Core Components:** - **`LLMSchema`**: Core schema builder with unified API - **`SchemaAdapterInterface`**: Simple contract for provider-specific schema converters - **`SchemaAdapterRegistry`**: Type-safe registry ensuring all providers are handled - **Provider Adapters**: Convert generic JSON Schema to provider-specific formats **How It Works:** ```typescript // The schema uses a unified API regardless of provider const schema = LLMSchema.object({ name: LLMSchema.string(), age: LLMSchema.number().min(1), }); // Internally, services call _toProviderSchema() which automatically // converts to the correct provider-specific format: schema._toProviderSchema(LLMProviders.OpenAI); // → OpenAI JSON Schema format schema._toProviderSchema(LLMProviders.GoogleGenerativeAI); // → Google Type-based format ``` **Provider Schema Adapters:** Each provider has its own schema adapter located in `src/providers/{Provider}/schemas/`: - **`OpenAISchemaAdapter`**: Converts to OpenAI's JSON Schema format, handles strict mode requirements - **`GoogleSchemaAdapter`**: Converts to Google's Type-based schema format using their `Type` enum **Type-Safe Registry:** Schema adapters are managed through a centralized, type-safe registry that ensures compile-time safety: ```typescript // src/utilities/schema/SchemaAdapterRegistry.ts export const SCHEMA_ADAPTER_REGISTRY = { OpenAI: new OpenAISchemaAdapter(), GoogleGenerativeAI: new GoogleSchemaAdapter(), } as const satisfies Record; ``` ### Adding a New Provider To add support for a new LLM provider, follow these steps: ### 1. Create Provider Directory Structure Create a new directory in `src/providers` with your provider name, following the established pattern: ``` src/providers/YourProvider/ ├── index.ts # Entry point for provider exports ├── YourProvider.constants.ts # Provider-specific constants ├── YourProvider.entity.ts # Provider-specific entity ├── YourProvider.typedefs.ts # TypeScript definitions ├── YourProviderService.factory.ts # Factory for your provider's services ├── schemas/ # Schema conversion adapters │ └── YourProviderSchemaAdapter.ts └── services/ # Provider service implementations ├── index.ts ├── YourProviderCompletionService.ts └── YourProviderAssistanceService.ts ``` ### 2. Add Provider to LLM Providers Enum Update the LLM providers enum in `src/LLMService.typedefs.ts`: ```typescript export enum LLMProviders { OpenAI = 'openai', GoogleGenerativeAI = 'google', YourProvider = 'your-provider-id', } ``` ### 3. Define Provider-Specific Types Create type definitions in `src/providers/YourProvider/YourProvider.typedefs.ts`: ```typescript // Define model names as an enum for type safety export enum YourProviderModelNames { MODEL_ONE = 'model-one', MODEL_TWO = 'model-extended', } // Define message roles if applicable export enum YourProviderRoles { User = 'user', Assistant = 'assistant', System = 'system', } // Add any other provider-specific enums or interfaces ``` Then ensure your provider is properly integrated in the main type system by updating the necessary type mappings in `src/LLMService.typedefs.ts`: ```typescript // Add import for your provider's types import { type YourProviderModelNames } from './providers/YourProvider/YourProvider.typedefs'; // Update the LLMProviders enum export enum LLMProviders { OpenAI = 'OpenAI', GoogleGenerativeAI = 'GoogleGenerativeAI', YourProvider = 'YourProvider', } // Update LLMInstances type mapping export type LLMInstances = { // ...existing code... [LLMProviders.YourProvider]: YourProviderClient; // Your provider's client type }; // Update LLMInstanceOptions type mapping export type LLMInstanceOptions = { // ...existing code... [LLMProviders.YourProvider]: { apiKey: string; // Add other provider-specific options }; }; // Update LLMModelName type mapping export type LLMModelName = { // ...existing code... [LLMProviders.YourProvider]: YourProviderModelNames; }; ``` ### 4. Create Schema Adapter Implement a schema adapter in `src/providers/YourProvider/schemas/YourProviderSchemaAdapter.ts`: ```typescript import { LLMProviders } from '@/LLMService.typedefs'; import type { SchemaAdapterInterface } from '@/utilities/schema/SchemaAdapterInterface'; export class YourProviderSchemaAdapter implements SchemaAdapterInterface { convertSchema(jsonSchema: any): any { // Convert JSON Schema to your provider's specific format // Example: transform to provider-specific schema structure return this.transformToYourProviderFormat(jsonSchema); } private transformToYourProviderFormat(jsonSchema: any): any { // Implement provider-specific schema transformation logic // Handle objects, arrays, strings, numbers, etc. // Return the schema in your provider's expected format if (jsonSchema.type === 'object') { // Handle object schemas return { // Your provider's object schema format }; } // Handle other schema types... return jsonSchema; } } ``` ### 5. Add Adapter to Schema Registry Update the schema adapter registry in `src/utilities/schema/SchemaAdapterRegistry.ts`: ```typescript import { YourProviderSchemaAdapter } from '@/providers/YourProvider/schemas/YourProviderSchemaAdapter'; export const SCHEMA_ADAPTER_REGISTRY = { OpenAI: new OpenAISchemaAdapter(), GoogleGenerativeAI: new GoogleSchemaAdapter(), YourProvider: new YourProviderSchemaAdapter(), // Add your adapter here // TypeScript will enforce that ALL providers have adapters } as const satisfies Record; ``` If your provider doesn't support structured output, set it to `null`: ```typescript YourProvider: null, // Provider doesn't support structured output ``` ### 6. Implement Provider Constants Define constants in `src/providers/YourProvider/YourProvider.constants.ts`: ```typescript import { type LLMProviderAvailableModels, type LLMProviderModelsByPurpose, type LLMProviders, LLMPurposes, type LLMServiceBuilder, } from '@/LLMService.typedefs'; import { YourProviderModelNames } from './YourProvider.typedefs'; import { YourProviderAssistanceService, YourProviderCompletionService, YourProviderSpeechToTextService, YourProviderTextToSpeechService, } from './services'; import { pick } from '@/utilities/functional.utils'; // Define available models with their capabilities, configurations and pricing const YOUR_PROVIDER_AVAILABLE_MODELS = { [YourProviderModelNames.MODEL_ONE]: { name: YourProviderModelNames.MODEL_ONE, limits: { maxInputTokens: 8_000, maxOutputTokens: 2_000, }, config: { temperature: 0.2, }, pricing: { getPriceForTextInput: (tokens) => 0.5 * tokens / 1_000_000, // Cost per million text input tokens getPriceForTextOutput: (tokens) => 1.5 * tokens / 1_000_000, // Cost per million text output tokens getPriceForAudioInput: (tokens) => 0, // Cost per million audio input tokens getPriceForAudioOutput: (tokens) => 0, // Cost per million audio output tokens currency: 'USD' as const, }, }, [YourProviderModelNames.MODEL_TWO]: { name: YourProviderModelNames.MODEL_TWO, limits: { maxInputTokens: 16_000, maxOutputTokens: 4_000, }, config: { temperature: 0.2, }, pricing: { getPriceForTextInput: (tokens) => 1 * tokens / 1_000_000, // Cost per million text input tokens getPriceForTextOutput: (tokens) => 3 * tokens / 1_000_000, // Cost per million text output tokens getPriceForAudioInput: (tokens) => 0, // Cost per million audio input tokens getPriceForAudioOutput: (tokens) => 0, // Cost per million audio output tokens currency: 'USD' as const, }, }, } as const satisfies LLMProviderAvailableModels< LLMProviders.YourProvider >; // Specify which models are available for each purpose export const YOUR_PROVIDER_MODELS = { [LLMPurposes.Completion]: pick( YOUR_PROVIDER_AVAILABLE_MODELS, [ YourProviderModelNames.MODEL_ONE, YourProviderModelNames.MODEL_TWO, ], ), [LLMPurposes.Assistance]: pick( YOUR_PROVIDER_AVAILABLE_MODELS, [ YourProviderModelNames.MODEL_TWO, // Only MODEL_TWO supports assistance ], ), [LLMPurposes.SpeechToText]: pick( YOUR_PROVIDER_AVAILABLE_MODELS, [ YourProviderModelNames.MODEL_ONE, // Speech-to-text capable model ], ), [LLMPurposes.TextToSpeech]: pick( YOUR_PROVIDER_AVAILABLE_MODELS, [ YourProviderModelNames.MODEL_ONE, // Text-to-speech capable model ], ), } as const satisfies LLMProviderModelsByPurpose< LLMPurposes, LLMProviders.YourProvider >; // Define service builders for each LLM purpose export const YOUR_PROVIDER_SERVICE_BUILDERS = { [LLMPurposes.Completion]: (logger, reporter, options) => ( new YourProviderCompletionService(logger, reporter, options) ), [LLMPurposes.Assistance]: (logger, reporter, options) => ( new YourProviderAssistanceService(logger, reporter, options) ), [LLMPurposes.SpeechToText]: (logger, reporter, options) => ( new YourProviderSpeechToTextService(logger, reporter, options) ), [LLMPurposes.TextToSpeech]: (logger, reporter, options) => ( new YourProviderTextToSpeechService(logger, reporter, options) ), } as const satisfies { [purpose in LLMPurposes]: ( LLMServiceBuilder | null ) }; ``` ### 6. Implement Provider Entity (if needed) Create the entity class in `src/providers/YourProvider/YourProvider.entity.ts`: ```typescript export class YourProviderEntity { // Implement provider-specific methods } ``` ### 7. Implement Service Classes Create service implementations in the `services` directory: **CompletionService (src/providers/YourProvider/services/YourProviderCompletionService.ts)**: ```typescript import { type LLMLoggerInterface } from '../../../utilities/logger/LLMLoggerInterface'; import { type LLMReporterInterface } from '../../../utilities/reporter/LLMReporterInterface'; import { LLMCompletionService } from '../../../services/LLMCompletionService.abstract'; import { type LLMInstanceOptions, LLMProviders } from '../../../LLMService.typedefs'; export class YourProviderCompletionService< Reporter extends LLMReporterInterface | undefined > extends LLMCompletionService { constructor( logger: LLMLoggerInterface | undefined, reporter: Reporter, options?: LLMInstanceOptions[LLMProviders.YourProvider], ) { super(LLMProviders.YourProvider, logger, reporter, options); } async sendMessage(options) { // Use built-in method from LLMBaseService for metrics initialization const metricsWriter = this.initMetricsWriter( 'sendMessage', // method name options.reporterContext ); try { this.logger?.info('Starting completion with YourProvider', { options }); // Implement provider-specific completion logic const response = await this.instance.complete({ // ... your provider's API call parameters ... }); // Calculate tokens from response const usage = { inputTextTokens: response.usage?.input_tokens || 0, outputTextTokens: response.usage?.output_tokens || 0, inputAudioTokens: 0, // If your provider supports audio outputAudioTokens: 0, // If your provider supports audio }; // Use built-in success metrics method which handles cost calculation await this.writeSuccessMetrics({ method: 'sendMessage', model: options.model, usage, writeMetrics: metricsWriter, }); return { text: response.text || 'Completed text', }; } catch (error) { // Write error metrics await this.writeErrorMetrics({ method: 'sendMessage', model: options.model, isAborted: error.name === 'AbortError', writeMetrics: metricsWriter, }); this.logger?.error('Error in YourProvider completion', { error }); throw error; } } // Implement instance getter with caching support get instance(): YourProviderClient { return this.getOrCreateInstance(() => new YourProviderClient(this.options)); } } ``` **AssistanceService (if applicable)**: ```typescript import { type LLMLoggerInterface } from '../../../utilities/logger/LLMLoggerInterface'; import { LLMAssistanceService } from '../../../services/LLMAssistanceService.abstract'; import { LLMAssistanceOptions, LLMAssistanceResult, LLMProviders } from '../../../LLMService.typedefs'; import { YourProviderEntity } from '../YourProvider.entity'; export class YourProviderAssistanceService< Reporter extends LLMReporterInterface | undefined > extends LLMAssistanceService { constructor( logger: LLMLoggerInterface | undefined, reporter: Reporter, options?: LLMInstanceOptions[LLMProviders.YourProvider], ) { super(LLMProviders.YourProvider, logger, reporter, options); } // Implement instance getter with caching support get instance(): YourProviderClient { return this.getOrCreateInstance(() => new YourProviderClient(this.options)); } // Implement required assistance methods async uploadFile(file: LLMUploadFile): Promise { // Implementation for file upload } async createFileStorage(options: LLMCreateFileStorageOptions): Promise { // Implementation for file storage creation } async createChat(options: LLMCreateChatOptions): Promise { // Implementation for chat creation } async assistInChat(options: LLMAssistanceOptions): Promise { // Implementation for chat assistance } } ``` **SpeechToTextService (if applicable)**: ```typescript import { type LLMLoggerInterface } from '../../../utilities/logger/LLMLoggerInterface'; import { LLMSpeechToTextService } from '../../../services/LLMSpeechToTextService.abstract'; import { LLMTranscribeOptions, LLMTranscribeResult, LLMProviders } from '../../../LLMService.typedefs'; import { YourProviderEntity } from '../YourProvider.entity'; export class YourProviderSpeechToTextService< Reporter extends LLMReporterInterface | undefined > extends LLMSpeechToTextService { constructor( logger: LLMLoggerInterface | undefined, reporter: Reporter, options?: LLMInstanceOptions[LLMProviders.YourProvider], ) { super(LLMProviders.YourProvider, logger, reporter, options); } // Implement instance getter with caching support get instance(): YourProviderClient { return this.getOrCreateInstance(() => new YourProviderClient(this.options)); } async transcribe(options: LLMTranscribeOptions): Promise { this.logger?.info('Starting transcription with YourProvider', { options }); try { // Implement provider-specific transcription logic return { text: 'Transcribed text from audio', }; } catch (error) { this.logger?.error('Error in YourProvider transcription', { error }); throw error; } } } ``` **TextToSpeechService (if applicable)**: ```typescript import { type LLMLoggerInterface } from '../../../utilities/logger/LLMLoggerInterface'; import { LLMTextToSpeechService } from '../../../services/LLMTextToSpeechService.abstract'; import { LLMCreateSpeechOptions, LLMCreateSpeechResult, LLMProviders } from '../../../LLMService.typedefs'; import { YourProviderEntity } from '../YourProvider.entity'; export class YourProviderTextToSpeechService< Reporter extends LLMReporterInterface | undefined > extends LLMTextToSpeechService { constructor( logger: LLMLoggerInterface | undefined, reporter: Reporter, options?: LLMInstanceOptions[LLMProviders.YourProvider], ) { super(LLMProviders.YourProvider, logger, reporter, options); } // Implement instance getter with caching support get instance(): YourProviderClient { return this.getOrCreateInstance(() => new YourProviderClient(this.options)); } async createSpeech(options: LLMCreateSpeechOptions): Promise { this.logger?.info('Starting speech creation with YourProvider', { options }); try { // Implement provider-specific speech creation logic return { audio: Buffer.from('audio data'), mimeType: 'audio/mp3', }; } catch (error) { this.logger?.error('Error in YourProvider speech creation', { error }); throw error; } } } ``` ### 8. Create Service Factory The service builders are already defined in the constants file (step 6). Now implement the service factory in `src/providers/YourProvider/YourProviderService.factory.ts`: ```typescript import { LLMProviders, type LLMPurposes, type LLMServiceByPurpose, } from '@/LLMService.typedefs'; import { LLMServicePurposeFactory, type LLMCreateServiceOptions } from '@/services'; import { YOUR_PROVIDER_SERVICE_BUILDERS } from '@/providers/YourProvider/YourProvider.constants'; export class YourProviderServiceFactory< Reporter extends LLMReporterInterface | undefined > extends LLMServicePurposeFactory< LLMProviders.YourProvider, Reporter > { createService< Purpose extends LLMPurposes >( serviceOptions: LLMCreateServiceOptions, ): LLMServiceByPurpose[Purpose] { const { purpose, logger, reporter, options, } = serviceOptions; const serviceBuilder = YOUR_PROVIDER_SERVICE_BUILDERS[purpose]; if (!serviceBuilder) { throw new Error(`Purpose [${purpose}] is not supported for [${LLMProviders.YourProvider}] service`); } return serviceBuilder(logger, reporter, options) as unknown as LLMServiceByPurpose[Purpose]; } } ``` ### 9. Update Entry Point Files Update the provider's `index.ts`: ```typescript export * from './YourProvider.constants'; export * from './YourProvider.entity'; export * from './YourProvider.typedefs'; export * from './YourProviderService.factory'; export * from './services'; ``` Update the main providers `index.ts` at `src/providers/index.ts`: ```typescript // ... other providers export * from './YourProvider'; ``` ### 10. Update LLM Service Factory Modify `src/LLMService.factory.ts` to include your new provider: ```typescript import { LLMProviders, LLMServiceOptions, YourProviderOptions, } from './LLMService.typedefs'; import { YourProviderServiceFactory } from './providers/YourProvider'; import { YourProviderOptions } from './providers/YourProvider/YourProvider.typedefs'; export class LLMServiceFactory { static resolveProviderOptions( provider: T, optionsMap: { // ... other providers [LLMProviders.YourProvider]?: YourProviderOptions; }, ) { return optionsMap[provider]; } static getCompletionService | undefined>( serviceOptions: LLMServiceOptions, ) { const { provider, options, logger, reporter } = serviceOptions; switch (provider) { // ... other providers case LLMProviders.YourProvider: return new YourProviderServiceFactory().createService({ purpose: LLMPurposes.Completion, logger, reporter, options }); default: throw new Error(`Unsupported provider: ${provider}`); } } static getAssistanceService | undefined>( serviceOptions: LLMServiceOptions, ) { const { provider, options, logger, reporter } = serviceOptions; switch (provider) { // ... other providers case LLMProviders.YourProvider: return new YourProviderServiceFactory().createService({ purpose: LLMPurposes.Assistance, logger, reporter, options }); default: throw new Error(`Unsupported provider: ${provider}`); } } static getSpeechToTextService( serviceOptions: LLMServiceOptions, ) { const { provider, options, logger } = serviceOptions; switch (provider) { // ... other providers case LLMProviders.YourProvider: return YourProviderServiceFactory.createService({ purpose: LLMPurposes.SpeechToText, logger, options }); default: throw new Error(`Unsupported provider: ${provider}`); } } static getTextToSpeechService( serviceOptions: LLMServiceOptions, ) { const { provider, options, logger } = serviceOptions; switch (provider) { // ... other providers case LLMProviders.YourProvider: return YourProviderServiceFactory.createService({ purpose: LLMPurposes.TextToSpeech, logger, options }); default: throw new Error(`Unsupported provider: ${provider}`); } } } ```