/** * kosha-discovery — Tokenizer-family inference for API-served models. * * Local runtimes (Ollama, llama.cpp) expose the tokenizer family via * {@link LocalRuntimeMetadata.tokenizerFamily}. For API-served models the * provider rarely publishes it explicitly, so we derive it from the * origin provider plus the model ID. The result is a best-effort hint * intended for tokenizer-aware downstream consumers (compression * libraries, routing policies, cost estimators) — not a guarantee. * @module */ /** * Infer a best-effort tokenizer-family identifier for a model. * * Returns a normalized string when the mapping is confident, or * `undefined` when no safe inference is available. Consumers must * treat the return value as a hint and fall back to their own * defaults when it is absent. * * Known families returned: * - `"o200k_base"` — OpenAI GPT-4o, GPT-4.1, o1/o3/o4 families * - `"cl100k_base"` — OpenAI GPT-4, GPT-3.5-turbo, text-embedding-3 * - `"claude"` — Anthropic Claude family (proprietary tokenizer) * - `"gemini"` — Google Gemini family (proprietary tokenizer) * - `"llama4"` — Meta Llama 4 family (~200k vocab BPE) * - `"llama3"` — Meta Llama 3.x family (128k vocab BPE) * - `"llama2"` — Meta Llama 2 / CodeLlama (32k vocab) * - `"mistral"` — Mistral/Mixtral family * - `"cohere"` — Cohere Command/Embed family * - `"deepseek"` — DeepSeek family * - `"qwen"` — Alibaba Qwen family * * @param originProvider - The model's creator (not the serving layer). * Examples: `"anthropic"`, `"openai"`, `"meta"`. * @param modelId - Provider-canonical model ID. * @returns Tokenizer-family hint, or `undefined`. */ export declare function inferTokenizerFamily(originProvider: string | undefined, modelId: string): string | undefined; //# sourceMappingURL=tokenizer-family.d.ts.map