/** * Which language a prompt is written in, to the extent it can be told cheaply. * * **Why the estimator needs this at all.** Measured against the counting * endpoint, `estimateTokens` is accurate on English (+1.0%) and badly low on every * other Latin language: German -37.3%, Spanish -22.9%, French -15.1%. Characters * per token says why — English 3.44, French 2.66, Spanish 2.53, German 2.02 — * while the estimator applied one divisor to all of them. Non-English text is * thinner in the merge table, so the same number of characters costs more tokens. * * **Accents are not the signal.** That was the first hypothesis and it was tested * and killed: a Spanish sample with zero accented characters comes out at -22.9%, * against -22.1% for accented Spanish. Diacritics correlate with non-English text * in a corpus and not in a prompt, and weighting them moved the figure by three * points. What separates these languages is which words they are made of. * * So this counts function words — the shortest, commonest, most language-specific * tokens there are. `the of and to` against `der die und ist` against * `que de la en`. * * **It answers `null` when unsure, and that is the important part.** A prompt is * not an essay: it can be three lines, or English instructions wrapped around a * Spanish example, or a JSON schema with no prose at all. Guessing on those would * apply a language's divisor to text that is not in that language, which is how a * fix for one case becomes a regression for four. `null` means "use the default", * and the default is the English-calibrated behaviour this has always had. */ /** Languages this can name. Exported so callers can key a table off it. */ export declare const DETECTABLE_LANGUAGES: readonly string[]; /** * The language of `text`, or `null` when no answer is safe. * * Case-insensitive, whole words only, and it stops reading after a bounded * prefix: a prompt can be a megabyte, the answer does not get better after a few * thousand words, and this runs inside `estimateTokens` on every call. */ export declare function detectTextLanguage(text: string): string | null; //# sourceMappingURL=language.d.ts.map