# πŸ“– API Reference **Grabit Engine β€” Full API Documentation**
--- ## πŸ“‘ Table of Contents - [ScrapePluginManager](#scrapepluginmanager) - [ProviderManagerConfig](#providermanagerconfig) - [ProviderSource](#providersource) - [ProvidersManifest](#providersmanifest) - [ScrapeRequester](#scraperequester) - [ProviderModuleManifest](#providermodulemanifest) - [ProviderMetrics](#providermetrics) - [ProviderHealthReport](#providerhealthreport) - [ProviderContext](#providercontext) - [ProviderFetchOptions](#providerfetchoptions) - [Media Input Types](#media-input-types) - [IBaseMedia](#ibasemedia) - [MovieMedia](#moviemedia) - [SerieMedia](#seriemedia) - [ChannelMedia](#channelmedia) - [Output Types](#output-types) - [SourceProvider\](#sourceprovidert) - [MediaSource](#mediasource) - [SubtitleSource](#subtitlesource) - [Provider Configuration](#provider-configuration) - [ProviderConfig](#providerconfig) - [TProviderEntryPatterns](#tproviderentrypatterns) - [EProviderQueryKey](#eproviderquerykey) - [TProviderSelectors](#tproviderselectors) - [Provider Class](#provider-class) - [Error Classes](#error-classes) - [Utility Functions](#utility-functions) - [Extractor](#extractor-utilsextractor) - [Path](#path-utilspath) - [Similarity](#similarity-utilssimilarity) - [Standard](#standard-utilsstandard) - [Services](#services) - [Unpacker](#unpacker-servicesunpacker) - [Crypto](#crypto-servicescrypto) - [tldts](#tldts) - [ISO 639-1](#iso-639-1) --- ## `ScrapePluginManager` The main orchestrator β€” creates, manages, and queries provider plugins. | Method | Returns | Description | | --------------------------------------- | -------------------------------------- | --------------------------------------------------------------------------------------------------------------------------- | | `ScrapePluginManager.create(config)` | `Promise` | Creates the manager and loads all your provider plugins. | | `getStreams(request)` | `Promise` | Gets streams from **all active providers** for the given media. Returns everything in one list. | | `getSubtitles(request)` | `Promise` | Gets subtitles from **all active providers** for the given media. | | `getStreamsByScheme(scheme, request)` | `Promise` | Gets streams from **one specific provider** by its scheme. | | `getSubtitlesByScheme(scheme, request)` | `Promise` | Gets subtitles from **one specific provider** by its scheme. | | `closeOperations()` | `Promise` | Cancels all in-progress and queued scrape operations. Useful for cleanup when navigating away or aborting. | | `getProvidersByRequest(type, request)` | `ProviderModule[]` | Returns the list of active providers that match the given type (`"media"` or `"subtitle"`) and request, sorted by priority. | | `getMetrics()` | `ReadonlyMap` | Returns health stats for each provider (errors, successes, last activity). | | `getMetricsReport()` | `ProviderHealthReport[]` | Returns a full health report for every loaded provider β€” error rate, status, and more. | ### `ProviderManagerConfig` The configuration object passed to `ScrapePluginManager.create(config)`. | Field | Type | Required | Default | Description | | ----------------------------------------- | ---------------- | -------- | ----------- | --------------------------------------------------------------------------------------------------------------------------------------------- | | `source` | `ProviderSource` | βœ… | β€” | Where to load providers from. See [ProviderSource](#providersource) below. | | `tmdbApiKeys` | `string[]` | βœ… | β€” | One or more TMDB API keys used for metadata lookups. | | `debug` | `boolean` | ❌ | `false` | Enables extra logging and error information for development. | | `strict` | `boolean` | ❌ | `false` | Throw on validation errors instead of warning. | | `autoInit` | `boolean` | ❌ | β€” | Auto-initialize providers on load. | | `autoUpdateIntervalMinutes` | `number` | ❌ | `15` | Interval (in minutes) for auto-updating providers from remote sources. Minimum is 5. **Only applies to remote sources.** | | `cache` | `object` | ❌ | β€” | Caching configuration. See below. | | `cache.enabled` | `boolean` | βœ… | `false` | Whether to enable caching of provider data. | | `cache.TTL` | `number` | βœ… | `0` | Cache expiration TTL in milliseconds for scraped data. | | `cache.MODULE_TTL` | `number` | ❌ | `900_000` | TTL in milliseconds for caching provider modules. Separate from data TTL to allow different strategies. | | `cache.TMDB_TTL` | `number` | ❌ | β€” | TMDB response cache TTL in milliseconds. | | `cache.maxEntries` | `number` | ❌ | `10_000` | Maximum number of entries to store in the cache. | | `scrapeConfig` | `object` | ❌ | β€” | Scraping behaviour configuration. See below. | | `scrapeConfig.concurrentOperations` | `number` | ❌ | `5` | Maximum number of concurrent provider scraping operations. | | `scrapeConfig.maxAttempts` | `number` | ❌ | `3` | Maximum retry attempts for failed scrapes. | | `scrapeConfig.operationTimeout` | `number` | ❌ | `15_000` | Global timeout in milliseconds for the entire operation. When elapsed, remaining tasks are cancelled and only collected results are returned. | | `scrapeConfig.successQuorum` | `number` | ❌ | `undefined` | Minimum successful provider results to short-circuit the operation. Remaining tasks are cancelled once the quorum is met. | | `scrapeConfig.errorThresholdRate` | `number` | ❌ | `0.7` | Error rate (0–1) above which a provider is automatically disabled. Only evaluated after `minOperationsForEvaluation` operations. | | `scrapeConfig.minOperationsForEvaluation` | `number` | ❌ | `10` | Minimum total operations before a provider's error rate is evaluated against the threshold. | ### `ProviderSource` Union type: `GithubSource | RegistrySource | LocalSource`. Determines where provider modules are loaded from. #### `GithubSource` Fetches providers from a GitHub repository. Works in Node 18+, browsers, and React Native. | Field | Type | Required | Default | Description | | ---------------- | ----------------------------------------------------------------- | -------- | -------- | -------------------------------------------------------------------------------------------------------------------------------- | | `type` | `"github"` | βœ… | β€” | Source discriminant. | | `url` | `string` | βœ… | β€” | GitHub repo URL or shorthand `"owner/repo"`. | | `author` | `string` | ❌ | β€” | Author name. | | `branch` | `string` | ❌ | `"main"` | Branch name. | | `rootDir` | `string` | ❌ | `"/"` | Root directory for the repository (e.g. `"dist"`). | | `token` | `string` | ❌ | β€” | Auth token for private repos. | | `moduleResolver` | `(scheme: string, sourceCode: string) => Promise` | ❌ | β€” | Custom resolver that converts fetched source into a module. Required in browser/React Native; Node falls back to dynamic import. | ```typescript // React Native example const manager = await ScrapePluginManager.create({ source: { type: "github", url: "https://github.com/username/providers-repo", branch: "main", moduleResolver: async (_scheme, sourceCode) => { const exports: Record = {}; const module = { exports }; new Function("module", "exports", sourceCode)(module, exports); return (module.exports as any).default ?? module.exports; } }, tmdbApiKeys: ["your-tmdb-key"] }); ``` #### `RegistrySource` Providers are passed as pre-imported modules. Works in any JS runtime. | Field | Type | Required | Description | | ----------- | -------------------------------- | -------- | --------------------------------------------- | | `type` | `"registry"` | βœ… | Source discriminant. | | `name` | `string` | βœ… | Library name. | | `author` | `string` | ❌ | Author name. | | `providers` | `Record` | βœ… | Map of scheme β†’ pre-imported provider module. | ```typescript import myProvider from "./providers/my-provider"; const manager = await ScrapePluginManager.create({ source: { type: "registry", name: "my-providers", providers: { "my-provider": myProvider } }, tmdbApiKeys: ["your-tmdb-key"] }); ``` #### `LocalSource` Auto-imports providers from a manifest using a user-supplied resolve function. Works in any JS runtime. | Field | Type | Required | Default | Description | | ---------- | ------------------------------------------------------------------- | -------- | ------- | ------------------------------------------------------------------------------------------------------------------------------------ | | `type` | `"local"` | βœ… | β€” | Source discriminant. | | `manifest` | `ProvidersManifest` | βœ… | β€” | The manifest object β€” import or require it yourself. | | `rootDir` | `string` | ❌ | `"./"` | Base directory prepended to every provider path in the manifest. Trailing slash added automatically. | | `resolve` | `(modulePath: string) => ProviderModule \| Promise` | βœ… | β€” | Module resolver called for each provider with the full path. Must return the `ProviderModule` (or a module whose `.default` is one). | ```typescript // Node.js const manager = await ScrapePluginManager.create({ source: { type: "local", manifest: require("./manifest.json"), rootDir: "./providers", resolve: (p) => require(p) }, tmdbApiKeys: ["your-tmdb-key"] }); ``` ### `ProvidersManifest` The manifest describing a provider library β€” used by `GithubSource` (loaded from the repo) and `LocalSource` (passed directly). | Field | Type | Required | Description | | ----------- | ---------------------------------------- | -------- | -------------------------------------------------------------------------------------- | | `name` | `string` | βœ… | Library name. | | `author` | `string` | ❌ | Author name. | | `providers` | `Record` | βœ… | Map of scheme β†’ provider manifest. The key is the provider's unique scheme identifier. | --- ## `ScrapeRequester` The request object accepted by the manager's `getStreams()` and `getSubtitles()` methods. You only need to provide the **minimum required fields** for your media type β€” the TMDB service automatically fills in any missing data (title, year, duration, IMDB ID, localized titles, etc.) before the scrape begins. > **How it works:** When you call `getStreams(requester)` or `getSubtitles(requester)`, the manager calls `TMDB.createRequesterMedia()` internally. This acts as a **polyfill** β€” it fetches metadata from TMDB and merges it with whatever you already provided. Fields you supply are **never overwritten** (except `localizedTitles`, which are always enriched from TMDB translations). If you provide a complete media object, TMDB still runs but only fills gaps. ### Minimum Required Fields | Media Type | Required Fields | Example | | ---------- | ------------------------------------- | ----------------------------------------------------------- | | `movie` | `type`, `tmdbId` | `{ type: "movie", tmdbId: "27205" }` | | `serie` | `type`, `tmdbId`, `season`, `episode` | `{ type: "serie", tmdbId: "1396", season: 1, episode: 1 }` | | `channel` | `type`, `channelId`, `channelName` | `{ type: "channel", channelId: "cnn", channelName: "CNN" }` | > **Note:** Channels don't use TMDB β€” they are passed through as-is. ### Fields | Field | Type | Required | Description | | ------------------- | ------------------------------------------------------------ | -------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `media` | `RequesterMovieMedia \| RequesterSerieMedia \| ChannelMedia` | βœ… | The movie, show, or channel you want to scrape. Can be a **partial** object β€” only `type` + `tmdbId` are required for movies; `type` + `tmdbId` + `season` + `episode` for series. TMDB fills the rest. | | `targetLanguageISO` | `string` | βœ… | Language code like `"en"` or `"fr"`. Used to fetch localized titles from TMDB. | | `userAgent` | `string` | ❌ | Custom user-agent string for requests. | | `proxyAgent` | `HttpsProxyAgent \| SocksProxyAgent \| HttpProxyAgent` | ❌ | Optional proxy for routing requests. | | `userIP` | `string` | ❌ | Optional user IP address of the requester. | ### Examples ```typescript // Minimal movie request β€” TMDB auto-fills title, year, duration, imdbId, etc. const streams = await manager.getStreams({ media: { type: "movie", tmdbId: "27205" }, targetLanguageISO: "en" }); // Minimal series request const serieStreams = await manager.getStreams({ media: { type: "serie", tmdbId: "1396", season: 1, episode: 1 }, targetLanguageISO: "en" }); // Full media object β€” TMDB only fills gaps (e.g. localizedTitles) const fullStreams = await manager.getStreams({ media: { type: "movie", title: "Inception", duration: 148, releaseYear: 2010, tmdbId: "27205", imdbId: "tt1375666" }, targetLanguageISO: "en" }); ``` --- ## `ProviderModuleManifest` Describes a provider's metadata inside a manifest. The provider's unique scheme identifier is **not** stored inside this object β€” it is the key under which this manifest is registered in the `providers` map (e.g. `{ "my-provider": { name: "...", ... } }`). | Field | Type | Required | Description | | --------------------- | ----------------------- | -------- | ---------------------------------------------------------------------------------- | | `name` | `string` | βœ… | Human-readable provider name. | | `version` | `string` | βœ… | Semver version string (e.g. `"1.0.0"`). | | `active` | `boolean` | βœ… | Whether the provider is enabled. | | `language` | `string \| string[]` | βœ… | ISO language code(s) β€” single string (e.g. `"en"`) or array (e.g. `["en", "fr"]`). | | `type` | `"media" \| "subtitle"` | βœ… | What the provider returns. | | `env` | `"node" \| "universal"` | βœ… | Runtime compatibility. | | `supportedMediaTypes` | `MediaType[]` | βœ… | `"movie"`, `"serie"`, `"channel"`. | | `priority` | `number` | ❌ | Lower = higher priority (default: `0`). | | `dir` | `string` | ❌ | Directory path for the provider folder. | --- ## `ProviderMetrics` Runtime health counters tracked per provider. | Field | Type | Required | Description | | --------------- | -------- | -------- | ----------------------------------- | | `errors` | `number` | βœ… | Total failed operations. | | `successes` | `number` | βœ… | Total successful operations. | | `lastOperation` | `Date` | βœ… | Timestamp of most recent operation. | --- ## `ProviderHealthReport` A detailed health snapshot returned by `getMetricsReport()`. | Field | Type | Required | Description | | ----------------- | --------- | -------- | ---------------------------------------- | | `moduleName` | `string` | βœ… | Provider module name. | | `errors` | `number` | βœ… | Total errors. | | `successes` | `number` | βœ… | Total successes. | | `totalOperations` | `number` | βœ… | Sum of errors + successes. | | `errorRate` | `number` | βœ… | Ratio `0.0` – `1.0`. | | `active` | `boolean` | βœ… | Whether the module is currently enabled. | | `lastOperation` | `Date` | βœ… | Timestamp of most recent operation. | --- ## `ProviderContext` The context object passed as the second argument to every `getStreams` / `getSubtitles` handler. | Property | Type | Required | Description | | --------------------- | ------------------------------------------------------ | -------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `xhr.fetch` | `(url, options, requester) => Promise` | βœ… | Makes an HTTP request, automatically applying the requester's user-agent and proxy. Supports timeout and retry options. | | `xhr.fetchResponse` | `(url, options, requester) => Promise` | βœ… | Like `fetch` but parses and returns the typed response body directly. | | `xhr.handleResponse` | `(response) => Promise` | βœ… | Parses a raw `Response` object into a typed value, throwing on error status codes. | | `xhr.status` | `(url, options, requester) => Promise<{ ok, status }>` | βœ… | Lightweight check β€” returns whether the request succeeded and its HTTP status code. | | `cheerio.$load` | `(html: string) => CheerioAPI` | βœ… | Direct access to `cheerio.load` for parsing raw HTML strings you already have, without making an HTTP request. | | `cheerio.load` | `(url, requester, xhrCtx) => Promise<{ $, response }>` | βœ… | Fetches a page and loads it into Cheerio for DOM traversal. Mimics a real browser request with appropriate headers. | | `cheerio.sortResults` | `($page, selectors, requester) => Promise` | βœ… | Scores and sorts search result elements by similarity to the requester's media (title, year, duration). Score range: 0–170 for movies/series, 0–100 for channels. | | `puppeteer.launch` | `(url, request) => Promise<{ browser, page }>` | βœ… | **Node.js only.** Launches a headless real browser via `puppeteer-real-browser`. Handles Cloudflare challenges automatically. | | `log` | `DebugLogger` | βœ… | Scoped debug logger bound to this provider's scheme. Provides `.info()`, `.warn()`, `.error()`, and `.debug()` methods. Output respects the manager's `debug` flag β€” always on in the `test-provider` CLI. | --- ## `ProviderFetchOptions` Options accepted by `ctx.xhr.fetch` / `ctx.xhr.fetchResponse` / `ctx.xhr.status`. | Field | Type | Required | Default | Description | | ----------------- | --------- | -------- | ------- | ------------------------------------------------------------------- | | `attachUserAgent` | `boolean` | ❌ | `false` | Attach the requester's `User-Agent` header to the request. | | `attachProxy` | `boolean` | ❌ | `true` | Route the request through the requester's proxy if one is provided. | > Also accepts all fields from `RequestInit`, `RequestRetryInit`, and `RequestTimeoutInit`. --- ## Media Input Types ### `IBaseMedia` Base fields shared by `MovieMedia` and `SerieMedia`. | Field | Type | Required | Description | | ------------------- | ---------- | -------- | ----------------------------------------------------------------- | | `original_language` | `string` | βœ… | Original language of the media (e.g. `"en"`, `"fr"`). | | `title` | `string` | βœ… | Original title (always in English). | | `localizedTitles` | `string[]` | βœ… | Localized titles in the requester's language (populated by TMDB). | | `duration` | `number` | βœ… | Duration in minutes. | | `releaseYear` | `number` | βœ… | Release year. | | `tmdbId` | `string` | βœ… | TMDB ID. | | `imdbId` | `string` | ❌ | IMDB ID. | > **Note:** When using the manager's `getStreams()` / `getSubtitles()`, you only need to provide `type` + `tmdbId` (for movies) or `type` + `tmdbId` + `season` + `episode` (for series). The TMDB service fills in all other fields automatically. See [ScrapeRequester](#scraperequester) for details. ### `MovieMedia` `IBaseMedia` with `type: "movie"`. No additional fields. ### `SerieMedia` `IBaseMedia` plus the following fields: | Field | Type | Required | Description | | ----------- | --------- | -------- | ------------------------------------------------------ | | `type` | `"serie"` | βœ… | Discriminant. | | `season` | `number` | βœ… | Season number. | | `episode` | `number` | βœ… | Episode number. | | `ep_tmdbId` | `string` | ❌ | Episode TMDB ID (auto-filled by TMDB if not provided). | | `ep_imdbId` | `string` | ❌ | Episode IMDB ID (auto-filled by TMDB if not provided). | ### `ChannelMedia` | Field | Type | Required | Description | | ------------- | ----------- | -------- | -------------------------- | | `type` | `"channel"` | βœ… | Discriminant. | | `channelId` | `string` | βœ… | Unique channel identifier. | | `channelName` | `string` | βœ… | Channel display name. | ### Type Aliases | Name | Definition | | ------------- | ------------------------------------------ | | `Media` | `MovieMedia \| SerieMedia \| ChannelMedia` | | `MediaType` | `"movie" \| "serie" \| "channel"` | | `MediaIdType` | `"tmdb" \| "imdb"` | --- ## Output Types ### `SourceProvider` Base interface extended by `MediaSource` and `SubtitleSource`. | Field | Type | Required | Description | | -------------------- | ------------------------ | -------- | -------------------------------------- | | `scheme` | `string` | βœ… | Provider scheme identifier. | | `providerName` | `string` | βœ… | Human-readable provider name. | | `language` | `string` | βœ… | ISO language code. | | `format` | `T` | βœ… | Media or subtitle format string. | | `xhr.haveCorsPolicy` | `boolean` | βœ… | Whether the source has a CORS policy. | | `xhr.headers` | `Record` | βœ… | Required request headers for playback. | ### `MediaSource` Extends `SourceProvider<"m3u8" | "dash" | "mp4" | "webm" | "mkv" | "flv" | "avi" | "mov">`. | Field | Type | Required | Description | | ---------- | --------------------------- | -------- | -------------------------------------------------------------- | | `fileName` | `string` | βœ… | Display file name. | | `playlist` | `string \| PlaylistEntry[]` | βœ… | Direct URL or an array of bandwidth/resolution/source entries. | ### `SubtitleSource` Extends `SourceProvider<"srt" | "vtt">` (without inherited `language`). | Field | Type | Required | Description | | -------------- | -------- | -------- | ------------------------------------------------ | | `fileName` | `string` | βœ… | Display file name. | | `language` | `string` | βœ… | ISO language code (e.g. `"en"`). | | `languageName` | `string` | βœ… | Human-readable language name (e.g. `"English"`). | | `url` | `string` | βœ… | Direct URL to the subtitle file. | ### Internal Types | Name | Definition | Description | | ------------------------ | -------------------------------------------------- | ---------------------------------------------------------- | | `InternalMediaSource` | `Omit` | Use this in `getStreams()` return value inside a provider. | | `InternalSubtitleSource` | `Omit` | Use this in `getSubtitles()`. | --- ## Provider Configuration ### `ProviderConfig` Configuration object used to define a provider's identity, endpoints, and behaviour. | Field | Type | Required | Description | | -------------------------------------- | ------------------------ | -------- | ---------------------------------------------------------------------------------------------------------------------------------------------- | | `scheme` | `string` | βœ… | Unique provider identifier (e.g. `"vidsrc"`). | | `name` | `string` | βœ… | Human-readable name. | | `language` | `string \| string[]` | βœ… | ISO language code(s) β€” single string (e.g. `"en"`) or array (e.g. `["en", "fr"]`). | | `baseUrl` | `string` | βœ… | Provider homepage URL. | | `entries` | `TProviderEntries` | βœ… | Media type β†’ endpoint pattern map. | | `mediaIds` | `MediaIdType[]` | ❌ | Preferred ID types, ordered by preference. Default: `["tmdb"]`. | | `contentAreCORSProtected` | `boolean` | ❌ | Whether content responses have CORS restrictions. | | `xhr.validateSources` | `boolean` | ❌ | When enabled, fetches each media URL before returning it to verify the source is accessible. Filters dead links at the cost of extra requests. | | `xhr.headers` | `Record` | ❌ | Custom headers sent with every request to this provider. | | `xhr.retries.maxAttempts` | `number` | ❌ | Max retry attempts per request. | | `xhr.retries.timeout` | `number` | ❌ | Per-attempt timeout in ms. | | `useSearchAlgorithm.enabled` | `boolean` | ❌ | Use the search-and-score algorithm to find media. | | `useSearchAlgorithm.minimumMatchScore` | `number` | ❌ | Minimum score (0–170) to accept a match. | ### `TProviderEntryPatterns` Defines how a media type maps to provider endpoints. | Field | Type | Required | Description | | ---------- | --------------------------------------------- | -------- | ------------------------------------------------------------------------------ | | `endpoint` | `string` | βœ… | URL path with placeholders, e.g. `"/embed/movie?tmdb={id:string}"`. | | `pattern` | `string` | ❌ | Extra pattern appended for search/matching, e.g. `"-{season:2}x{episode:2}/"`. | | `queries` | `Record` | ❌ | Additional static query parameters. | #### Supported Placeholder Formats | Syntax | Example | Result | | ---------------- | ------------------ | ---------------------------------------- | | `{key:string}` | `{id:string}` | Raw string value. | | `{key:uri}` | `{title:uri}` | `encodeURIComponent`. | | `{key:form-uri}` | `{title:form-uri}` | `encodeURIComponent` with spaces as `+`. | | `{key:N}` | `{season:2}` | Zero-padded to N digits. | | `{N}` | `{0}` | Indexed arg via `EProviderQueryKey`. | ### `EProviderQueryKey` Enum mapping numeric index placeholders (`{0}`, `{1}`, …) to media fields. | Key | Index | Resolves To | | --------- | ----- | ---------------------------------- | | `id` | 0 | Preferred media ID (TMDB or IMDB). | | `tmdb` | 1 | TMDB ID. | | `imdb` | 2 | IMDB ID. | | `title` | 3 | Media title. | | `year` | 4 | Release year. | | `season` | 5 | Season number. | | `episode` | 6 | Episode number. | | `ep_id` | 7 | Preferred episode ID. | | `ep_tmdb` | 8 | Episode TMDB ID. | | `ep_imdb` | 9 | Episode IMDB ID. | ### `TProviderSelectors` Cheerio selectors used by `cheerio.sortResults`. All values are CSS selector strings. | Field | Type | Required | Description | | ------------------ | -------- | -------- | -------------------------------------------------- | | `$results` | `string` | βœ… | Selector for the results container. | | `$result_entry` | `string` | βœ… | Selector for each result row inside the container. | | `$result_title` | `string` | βœ… | Selector for the title element inside a result. | | `$result_year` | `string` | ❌ | Selector for the year element. | | `$result_date` | `string` | ❌ | Selector for the release date element. | | `$result_duration` | `string` | ❌ | Selector for the duration element. | --- ## `Provider` Class The runtime class built from a `ProviderConfig`. Constructed via the static factory method. ```typescript const provider = Provider.create(config); ``` | Method | Returns | Description | | -------------------------------------------------- | ---------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `Provider.create(config)` | `Provider` | Creates a `Provider` instance from a `ProviderConfig`. | | `createResourceURL(requester, useLocalizedTitle?)` | `URL` | Builds the full scrape URL by substituting endpoint placeholders with media data from the requester. `useLocalizedTitle` (default: `true`) controls whether the localized title is used instead of the original title (only applies when the provider's language is not English). | --- ## Error Classes Custom error types thrown during scraping operations. Both extend `Error` and can be identified with the `isCustomError()` utility. ### `HttpError` Thrown when an HTTP request fails. | Field | Type | Required | Description | | ------------ | -------- | -------- | ----------------- | | `statusCode` | `number` | βœ… | HTTP status code. | | `message` | `string` | βœ… | Error message. | ### `ProcessError` Thrown when a provider's scraping logic encounters a non-HTTP error. | Field | Type | Required | Description | | --------- | -------- | -------- | -------------- | | `message` | `string` | βœ… | Error message. | --- ## Utility Functions All utilities below are exported from the package root (`import { ... } from "grabit-engine"`). ### Extractor (`utils/extractor`) Helpers for extracting data from HTML and JavaScript source strings. | Function | Signature | Description | | -------------------------------------- | ----------------------------------------- | ------------------------------------------------------------------------------------------------------ | | `extractYearFromText` | `(text) β†’ number \| null` | Extracts the first 4-digit year (1900–2099) from a string. | | `extractSetCookies` | `(headers) β†’ string[]` | Normalises `Set-Cookie` headers into a `string[]`, handles `Headers`-like objects. | | `extractEvalCode` | `(source) β†’ string \| null` | Returns the first `eval(…)` call found in source. | | `extractVariableByJSONKey` | `(source, requiredKeys) β†’ object \| null` | Finds a `var/let/const` declaration whose object value contains **all** required keys. | | `extractContructorJSONArguments` | `(codeString) β†’ object \| null` | Parses the first function/constructor call's arguments from a JS snippet. | | `extractContructorJSONArgumentsByName` | `(source, functionName) β†’ object \| null` | Same as above but searches for a specific named function in a larger source. | | `extractVariableJSON` | `(source, varName) β†’ object \| null` | Extracts and parses a `var/let/const varName = { … }` object. | | `extractVariableValue` | `(source, varName) β†’ string \| null` | Extracts a scalar value (string/number/bool/null) from `const x = …` or bare `x.prop = …` assignments. | ### Path (`utils/path`) URL and path construction utilities. | Function | Signature | Description | | ------------------- | ------------------------------------------- | --------------------------------------------------------------------------------- | | `stringFromPattern` | `(pattern, params) β†’ string` | Replaces `{key:type}` placeholders in a pattern string with values from `params`. | | `formatString` | `(pattern, args) β†’ string` | Replaces indexed `{0}`, `{1}` … placeholders with positional args. | | `encodeURI` | `(str, type?) β†’ string` | URI-encodes a string; `type: "form-uri"` encodes spaces as `+`. | | `buildRelativePath` | `(entry, params, includePattern?) β†’ string` | Builds a provider relative URL from a `TProviderEntryPatterns` entry. | | `pathJoin` | `(...parts) β†’ string` | Joins path segments, deduplicating slashes. | ### Similarity (`utils/similarity`) String and media similarity scoring. | Function | Signature | Description | | ---------------------------- | --------------------------------- | ----------------------------------------------------------------------------------------------------------------------- | | `calculateMatchScore` | `(criteria, media) β†’ number` | Scores a candidate (title, year, duration) against a `Media` object. Range 0–170 for movies/series, 0–100 for channels. | | `advanceLevenshteinDistance` | `(itemName, targetName) β†’ number` | Levenshtein distance with bonuses for prefix/word matches (lower = more similar). | | `levenshteinDistance` | `(a, b) β†’ number` | Standard Levenshtein edit distance. | | `cosineSimilarity` | `(a, b) β†’ number` | Cosine similarity of word-frequency vectors. Returns 0–1. | ### Standard (`utils/standard`) General-purpose runtime helpers. | Function | Signature | Description | | ----------------------- | ------------------------------------------------ | ------------------------------------------------------------------------- | | `isDevelopment` | `() β†’ boolean` | `true` when `process.env.ENV !== "production"`. | | `isNode` | `() β†’ boolean` | `true` when running in a Node.js environment. | | `isCustomError` | `(error) β†’ error is HttpError \| ProcessError` | Type guard for custom error classes. | | `sanitizeMessage` | `(value) β†’ string` | Strips escaped quotes and normalises whitespace. | | `minutesToMilliseconds` | `(minutes) β†’ number` | Converts minutes β†’ ms. | | `hoursToMilliseconds` | `(hours) β†’ number` | Converts hours β†’ ms. | | `secondsToMilliseconds` | `(seconds) β†’ number` | Converts seconds β†’ ms. | | `customParseInt` | `(input) β†’ number` | Parses a digit-only string; returns `NaN` for anything else. | | `commaSplitter` | `(input) β†’ string[]` | Splits a comma-separated string, trimming each part. | | `delay` | `(ms) β†’ Promise` | Awaitable sleep. | | `retriesCount` | `(attempts, maxAttempts, retryScore?) β†’ number` | Computes the logical retry row number. | | `excuteWithRetries` | `(fn, maxAttempts?, backoffDelay?) β†’ Promise` | Runs `fn` up to `maxAttempts` times with optional delay between attempts. | | `sorter` | `(items, compareFn) β†’ Promise` | Async merge-sort with an async comparator. | | `createCookiesFromSet` | `(headers) β†’ string` | Converts `Set-Cookie` headers into a single `Cookie` header string. | | `joinCookies` | `(existingCookies, newCookies) β†’ string` | Merges two cookie strings, deduplicating entries. | | `attachExtension` | `(extension, urlOrPath) β†’ string` | Appends or replaces the file extension on a URL or path. | | `shuffleArray` | `(array) β†’ T[]` | Returns a new array with elements randomly shuffled (Fisher–Yates). | --- ## Services ### Unpacker (`services/unpacker`) Utilities for handling P.A.C.K.E.R.-obfuscated JavaScript. | Function | Signature | Description | | -------------- | -------------------- | --------------------------------------------------- | | `detectPacked` | `(source) β†’ boolean` | Returns `true` if `source` is P.A.C.K.E.R. encoded. | | `unpackV1` | `(code) β†’ string` | Unpacks a P.A.C.K.E.R. v1 encoded JS string. | ### Crypto (`services/crypto`) Re-exports Node's built-in `crypto` module as a named export for cross-environment use. For **React Native**, the native `crypto` module is not available. Install [`react-native-quick-crypto`](https://www.npmjs.com/package/react-native-quick-crypto) as a drop-in polyfill: ```bash npm install react-native-quick-crypto ``` This module also polyfills `atob` / `btoa` for environments that don't expose them globally (e.g. React Native < 0.74), using the optional peer dependency [`base-64`](https://www.npmjs.com/package/base-64). ```typescript import { Crypto } from "grabit-engine"; const hash = Crypto.createHash("md5").update("hello").digest("hex"); ``` ### tldts Re-exported [`tldts`](https://www.npmjs.com/package/tldts) for URL hostname parsing, domain extraction, and public suffix lookups. ```typescript import { tldts } from "grabit-engine"; tldts.parse("https://www.example.co.uk/path"); // β†’ { hostname: 'www.example.co.uk', domain: 'example.co.uk', publicSuffix: 'co.uk', ... } tldts.getDomain("https://www.example.co.uk"); // β†’ 'example.co.uk' ``` ### ISO 639-1 Re-exported [`iso-639-1`](https://www.npmjs.com/package/iso-639-1) for convenience. ```typescript import { ISO6391 } from "grabit-engine"; ISO6391.getName("fr"); // β†’ "French" ISO6391.validate("en"); // β†’ true ```