# webnovel-downloader

[![npm version](https://img.shields.io/npm/v/@duyquangnvx/webnovel-downloader.svg)](https://www.npmjs.com/package/@duyquangnvx/webnovel-downloader)
[![license: MIT](https://img.shields.io/npm/l/@duyquangnvx/webnovel-downloader.svg)](./LICENSE)
[![node](https://img.shields.io/node/v/@duyquangnvx/webnovel-downloader.svg)](https://nodejs.org)
[![types: included](https://img.shields.io/npm/types/@duyquangnvx/webnovel-downloader.svg)](./dist)

A pluggable, type-safe webnovel downloader for Node.js. Point it at a novel's
URL on a supported site and get back structured data — metadata plus ordered,
normalized chapters — ready to serialize into EPUB, TXT, JSON, or Markdown with
a formatter of your choice.

Part of the [`webnovel-studio`](https://github.com/duyquangnvx/webnovel-studio)
toolkit, but usable standalone.

> **Status:** Pre-alpha. The public API may still change between minor versions.

## Features

- **Pluggable adapters** — each site is a self-contained `SiteAdapter`, resolved automatically from the URL.
- **Type-safe end to end** — every external response is validated with [Zod](https://zod.dev); results are discriminated unions you narrow with `switch` (no `any`, no casts).
- **Concurrent, but polite** — parallel chapter fetches behind a per-host token-bucket rate limiter (default ~2 req/s) with automatic retries and exponential backoff.
- **Resumable** — crash-safe state; re-run to skip finished chapters and retry only the failures, or resume a `partial` run from a token.
- **Browser tier on demand** — transparently escalates to a real (headless) browser for Cloudflare-fronted or JS-rendered sites; plain HTTP everywhere else.
- **Format-agnostic output** — normalized plain-text chapters; you own the file writing.
- **Ships ESM + CommonJS** with full type declarations. `import` and `require` both work. Node ≥ 20.

## Install

```bash
npm install @duyquangnvx/webnovel-downloader
# optional — only needed for Cloudflare / JS-rendered sites (e.g. wikicv):
npm install patchright
```

`patchright` (or `playwright`) is an optional peer dependency for the browser
transport tier. `truyenfull` and `metruyenchu` work over plain HTTP without it.

## Quick start

```ts
import { downloader } from "@duyquangnvx/webnovel-downloader";

const result = await downloader.download("https://truyenfull.today/tien-nghich/");

if (result.status === "success") {
  const { metadata, chapters } = result.data;
  console.log(`${metadata.title} — ${chapters.length} chapters`);
}
```

`downloader` is a shared, process-wide singleton with sensible defaults (auto
transport, headless, ~2 req/s per host). Use it for the common case. When you
need custom throttling, retries, a headed browser, or deterministic teardown,
build your own instance with [`createDownloader()`](#configuration).

## Supported sites

Pass the novel's **landing page** (its table-of-contents URL). The adapter is
resolved from the host automatically — you never name it.

| Site | Adapter id | Transport | Notes |
|---|---|---|---|
| `truyenfull.today` (+ `truyenfull.vision`, `truyenfull.vn`) | `truyenfull` | HTTP | Legacy `.vision` / `.vn` hosts auto-rewrite to `.today`. |
| `metruyenchu.com.vn` | `metruyenchu-com-vn` | HTTP | Distinct from the dead `metruyenchu.com` brand. |
| `wikicv.net` (+ `truyenwikidich.net`) | `wikicv` | **Browser required** | Needs `patchright`/`playwright`. May hit Cloudflare. `truyenwikidich.net` auto-rewrites. |
| `tangthuvien.net` | `tangthuvien` | — | Scaffold only — parsers TBD, not yet functional. |

Check support at runtime with `downloader.canHandle(url)` (boolean) or enumerate
sites with `downloader.supportedSites()`.

## Core API

Methods on a `Downloader` (the `downloader` singleton or a `createDownloader()` instance):

| Method | Returns | Description |
|---|---|---|
| `download(url, options?)` | `Promise<DownloadResult>` | Fetch the whole novel. Returns a result envelope; throws only on abort or browser setup failure. |
| `fetchMetadata(url, options?)` | `Promise<NovelMetadata>` | Metadata only (title, author, cover, `totalChapters`, …). Returns a bare value and **throws** on failure. |
| `fetchChapterList(url, options?)` | `Promise<readonly ChapterRef[]>` | The table of contents (index/title/url) **without** downloading any chapter bodies. Honors `chapterRange`. **Throws** on failure. |
| `canHandle(url)` | `boolean` | Whether a registered adapter can handle this URL. |
| `supportedSites()` | `{ id, displayName, hostnames }[]` | Plain, serializable list of registered sites. |
| `dispose()` | `Promise<void>` | Release held resources (browser pool, HTTP sockets). Call on `createDownloader()` instances when done. |

### The result envelope

`download()` returns a discriminated union — always narrow on `status` before
reading `data`:

```ts
const result = await downloader.download(url);

switch (result.status) {
  case "success":
    // Every in-range chapter was fetched.
    console.log(result.data.chapters.length);
    break;
  case "partial":
    // Some chapters permanently failed. `data` has the rest (with gaps at the
    // failed indices); `failures` lists what broke. Resumable when you passed
    // `resume` — narrow on `resumable` before reading `resumeToken`.
    console.warn(`${result.failures.length} chapters failed`);
    if (result.resumable) {
      await downloader.download(url, { resume: { token: result.resumeToken } });
    }
    break;
  case "error":
    // No novel produced (bad/unknown URL, invalid range, metadata/TOC failure).
    // No `data`. `error.code` is a stable string you can switch on.
    console.error(result.error.code, result.error.message);
    break;
}
```

`download()` **throws** only when it can't reach the envelope stage: an abort via
`options.signal` (`CancelledError`), or a browser-tier site whose browser can't
start — no `patchright`/`playwright` installed (`BrowserModuleNotInstalledError`)
or `transport: "http-only"` (`ParseError`). Wrap those in `try/catch`.

### Output shape

```ts
NovelData    = { metadata: NovelMetadata; chapters: readonly Chapter[] }
NovelMetadata = { sourceUrl, sourceSite, title, author, description,
                  coverUrl?, genres, status, totalChapters?, fetchedAt }
Chapter       = { index, title, url, volume?, content, wordCount, fetchedAt }
```

`chapters` is sorted by source `index` (0-based); `content` is normalized plain
text with paragraphs separated by blank lines. `totalChapters` is the site's
reported count and may be absent — never inferred from what was fetched.

## Options

Per-call options for `download()` / `fetchMetadata()` / `fetchChapterList()` —
all optional:

| Option | Type | Purpose |
|---|---|---|
| `concurrency` | `number` | Parallel chapter fetches (default `4`). Lower it to be gentler. |
| `chapterRange` | `{ from?, to? }` | Fetch a contiguous slice. **0-based, inclusive** — chapter 1 is index 0, so `{ from: 0, to: 9 }` is the first 10. Invalid bound ⇒ `{ status: "error" }`. |
| `cache` | `{ dir, maxAgeMs? }` | On-disk HTTP response cache (ETag/304) so reruns skip refetching. |
| `resume` | `true \| { stateFile } \| { token }` | Crash-safe resume (see below). |
| `signal` | `AbortSignal` | Cancel mid-run; `download()` then rejects with `CancelledError`. |
| `onEvent` | `(e: DownloadEvent) => void` | Progress/lifecycle callback (see below). |
| `adapter` | `SiteAdapter` | Force a specific adapter instead of resolving from the URL. |

`rateLimit`, `retry`, and `transport` are **instance-level**, not per-call — set
them via `createDownloader()`.

### Progress events

Pass `onEvent` and discriminate on `type`:

```ts
await downloader.download(url, {
  onEvent: (e) => {
    if (e.type === "toc:complete") console.log(`found ${e.total} chapters`);
    if (e.type === "progress") console.log(`${e.completed}/${e.total}`);
  },
});
```

Event types: `metadata:fetched`, `toc:progress`, `toc:complete`,
`chapter:start`, `chapter:success`, `chapter:failed`, `progress`, plus transport
signals `cache:hit`, `rate-limit:wait`, and `http:retry`.

## Configuration

`createDownloader()` builds an instance wired with the built-in adapters, but
lets you set transport, throttling, retries, and logging:

```ts
import { createDownloader } from "@duyquangnvx/webnovel-downloader";

const dl = createDownloader({
  rateLimit: { requestsPerSecond: 1 },        // per host; default 2
  retry: { retries: 5, backoff: "exponential" },
  transport: { mode: "auto" },                // "auto" | "http-only" | "browser-required"
  logLevel: "info",                            // pino level shortcut
});
try {
  const result = await dl.download(url);
} finally {
  await dl.dispose();  // releases the browser pool and HTTP sockets
}
```

It also accepts `http` (a custom `HttpClient` replacing the whole transport
stack), `logger` (a pino instance), and `adapters` (override the registered set —
combine `builtinAdapters()` with your own).

## Transport tier (Cloudflare / JS-rendered sites)

Sites behind Cloudflare or rendering content via JS need a real browser. Install
one peer:

```bash
npm install patchright   # recommended — built to slip past Cloudflare unattended
npm install playwright   # works for non-protected pages
```

The module is resolved at runtime (`patchright` first, then `playwright`) and the
default `auto` transport launches it **headless** only when an adapter needs it.
If a Cloudflare challenge blocks the headless run, launch a **headed** window and
solve the checkbox once (cookies then cache ~30 min within that instance):

```ts
const dl = createDownloader({
  transport: { mode: "auto", browserOptions: { headed: true } },
});
```

Headed mode needs a display — run it where a human can click, not on a headless
CI box. Transport modes:

- `"auto"` — undici first, escalate to the browser on a Cloudflare challenge.
- `"http-only"` — never launch a browser; browser-only adapters throw a `ParseError`.
- `"browser-required"` — route every request through the browser.

A challenge the headless run can't clear returns `{ status: "error" }` with
`error.code === "CHALLENGE_UNRESOLVED"` (and a `hint`) — do a headed solve.

## Resume & partial downloads

```ts
await downloader.download(url, { resume: true });
```

With `resume: true`, state and per-chapter files live under the OS cache dir
(`%LOCALAPPDATA%\...` on Windows, `~/Library/Caches/...` on macOS,
`$XDG_CACHE_HOME`/`~/.cache/...` on Linux — all under
`webnovel-downloader/state/`). Re-running skips finished chapters and retries the
failed ones. To resume a specific `partial` run instead, pass its token:

```ts
const result = await downloader.download(url, { resume: true });
if (result.status === "partial" && result.resumable) {
  await downloader.download(url, { resume: { token: result.resumeToken } });
}
```

## Error handling

Every error carries a stable, machine-readable `code`. The error channel is a
discriminated union, so a `switch` narrows to the concrete error and its typed
fields:

```ts
if (result.status === "error") {
  switch (result.error.code) {
    case "HTTP_ERROR":
      console.error(`HTTP ${result.error.status} for ${result.error.url}`);
      break;
    case "RATE_LIMITED":
      console.error(`rate limited; retry after ${result.error.retryAfterMs}ms`);
      break;
    case "CHALLENGE_UNRESOLVED":
      console.error(`Cloudflare challenge (${result.error.hint})`);
      break;
    default:
      console.error(result.error.message);
  }
}
```

Codes: `ADAPTER_NOT_FOUND`, `HTTP_ERROR`, `RATE_LIMITED`, `PARSE_ERROR`,
`CHAPTER_FETCH_FAILED`, `TIMEOUT`, `CANCELLED`, `BROWSER_MODULE_NOT_INSTALLED`,
`CHALLENGE_UNRESOLVED`.

## Examples

Runnable scripts in [`examples/`](./examples):

- [`basic.ts`](./examples/basic.ts) — download with live progress logging.
- [`dump.ts`](./examples/dump.ts) — download and serialize to `.txt` files + `summary.json`.
- [`custom-transport.ts`](./examples/custom-transport.ts) — plug in a SaaS render service via a custom `HttpClient`.

```bash
pnpm example <url>          # basic.ts
pnpm dump <url> [outDir]    # dump.ts
```

## Documentation

Deep dives live in [`docs/`](./docs):

- [`architecture.md`](./docs/architecture.md) — layers and the big picture.
- [`data-model.md`](./docs/data-model.md) — core types and contracts.
- [`pipeline.md`](./docs/pipeline.md) — download flow, events, errors, resume.
- [`adapter-spec.md`](./docs/adapter-spec.md) — how to add a new site.

Adding a new site = implementing a single `SiteAdapter`; see `adapter-spec.md`.

## Development

```bash
pnpm install
pnpm test            # vitest
pnpm typecheck       # tsc --noEmit
pnpm build           # tsup → dist/ (ESM + CJS + d.ts)
pnpm smoke:live      # live end-to-end check across active adapters
```

## Publishing

Maintainers only. Bump `version`, then `pnpm release` (runs `prepublishOnly` to
build `dist/`, then `pnpm publish --access public`). Only `dist/` ships. Publish
from a clean `main`.

## License

MIT — see [`LICENSE`](./LICENSE).