# ftfy: fixes text for you — TypeScript port

> [!IMPORTANT]
> This repository is intentionally a TypeScript port and derived work of
> [Robyn Speer's `rspeer/python-ftfy`](https://github.com/rspeer/python-ftfy).
> Its algorithms, behavior, tests, examples, and documentation are based on
> upstream commit
> [`74dd0452b48286a3770013b3a02755313bd5575e`](https://github.com/rspeer/python-ftfy/commit/74dd0452b48286a3770013b3a02755313bd5575e).
> This project does not claim independent authorship of ftfy. See
> [Attribution and license](#attribution-and-license) and [UPSTREAM.md](UPSTREAM.md).

```ts
import { fix_encoding } from "ftfy-js";

console.log(fix_encoding("(ง'⌣')ง"));
// (ง'⌣')ง
```

`ftfy` fixes Unicode text that is broken in various ways. Its main goal is to
take in bad Unicode and output good Unicode, particularly by repairing
*mojibake*: text that was decoded using the wrong character encoding.

This package mirrors python-ftfy 6.3.1's Python-style API names so behavior and
examples map directly between the two projects. It ships ESM, CommonJS,
TypeScript declarations, browser-compatible core functions, and a Node.js CLI.
It is published to npm as `ftfy-js`.

## Install

```sh
npm install ftfy-js
```

For local development:

```sh
npm install
npm test
```

Node.js 20 or newer is supported.

## What it does

ftfy detects character patterns that were probably intended to be UTF-8 but
were decoded as another encoding:

```ts
import { fix_text } from "ftfy-js";

fix_text("✔ No problems");
// "✔ No problems"
```

It can repair multiple layers of mojibake:

```ts
fix_text("The Mona Lisa doesn’t have eyebrows.");
// "The Mona Lisa doesn't have eyebrows."
```

It can fix mojibake after curly quotes have been applied:

```ts
fix_text("l’humanité");
// "l'humanité"
```

It can recover a non-breaking-space byte that was changed into an ASCII space:

```ts
fix_text("à perturber la réflexion");
// "à perturber la réflexion"

fix_text("à perturber la réflexion");
// "à perturber la réflexion"
```

It can decode HTML entities outside HTML, including carefully selected
incorrectly-capitalized forms:

```ts
fix_text("PÉREZ");
// "PÉREZ"
```

The fixes are deliberately conservative. Correctly decoded text that merely
*could* be interpreted as mojibake is left alone:

```ts
fix_text("IL Y MARQUÉ…");
// "IL Y MARQUÉ…"
```

## API

The common functions are:

```ts
import {
  TextFixerConfig,
  apply_plan,
  fix_and_explain,
  fix_encoding,
  fix_encoding_and_explain,
  fix_text,
  fix_text_segment,
  guess_bytes,
} from "ftfy-js";
```

- `fix_text(text, config?)` applies all enabled fixes, independently by line.
- `fix_text_segment(text, config?)` applies a single consistent sequence of
  fixes to the whole input.
- `fix_encoding(text, config?)` repairs encoding mix-ups only.
- `fix_and_explain(text, config?)` and
  `fix_encoding_and_explain(text, config?)` return an array that can be
  destructured as `[text, explanation]` and also has `.text` and
  `.explanation` properties.
- `apply_plan(text, explanation)` replays an explanation.
- `guess_bytes(bytes)` implements upstream's deliberately limited byte guesser.
- `TextFixerConfig` exposes the same snake_case settings as python-ftfy.

```ts
const [fixed, explanation] = fix_and_explain("só");

fixed;
// "só"

explanation;
// [["encode", "latin-1"], ["decode", "utf-8"]]

apply_plan("só", explanation ?? []);
// "só"
```

Configuration can be passed as a plain object or as a `TextFixerConfig`:

```ts
fix_text("“text”", { uncurl_quotes: false });

const config = new TextFixerConfig({
  unescape_html: false,
  normalization: "NFKC",
});
fix_text("text", config);
```

Individual fixers are exported from `ftfy-js/fixes`; heuristic functions from
`ftfy-js/badness`; encoding data from `ftfy-js/chardata`; terminal-width helpers
from `ftfy-js/formatting`; and codec helpers from `ftfy-js/bad-codecs`.

See the [full documentation](docs/index.md).

## Command line

The package provides the same `ftfy` command-line shape as upstream:

```sh
ftfy input.txt
ftfy -e latin-1 input.txt -o output.txt
ftfy -g input.txt
cat input.txt | ftfy
```

Run `ftfy --help` for all options. The CLI reads and writes UTF-8 by default.

## Compatibility and verification

The port is pinned to the upstream revision documented in [UPSTREAM.md](UPSTREAM.md).
Its test suite directly ports all upstream unit tests and all five upstream JSON
corpora. Known upstream failures remain explicit tests and are not presented as
successes. The codec, HTML entity, Unicode-name, Unicode-category, and display
width tables are generated from the pinned upstream Python environment instead
of being approximated with unrelated JavaScript packages.

Run the complete release check with:

```sh
npm run check
```

This runs strict TypeScript checking, all tests, both ESM and CommonJS builds,
the CLI build, and an npm package dry run. npm publishing is intentionally not
part of any script or workflow.

## Who created ftfy?

ftfy was designed and created by **Robyn Speer**, also known as Elia Robyn
Lake. The canonical project is
[`rspeer/python-ftfy`](https://github.com/rspeer/python-ftfy). Robyn's projects
and writing are available [on GitHub](https://github.com/rspeer) and at
[posts.arborelia.net](https://posts.arborelia.net).

This TypeScript repository is a language port of that work.

## Citing ftfy

ftfy has been used as a crucial data-processing step in major NLP research.
When using this port in research, cite the original work:

```text
Robyn Speer. (2019). ftfy (Version 5.5). Zenodo.
https://doi.org/10.5281/zenodo.2591652
```

```bibtex
@misc{speer-2019-ftfy,
  author       = {Robyn Speer},
  title        = {ftfy},
  note         = {Version 5.5},
  year         = 2019,
  howpublished = {Zenodo},
  doi          = {10.5281/zenodo.2591652},
  url          = {https://doi.org/10.5281/zenodo.2591652}
}
```

## Attribution and license

This repository follows the **Important license clarifications** in
python-ftfy's README.

The Apache License 2.0 is what grants permission to use, copy, modify, and
distribute ftfy. Following that license is required. In particular, derived
works must retain the required notices and must correctly attribute ftfy's
author, **Robyn Speer**. A derived work must not obscure ftfy's authorship.

This repository therefore:

- identifies itself prominently as a TypeScript port and derived work;
- names Robyn Speer as the original author;
- links to the canonical python-ftfy repository;
- records the exact upstream commit;
- reproduces the upstream Apache-2.0 license in [LICENSE.txt](LICENSE.txt); and
- includes attribution in [NOTICE](NOTICE), generated files, tests, and docs.

See the upstream
[Important license clarifications](https://github.com/rspeer/python-ftfy#important-license-clarifications)
for the author's complete statement. If this project is redistributed or
further modified, do not remove or obscure this attribution.