# Heuristics for detecting mojibake

## The badness heuristic

`badness(text)` counts unlikely Unicode character sequences associated with
mojibake. `is_bad(text)` stops after the first match and answers whether a
string looks broken.

```ts
import { badness, is_bad } from "ftfy-js/badness";
```

The character categories and adjacency rules are direct ports of upstream's
hand-designed `BADNESS_RE`.

## The UTF-8 detector

`UTF8_DETECTOR_RE` in `ftfy-js/chardata` recognizes mojibake sequences produced
by supported Latin-1-like encodings. `decode_inconsistent_utf8` uses it to fix
embedded sequences even when the entire string cannot be re-decoded.

The expression avoids starting after a likely UTF-8 continuation character.
This prevents it from decoding a small fragment inside a much larger garbled
sequence and making the result more confusing.

## Lossy UTF-8

`LOSSY_UTF8_RE` recognizes UTF-8-like byte sequences in which continuation
bytes became `?` or U+FFFD through a prior lossy decode. The
`replace_lossy_sequences` fixer turns each matched sequence into the UTF-8 bytes
for U+FFFD so the surrounding repair can continue.