# 82excel conformance suite

Language-neutral test cases for `validate(rows, rules)` (spec §5 G). The Node implementation
(`packages/core`) passes 100% of them. **A new language SDK (Java, PHP, Python, C#, Go, …) must pass
100% of these files, unchanged**, before it can claim the same results as the browser `check()`.

## Where it lives

In the repository: `spec/conformance/`. In the npm package: `node_modules/82excel/dist/conformance/`
(`npm i 82excel`), or browse it at https://cdn.jsdelivr.net/npm/82excel@0.2.0/dist/conformance/. The
layout below is the same in both places.

## Layout

```
spec/conformance/
  cases/<group>/<name>.json   one case per file
  tools/build-cases.mjs       writes cases/ from a hand-authored table (node spec/conformance/tools/build-cases.mjs)
```

(In the npm package the root is `dist/conformance/`. `tools/build-cases.mjs` only needs the folder it sits in,
but it is a maintainer tool: it deletes and rewrites `cases/`, so run it in the repository, not in `node_modules`.)

Groups: `types/` (every column type: valid, invalid, canonical re-input, Excel error values), `rules/` (column rules, `failOn`),
`rowrules/` (`when` / `compare` / `uniqueWith`), `blank/` (`rows.blank`), `injection/` (`formulaInjection`),
`pii/` (detection, forbid/allow, masking), `normalize/` (the `normalize` switches), `idempotency/`
(cleaned rows fed back), `edge/` (row numbers, boundaries, `rows.min`/`rows.max`), `invalid/` (rules that must be rejected).

The JSON files are generated; edit `tools/build-cases.mjs` and re-run it. Expected values there are written
by hand from the spec, the research notes and the controller rulings, never pasted from engine output. About
a third of the cases carry a `why` field naming the spec section, research line or ruling behind the
expectation.

## Case format

```json
{
  "name": "types/date-dots",
  "why": "research §3 Korean dates: 2024.01.05",
  "rules": { "columns": [{ "key": "v", "type": "date" }] },
  "rows": [{ "v": "2024.01.05" }],
  "expect": {
    "ok": true,
    "errors": [],
    "warnings": [],
    "rows": [{ "$row": 2, "v": "2024-01-05" }],
    "masked": [{ "row": 2, "key": "v", "raw": "900101-1******", "value": "900101-1******" }]
  }
}
```

| field | meaning |
|---|---|
| `name` | unique id, `<group>/<file name>` |
| `why` | optional; where the expectation comes from |
| `rules` | 0.2 rules JSON (spec §4), passed to `validate` as is |
| `rows` | input row objects keyed by column `key`; may carry `$row` |
| `expect.ok` | `true` when there are no errors |
| `expect.errors`, `expect.warnings` | lists of `{ row, key?, code }`; `key` is absent for row-level issues (`BLANK_ROW`, `ROWS_TOO_FEW`, `ROWS_TOO_MANY`) |
| `expect.rows` | the cleaned result rows, each with `$row` |
| `expect.masked` | optional (pii cases); `{ row, key, raw, value? }` — the reported (masked) `raw`, and `value` when given, of the issues at that cell |
| `expect.throws` | optional (`invalid/` cases); when present it is the only field of `expect`: `validate` must throw an error whose message starts with `82excel: ` and contains this text |

Inputs are plain JSON only: strings, numbers, booleans, `null`, or a missing key. There are no parsed cell
objects (`{ t, v, fmt }`); those exist only when reading files and are out of scope.

## Comparison rules

1. Call `validate(case.rows, case.rules)` with no options.
2. `errors`: compare the **set** of `(row, key, code)` of the result's errors with `expect.errors`. Order does
   not matter; a missing `key` compares as null. Same for `warnings`. Severity is implied by which list an
   issue is in.
3. `ok` must equal `expect.ok`.
4. `rows`: strict deep-equal, in order — no missing or extra keys. Numbers compare as numbers (`1000` =
   `1000.0`), strings exactly, `null` as null, booleans as booleans. Each row has exactly `$row` plus one
   entry per column `key`.
5. `masked` (only when present): for every entry, at least one issue exists at that `(row, key)` that carries
   a `raw` field, and every such issue's `raw` equals `entry.raw`. When the entry has `value`, at least one
   issue there carries `value`, and every such issue's `value` equals `entry.value`. Outside the pii cases,
   `raw`/`value` are not compared.
6. `throws` (only when present): `validate` throws; the message starts with `82excel: ` and contains the
   given text. Nothing else is compared. The given substrings **are part of the contract**: an SDK's error
   message must contain them verbatim.
7. Messages (`message`), `rule`, `col`, `header`, `summary` and other issue fields are **not** compared.
   Issues that have a `key` also carry `header` (the column's header text from the rules); it is not compared.

## How to run

Node (reference implementation, in the repository):

```
pnpm --filter 82excel exec vitest run test/conformance.test.ts
```

`packages/core/test/conformance.test.ts` loads every file under `cases/` and runs one test per case.

Other languages (from the repository or from `node_modules/82excel/dist/conformance/`): walk `cases/**/*.json`, apply the comparison rules above, and report each case by `name`.

## Behaviour this suite pins down

Most cases follow the spec text directly. Where the spec is silent, the suite fixes the behaviour below, and
an SDK must match it:

- **Result rows.** Every non-blank input row gives one result row: `$row` plus every column key, in input
  order. An empty cell is `null`. A cell that fails its type keeps its raw input as given (not trimmed). A
  cell that passes its type but fails enum / pattern / range / length / unique keeps its cleaned value. Input
  keys that are not columns are dropped. Blank rows are never result rows under any policy (Ruling V18).
  With `rows.blank: "stop"`, the first blank row and every row after it give no result row and no issue.
- **Row numbers.** `$row` when it is a positive integer (a JSON number with no fraction, ≥ 1), else
  `index + headerRow + 1` (`headerRow` defaults to 1). `0`, `-1`, `2.5` and the string `"5"` fall back.
- **Emptiness.** `null`, a missing key, or a string that is `''` after cleanup. The number `0` and `false` are
  not empty. A row whose every column is empty is blank. Blank rows after the last non-blank row are dropped
  without an issue whatever `rows.blank` says.
- **Cleanup.** Applied in this order to string input:
  1. `invisible`: remove U+200B, U+200C, U+200D, U+FEFF; replace U+00A0 (NBSP) with U+0020.
  2. `trim`: remove from both ends every code point of ECMAScript `String.prototype.trim`: U+0009–U+000D,
     U+0020, U+00A0, U+1680, U+2000–U+200A, U+2028, U+2029, U+202F, U+205F, U+3000, U+FEFF. U+200B is
     **not** whitespace.
  3. `fullwidthDigits`: map U+FF10–U+FF19 and `， ． － ＋ （ ）` (U+FF0C, U+FF0E, U+FF0D, U+FF0B, U+FF08,
     U+FF09) to ASCII. Never applied to `text` or `email` output (Ruling V9).

  **The `normalize` switches govern `text` columns only (Ruling V32).** Every other type (number, integer,
  date, boolean, email, kr.*) always applies all three steps when parsing and when deciding whether a cell is
  empty, whatever `normalize` says. No NFKC anywhere.
- **Check order per cell.** empty → type → enum → pattern → range → length → unique. A type failure stops the
  remaining checks for that cell. Pattern and length apply to string values, min/max to number values
  (Ruling V18). Values longer than `maxLength` skip the pattern. Enum and `when.eq/neq/in` use strict
  equality on the cleaned value (`1` ≠ `"1"`); a `when` condition never matches an empty cell except `empty`.
- **Issue keys.** Cell issues carry their column key. `ROW_REQUIRE` / `ROW_FORBID` carry the required or
  forbidden column, `ROW_COMPARE` the left column, `ROW_UNIQUE_WITH` the first column of the set.
  `CELL_UNIQUE` and `ROW_UNIQUE_WITH` are reported on every repeat after the first occurrence, not on the
  first. `compare` is skipped when either cell is empty or failed its type. `BLANK_ROW` has no key.
- **Row counts (Ruling V40).** `rows.min` / `rows.max` count the result rows (blank rows and rows after a
  `stop` are not result rows). A count outside them is an error `ROWS_TOO_FEW` / `ROWS_TOO_MANY` with no key,
  reported at row `headerRow` (default 1), alongside every other issue.
- **failOn (Ruling V40).** A warning whose code is listed in `failOn` is reported as an error instead, exactly
  as the browser `check()` does. Other codes keep their severity.
- **Excel error values (Ruling V41).** A string that equals `#NULL!`, `#DIV/0!`, `#VALUE!`, `#REF!`, `#NAME?`,
  `#NUM!` or `#N/A` after cleanup is `CELL_TYPE` in every column type (it was an error cell in the file).
  Text that only contains one (`see #REF!`) is ordinary text.
- **Severity.** Column rule and type failures take the column `severity` (default `error`), including
  `LEADING_ZERO_LOSS` from `kr.phone`/`kr.zip` (Ruling V10). `RRN_CHECKSUM` and `CRN_CHECKSUM` are always
  warnings; `PERSONAL_DATA` is a warning; `PII_FORBIDDEN` is an error; dangerous formulas are errors.
- **Formula injection** applies to string values of `text` and `email` columns that start with `= + - @`,
  tab or CR, unless the value reads as a number, a phone number or an amount (`-1000`, `+82 10-1234-5678`).
  A value starting with `'` is already escaped. `HYPERLINK(`, `DDE(`, `cmd|`, `IMPORTXML(`, `IMPORTDATA(`
  (any case) are errors under every mode. Under `escape` the result value gets a leading `'`, and unique /
  row rules compare the value without it.
- **Personal data** is looked for in `text` columns only, for the kinds in `pii.detect`, in the fixed order
  rrn, card, phone, email. One `PERSONAL_DATA` per column, at the first row. RRN detection checks format and
  birth date, not the checksum (Ruling V13). Candidates: RRN `\d{6}[ \t]?-?[ \t]?\d{7}`; card
  `\d(?:[ -]?\d){12,18}` passing Luhn; mobile phone `(?:\+82[ .-]?0?|0)1[016789][ .-]?\d{3,4}[ .-]?\d{4}`;
  email `[\w.%+-]{1,64}@[A-Za-z0-9.-]{1,253}\.[A-Za-z]{2,24}` (ASCII `\w`). A candidate touching another digit
  on either side is not a hit. Values longer than the column's `maxLength` give no `PERSONAL_DATA` /
  `PII_FORBIDDEN` (they are already `CELL_LENGTH`; their reported `raw` is still masked). Masking applies to issues of `kr.rrn`, `kr.phone`, `email` columns always and to detected
  values elsewhere, never to result rows.
- **Linear time.** Detection and type parsing must run in time linear in the input: servers run `validate`
  on untrusted rows, and a cell may hold 32,767 characters. Use bounded quantifiers or hand-written scanners;
  a regex with nested or adjacent unbounded repeats (`\s*-?\s*`, `[\w.]+@…`) backtracks quadratically on
  inputs like six digits followed by 32k spaces. The Node package has perf tests for these inputs
  (`packages/core/test/perf.test.ts`); an SDK should have the same.
- **Patterns** are ECMAScript regular expressions compiled with the `u` flag and tested with
  `RegExp.prototype.test` semantics: the pattern may match anywhere in the value unless it is anchored
  (Python: `re.search`, not `re.match`/`fullmatch`). An SDK must translate the pattern or use a compatible
  engine. Cases that use Unicode property escapes (`\p{…}`): `rules/pattern-unicode-flag`,
  `rules/pattern-unicode-flag-fail`. `rules/pattern-is-a-search` checks the unanchored semantics.
- **Phone shapes** (`kr.phone`): mobile `010` + 4 + 4 digits (a 3-digit middle group after 010 is invalid,
  Ruling V33), `011/016/017/018/019` + 3–4 + 4; landline `02`, `031–033`, `041–044`, `051–055`, `061–064`
  + 3–4 + 4, `070` + 4 + 4, `0502–0508` + 3–4 + 4; representative `15xx/16xx/18xx` + 4 (kind `any` only).
- **Dates.** Years 0001–9999 (Ruling V33), proleptic Gregorian leap years, for both the canonical form and
  parsed formats.
- **JSON numbers as text.** Where a number must be read as text — a `text` column, boolean map lookup,
  date digits (`20240105`) — it is the shortest round-trip decimal of ECMAScript `Number.prototype.toString`
  (`1` → `"1"`, `1.5` → `"1.5"`). The digit-based kr.* types accept only non-negative safe integers as
  numbers.
- **Mask forms.** Each detected hit in a text value is replaced by its masked form and the rest of the text
  is kept: RRN → `YYMMDD-G******` (first 6 digits, `-`, 7th digit, six `*`); card → its digits with all but
  the first 4 and last 4 starred, grouped by 4 with `-` (`4111-****-****-1111`); phone → the normalized
  hyphen form with the middle group starred (`010-****-5678`); email → first character + `***` + `@domain`.
  Values of `kr.rrn`, `kr.phone`, `email` columns are masked the same way (an unparseable RRN keeps its first
  7 digits and stars the rest), and then scanned like free text, so an RRN typed into a phone or email
  column is masked as an RRN (Ruling V43). `raw` and `value` are both masked.
- **Lengths** count UTF-16 code units, which equals characters for Hangul and other BMP text. No case uses
  characters outside the BMP.

## Out of scope

These need the file and are tested in the Node package, not here (spec §5 P3):

- file-level checks: password, encoding, hidden rows/columns, merged cells, macros, sheet selection,
  header mapping (`aliases`, `required`, `columnsOrder`, `extraColumns`), totals rows;
- H warnings for stored cells (`PRECISION_LOSS`, `SCIENTIFIC_NOTATION`, a number cell in a text column);
- the card "suspect" warning for numeric cells of 16+ digits (needs the stored cell; JSON numbers that long
  lose precision anyway);
- real formula cells in text columns (a string starting with `=` is covered; a stored formula is not);
- Excel date serials and the 1904 date system: they depend on the cell's number format, which JSON rows do
  not carry. `check()` converts date cells to ISO before calling `validate`, so a server never sees a serial
  (Ruling V19). A bare JSON number in a date column is read as digits (`20240105` → `2024-01-05`).
- parsed cell objects as input;
- message text and locales (`messages`, `locale`, column `message`), `summary`, `col` letters;
- JS row hooks (`rowRule(fn)`), which each server implements itself.