# MQTT compatibility tests

Measures Opifex's **actual** MQTT 5.0 and 3.1.1 protocol compatibility by
running the vendor-neutral [Eclipse Paho interoperability suite][paho] against a
broker booted from this repo, and reports an honest pass percentage. Wired into
CI by
[`.github/workflows/mqtt-compat.yml`](../../.github/workflows/mqtt-compat.yml),
which posts the result as a sticky PR comment and a job summary.

## Files

| File            | Purpose                                                                                                            |
| --------------- | ------------------------------------------------------------------------------------------------------------------ |
| `broker.js`     | Boots Opifex on a plain-TCP port (`MQTT_PORT`, default 1883) with the v5 capabilities the suite exercises enabled. |
| `run_compat.py` | Runs each Paho test in an isolated subprocess against the broker, emits `report.md` + `result.json`. Pure stdlib.  |

## Running locally

```bash
# 1. Fetch the Paho suite, pinned to the same commit CI uses so local results
#    match. The pin (PAHO_REF) lives in .github/workflows/mqtt-compat.yml.
git clone https://github.com/eclipse-paho/paho.mqtt.testing /tmp/paho
git -C /tmp/paho checkout "$(grep -oP 'PAHO_REF:\s*\K\S+' ../../.github/workflows/mqtt-compat.yml)"

# 2. Start the broker
node ./broker.ts &

# 3. Run the suites
python3 ./run_compat.py --paho /tmp/paho/interoperability --host localhost --port 1883 --out report.md --json result.json
```

`--protocols v5,v3` selects the suites; `--timeout` bounds each test (default
45s).

### Individual test

Version 3.11:

```bash
python3 /tmp/paho/interoperability/client_test.py Test.test_retained_messages
```

Version 5:

```bash
python3 /tmp/paho/interoperability/client_test5.py Test.test_retained_messages
```

## How the score is computed

Each Paho `Test` method runs in its **own subprocess** with a hard timeout. This
matters because the Paho clients do blocking waits with no internal timeout: a
case where the broker never sends an expected packet would otherwise hang the
whole run, and a hung test would corrupt the class-level shared clients later
tests reuse. A timed-out test is recorded as a bounded failure.

The percentage is the raw pass rate (`passed / evaluated`). Two categories are
handled specially:

- **Expected gaps** (`EXPECTED_GAPS` in `run_compat.py`) — features Opifex
  intentionally does not implement yet (e.g. shared subscriptions, which the
  broker advertises as `sharedSubscriptionAvailable=false`). These still count
  as failures; they are only annotated with the reason so the table is readable.
  Because v5 is a work in progress, the harness also detects the reverse: when
  an `EXPECTED_GAPS` test starts **passing** (the feature got implemented), the
  report highlights it with a 🎉 banner and the workflow emits a CI warning,
  prompting you to remove the now-stale entry from `EXPECTED_GAPS`.
- **Harness-limited** (`HARNESS_LIMITED`) — tests that cannot be evaluated under
  per-test isolation against _any_ broker (currently only `test_flow_control2`,
  which depends on the persistent client the suite sets up only in
  single-process mode). These are **excluded** from the denominator. The
  capability they probe is still measured elsewhere (`test_flow_control1` for
  receive-maximum).

### Adding or moving a test category

Both lists in `run_compat.py` are keyed
`protocol -> exact Paho method name ->
reason string`. Pick the dict by what the
failure _means_:

- A genuine Opifex gap that should drag the score down → **`EXPECTED_GAPS`**. It
  still runs and counts as a failure; when Opifex later implements it the
  harness flags the 🎉 xpass so you remove the entry.
- A test the harness can't fairly evaluate against _any_ broker →
  **`HARNESS_LIMITED`**. It is skipped before the gap logic and excluded from
  the denominator. A method must live in **exactly one** list (`HARNESS_LIMITED`
  is checked first, so a duplicate `EXPECTED_GAPS` entry would be dead code).

## Coverage scope — read the percentage correctly

The percentage is **"% of the Paho functional suite that passes"**, not "%
MQTT-5.0-compliant". The Paho suite is a third-party **happy-path functional**
test: it exercises the user-facing v5 features well (session expiry, will + will
delay, message expiry, RH/RAP/NL, request/response, payload format, assigned
client id, server keep alive, inbound topic aliases, max-packet-size rejection,
PUBLISH user properties), but it touches only 5 reason codes and deliberately
omits large parts of the spec:

- **Error / negative paths** — most of Opifex's reason codes
  (`lib/constants.js`): `0x87` not-authorized acks, `0x8E` session takeover,
  `0x82` protocol errors, `0x8C` bad auth method, `0x9E`, the CONNACK reject
  codes, broker-initiated DISCONNECT with ReasonString, the oversized-frame read
  latch, wildcard ResponseTopic rejection. These are covered by
  **`test/mqtt5.js`**, which is the authoritative place Opifex's v5 error
  semantics are pinned.
- **Whole spec chapters** — AUTH / enhanced auth (§4.12), Server Reference
  (`0x9C`/`0x9D`), CONNACK capability flags, ReasonString round-trips, and
  UserProperty on non-PUBLISH packets. Some of these Opifex does not implement;
  the test gap left by _both_ suites is tracked in
  [#1096](https://github.com/moscajs/Opifex/issues/1096).

So treat this workflow as the **cross-implementation functional gauge** and
`test/mqtt5.js` as the **gap-filler** — they are complementary, not redundant.

Some Paho failures are intentional Opifex design decisions rather than bugs (see
`EXPECTED_GAPS`), and at least one is a genuine spec bug found by this suite:
[#1095](https://github.com/moscajs/Opifex/issues/1095) (zero-length ClientId +
CleanSession=0 must be rejected on v3.1.1).

## Harness validation

The harness was validated against the Paho project's own fully-compliant
reference broker (`startbroker.py`): it scores **100%** of evaluated v5 tests
(with `test_flow_control2` excluded as above). Every test Opifex fails, the
reference broker passes under the identical harness — so a failure here reflects
a real difference between Opifex and a spec-compliant broker, not a harness
artifact.

[paho]: https://github.com/eclipse-paho/paho.mqtt.testing