# Cross-building native extensions

The pipeline that turns a package's own source into a SandboxedJs wheel, and
the list of things it still assumes about a package — so each of them can be
removed deliberately rather than discovered by a failure.

## What it does

`python-runtime/scripts/build_extension.py` takes a recipe and produces a
wheel tagged `cp313-cp313-emscripten_5_0_6_wasm32` with a `Build-ABI` line
naming the `abiId` computed from `abi/extension-abi.json`. Three recipe kinds
exist:

| kind | source of truth for *what* to build | covers |
| --- | --- | --- |
| `setuptools` | the package's own `setup.py` / `pyproject.toml` | plain C extensions, Cython, anything setuptools builds |
| `pep517` | the package's declared build backend and hooks | pyproject-only projects, in-tree backends |
| `meson` | the project's `meson.build`, via meson-python | Meson projects, including NumPy |
| `pyo3` | the crate's `Cargo.toml` | Rust extensions |
| `c-extension` | the recipe's own `sources` list | probes only — see the assumptions below |

`kind` selects a backend explicitly; nothing is inferred from which files are
present. A project with both a `setup.py` and a `pyproject.toml` can be built
either way, and guessing would silently change how a package is built the day
upstream adds a file. Recipes are schema-validated before anything is fetched
or compiled, so a malformed one costs a second rather than a Rust build.

In every case the *how* — compiler, target, PIC, threads, exceptions,
extension suffix, wheel tag — comes from `abi/extension-abi.json` and only
from there. A recipe cannot contribute a compile or link flag. This is not
tidiness: a recipe that could set its own `-pthread` could produce an artifact
that links against a main module with a different memory model and then
corrupts memory, and the mismatch would be invisible in the wheel.

## How the setuptools path works

`sysconfig` loads its table of build variables from a module named by the
`_PYTHON_SYSCONFIGDATA_NAME` environment variable, and
`distutils.command.build_ext` — which setuptools vendors — takes the compiler,
its flags and the extension suffix from that table. So the package's own
`setup.py build` runs on the *build machine's* interpreter with `sysconfig`
answering for the *target*. This is the mechanism CPython uses for its own
cross builds and the one `crossenv` automates; nothing about it is specific to
this project.

The CPython cross build already writes the target's table
(`_sysconfigdata__emscripten_wasm32-emscripten.py`). `scripts/crossenv.py`
copies it and overrides exactly three groups of values, all from the ABI
contract:

* `INCLUDEPY` / `CONFINCLUDEPY` — the shipped table names
  `$prefix/include/python3.13`, where the headers would be if the interpreter
  had been installed. It has not been; they are in the source tree.
* `LDSHARED` / `BLDSHARED` / `LDCXXSHARED` — the shipped link line has no
  `-sSIDE_MODULE=1`, because CPython adds it in a rule outside `LDSHARED`.
  Without it the artifact is a complete program and `dlopen` cannot load it.
* `CCSHARED` — the ABI's `-pthread -fwasm-exceptions -fPIC`, which must match
  the main module exactly. `CCSHARED` rather than `CFLAGS` so the flags apply
  to extension objects and not to host tools a package builds during setup.

Cython needs no support at all: `cythonize()` runs inside the package's
`setup.py` on the build machine and hands generated C to the same `build_ext`.
The only difference between the plain and Cython fixtures is a pinned
`buildRequires`.

## Validation

Three checks, in order of how misleading their failure would otherwise be:

1. `verify_toolchain` refuses an Emscripten other than the pinned one. A side
   module built by a different Emscripten links and then fails at run time.
2. `verify_side_module` checks the artifact is a WebAssembly binary, has a
   `dylink` section, and exports `PyInit_<last component of the module name>`.
   A missing dylink section fails at `dlopen` with a message about the file; a
   missing init symbol fails with "dynamic module does not define module
   export function", which reads like a source error and is in fact a link
   setting.
3. `test/python-runtime/extensions.test.ts` installs each wheel from an HTTP
   index with the container's own `pip` and imports it with CPython's ordinary
   import machinery. This is the only check that proves the ABI, because the
   first two can pass on a module compiled against the wrong headers.

## Package-specific assumptions, and how to remove each

Each of these is a place the pipeline knows, or requires a recipe to state,
something a package already states about itself.

**1. Target dependencies are not resolved.** *Resolved.* Native target
dependencies are declared by `dependency.json` manifests under
`python-runtime/native-deps/` (real libraries) and
`python-runtime/fixtures/native/` (test ones), built by
`scripts/native_deps.py` into the profile's sysroot with the ABI's own flags,
and reached by extensions through the generated cross configuration — so
`libraries=["yaml"]` in a package's `setup.py` resolves with no path or flag
in the recipe. A recipe names what it needs in `nativeRequires`, which is a
separate namespace from `requires` (Python) and `buildRequires` (build
machine) so that a build-machine library can never satisfy a target link.

Reuse is decided by a stamp recording the source digest, profile, exact
compiler flags and the digest of every installed output — not by whether a
file exists, which is what let an archive built for one profile satisfy
another. What remains: only `make` and `configure` build systems are
supported, and CMake and Meson libraries will need a third; and the libraries
CPython itself links are still built by the older hand-written
`scripts/build_dependencies.py` rather than through manifests.

**2. Wheel metadata is synthesized from the recipe on the `pyo3` path only.**
*Resolved for `setuptools`.* That path now runs the package's own `dist_info`
under the cross environment and ships the `METADATA` it generates, so markers,
extras and `Requires-Python` are upstream's and a recipe has no `requires`
field to drift from them. Each wheel records which of the two it used in its
`WHEEL` file as `Metadata-Source: package|recipe`, so the remaining cases are
visible in the artifacts rather than only here.

A recipe may still *remove* a requirement through `dependencyOverrides`, which
must carry a `because`, must name a requirement the package actually has, and
is recorded in the built wheel as `Build-Dropped-Requirement`. *Remove the
remaining `recipe` cases by:* reading `[project]` from the crate's
`pyproject.toml` for maturin-built extensions, or by driving maturin itself
under the cross environment.

**3. `pythonRoots` exists only because the `pyo3` path has no `build_py`.** It
is a hand-written statement of which directory holds the package's Python
half; the pydantic-core recipe encodes maturin's `python/` convention. *Remove
by:* reading `[tool.maturin] python-source` from the crate's `pyproject.toml`,
or by driving maturin itself under the cross environment.

**4. The `pyo3` path declares module layout.** `crateName` and `package` state
where the artifact must sit for `from ._x import ...` to work. The setuptools
path derives the same thing from the build tree. *Remove by:* same as 3.

**5. The `pyo3` recipe hard-codes an unpacked source path.**
`recipes/pydantic-core/recipe.json` names
`../../out/ports/pydantic_core-2.46.5`, duplicating the version that appears
three other times in the same file. *Remove by:* deriving the crate path from
`source.unpackTo` and the archive's own stem, which `fetch_source` already
computes.

**6. `kind: "c-extension"` is a transcription.** The recipe lists sources the
package's build system already lists, so the two can disagree silently. It is
retained only for `fixtures/sbx_c_probe`, which deliberately has no build
system. *Remove by:* giving the fixture a `setup.py` and deleting the kind and
`compile_c_module` with it.

**7. `setup.py` is assumed to be cross-safe.** *Mitigated, not removable.* A
`setup.py` that probes the build machine (compiles a test program, runs the
extension it is building, reads `platform.machine()`) will describe the build
machine. Nothing detects this; the build succeeds and produces a wrong
artifact. It is a property of the package, so it cannot be fixed generically —
but it now has one designated place to be fixed in. A recipe declares
`patches: {"dir": …}`, and the patch set states the package, the version, the
digest of the source tree it was written against, and per patch a reason, the
target fact being substituted, and the test that proves it. Patches apply to a
staged copy with `git apply` at exact context; a drifted source is refused
rather than fuzzily patched. `fixtures/sbx_patched_probe` is exactly this
failure — `platform.machine()` declaring 64-bit pointers for a 32-bit target —
and its patch is the worked example.

**8. Build tools come from PyPI at first build.** *Resolved.*
`build-tools.lock` pins every build tool — including `setuptools` itself, which
was previously taken from whatever the build machine had — and records the
sha256 of every distribution published for that version, so one lock serves any
build machine. Installation uses `--require-hashes`, which also refuses a tool
that grows an unpinned dependency. Regenerate with
`scripts/lock_build_tools.py`; a build reads the lock and never refreshes it.

**9. One ABI, no matrix.** `abiId` covers exactly one configuration; there is
no way to build the same recipe for a second profile. *Remove by:* taking the
profile from the command line into the wheel's local version segment, once a
second profile exists to want it.

**10. `--build-lib` is assumed to be honoured.** Projects with a custom
`build` command that ignores it would stage nowhere the collector looks; the
build "succeeds" and produces no modules. This is caught — the empty-module
case raises — but only after the fact.

## Native target dependencies

Three kinds of dependency exist and are deliberately kept in three namespaces,
because a single list would let a build machine's library satisfy a target
link — producing a module that links cleanly and then faults:

| recipe field | what it is | where it goes |
| --- | --- | --- |
| `requires` | Python runtime dependency | installed in the container, imported |
| `buildRequires` | build-machine tool (Cython, setuptools_scm) | a per-recipe venv on this machine |
| `nativeRequires` | C library cross compiled for wasm32 | `out/sysroot-<profile>`, linked into the side module |

A native dependency is declared by a `dependency.json`:

```json
{
  "name": "yaml", "version": "0.2.5",
  "license": "MIT", "licenseFiles": ["License"],
  "provenance": { "url": "…", "sha256": "…", "unpackDir": "yaml-0.2.5" },
  "profiles": ["dynamic"], "dependsOn": [],
  "build": { "system": "configure", "configureArgs": ["--disable-shared"] },
  "outputs": { "libraries": ["libyaml.a"], "headers": ["yaml.h"] }
}
```

`name` must be the name a linker is given (`-lyaml`), not the project's title:
a manifest called `libyaml` would build the right library and leave the
extension unable to ask for it.

`outputs` is checked after the build and recorded in the stamp. A build system
that quietly produces nothing — a `configure` that disabled the library, a
`make` that built only tools — otherwise looks like a success until an
extension fails to link with a message about a missing symbol.

Provenance is either `local` (source in this repository, still digested so an
edit rebuilds) or a pinned `url` and `sha256`, verified before anything is
unpacked. These are kept out of `sources.lock` on purpose: that file pins what
the *interpreter* links, and a library that exists for packages should not
become something the runtime build has to download.

`build.host` exists because autotools packages vendor a `config.sub` frozen at
their release date. libyaml 0.2.5's is from 2018 and rejects
`wasm32-unknown-emscripten` outright; it accepts `wasm32-unknown-none`, which
is sufficient because `--host` only tells `configure` that this is a cross
build — `emconfigure` has already supplied the tools. That is a property of
the package, so it is declared per dependency rather than worked around for
everybody.

## Reproducible inputs, patches and provenance

Builds never run in the source tree. The project is copied to a staging
directory first, then patched, then built. Two failures follow from doing
otherwise: a patch applied in place leaves the checkout modified, so a second
build starts from different source and the patch no longer applies; and a build
leaves generated files behind — Cython's `.c`, `.egg-info` — which then become
inputs to the next one.

Every input is pinned and verified before use:

| input | pinned by | verified by |
| --- | --- | --- |
| interpreter and its libraries | `sources.lock` | sha256 before unpacking |
| build tools (setuptools, Cython) | `build-tools.lock` | `pip --require-hashes` |
| native target libraries | `dependency.json` provenance | sha256 before unpacking |
| package source | recipe `source`, or in-repo | sha256, or tree digest |
| patches | `patches.json` `sourceDigest` | exact-context `git apply` |

Wheels are byte-reproducible: every zip member is stamped with the zip epoch
rather than the build time and RECORD is sorted, so two builds from the same
inputs produce identical files. This is stronger than the "documented timestamp
normalization" the task allowed, and it is what makes "did anything actually
change?" answerable by comparing digests. Confirmed by rebuilding every wheel
and diffing digests, and by `test/python-runtime/reproducible-builds.test.ts`.

Each wheel carries `dist-info/sandboxedjs-provenance.json`, which answers, for
an artifact held by someone without this repository: which source tree digest
produced it, which build tools and native libraries went into it, which ABI id
and recipe revision were used, and which patches were applied. The index
summarises those fields so they can be compared across wheels without
downloading any.

## Build backends

`scripts/backends.py` is the only place a package's build system is known
about. The orchestrator verifies inputs, selects a backend by the name the
recipe stated, hands it a `BuildRequest`, and packages the `BuildResult` that
comes back. It decides nothing else — which is what keeps a package-name check
from appearing in it, since once an orchestrator is already making build
decisions, one more looks harmless.

A backend receives verified staged source, the locked build-tool interpreter,
the cross environment, the declared native dependencies, a staging directory
and a read-only view of the ABI. It returns a wheel-layout tree, the package's
own `.dist-info`, and evidence recorded in the wheel's provenance. It cannot
contribute a compiler flag, and the orchestrator digests the ABI contract
before and after the build — a backend that mutated it would produce a
correctly-tagged wheel built for a different ABI.

The `pep517` backend calls the package's declared hooks the way a frontend
does, including resolving an in-tree backend through `backend-path`. It does
**not** use `python -m build`, whose isolation installs whatever versions
upstream publishes on the day of the build — the unpinned input the rest of
this pipeline refuses. The environment is assembled from `build-tools.lock`,
and a build requirement absent from the lock is refused rather than fetched.
The wheel the hooks produce is unpacked rather than shipped: it carries
whatever tag the backend chose, and this pipeline tags a wheel for what it
actually is.

Build tools run on an interpreter whose feature version matches the target's.
Most of the cross build does not care, because `sysconfig` answers for the
target either way — but `bdist_wheel` composes a tag from the *running*
interpreter's version and the *target's* ABI tag. Building for 3.13 from 3.14
yields `('cp314', 'cp313', 'emscripten_5_0_6_wasm32')`, which fails an
assertion inside setuptools rather than anywhere that names the cause.

## Meson

`kind: "meson"` drives a project through meson-python's PEP 517 hooks. It is
its own backend rather than a `pep517` recipe with extra settings because a
cross file is backend mechanics, not a package quirk — a recipe that passed its
own `--cross-file` could describe a different target than the wheel is tagged
for.

Meson does not read `sysconfig`; it is told about a target by a file. So the
same facts the cross table states are stated again in Meson's format, generated
from `abi/extension-abi.json` by `write_meson_cross_file`. Three settings are
supplied by the backend because they are facts about the target, not choices:

- `needs_exe_wrapper = true`, or Meson believes it can run what it builds and
  every compile-and-run check silently tests the build machine.
- `--wrap-mode=nodownload`. Meson otherwise resolves a missing subproject by
  cloning it mid-build — an unpinned input arriving over the network at the one
  moment nothing is watching.
- `-Ddefault_library=static`. There is no such thing as a shared library here;
  `ld.wasm` refuses, and Meson reports it at configure time as a message about
  the linker rather than about the default that reached it.
- `longdouble_format`, derived by asking the pinned compiler for its
  `__LDBL_MANT_DIG__` and byte order rather than being written down. Meson
  cannot run a program on the target to find out, and a project that inspects
  float layouts — NumPy does — will not configure without it. Deriving it
  matters more than the convenience: a hand-written value that disagreed with
  the toolchain would produce a library that builds, imports, and computes
  wrong answers.

A project whose subprojects were previously downloaded supplies them through
`vendoredSources`: hash-verified archives unpacked at declared paths inside the
staged tree, recorded in the wheel's provenance because they are compiled into
it. They cannot escape the staged tree.

### Two things that were failures first

**Split headers.** A cross build leaves `Include/` in the source tree and the
generated `pyconfig.h` in the build directory; an *installed* interpreter has
them in one place, and tools assume the installed shape. `Python.h` includes
`"pyconfig.h"` in quotes, so the compiler looks beside `Python.h` and then walks
the `-I` list — and finds the *build machine's* `pyconfig.h` from whichever host
include directory a tool added. The build then fails with `LONG_BIT definition
appears wrong for platform`, which reads like a broken toolchain and is in fact
a 64-bit header describing a 32-bit target. Meson adds such a directory;
setuptools happens not to. Both now compile against one staged directory
holding every target header.

**Paths in compiled output.** This one recurred three times before the rule was
clear: *nothing a compiler sees may live at a per-build path.*

First the staged include directory, created inside the temporary build
directory — the compile line carries `-g`, so the varying path reached debug
information. Then Meson's own `.mesonpy-<random>` build directory, which also
has to be pinned together with the cross file, because Meson caches the cross
file's path inside the build directory and a stale one fails on the next run
with a `FileNotFoundError` about a file nobody asked for.

Then the staged *source* directory, which was the subtlest: Cython writes the
`.pyx` path into its generated C so a traceback can name a line. NumPy exposed
it — 869 of the 874 files in its wheel matched between builds, and the five
that differed were its Cython-generated modules, all at identical sizes. The
staged source is now a stable path too, and `reproducible-builds.test.ts`
checks that the generated cross configuration carries no temporary path.