# Native extensions for the owned CPython runtime

This describes how a compiled CPython extension — a C module, or a Rust one
built with PyO3 — is built, tagged, installed and loaded by the SandboxedJs
Python distribution.

It is a general pipeline. Nothing in the runtime, the installer or the import
machinery knows about any particular package. Adding support for a package
means adding a *recipe*; it never means adding a runtime adapter.

## The contract

`python-runtime/abi/extension-abi.json` is the single source of truth. The
wheel tag, the compiler flags side modules are built with, the identifier the
wheel cache is keyed on and the checks the builder makes are all generated
from it by `scripts/generate_extension_abi.py` into:

- `python-runtime/abi/sbx_ext_abi.h` — for C extensions
- `src/runtime/python/extension-abi.ts` — for the installer and loader

Regenerate with `make -C python-runtime extension-abi` and commit the output.

### What the contract fixes

| Field | Value |
| --- | --- |
| Implementation | CPython 3.13.5 |
| ABI tag | `cp313` |
| SOABI | `cpython-313-wasm32-emscripten` |
| Extension suffix | `.cpython-313-wasm32-emscripten.so` |
| Target triple | `wasm32-unknown-emscripten` |
| Emscripten | 5.0.6 |
| Threads | pthreads, shared memory |
| Memory | 256 MiB initial, growth enabled |
| Stack | 8 MiB |
| Exceptions | WebAssembly EH (`-fwasm-exceptions`), longjmp lowered to wasm |
| Dynamic linking | CPython is `MAIN_MODULE=1`; extensions are `SIDE_MODULE=1` |
| Wheel tag | `cp313-cp313-emscripten_5_0_6_wasm32` |

The wheel tag is the PEP 425 platform tag CPython itself reports on
Emscripten, so `pip` and `packaging` accept it without a project-specific
patch. **Linux and macOS wheels are never accepted.** A `manylinux` wheel
contains ELF objects this interpreter cannot load; installing one would turn a
clear resolution failure into an `ImportError` inside the user's program.

### Why there is an ABI id as well as a tag

`abiId` (currently `sbxabi1-…`) is a hash of every field that affects binary
compatibility. It is deliberately finer-grained than the wheel tag: the
platform tag names only the Emscripten version, but two builds from the same
Emscripten with different pthread or memory settings are still incompatible.

The tag is what the packaging ecosystem understands. The id is what this
project caches and verifies on. Change any field that matters and the id
changes with it, so every previously built wheel stops matching rather than
being loaded against an interpreter it no longer fits.

The id is a hash rather than a hand-maintained version number because a
version number is bumped when someone remembers to bump it, which is not the
same occasion as the ABI actually changing.

## Build profiles

`--enable-wasm-dynamic-linking` is what makes extension loading possible at
all, and it is a separate profile rather than a flag on the existing one:

| Profile | Threads | Native extensions |
| --- | --- | --- |
| `core` | no | linked in, fixed |
| `threaded-fixed` | yes | linked in, fixed |
| `dynamic` | yes | **loaded at import** |

`dynamic` is the profile shipped in the npm package. Build it with:

```bash
make -C python-runtime python PROFILE=dynamic
make -C python-runtime package PROFILE=dynamic
```

`dynamic` has its own OpenSSL sysroot (`out/sysroot-dynamic`) because every
object linked into a `MAIN_MODULE` must be position independent.

## Adding a package

Write a recipe and build it:

```bash
python3 python-runtime/scripts/build_extension.py path/to/recipe.json
```

A recipe names sources or a crate, and nothing else:

Pydantic is version-pinned at the native boundary. The repository carries both
`pydantic-core==2.23.2` for `pydantic==2.9.x` and `pydantic-core==2.46.5` for
newer Pydantic releases that require it. Keep each exact pair in the wheel
index; a newer core cannot satisfy an older Pydantic requirement.

```json
{
  "name": "sbx-c-probe",
  "version": "1.0.0",
  "kind": "c-extension",
  "modules": [{ "name": "sbx_c_probe", "sources": ["sbx_c_probe.c"] }]
}
```

```json
{
  "name": "sbx-rust-probe",
  "version": "1.0.0",
  "kind": "pyo3",
  "modules": [{ "name": "sbx_rust_probe", "crate": ".", "crateName": "sbx_rust_probe" }]
}
```

**A recipe cannot set compiler or link flags.** They come from the contract.
A recipe that could set its own `-pthread` could produce an artifact that
links and then corrupts memory, and the mismatch would be invisible in the
wheel it produced.

The builder refuses to produce a wheel unless the artifact is a WebAssembly
binary, declares a `dylink` section, and exports `PyInit_<module>`. All three
failures are misleading at import time, so they are caught at build time and
named.

## Rust and PyO3

Two properties of the Rust toolchain decide how this works, and both were
found as link errors rather than documented limitations.

**Rust's shipped `std` for `wasm32-unknown-emscripten` has no atomics.** It is
built without the `atomics` and `bulk-memory` features, so it cannot be linked
into a shared-memory module — and this runtime's main module has pthreads. The
error names an `.rcgu.o` file:

```
wasm-ld: error: --shared-memory is disallowed by …rcgu.o because it was not
compiled with 'atomics' or 'bulk-memory' features.
```

So `std` is rebuilt from source with those features, which needs a nightly
toolchain for `-Z build-std`. This is a property of the threading model, not a
preference: a runtime built without pthreads would link stable Rust's shipped
`std` unchanged.

**Rust panics unwind using the WebAssembly exception proposal.** A `cargo`
built extension imports a `__cpp_exception` tag, and a main module linked
without exception handling cannot supply one:

```
LinkError: WebAssembly.Instance(): Import #420 "env" "__cpp_exception":
tag import requires a WebAssembly.Tag
```

CPython is therefore linked with `-fwasm-exceptions` (and
`-sSUPPORT_LONGJMP=wasm`, since the two share a mechanism). Enabling it in the
main module is preferred over forcing `panic = "abort"` on every Rust
extension, because PyO3 turns a Rust panic into a Python exception by catching
the unwind — an aborting build would take the process down instead of raising.

PyO3 is cross-compiled through a generated config file rather than allowed to
probe, because probing describes the *build machine's* interpreter, which is
how a cross build silently produces a host artifact.
`suppress_build_script_link_lines` matters most: without it PyO3 emits
`-lpython3.13`, and there is no such library here — the CPython symbols come
from the main module at load time, which is what a side module's undefined
symbols are for.

## Installation

The installer resolves before it downloads, and downloads before it writes:

1. Already-installed distributions are skipped.
2. Dependencies are resolved with constraint propagation and conflict
   learning (`src/runtime/python/resolver.ts`).
3. A compatible pure-Python wheel is preferred.
4. Then a wheel tagged for this ABI.
5. Then, if enabled, a source build.
6. Every download is verified against the digest the index published. An
   artifact published without one is refused rather than installed unverified.
7. The whole solved graph is staged and committed as one transaction.

Resolution reads dependencies from **PEP 658 metadata sidecars**, a few
kilobytes each, rather than downloading whole wheels to read `METADATA`. That
is what removed the previous installer's arbitrary 96-candidate cap: the cap
existed because discovering that forty releases all needed the same
unavailable extension cost forty multi-megabyte downloads. What bounds the
search now is a deadline and a cancellation signal, which are honest limits —
they say the search ran out of time, not out of an arbitrary allowance.

## Security model

- Wheels are fetched only when the container's `allowOutbound` policy permits
  it. A container that refuses `curl` cannot install packages.
- Every artifact is verified by SHA-256 against the index's published digest
  before any of it is written.
- Extension code runs inside the same WebAssembly sandbox as the interpreter.
  It has no more access to the host than Python does: the filesystem it sees
  is the container's, served over the host ABI.
- The build toolchain is not shipped to consumers. The npm package contains
  the runtime and the loader.

## "The ASGI callable works" is not "Uvicorn serves"

These are different claims, and only the first is proven.

**Proven.** FastAPI imports, Pydantic 2 validates a request body through the
compiled `pydantic_core` extension, and the application is driven through the
public ASGI protocol — an ordinary `await app(scope, receive, send)` — with the
response status and body asserted. Installing `fastapi[standard]` also installs
the FastAPI CLI, whose `fastapi dev main.py` and `fastapi run main.py` commands
are available. A synchronous endpoint is exercised, so Starlette's threadpool
path runs on real worker threads.

**Not proven.** `uvicorn main:app` binding a port that the container's
networking can route a request to. This is not merely untested; the layer is
missing, and the way it is missing is worth stating precisely because it looks
like success:

```python
s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.bind(("127.0.0.1", 8123)); s.listen(1)   # succeeds
```

`bind` and `listen` return cleanly, and `asyncio`'s event loop has
`create_server`. But that socket belongs to **Emscripten's own socket
emulation**, which is not connected to `VirtualHttpRouter` — the router that
already routes `curl localhost:3000` to a Node server in the same container. So
a server would appear to start, report itself listening, and never receive a
request. A test that asserted "uvicorn started" would pass while proving
nothing.

Closing this means bridging CPython's socket layer to the container network the
way `sbxfs.ts` bridges its filesystem: `listen` registers a virtual port with
the router, and an accepted connection becomes a socket whose reads and writes
cross the host ABI. It belongs with the socket layer, not with Uvicorn —
patching Uvicorn would make one server work and leave every other one broken.

Note also that `socket.SO_REUSEADDR` is absent from this build, which some
servers set unconditionally.

## A resolution behaviour worth knowing

`pip install pydantic` **succeeds** without a wheel index, by backtracking to
the 1.x line — which is pure Python and needs no compiled core. That is correct
of the resolver and is what pip does too, but it is unlikely to be what someone
asking for Pydantic today wants. Constrain the requirement (`pydantic>=2`) when
the 2.x line is what you mean; with the bound, an unavailable native dependency
is reported rather than worked around.

## Current limitations

- Building extensions requires Emscripten 5.0.6, a nightly Rust with
  `rust-src`, and the CPython build tree. Consumers install prebuilt wheels.
- Source distributions are resolved but not built in-container; a resolution
  that lands on one reports that no builder is configured, rather than
  claiming an install. The optional local and remote builders described in the
  architecture are not implemented.
- **Uvicorn cannot serve.** See the section above; the ASGI callable works.
- Emscripten documents dynamic linking with pthreads as experimental.
- The default package wheel index carries the ABI-matched Pydantic Core wheels
  beside the interpreter. Consumers can override it with
  `configurePython({ wheelIndex })` to add privately built extensions.
- `socket.SO_REUSEADDR` is absent from this build.

## Which tests prove what

| Claim | Test |
| --- | --- |
| A C extension imports and runs | `test/python-runtime/extensions.test.ts` |
| A PyO3 extension imports and runs | `test/python-runtime/extensions.test.ts` |
| The ABI id is derived, not hand-written | `test/python-runtime/extension-abi.test.ts` |
| Linux wheels are refused | `test/python-runtime/extension-abi.test.ts` |
| Resolution has no candidate cap | `test/python-runtime/resolver.test.ts` |
| Constraints propagate, conflicts explain | `test/python-runtime/resolver.test.ts` |
| A failed native install leaves nothing | `test/python-runtime/extensions.test.ts` |
| Real pydantic-core validates data | `test/python-runtime/fastapi.test.ts` |
| Pydantic 2 + FastAPI + ASGI + console script | `test/python-runtime/fastapi.test.ts` |

The FastAPI test needs `SANDBOXEDJS_CLEAN_NETWORK_TESTS=1`; the extension
tests need `make -C python-runtime extensions` to have been run, and skip with
a reason otherwise.