# Managed cloud operations (0.4.0)

## Scope and evidence

Historical managed-cloud work was merged into `rebuild/0.3-native` as `2382dc5`; subsequent evidence and current dirty-worktree acceptance must be identified separately. **Current frozen-candidate acceptance passed four real scenarios on both providers**; see [the release report](RELEASE-ACCEPTANCE.md) for the artifact hash, exact scope, metrics, original harness failures and retained empty Modal Apps. No resident service, Bash interception, source writeback, AI client or new provider was introduced.

The task pipeline remains: top-level command admission (not a recursive script audit) → one verified source snapshot → authorized resource allocation → existing remote supervisor → validated task receipt. `src/resources.ts` adds only the shared account lease, local estimate ledger and resource ownership/recovery boundary. Provider SDK details stay in each adapter. `reroute_run` / `reroute_status` remain the only registered model tools. Their static guidance is intentionally short: choose based on task context, prefer Reroute for heavy builds/full tests, keep quick checks/Git/files with Pi/AFT, and inspect failures with `reroute_status`. A per-turn hook may add at most three recent confirmed observations (no raw commands/output and no speedup claim); it never calls a model and never changes AFT/Bash ownership.

Locked SDK evidence was checked against installed declarations **and implementation**:

- `@daytona/sdk` **0.223.1**: `Daytona.list({labels})`, `create({image,resources,volumes,labels,autoStopInterval,autoPauseInterval,autoDeleteInterval})`, `volume.get(name,true)`, `Sandbox.refreshData/refreshActivity/setAutostopInterval/start/stop/delete`, `Image.base().dockerfileCommands()`. No guessed resize or billing API is used.
- `modal` **0.11.0**: `apps.fromName`, `sandboxes.list/create/fromId`, tags, CPU/memory reservations **and hard limits**, `timeoutMs`, named Volume mounts, registry image layers, `terminate({wait:true})`. This SDK defaults to its V2 backend; its actual `list`/`create` forwarding, rather than outdated V1-only comments, was checked. No private billing RPC is used.
- Official [Daytona Volumes](https://www.daytona.io/docs/en/volumes/) confirms S3-compatible object storage, independent persistence and project/subpath isolation. This is why random-write Cargo target directories are not placed on the Volume.
- Official [Modal Sandbox files](https://modal.com/docs/guide/sandbox-files) and [Volumes](https://modal.com/docs/guide/volumes) describe background commits and a final commit on Sandbox termination; explicit `sync` commit is **Volume-v2-only**. The implementation relies on the documented final commit and waits for termination, not an invented JS `Volume.commit()`. Lightweight and real-repository live acceptance has verified final-commit persistence; broad performance and failure recovery remain unverified.
- Public pricing fetched 2026-10-21: [Daytona](https://www.daytona.io/pricing/) vCPU $0.0504/hour and memory $0.0162/GiB-hour; [Modal](https://modal.com/pricing) **Sandbox/non-preemptible** physical CPU $0.00003942/core-second and memory $0.00000667/GiB-second (hourly $0.141912/$0.024012). Modal's cheaper preemptible Function rates must not be substituted. Rates are reviewable defaults, not a guaranteed quote or proof of free-credit eligibility.

**Daytona has since been verified live** against a real account: repeated smoke and cache cold/warm runs, a real-model plus real-AFT workflow, and a full acceptance on the exact merged commit (`f713f52`) at 2 vCPU / 4 GiB covering first-run image build, cache save then hit, intentional failure and retest, canonical identity-file exclusion, and detached-descendant cleanup. Evidence lives in `.artifacts/live-clean-commit-acceptance.json` and the earlier artifacts named in [DELIVERY.md](../DELIVERY.md). All created resources were then deleted; the account now holds zero sandboxes and zero volumes.

**Modal has now passed lightweight real-service checks** at 1 physical CPU / 2 GiB using `python:3.12-slim-bookworm`: separate Unicode output, task exit codes 0/7, confirmed compute release, direct Volume persistence, and (after the mount-alias fix) cold save / warm restore of a synthetic download-cache marker across different Sandboxes. Evidence: `.artifacts/modal-light-summary.md` and `.artifacts/modal-cache-fixed-result.json`. Subsequent acceptance SHA `e6b68ad176bd6cbec4fea4e30d5eec54e207e89a` also passed the default Node/Rust/Python toolchain and two identical real-repository install/typecheck/build/test/smoke loads at 2 CPU / 4 GiB (200 passed, 0 failed, 2 original installed-host-AFT skips each). A single-Sandbox diagnostic confirmed `bash -lc` discarded image PATH; Modal now uses `bash -c`, without image-recipe changes. Real download-cache restore passed in a new Sandbox; warm install was faster but full execution/end-to-end was slower. Evidence: `.artifacts/modal-project-acceptance-UQxc8A/SUMMARY.md`, `acceptance-result.json`, and `all-session-resources.json`. All nine session Sandboxes were stopped; five Volumes/Apps retained. Broad speedup and live failure/recovery acceptance are not claimed. **Billing is unverified for both providers**: every artifact records `actualBilledUsd: null`, and the only spending figure available is a rounded cumulative dashboard value explicitly marked as not that run's charge.

## Configuration and cost boundaries

Use the masked `reroute setup` CLI, or native/environment authentication. Pi's `/reroute` only edits non-secret settings. Named credentials are outside projects in `~/.config/reroute/accounts/NAME.json`; explicitly selecting one disables all fallback to another account. Never pass keys as CLI flags or paste them into Pi text/editor fields. Masked setup refuses storage inside the source project/repository; snapshots also exclude the actual relocated credential directory and native `.modal.toml`, even if someone has tracked them. This is defensive exclusion, not a general secrets scanner.

User defaults and named accounts are shared across projects, not execution consent or cloud ownership. `reroute setup` saves user defaults; `--project` opts into explicit project overrides. Project top-level settings replace user defaults; named profiles replace whole named profiles. Existing project files are retained. Before **any** remote execution, approve the canonical project/account in Pi or explicitly run `reroute authorize --yes`; `authorize --revoke` removes that project's reusable local consents. Private consent receipts live outside project source under the existing state directory. Changing accounts/targets needs new consent. The managed policy below is a **second**, independent spending boundary.

A managed profile uses `id: "auto"`, `compute: {cpu,memoryGiB}`, an image and the `managed` policy. User authorization defaults to false. `--authorize`, or an explicitly confirmed Pi settings edit, grants future create/start only within the configured local envelope. Editing through CLI setup requires reviewing rates again and does not silently carry prior authorization forward.

`reroute:node-rust-v1` builds a starter image with official SDK image layers: Rust stable, Node 22/npm, Python 3, Git/rsync and build tools. It includes **no local source context**. First image building may take time and can fail; Modal has now built/started this recipe and passed actual Rust compilation plus Node/Python checks. The executor must preserve image PATH rather than source PATH-resetting login profiles. A registry image, preferably digest-pinned and containing the project's exact toolchains, can replace it. pnpm/yarn/bun or nonstandard Rust versions must be supplied in your own image; the starter does not pretend to provide every package manager.

Daytona requires whole vCPU/GiB. Modal uses physical CPU millicores, at least 0.125 cores, and whole MiB memory with a 128 MiB minimum. Provider limits/quotas are still authoritative. SDK/API errors are failures, not a silent smaller specification. Attached targets are not resized; per-task overrides require managed mode.

Local reservation:

```
hourly = cpu * reviewedCpuRate + memoryGiB * reviewedMemoryRate
minutes = ceil(taskTimeoutSeconds / 60) + idleMinutes + 5
reservedUsd = hourly * minutes / 60 + storageMonthUsd
```

The ledger reserves this before billable API calls, serializes tasks using the same local credential scope, and checks both task and current-month estimate limits. The storage allowance is charged to the **estimate reservation on every task**: conservative by design, not a measured storage invoice. Failed/uncertain reservations are not silently refunded. `runtimeSeconds` measures client-observed allocation/execution elapsed time, not full provider billing uptime.

Official balance/remaining gift credit: **unavailable**. Actual billing: **unavailable**. Cost shown by Reroute: **local estimate**. An active named gift account is merely the account the user chose. The default $3 estimate budget does not mean a guaranteed RMB 20–30 self-pay ceiling. Retained Volumes, stopped disks/images, egress, image building, external resource use, stale local state, rate changes and provider failures may produce costs outside this reservation. Use available native provider account spending controls and inspect your actual balance before enabling paid operations.

## Lifecycle and recovery

An installation UUID plus canonical project/profile identity forms the ownership marker. Two local installations with the same project path must not accidentally share lifecycle ownership. Keep `~/.local/state/reroute-v3/cloud` (or `REROUTE_STATE_DIR` / XDG state equivalent): it contains ownership ID, account lease, allocation intents and reservations, **not API keys**.

- **Daytona:** label/name discovery, create or reuse matching compute, explicitly authorized start of stopped/archived owned compute. No hot resize. A provider-side auto-stop window covers task timeout plus safety margin; it is refreshed immediately before execution. After confirmed completion it shortens to the configured idle delay. Auto-delete is disabled so a temporary stop cannot delete working data. The project Volume survives explicit Sandbox destruction.
- **Modal:** App/tags/name discovery and a named Volume. CPU/memory hard limits accompany reservations. A lifetime guard covers execution; an expiring Sandbox is not silently reused for a task that cannot fit. The SDK cannot extend lifetime in place. Confirmed tasks release managed compute immediately with `terminate({wait:true})`; `idleMinutes` is a fallback lifetime allowance, not a mutable Modal idle setting. The next task recreates compute with the same Volume. Running/unknown tasks are never proactively terminated just to save costs. Attached legacy Sandboxes retain explicit filesystem-snapshot stop/resume, with the additional pending-resume checkpoint and old-resource termination check.
- `reroute resource stop --yes` / `destroy --yes` operate on owned allocations for the active project/profile and protect active/unresolved jobs. Repeated confirmed stop/destroy of an already released allocation is a no-op. They **do not delete Volumes or Apps/images**.
- `reroute resource recover --yes` only reclaims a dead local client's lease and looks up the **same** allocation. It never creates/starts compute, clears remote task locks, retries execution, or invents an exit code. Live/foreign/unverifiable lease owners are refused. A recovered resource being online does not resolve its jobs; use `reroute_status` / `reroute status JOB_ID` separately.
- A lost creation acknowledgement retains an intent. If discovery cannot find exactly one matching allocation, Reroute refuses a second create. Likewise a legacy Modal `pendingResume` marker blocks another snapshot recreation; locate its recorded name in the provider and explicitly attach the confirmed resource ID.
- **Manual-support boundary:** disconnected/unknown preparation or a vanished allocation may not be recoverable from a receipt. There is deliberately no “force clear unknown” button. Verify the exact resource/name and remote task state in the provider, preserve/back up local state, and resolve it manually before removing locks or authorizing another run. Do not delete the entire state directory to bypass a block. Provider stop/absence alone is not a fabricated task-exit receipt.

Native guard windows mitigate abandoned compute but cannot prove the provider honored them. Storage is intentionally retained and can keep billing until the user verifies it is unused and removes it with native provider controls. Reroute does not claim a zero-leak or absolute billing guarantee.

## Cache design and measurements

Local scratch holds source, task work/receipts/artifacts, Cargo target, installed node_modules and toolchain writes. On a confirmed task, only npm `_cacache`, pnpm/bun stores and Cargo `registry`/`git` stores are archived onto `/var/reroute-cache`. Each project has a separate owned Volume; lockfiles, toolchain inputs, provider and image namespace archive keys. Modal exposes the fixed cache mount as a symlink to `/__modal/volumes/vo-<ID>`; only that exact alias/target layout is accepted, with link-free target ancestors and unchanged archive-file/member checks. Unexpected layouts remain warned misses.

Archives are at most 256 MiB compressed / 1 GiB expanded. Each restore/save phase has a 120-second supervisor deadline; transport and provider guards include a five-minute margin so a slow Volume does not consume the build deadline. Timed-out cache I/O is a warning/miss, not a fabricated build timeout. Uninterruptible platform I/O and service failures still require live fault testing. Restore stages and validates before replacing local stores; traversal, links, special files, truncation and checksum damage are rejected. A corrupt/unavailable archive is an explicit warned miss, not a fake hit. Failed persistence does not replace the task's exit code. Unknown termination/changed work ownership never publishes a cache or releases the task lock.

Archives are built in a private scratch directory, then written as new ordinary files with an embedded SHA-256 in the filename. The Volume path is not chmod/chown-ed and publication uses no Volume temporary directories, symlinks or rename/overwrite. Only closed, readback-verified generations are reported saved; restore skips partial/checksum-mismatched generations and can still read legacy `KEY.tgz` archives. One checksum-verified previous generation is retained as fallback; older/incomplete generations are pruned only after new-publication verification. Prune failures warn separately and may retain extra files. Legacy archives and other cache keys remain retained. This avoids depending on transactional behavior that [Daytona FUSE Volumes do not promise](https://www.daytona.io/docs/en/volumes/#share-data-between-sandboxes). Close/readback is not proof of persistence after stop/recreation; that still requires authorized live acceptance.

`job.metrics` records `provisionMs`, `transferMs`, `executionMs`, `cacheHit`, `cacheSaved`, `cacheRestoreMs` and `cacheSaveMs`. `cacheSaved=true` requires successful publication/readback; `cacheSaveMs` alone is just elapsed time, including failures, and older receipts may omit `cacheSaved`. `job.cacheWarning` is distinct from execution failure. Cache failures report the operation stage (such as `volume-open`, `volume-write`, `volume-close`, `volume-readback`) and OS errno, not raw exception paths or credentials. Volume archive hit is not evidence of a faster build. Cargo target survives **Daytona stop/restart**, but is deliberately recompiled after Sandbox destruction; Modal target is scratch and therefore does not persist between normally released tasks. Do not assume dependent Node commands carry `dist` across fresh task workspaces: use a reviewed project script that does the required sequence in one task (explicit trusted prefix/CLI exec when the conservative router refuses it), rather than weakening shell routing or relying on source writeback.

Offline tests use real Python processes/files, simulated mounted storage and scratch reconstruction, plus injected official-SDK-shape contracts. They test semantics, not network Volume throughput. A live cold/warm speedup or on-device resource saving is **not yet established**.

## Authorized live acceptance

**Do not execute this section without privately configured credentials and explicit resource/billing authorization. No keys need to be sent to an assistant.** Use a small disposable project containing a known `cargo test` or locked Node build; inspect what the source snapshot will include.

1. Install this local checkout using [README](../README.md#managed-cloud-profiles). Run masked `setup`, use Space to select Daytona and optionally Modal together, then Enter. New profiles use separate named credential references; existing profiles can be kept unchanged or another account added. Review the small preset/toolchain defaults and per-account rates/budgets, or choose Customize. Choose the project default; `setup --advanced` retains existing Sandbox/account edits and other providers. Use a short task timeout for acceptance. Keep authorization off initially. Approve this disposable project/account explicitly (`reroute authorize --yes` or the Pi confirmation), independently of the spending policy. Run `reroute plan --timeout 120 -- cargo test` and inspect `reroute credentials --credential ACCOUNT`. Neither command creates compute. Verify the actual provider balance and available native budget controls, then explicitly authorize the managed policy.
2. From that project, use the optional CLI's explicit shell mode for the split-output/exit probe:
   ```sh
   reroute exec --timeout 60 --shell 'printf "live stdout\n"; printf "live stderr\n" >&2; exit 7'
   ```
   Preserve stdout, stderr, shell exit **7**, emitted Job ID, the local job JSON and provider resource name/ID. This is intentionally an explicit CLI probe, not an ineligible compound `reroute_run` call. The old `npm run test:live` remains **existing-resource-only** and now refuses managed profiles, so its previous opt-in cannot accidentally authorize new compute.
3. Run a single eligible real build/test twice, e.g. `reroute exec --timeout 120 -- cargo test`. Save each Job ID and metrics, verify actual build success, compare cold/warm restore/save/execution time and provider traces. In Pi, use `reroute_run({command:"cargo test",timeout:120})`; confirm ordinary Bash remains AFT-owned. Do not claim cache benefit if the measured warm case is slower.
4. On Daytona, `reroute resource stop --yes`, then run another authorized task. Verify reuse of the same resource and cache. After confirming **all tasks terminal**, `reroute resource destroy --yes`, then run again: verify a new Sandbox, the same project Volume, a download-cache hit and the expected **missing scratch target**. Modal's normal task release already exercises recreation; verify actual Volume data after the previous `terminate(wait:true)` has completed. A later status query on released Modal compute correctly returns history with `remoteConfirmed=false`, not fake fresh verification; the execution tool obtained its task receipt before release.
5. Override one task with `--cpu 4 --memory 8` (shorten timeout or raise a reviewed task cap if the estimate rejects it). Confirm provider-reported units/specification and unchanged saved defaults. Set up a separate Modal profile/account, `reroute profile NAME`, repeat steps 2–4. Daytona vCPU and Modal physical cores are not claimed to be equivalent performance.
6. Exercise a failing build, a confirmed short timeout, SIGINT cancellation, and a deliberate client disconnect/crash while watching provider-side state. Do not rerun the disconnected task. Recover the dead client lease, query its Job ID, and confirm no duplicate resource/task was created. Test creation-response loss in a controlled environment; inspect the recorded allocation name rather than generating a second allocation. If state stays unknown, retain it for manual provider inspection.
7. Confirm compute is stopped/released after completion or its fallback guard, including after a killed client. Inspect the native provider resource list and billing dashboard. Stop unused allocations profile by profile. Delete only verified-unused retained Volumes/images through native controls. Preserve redacted evidence with exact checkout/tarball integrity, SDK versions, Job IDs, resource IDs, metrics and actual billing observations.

Lightweight Modal execution/release, synthetic cache persistence, starter image/toolchain and actual repository cold/warm builds have historical evidence at the SHAs above. Broader quota/failure acceptance, Volume performance, service-side auto-stop after client death and real billing remain pending. The frozen global-default/project-consent and packed source-transfer candidate has since passed separately authorized real-service rechecks on both providers; see [release acceptance](RELEASE-ACCEPTANCE.md). Old cloud evidence alone still does not qualify a new checkout. No fake cloud receipts or sandbox IDs are provided as evidence.

## Pi/AFT workflow acceptance (no cloud)

`tests/pi-history.test.mjs` exercises bounded, sanitized history and failure isolation. `tests/pi-coexistence.test.mjs` checks the actual installed AFT backend in both loading orders, ordinary Bash execution and per-turn prompt integration without model calls. `tests/pi-model-selection.mjs` is a separate explicit opt-in real-model probe, excluded from `npm test`; it uses the user-configured model via Pi’s official SDK with read-only auth, isolated settings, no personal skills/context, no provider fallback and no cloud credentials/transport. Its three Chinese requests do not name tools: light Git/version checks, complete tests then build, and failed tests stopping before build/local retry.

The historical version-pinned probe passed the light case before a model-provider request error blocked heavy/failure follow-up; `.artifacts/pi-workflow-selection.json` retains that partial result. It is superseded for current acceptance by the **real Pi 1.4 / AFT 0.62 six-category model probe** and independently authorized light-only harness recheck: 15 total requests, heavy work selected Reroute first, failure/unknown states queried status without local retry. The original 5/6 harness failure remains unchanged. These are isolated local-Provider executions, **not model-driven cloud or universal selection-stability evidence**. Current scripts and receipts are in `.artifacts/release-acceptance-1oje2Y/model-live/`; read [release acceptance](RELEASE-ACCEPTANCE.md) before arranging any new explicitly authorized paid probe.


### Transfer measurements in this checkout

The historical Modal acceptance had 72 input files / 742,267 bytes; snapshot preparation was 428–446 ms but combined transfer was 82.3–123.7 s. That older counter combines request/stage, sequential per-file SDK puts and seal/verification, so it does not establish a network-only bottleneck. SDK providers now send at most one manifest-indexed gzip byte stream for needed files plus the request; native rsync stays unchanged. Remote private staging validates bounds, full compressed stream, paths and hashes before baseline publication. No cross-project cache/source reuse or bidirectional protocol is introduced.

Local latency-injected evidence (128 repetitive 64-KiB files, 10 ms per put, one sample) reduces cold puts 129→2 and combined transfer 1,612.5→419.0 ms; unchanged input varied 179.9→239.2 ms, so no universal speedup is claimed. See the current [delivery evidence](../DELIVERY.md#daily-use-optimization--unreleased-working-tree), `.artifacts/transfer-performance.md` and `tests/transfer-measurement.mjs`. **A subsequent authorized recheck passed four real tasks on each provider** using one frozen installation. New Modal combined transfer was 12.65–16.32 seconds, not a controlled speedup factor against the different historical dataset. Daytona unchanged input uploaded zero source files, and its one-file case uploaded one file. See [release acceptance](RELEASE-ACCEPTANCE.md) for measurements, cloud cleanup and retained App metadata.

Current receipts separate input snapshot, pack, manifest upload, source upload, remote stage/unpack/verify/work preparation, cache restore/save, execution, descendant/work cleanup and provider release. Byte figures distinguish uncompressed changed source from compressed archive/request; they are not SDK wire traffic. Enclosing wall times can overlap subphase counters. Pi tool updates show actual fixed stage transitions, not estimated percentages; output/error/receipt safety is unchanged.