---
sidebar_position: 5
title: Troubleshooting
---

# Troubleshooting

A self-hosted box is **one VM** — control-plane container + Scylla + object store + ephemeral run containers. When something is slow or failing, diagnose from the box's own logs; the reflex "it doesn't scale / the VM is overloaded" is almost always wrong (a `top` usually shows the box near-idle).

## First moves

```bash
top                                          # CPU / memory — usually reveals it's idle
docker ps -a                                 # are containers Up? did a run container Exit?
docker logs zibby-control-plane --tail 100   # the executor + reaper + API
docker logs zibby-bootstrap                  # first-boot template sync + the seeded token
curl -s http://localhost:3001/health         # control-plane health
curl -s http://localhost:3001/selfhost/version   # the running release (baked, authoritative)
```

To query the box's own API, use its `PROJECT_API_TOKEN` from `.env`:

```bash
curl -H "Authorization: Bearer <zby_...>" \
  "http://localhost:3001/executions?projectId=selfhosted-project"
```

A failed run's `reapedReason` field tells you why it was reaped. (The cloud Zibby MCP/CLI talks to the *cloud* backend — it cannot reach a self-host box; you need the box's own token.)

## Failure signatures worth recognizing

**Runs "fail" with empty steps + `pull access denied for zibby-agent … repository does not exist`.** An agent-image tag mismatch after an upgrade — the executor is launching a tag the reused `.env` doesn't have. Fixed at the root in installer 0.1.53+; immediate unblock:

```bash
docker tag zibby-agent:slim zibby-agent:local
```

**The dashboard shows the new version but the API still behaves like the old one.** `docker compose up -d` can judge a service "up to date" when only the image *content* changed. The installer force-recreates the control-plane container to avoid this; if you hit it, re-run the installer (or `docker compose up -d --no-deps --force-recreate control-plane`). Because `/selfhost/version` reads the **baked** file, a genuine version mismatch is *visible* rather than hidden behind a lying env-var label.

**Control plane won't become healthy right after boot.** Scylla cold-boots slowly (~1–2 min). The installer waits up to ~5 min; if it's still unhealthy, read `docker logs zibby-control-plane` — a broken data path shows up there, not in `top`.

**An upgrade ended at `rolled_back`.** The box is safe (back on the prior version). Read `GET /selfhost/upgrade/status` `logTail` for the failing phase, and check `docker logs zibby-control-plane`. Re-attempt with a **pinned** version (see [Upgrade & rollback](./upgrade)).

## Where things live

- **Data** — docker volumes `scylla-data` + the object store volume. Back them up (see [Backup & restore](./backup-restore)); they survive restarts but not `down -v`.
- **`.env`** — the runtime source of truth (`JWT_SECRET`, `ENCRYPTION_KEY`, tokens, `MAX_CONCURRENT_RUNS`, `AGENT_IMAGE`). Reused untouched on upgrade.
- **Upgrade backups + logs** — `<install-dir>/backups/upgrade-<id>/`.