---
sidebar_position: 5
title: Agent operator
---

# The agent-ops sidecar

Every Zibby Managed App ships with **agent-ops**, an autonomous daemon sidecar that runs alongside the app container. It does what a human operator would do — checks health, restarts on crash, prunes disk, rolls upgrades — only it does it every hour, never sleeps, and never forgets to file a runbook.

This is the structural difference between "deploy button on a VM" and **Zibby**.

## What it does

| Task | Cadence | What happens |
|---|---|---|
| **Hourly health check** | every 60 min | HTTP probe + container state + EFS usage. Recorded as a structured run record. |
| **Self-heal on OOM** | event-driven | Container exits with OOMKilled → agent-ops triggers an ECS restart, records the recovery. |
| **Disk-pressure prune** | when EFS > 90% | Removes safe-to-delete caches (e.g. n8n execution history older than 30 days). Configurable. |
| **Upgrade orchestration** | on schedule | When a new app version lands in the catalog, agent-ops can run the in-place upgrade on a cron you set. |
| **Activity log** | every action | One row in the app's "Agent activity" tab, with structured fields you can grep / chart. |

Every action lands in DynamoDB as an `app-runs` record — queryable by anything from an agent node to a Grafana dashboard.

## See it in action

```bash
zibby app activity a1b2c3d4
```

```
Time         Action                      Status   Duration   Notes
14:00:01     hourly_health_check         ok       1.2s
13:00:01     hourly_health_check         ok       1.1s
12:04:38     oom_recovery                ok       4.8s       restarted container after OOMKilled
12:00:01     hourly_health_check         warn     2.1s       container restarting
11:00:01     hourly_health_check         ok       1.3s
10:00:01     hourly_health_check         ok       1.0s
```

The dashboard's "Agent activity" tab shows the same records with extra context (HTTP status codes, container logs at failure time, recovery diff).

## Run records ≠ logs

A run record is **structured metadata** about something agent-ops did:

```json
{
  "instanceId": "a1b2c3d4",
  "runId": "01J9KZQF...",
  "action": "hourly_health_check",
  "status": "ok",
  "startedAt": "2026-05-30T14:00:01Z",
  "duration_ms": 1234,
  "httpStatus": 200,
  "containerState": "RUNNING"
}
```

Logs are unstructured text. Run records are queryable, chartable, and aggregatable — that's why the Agent activity tab can show a 30-day uptime % without you running grep.

The CLI exposes them:

```bash
zibby app activity a1b2c3d4 --since 7d
zibby app activity a1b2c3d4 --action oom_recovery
zibby app activity a1b2c3d4 --json | jq '.[] | select(.status == "warn")'
```

## Why a sidecar, not a centralized controller?

Three properties only this shape gives you:

1. **No noisy-neighbor failure mode** — your instance's agent-ops can't be blocked by another instance's slow health check
2. **Per-instance customization without a central feature flag** — env vars (`AGENT_OPS_CHECK_INTERVAL_MIN=15`, `AGENT_OPS_PRUNE_THRESHOLD=80`) live on your instance and just work
3. **Egress identity matches the app** — outbound calls from agent-ops use the same ENI / NAT path as the app itself, so when the app has a dedicated egress IP, agent-ops's webhook callbacks come from the same IP

## Customize via env

Per-instance agent-ops behavior is tunable via env vars (set on the app instance — Apps → ENV tab — or via `zibby app env set`):

| Env var | Default | What it controls |
|---|---|---|
| `AGENT_OPS_CHECK_INTERVAL_MIN` | 60 | Minutes between hourly health checks |
| `AGENT_OPS_PRUNE_THRESHOLD` | 90 | EFS usage % that triggers disk-pressure prune |
| `AGENT_OPS_AUTO_UPGRADE` | `false` | If `true`, upgrade automatically when catalog publishes a new version |
| `AGENT_OPS_NOTIFY_WEBHOOK` | — | URL to POST run records to (any HTTPS endpoint — your own backend, n8n, etc.) |

`AGENT_OPS_NOTIFY_WEBHOOK` is how you wire agent-ops into your existing observability stack — fire every run record into your team's #ops Slack via an agent trigger, into Datadog via their webhook receiver, or into your own database.

## Hooking agent-ops into an agent

The most powerful pattern: a Zibby agent that runs **on agent-ops events**.

Example: when an `oom_recovery` fires, run an agent that pulls the container's last-100-lines, classifies the crash, and pages whoever owns this app:

```bash
zibby app env set a1b2c3d4 \
  AGENT_OPS_NOTIFY_WEBHOOK=https://api-prod.zibby.app/v1/workflows/<wf-uuid>/trigger
```

The agent receives the run record as `input`, can call back to `zibby app logs` / `zibby app status`, and decides what to do. Agent-ops + agents compose into a self-operating fleet — humans only get pinged for genuinely novel failure modes.

## Upgrade orchestration

When you `zibby app upgrade <id>` manually, agent-ops watches the rollout and rolls back if the new task fails health checks twice in a row. With `AGENT_OPS_AUTO_UPGRADE=true` set, the upgrade fires on a cron (default: weekly, Sunday 04:00 UTC) — agent-ops runs the same flow:

1. Register new task definition revision (catalog's latest)
2. Update service, watch the rollout
3. If 2 consecutive health checks pass on the new revision → keep it
4. If 2 fail → roll back to the previous revision, log a `failed_upgrade` run record

The activity log shows the full attempted upgrade timeline so you can see why a rollback happened.

## How goal-mode deploys are supervised

When you `zibby app deploy --goal "..."` (see [Goal-mode deploys](./goal-mode)), agent-ops switches into a different mode at task start: `AGENT_OPS_BOOTSTRAP_MODE=agent_script`. Instead of "run a known image and health-check it", it does:

1. **Plan** — Claude (Write+Read tools only, no Bash) writes a complete `/tmp/install.sh` from the customer's goal text + the house rules. ~2 turns, ~$0.05 in tokens.
2. **Supervise** — agent-ops execs the script in a process group, then every 30s sends Claude (text-only) a snapshot of stdout/stderr tail + proc status. Claude returns one JSON line: `continue`, `done`, or `intervene`. On `intervene`, agent-ops SIGTERMs the process group, gives it 5s grace, SIGKILLs, and replans with the previous failure context.
3. **Auto-short-circuit** — if the script exits with code 0 AND the verify port returns 2xx-499, agent-ops declares `done` without consulting the supervisor. This catches the false-positive `intervene` that fires when the planner backgrounds the app with nohup and Claude can't see "startup logs" in the snapshot.

Hard caps: 5 iterations, 30 min wall-clock, $1.00 token budget — whichever hits first wins. Every iteration's script, supervisor turns, and final status are persisted under `/var/lib/agent-ops/agent_script-state/` on the per-instance EFS volume — `zibby app logs <id>` surfaces the timeline.

### House rules

agent-ops reads `AGENT_OPS_BOOTSTRAP_SYSTEM_RULES` at task start and prepends it to the planner's system prompt. We curate ~6 rules in the backend covering things like: "Write COMPLETE bash scripts per turn, never run individual commands", "Never background installs; only the final long-running app process", "End with a curl health-check loop on the target port", "If install > 15 min, switch to a pre-built distribution". The rules are env-driven so we can iterate wording without rebuilding agent-ops — and it's why goal-mode deploys tend to converge reliably rather than oscillate between "almost done" and "container exited".

You don't set these yourself; they're operator-curated. But they're the reason the planner consistently emits an install script that ends in a working `curl localhost:<port>` health check, not a half-finished session of individual commands.

→ Done with apps. See [Goal-mode deploys](./goal-mode), [Auth proxy](./auth), or [CLI Reference](../cli-reference#app-commands) for the full `zibby app` command surface.