# Alerting — operator guide Getting celilo to tell you when something is wrong, on a channel you actually read, without teaching you to ignore it. This guide is task-ordered: set up who gets told, how they get told, and in what order; then watch a module; then handle an alert when it arrives. --- ## The shape of it ``` monitor ──runs a check every N minutes──▶ alert ──escalation policy──▶ route ──▶ your phone │ │ │ └─ a module's health_check hook, └─ one per PROBLEM, not per run, └─ you reply or a built-in (machines_reachable) identified by a stable key `ack` ``` Four nouns, and they are worth keeping straight: | noun | what it is | |------|-----------| | **person** | a human, with a timezone and optional quiet hours | | **route** | one way to reach a person (a Signal number). A person may have several | | **escalation policy** | an ordered list of *steps*: which route, how many minutes in | | **monitor** | a check that runs on a schedule and produces alerts | An alert's identity is its **key** — `module:caddy/check:cert-validity`. The same problem across a hundred runs is one alert, so a slow-burning failure does not become a hundred messages. --- ## 1. Add yourself ```bash celilo person add peba --timezone America/Los_Angeles ``` Quiet hours are optional and per-person. They defer the *message*, never the escalation clock — steps keep advancing while you sleep, so if nobody acknowledges by 03:00 the chain has already reached whoever is awake: ```bash celilo person add peba --timezone America/Los_Angeles \ --quiet-hours 22:00-07:00 ``` A policy may be marked to page through quiet hours anyway (`escalation_policies.bypass_quiet_hours`); `celilo escalation-policy list` shows which do. Use it sparingly — a channel that wakes you for a warning is a channel you will mute. ## 2. Add a route A route needs a transport module. Deploy `signal` first (see [the signal module's README](../../../modules/signal/README.md)); then: ```bash celilo route add peba signal --address +15551234567 --can-ack ``` `--can-ack` says replies from this address are trusted to acknowledge alerts. A route without it is send-only — it still gets paged, it just gets no reply token, because a reply instruction nobody can follow is worse than none. `--severity-floor critical` keeps a route out of everything below critical. That is how "hazel's phone is for real emergencies only" is expressed. ```bash celilo route list ``` ## 3. Build an escalation policy Steps are `--after` minutes from when escalation *begins*, not from the previous step — so the numbers read as a timeline rather than an accumulation. A step names a person and a transport, which together identify the route. ```bash celilo escalation-policy add oncall celilo escalation-policy step oncall peba signal --after 0 celilo escalation-policy step oncall hazel signal --after 15 ``` That is: page me immediately; if nobody has acknowledged 15 minutes later, page hazel. A step whose route's severity floor excludes the alert is skipped *immediately* rather than costing its delay in silence. ## 4. Watch something A module that declares `hooks.health_check` gets a monitor automatically on its first deploy. To see what is watched: ```bash celilo monitor list ``` To add one by hand, and to point it at a policy: ```bash celilo monitor add caddy --interval 5m celilo monitor add machines_reachable --interval 5m celilo escalation-policy assign oncall caddy ``` The target says which kind it is: an audit category (`machines_reachable`, `health_coverage`) is a built-in check, anything else is a module's `health_check` hook. `celilo monitor run ` runs one now. `machines_reachable` is deliberately **unsuppressible**: it is how you find out the fleet is unreachable, so nothing may explain it away. Modules with no health check at all show up under the built-in `health_coverage` check — "not observed" is a state worth being able to see. --- ## Living with it ### What you will see ```bash celilo alerts list celilo alerts list --json # for scripts celilo module list # health column, per module ``` The health column reads `healthy` / `failing(n)` / `suppressed` / `not observed`. "not observed" is not "healthy" and is not printed as if it were. ### When one arrives The message names the key first — on a phone that is what shows in the notification, and half-awake you need to know *what* before *why*. ``` 🔴 module:caddy/check:cert-validity certificate for www.example.com expires in 2 days reply K3M9QX ``` Reply with the token, or just `ack` if only one alert is outstanding for you. Everyone else who was paged is told **who** took it, so the secondary can go back to bed. From a terminal: ```bash celilo alerts ack module:caddy/check:cert-validity --as peba ``` ### ack vs silence vs resolve These look similar and mean entirely different things. Collapsing any two is how an alerting system stops being trusted. | | means | escalation | alert state | |---|---|---|---| | **ack** | "I have this" | stops | still **firing** — the problem is still happening | | **silence** | "I know, stop telling me until X" | stops until expiry | firing, silenced | | **resolve** | "the condition is gone" | stops | resolved | ```bash celilo alerts silence module:caddy/check:cert-validity --for 2h celilo alerts resolve module:caddy/check:cert-validity ``` `celilo monitor disable ` is the bigger hammer: it stops the check running at all, rather than silencing what it finds. Silence always expires — that is what makes it safe to offer. ### What `resolve` is actually for Reasonable question: if the check is still failing, what does resolving it achieve? Nothing — and that is the point. **Resolve does not override the monitor.** The next successful run is authoritative, so if the condition is still there the alert comes straight back on the next sweep. Its use is narrower than it looks: **you have just fixed something, and the monitor's interval is fifteen minutes.** Without resolve you stare at a stale `firing` row — and keep escalating against it — until the next run notices. Resolve says "I know this one is done, stop counting" and costs nothing if you are wrong, because the monitor corrects you. If you find yourself resolving the same key repeatedly, the check interval is too long or the check is wrong. That is the signal, not the resolve. ### Replies can only acknowledge The reply grammar is deliberately one verb. A bare token acknowledges; so does `ack`, `ok`, `k`, or 👍: ``` K3M9QX → acknowledged K3M9QX ack → acknowledged K3M9QX resolve → REJECTED, and nothing happens ``` `resolve` and `silence` are terminal operations, not reply verbs — every extra verb is another thing to mistype at 3am, and a reply that silently did the wrong thing would be worse than one that did nothing. An unrecognised verb is rejected and counted, so `celilo alerts poll` reports it rather than swallowing it. ### Why didn't I get paged? Every reason to stay silent is recorded rather than being an early return. ```bash celilo alerts list --json | jq '.[] | {key, state, suppressedBy: .suppressedByAlertId}' celilo monitor run caddy # run one check now, see what it reports ``` The usual answers: - **within grace** — a new failure waits out a grace window before it may page, so a check that flaps for thirty seconds never reaches you. - **suppressed** — something upstream explains it. A firing machine explains every module on it; a firing capability provider explains its zone's consumers. This is derived from the deployment topology, never configured, so it cannot drift from reality. - **deploy window** — a deploy suppresses its own module. Restarting a service fails its own health check, and paging you about the deploy you are personally running is how a channel gets muted. - **awaiting confirmation** — suppression just lifted. The alert waits for one more successful run before paging: if it recovered along with its cause, it resolves quietly. - **quiet hours** — deferred, not dropped. It arrives when the window ends, *if* it is still a problem then. ### What happens when the check itself breaks A check that could not **run** tells you nothing about what it was watching. So an errored run resolves nothing — every alert it owns is frozen exactly as it was — and raises one module-level alert saying the check is broken. The alternative is the single most dangerous failure an alerting system can have: reading "could not run" as "found nothing wrong", clearing every alert, and telling you everything is fine at the moment you can see least. --- ## Running it Two commands, both dispatcher-driven, both safe to run by hand: ```bash celilo alerts sweep # timer.tick.5m — run due monitors, escalate, notify celilo alerts poll # read replies off every bidirectional transport ``` `poll` runs far more often than the sweep: "I texted ack and nothing happened" is the experience the whole design exists to prevent. celilo-mgr polls the transport; nothing calls in. That direction is deliberate — an inbound webhook would need public HTTPS, which means caddy, DNS and a certificate, so the ack path would depend on the very infrastructure being paged about. --- ## Answering deploy questions from your phone Because an alert and an interview question are the same message with different reply semantics, the same transport carries both. A headless deploy that needs a value can ask you: ``` ❓ caddy needs: admin_email reply K3M9QX ``` Two rules, both enforced before anything is sent: 1. **A terminal wins.** If stdin is a TTY, the question is answered there — a deploy started from your laptop must not appear to hang while silently waiting for a text. 2. **Secrets are refused.** `secret.*` questions never travel over a messaging transport, on a headless box or anywhere else. Typing a credential into a chat puts it in the transport's message store and the device's history before the responder ever sees it. Answer those at a terminal. --- ## See also - **Rolling this out to a live fleet**: `openspec/changes/add-alerting/ROLLOUT.md` — ordered stages, what to verify at each, and how to back out. - **Building the libsignal aarch64 native** (needed on ARM hosts): `modules/signal/build/README.md` - Design and the decisions behind it: `openspec/changes/add-alerting/design.md` - Subsystem map: [CELILO_SUBSYSTEMS.md](../CELILO_SUBSYSTEMS.md) § Alerting & notifications - The Signal transport: `modules/signal/README.md`