# Upgrading MCP Hangar ## Upgrade to 2.21.0 ### giving up on a server is recorded as a stop When the recovery saga gives up on a server, the server now records `McpServerStopped` with `reason: max_retries_exceeded`, and right after it the `McpServerStateChanged` that moves it to `dead`. The server still ends `dead`, live and when its stream is replayed. - `mcp_hangar_mcp_server_stops_total{reason="max_retries_exceeded"}` counts each give-up once, as in 2.20.0. It is now counted from the event, the way `idle` and `shutdown` are. To alert on a give-up, use `increase(mcp_hangar_mcp_server_stops_total{reason="max_retries_exceeded"}[1h]) > 0`. A `reason!="idle"` rule also matches every operator stop. - The `reason` label is a closed set, and the metric's HELP line lists it: `idle`, `shutdown`, `user_request`, `manual`, `failback`, `compensation`, `detection_enforcement:block` and `max_retries_exceeded`. A REST stop whose body names any other reason is now counted as `manual`. The response still returns the reason it was given. - The audit log and the event store hold the new `McpServerStopped` record. A consumer that reads every `McpServerStopped` as `cold` should read the state change that follows it. - The alert handler's warning for an unexpected stop now fires on a give-up. - A failover whose backup is given up on ends, as it does when the backup is stopped. The primary's recovery no longer schedules a failback stop that would have turned the dead backup `cold`. ### a front-door call the gateway cannot answer is a tool error, not `-32602` On a front door, a flat `tools/call` whose upstream answer could not be turned into a valid `CallToolResult` -- one with no `content` -- reached the client as a JSON-RPC error with code `-32602` ("Invalid params"), which blamed the caller's arguments for an upstream's answer. The gateway's own per-call log line said `outcome: ok` for that same call, so the log and the client disagreed about it. Such a call is now answered the way every other failure on this surface is: a result with `isError: true`, whose text says the upstream result could not be returned. The upstream's payload is not quoted in it, and a valid result is unaffected. - A client that watched for the `-32602` now sees an ordinary tool error, and has to read `isError` to notice the failure. - The `front_door_tool_call` line for that call now reads `outcome: tool_error` with `reason: invalid_result`. An alert that counts non-`ok` outcomes will see calls it used to count as `ok` -- they were already failing for the client. ### an upstream task is handed only to a caller that can poll it When an upstream answers a `tools/call` with a task, Hangar now hands the task only to a caller that can poll it: a client on the 2026-07-28 revision that declared the `io.modelcontextprotocol/tasks` extension. That is the caller `tasks/get`, `tasks/update` and `tasks/cancel` already served. - **Any other caller gets a tool error, not a task.** On the front door's flat `tools/call` it is a result with `isError: true` whose text names the extension to declare. In a `hangar_call` batch the call fails with `error_type: "TasksNotNegotiated"`. Before, such a caller was handed the task, and `tasks/*` then refused it (`-32021` or `-32601`), so it could never read the task. - **The upstream has still made the task.** Hangar does not record it and does not cancel it. - **An upstream task in the current flat shape now works.** A task an upstream answers with as `resultType: "task"` (SEP-2663) is recorded and relayed like one in the older nested shape. On `hangar_call` the flat handle used to come back unrecorded, so `tasks/*` answered "Task not found", and the front door answered `-32602`. To keep receiving tasks, declare `io.modelcontextprotocol/tasks` under `clientCapabilities.extensions` on each request. ### a group member is judged by its own health A call through a group used to count against the member that took it whenever the call did not succeed. It now counts only when the outcome is evidence the member is unwell. **Nothing to change in configuration.** What you may see: - Fewer members leave rotation, and a group's circuit opens less often. `mcp_hangar_group_circuit_open` is at `1` less often, and `hangar_group_list` shows `consecutive_failures` at `0` for a member that only answered bad requests. - **A member that answered the request counts as healthy**, as a success does: a tool result with `isError: true`, a JSON-RPC error `-32602` (invalid params) or `-32601` (method not found), and a JSON-RPC error whose code is outside the reserved range `-32768` to `-32000`, such as a tool's own `-1`. - **A refusal Hangar makes before asking the member counts as nothing**: the command bus's `rate_limit`, an egress rule, or a tool the member does not list. Refusals by Hangar's gates (tool access, pins, validators, approval, the tenant's budget) already counted as nothing. - **These still count against the member, as before**: a transport failure, a timeout, a failed start, the member's own backoff or dead state, and a JSON-RPC error in the reserved range, such as `-32603` (internal error), `-32700` (parse error) or `-32000` (server error). An HTTP upstream's `4xx` or `5xx` status reaches the group as `-32000`, so it counts. - An alert that fired when callers sent bad requests through a group no longer fires for them. Alert on the tool errors themselves instead. ### a reload restarts only the servers whose settings changed A configuration reload used to restart every server whose entry left `resources` out, which is most of them, even when nothing in the file had changed. That applied to a reload over `POST /api/config/reload`, SIGHUP or the file watcher alike. Each reload stopped those servers and dropped their in-flight calls, and their next call started a new process. A reload now keeps a server running, with the same process, sessions, health and circuit state, unless the file changes something the server is built from: `mode`, `command`, `args`, `image`, `build`, `endpoint`, `env`, `volumes`, `resources`, `network`, `read_only`, `user`, `description`, `idle_ttl_s`, `health_check_interval_s`, `max_consecutive_failures`, a list of predefined `tools`, `auth`, `tls`, `http` or `capabilities`. Defaults count as set: leaving a setting out and spelling out its default are the same server. A group's inline members follow the same rule. What changes for you: - **A governance change no longer restarts a server.** A reload applies a change to a server's `tools` access block, `access`, `tool_access`, `tool_projection` or `header_exposure`, or to a member's `weight` or `priority` in its group, and the server keeps running. Before, such a change restarted the server. - **`mcp_servers_updated` lists only the servers the reload restarted**, in the reload response and in the `ConfigurationReloaded` event. The servers it kept are in `mcp_servers_unchanged`. - **A reload no longer restarts a server whose settings did not change.** To restart one, stop it with `hangar_stop` or `POST /api/mcp_servers/{id}/stop`. Its next call starts it again. - A server whose `env`, `description` or intervals were edited over the REST API is still restarted with the file's values, as before. ### the Python facade stops idle servers and health-checks them `Hangar` and `SyncHangar` now start the background workers `mcp-hangar serve` starts, and stop them on `stop()` or at the end of the `async with` or `with` block. No code change is needed, but an embedded gateway now does what a served one does: - A server idle for longer than its `idle_ttl_s` is stopped by the GC worker, which runs every 30 seconds. `HangarConfig.add_mcp_server()` defaults `idle_ttl_s` to 300. The next call starts the server again, and pays its start-up time. To keep a server running for the life of the host, give it a larger `idle_ttl_s`, up to 86400. - Every running server is health checked by the health-check worker, every 60 seconds, so a failing server is noticed, and one that recovers is returned to rotation, without a call. - The metrics snapshot worker records metrics history under `./data`, as it does under `serve`. - A facade started from a config file, `Hangar.from_config()` or `SyncHangar.from_config()`, watches that file, and a change to it reloads the configuration. To keep the file from being reloaded, set: ```yaml config_reload: enabled: false ``` `stop()` now waits for the workers' threads to end, up to 10 seconds in total. ### a group member is its top-level server, whatever the order in the file A group member whose `id` names a top-level server in `mcp_servers` is now that server, wherever the group appears in the file, at startup and on a reload. Before, a group listed above the server got a member built from the member entry alone. With only an `id`, that was a server with no command, which could not start, and not the server the rest of Hangar knew by that id. Two configurations now load differently: - **A member that names no server is refused.** A member whose id is not a top-level server, and whose entry does not say how to run one, fails the load: `Group 'pool' member 'm1' names no server`. Before, it loaded and failed only when a call was routed to it. A reload with such a member is refused and changes nothing. Declare the server under `mcp_servers`, or give the member entry what its mode needs: `command` for `subprocess`, `image` or `build` for `docker`, `endpoint` for `remote`. - **A member entry cannot redefine a top-level server.** When a member entry sets server fields such as `command`, `env` or `mode`, and a top-level server has the same id, the member is the top-level server and the entry's server fields are ignored. Before, which of the two the group held depended on the order in the file. The ignored fields are named in a `group_member_entry_settings_ignored` warning. `weight`, `priority` and `tools` on a member entry still apply. ### per-tenant execution budgets `execution.max_concurrency` bounds the whole process, so one tenant's burst could take every execution slot. The new, optional `execution.tenant_limits` section bounds each tenant on its own. **With no `tenant_limits` section, nothing changes.** ```yaml execution: max_concurrency: 50 tenant_limits: "tenant:a": {max_concurrency: 4, rps: 10, burst: 20} "*": {max_concurrency: 2, rps: 5, burst: 10} ``` - `max_concurrency` is how many of the tenant's calls may be executing at once. `rps` and `burst` are a token bucket: calls start at `rps` per second on average, and at most `burst` at once. All three are required. `max_concurrency` and `burst` are integers from 1 to 1000000000, and `rps` is above 0 and at most 1000000. A misspelt or extra key, or a value out of range, refuses the configuration. - **A tenant that is not listed** gets a budget of its own, built from the `"*"` entry. `"*"` is a template, not a pool that unlisted tenants share. Callers with no tenant share one such budget. **With authentication off, no caller has a tenant, so every caller shares that single `"*"` budget.** - **With budgets configured and no `"*"` entry, an unlisted tenant and a caller with no tenant are refused.** If you add `tenant_limits` for a few tenants, add a `"*"` entry too, unless refusing everyone else is what you want. - `none` cannot be a tenant id in `tenant_limits`: it is the metric's label for a call that no entry applied to. - **Where the budget is taken.** Both parts come after the policy gates (tool access, withdrawal, pins, the circuit breaker and validators), so a call they refuse spends nothing. - The token, and the check that the caller has a budget at all, come before the approval hold and the cold start. A caller with no budget, or over its rate, is refused before anyone is asked to approve the call, and before the call starts a stopped server. On a server that has not started yet, a pinned tool's pin can only be checked once the server starts, so such a caller is refused with `TenantQuotaExceeded`, not a pin mismatch. A call refused after that point (denied or expired at approval, no longer valid after the hold, or its server failing to start) gets its token back. - The slot comes last, just before the upstream call. A call held for approval, or waiting on a cold start, holds no slot. The slot is given back when the call returns. A call that the upstream answers with a task handle returns with the handle, so a task the upstream keeps running does not hold a slot. - **An approved call can still be refused** if all of its tenant's slots are taken when it is dispatched. The approval is then spent: running the call again needs a new one. The `tenant_quota_exceeded` log line, a warning in this case, names the approval. Leave room in `max_concurrency` for tools that need approval. - A tenant at its concurrency limit can still start a stopped server, with a call that is then refused. - A call over its budget is refused at once, never queued or retried, with the error type `TenantQuotaExceeded`. The front-door call log records it as `denied`, and `mcp_hangar_tenant_quota_refusals_total{budget,reason}` counts it, with `reason` one of `no_budget`, `concurrency` or `rate`. - **Budgets are counted per process**, like `execution.max_concurrency`. With N replicas a tenant can run up to N times its budget, so size each budget for your replica count. - A reload keeps the budget of every tenant whose limits did not change, with its calls in flight and its spent tokens. A tenant whose limits changed keeps counting the calls it has in flight, so lowering a limit never lets more than the new limit start. A tenant removed from `tenant_limits` is refused from then on, and calls it still has running are counted again if it is added back. Calls already running when budgets are first turned on are not counted. ### a front door can wait for its catalogue before it is ready Opt-in: nothing changes unless you add `tool_access.required_catalogue`, and it only takes effect with `tool_access.mode: front_door`. ```yaml tool_access: mode: front_door required_catalogue: servers: [payments, search-pool] retry_for_s: 600 ``` - `retry_for_s`, 600 by default, is a window that opens when the replica first applies its configuration. Inside it, `/health/ready` answers 503 until the boot warm-up has projected every listed server once. When they all have been, or when the window ends, readiness stops depending on the catalogue and goes back to today's rule. A replica is held out of the Service for at most `retry_for_s`, even if a listed backend never comes back. - The window counts from that first apply, before the rest of boot and the warm-up, so set `retry_for_s` to cover boot plus the warm-up. If they outlast it, the retry still gives each missing server one attempt before it ends; readiness does not wait for that attempt. - The readiness endpoint is unauthenticated, so its `catalogue` field reports counts and state only: `complete`, `holds_readiness`, `required`, `projected`, `missing_count`, `not_retried_count`, and `retry`. - The missing ids, and the reason the retry will not start a server, are logged in a `required_catalogue_waiting` line each time they change, and `hangar_health` returns them under `catalogue`. Both keep reporting after the window ends. - Within the window, a server the warm-up could not start is retried. The retry starts a server the way a call does, so a dead server waits out its backoff. It never starts a server that is `dead` for `given_up` or `capability_blocked`, it leaves a `degraded` server to the recovery saga, and it never starts a server this replica has already projected, so a server stopped for being idle stays stopped. Each attempt writes a `required_catalogue_retry` log line and one sample of `mcp_hangar_catalogue_retries_total{mcp_server, outcome}`. - The retry ends in a final state: `finished` once the list is met, `blocked` as soon as every server still missing is one it may not start, `exhausted` when the window ends, and `stopped` at shutdown or when a reload removes the list. After `blocked`, readiness still waits for the rest of the window. - Once every listed server has been projected, readiness never depends on the catalogue again: a backend that stops, goes idle or fails later does not make the replica not ready. - A group id is satisfied once any one of its members has been projected. A member defined only inline in its group can be listed too. - The block is checked at load, from a file and from a dict alike. These refuse the configuration: - an id that is not a server or group in `mcp_servers`; - a group with no members; - a key other than `servers` and `retry_for_s`; - a `retry_for_s` that is not a number above 0; - with a `coordination:` block or a shared storage backend, a listed server in a local mode (`subprocess`, `docker`, `container`, `podman`), or a group whose members all are. Only the replica holding the management lease may start one, so the others could never project it. Use `remote` mode for a server every replica must serve. - A persistence backend registered by a plugin is treated as shared, so a local-mode server is refused there too, as a precaution. - A single-replica deployment on a shared backend, such as `postgresql`, cannot require a local-mode server either: the configuration cannot know how many replicas will run it. In `egress` it is checked and then ignored. - A reload checks the block like any other key, and never moves the window. Inside the window, a reload replaces the list, and one that removes the block releases the wait. A replica that has met its list stays ready. After the window has ended, or on a replica that booted without a list, a reload that adds a required server does not hold readiness and does not start a retry: the server is reported in the log and in `hangar_health`, and is started by a call or a deliberate start. **A call's cold start now follows the call rules**, in every topology. The batch executor starts a cold or dead server as a call, not as a deliberate start, so a server that became capability-blocked, or went back into its backoff, after the executor checked it is not started by the call. The call is refused with the code the executor's own check gives the same condition: `CircuitBreakerOpen` inside a backoff, `CannotStartMcpServerError` for a capability block. **If readiness stays 503.** Read the `required_catalogue_waiting` log line, or `hangar_health`, for the ids. A server under `not_retried` is one Hangar gave up on or blocked for a capability drift. Fix it, then start it deliberately (`hangar_start`, or a start through the REST API), or take it off the list; otherwise readiness falls back when the window ends. **Probe timings.** A replica can now stay not ready for up to `retry_for_s` after it starts. A failing readiness probe does not restart a pod, but a rollout waits for it, so keep the Deployment's `progressDeadlineSeconds` above `retry_for_s`. ### the HTTP graceful-shutdown bound can be set `serve --http` reads a new key, `http.graceful_shutdown_timeout_s`. It is how many seconds a stop waits for the requests already in flight before it cancels them. ```yaml http: graceful_shutdown_timeout_s: 90 ``` Nothing changes unless you set it. Unset, Hangar passes uvicorn its own default, `None`, which waits for in-flight requests without a bound. The process then ends when they finish, or when something kills it. In Kubernetes that is the kubelet's SIGKILL at the end of the pod's `terminationGracePeriodSeconds`, 30 seconds by default. - The value is a positive whole number of seconds. Any other value, or an `http` that is not a mapping, refuses to start. A reload with such a value is refused too, and everything keeps running as it was. - The bound is read when the HTTP server starts. A reload checks it, but the running server keeps the bound it started with. Restart to change it. - Stdio mode has no HTTP server, and ignores the key. - `starting_http_server` logs the bound in force as `graceful_shutdown_timeout_s`, and logs `null` when it is unset. **In Kubernetes**, the bound only helps if the pod lives long enough to use it. The kubelet counts the grace period from the start of the `preStop` hook, so set `terminationGracePeriodSeconds` longer than the `preStop` delay plus the bound, with room for Hangar's own cleanup after it. The mcp-hangar Helm chart sets all three from its `shutdown` values, and refuses to render a grace period that is too short. ### the Python facade's `invoke` applies the configured controls `Hangar.invoke` and `SyncHangar.invoke` called the server directly, so none of the call-time controls in your configuration applied to them. They now run the call through the same executor as `hangar_call`: tool access and withdrawals, digest pins, validators and interceptors, approval, the global and per-server concurrency limits, and tenant budgets all apply. **Pass the caller.** `invoke` takes a new, optional `principal=`. The call is authorized for `tool:invoke` as an authenticated `hangar_call` caller is, by the roles your configuration gives that principal id and its groups. The principal's `tenant_id` is the tenant the per-tenant controls are applied for. ```python from mcp_hangar import Hangar from mcp_hangar.domain.value_objects import Principal, PrincipalId, PrincipalType caller = Principal( id=PrincipalId("agent-1"), type=PrincipalType.SERVICE_ACCOUNT, tenant_id="team-a", ) async with Hangar.from_config("config.yaml") as hangar: result = await hangar.invoke("math", "add", {"a": 1, "b": 2}, principal=caller) ``` `SyncHangar.invoke` takes the same `principal=`. Nothing verifies the principal: your application vouches for its id, groups and tenant, as an authenticator does for a request. `Principal.system()` is refused with `ValueError`, because authorization grants the system principal every permission. **Without a principal, the call is an anonymous caller's**, the same as an unauthenticated `hangar_call`: - With authentication configured, it is refused with the code `AuthorizationDenied`. - It carries no tenant. With `execution.tenant_limits` set, it shares the budget of callers with no tenant, built from the `"*"` entry, and is refused with `TenantQuotaExceeded` when there is no `"*"` entry. There is no way to make an unchecked call. If your configuration refuses anonymous callers, pass a principal. **What a call that does not succeed raises.** - A call a control refuses, or one that fails upstream, raises the new `ToolCallFailedError`. It is a `ToolInvocationError`, so an `except ToolInvocationError` still catches it. Its `code` is the `error_type` that `hangar_call` reports for the same call: for example `ToolAccessDeniedError`, `ToolWithdrawnError`, `ValidatorDenied`, `TenantQuotaExceeded` or `CircuitBreakerOpen`. Its message is the text `hangar_call` reports. - An upstream failure that `invoke` let through as its own exception type, such as `ToolTimeoutError` or `ClientError`, is now a `ToolCallFailedError` whose `code` names that type. - `McpServerNotFoundError`, `ToolNotFoundError` and `TimeoutError` are raised as before. - A server that fails to start used to raise `McpServerStartError`. It now raises `ToolCallFailedError` with `code == "McpServerStartError"`, so an `except McpServerStartError` no longer catches it. - The result is still returned whole. The per-call size limit (10 MB) and a `truncation:` section cut `hangar_call` results, not the results `invoke` returns, and no continuation is stored for an `invoke` call. **A tool that needs approval.** - `invoke` raises `TimeoutError` at `timeout_s`, and the event loop is not blocked while the call waits. - The call still holds one of the facade's pool threads until the approval is decided or expires (`approval_timeout_seconds`, 300 seconds by default). An approval given after `invoke` timed out is refused, so the tool does not run. - That pool also runs `stop()` and `health()`, and each pending approval takes one of its threads. Size it with `HangarConfig().max_concurrency(...)` for the approvals that can be pending at once. - `SyncHangar.invoke` blocks the calling thread for up to `timeout_s`. **Also:** - `invoke` accepts a group id, as `hangar_call` does, and the call goes to the member the group selects. - `timeout_s` still bounds the wait. The call itself is given `timeout_s` clamped to 1-300 seconds, as `hangar_call` clamps its `timeout`. - A facade call now writes the `hangar_call` span and log lines, and is counted in the batch metrics. - A call through `invoke` has no session and no request headers: session suspension does not apply to it, and an L7 rule that selects on `Mcp-Param-*` does not fire, as for `hangar_call` over stdio. ### a cluster refuses mode-less servers and local inline group members A configuration with a `coordination:` block is refused at startup when it declares a server that runs as a child process of one gateway, because only the replica holding the management lease can run it. The check read only top-level `mcp_servers` entries whose `mode` was written as `subprocess`, `docker` or `container`. It now also refuses: - a server with no `mode`. The loader builds it as `subprocess`, with or without an `endpoint`; - a group member whose id names no server under `mcp_servers`, when its own entry in the group's `members:` list defines a server with a local or missing `mode`. The error names it `/`. An entry that says neither is still refused by the loader as naming no server; - `mode: podman`. That configuration already failed to load, later, with `'podman' is not a valid McpServerMode`. To fix a refused configuration, give each server it names `mode: remote` and an `endpoint`. For a group member, do that in the member's entry, or declare the server under `mcp_servers` and name it by its id. If this is one gateway, remove the `coordination:` block. A configuration without `coordination:` is unaffected. ### docker discovery no longer waits for Docker With docker discovery enabled and Docker unreachable, `serve` and `Hangar.start()` no longer wait out the docker source's connection retries, and a stop no longer waits for a connection attempt in flight. **Nothing needs changing.** Two things read differently: - **The docker source's `is_healthy`** in `GET /discovery/sources` and `hangar_sources` reports whether discovery's last connection or scan reached Docker. It no longer connects and pings from the request. It is `false` until the first connection completes, and it follows a Docker that goes away or comes back at the next discovery cycle, not at the next listing. - **Each connection attempt times out sooner.** Its requests, the API version check and the ping, time out after 5 s instead of the Docker client's 60 s. A Docker that takes longer than that to answer is treated as unreachable. Calls after the connection keep the client's default timeout. ### the Python facade takes the management lease and warms a front door `Hangar.start()` and `SyncHangar.start()` now start two things that `serve` starts and the facade did not: - **Under a `coordination:` block**, the management lease keeper and the event tailer. An embedded gateway now takes and renews the lease, and follows the shared event log, as a served replica does. `stop()` stops the tailer, then shuts down, then releases the lease. - **In front-door mode** (`tool_access.mode: front_door`), the catalogue warm-up. Every configured server is started at `start()`, on a thread of its own, so `start()` does not wait for it. With `tool_access.required_catalogue` set, the required-catalogue retry runs after it. `stop()` stops the retry and waits up to 10 s for the warm-up to finish. In egress mode, and without a `coordination:` block, nothing changes. **What to check.** An embedded front door now starts every configured server when it starts, not on first use, as `serve` does. `stop()` can take up to 10 s longer while a warm-up is still starting servers. **Concurrent starts.** Concurrent `start()` calls now share one bootstrap: the later calls return once the first has finished. A `stop()` made while a `start()` is in flight waits for it, then stops everything it started. `SyncHangar.start()` and `SyncHangar.stop()` are serialised across threads in the same way. Before, two concurrent starts could both bootstrap, and the first context was never stopped. ### each server stop is counted once `mcp_hangar_mcp_server_stops_total` counted some stops twice. An idle reap added 2 under `reason="idle"`. A stop through `hangar_stop`, the REST stop or block, or the failover saga added 1 under its own reason and 1 more under `reason="shutdown"`. Each stop now adds 1, under the reason it was made for. **Stop rates drop after the upgrade**, with no change in how often servers stop: - `reason="idle"` halves. - `reason="shutdown"` no longer counts a stop made through the stop command. It counts the stops Hangar makes by itself: a reload, an unload or delete, a group's `stop_all`, process exit. - `user_request`, `manual`, `failback`, `compensation`, `detection_enforcement:block` and `max_retries_exceeded` count as before. - A sum over every reason drops by one for each stop that was counted twice. Review the alerts and recording rules on this counter, such as a `rate(mcp_hangar_mcp_server_stops_total[5m])` threshold or a ratio against `reason="shutdown"`. **`McpServerStopped` carries the stop command's reason.** A stop through `hangar_stop` is now recorded as `reason: user_request` instead of `shutdown`. A REST stop is recorded under the reason its body names when the counter lists that reason, and as `manual` otherwise; the response still repeats the reason as given. A failback is recorded as `failback`, a compensation as `compensation`, a block as `detection_enforcement:block`. The event store, the audit log and the event stream show these values. The alert handler still raises no alert for these stops, and the recovery saga still clears a server's retry state after them. ### each caller can have its own command-bus rate limit `rate_limit` keeps its meaning: `rps` and `burst` are the budget all callers share, one per command type. Without the new key below, it admits and refuses what it did before, except for the read-only tools. **Nothing needs changing.** What is new: - **`rate_limit.per_caller`** gives each caller a budget of its own, per command type, under the shared one. A caller is its tenant and its principal. Callers with neither are one caller and share one budget: anonymous callers, every caller when authentication is off, and work no request started. A caller that has used up its own budget is refused without spending the shared one, so the other callers keep theirs. Both keys are required, and a malformed `per_caller` stops startup. It is read at startup and counted per replica, as `rate_limit` is. Set it below the shared budget: at or above it, it never refuses a call the shared budget would not. - **The read-only tools are never refused** by the rate limit: `hangar_list`, `hangar_status`, `hangar_details`, `hangar_group_list`, `hangar_health`, `hangar_metrics`, `hangar_discovered` and `hangar_quarantine`. They read Hangar's own state and call nothing outside it. `hangar_tools` is still limited, because it may start a stopped server. So is `hangar_sources`, because it runs every discovery source's health check, and the Kubernetes source's check is a call to the cluster's API. - **A refusal reads the same on every path.** Its message names the code, the budget, the limit and when to retry, as in the example below. A `hangar_call` result has `error_type` `RateLimitExceeded`. The HTTP API answers `429` with a `Retry-After` header, and its `details` hold `retry_after` and `scope` (`caller` or `all_callers`). A client or an alert that matched the old text, `Rate limit exceeded: N requests per Ms`, needs the new one. ```yaml rate_limit: rps: 50 # all callers together, per command type burst: 100 per_caller: # each caller, per command type rps: 5 burst: 10 ``` ```text RateLimitExceeded: this caller's rate limit for InvokeToolCommand is used up (10 at once, refilled at 5 per second). Retry after 0.20s. ``` `per_caller` is a key of its own so that no deployment admits more than it did. Making `rate_limit` itself per caller would have multiplied what a gateway admits in total by its number of callers, without the operator asking for it. ### a failed start's degraded event and initialize log carry no upstream text - When an upstream process answers `initialize` with an error, Hangar logs one `mcp_server_initialize_refused` line with `mcp_server_id`, `exit_code` and `stderr_bytes`. The `mcp_server_process_exit_code: ` and `mcp_server_stderr: ` lines are gone. The stderr still reaches the caller, in the `McpServerStartError` the start raises. A log query or alert that matched the old lines needs updating. - `McpServerDegraded.reason` for a failed start is the error's type, such as `McpServerStartError` or `ConnectionError`, where it was the error's text. The event store, the audit log, the alert handler's `details.reason` and the security handler's `details.reason` carry the new value. A server that health checks degraded still reads `health_check_failures`. Events recorded by an earlier release keep the text they were written with. - A failed tool call's security record (`log_validation_failed`, field `tool`) has the error's type as its message, where it was `: `. ### a task's follow-ups follow its tool's current access A relayed task's follow-ups are checked against the tool access in force when they arrive, as a new call of the task's tool is: the tool policy, the withdrawals and the caller's tenant. **Nothing needs changing in configuration.** What a client may see when a tool is withdrawn, or the policy stops allowing it for the caller's tenant, while its task runs: - **`tasks/update` is refused** with `-32602`, the call's message, and `data.error_type` set to the call's refusal: `ToolWithdrawnError` or `ToolAccessDeniedError`. The upstream is not sent the input. - **`tasks/get` still answers with the task's status.** A poll that would hand over what the tool produced, a result, an error or input requests, is refused the same way. - **`tasks/cancel` is always served**, so a task whose tool was taken away can still be stopped. A tool that is still allowed is followed up as before. ### a member of several groups is governed by each of them on the front door On the front door, a server that is a member of several groups is now governed by every one of those groups, as `hangar_call` naming that server already was. Before, the front door kept one group per member, whichever group the file declared last. The other groups' `tools` policy, `tool_projection` withdrawals and pins, and `header_exposure` block did not apply to that member there. **A server in one group, or in none, is unaffected.** For a server in several groups: - **It may now be refused a tool it was served before.** A tool that any of its groups denies or withdraws, for every tenant or for the caller's, is no longer listed for it, and a call to it is refused. A call to a tool any of its groups pins is checked against that pin. This is the decision `hangar_call` already gave for the same config. - **A call is routed to the server itself**, not through one of its groups, so no group's `strategy` picks the member that answers. A server in one group is still routed through its group. - **Its tools are counted under its own id** in `mcp_hangar_projected_upstream_bytes`, not under a group's. To serve such a tool again, allow it in every group the server is a member of, or take the server out of the group that refuses it. ### a `hangar_*` tool call is charged to one rate-limit budget `rate_limit` and `rate_limit.per_caller` keep their keys and their meaning. **Nothing needs changing** in a configuration. What they admit changes in three ways, each toward the configured numbers: - **A server or group a call names no longer has a budget of its own.** The limited tools used to keep one per id, so starting four servers admitted four times the budget. Now a tool whose work is a command is charged at the command bus, once per command type: `hangar_start` and `hangar_stop` of a server, `hangar_tools`, `hangar_warm` and `hangar_reload_config`, whose budgets are `StartMcpServerCommand`, `StopMcpServerCommand` and `ReloadConfigurationCommand`. A tool whose work never reaches the bus is charged once to a budget named after it, whatever it names: `hangar_load`, `hangar_unload`, `hangar_approve`, `hangar_discover`, `hangar_sources`, `hangar_group_rebalance`, the continuation tools, and `hangar_start` and `hangar_stop` of a group. - **A call is charged once.** The limited tools were charged by the deprecated `check_rate_limit()` and again by the command bus. A `hangar_start` of a server now spends one token, not two, and a refusal names the command, as in `... rate limit for StartMcpServerCommand is used up ...`. A call that names no server spends nothing, since it does no work. - **Waiting no longer refills a bucket past its rate.** A bucket idle for the cleanup window (60 seconds) was dropped and came back full. It is now dropped only once it has refilled, so after a wait a caller gets what `rps` refilled in that time. `mcp_hangar.server.validation.check_rate_limit()` is removed. It was deprecated and nothing in Hangar calls it now. Code that called it can call `charge_tool()` from the same module, which charges the caller's budget and the shared one for that name. ### a front door relays tasks from a current-spec upstream A front door's flat `tools/call` now forwards the caller's `io.modelcontextprotocol/tasks` declaration to the upstream, as `hangar_call` already did. SEP-2663 leaves task creation to the upstream and gates it on the caller having declared the extension, so an upstream that follows the spec used to answer a front-door call with an ordinary tool result and never a task. Such a call can now come back as a task result (`resultType: "task"`), which the caller polls with `tasks/get`. - **What changes for a caller.** A client that declares the extension under `clientCapabilities.extensions` on each request may now receive a task where it previously received a tool result, from upstreams that offer one. A client that declares nothing sees no change. Declaring the extension has always been the opt-in; on a front door it now has effect. - **The same request context also carries two things it did not.** The caller's W3C trace context (`traceparent` / `tracestate` in `params._meta`) now parents the flat call's spans, so a front-door call joins the caller's trace instead of starting its own; and the request's `Mcp-Param-*` routing headers reach the L7 egress evaluator on this path, so a policy selecting on them can now fire for a flat call. - **A caller on an older revision is no longer spoken for.** Whether the caller declared the extension is read once per request, together with whether its protocol revision has `tasks/*` at all. A caller without `tasks/*` has no declaration forwarded on its behalf, since it could not poll the resulting task. ### a task no caller is handed is cancelled upstream When an upstream answers a call with a task that Hangar will not hand over -- the caller cannot poll one -- Hangar now sends that upstream a best-effort `tasks/cancel` for it. The caller still gets the same refusal it got before, and it gets it without waiting: the cancel is sent off the request path, with one attempt and a bounded timeout, and its outcome is logged (`cancelled`, `refused` or `failed`) rather than surfaced. Cancellation is cooperative under SEP-2663, so an upstream may decline. An upstream that does honour it will see `tasks/cancel` arrive for a task it just created, shortly after creating it, with no `tasks/get` in between. Before, that task was left to run until its own TTL with nobody able to collect the result. ### `error_type` names the failure in every MCP tool error payload A tool error payload used to name the failure under `type`, while a `hangar_call` result named it under `error_type`. Both now use **`error_type`**, so a client reads one key whichever tool answered. ```json {"error": "...", "error_type": "RateLimitExceeded", "details": {}} ``` **A client that matched `type` on a tool error payload has to read `error_type` instead.** The `error` and `details` keys are unchanged. This applies to every tool error, not only a rate-limit refusal. A `hangar_call` result already used `error_type` and does not change. A rate-limit refusal also had more than one shape, depending on which limiter refused the call: - The tool wrapper's own check (`hangar_load`, `hangar_fetch_continuation` and the other tools charged by name) raised out of the wrapper as an **MCP error**, so the call came back as a JSON-RPC error rather than a tool result. It is now the same error payload every other path returns. **A client that caught a JSON-RPC error to detect this refusal now sees a normal tool result whose `error_type` is `RateLimitExceeded`.** - A refusal raised inside a tool body -- by the command bus, or by `charge_tool` for work a tool does itself -- was already a payload, and keeps being one. `details` now carries the refusal's own fields on every path, where before a tool payload dropped them and left only the message text: ```json { "error": "RateLimitExceeded: this caller's rate limit for hangar_load is used up (2 at once, refilled at 5 per second). Retry after 0.20s.", "error_type": "RateLimitExceeded", "details": { "limit": 2, "window_seconds": 1, "retry_after": 0.2, "scope": "caller", "key": "hangar_load", "rps": 5 } } ``` `scope` is `caller` when the caller's own budget was used up and `all_callers` when the shared one was, as it already was over the HTTP API. Only a rate-limit refusal carries `details`; every other tool error still has `details: {}`. **Security log.** A refusal by the command bus's limiter now reaches the security handler, which recorded only the tool-level ones before. Each refusal is recorded once, as `rate_limit_exceeded`, and its details gained `scope`, the `key_kind` (`tool` or `command`) and the `key` it names. These are values Hangar chose: no argument value and no caller's own text is recorded. A refusal is no longer also recorded as a `validation_failed` event. ### an L7 egress policy set over the API survives a reload An L7 egress policy is set at runtime, through `POST /api/mcp_servers/{id}/l7_policy` or the fleet projection, and no configuration file declares one. A reload that rebuilt a server -- because a setting it is built from, such as `env`, changed -- dropped that policy, and reported success. A rebuild now carries the running server's policy onto its replacement, and logs `l7_policy_carried_to_rebuilt_mcp_server` when it does. A deployment that relied on a reload clearing a policy must now clear it explicitly, with `DELETE /api/mcp_servers/{id}/l7_policy`. ### a call naming a group waits for the approver its member's L7 policy asks for A call that names a group is routed to one of the group's members. When that member's L7 egress policy (`MCPEgressPolicy`) is in `Enforce` mode and routes the tool to `requireApproval`, the call is now held and an approval is raised, exactly as for a call that names the member directly. Before, the approval gate looked the L7 policy up by the id the call named. A group id is not a server id, so a call naming a group found no policy and nobody was asked; the member's own check then refused the call at invoke with `EgressPolicyApprovalRequiredError`. **A call that was refused immediately may now wait.** With an approval gate configured, such a call blocks until someone decides it, or until the timeout: - approved -- the call runs, and its result comes back as usual; - denied -- it is refused with `approval_denied`; - nobody answers -- it is refused with `approval_timeout` after `approval_timeout_seconds`, which is 300 seconds when the rule comes from the L7 policy alone and no tool-access policy narrows it. If a caller of yours relied on the immediate refusal, give it a timeout, or change the rule from `requireApproval` to `deny` where you meant "refuse" rather than "ask". Check as well that the configured channel reaches a person (`approvals.channel`): on `noop` the hold is real and nobody is told, so the call runs out its timeout. **Nothing changes without an approval gate.** With approvals disabled, or with no gate wired, the call is still refused at invoke with `EgressPolicyApprovalRequiredError`, as before. The member's own invoke-time L7 check is untouched and still refuses first, so `deny` still wins and `Audit` mode still asks nobody. A call that names the member directly behaves as it did. ### every error payload a tool answers with is `error`, `error_type`, `details` #### a failed `hangar_reload_config` answers the tool error payload A reload that failed used to answer a layout of its own: ```json { "status": "failed", "message": "Configuration reload failed: ...", "error_type": "ConfigurationError" } ``` It now answers the payload every other tool failure uses: ```json { "error": "Configuration reload failed: ...", "error_type": "ConfigurationError", "details": {} } ``` **A client that detected a failed reload by `status == "failed"` reads `error` (or `error_type`) instead**, and takes the message text from `error` rather than `message`. The text itself is unchanged, and `error_type` still names the exception class. A successful reload is untouched: it still answers `status: "success"`, `message`, the four `mcp_servers_*` lists and `duration_ms`. #### `MCPError.to_dict()` is removed `mcp_hangar.MCPError` -- and every domain exception under it -- carried a `to_dict()` that rendered: ```json { "error": "...", "mcp_server_id": "...", "operation": "...", "details": {}, "type": "ConfigurationError" } ``` Nothing in Hangar called it. The REST API builds its own envelope (`{"error": {"code", "message", "details"}}`) from the exception's attributes, and an MCP tool error is the three-key payload above, so this was a second serializer that named the failure under `type` where the answering surfaces named it `code` and `error_type`. **An embedder that called `exc.to_dict()` builds the dict itself:** ```python { "error": exc.message, "mcp_server_id": exc.mcp_server_id, "operation": exc.operation, "details": exc.details, "error_type": type(exc).__name__, } ``` The attributes it read -- `message`, `mcp_server_id`, `operation`, `details` -- are unchanged, as is every exception class and what raises it. ## Upgrade to 2.20.0 ### `block` and `quarantine` stop a server whose tools drift A server with `capabilities.enforcement_mode` set to `block` or `quarantine` whose upstream serves a tool that is not in `capabilities.tools.expected_tools` now serves nothing. Before, block mode detected the drift and marked the server `dead`, but the call that started it still ran the tool it asked for, and the upstream process was left running. Quarantine did not act on the drift at all and served the server as `alert` does. What happens now, in both modes: - The start that finds the drift fails with `CapabilityBlockedError`. Hangar closes its connection, records `CapabilityViolationDetected`, and moves the server to `dead` for a capability block (`capability_blocked`). It records no `McpServerStarted`. Quarantine also records `McpServerCapabilityQuarantined`. - No call starts the server again: every later call to any of its tools, declared ones included, is refused with `CannotStartMcpServerError`. The recovery saga does not retry it, and a group does not put it in rotation. - A tool that appears after a clean start, through a refresh or `tools/list_changed`, blocks the server in the same way at its next call. Hangar closes the connection at once, so a call already in flight on it fails. - `alert` mode is unchanged. If you set `quarantine` expecting it to keep serving, as it did, it no longer does: use `alert` for that. The error does not name the undeclared tools, because the caller it reaches is not the operator who has to act on them. The `capability_drift_detected` warning and the `CapabilityViolationDetected` event name them. **To bring a blocked server back**, fix the upstream or add the tool to `expected_tools`, then start the server deliberately: `hangar_start`, `hangar_warm` naming it, or a start through the REST API. That start checks the tools again, and fails the same way while they still drift. A stop leaves the server `cold`, and the next start, a call's included, checks them too. **Prompts, resources and task relays** to a server now need it to be `ready`, the same as a tool call. Before, they were forwarded to any server with a live connection, a `degraded` one included. ### egress calls are governed with their group and tenant scope A `hangar_call` that names a group member by its own server id, instead of naming the group, is now governed by that group. Before, it was governed as if the member were a standalone server. For each group that owns the member, a call naming the member is now checked the way a call naming that group and routed to that member is checked: | Declared on the group | Before | Now | | --- | --- | --- | | `tools:` (allow, deny) | not applied | applied | | `members[].tools:` for this member | not applied | applied | | `tool_projection.withdrawn`, per tenant or for all, and a runtime withdrawal of the group id | not applied | applied | | `tool_projection.pins`, in the group's `digest_enforcement` mode | not applied | applied | The call is still sent to the member it names. The group's member selection does not run. Everything the member's own server id declares applies as it did before. A server that is a member of more than one group is governed by all of them, and deny wins. A tool that any one of them denies, withdraws, or pins to a digest the tool does not match is refused. **Approval lists declared on groups or tenants now take effect.** The approval gate now reads approval lists with the same scope as the access policy. Before, it read only the `approval_list` of the server a call named, and asked without the caller's tenant. | Approval list | Before | Now | | --- | --- | --- | | a server's own `tools.approval_list`, egress | applied | applied | | a group's `tools.approval_list`, or `members[].tools.approval_list` | not applied | applied, on a call naming the group or the member | | `tool_access.member..approval_list` | not applied | applied to that tenant | | any approval list, `front_door` | not applied | applied | A tool on any list that applies needs approval. The first list that applies supplies the timeout and channel. Approval routed by an L7 egress policy's `requireApproval` is unchanged. **Withdrawal is re-checked after an approval hold.** A tool withdrawn while a call waits for approval, on the server, on an owning group, for every tenant or for the caller's, is now refused with `ToolWithdrawnError` once the call is approved. Before, it ran. **What to check.** - A caller that relied on naming a member to reach a tool its group denies, withdraws or pins now gets `ToolAccessDeniedError`, `ToolWithdrawnError` or `ToolDigestMismatchError`, as a call naming the group does. If that caller should reach the tool, allow it on the group, or move the server out of the group. - If you declared an `approval_list` on a group, on a group member, or for a tenant, or you run `front_door` with approval lists, the tools on those lists now wait for approval. With no approval channel configured, a held call times out after `approval_timeout_seconds`. Remove a list you do not want enforced. - On `front_door`, a call has a fixed 30 s budget, so a held front-door call must be approved within 30 s. An approval that arrives later is refused, and the call does not run. Calls that name a group are governed as before, apart from their approval lists. So are servers that are in no group. ### a continuation answers only the caller that made the call With response truncation on (`truncation.enabled`), the rest of a truncated `hangar_call` result is kept in the continuation cache. `hangar_fetch_continuation` and `hangar_delete_continuation` now serve it only to the caller whose `hangar_call` produced it: the same tenant and the same principal. Anyone else who presents the id gets the answer for an id that does not exist. | Tool | Answer to any caller but the one that made the call | | --- | --- | | `hangar_fetch_continuation` | `{"found": false, "error": "Continuation not found (may have expired)"}` | | `hangar_delete_continuation` | `{"deleted": false, "continuation_id": ""}`, and nothing is deleted | **What may need a change.** - A client that fetched a continuation as a different principal from the one that made the call now gets not found, even within the same tenant. Fetch it with the credentials that made the call. - A caller with no tenant does not match a continuation stored with one, and the reverse. - The `result_truncated` log line no longer carries `continuation_id`. It has `batch_id`, `call_index` and `continuation_advertised`. The cache and continuation-tool log lines carry `continuation_ref` in place of `continuation_id`: the id without its random suffix. - A fetch or delete of another caller's continuation logs a `continuation_owner_mismatch` warning naming the owner's and the caller's tenant. **With auth off**, neither the caller that makes the call nor the caller that fetches has an identity. Both are anonymous, and continuations work as before. Nothing authenticates a caller there, so this is not a boundary. With the memory cache, a continuation is still fetchable only on the replica that truncated the result. **Upgrading with the Redis cache.** Values are now stored with their owner. A value written before the upgrade has no owner and is read as anonymous: with auth off it is still served, and with auth on it is not found. Either way it expires within `cache_ttl_s`. Until every replica runs this version, a replica on an older version still serves any continuation to any caller that holds its id, and returns a value written by an upgraded replica with the owner in front of the payload. ### `tool_access.rules` is refused as a key nothing reads `tool_access.rules` was never read. The config schema listed it next to `tool_access.mode`, so a `rules:` block passed `mcp-hangar config check` and `HANGAR_CONFIG_STRICT=1`, loaded without a warning, and restricted nothing. No document or example described it (#1422). Delete it: ```yaml tool_access: mode: front_door rules: [] # delete this key, and anything nested under it ``` A config that still sets it loads, and logs `unknown_config_key` saying the key was never read. `HANGAR_CONFIG_STRICT=1` and `mcp-hangar config check` refuse it, as they refuse any key nothing reads: under strict mode a gateway whose config still sets it does not start, so delete the key before upgrading. This applies to a file and to `bootstrap(config_dict=...)` alike. Nothing that restricts a tool changes. Tool access is set by the `tools:` allow and deny lists of a server, a group and a group member, and `tool_access.mode` still selects the `egress` or `front_door` topology. ### a reload applies the whole configuration, and keeps the topology mode This affects every configuration reload: `POST /api/config/reload`, `hangar_reload_config`, SIGHUP, and the config file watcher. A reload used to reset `tool_access.mode` to `egress` and apply only `mcp_servers`. A reload now applies every section startup applies from the configuration: - `mcp_servers`, as before - `interceptors.validators` - `ui_resources`, the `ui://` allow list - `headers.param_validation` - `resource_links` - `execution`, the concurrency limits A section you delete from the file goes back to its default when you reload. Before, it stayed in force until the next restart. Every section is checked before any server is stopped. A bad value refuses the reload, and nothing changes. Sections that startup reads only once, such as `auth`, `persistence`, `event_store`, `discovery` and `logging`, still need a restart, as before. ### A reload that changes `tool_access.mode` is refused A reload keeps the mode the gateway started with. If the file sets a different `tool_access.mode`, the reload is refused and nothing changes, because the front-door tool surface is built at startup. Restart the gateway to change the mode. | Trigger | What you see when the mode changed | | --- | --- | | `POST /api/config/reload` | HTTP 409, `ConfigurationRestartRequiredError` | | `hangar_reload_config` | `status: failed`, with the same message | | SIGHUP, file watcher | `configuration_reload_failed` in the log | The workaround for a `front_door` gateway, `config_reload.enabled: false` and a restart for every change, is no longer needed. ### Policies set at runtime Before, a reload removed every tool-access policy set at runtime. Now: - The policies stored by the REST policy endpoint are replayed after the file, as a restart does. On a scope that both the file and the REST endpoint define, the stored policy applies, after a reload as after a restart. If the policy store cannot be read, the reload is refused and nothing changes; `POST /api/config/reload` answers HTTP 503. - Any other policy set at runtime, such as one `hangar_load` set, is kept unless the file now defines the same scope. The file's policy then replaces it. - A server that the reload removes takes its policies with it, as `hangar_unload` does. A policy the REST endpoint stored for it comes back at the next restart, as before. A group's inline members count as declared by the file. A reload no longer stops them as removed servers or strips the policies set on them at runtime. The groups, and the policies, withdrawals, pins and `header_exposure` blocks, are replaced rather than cleared and registered again, so a call made during a reload never finds them empty. A reload builds and checks every server and group before it stops any, so a bad block refuses the reload and changes nothing. ### a config dict gets every setting it passes This affects code that calls `bootstrap(config_dict=...)` directly, such as embedders and test harnesses. `Hangar.from_config()` and `mcp-hangar serve` read a file and are unchanged. `Hangar.from_builder()` passes a dict and is covered at the end of this section. A dict is now applied the same way as the same document in a file. Before, the dict path dropped these settings without logging anything: - `tool_access.mode`, so a dict that asked for `front_door` came up in `egress` - `interceptors.validators`, so no parameter validator ran - `ui_resources`, the `ui://` allow list - `headers.param_validation` - `resource_links` - `execution`, the concurrency limits A dict now gets all of them. If a harness passed one of these and relied on it being ignored, remove it from the dict. The schema check now runs on a dict too. An unknown or removed key logs `unknown_config_key`, and under `HANGAR_CONFIG_STRICT=1` the boot refuses, as it does for a file. Four more cases used to be accepted without a word: | A dict that | Before | Now | | --- | --- | --- | | is passed while `MCP_CONFIG` or `./config.yaml` exists | was laid over that file: the file's topology, validators and `ui://` allow list applied, and the dict replaced the file's other sections | is the whole configuration, and no file is read | | has no `mcp_servers` section | booted the built-in example server | is refused, as a file is, unless `discovery.enabled` is true | | enables `config_reload` | built a reload watcher with no file, which did nothing | is refused: set `config_reload.enabled: false`, or pass a file | | is passed together with `config_path` | ran the dict, while reload watched the file | is refused | Relative paths in a dict resolve against the working directory, as they do in a file. `Hangar.from_builder()` no longer passes its own `max_concurrency` to the gateway, which never read it. It still sizes the facade's thread pool. A builder that calls `enable_discovery()`, or adds a server with `mode="remote"` and `url=...`, produces keys the gateway does not read. Those settings were never applied. They now log `unknown_config_key`, and under strict mode the boot refuses. ### remote servers and discovery from the builder take effect This affects code that builds its configuration with `HangarConfig` and runs it with `Hangar.from_builder()`. Code that calls `Hangar.from_config()` on a file that enables discovery is affected by the `Hangar.start()` change below. Two builder features wrote keys the gateway does not read, so neither was ever applied. Both now take effect: - `add_mcp_server(..., mode="remote", url=...)` writes the address as `endpoint`, the key the gateway reads. A remote server now boots with its address and answers calls. The argument is still called `url=`. - `enable_discovery(...)` writes `discovery: {enabled: true, sources: [...]}`, one `additive` source per requested type. Additive sources add the servers they find and never remove one. `Hangar.start()` now runs discovery when the configuration enables it, and `Hangar.stop()` stops it. `bootstrap()` builds the discovery sources and starts nothing, and only `mcp-hangar serve` used to start them. So under the facade, a `discovery` section built its sources and never ran them, whether it came from the builder or from a file. If you enabled discovery and relied on it doing nothing, remove the call or the section: discovery now registers the servers its sources report. The builder now raises `ConfigurationError` on these calls. Each used to be stored and never applied: | Call | Why | | --- | --- | | `enable_discovery(filesystem=[a, b])` | The gateway keeps one source per type, so the second directory replaced the first. Pass one directory. | | `enable_discovery()` with no source | It enabled nothing. | | `add_mcp_server(..., mode="group")` | The builder cannot declare a group's members. Declare the group in a config file. | | `add_mcp_server(..., mode="container")` without `image=` | The launcher refused it only when the server started. Pass the image. | | An option the mode does not read, such as `url=` on a subprocess server, or `env=` or `command=` on a remote one | The gateway ignored it. Remove the option. | | `set_intervals(...)` | No configuration key sets the GC or health-check interval, so the value was never applied. Remove the call. | `build()` now checks the configuration against the gateway's schema, and raises on a key the gateway does not read. `to_dict()` no longer includes `max_concurrency`, which sizes the facade's thread pool and is not a gateway setting. `HangarConfigData` no longer has `gc_interval_s` or `health_check_interval_s`. ### a server Hangar gives up on reads `dead`, not `cold` When the recovery saga runs out of retries, the server now goes to `dead`. Before, giving up was a stop, so the server went to `cold`: the state of a server nobody has called yet. Two other failures already reached `dead` but published no state change, so the gauges kept their last values. A crashed process read `ready` and `up` 1 until something called it, and a start that failed below `max_consecutive_failures` read `initializing`. All three now read `dead`. This affects anything that reads `mcp_hangar_mcp_server_state`, `mcp_hangar_mcp_server_up` or `mcp_hangar_mcp_server_initialized`, and anything that reads a server's `state` from `hangar_list`, `hangar_status` or `GET /api/mcp_servers`. | Server | `state` before | now | `up` before | now | `initialized` before | now | | --- | --- | --- | --- | --- | --- | --- | | the recovery saga gave up on it | `0` (cold) | `4` (dead) | `0` | `0` | `0` | `1` | | its process crashed | `2` (ready) | `4` | `1` | `0` | `1` | `1` | | its start failed, below `max_consecutive_failures` | `1` (initializing) | `4` | `0` | `0` | `1` | `1` | | never started, stopped, or reaped for being idle | `0` | `0` | `0` | `0` | `0` | `0` | **An alert on `mcp_hangar_mcp_server_state == 0` no longer fires for a server Hangar gave up on.** `0` now means only that the server is not running and is not failing. Use one of these instead: - `mcp_hangar_mcp_server_state == 4` fires when a server is dead. - The new gauge `mcp_hangar_mcp_server_last_healthy_timestamp_seconds` holds when Hangar last saw the server working: a passing health check, a completed start or a successful tool call. It is kept when the server goes cold or dead. ```promql time() - mcp_hangar_mcp_server_last_healthy_timestamp_seconds > 900 unless mcp_hangar_mcp_server_state == 0 ``` Keep the `unless`, and keep its default matching. A cold server is not probed, so without the `unless` the rule also fires for every server reaped for being idle more than 15 minutes ago. `unless on(mcp_server)` would drop the `instance` label, so with more than one replica a server that is cold on one replica would hide it being dead on another. A server that was never healthy has no series, so pair the rule with `state == 4`. **`sum(mcp_hangar_mcp_server_up) == 0` can newly fire.** A crashed server used to keep reading `up` 1; it now reads 0. A pool whose only servers reading `up` had crashed now reads 0. ### What starts a dead server again Why it died decides. A call is refused while the server's backoff lasts and is told how long to wait: through `hangar_call`, `CircuitBreakerOpen` with the time to retry. Once the backoff has passed, the call starts it. | Why it died | A call through a group | A call naming it | A deliberate start | | --- | --- | --- | --- | | the recovery saga gave up on it | never | yes, after its backoff | yes | | a capability block stopped it | never | never | yes | | its process crashed, or its start failed | yes, after its backoff | yes, after its backoff | yes | The deliberate starts are `hangar_start` on the server or on its group, `POST /api/mcp_servers/{id}/start`, `hangar_warm` naming the server, a group adding the server with auto-start, and a failover saga starting the backup it was configured with. Nothing else starts a dead server: - The health worker does not check it. It used to, every 60s, and counted `mcp_hangar_health_checks_total{result="unhealthy"}` for a check that sent nothing, so that series stops moving while a server is dead. - The recovery saga cancels the restarts it has scheduled when it gives up, and when the server starts or stops. - The bulk warm-ups skip it: the front door's at boot, and `hangar_warm` with no names, which now lists it under `skipped_dead`. - `hangar_tools` lists a dead server, or a group's dead member, without starting it. - The GC acts only on servers that are `ready`. A group does not count a dead member as healthy. A member Hangar gave up on, or one a capability block stopped, leaves rotation, and a successful start puts it back, subject to the group's `healthy_threshold`. A member whose process crashed stays in rotation, so the next call through the group restarts it, as it always did. When that restart fails, or the call is refused inside the member's backoff, the call counts as the member's failure, so a member whose restart keeps failing leaves rotation and the group fails over. ### Other changes - A give-up no longer also counts `mcp_hangar_mcp_server_stops_total{reason="shutdown"}`. It still counts `reason="max_retries_exceeded"`. - `hangar_stop` on a dead server makes it `cold`. - A server that is deleted, unloaded or reloaded away loses its lifecycle gauges, so a removed dead server stops reading `4`. A deletion removes them on every replica. Its counters stay. - With a durable event store, a server restored `dead` or `degraded` reads so from boot. A server restored `ready` has no connection in the new process: it has no series until its first health check or call, which make it `cold`, not `dead`. ### a group's `circuit_breaker.reset_timeout_s` is removed It never did anything. An open group circuit did not half-open once the timeout passed, however long it waited: a breaker half-opens only when asked whether to let a request through, and a group never asks. The circuit closes once `min_healthy` members are back in rotation, after a passing health check or a successful call, and that is unchanged. A timed probe would have been a second way out, competing with that one, so the option was removed rather than honoured (#1398). Delete it from every group: ```yaml mcp_servers: pool: mode: group circuit_breaker: failure_threshold: 10 reset_timeout_s: 60 # delete this line ``` A config that still sets it loads, and logs `unknown_config_key` naming the group and the key. `HANGAR_CONFIG_STRICT=1` and `mcp-hangar config check` refuse it, as they refuse any key nothing reads: under strict mode a gateway whose config still sets it does not start, so delete the key before upgrading. The flat spelling `circuit_reset_timeout_s` is reported the same way. In Python, `McpServerGroup(...)` no longer accepts `circuit_reset_timeout_s`: passing it raises `TypeError`. ### a group's `healthy_count` counts members that are `ready` A group's `healthy_count` used to count every member in rotation that was not `dead`, `cold` ones included. It now counts the members that are `ready` and in rotation. The number of members in rotation, in any state, is a new field, `members_in_rotation_count`: the length of the `members_in_rotation` list that `hangar_group_rebalance` returns. `healthy_count` changes meaning, so its value drops for any group with a member in rotation that is not `ready`. The common case is a group whose members the GC reaped for being idle. Before, it read `healthy_count: 2` with nothing running; it now reads `healthy_count: 0` and `members_in_rotation_count: 2` until a call through the group starts a member. This affects anything that reads a group's `healthy_count` from `GET /api/groups`, `GET /api/groups/{id}`, `hangar_details`, `hangar_group_list`, `hangar_list`, `hangar_start` or `hangar_group_rebalance`, or its `healthy_members` from `hangar_status`, `hangar_health` or `hangar_metrics`. The `GroupStateChanged` event's `healthy_count` changes the same way and gains `members_in_rotation_count`. No metric reports either count. **A check that treats `healthy_count: 0` as a group that cannot serve now fires for idle groups that serve fine.** A group routes as long as `is_available` is true. To ask whether a group can take a call, read `is_available`. To ask whether members are in rotation, read `members_in_rotation_count`. What the group decides is unchanged. `is_available`, the group `state` and the `min_healthy` rule that closes an open circuit count the members in rotation that are not `dead`, as `healthy_count` did before. ## Upgrade to 2.19.1 ### a suspended session is refused `POST /api/sessions/{id}/suspend` now blocks the session it names. Before, it answered 200 and replicated the suspension to every replica, and no request path read it, so the session went on calling tools. A request that carries a suspended session id is refused before Hangar does anything for it: no validation, authorization, approval, cold start or upstream call. A replica refuses the session once it has read the suspension from the shared event log. `DELETE /api/sessions/{id}/suspend` lifts it. Each refusal logs a `session_suspended_call_refused` warning naming the session. | Request | Refused with | | --- | --- | | `hangar_call`, every other `hangar_*` tool, a front door's flat tool call | a tool error: `Session suspended: this call was refused.` | | `tasks/get`, `tasks/cancel`, `tasks/update` | JSON-RPC error `-32600`, the same message, `data.reason: session_suspended` | | on a front door: `prompts/list`, `prompts/get`, `completion/complete`, `resources/list`, `resources/templates/list`, `resources/read`, `subscriptions/listen` | the same JSON-RPC error | **Where a caller's session id comes from.** A suspension refuses only a caller that carries the session id it names. Hangar reads it from one of two places, in this order: | Source | Honoured when | | --- | --- | | the session-id claim of an OIDC bearer token (`sid` by default) | the token is verified. A header cannot replace it. | | `x-session-id` request header | the connecting peer is listed in `MCP_TRUSTED_PROXIES` | If your identity provider puts the session id in another claim, name it with `auth.oidc.session_id_claim`, or per issuer with `session_id_claim` on an `auth.oidc.issuers` entry. A session id must match `[A-Za-z0-9_-]{1,128}`, the shape the suspend route accepts. Anything else is ignored. `Mcp-Session-Id` is not read. To suspend API-key callers by session, put a proxy in front of Hangar that sets `x-session-id` and removes any the client sent. List the proxy's address in `MCP_TRUSTED_PROXIES`, which defaults to `127.0.0.1,::1`. **What is not covered.** - A caller with no session id carries nothing to match, and is not refused. Suspension does not cut off a principal. To do that, revoke its API key or disable it at the identity provider. - A stdio session has no session id: a pipe carries neither source. - With auth disabled, nothing is refused. The suspend route is then open to every caller, so it was never a control there. - `tools/list`, the REST API and `/ws/events` do not check suspensions. The tool listing answers from Hangar's own catalogue and reaches no upstream. - A replica that starts after a suspension does not learn of it. - A suspension expires after 24 hours. Each replica holds at most 10,000. ### only `MCP_TRUSTED_PROXIES` decides the forwarded client address Hangar now starts its HTTP server with uvicorn's forwarded-header handling turned off. Before, uvicorn rewrote the client address from `X-Forwarded-For`, and the scheme from `X-Forwarded-Proto`, for any peer in `FORWARDED_ALLOW_IPS` (default `127.0.0.1`), before Hangar saw the request. Hangar then applied its own `MCP_TRUSTED_PROXIES` to what was left. Now Hangar sees the address that actually connected. It believes `X-Forwarded-For` only when that address is in `MCP_TRUSTED_PROXIES` (default `127.0.0.1,::1`). The client address is the rightmost forwarded entry that is not itself a trusted proxy, which is the rule uvicorn applied. Before, behind a proxy outside `FORWARDED_ALLOW_IPS`, Hangar took the leftmost entry, which the client can write. | Connection | Address that rate limiting, audit and security events see | | --- | --- | | directly from a client | the client's address | | from a trusted proxy, with `X-Forwarded-For` | the rightmost forwarded address that is not a trusted proxy | | from any other peer, with `X-Forwarded-For` | the peer's address. The header is ignored. | What to do: - If you set `FORWARDED_ALLOW_IPS` for Hangar, put the same addresses in `MCP_TRUSTED_PROXIES`. Hangar no longer reads `FORWARDED_ALLOW_IPS`. - If requests pass through more than one proxy, list every proxy in `MCP_TRUSTED_PROXIES`. An unlisted hop is taken to be the client. - A proxy on loopback needs nothing, because loopback is trusted by default. Its `x-session-id` is now honoured even when it also sends `X-Forwarded-For`. Before, the rewritten address made Hangar ignore it. - The protected-resource metadata already read `X-Forwarded-Proto` itself, so the scheme it advertises is unchanged. ### traces, the security log and Langfuse carry no error text Hangar's telemetry no longer says what a failure said: a tool's error message can hold whatever the tool returned. A failed span ends in ERROR with an empty status description and a bounded `error.type`. Where an exception escaped the span, or a fault barrier handled one, the span has an `exception` event whose only attribute is `exception.type`, with no `exception.message` and no `exception.stacktrace`. | Where | Before | Now | | --- | --- | --- | | `batch.call.` status description | the call's error message | empty; `error.type` is the error's class, e.g. `ToolInvocationError` | | any other Hangar span an exception escapes | `exception` event with type, message and stacktrace; description `": "` | `exception` event with `exception.type` only; empty description; `error.type` is the exception's class | | a failure a fault barrier handled (event store append, discovery, cold start) | `exception` event with type, message and stacktrace | `exception` event with `exception.type` only; `error.type` as before | | security log, failed tool call | `details.error`: the message | `details.error_type` | | security log, repeated health-check failures | `details.error`: the message | no error field; `details.consecutive_failures` as before | | `health_check_failed` warning log | `error=` | `error_type=` | | Langfuse, failed tool call | output `{"error": , "type": }`, status message `Tool invocation failed: `, `tool_success` score comment `` | output `{"type": }`, status message `Tool invocation failed: `, score comment `` | | `error_type` of an upstream JSON-RPC error whose `code` is not an integer | the value sent, or `unknown` when absent | `_OTHER` | An error type that does not look like a class name or a code is recorded as `_OTHER`. Langfuse still receives tool arguments and results. If a dashboard or alert matched on span status descriptions, on `exception.message` or `exception.stacktrace`, on the security log's `details.error`, or on the Langfuse error output, match on `error.type` or `exception.type` instead. For the message itself, look up the call's `ToolInvocationFailed` event in the event store, or the result the caller received. Neither has changed. ### `scrub_baggage_for_tenant` is removed, and baggage is not forwarded `scrub_baggage_for_tenant` is removed from `mcp_hangar.observability`, with no replacement. If you call it, delete the call. There is nothing to do instead: Hangar forwards no baggage. Hangar propagates `traceparent` and `tracestate` only, over HTTP and stdio alike, and ignores `baggage` on inbound requests. Hangar sets no baggage itself, so this changes something only where host instrumentation or an embedding application attached baggage to the context, or a request carried a `baggage` key in `_meta`. Those entries no longer reach upstream servers. ## Upgrade to 2.19.0 ### tenant-scoped role grants are limited to their tenant A role bound at `tenant:` is now a grant within that tenant only. That covers an `auth.role_assignments` entry with `scope: "tenant:"`, `assign-role --scope`, and `POST /api/auth/roles/assign` with a `scope`. Before, such a grant authorized every REST route, the `/ws/events` stream and the `hangar_*` management tools as if it were global. | Surface | With a tenant-scoped grant | | --- | --- | | `/ws/events` | only events that name the tenant | | `GET /api/mcp_servers/{id}/tools/history` | only that tenant's invocations | | `POST /api/admin/tools/{server}/{tool}/withdraw` and `restore` | that tenant only; no `tenant_id` means that tenant; an all-tenant withdrawal cannot be lifted | | `/api/approvals` (list, get, resolve) | only approvals that name the tenant; approvals naming no tenant are withheld | | every other route | 403, reason `tenant-scoped grant; route requires a global grant` | | `hangar_*` management tools | refused, and not listed on a front door | `hangar_call` and the continuation tools are unchanged. If a principal needs fleet-wide access, bind its role at `global` scope. That includes listing, starting and stopping servers, and reading configuration. It also includes managing groups, discovery or credentials. ```yaml auth: role_assignments: - principal: "group:platform-engineering" role: developer scope: global # was: tenant:platform ``` Global grants, and deployments with auth disabled, behave exactly as before. ## Upgrade to 2.18.0 ### adopting the enforcement `init` now writes Nothing is required. An existing configuration is served exactly as it was in 2.17.1: no identity is declared, so a stdio caller stays anonymous, and no pins are written, so no tool is checked against one. What follows is how to adopt what the new `init` writes, on a configuration you already have. ### Pin what your servers serve ```bash mcp-hangar pin --check # what does the file claim, and what do the servers serve? mcp-hangar pin --write # adopt what they serve now ``` `--write` rewrites the file through PyYAML, which keeps values but **not comments or key order**, and leaves the previous file beside it as `.bak`. Bare `mcp-hangar pin` prints the same digests if you would rather paste them into a file you maintain by hand. `digest_enforcement` already defaults to `block`, so a pinned tool whose schema or description changes is refused from the moment you write the pin. That is the point, and it is also the one thing to know before writing pins on a busy gateway: an upstream that changes a tool stops serving it through Hangar until you re-pin. ### declaring who calls you over stdio `tool_access.mode: front_door` serves the upstreams' own tool names instead of Hangar's `hangar_*` API. It is fail-closed on identity, and identity arrives through HTTP middleware that a stdio process never enters -- so over stdio it needs a declared caller, or it serves an empty list: ```yaml tool_access: mode: front_door auth: stdio: principal: id: local-user tenant_id: local roles: [viewer] # read-only; `[]` projects no hangar_* tools at all ``` Nothing is checked here and nothing can be: a stdio server is not listening on anything, so the process that spawned it is the trust boundary ([ADR-026](https://mcp-hangar.io/docs/adr/ADR-026-stdio-is-an-authenticated-transport)). Over HTTP the block is ignored and the credential channel is unchanged. Per-tenant pins for the declared `tenant_id` become matchable, so a configuration that previously refused to boot with `PinnedToolsNeedAnIdentityError` for that tenant now starts. ### the CLI stopped printing `mcp_server` The CLI no longer prints `mcp_server` at people: help text says "MCP server", and `init --mcp_servers` is now `init --servers`. The old flag still parses, so scripts keep working. If you assert on `--help` output, it changed. ### a config generated by an older `init` Two more things moved. `init` no longer writes a `health_check:` block -- nothing has ever read it, so every generated config logged `unknown_config_key` and tuning `interval_s` tuned nothing. And the servers it configures now use their real npm packages where no official PyPI distribution exists; if you have a config from an earlier `init` with `command: [uvx, "mcp-server-filesystem"]` or `mcp-server-memory`, those never resolved -- replace them with `[npx, -y, "@modelcontextprotocol/server-filesystem"]` and `[npx, -y, "@modelcontextprotocol/server-memory"]`. ## Upgrade to 2.14.0 ### A front-door tool can disappear, with no config change (#1056) SEP-2243 makes it a client-side **MUST**: a client drops any tool whose `x-mcp-header` annotations are invalid. Hangar forwarded the upstream definition verbatim, so it advertised such a tool and counted it as surface delivered while every conforming client silently dropped it. Since this release the tool is withheld at the projection instead: absent from `tools/list`, and `-32601` on the call. **If an upstream ships a malformed annotation, a tool that was listed yesterday is gone today**, and nothing in your configuration changed. The reason is in the log (`tool_withheld_invalid_x_mcp_header`, naming the tool and what was wrong) and on `mcp_hangar_projection_withdrawals_total{reason="invalid_x_mcp_header"}`. Scrape that counter after the upgrade if you want to know before your users do. An annotation is valid only when it sits on a property reachable through a pure `properties` chain, names an RFC 9110 token, is on an `integer`/`string`/ `boolean` property, and is unique across the schema. The fix belongs upstream; Hangar does not edit a projected schema, because that would move its digest and break every pin. ### A legacy-era header/body mismatch now answers `-32020` (#1051) The handshake-era front door refused a mismatch with `-32600` (`InvalidRequest`) while `tasks/*` used `HEADER_MISMATCH` (`-32020`) for the same class of failure. Both now answer `-32020`; the HTTP status stays 400 and nothing that was refused before is accepted now. **Update any client branch or log query keyed on `-32600` for this case.** Modern-protocol clients are unaffected — the SDK ladder already answered `-32020` there. ### Eight metrics appear on `/metrics` for the first time (#1059, #1049) These were defined and incremented on the live path, and never registered, so they were absent from every scrape — indistinguishable from a feature that was never built: `mcp_hangar_approval_requests_total`, `mcp_hangar_approval_deliveries_total`, `mcp_hangar_approval_decisions_total`, `mcp_hangar_egress_policy_violations_observed_total`, `mcp_hangar_projected_tools`, `mcp_hangar_empty_projection_total`, `mcp_hangar_param_header_validation_skipped_total`, `mcp_hangar_projection_withdrawals_total`. A panel or alert that has been quietly returning no data starts returning rows. The three approval counters have been dead since 2.10.0, and the queries in the observability guide were written against them — those work now. The Audit-mode egress counter is the signal ADR-013 calls the safe adoption path for an `MCPEgressPolicy`, so an Audit rollout is measurable for the first time. Nothing to change; expect graphs that were flat to start moving. ### Optional: govern what an upstream may ask a client to expose (#1057) A new `header_exposure:` block sits beside `tool_projection:` on an mcp_server or a group. It is **off unless you configure it**, and its default action is `warn`, so adopting it does not change what any client sees until you say otherwise. ```yaml mcp_servers: payments: header_exposure: deny_annotated: ["*token*", "*secret*", "*password*", "api_key", "*_key"] on_violation: warn # warn | withdraw | refuse_boot ``` `refuse_boot` refuses to **serve the catalogue**, not to start the process: the violation is only knowable after discovery. An unknown `on_violation` is refused at parse rather than defaulted. ### Optional: `Mcp-Param-*` selectors in an egress policy (#1058) `L7Policy` can now select on SEP-2243 `Mcp-Param-*` headers with the same glob precedence as the tool-name rules. A request whose `MCP-Protocol-Version` predates mandatory header-body validation never satisfies such a selector. In this release the selector is reachable through the REST channel (`PUT /api/mcp_servers/{id}/l7_policy`) only. The matching `MCPEgressPolicy` CRD field ships separately in the operator ([operator#160](https://github.com/mcp-hangar/mcp-hangar-operator/issues/160)), so a GitOps deployment cannot express it yet. ## Upgrade to 2.12.0 ### `truncation.cache_driver: redis` now fails closed (#1007) If Redis cannot actually serve the continuation cache -- the `redis` package is missing, the URL does not parse, or the server cannot `SETEX` (a Sentinel listen port) -- the gateway now **refuses to start** instead of silently falling back to the per-replica memory cache. If your deployment booted with `cache_driver: redis` before this release, Redis was never actually in use; either fix the connection (the image now ships the `redis` extra, #1008) or set `cache_driver: memory` explicitly. A truncated response no longer carries a `continuation_id` unless the full payload was actually stored. `pip install mcp-hangar[redis]` provides the client; it is deliberately not part of the base install or the `full` extra. ## Upgrade to 2.11.0 ### the unused-surface sweep (#969) Nine verified-dead surfaces left over from the factory cut are gone. None had a caller in `src/`; if you imported them in your own code, the replacements are listed: - **`HangarError` / `Rich*` errors, factories, `ErrorClassifier`** (`mcp_hangar.errors`, also re-exported from the package root). The live hierarchy is `mcp_hangar.domain.exceptions`; `is_retryable` stays and keeps matching timeout/connection-style exceptions by pattern. - **`ProgressTracker` / `create_progress_tracker`** (`mcp_hangar.progress`, module deleted). MCP progress notifications are a different, live feature. - **`HealthEndpoint` / `HealthCheck` / `get_health_endpoint`** (`mcp_hangar.observability`). The live probes are the `/health/*` routes; event-store durability get/set remains in `observability.health`. - **`mcp_hangar.domain.bundles`** (starter/developer/data bundle catalog). Hot-loading from the registry (`hangar_load`) is the live path. - **`AuditService`** (`domain.services`). Live audit is `AuditEventHandler` over `IAuditRepository`. - **Tenant/catalog/package exception cluster** (`TenantNotFoundError`, `QuotaExceededError`, `CatalogItemNotFoundError`, `PackageVerificationError`, ...) plus `McpServerEntry` and `CatalogItemId`. There is no catalog API these could describe. - **`HangarLoadResult` / `HangarUnloadResult`** and REST `serialize_tool_info` / `serialize_health_info`; the tools return dicts and REST serializes via `.to_dict()`. - **Metrics helpers** `init_metrics`, `timed`, `record_*` for detection/behavioral features that never shipped producers. - **`initialize_runtime` / `shutdown_runtime`** (`bootstrap.runtime`) and the `trace_tool_invocation` decorator; `create_runtime`, `init_tracing` and `get_tracer` stay. ## Upgrade to 2.10.0 ### `config.yaml` warns about a key nothing reads Unknown keys were kept and ignored at every level. They are now reported, with the offending key and the allowed set named: ```text auth has unknown key(s) ['enabledd']; allowed keys: ['allow_anonymous', 'api_key', 'enabled', 'oidc', 'opa', 'rate_limit', 'role_assignments', 'storage'] ``` **This release warns and starts anyway.** Refusing is correct -- a misspelled `auth` key is a gateway that believes it enabled authentication -- and is also a breaking change for anyone carrying a stale key, so it gets a release of notice instead of arriving in a patch. - `HANGAR_CONFIG_STRICT=1` refuses now, which is what to set in CI. - **the default becomes refusal in 3.0.0.** Any `unknown_config_key` warning in your logs today is a config that will not load then. Checked: top-level section names, the direct keys of each section, and the keys of an `mcp_servers.` spec. Not checked: anything deeper. That is where a single reader exists to enumerate from -- below it the keys live in around twenty modules, and a schema hand-copied from twenty readers drifts into rejecting valid configuration, which is worse than accepting a typo. **New: `mcp-hangar config check [path]`.** Answers the same question without starting a gateway, and is always strict. Exit 0 clean, 1 unknown key, 2 the file is missing or is not YAML. It defaults to `$MCP_CONFIG`, then `config.yaml`. ```console $ mcp-hangar config check config.yaml FAIL config.yaml: 1 key(s) nothing reads: mcp_servers.math has unknown key(s) ['commandd']; allowed keys: [...] ``` ## Upgrade to 2.9.0 Drop-in for every deployment that runs the gateway. Nothing about a served Hangar changes. One tool starts working, and a Python API that no shipped code path ever executed is gone. ### `hangar_load` can now succeed, and wants `uvx` or `npx` on PATH Hot-loading has been enabled by default and unable to complete since it shipped: bootstrap handed its resolver a runtime table with every entry `False` and an empty installer list, so every call answered ```json {"status": "failed", "message": "No compatible package found (missing runtime?)", "warnings": ["Available runtimes: []"]} ``` It now resolves `pypi` packages through `uvx` and `npm` packages through `npx`, and reports availability from the installers rather than from a hardcoded table. The runtime has to be on the gateway process's PATH, which is the part to check before expecting different behaviour: - **the published container image carries neither `uvx` nor `npx`**, so hot-loading still fails there — now with a message naming what is missing rather than an empty list. Derive your own image from ours and add `uv`, Node, or both if you want it working in a container. - running from a `pip install`, install [uv](https://docs.astral.sh/uv/) for PyPI-published servers and Node for npm-published ones. Either alone is fine. `oci` and `mcpb` packages remain unloadable, deliberately: OCI needs a container runtime the image does not ship, and `mcpb` has no defined install path. Both are now reported *unavailable* instead of being selected and then dropped. Nothing to do if you do not use `hangar_load`. `hot_loading.enabled: false` keeps the tool switched off. ### The `fastmcp_server` factory stack is gone Removed from `mcp_hangar.fastmcp_server`: `MCPServerFactory` with its `builder()` and `create_asgi_app()`, `MCPServerFactoryBuilder`, `HangarFunctions`, `ServerConfig`, the thirteen `Hangar*Fn` protocols, and the ASGI combiners `create_health_routes`, `create_combined_asgi_app` and `create_auth_combined_app`. **Nothing about a running Hangar changes.** No shipped code constructed any of it. `serve --http` builds its MCP server in `mcp_hangar.server.bootstrap` and its ASGI app in `mcp_hangar.server.lifecycle.mcp_app_for_serving`, and has never gone through the factory. The two assemblies had drifted far enough to prove it: the factory mounted flat `/health` and `/ready`, while a running Hangar serves `/health/live`, `/health/ready`, `/health/startup` and `/metrics`. Keeping a second construction path that looked serviceable is what made four bugs possible (#592, #594, #595, #596): each was a capability wired into the factory, which made it appear wired and shipped it dead. **If you were embedding through the factory** there is no drop-in replacement, because the factory was never how the product ran. Either run the gateway (`mcp-hangar serve --http`) and drive it over MCP or the REST API, or call the composition root the CLI itself uses: `server.bootstrap` to build and register, `lifecycle.mcp_app_for_serving` for the ASGI app, and `server.api.middleware.create_auth_enforced_app` to apply the same authentication. Those are tested on every PR and are what a released Hangar executes. `HANGAR_SERVER_NAME` is unchanged and still exported from `mcp_hangar.fastmcp_server`. The v0.4.0 note further down names the factory as the successor to `setup_fastmcp_server()`; it describes what that release did and stays as history. ## Upgrade to 2.8.0 Two things can break a build rather than a deployment: an extra that no longer exists, and a bundled monitoring stack that has moved to the Helm chart. ### `pip install mcp-hangar[containers]` now fails The `containers` extra is gone. It installed `testcontainers` for a test tier that never ran — those tests were gated behind `--run-containers` / `--run-slow` and no CI job, `Makefile` target or script ever passed either flag, so every one of them reported `skipped` on every run. Nothing in the shipped package imported it. Drop `[containers]` from your install line. If you depended on `testcontainers` yourself, depend on it directly. ### The bundled compose monitoring stack is gone `monitoring/` and `docker-compose.monitoring.yml` are removed from the repository. The four Grafana dashboards and the 30 Prometheus alert rules ship with the Helm chart instead: `dashboards.enabled` renders them as sidecar-labelled ConfigMaps, `prometheusRule.enabled` renders a `PrometheusRule`. Instrumentation is untouched — `/metrics`, tracing and the OTLP exporter are unchanged; only bundled config moved. There is no one-command local Grafana any more. If you were running it, either use the chart or keep a copy of the compose file from the 2.7.0 tag. ### The published container runs Python 3.14 `pip install` still supports 3.11 through 3.14, and 3.14 is now a required CI citizen rather than an advisory one. Relevant only if you build on top of our image and pin something against the interpreter version. ### Three unused symbols left the application layer `CallbackAlertSink` and `LogAuditStore` are gone from `mcp_hangar.application.event_handlers`, and `detect_runtime_availability` with its `IRuntimeChecker` protocol from `mcp_hangar.application.services`. None had a caller outside this repository's own tests. - **`CallbackAlertSink`** — production `get_alert_handler()` builds a `LogAlertSink`. To capture alerts in your own code, implement the ABC: ```python from mcp_hangar.application.event_handlers.alert_handler import Alert, AlertSink class CapturingSink(AlertSink): def __init__(self) -> None: self.alerts: list[Alert] = [] def send(self, alert: Alert) -> None: self.alerts.append(alert) ``` - **`LogAuditStore`** — it could not have served as an audit store: `query()` raised `NotImplementedError`, because a log sink cannot answer a query. Write the sink you want against the `AuditStore` ABC, or use the OTLP exporter path (`OTLPAuditEventHandler` / `IAuditExporter`), which is built for shipping audit records off the box. - **`detect_runtime_availability`** — no replacement, deliberately. Ask the installer you care about (`is_runtime_available()`) and construct the `RuntimeAvailability` yourself, which is all the removed function did, in a fixed order, for a list it did not validate. `AlertSink`, `Alert`, `LogAlertSink`, `AlertEventHandler`, `get_alert_handler`, `AuditRecord`, `AuditStore`, `InMemoryAuditStore`, `AuditEventHandler`, `get_audit_handler`, `PackageResolver` and `RuntimeAvailability` are unchanged. ## Upgrade to 2.7.0 Drop-in for most deployments. Two behaviours change without a config change: `approval_channel`, which was recorded and ignored, now selects where approvals are notified; and the MCP endpoint stops handing out session ids. Read the `approval_channel` section if any of your policies set it, and the session section if anything in front of your pods pins traffic. ### The MCP endpoint no longer hands out a session id `initialize` returns no `Mcp-Session-Id`, and no request needs one. A session lived in one replica's memory, so a client that initialized against one pod and called against another was told `Session not found` -- 13 of 15 attempts through a three-replica Service. Session affinity papered over that and could not fix it: a pin does not outlive its pod, so a rolling restart or a scale-down took the session with it. | | before | from 2.7.0 | |---|---|---| | `initialize` | returns `Mcp-Session-Id` | returns no session id | | a request carrying a stale or foreign `Mcp-Session-Id` | `Session not found` | served; the header is ignored | | `DELETE /mcp` | `200` | **`405 Method Not Allowed`** | The last row is the only one that can surface in a client's logs. There is no session to terminate, so teardown is refused rather than acknowledged. **What this does not change.** Nothing about the 2026-07-28 revision, which has no sessions at all and was already served this way. Nothing about session *suspension* (`/api/sessions`), which keys on the caller identity from `x-session-id` or the JWT `sid` claim and never on the transport. Nothing about authorization, which is per request. **Deployments.** Sticky routing is no longer a requirement for a replica set -- see [running more than one replica](cookbook/25-multiple-replicas.md). Existing pinning is now merely unhelpful rather than wrong, so there is no rush to remove it; leave it if you still run an older gateway behind the same ingress. ### `front_door` no longer serves an empty tool list after a restart Also fixed here, and worth knowing whether it happened to you. In `tool_access.mode: front_door`, `tools/list` **is** the per-tenant projection, and the projection was built from whatever that replica had started. A replica that had started nothing served an empty list to a valid tenant, with no client- reachable way to fix it, and two replicas that had warmed different servers answered the same tenant differently. A `front_door` gateway now starts every configured mcp_server at boot, on its own thread so readiness never waits on a backend handshake. A backend that fails to start is logged as `front_door_warmup_failed` rather than costing the others their projection. `egress` is unchanged: backends still start lazily on first use. ### A consent gate no longer disappears on restart Fixed, not a migration step — but worth knowing whether it happened to you. The tool-access-policy store held `allow_list` and `deny_list` and nothing else, and the startup replay rebuilt policies from those two fields, assigning over whatever the YAML had already registered. A server with `tools.approval_list` in its config and **any** prior policy update over the REST API came back **ungated** after a restart: the tools it named ran without being held, and the startup check that guards this class saw no `approval_list` left to demand a gate, so the boot was clean. The store now persists the approval fields and the replay hands back whole policies. An existing database is widened in place on first open. A row written by an older build carries no approval columns; rather than let that erase a gate one last time, the replay carries the in-force gate forward and logs `tap_replay_carried_approval_gate`. Nothing to do. If a gate was lost to this, it is back on the next restart — the YAML declaration was never what went missing. If you keep audit records, calls to `approval_list` tools between an affected restart and this upgrade ran without a human decision. ### `approval_channel` now routes, and the built-in channel is renamed `approval_channel` was documented as a policy's delivery channel and merged carefully across scope narrowing — and dispatched nowhere. One delivery, built from the global `approvals.channel`, handled every approval whichever policy raised it. A config that set `approval_channel: slack` on one server and something else on another got one channel, silently. They now route as written. **Check your policies before upgrading**: if two servers name different channels and only one adapter is installed, the other now degrades to `noop` where it previously borrowed the global channel. The core channel formerly called `dashboard` is now `event_stream`. It was named after a management UI that shipped with the Hangar Cloud tier and was archived with it, and it never pushed to that UI anyway — its `send` wrote a log line while its docstring claimed a WebSocket integration that was never wired. The new name points at the surface that does carry the notification: the `ToolApprovalRequested` domain event on `/api/ws/events`. `channel: dashboard` still resolves, to the same delivery, and logs `approval_delivery_channel_renamed` once at boot. No config change is required. ### An armed gate now says when nobody is listening A policy that gates a tool while its channel reaches nothing outside the process — `noop`, or a vendor name no installed package claims — is now reported at startup: ```text subsystem_configured_but_unreachable subsystem=approval_delivery required_by="tools.approval_list on mcp_server:payments (channel 'slack')" fail_closed=False ``` The gateway still starts. The gate is fail-closed by timeout, so what is missing is a signal rather than enforcement, and refusing the boot over a notification channel would trade a degraded notify path for an outage. A deployment that wants the refusal opts in: ```yaml approvals: delivery: required: true ``` Three metrics land with it — `mcp_hangar_approval_requests`, `mcp_hangar_approval_deliveries` and `mcp_hangar_approval_decisions`, all labelled by channel. See [Observability → Approval Gate](guides/OBSERVABILITY.md). ### Removed `hangar_approve_prompt`, an MCP tool nothing registered, whose docstring pointed at an `approvals.channel: mcp_prompt` that no builtin or entry point has provided since 2.0. If you were calling it, you were getting a `tool not found`. ## 2.6.0 — three things to check before you roll out Two of these can stop a deployment that works today, and both are in the same place: enforcement that was advertised and did not run now runs. Read the first two before upgrading. The third is additive. Nothing changes for a gateway with authentication off, except that it may refuse to start for the reason in §1. ### 1. Per-tenant digest pins with authentication off now refuse the boot A digest pin could only be declared under a tenant: ```yaml tool_projection: digest_enforcement: block tenant_overrides: "tenant:a": pins: { refund: } ``` and the tenant reaches the enforcement path from the authenticated principal and nowhere else. So on a gateway with `auth.enabled: false` — where every caller is anonymous and carries no tenant — **no pin was ever matched**. Drift stayed computable and nothing stopped it, while `initialize` went on advertising `io.mcp-hangar.digest-pinning` with all three enforcement modes. The same miss took the task path with it: nothing bound a relayed task to a digest, so the fail-closed re-verification on result retrieval had nothing to check. That configuration now fails the boot rather than serving a guarantee it cannot keep: ```text digest pins are declared per tenant (payments.tenant:a.refund) and authentication is disabled (`auth.enabled: false`), so no caller carries a tenant id and not one of those pins can ever be matched -- schema drift would be counted and nothing would be stopped. Either enable authentication so callers arrive with the tenant these pins name, or move them to the all-tenants `tool_projection.pins:` block, which holds every caller including an anonymous one. ``` **What to do.** Either turn authentication on, so callers arrive carrying the tenant the pins name, or move the pins to the new all-tenants block, which holds every caller including an anonymous one: ```yaml tool_projection: digest_enforcement: block pins: # every caller refund: ``` Both forms may be used together. A pin declared for a tenant wins over the all-tenants one for that tenant — narrowest first, the order the tool-access policies already resolve in. **Who is unaffected.** A gateway with authentication on. Its per-tenant pins were being enforced and continue to be, unchanged. ### 2. The `hangar_*` tools now require the permission their REST twin requires `hangar_call` authorized every call it dispatched. The other twenty-one `hangar_*` tools authorized **nothing** — so with authentication on, any valid credential could stop a server, load one, reload the configuration or approve a discovered upstream over MCP, while the same operations over REST were refused for the same identity in the same process. Authorization is now resolved from the tool name, mirroring the REST route that performs the same operation. No permission was invented and no role changed: | Tool | Permission | Built-in roles that hold it | | --- | --- | --- | | `hangar_list`, `hangar_status`, `hangar_details`, `hangar_tools`, `hangar_health` | `mcp_servers:read` | admin, provider-admin, developer, viewer | | `hangar_start`, `hangar_stop`, `hangar_warm` | `mcp_servers:lifecycle` | admin, developer | | `hangar_load`, `hangar_unload` | `mcp_servers:write` | admin, developer | | `hangar_reload_config` | `config:reload` | admin | | `hangar_discovered`, `hangar_sources` | `discovery:read` | admin, provider-admin, developer, viewer, auditor | | `hangar_discover` | `discovery:trigger` | admin, provider-admin | | `hangar_approve`, `hangar_quarantine` | `discovery:approve` | admin, provider-admin | | `hangar_group_list` | `group:read` | admin, provider-admin, developer, viewer | | `hangar_group_rebalance` | `group:update` | admin, provider-admin | | `hangar_metrics` | `metrics:read` | admin, provider-admin, viewer, auditor | | `hangar_fetch_continuation`, `hangar_delete_continuation` | `tool:invoke` | admin, provider-admin, developer, service-account | `hangar_call` is unchanged: it still checks `tool:invoke` per call in the batch, which is finer than one entry here could express. **What to check.** Any API key or token that drives the fleet over MCP. If it was working because MCP asked for nothing, it now needs the role its REST equivalent has always needed. Two combinations surprise people: - **`provider-admin` cannot start, stop, load, unload or reload.** It holds `mcp_servers:read` and not `:write` or `:lifecycle`, and not `config:reload` — deliberately, and it could not do those things through the REST API either. An operator key that needs lifecycle wants `admin`, or a custom role holding exactly the permissions above. - **`developer` cannot approve, quarantine, trigger discovery, rebalance a group or read metrics.** It holds fleet read, write and lifecycle, and none of the discovery-approval, group-update or metrics permissions. A refused call answers with the permission it wanted, so the log names what to grant: ```text Not authorized to call 'hangar_stop': mcp_servers:lifecycle permission required ``` **A gateway with authentication off is unchanged** — every call is allowed, as it already was on the `hangar_call` path. `--unsafe-no-auth` behaves exactly as before. `metrics:read` is now enforced on `hangar_metrics`. The unauthenticated `/metrics` scrape endpoint is untouched; it is on the auth skip list and stays there. ### 3. `front_door` shows an operator a control plane Additive, and nothing to do. `front_door` served flat upstream names and no `hangar_*` to anybody, so an operator on a front door had no management surface over MCP at all and had to run a second instance in `egress` to get one. A management tool is now listed exactly when the caller is authorized to call it — the same table as §2, the same authorizer. An agent principal sees what it saw before; an operator's list grows by the tools its role permits. Nothing is shown that could not be called, and a name that is not shown is still `-32601`. With authentication off the management surface stays empty, which is stricter than the invoke path is on the same gateway. That is deliberate: a front door that shows an unauthenticated caller nothing today should not start showing it a control plane. `egress` is untouched and still serves every caller the whole meta-API. There it is not a management surface that happens to be visible — it *is* the surface, and a client without `hangar_call` reaches no upstream tool at all. New metric: `mcp_hangar_projected_tools`, a histogram of how many tools a front-door `tools/list` returned, labelled `kind=governed|management`. ### Also: one log line per config-file remote upstream A `remote` server declared in `config.yaml` is outside the SSRF policy — it gets neither the registration check that answers `400 ssrf_blocked` on the REST path, nor the connect-time re-resolution added in 2.5.0 that closes DNS rebinding. That exclusion is deliberate (the file is trusted input; see ADR-021) and has not changed. What has changed is that startup now says so, once per such upstream, naming it and its endpoint. Nothing is refused and no request path is affected. If you want an endpoint checked, register it through the REST API instead of the file. ## 2.5.3 — two things a client may notice Every change in this release is a defect fix and nothing you wrote has to change. Two of them are visible from outside the gateway, so they are worth acting on rather than reading past. ### `prompts/list` and `resources/list` now answer `-32601` The gateway advertised the `prompts` and `resources` capabilities on every deployment and served neither. Nothing hard-coded that claim — the SDK derives each capability from whether its handler is registered, and the framework registers both unconditionally, empty or not. The cost was not the missing feature; it was the lie. `{"prompts": []}` tells a conformant client *this server has no prompts*, which is a different statement from *this gateway does not carry prompts*, and nothing on the wire distinguished them. Registered upstreams with prompts and resources were invisible with no error anywhere. | | before | after | | --- | --- | --- | | `initialize` capabilities | `prompts`, `resources` advertised | neither advertised | | `prompts/list`, `resources/list` | `200` with an empty list | `-32601` Method not found | **Who is affected.** A client that reads the advertised capabilities before calling — which the specification tells it to do — sees no `prompts` capability and does not call. Nothing to change. A client that calls these methods unconditionally and treats a JSON-RPC error as fatal will now fail where it previously received an empty list; it needs to check capabilities first. This is derived rather than inverted: when the gateway proxies an upstream's prompts and resources (#889), the capabilities return on their own. ### An upstream's tool catalogue may grow The gateway never finished the MCP handshake — it sent `initialize` and went straight to `tools/list`, skipping the `notifications/initialized` the lifecycle requires. A server is entitled to defer work until that notification arrives, and servers do: against the official reference server, a tool registered in its `oninitialized` handler was neither listed nor callable. **What to expect.** If your upstream registers tools on initialization, this release discovers them for the first time, and its catalogue legitimately grows after the upgrade. Anything asserting on a **tool count** will notice. Tool **digests** are unaffected: the pinned surface is unchanged, so no existing pin moves — including the ones on tools that carry the newly-forwarded `title`, `annotations`, `execution`, `icons` and `_meta`. The notification is best-effort. An upstream that mishandles it gets a warning in the log rather than a failed start. ## 2.5.1 — restart to re-arm a defence 2.5.0 lost, and check one configuration Drop-in from 2.5.0 in the sense that nothing you wrote has to change. Two things are worth acting on rather than reading past. ### The connect-time SSRF guard was not surviving a restart 2.5.0 added a second SSRF check on every outbound connection, with the connection pinned to the validated address, so a hostname that passed registration could not be re-pointed at `169.254.169.254`, `10.x` or `127.0.0.1` before the next tool call. The flag that arms it was never written to the stored server record, so **any server rebuilt from that record came back without it** — after every restart, and on every replica that learned of the registration from the shared log rather than performing it. **What that means for a gateway running 2.5.0.** If it has restarted since a `remote` server was registered through the REST API or by discovery, that upstream has been reached with registration-time validation only. The endpoint was still checked once, when it was registered; what lapsed is the re-check that defends against the name being re-pointed afterwards. In a replica set, only the replica that handled the registration ever had the guard. **What to do: upgrade and restart.** The guard is restored for servers already in the store — no re-registration, no edit to the database. One deliberate exception: a stored endpoint that is a private literal keeps 2.5.0's behaviour, because such a row can only have come from discovery reporting a container address, and arming the strict policy over it would refuse an upstream that works today. Re-registering such a server writes the provenance that scopes the guard correctly. **One behaviour to expect once it is armed.** Guarding is also pinning: a guarded connection goes to one validated address rather than letting the client walk a multi-address DNS answer, so a dead address behind a healthy name fails the call instead of being skipped. That shipped in 2.5.0; what changes here is how much of your fleet it covers. ### A `coordination:` block with no `persistence.backend` is now refused 2.5.0 refused a declared cluster on a backend the replicas cannot share, and said nothing when no backend had been selected at all. The outcome is the same either way — no lease keeper, every replica managing the fleet, every one reporting `manages_fleet: true` — so it is now refused too. **Who is affected:** a configuration carrying `coordination:` while storage is still configured through the legacy per-subsystem keys (`event_store.driver`, `auth.storage.driver`). That deployment may well share one PostgreSQL, and it was never coordinating through it. It booted on 2.5.0 and will not boot on 2.5.1 until it says where it persists: ``` this gateway is configured as part of a cluster (`coordination:`), and no storage backend has been selected. ... Set `persistence.backend: postgresql`, or remove the `coordination:` block to run this as a single gateway. ``` Both ways out are in the message. A single gateway that never declared `coordination:` is unaffected, whatever its storage. ## 2.5.0 — nothing changes until you select a storage backend The release adds `persistence.backend` and multi-replica coordination. **An existing configuration that sets neither is unaffected**: omitting `persistence` keeps the per-subsystem storage behaviour exactly as it was, which is deliberate — a storage rewiring must not change what a running deployment does. Everything below applies only once you opt in. ### Selecting a backend takes over every persisted concern `persistence.backend: sqlite | postgresql` chooses storage for all of it at once: the event log and its delivery mark, server configuration, the audit trail, saga state, approvals, API keys, roles, tool-access policies, metric history and the management lease. A backend serves every one of them or selection is refused, which is what makes the half-configured deployment unrepresentable — before 2.5.0 you could select the PostgreSQL auth driver and silently lose tool-access policy management with it. Two consequences to check before you roll out: - **`${VAR}` in configuration is interpolated everywhere.** It used to work inside `mcp_servers..auth` and nowhere else, while the documentation described it as a property of configuration. If you kept a secret out of the file the way the production checklist says to, and it silently arrived as the literal characters, this is why. The refusal moved with it: a `${VAR}` with no value and no `:-default` has always been fail-closed, and now fails the whole boot rather than only the `auth` sub-block. Check the keys you never had to set before -- `${VAR:-}` allows an empty value explicitly. A value that *contains* a literal `${...}`, such as a generated password, is safe: the document is interpolated once, so the substituted text is never rescanned. - **A per-subsystem key that names a different backend now refuses startup.** `auth.storage.driver` and `event_store.driver` are compared against your selection, and a contradiction fails the boot rather than being resolved by a precedence rule. Whichever way such a rule fell, half of what you wrote would be ignored — and the half that loses is the one written most recently. `memory` is exempt: it is a testing choice, not a storage backend. - **`event_store.allow_memory_fallback` no longer has anything to decide.** With a backend selected, the log and its delivery mark come from it, and a backend is durable as a whole. Keep the key if you are not selecting a backend; it still fails a non-durable store fast there. **There is no migration between backends.** Selecting PostgreSQL on a gateway that has been running on SQLite starts an empty database — it does not move what is in the file. ### Selecting PostgreSQL turns coordination on, at one replica This is the one that can surprise a single-node deployment. Coordination keys off whether the storage **can be shared**, not off how many replicas you run, so a single gateway on PostgreSQL takes a management lease and reports `coordinates_with_peers: true`. It manages the fleet, because it is the holder — nothing stops working. What does change on that deployment: **registering a `subprocess`, `docker` or `container` server through the API is refused** (HTTP 422), because those modes attach a child process's stdio to one gateway and any peer that learned of such a server would start its own copy. Servers already declared in `config.yaml` keep working — the refusal is on the registration path, not the startup one. If that deployment is genuinely single-node and wants to keep registering local modes at runtime, stay on `persistence.backend: sqlite`, which is not shareable and therefore not coordinated. The message says so: it names the condition — storage peers can share — and offers `persistence.backend: sqlite` as the alternative to `remote` mode. It used to end "or run a single instance", which read oddly when you already were one. ### A declared cluster refuses a child-process server outright The paragraph above is about *runtime registration*. Servers declared in `config.yaml` take a different path, and when the deployment declares a `coordination:` block they are now refused **at startup**, naming every offender at once: ``` this gateway is configured as part of a cluster (`coordination:`), and 'reports' is 'subprocess'. ... Use `remote` mode for servers several replicas must serve, or remove the `coordination:` block to run this as a single gateway. ``` Without the block nothing here fires, which is the whole point of asking on that axis: a single gateway that merely uses PostgreSQL keeps running its child processes exactly as before. What this replaced is worth knowing if you ran an earlier candidate. Such a server loaded on every replica and only the lease holder could start it, so `GET /api/mcp_servers//tools` answered with the server's tools on one pod and an empty list on the others, and starting it on any other pod returned a `409`. ### A `coordination:` block requires PostgreSQL Adding `coordination:` is the statement that these replicas are meant to be **one** gateway. On a file-backed backend it refuses to start, because replicas that cannot share storage are not a cluster — each would hold its own fleet and its own lease and never notice the others. That is not hypothetical: three replicas on SQLite each reported `manages_fleet: true`, with every health check green. Running many pods each with their own storage stays legitimate — that is many gateways, and nobody's business but yours. What is refused is calling them one. ### If you already run more than one replica Through 2.4.0 the documentation said not to, and the failure was silent rather than loud. On 2.5.0, to make a replica set safe you need all three of: one PostgreSQL every replica shares, a `coordination:` block, and `remote`-mode servers. Then check it pod by pod rather than through the Service — exactly one should answer `manages_fleet: true` at `GET /api/system`. Two costs are worth knowing before the rollout rather than after: rate limits are counted **per instance** (three replicas admit three times the configured rate — a fleet-wide cap belongs at the ingress), and anything travelling by the shared log reaches peers within a poll interval rather than immediately. The full recipe is [cookbook 25](https://mcp-hangar.io/docs/cookbook/25-multiple-replicas); the decisions and their failure modes are in [ADR-020](https://mcp-hangar.io/docs/adr/ADR-020-high-availability). ## Discovery: namespace policy moves to the Kubernetes source `discovery.security.allowed_namespaces` and `discovery.security.denied_namespaces` belong to the Kubernetes source now, not to the core's security config. They were the only source-specific rules in a component that is otherwise source-agnostic, and they were applied behind a check on the source's name -- so any other source passed that stage with nothing validated and nothing said. **Who is affected:** deployments that set either key. **What to do:** move them under the kubernetes source's own entry: ```yaml discovery: sources: - type: kubernetes namespaces: [apps] denied_namespaces: [kube-system, default] # was discovery.security.* ``` The old location still works and is applied when the new one is absent, with a `discovery_namespace_policy_deprecated_location` warning at startup. It will be removed in a later release. Moving a security setting silently is the one migration that must not happen quietly: a deployment that denied `kube-system` must not start accepting it because a key changed address. Defaults are unchanged -- `kube-system` and `default` are still denied when nothing is configured. A source can now declare its own rules through `DiscoverySource.policy_violation`, which is optional: a source that does not implement it raises no objection, so existing third-party sources keep working untouched. ## Discovery: an unknown `source_type` now refuses to start A discovery source configured with a type nothing provides used to be skipped with a warning, and the gateway carried on with that source absent. A typo in `type:` therefore produced a running gateway with no discovery and one line in the log. It now raises at startup, the way an unknown `event_store.driver` already did. **Who is affected:** deployments whose configuration names a source type that is not installed. They were already not getting that source; now they are told. **What to do:** fix the type, or install the package that provides it. The error lists the types that are registered. A missing *optional dependency* is unchanged -- `ImportError` still degrades with a warning, because that is a deployment shape rather than a mistake in the configuration. Third-party sources now register under the `mcp_hangar.discovery_sources` entry point group, so adding one no longer means patching the core. ## Discovery: an unknown `mode` now refuses to start A discovery source whose `mode` was misspelled used to fall back to `additive`. That was especially dangerous for a source intended to be `authoritative`: the gateway stayed up, but never removed servers that disappeared from discovery. The configured value is now parsed as a `DiscoveryMode` and startup refuses unknown values. **Who is affected:** deployments with a discovery source whose `mode` is not `additive` or `authoritative`, including capitalization variants such as `Authoritative`. **What to do:** correct the value in the source configuration. An omitted mode still defaults to `additive`. ## `auth bootstrap-admin` requires `--show-key` when API keys are the only way in On a deployment with no trusted OIDC issuer, omitting `--show-key` is now refused before anything is written: ``` Error: Nothing could use this administrator: API keys are the only authenticator, and the key's secret would not be printed. ``` **Who is affected:** anything that scripts `mcp-hangar auth bootstrap-admin` against a config without an `auth.oidc` block. Add `--show-key` and capture the secret the run prints. **Why it refuses rather than warns.** The claim is one-shot and the key is stored hashed, so a run that ends without printing the secret can be neither repeated nor recovered from. It used to end by advising a re-run with `--show-key`, at the exact moment re-running had become impossible: the second run answers "The initial administrator has already been bootstrapped", and `bootstrap-admin` is the only subcommand in the auth CLI. The refusal costs one command; the advice cost the deployment. Nothing changes for a deployment that trusts an OIDC issuer -- the principal authenticates on its own identity and needs no secret. That run's closing message no longer suggests a second chance exists, because it does not. A store whose claim has already been spent with the secret discarded is recovered by clearing its `initial_admin_bootstrap` row, or by starting from a fresh auth store, and re-running with the flag. ## Removed: `EventBus.on_error` The hook that registered a callback for exceptions raised inside event handlers is gone, along with the list it appended to. **Who is affected:** only code that calls `EventBus.on_error(...)`. Nothing in Hangar ever did, so the loop that invoked those callbacks ran zero times on every handler failure -- dead code in the one path that only executes when something is already wrong. `IEventBus`, the port the application layer depends on, never declared it. **What to do:** nothing, unless you registered a callback. If you did, the information it carried is now a metric: a handler that raises increments `mcp_hangar_errors{component="event_handler"}`, labelled with the exception type, and the `event_handler_error` log line now names the failing handler. The fault barrier itself is unchanged -- one failing handler still does not stop the others, and `publish()` still does not raise. ## Removed: eight domain event classes that nothing emitted `CatalogItemApproved`, `CatalogItemDeprecated`, `CatalogItemPublished`, `CatalogItemRejected`, `ToolSchemaChanged`, `ToolSchemaDriftDetected`, `BehavioralModeChanged` and `CapabilityDeclarationMissing` are gone from `mcp_hangar.domain.events`. **Who is affected:** only code that imports one of those names. No deployment can have received one of these events, because nothing in Hangar has ever constructed one -- they were vocabulary for features that were never built, and an audit found them with no producer and no consumer anywhere in the tree. **What to do:** delete the import and any handler registered against it. Such a handler has never been called, so removing it changes no behaviour. A stream cannot contain one either, so no event store needs migrating. The remaining unemitted events -- the five discovery ones, four quarantine ones and `PolicyPushRejected` -- are deliberately kept: those features are live or planned, and the missing emitter is tracked rather than papered over. ## 2.3.0 — two things to check before you roll out Neither affects a default deployment. The first matters only if you import the concrete launchers from the domain layer; the second only if you set `auth.storage.driver: event_sourcing`. > The work below was written against a planned 2.2.2. That release was never > cut -- it became 2.3.0 when the launcher removal landed, so everything here > ships in 2.3.0. ### The deprecated launcher import paths are gone Only affects code that imports the concrete launcher classes from the domain layer. If you import them from `mcp_hangar.infrastructure.launchers`, which is where they live and what the deprecation warning has been telling you since **v1.0.2**, nothing changes. Two import paths were removed: ```python # Both of these now raise. from mcp_hangar.domain.services.mcp_server_launcher import DockerLauncher from mcp_hangar.domain.services import DockerLauncher # This is the one to use, and always was: from mcp_hangar.infrastructure.launchers import DockerLauncher ``` The same applies to `SubprocessLauncher`, `ContainerLauncher`, `HttpLauncher`, `ContainerConfig`, `McpServerLauncher` and `get_launcher`. `mcp_hangar.domain.services` still exports the launcher **port**, `IMcpServerLauncher`, along with `LaunchResult` and `TransportClient`. It is the concrete implementations that moved out — a domain package re-exporting infrastructure classes is what the deprecation was about. The shim emitted a `DeprecationWarning` on import from v1.0.2 onward, so a run of your test suite with warnings visible will list every call site: ```bash python -W error::DeprecationWarning -m pytest ``` Removing it also broke a real import cycle: the domain reaching for the concrete launchers is what forced two sagas to import their saga manager inside a function body rather than at module level. ### If you run `auth.storage.driver: event_sourcing`, read this before upgrading On that driver, API keys and role assignments were written to the event store correctly and could not be read back: the writer accepts any domain event, the reader looked the class up in a hand-maintained table that listed 30 of the 116 event types, and all five the auth aggregates emit were missing. Every API key stopped authenticating across a restart, and role assignments were invisible after one. Affected from **1.2.2** (when the driver landed) through **2.2.1**; `memory` is the default and `sqlite`/`postgresql` were never affected. **Nothing was lost** — only the read path failed — which is exactly why this needs planning rather than celebration: > Credentials and role assignments you believed were gone start working again > the moment you upgrade, including any `admin` assignment made in that window. Revocations are events too and replay in order, so anything you revoked stays revoked. Look at what is dormant before you roll out, and revoke what you do not want live. The canonical guide has the two `sqlite3` commands for that: . Also in this release, and needing no configuration change: events written before the `provider` -> `mcp_server` rename (stores from 1.0.1 or earlier) reach their handlers again instead of replaying into nothing, and a `datetime` field on a persisted event comes back as a `datetime` rather than a string. --- ## 2.2.0 — action required before you roll out 2.2.0 is a security release. It is a minor rather than a patch because it changes behaviour that working deployments rely on. Three things can break a working deployment, and two of them fail silently: 1. **Operator API key.** `POST`/`DELETE /api/mcp_servers/{id}/l7_policy` now requires `policy:write` instead of `mcp_servers:write`. A `developer` token stops delivering compiled egress policy — the CRD still reports `Compiled` and nothing reaches the enforcement point. Move operator keys to `provider-admin`, which gained `mcp_servers:read` + `policy:write` for exactly this. 2. **OPA policies.** A non-boolean verdict was treated as *allow* (including a policy returning the string `"deny"`). It is now a denial. A policy that returns an object or a string flips from allowing everything to denying everything. 3. **`tool_access.mode`.** A misspelled value used to fall back to `egress` with a warning; the server now refuses to start. An absent key still means `egress`. Also changed: REST authorization is enforced on every route (`/config`, `/discovery`, `/groups`, `/sessions`, `/tools`, `/approvals` reads and the whole `/auth` subtree previously made no authorization decision at all); `POST /api/config/reload` no longer accepts a caller-supplied `config_path`; approvals pending across the upgrade are refused and must be re-requested. The full guide, with the role-compatibility table and the per-item rationale, is the canonical one: . --- ## Upgrading to MCP Hangar v1.0 This guide covers upgrading from v0.12.x (or earlier) to v1.0.0. If you are upgrading from a version older than v0.4.0, read every section. If you are already on v0.12.x, skip to the [Pre-flight checklist](#pre-flight-checklist) and then review only the sections marked with your starting version. --- ## Pre-flight checklist Run through this list before you begin. Every item should be green before you upgrade production. 1. **Back up your configuration.** Copy `config.yaml`, `.env`, and any Kubernetes manifests (MCPProvider, MCPProviderGroup, MCPDiscoverySource CRs). 2. **Back up your event store.** If you use SQLite or Postgres event sourcing, take a snapshot or dump before upgrading. 3. **Note your current version.** Run `mcp-hangar --version` or check `pyproject.toml`. 4. **Check Python version.** v1.0 requires Python 3.11+. Run `python3 --version` to confirm. 5. **Review deprecation warnings.** Run your test suite and check logs for deprecation warnings introduced in v0.4.0 through v0.12.0. 6. **Read the sections below** that apply to your starting version. 7. **Test in staging** before promoting to production. --- ## Version upgrade paths | Starting version | Path | |-----------------|------| | v0.1.x - v0.3.x | Read ALL sections below in order. | | v0.4.x - v0.6.x | Start at [Environment variables](#environment-variables). | | v0.7.x - v0.12.x | Start at [Configuration changes](#configuration-changes-v060). | | v0.12.x | Start at [Enterprise module split](#enterprise-module-split). | --- ## Python version requirement **Applies to:** all versions before v0.3.0 MCP Hangar v1.0 requires Python 3.11 or later. Earlier versions were compatible with Python 3.10. If you are running 3.10, upgrade Python first. ```bash python3 --version # Must be 3.11.x or later ``` --- ## Rebrand: "registry" to "hangar" (v0.4.0) **Applies to:** upgrading from v0.3.x or earlier v0.4.0 renamed the project from "MCP Registry" to "MCP Hangar". This is the single largest breaking change in the project's history. All backward compatibility aliases were removed in v0.4.0. ### MCP tool renames All 14 MCP tools changed prefix from `registry_*` to `hangar_*`: | Old (removed) | New | |---------------|-----| | `registry_list` | `hangar_list` | | `registry_start` | `hangar_start` | | `registry_stop` | `hangar_stop` | | `registry_invoke` | `hangar_invoke` | | `registry_tools` | `hangar_tools` | | `registry_details` | `hangar_details` | | `registry_health` | `hangar_health` | | `registry_discover` | `hangar_discover` | | `registry_discovered` | `hangar_discovered` | | `registry_quarantine` | `hangar_quarantine` | | `registry_approve` | `hangar_approve` | | `registry_sources` | `hangar_sources` | | `registry_metrics` | `hangar_metrics` | | `registry_group_list` | `hangar_group_list` | | `registry_group_rebalance` | `hangar_group_rebalance` | **Action:** Update any AI assistant system prompts, scripts, or integrations that reference tool names. ### Python API renames | Old (removed) | New | |---------------|-----| | `RegistryFunctions` | `HangarFunctions` | | `RegistryListFn` | `HangarListFn` | | `RegistryStartFn` | `HangarStartFn` | | `RegistryStopFn` | `HangarStopFn` | | `RegistryInvokeFn` | `HangarInvokeFn` | | `RegistryToolsFn` | `HangarToolsFn` | | `RegistryDetailsFn` | `HangarDetailsFn` | | `RegistryHealthFn` | `HangarHealthFn` | | `RegistryDiscoverFn` | `HangarDiscoverFn` | | `RegistryDiscoveredFn` | `HangarDiscoveredFn` | | `RegistryQuarantineFn` | `HangarQuarantineFn` | | `RegistryApproveFn` | `HangarApproveFn` | | `RegistrySourcesFn` | `HangarSourcesFn` | | `RegistryMetricsFn` | `HangarMetricsFn` | | `with_registry()` | `with_hangar()` | | `factory.registry` | `factory.hangar` | **Action:** Search your code for `Registry` and `with_registry` and replace. ### Removed factory functions These convenience functions were removed in v0.4.0: | Removed | Replacement | |---------|-------------| | `setup_fastmcp_server()` | `MCPServerFactory` | | `create_fastmcp_server()` | `MCPServerFactory.create_server()` | | `run_fastmcp_server()` | `MCPServerFactory.create_asgi_app()` | ```python # Before (removed) from mcp_hangar import setup_fastmcp_server server = setup_fastmcp_server(config_path="config.yaml") # After from mcp_hangar.fastmcp_server import MCPServerFactory factory = MCPServerFactory() server = factory.create_server(config_path="config.yaml") ``` ### Prometheus metric renames All metrics changed prefix from `mcp_registry_*` to `mcp_hangar_*`: | Old | New | |-----|-----| | `mcp_registry_tool_calls_total` | `mcp_hangar_tool_calls_total` | | `mcp_registry_tool_call_duration_seconds` | `mcp_hangar_tool_call_duration_seconds` | | `mcp_registry_provider_state` | `mcp_hangar_provider_state` | | `mcp_registry_cold_starts_total` | `mcp_hangar_cold_starts_total` | | `mcp_registry_health_checks` | `mcp_hangar_health_checks` | | `mcp_registry_circuit_breaker_state` | `mcp_hangar_circuit_breaker_state` | **Action:** Update Grafana dashboards, Prometheus recording rules, and alert rules. If you use the bundled dashboards from `monitoring/`, update them from the latest release. --- ## Kubernetes operator API group (v0.2.0) **Applies to:** upgrading from v0.1.x The CRD API group changed from `mcp.hangar.io` to `mcp-hangar.io` in v0.2.0. ```yaml # Before (v0.1.x) apiVersion: mcp.hangar.io/v1alpha1 kind: MCPProvider # After (v0.2.0+) apiVersion: mcp-hangar.io/v1alpha1 kind: MCPProvider ``` **Action:** 1. Update all MCPProvider, MCPProviderGroup, and MCPDiscoverySource manifests. 2. Delete old CRDs: `kubectl delete crd mcpproviders.mcp.hangar.io` 3. Install new CRDs from the updated Helm chart or `make install` in the operator directory. 4. Re-apply your custom resources with the new API group. --- ## Environment variables **Applies to:** all versions ### Prefix migration: HANGAR\_\* to MCP\_\* The canonical environment variable prefix is `MCP_*`. The old `HANGAR_*` prefix is not supported in v1.0. | Old | New | |-----|-----| | `HANGAR_CONFIG` | `MCP_CONFIG` | | `HANGAR_MODE` | `MCP_MODE` | | `HANGAR_HTTP_HOST` | `MCP_HTTP_HOST` | | `HANGAR_HTTP_PORT` | `MCP_HTTP_PORT` | | `HANGAR_LOG_LEVEL` | `MCP_LOG_LEVEL` | | `HANGAR_JSON_LOGS` | `MCP_JSON_LOGS` | **Action:** Search your shell profiles, `.env` files, Docker Compose files, Kubernetes ConfigMaps/Secrets, and CI pipelines for `HANGAR_` and replace with `MCP_`. ### Langfuse environment variables The Langfuse integration variables also follow the `MCP_*` convention in v1.0: | Old | New | |-----|-----| | `HANGAR_LANGFUSE_ENABLED` | `MCP_LANGFUSE_ENABLED` | | `HANGAR_LANGFUSE_SAMPLE_RATE` | `MCP_LANGFUSE_SAMPLE_RATE` | | `HANGAR_LANGFUSE_SCRUB_INPUTS` | `MCP_LANGFUSE_SCRUB_INPUTS` | | `HANGAR_LANGFUSE_SCRUB_OUTPUTS` | `MCP_LANGFUSE_SCRUB_OUTPUTS` | --- ## Repository URL migration (v0.7.0) **Applies to:** upgrading from v0.6.x or earlier All repository URLs migrated from `github.com/mapyr` to `github.com/mcp-hangar` in v0.7.0. This affects: - Git remote URLs - Go module import paths - Container image references (GHCR) - Documentation links - Helm chart source URLs **Action:** Update any pinned references to the old GitHub organization. ```bash # Check for old references grep -r "mapyr" . --include="*.yaml" --include="*.yml" --include="*.toml" # Go modules: update go.mod # Old: github.com/mapyr/... # New: github.com/mcp-hangar/... ``` --- ## Configuration changes (v0.6.0+) Several new configuration sections were added between v0.6.0 and v0.8.0. These are all opt-in with sensible defaults, so existing config files continue to work. Review these if you want to take advantage of new capabilities. ### Hot-reload configuration (v0.6.6) ```yaml # New section -- optional, enabled by default config_reload: enabled: true use_watchdog: true interval_s: 5 ``` ### Response truncation (v0.6.3) ```yaml # New section -- optional, disabled by default truncation: enabled: false max_batch_size_bytes: 950000 cache_driver: memory # memory | redis cache_ttl_s: 300 ``` ### Execution concurrency (v0.7.0) ```yaml # New section -- optional execution: max_concurrency: 50 # global limit default_provider_concurrency: 10 # per-provider default providers: my_provider: max_concurrency: 5 # per-provider override ``` ### Tool access filtering (v0.8.0) ```yaml # New per-provider section -- optional providers: grafana: tools: deny_list: - "delete_*" - "create_alert_rule" allow_list: - "query_*" ``` --- ## bootstrap() API change (v0.3.0) **Applies to:** upgrading from v0.2.x or earlier The `bootstrap()` function now accepts an optional `config_dict` parameter for programmatic configuration. This is backward compatible -- existing calls without the parameter continue to work. If you were monkey-patching configuration, use this parameter instead: ```python # Before import mcp_hangar.server.config as cfg cfg._global_config = my_config bootstrap() # After bootstrap(config_dict=my_config) ``` --- ## Enterprise module split **Applies to:** v1.0 (new in this release) Starting with v0.13.0, enterprise features (auth, RBAC, behavioral profiling, compliance export, Langfuse integration) moved from the core package to the `enterprise/` directory. As of v1.3.0, the `enterprise/` directory was absorbed back into `src/mcp_hangar/` and the entire codebase is licensed under MIT. ### What moved | Feature | Old location | New location | |---------|-------------|--------------| | API key stores, JWT/OIDC, RBAC | `src/mcp_hangar/infrastructure/auth/` | `enterprise/auth/` | | Role definitions | `src/mcp_hangar/domain/security/roles.py` | `enterprise/auth/roles.py` | | Auth REST endpoints | `src/mcp_hangar/server/api/auth/` | `enterprise/auth/api/` | | Auth bootstrap wiring | `src/mcp_hangar/server/auth_bootstrap.py` | `enterprise/auth/bootstrap.py` | | Tool access policy enforcement | `src/mcp_hangar/domain/value_objects/tool_access_policy.py` | `enterprise/policies/` (interface stays in core) | | SQLite/Postgres event stores | `src/mcp_hangar/infrastructure/persistence/event_store.py` | `enterprise/persistence/` | | Langfuse integration | `src/mcp_hangar/infrastructure/observability/langfuse_adapter.py` | `enterprise/integrations/langfuse.py` | ### Impact on deployments - **All users:** No license key or tier distinction applies. All features (provider lifecycle, health checks, circuit breaker, groups, load balancing, failover, Prometheus metrics, OTEL export, CLI, hot-reload, batch invocations, auth, RBAC, behavioral profiling, compliance export, Langfuse integration) are unconditionally available under MIT. ### Import boundary Core code never imports from `enterprise/`. If you have custom code that imports from internal paths that moved to `enterprise/`, update your imports: ```python # Before (if you imported internal auth modules directly) from mcp_hangar.infrastructure.auth.api_key_store import SQLiteApiKeyStore # After -- use the contract interface from core from mcp_hangar.domain.contracts import IApiKeyStore # The concrete implementation is loaded by bootstrap when licensed ``` --- ## Deprecated patterns removed in v1.0 The following were deprecated in earlier versions and are removed in v1.0: | Deprecated | Replacement | Removed in | |-----------|-------------|------------| | `provider_manager.py` | `Provider` aggregate | v1.0 | | `ProviderSpec` | `Provider` constructor | v1.0 | | `ProviderConnection` | `Provider` aggregate | v1.0 | | `ProviderHealth` in `models.py` | `HealthTracker` | v1.0 | | `setup_fastmcp_server()` | `MCPServerFactory` | v0.4.0 | | `create_fastmcp_server()` | `MCPServerFactory.create_server()` | v0.4.0 | | `run_fastmcp_server()` | `MCPServerFactory.create_asgi_app()` | v0.4.0 | | `RegistryFunctions` | `HangarFunctions` | v0.4.0 | | `with_registry()` | `with_hangar()` | v0.4.0 | **Action:** Search your code for these names. If any are found, replace them before upgrading. ```bash # Quick check for deprecated patterns grep -rn "ProviderSpec\|ProviderConnection\|ProviderHealth\|provider_manager" \ --include="*.py" your_project/ grep -rn "setup_fastmcp_server\|create_fastmcp_server\|run_fastmcp_server" \ --include="*.py" your_project/ grep -rn "RegistryFunctions\|with_registry\|registry_list\|registry_invoke" \ --include="*.py" your_project/ ``` --- ## Kubernetes operator upgrade **Applies to:** users running the MCP Hangar operator in Kubernetes ### CRD updates The operator CRDs remain at `v1alpha1` in v1.0. A future release will introduce `v1beta1` with conversion webhooks (tracked as task 11.10). If upgrading from v0.1.x, you must update the API group as described in [Kubernetes operator API group](#kubernetes-operator-api-group-v020). ### Helm chart upgrade ```bash # 1. Back up current values helm get values mcp-hangar -n mcp-hangar > values-backup.yaml # 2. Update the chart repository helm repo update mcp-hangar # 3. Review changes helm diff upgrade mcp-hangar mcp-hangar/mcp-hangar \ -n mcp-hangar -f values-backup.yaml # 4. Apply helm upgrade mcp-hangar mcp-hangar/mcp-hangar \ -n mcp-hangar -f values-backup.yaml ``` ### Helm values changes Review your `values.yaml` for these additions in the mcp-hangar chart: ```yaml # Authentication (required for enterprise features) config: auth: jwtSecret: "" # Set via secret reference, not plaintext # Database (if using Postgres event store) postgresql: enabled: true # Autoscaling (new) autoscaling: enabled: false minReplicas: 2 maxReplicas: 10 ``` --- ## Observability upgrade ### Grafana dashboards If you use the bundled Grafana dashboards from `monitoring/`, replace them with the versions from v1.0. Key changes since v0.4.0: - All metric names use `mcp_hangar_*` prefix (not `mcp_registry_*`). - New dashboards: `alerts.json`, `provider-details.json` (added v0.6.4). - Alert count reduced from 28 to 19 in v0.6.4 (removed alerts for not-yet-populated metrics). - Updated thresholds: P95 latency 5s to 3s, P99 10s to 5s, batch slow 60s to 30s. ### Prometheus alert rules Replace `monitoring/alerts.yaml` with the v1.0 version. If you have custom rules, update metric names: ```yaml # Before - alert: MCPRegistryToolCallSlow expr: mcp_registry_tool_call_duration_seconds > 5 # After - alert: MCPHangarToolCallSlow expr: mcp_hangar_tool_call_duration_seconds > 5 ``` ### New metrics (v0.5.0 - v0.12.0) These metrics were added after v0.4.0. They are available automatically -- no configuration change needed, but you may want to add dashboard panels: | Metric | Added in | Description | |--------|----------|-------------| | `mcp_hangar_batch_calls_total` | v0.5.0 | Batch invocation count | | `mcp_hangar_batch_duration_seconds` | v0.5.0 | Batch execution time | | `mcp_hangar_batch_concurrency_gauge` | v0.5.0 | Current parallel executions | | `mcp_hangar_batch_inflight_calls` | v0.7.0 | Global in-flight call gauge | | `mcp_hangar_batch_concurrency_wait_seconds` | v0.7.0 | Slot acquisition wait time | | `mcp_hangar_tool_access_denied_total` | v0.8.0 | Tool access policy denials | | `mcp_hangar_tool_access_policy_evaluations_total` | v0.8.0 | Policy evaluations | | `mcp_hangar_rate_limit_hits_total` | v0.6.5 | Rate limiter triggers | | `mcp_hangar_http_requests_total` | v0.6.5 | HTTP client requests | --- ## Step-by-step upgrade procedure ### PyPI package users ```bash # 1. Check current version pip show mcp-hangar # 2. Upgrade pip install --upgrade mcp-hangar==1.0.0 # or with uv: uv pip install mcp-hangar==1.0.0 # 3. Verify mcp-hangar --version # 4. Test configuration mcp-hangar serve --dry-run # if available, or start and check logs # 5. Update environment variables (see sections above) # 6. Update any custom code imports (see sections above) # 7. Restart mcp-hangar serve ``` ### Docker users ```bash # 1. Pull new image docker pull ghcr.io/mcp-hangar/mcp-hangar:1.0.0 # 2. Update docker-compose.yml image tag # image: ghcr.io/mcp-hangar/mcp-hangar:1.0.0 # 3. Update environment variables in docker-compose.yml # Replace HANGAR_* with MCP_* # 4. Restart docker compose up -d ``` ### Kubernetes users ```bash # 1. Back up CRDs and custom resources kubectl get mcpproviders -A -o yaml > mcpproviders-backup.yaml kubectl get mcpprovidergroups -A -o yaml > mcpprovidergroups-backup.yaml kubectl get mcpdiscoverysources -A -o yaml > mcpdiscoverysources-backup.yaml # 2. Update Helm chart helm repo update helm upgrade mcp-hangar mcp-hangar/mcp-hangar -n mcp-hangar -f values.yaml # 3. Verify operator is running kubectl get pods -n mcp-hangar kubectl logs -n mcp-hangar deploy/mcp-hangar-operator # 4. Verify CRDs kubectl get crd | grep mcp-hangar # 5. Check provider status kubectl get mcpproviders -A ``` --- ## Troubleshooting ### "ModuleNotFoundError: No module named 'mcp_hangar.provider_manager'" The `provider_manager` module was removed. See [Deprecated patterns removed in v1.0](#deprecated-patterns-removed-in-v10). ### "NameError: name 'RegistryFunctions' is not defined" The old registry names were removed in v0.4.0. See [Rebrand: registry to hangar](#rebrand-registry-to-hangar-v040). ### "Unknown environment variable HANGAR_*" v1.0 only reads `MCP_*` variables. See [Environment variables](#environment-variables). ### CRD conflicts after operator upgrade If old CRDs from the `mcp.hangar.io` API group remain, delete them manually: ```bash kubectl delete crd mcpproviders.mcp.hangar.io kubectl delete crd mcpprovidergroups.mcp.hangar.io kubectl delete crd mcpdiscoverysources.mcp.hangar.io ``` Then reinstall CRDs from the updated chart. --- ## Getting help - GitHub Issues: https://github.com/mcp-hangar/mcp-hangar/issues - Changelog: See `CHANGELOG.md` for the complete version history. - Architecture: See `ARCHITECTURE.md` for system design documentation.