# ๐Ÿ”„ pi-retry **Automatic retry for every error in [pi](https://github.com/earendil-works/pi-coding-agent)** _400/413, connection errors, credit errors, stream exhaustion โ€” retry them all._ [![pi extension](https://img.shields.io/badge/pi-extension-blueviolet)](https://github.com/earendil-works/pi-coding-agent) [![license](https://img.shields.io/badge/license-MIT-blue)](./LICENSE)
--- --- ## Overview This extension automatically detects and retries **all** errors by default, with only a tiny blacklist of known permanent failures (invalid API key, missing model, etc.). | Error Type | Retry Behavior | Use Case | |------------|----------------|----------| | **Any retryable error** (catch-all) | **Indefinite** with capped backoff | Everything else โ€” provider hiccups, stream exhaustion, credit issues, unknown errors | | HTTP 400/413 | **Indefinite** with capped backoff, NO compaction | Transient context overflow that might resolve | | Credit / payment errors | **Indefinite** with capped backoff | "Not Enough Credits", insufficient balance, 402 โ€” top up and the retry loop auto-resumes | | **Quota / session-limit / budget exhaustion** | **Not retried** โ€” notify + stop | "You've hit your limit", `insufficient_quota`, "out of budget", suspended accounts | | Connection errors | **Indefinite** with capped backoff | Network hiccups, connection drops, socket errors, stream exhaustion | | Max tokens (`stopReason: "length"`) | **Auto-continue** indefinitely with hidden continuation turns | Model hits output token limit mid-generation | | Empty / think-only stop (`stopReason: "stop"` with no text or tool calls) | **Nudge once** with a hidden continuation, then give up | Model ends its turn with no usable output (Anthropic empty responses with end_turn, thinking-only turns) | --- ## The Problem By default, pi has built-in retry for some errors (rate limits, 5xx, overloaded), but: 1. **400/413 errors** are treated as context overflow โ†’ triggers compaction but NO retry 2. **Connection errors** sometimes get only limited retries before giving up 3. **Credit errors** ("Not Enough Credits") are never retried 4. **Stream exhaustion** ("Max outbound streams") and other provider-specific errors are never retried 5. **Any unknown error** from a new provider is silently ignored ## The Solution This extension provides **automatic** infinite retry with sensible exponential backoff (2s โ†’ 4s โ†’ 8s โ†’ ... โ†’ 60s max). **Philosophy: retry EVERYTHING by default.** The only things we skip are a tiny blacklist of known permanent failures (invalid API key, model not found, unsupported model, etc.). **Features:** - **Catch-all retry** โ€” Any `stopReason: "error"` is retried, regardless of error message - Automatic detection of 400/413, connection, credit, and stream exhaustion errors - **Auto-continuation** when the model hits its max output tokens (`stopReason: "length"`) โ€” indefinite, no cap, hidden from the TUI - **Indefinite retry** โ€” Keeps retrying until success - **Auto-stop on quota/budget exhaustion** โ€” Session limits, plan quotas, and budget caps ("You've hit your limit", "out of budget", `insufficient_quota`, suspended accounts) are detected and **not** retried, with a notification explaining why - Exponential backoff with cap: max 60s between retries - **Hidden triggers** โ€” provider-valid custom messages use `display: false`, so retries do not add TUI clutter - Manual controls via unified `/retry` command - Non-retryable errors are explicitly logged so you know why we didn't retry --- ## Installation ### Option 1: Install via pi package (Recommended) Install directly from GitHub as a pi package: ```bash pi install https://github.com/monotykamary/pi-retry ``` Or add to your `settings.json`: ```json { "packages": [ "https://github.com/monotykamary/pi-retry" ] } ``` ### Option 2: Global Installation Copy the extension to pi's global extensions directory: ```bash cp retry.ts ~/.pi/agent/extensions/ ``` ### Option 3: Project-Local Installation Copy to your project's `.pi/extensions/` directory: ```bash mkdir -p .pi/extensions cp retry.ts .pi/extensions/ ``` ### Option 4: Quick Test ```bash pi -e ./retry.ts ``` --- ## Usage Once loaded, the extension **automatically** detects and retries all errors. ### Manual Controls | Command | Description | |---------|-------------| | `/retry` | Manually trigger immediate retry (auto-detects: 400/413, credit, connection, max_tokens, or any other error) | | `/retry status` | Show current retry diagnostics for all error types + continuation state | | `/retry reset` | Reset all retry counters and state | --- ## Configuration Edit the constants at the top of `retry.ts`: ```typescript const BASE_DELAY_MS = 2000; // Start with 2 seconds const MAX_DELAY_MS = 60000; // Cap at 60 seconds const BACKOFF_MULTIPLIER = 2; // Double each time // Continuations use a hidden provider-valid custom message ``` --- ## How It Works 1. **Listen to `agent_end` event** โ€” Fires after each agent turn completes 2. **Check for any error** โ€” Examine the last assistant message for `stopReason === "error"` 3. **Blacklist check** โ€” Skip known permanent failures (invalid API key, model not found, quota/session-limit/budget exhaustion, suspended accounts, etc.) 4. **Categorize for messaging** โ€” Classify into 400/413, credit, connection, or other for nice UI notifications 5. **Retry or continue with hidden turns** โ€” Wait with exponential backoff, then trigger a provider-valid custom user turn via `pi.sendMessage()` with `display: false` and `triggerTurn: true` 6. **Valid provider context** โ€” Hidden retry and continuation messages remain in context so providers never receive a trailing assistant message 7. **Indefinite continuation** โ€” Max_tokens auto-continues are uncapped; repeated `length` stops keep producing continuation turns until the model terminates normally 8. **Empty-stop recovery** โ€” A `stop` turn with no usable output (only thinking, or nothing at all) gets exactly one hidden "nudge" continuation, then gives up. Matches Anthropic's documented "empty responses with `end_turn`" remedy (continuation prompt in a new user message) โ€” retrying an empty response in place doesn't help because the model has already decided it's done. 8. **Lifecycle exposure** โ€” Emits `pi-retry:started`, `pi-retry:completed`, and `pi-retry:cancelled` on Pi's shared extension event bus with a matching `retryId`, allowing status integrations to suppress intermediate completion signals The pi's built-in `transform-messages` already strips aborted/errored assistant messages from the LLM context, so the model never sees the failed attempts. --- ## Detected Error Patterns ### Catch-All (Any Error) - **Any** assistant message with `stopReason === "error"` is retried by default - Unknown provider errors, stream errors, unexpected failures โ€” all handled automatically - Only skipped if it matches a known permanent failure (invalid API key, missing model, etc.) ### Non-Retryable (Permanent Failures) These are explicitly **not** retried: - Invalid API key / invalid authentication - API key not found / missing / revoked - Model not found / unknown model / no such model / model does not exist - Unsupported model ### Non-Retryable (Quota / Session Limit / Budget) Exhausted quotas, session limits, and budgets are auto-detected and stop the retry loop (with an explanatory notification), because retrying is pointless until you act or the reset window passes: - **Usage / session limits with reset windows** โ€” "You've hit your limit ยท resets โ€ฆ" and "5-hour limit reached" (Claude Code), "You've hit your usage limit" / "You've exceeded your usage limit" (Codex), "You have hit your ChatGPT usage limit (plus plan)" (ChatGPT subscription caps via the Codex backend, also `usage_limit_reached`) - **Plan / billing quotas** โ€” OpenAI `insufficient_quota`, "You exceeded your current quota, please check your plan and billing details" (OpenAI, Gemini โ€” reached only after pi's built-in 429 retry gives up) - **Google subscription caps** โ€” "You have exhausted your capacity on this model. Your quota will reset after โ€ฆ" (Gemini Code Assist), "You have reached the quota limit for โ€ฆ" / "You can resume using this model at โ€ฆ" (Antigravity) - **Hard allotments** โ€” OpenRouter `free-models-per-day`, Alibaba Coding Plan window quotas "hour/week/month allocated quota exceeded" and free-quota exhaustion "free allocated quota exceeded" (the bare "Allocated quota exceeded" is TPM rate limiting and stays retryable), GitHub Copilot "premium request allowance", z.ai GLM Coding Plan "Usage limit reached for 5 hour" / "no resource package" - **Budget exhaustion** โ€” "out of budget", "Budget has been exceeded" (LiteLLM-style proxies), max/spending/monthly limits - **Suspended accounts** โ€” "Your account โ€ฆ is suspended" (Kimi `exceeded_current_quota_error`) Deliberate distinction: plain pay-as-you-go **balance** errors stay retryable โ€” DeepSeek 402 "Insufficient Balance", OpenRouter 402 "Insufficient credits", Kimi "exceeded your current token quota". A mid-session top-up lets the retry loop auto-resume, whereas session limits and budgets do not self-resolve for hours. ### Max Tokens (stopReason: "length") - The model hit its `max_tokens` / output token limit - The model's response was truncated mid-generation - Auto-continuation sends a provider-valid custom message hidden from the TUI ### 400/413 Errors - HTTP 400 Bad Request - HTTP 413 Payload Too Large - "bad request" messages - "payload too large" messages ### Credit / Payment Errors - "Not Enough Credits" - "insufficient credits" - "insufficient balance" - "out of credits" - "Payment Required" - HTTP 402 status code ### Connection Errors - Connection / network errors - Fetch failures - Socket hang up / socket errors - `ECONNRESET`, `ECONNREFUSED`, `ETIMEDOUT`, `ENOTFOUND` - DNS lookup failures - "Request ended without sending any chunks" - Upstream connect errors - TLS handshake errors - Timeouts awaiting response - Stream exhaustion ("Max outbound streams is 100, 100 open") - Stream limit errors --- ## Development ### Running Tests ```bash # Run all tests npm test # Run tests in watch mode npm run test:watch # Run tests with coverage npm run test:coverage # Type check npm run typecheck # Dead code detection npm run lint:dead ``` ### Project Structure ``` . โ”œโ”€โ”€ retry.ts # Main unified extension โ”œโ”€โ”€ src/ # Shared utilities (testable, DRY) โ”‚ โ”œโ”€โ”€ error-patterns.ts # Error pattern matching, custom types, hasMaxTokensStop โ”‚ โ”œโ”€โ”€ retry-logic.ts # Retry utilities (calculateDelay, RetryState, ContinuationState, etc.) โ”‚ โ””โ”€โ”€ index.ts # Barrel exports โ”œโ”€โ”€ __tests__/ # Unit tests โ”‚ โ””โ”€โ”€ unit/ โ”‚ โ”œโ”€โ”€ error-patterns.test.ts โ”‚ โ””โ”€โ”€ retry-logic.test.ts โ”œโ”€โ”€ vitest.config.ts # Test configuration โ””โ”€โ”€ knip.json # Dead code detection config ``` ### Code Quality ```bash # Run all quality checks npm test # 99 unit tests npm run typecheck # TypeScript type checking npm run lint:dead # Dead code detection with knip ``` --- ## Troubleshooting ### Extension not working? Check that it's loaded in the startup header: ``` Loaded extensions: retry.ts ``` ### Retry not triggering? Use the status command to diagnose: ``` /retry status ``` ### Want to see what's happening? The extensions send notifications on retry attempts. Look at the footer status line for retry status updates. Non-retryable errors are logged as errors so you know why we stopped. ### Too many retries? Use `/retry reset` to clear the counters, or press `Ctrl+C` to abort the session. --- ## Comparison with @georgebashi/pi-retry The npm package `@georgebashi/pi-retry` handles "aborted" streaming errors but explicitly excludes "connection error" (assuming pi's built-in retry handles it). This extension: 1. **Handles ALL errors** via a catch-all โ€” no more playing whack-a-mole with new error patterns 2. **Handles connection errors** that pi might not retry sufficiently 3. **Handles 400/413 errors** without compaction 4. **Handles credit errors** and stream exhaustion They can work together for maximum coverage: ```bash pi install npm:@georgebashi/pi-retry # Plus install this extension ``` --- ## Limitations - Extensions cannot override pi's internal `isRetryableError()` check โ€” they run *after* pi decides not to auto-retry - Error messages remain in the session history (but are invisible to the LLM) - May hit the same error repeatedly if the issue is persistent (use `Ctrl+C` to abort) - **Warning**: Retrying 400/413 without reducing context may fail repeatedly if the payload is genuinely too large - Non-retryable errors (invalid API key, missing model, quota/session-limit/budget exhaustion) are logged but not retried โ€” you'll need to fix the underlying issue, then use `/retry` --- ## Related - [Pi Coding Agent Extensions Docs](https://github.com/badlogic/pi/tree/main/packages/coding-agent/docs/extensions.md) - [@georgebashi/pi-retry](https://github.com/georgebashi/pi-retry) โ€” Handles "aborted" streaming errors - [Issue #252: Connection error with no retry](https://github.com/badlogic/pi-mono/issues/252) ## License MIT