Ludicrous Gateway

Ludicrous Gateway

Move over, single-provider peasants. We're going LUDICROUS SPEED.

Quick Start • Install • Why? • Features • Providers • Routing • CLI

8 Providers Content-Based Routing SSE Streaming npm version MIT License

--- ## What is this? **Ludicrous Gateway** is an LLM API gateway that sits between Claude Code and multiple AI providers. It analyzes your requests and automatically routes them to the best provider based on content, cost, and privacy rules. ``` You: "Claude, review this code for security vulnerabilities" Gateway: *analyzes request* "Security task detected, routing to Claude Opus..." You: "Write unit tests for this function" Gateway: *analyzes request* "Coding task, routing to GLM for cost savings..." You: "Here's my .env file, help me configure it" Gateway: *analyzes request* "Sensitive content! Routing to local Ollama..." ``` ### The Architecture ``` ┌─────────────────────────────────────────────────────────────────────┐ │ │ │ Claude Code ──────▶ LUDICROUS GATEWAY ──────▶ Providers │ │ │ │ │ ANTHROPIC_BASE_URL │ ┌─────────────┐ │ │ localhost:8787 │ │ Analyzer │ │ │ │ │ ─────────── │ │ │ │ │ • Tokens │ ┌──────────┐ │ │ │ │ • Keywords │──▶│ Anthropic│ │ │ │ │ • Patterns │ ├──────────┤ │ │ │ │ • Roles │──▶│ OpenAI │ │ │ │ └─────────────┘ ├──────────┤ │ │ │ │ │ Gemini │ │ │ │ ▼ ├──────────┤ │ │ │ ┌─────────────┐ │ Grok │ │ │ │ │ Router │──▶├──────────┤ │ │ │ │ Rule Engine │ │ Mistral │ │ │ │ └─────────────┘ ├──────────┤ │ │ │ │ DeepSeek │ │ │ │ ├──────────┤ │ │ │ │ Z.AI │ │ │ │ ├──────────┤ │ │ │ │ Ollama │ │ │ │ └──────────┘ │ └─────────────────────────────────────────────────────────────────────┘ ``` --- ## Why Ludicrous? | Problem | Ludicrous Solution | |---------|-------------------| | Claude Max is expensive for simple coding tasks | Route coding to cheaper GLM providers | | Sensitive data shouldn't leave your machine | Auto-route secrets to local Ollama | | Different tasks need different models | Role-based routing (Planner → Claude, Coder → GLM) | | Provider goes down mid-project | Circuit breaker with automatic failover | | Want to try multiple providers | One config, 8 providers, zero code changes | --- ## Quick Start ### 1. Install ```bash # Install globally (recommended) npm install -g ludicrous-gateway # Or from source git clone https://github.com/sashabogi/ludicrous-gateway cd ludicrous-gateway && pnpm install && pnpm build ``` ### 2. Setup (Interactive Wizard) ```bash ludicrous-gateway setup ``` The wizard walks you through: - Selecting and configuring providers - Testing connections - Setting up routing rules - Creating your config file ### 3. Start the Gateway ```bash ludicrous-gateway serve ``` ### 4. Point Claude Code at It ```bash export ANTHROPIC_BASE_URL="http://localhost:8787" claude "Hello, routed world!" ``` That's it. Claude Code now routes through your gateway. --- ## Installation Scope ### System-Wide (Recommended) Install once, use everywhere. Config lives at `~/.config/ludicrous-gateway/config.yaml`. ```bash npm install -g ludicrous-gateway ludicrous-gateway setup # Creates system-wide config ludicrous-gateway serve # Run from anywhere ``` All your projects share the same gateway and routing rules. ### Per-Project Need different routing rules for different projects? Create a local config. ```bash cd my-project ludicrous-gateway setup # Or create ./config/local.yaml manually ludicrous-gateway serve # Uses project config (takes priority) ``` **Config priority (highest first):** 1. `./config/local.yaml` - project-level 2. `./config/config.yaml` - project-level 3. `./ludicrous-gateway.yaml` - project-level 4. `~/.config/ludicrous-gateway/config.yaml` - system-wide fallback **Use case examples:** - **System-wide**: Most users - set up once, works everywhere - **Per-project**: Air-gapped project that must use only Ollama - **Per-project**: Client project with specific provider requirements --- ## Features ### Content-Based Routing Route requests based on what's in them: ```yaml # Privacy: Keep secrets local - name: "privacy-first" conditions: keywords: ["password", "API_KEY", ".env", "secret"] route_to: provider: "ollama" priority: 100 # Cost: Big requests → cheaper providers - name: "cost-saver" conditions: min_tokens: 10000 route_to: provider: "zai" priority: 80 ``` ### Role-Based Routing The setup wizard includes 10 pre-configured agent roles: | Role | Default Provider | Why | |------|-----------------|-----| | Planner/Architect | Anthropic | Complex reasoning needs the best | | Security Analyst | Anthropic | Security is not where you cut corners | | Adversarial/Red Team | DeepSeek | Great at finding flaws | | Debugger | Anthropic | Needs deep understanding | | Code Reviewer | Anthropic | Quality matters | | Researcher | Z.AI | Good enough, way cheaper | | Tester | Z.AI | Test writing is mechanical | | Coder | Z.AI | Most coding is straightforward | | Refactorer | Z.AI | Pattern-based work | | Documentation | Z.AI | Prose generation | ### Subscription Passthrough Using Claude Max? The gateway passes your subscription headers directly to Anthropic - no API key needed. ```yaml providers: - name: "anthropic" type: "anthropic" subscription_passthrough: true # Magic! ``` ### Circuit Breaker Provider having a bad day? Gateway detects failures and routes around them: ```yaml health: circuit_breaker: failure_threshold: 3 # 3 strikes... recovery_timeout_seconds: 60 # ...then timeout ``` --- ## Providers | Provider | Type | Best For | |----------|------|----------| | **Anthropic** | `anthropic` | Complex reasoning, architecture, security | | **OpenAI** | `openai` | General tasks, good all-rounder | | **Gemini** | `gemini` | Fast responses, good for iteration | | **Grok** | `grok` | Real-time knowledge, creative tasks | | **Mistral** | `mistral` | Code generation (Codestral) | | **DeepSeek** | `deepseek` | Reasoning (R1), cost-effective | | **Z.AI** | `zai` | GLM models, very cost-effective | | **Ollama** | `ollama` | Privacy, offline, free | ### Quick Provider Setup **Anthropic (Claude Max passthrough):** ```yaml - name: "anthropic" type: "anthropic" subscription_passthrough: true ``` **Z.AI (budget coding):** ```yaml - name: "zai" type: "zai" keys: - key: "${ZAI_API_KEY}" model_aliases: claude-opus-4-5-20251101: "glm-4.7" ``` **Ollama (local/private):** ```yaml - name: "ollama" type: "ollama" base_url: "http://localhost:11434" default_model: "qwen3:32b" ``` --- ## Routing Rules Rules are evaluated by priority (highest first). First match wins. ### Condition Types | Condition | What it does | Example | |-----------|--------------|---------| | `keywords` | Match words in content | `["password", "secret"]` | | `patterns` | Regex matching | `["architect.*system"]` | | `min_tokens` | Minimum token count | `10000` | | `max_tokens` | Maximum token count | `1000` | | `models` | Match model names | `["claude-opus-4-5-20251101"]` | ### Example: Multi-Layer Routing ```yaml routing: default_provider: "zai" # Cheap default rules: # Layer 1: Privacy (highest priority) - name: "keep-secrets-local" conditions: keywords: ["password", "API_KEY", ".env", "credentials"] route_to: provider: "ollama" priority: 100 # Layer 2: Complex reasoning - name: "thinking-hard" conditions: patterns: ["architect", "security", "debug", "why is"] route_to: provider: "anthropic" priority: 90 # Layer 3: Code generation (falls through to default) # Everything else → zai (cheap and cheerful) ``` --- ## CLI ```bash # The magic wizard ludicrous-gateway setup # Start serving ludicrous-gateway serve ludicrous-gateway serve --port 9000 --log-level debug # Check what's happening ludicrous-gateway status # Test your routing rules ludicrous-gateway route --tokens 15000 --keywords "password,debug" # Provider management ludicrous-gateway provider list ludicrous-gateway provider add ludicrous-gateway provider test anthropic # Validate config ludicrous-gateway config --print ``` --- ## Configuration Config file locations (searched in order): 1. `./config/local.yaml` 2. `./config/config.yaml` 3. `./ludicrous-gateway.yaml` 4. `~/.config/ludicrous-gateway/config.yaml` Environment variables work with `${VAR_NAME}` syntax. ### Full Example ```yaml server: listen: "127.0.0.1:8787" timeout_ms: 300000 routing: default_provider: "zai" rules: - name: "privacy" conditions: keywords: ["password", "secret", ".env"] route_to: provider: "ollama" priority: 100 providers: - name: "anthropic" type: "anthropic" enabled: true subscription_passthrough: true - name: "zai" type: "zai" enabled: true keys: - key: "${ZAI_API_KEY}" - name: "ollama" type: "ollama" enabled: true base_url: "http://localhost:11434" health: check_interval_seconds: 30 circuit_breaker: failure_threshold: 3 recovery_timeout_seconds: 60 logging: level: "info" format: "text" ``` --- ## API Endpoints | Endpoint | Description | |----------|-------------| | `POST /v1/messages` | Main API (Anthropic-compatible) | | `GET /health` | Provider health status | | `GET /stats` | Request metrics | | `POST /debug/route` | Test routing without executing | --- ## Environment Variables | Variable | Provider | |----------|----------| | `ANTHROPIC_API_KEY` | Anthropic | | `OPENAI_API_KEY` | OpenAI | | `GOOGLE_API_KEY` | Gemini | | `XAI_API_KEY` | Grok | | `MISTRAL_API_KEY` | Mistral | | `DEEPSEEK_API_KEY` | DeepSeek | | `ZAI_API_KEY` | Z.AI | --- ## Development ```bash pnpm install # Install deps pnpm dev # Dev mode with hot reload pnpm build # Build for production pnpm test # Run tests pnpm typecheck # Type check pnpm lint # Lint ``` --- ## Contributing PRs welcome! See [CONTRIBUTING.md](CONTRIBUTING.md). --- ## Inspiration This project was inspired by [cc-relay](https://github.com/omarluq/cc-relay), a Go-based LLM API gateway. We loved the concept but wanted to take it further with: - **TypeScript** for easier contribution and type safety - **Content-based routing** with token estimation, keyword detection, and pattern matching - **Role-based routing** for multi-agent workflows - **8 providers** out of the box (vs 2) - **Interactive setup wizard** for easy onboarding - **Circuit breaker** for automatic failover Thanks to [@omarluq](https://github.com/omarluq) for the original idea of putting a smart proxy between Claude Code and LLM providers. --- ## License MIT - do whatever you want. ---

"I'm going so fast, I'm routing to multiple providers!" - Ludacris, probably