# Shimmy โ Local Inference, OpenAI-Compatible
### ๐ Local GPU inference โ one binary, 100% Rust ๐
[](https://opensource.org/licenses/MIT)
[](https://github.com/Michael-A-Kuykendall/shimmy/security)
[](https://github.com/Michael-A-Kuykendall/shimmy/actions/workflows/ci.yml)
[](https://crates.io/crates/shimmy)
[](https://crates.io/crates/shimmy)
[](https://rustup.rs/)
[](https://github.com/Michael-A-Kuykendall/shimmy/stargazers)
[](https://github.com/sponsors/Michael-A-Kuykendall)
[](https://translifeline.org/)
[](https://www.thetrevorproject.org/)
**Languages:** [็ฎไฝไธญๆ](docs/zh-CN/README.md) ยท [็น้ซไธญๆ](docs/zh-TW/README.md)
Shimmy is independently maintained and free forever. [Sponsorship](https://github.com/sponsors/Michael-A-Kuykendall) funds certification, compatibility work, and releases.
**Shimmy will be free forever.** No asterisks. No "free for now." No pivot to paid.
---
## What Is Shimmy?
Shimmy is a **single-binary** OpenAI-compatible inference server for GGUF models. Point your existing AI tools at Shimmy and they just work โ locally, privately, and free.
**Shimmy is the server. Airframe is the engine.** Under the hood, Shimmy runs on [**Airframe**](https://github.com/Michael-A-Kuykendall/airframe) (v0.4.4), a pure-Rust WebGPU (WGSL) transformer engine. No C++ toolchain, no Python runtime, no backend flags. 26 models certified across 12 families. Version history: [CHANGELOG](CHANGELOG.md) ยท [Airframe CHANGELOG](https://github.com/Michael-A-Kuykendall/airframe/blob/main/CHANGELOG.md).
**Why this matters:**
- No Python runtime or C++ toolchain โ Rust only, top to bottom
- F32 accumulation precision with deterministic output (same model + seed + params โ same output)
- WGSL compute shaders via WebGPU โ NVIDIA, AMD, Intel, integrated GPUs, Apple Silicon
- Model spec auto-derived from GGUF metadata โ no hardcoded per-model constants
- YaRN RoPE scaling for extended context via `SHIMMY_MAX_CTX` (see [Extended Context](docs/EXTENDED_CONTEXT.md))
---
## ๐ฏ Supported Models
**12 model families ยท 26 certified model/quant combinations** โ every model below passes Shimmy's 3-box certification regimen (MATH + INFERENCE + DETERMINISM) against the certification ledger. Certification applies to the named model/quant combination; architecture recognition does not automatically mean certification. GGUF files load as-is; no recompilation, no hardcoded per-model constants.
| Family | Model | Quants |
|---|---|---|
| **Llama** | [Llama-3.2-1B-Instruct](https://huggingface.co/bartowski/Llama-3.2-1B-Instruct-GGUF) | Q4_K_M ยท Q6_K |
| | [Llama-3.2-3B-Instruct](https://huggingface.co/bartowski/Llama-3.2-3B-Instruct-GGUF) | Q4_K_M |
| | [Llama-3.1-8B-Instruct](https://huggingface.co/bartowski/Llama-3.1-8B-Instruct-GGUF) | Q4_K_M |
| | [TinyLlama-1.1B-Chat](https://huggingface.co/TheBloke/TinyLlama-1.1B-Chat-v1.0-GGUF) | Q4_0 ยท Q5_K_M ยท Q6_K |
| **Qwen3** | [Qwen3-0.6B](https://huggingface.co/Qwen/Qwen3-0.6B-GGUF) | Q4_K_M |
| | [Qwen3-1.7B](https://huggingface.co/Qwen/Qwen3-1.7B-GGUF) | Q4_K_M |
| | [Qwen3-4B](https://huggingface.co/Qwen/Qwen3-4B-GGUF) | Q4_K_M |
| | [Qwen3-4B-Thinking](https://huggingface.co/Qwen/Qwen3-4B-Thinking-GGUF) | Q4_K_M |
| | [Qwen3-8B](https://huggingface.co/Qwen/Qwen3-8B-GGUF) | Q4_K_M |
| **Qwen2** | [Qwen2-0.5B-Instruct](https://huggingface.co/Qwen/Qwen2-0.5B-Instruct-GGUF) | Q4_K_M |
| | [Qwen2-1.5B-Instruct](https://huggingface.co/Qwen/Qwen2-1.5B-Instruct-GGUF) | Q4_K_M |
| | [Qwen2-7B-Instruct](https://huggingface.co/Qwen/Qwen2-7B-Instruct-GGUF) | Q4_K_M |
| **Qwen3.5** | [Qwen3.5-9B](https://huggingface.co/Qwen/Qwen3.5-9B-GGUF) | Q4_K_M |
| **Phi-3** | [Phi-3.5-mini-Instruct](https://huggingface.co/microsoft/Phi-3.5-mini-instruct-gguf) | Q4_K_M |
| | [Phi-3-mini-4k-Instruct](https://huggingface.co/microsoft/Phi-3-mini-4k-instruct-gguf) | Q4_0 |
| **Phi-2** | [Phi-2](https://huggingface.co/TheBloke/phi-2-GGUF) | Q4_K_M |
| **Gemma-2** | [Gemma-2-2B-it](https://huggingface.co/bartowski/gemma-2-2b-it-GGUF) | Q4_K_M |
| | [Gemma-2-9B-it](https://huggingface.co/bartowski/gemma-2-9b-it-GGUF) | Q4_K_M (supported; cert: see [v2-roadmap](docs/v2-roadmap.md)) |
| **Gemma-4** | [Gemma-4-12B-coder](https://huggingface.co/google/gemma-4-12B-coder-GGUF) | Q4_K_M |
| | [Gemma-4-E4B](https://huggingface.co/google/gemma-4-E4B-it-GGUF) | Q4_K_M |
| **DeepSeek-R1** | [DeepSeek-R1-0528-Qwen3-8B](https://huggingface.co/deepseek-ai/DeepSeek-R1-Distill-Qwen-8B-GGUF) | Q4_K_M |
| **Ministral** | [Ministral-3-14B-Reasoning](https://huggingface.co/bartowski/Ministral-3-14B-Reasoning-GGUF) | Q4_K_M |
| **StarCoder2** | [StarCoder2-3B](https://huggingface.co/second-state/StarCoder2-3B-GGUF) | Q4_K_M |
**SafeTensors format** (`.safetensors`) is supported for model loading via `safetensors_native`. Full Airframe-native inference for SafeTensors remains roadmap work; see [docs/v2-roadmap.md](docs/v2-roadmap.md).
## Features
- **โก [TurboShimmy INT4 KV Cache](docs/turboshimmy.md)** โ About 7ร lower KV-cache memory in tested configurations. Run Llama-3.2-3B on 4 GB GPUs.
- **๐ [OpenAI SDK Compatibility](docs/INTEGRATION.md)** โ Chat completions, text completions, streaming, and model endpoints. Works with OpenAI SDKs and tools using that surface.
- **๐ง [Extended Context](docs/EXTENDED_CONTEXT.md)** โ YaRN RoPE scaling via `SHIMMY_MAX_CTX`.
- **๐ฆ [Migrating from v1.x](docs/MIGRATION_v2.md)** โ llama.cpp, MLX, HuggingFace, and RustChain backends removed in v2.0+. Shimmy is now a pure Airframe product.
- **๐ Certification** โ Every model passes a 3-box certification regimen (MATH + INFERENCE + DETERMINISM). See [docs/CERTIFICATION.md](docs/CERTIFICATION.md).
- **๐ง MOE support** โ Mixture-of-Experts CPU offloading is on the Airframe roadmap.
---
## Quick Start
Local loopback access needs no key. Network binds require `SHIMMY_API_KEY`;
browser clients must also be listed in `SHIMMY_ALLOWED_ORIGINS`.
See the [network-access instructions](docs/quickstart.md#network-access-and-api-keys).
```bash
cargo install shimmy --locked
shimmy serve --model-path /absolute/path/to/model.gguf --bind 127.0.0.1:11435
```
Then in another terminal:
```bash
shimmy list --short
curl -s http://127.0.0.1:11435/v1/chat/completions \
-H 'Content-Type: application/json' \
-d '{"model":"tinyllama-1.1b","messages":[{"role":"user","content":"Say hi in 5 words."}],"max_tokens":32}'
```
Full install, model acquisition, GPU, VRAM sizing, platform-specific builds: **[docs/quickstart.md](docs/quickstart.md)**
---
## Documentation
| Start here | What you need |
|---|---|
| [Quick Start](docs/quickstart.md) | Install, models, GPU, VRAM |
| [Supported Models](docs/SUPPORTED_MODELS.md) | Certified models and quantization |
| [API Compatibility](docs/OPENAI_COMPAT.md) | Endpoints, SDKs, integration |
| [Configuration](docs/CONFIGURATION.md) | Env vars and config options |
| [Troubleshooting](docs/TROUBLESHOOTING.md) | GPU errors, model failures |