English | ็ฎไฝไธญๆ | ๆฅๆฌ่ช
flowchart LR
C["Retry Current Model"] -->|"fail"| D["Fallback Next Model"];
D -->|"next"| C;
C -->|"success"| E["โ
Return Response"];
D -->|"exhausted"| F["โ Return Error"];
classDef muted fill:#F8FAFC,stroke:#CBD5E1,color:#64748B,stroke-width:1px;
classDef flow fill:#FFF7ED,stroke:#EA580C,color:#7C2D12,stroke-width:3px;
classDef flow2 fill:#FEF3C7,stroke:#D97706,color:#78350F,stroke-width:3px;
classDef ok fill:#DCFCE7,stroke:#16A34A,color:#14532D,stroke-width:1.5px;
classDef bad fill:#FEE2E2,stroke:#DC2626,color:#7F1D1D,stroke-width:1.5px;
class C flow;
class D flow2;
class E ok;
class F bad;
HydraLLM is a high-performance LLM API proxy with automatic retry and model fallback across OpenAI-compatible, Anthropic, and AWS Bedrock providers.
When a request fails, HydraLLM retries the current model, then falls back to the next configured model until success or exhaustion.
Tip
For complete configuration fields and examples, see CONFIGURATION.md.
- Automatic retry + fallback for coding and agent workloads.
- Multi-provider support: OpenAI-compatible, Anthropic, and AWS Bedrock.
- Single local endpoint with stable client integration while model chains evolve.
brew install fang2hou/tap/hydrallmgo install github.com/fang2hou/hydrallm@latestDownload from GitHub Releases.
The GLM showcase included with this project exposes two listeners in one config:
- OpenAI-compatible API on
http://127.0.0.1:8101 - Anthropic-compatible API on
http://127.0.0.1:8102
macOS / Linux:
mkdir -p ~/.config/hydrallm
curl -o ~/.config/hydrallm/config.toml \
https://raw.githubusercontent.com/fang2hou/hydrallm/main/showcases/glm-coding-plan.tomlWindows (PowerShell):
New-Item -ItemType Directory -Force -Path "$env:USERPROFILE\.config\hydrallm"
Invoke-WebRequest -Uri "https://raw.githubusercontent.com/fang2hou/hydrallm/main/showcases/glm-coding-plan.toml" -OutFile "$env:USERPROFILE\.config\hydrallm\config.toml"macOS / Linux:
export ZAI_API_KEY="your-api-key"Windows (PowerShell):
$env:ZAI_API_KEY = "your-api-key"hydrallmOpenAI-compatible listener (8101)
curl http://127.0.0.1:8101/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "placeholder",
"messages": [{"role": "user", "content": "Say hello"}]
}'Anthropic-compatible listener (8102)
curl http://127.0.0.1:8102/v1/messages \
-H "Content-Type: application/json" \
-d '{
"model": "placeholder",
"max_tokens": 64,
"messages": [{"role": "user", "content": "Say hello"}]
}'Note
HydraLLM overrides request model with the configured model chain for each listener.
Use Homebrew services for auto-start.
Note
For brew services, configure api_key explicitly in the config file; do not rely on shell environment variables.
brew services start hydrallm
brew services info hydrallm
brew services restart hydrallm
brew services stop hydrallm- macOS:
launchd(starts automatically after login) - Linux:
systemd
| Command | Description |
|---|---|
hydrallm |
Start server |
hydrallm serve |
Start proxy |
hydrallm edit |
Open config in $EDITOR |
hydrallm version |
Print version info |
hydrallm --help |
Show help |
Global flags: --config /path/to/config.toml, --log-level info
Quick diagnostics:
hydrallm --config /path/to/config.toml --log-level debug
brew services list | grep hydrallmconfig validation failed: at least one model must be configured
Add at least one model under [models.<id>].
model "...": provider "..." not found
Ensure each model provider matches a key under [providers.<name>].
listener "...": mixed model types are not allowed
Each listener must contain models of a single API type (openai, anthropic, or bedrock).
Split mixed types across multiple listeners.
Requests return upstream 4xx/5xx
Temporarily set log.include_error_body = true to inspect upstream error details.
MIT