llm-gateway
Source on GitHubOne OpenAI-compatible endpoint in front of OpenAI, Gemini, Anthropic and Ollama. Apps ask for an alias, the gateway picks a provider and moves to the next one when it fails.
POST /v1/chat/completions model: demo → gemini/gemini-3.5-flash-lite
The answer shows up here with the gateway's response headers. x-gateway-provider says which provider served it.
What the gateway does
- Fallback chains
gateway.yamlmaps an alias to a list of provider/model targets, tried top to bottom.- Retries
- 429s, 5xx and network errors are retried with backoff, then the next provider takes over.
- One format
- Gemini SSE, Anthropic events and Ollama NDJSON all come out as OpenAI chunks.
- Rate limits and cache
- Limits per client, plus an exact-match cache for repeated requests.
- Cost log
- Every request records tokens, latency and estimated cost in Postgres.
Call it yourself
curl -i ORIGIN/v1/chat/completions \
-H 'content-type: application/json' \
-d '{"model":"demo","messages":[{"role":"user","content":"hi"}]}'
Limits of this demo
This is the real gateway with one alias on a free Gemini key, so no API key is needed here. Other models and tool calls are refused. Messages can be 2000 bytes, answers 256 tokens. Each IP gets 5 requests a minute and 20 a day, and the whole demo stops at 100 a day (UTC).
Prompts are not stored here, but they do go to Google's free API, so leave out anything private.