llm-gateway

Source on GitHub

One OpenAI-compatible endpoint in front of OpenAI, Gemini, Anthropic and Ollama. Apps ask for an alias, the gateway picks a provider and moves to the next one when it fails.

POST /v1/chat/completions  model: demo → gemini/gemini-3.5-flash-lite

Don't enter personal information, prompts go to Google's free API.

The answer shows up here with the gateway's response headers. x-gateway-provider says which provider served it.

What the gateway does

Fallback chains
gateway.yaml maps an alias to a list of provider/model targets, tried top to bottom.
Retries
429s, 5xx and network errors are retried with backoff, then the next provider takes over.
One format
Gemini SSE, Anthropic events and Ollama NDJSON all come out as OpenAI chunks.
Rate limits and cache
Limits per client, plus an exact-match cache for repeated requests.
Cost log
Every request records tokens, latency and estimated cost in Postgres.

Call it yourself

curl -i ORIGIN/v1/chat/completions \
  -H 'content-type: application/json' \
  -d '{"model":"demo","messages":[{"role":"user","content":"hi"}]}'

Limits of this demo

This is the real gateway with one alias on a free Gemini key, so no API key is needed here. Other models and tool calls are refused. Messages can be 2000 bytes, answers 256 tokens. Each IP gets 5 requests a minute and 20 a day, and the whole demo stops at 100 a day (UTC).

Prompts are not stored here, but they do go to Google's free API, so leave out anything private.