LiteLLM
Self-hosted gateway that tracks spend per key, team and model, and stops requests at budget.
What it is
LiteLLM is Berrie AI's open-source AI gateway that puts OpenAI, Anthropic, Vertex AI, Bedrock and 140+ other providers and 1,800+ models behind a single OpenAI-compatible API. It ships with an admin UI for issuing keys, splitting teams, watching spend and capping budgets. Teams no longer need a separate SDK and a separate set of credentials for every model.
How it runs
The shortest path is Docker — two commands bring up the gateway itself:
docker pull docker.litellm.ai/berriai/litellm:latest
docker run -p 4000:4000 litellmThe page also offers a Helm chart and a Terraform module, plus database-bundled and non-root image variants; you can connect your own Postgres and Redis, autoscale with Kubernetes, or deploy in a fully air-gapped environment. The Rust gateway is live on the page as well.
Core mechanics
- Unified interface: swapping models only changes gateway configuration, application code stays put; new models are picked up day-zero.
- Accounting and hard budgets: usage and spend are attributed by key, user, team, org, model and MCP; budgets reset daily or monthly and requests stop once the cap is hit; keys can also carry TPM/RPM limits and leaked-key protection.
- Routing for lower cost: load balancing across providers, regions and keys, lowest-cost deployment routing, Auto Routing that sends simple prompts to cheaper models, plus semantic caching (Redis/S3/GCS) and prompt compression.
- Guardrails and audit: PII masking, prompt injection, secret detection and other guardrails run at pre-call / during-call / post-call, and every request lands in an audit log.
- More than LLMs: MCP servers and agents go through the same gateway and the same key.
Who it is for
First, platform teams that want to spread model access across the whole company without every step going through procurement and security review; second, teams that switch between several providers and need to charge AI spend back to each business line.
Boundaries
The open-source core is MIT-licensed, and self-hosting means running your own Postgres and Redis (or choosing the database-bundled image). SSO, RBAC, audit logs, SCIM and SLA belong to the paid Enterprise offering, not to a different core. The performance numbers come from the vendor's own AI Gateway Bench: 0.66 ms added p99 latency.