One OpenAI-shaped API in front of every model, with per-key budgets, rate limits, caching, and fallbacks you configure at a boundary instead of in your code.
What it is
LiteLLM is an open-source proxy that exposes 100+ providers and 2500+ models through the OpenAI API format, so your call sites never change when the model does. On top of that translation layer it adds the operational plumbing a fleet of models needs: per-key and per-team budgets, rate limits, response caching, and automatic fallbacks. You self-host it with Docker and get a dashboard for keys, spend, and usage.
The one job
It gives you a single OpenAI-shaped endpoint with budgets and fallbacks attached, so provider choice, cost limits, and failover become configuration at the edge rather than branches scattered through your application.
Reach for it when
Skip it when
A minimal look
import OpenAI from "openai";
// point the OpenAI client at your LiteLLM proxy, not at OpenAI
const client = new OpenAI({
baseURL: "http://localhost:4000", // the LiteLLM proxy
apiKey: process.env.LITELLM_KEY, // a virtual key with a budget
});
// "model" is now any provider LiteLLM knows; budgets + fallbacks apply
const res = await client.chat.completions.create({
model: "claude-sonnet",
messages: [{ role: "user", content: "summarise this ticket" }],
});The principle it teaches
Routing and budgets belong at a boundary, not sprinkled through the code. LiteLLM is Week 3's cost lever made concrete; the course's common/llm.ts seam is the tiny, hand-rolled version of the same idea.
Where it fits your labs
This is a Week 3 tool. When Lab 3 asks you to cap spend and fail over from an expensive model to a cheaper one, LiteLLM is where that budget-and-fallback ladder lives, one config away from the app.