vLLM & Ollama

What it is

Your laptop, or your production GPUs.

These are two ways to run open-weight models on your own hardware. Ollama is a dead-simple local runner for development and prototyping: install it, pull a model, and it is serving. vLLM is a high-throughput production inference server, with paged attention and continuous batching, built to serve many concurrent requests at scale. They sit at opposite ends of the same job, and both expose OpenAI-compatible endpoints, so your client code does not change between them.

The one job

Serve open models yourself.

Run open-weight models on your own infrastructure: Ollama on your laptop for development, vLLM for production throughput. Same models, same API shape, sized to the moment you are in.

Reach for it when

• Your data cannot leave your infrastructure for privacy or compliance.
• You run high volume where per-token API cost dominates the bill.
• You need latency control that a shared hosted API cannot promise.
• You want offline or local development, which Ollama makes trivial.

Skip it when

• Volume is low: managed APIs are cheaper once you count operations.
• You need frontier quality, where open models still trail on the hardest tasks.
• You have no GPU or ops capacity to run and maintain the server.

A minimal look

Point the same client at localhost.

import OpenAI from "openai";

// Ollama (11434) or vLLM (8000): both speak the OpenAI API
const client = new OpenAI({
  baseURL: "http://localhost:11434/v1",  // local Ollama server
  apiKey: "ollama",                       // unused, but required by the SDK
});

const res = await client.chat.completions.create({
  model: "llama3.1",
  messages: [{ role: "user", content: "classify this ticket" }],
});

The principle it teaches

Where it fits your labs

This is a Week 3 tool. When you weigh cost against control, self-hosting is the "build" side of the ledger: Ollama lets you try an open model locally in minutes, and the same client code would point at vLLM in production.