LLM Serving & Inference
Most agents start by calling an API. Self-hosting is a different muscle — you become the SRE for inference. This lesson is the decision-and-assembly path: when self-hosting pays, then the engine, precision, parallelism, and benchmarking choices that turn a model checkpoint into a production endpoint.
The serving stack
Self-hosting is not one decision; it is a small stack of decisions that compound:
The lesson walks through each layer in order. The benchmark step is what closes the loop — if you skip it, the rest are guesses.
Workload profile: API vs self-hosted
The first question is whether to self-host at all. API providers (Anthropic, OpenAI, Google) handle every layer below for you. Self-host only when one of these is true:
Use API providers when:
✓ Low or bursty volume (often < 500K req/month unless prompts are very long)
✓ Need frontier model quality (GPT-5.x, Claude Opus/Sonnet, Gemini)
✓ Variable / unpredictable traffic
✓ No data residency requirements
✓ Team has no ML infra experience
Self-host when:
✓ High steady volume with a fixed model and high GPU utilization
✓ Strict data privacy / air-gapped environments
✓ Custom fine-tuned models that no API exposes
✓ Latency-critical (< 100ms first token)
✓ Predictable, steady traffic patternsCost crossover. Approximate June 2026 estimates, assuming ~1K input + 1K output tokens per request; re-derive these for your workload because both API and GPU prices move fast.