Running AI in Production: A Handbook

The short version

Shipping an AI feature is the start of the work, not the end. Models drift, providers change, prompts rot, and quality degrades silently — there's no stack trace for "the answers got worse". Running it well rests on four habits: 1. **Monitor proxies for quality** — correction rates, refusals, latency, cost-per-call — because you rarely have ground truth in real time. 2. **Run a small online eval** on a golden set, on a schedule, so regressions surface before users complain. 3. **Have an incident runbook** — the levers to pull when it misbehaves, in order. 4. **Build for handover** — so the owning team can run, change, and debug it without the original builders. That's autonomy, and it's the point. The full handbook covers each habit with concrete metrics, thresholds, an incident template, and a keys-in-hand handover checklist.