When self-hosting an AI model is worth it

By Anwar Benhamada · August 6, 2026

The pitch is compelling: stop paying per token, run an open model on your own hardware, own the whole stack.

Sometimes that’s right. Usually the arithmetic doesn’t survive contact with the costs people leave out.

The comparison people run

API costs $X per million tokens. A GPU costs $Y per month. At my volume, $Y < $X, therefore self-host.

The problem is that $Y is never just the GPU.

What’s missing from $Y

Utilisation. You pay for the GPU continuously. You use it in bursts. If your workload runs eight hours a day, your effective cost is three times the sticker price — and most workloads are far spikier than that.

Engineering time. Deployment, model updates, monitoring, scaling, the inference server, quantisation decisions. This is the largest cost and the one never counted. A day a month of engineering time exceeds most teams’ entire API bill.

Redundancy. One GPU is a single point of failure. Two is double the cost.

Capability gap. Open models are good and generally behind frontier models. If output quality drops enough to need more human review, you’ve moved cost from a line item to your team’s time, where it’s invisible and larger.

Cold starts. Loading a large model takes time. Keeping it warm means paying for idle.

When self-hosting genuinely wins

1. Data cannot leave your infrastructure. Regulatory, contractual, or customer-mandated. Then it’s not a cost decision — the API isn’t an option, and you’re comparing self-hosting to not doing it at all.

2. Sustained high, predictable volume. Millions of requests, running continuously, on a task where a smaller open model is genuinely sufficient. High utilisation is what makes the arithmetic work, and it’s exactly what most workloads lack.

3. The task is narrow and a small model is enough. Classification, entity extraction, structured tagging. A fine-tuned small model can beat a frontier model on a narrow task while costing a fraction — and this is by far the most underrated case. You don’t need a large model to decide whether an email is a complaint.

When it doesn’t

• Spiky or unpredictable volume
• General-purpose reasoning where the capability gap matters
• Small teams without someone who wants to own inference infrastructure
• Early-stage work where requirements are still moving

That last one especially: self-hosting locks in an architecture before you know what you need.

The honest calculation

Self-host  = GPU (24/7) + storage + egress
           + (engineering hours × loaded rate)
           + redundancy
           + quality-gap review time

API        = tokens × price + near-zero ops

Run it at your realistic utilisation, not your peak. Most people are surprised by how far ahead the API stays.

The middle path

Route by task. Self-host the narrow, high-volume, easy work — classification, extraction, routing. Send the hard general work to an API.

This is where the real savings are, and it doesn’t require betting your whole stack on one decision. Start by measuring which of your calls are actually simple; in most workloads it’s the majority, and they’re consuming a disproportionate share of the bill.

Before you commit

Ask what makes this reversible. Keep your prompts and evaluation sets provider-agnostic, and put an abstraction between your application and whatever serves the model.

Then self-hosting is an experiment you can run and undo — which is a much better position than discovering six months in that your entire pipeline assumes a model you no longer want to operate.