A month of one rented GPU costs what 21 million API chats do

In May, a blog post titled “Apple Silicon costs more than OpenRouter” reached 355 points on Hacker News, and its comments argued over whether running a model yourself ever comes out cheaper. The version most builders weigh is bigger: rent a data-center GPU, load an open-weight model, and stop paying an API by the token.

For a one-person product that switch costs more until traffic is heavy and steady, because a rented GPU bills for every hour of the month and only gets cheap when it is answering a crowd at the same instant.

We priced GPUs and APIs on the same day, took throughput from published benchmarks, and worked out where the two lines cross.

The short version

• A month of GPU rent buys about 21 million API chats. One B200 at RunPod’s community price of $5.98 an hour costs $4,365 for the month. DeepInfra sells gpt-oss-120b at $0.037 per million tokens in and $0.17 out, so $4,365 pays for about 21 million chats of 1,000 tokens each way.
• A GPU serving a lone user leaves most of itself unused. In NVIDIA’s own tests an H100 running Llama 3.1 8B wrote 220 tokens a second for a single user and 11,553 for 200 users, 52 times as many. Rented at $2.69 an hour, that is $3.39 against $0.065 per 1,000 chats.
• Break-even is a count of requests in flight. In SemiAnalysis’s vLLM runs, gpt-oss-120b on one B200 matches DeepInfra at about 96 requests running at once, all day and all night, and Together, Fireworks or Groq at about 11. Llama 3.3 70B on two H100s needs about 170 against DeepInfra. Llama 3.1 8B on an H100 gets under DeepInfra’s $0.06 only in NVIDIA’s flat-out ceiling test.
• A full GPU on NVIDIA’s own engine beats even DeepInfra. Running TensorRT-LLM, one B200 kept 256 requests in flight at 14,725 tokens a second, about $0.11 per 1,000 chats, roughly half DeepInfra’s price. Holding it there takes about 39 million chats a month.
• Real traffic is lumpy, and lumps hurt the GPU. Azure traces in a Microsoft and University of Illinois paper show peaks 1.7 times the average for a chat service. Size a B200 for that peak on the vLLM curve and the best it can average is about $0.24 per 1,000 gpt-oss chats, above DeepInfra’s $0.207.
• Per-second GPUs trim idle hours and keep the low-load penalty. Modal rents an H100 for $3.95 an hour, billed by the second. Serving users one by one comes to roughly $5 per 1,000 chats, before a 60-second idle window and a cold start Modal says can take minutes.
• A gaming PC you own can beat the API on the power bill, with a batching server. If an RTX 5090 drew its full rated 575 watts at the US average home electricity price, serving a lone user would cost up to about $0.19 of power per million tokens written, three times what DeepInfra charges for the same work on the same 8B model. Serving eight requests together with a batching server cuts that to about $0.03. Under Ollama it barely moves.
• The case for self-hosting is control. Weights you run cannot be retired by a lab, keep data on your machines, and keep a fine-tune trainable as OpenAI closes self-serve fine-tuning to new jobs on January 6, 2027.

Everything below is the evidence, and the cost model ships with this piece as a spreadsheet you can rerun with your own prices.

How we priced it

A “chat” here is one request with 1,000 tokens in and 1,000 tokens out, about a page of context and a page of answer. It is the standard load in the benchmarks we used, so the API price and the GPU throughput describe the same work. Every price is per 1,000 chats.

On the API side we read each provider’s list price on October 9. On the GPU side we took the on-demand hourly price and divided it by the tokens the GPU produces per hour at a given number of simultaneous requests. Data-center throughput comes from two published sources: NVIDIA’s benchmark tables for its NIM containers (Llama 3.1 8B and Llama 3.3 70B) and SemiAnalysis’s InferenceX runs of gpt-oss-120b, mostly on vLLM. NVIDIA’s TensorRT-LLM tables serve as a flat-out ceiling.

Most assumptions lean toward the GPU. Traffic arrives evenly around the clock, and the bill leaves out the engineer’s time, storage, monitoring and the minutes spent loading a model. Two lean the other way: we used on-demand prices, while RunPod sells three- and six-month terms at a discount, and published benchmark settings, which a faster engine can beat.

What the APIs charge

The open models are sold by several hosts at once, and for the same model the price varies by a factor of three to five.

Model	Seller	Per million tokens, in / out	Per 1,000 chats
Llama 3.1 8B	DeepInfra	$0.02 / $0.04	$0.06
gpt-oss-120b	DeepInfra	$0.037 / $0.17	$0.207
gpt-oss-120b	Together, Fireworks or Groq	$0.15 / $0.60	$0.75
Llama 3.3 70B	DeepInfra (FP8)	$0.10 / $0.32	$0.42
Llama 3.3 70B	Together	$1.04 / $1.04	$2.08
Gemini 2.5 Flash-Lite	Google	$0.10 / $0.40	$0.50
GPT-6 Luna	OpenAI	$0.10 / $0.50	$0.60
Claude Haiku 5.5	Anthropic	$0.10 / $0.50	$0.60

Two notes on the closed models. OpenAI’s September 22 price news was the launch of two new models, GPT-6 Sol and GPT-6 Luna, at roughly half the promotional prices of their GPT-5.6 versions; the GPT-5.6 models kept their prices. Haiku’s rate applies to prompts up to 100,000 tokens. Among the open-model hosts, OpenRouter also lists a CoreWeave endpoint for gpt-oss-120b at $0.03 in and $0.17 out; after OpenRouter’s 5.5 percent platform fee it lands about level with DeepInfra. OpenAI, Anthropic and Google also run batch APIs at half price for work that can wait up to a day.

What the GPUs charge

GPU	RunPod community	RunPod secure	Lambda	Vast.ai median	Modal (per second)	AWS
H100	$2.69	$3.99	$4.29	$2.40	$3.95	$6.88
H200	$3.59	$5.29	none listed	$5.00	$4.54	$7.91
B200	$5.98	$7.99	$6.99	$7.81	$6.25	$14.24
RTX 5090	$0.69	$1.19	none listed	$0.61	none listed	none listed

On-demand price per GPU per hour, read October 9. AWS is the instance price divided by its GPU count.

We used RunPod’s community price as the base case: it sits near the bottom of the table, bills by the second and charges nothing for data leaving the machine. Kept running all month, one H100 there costs $1,964 and one B200 $4,365. Vast.ai’s marketplace is cheaper still for some cards, and its interruptible offers, for work that can be stopped, are 50 percent or more cheaper by Vast’s own account.

A GPU answering one request is mostly waiting

A GPU writing tokens for one user spends most of each step moving the model’s weights through memory, and the same pass can serve dozens of users at almost no extra cost. Throughput climbs with the number of requests running at once until the memory fills.

NVIDIA’s own figures show how large that effect is. Llama 3.1 8B on one H100 managed 220 tokens a second serving one person, 4,098 serving 25 and 11,553 serving 200. Llama 3.3 70B on two H100s went from 53 tokens a second for one user to 4,167 for 250, a factor of 78. Rent stays the same at every load, so the cost of a chat falls almost in proportion.

In NVIDIA’s NIM test the 8B model never gets below DeepInfra’s $0.06; its best is $0.065 at 200 at once. NVIDIA’s TensorRT-LLM tables, which measure a flat-out ceiling with no limit on how long each answer takes, put the same H100 at about 15,000 tokens a second, or $0.050 per 1,000 chats: just under the API, with no headroom left for a busy hour. The comparison with GPT-6 Luna looks better for the GPU, which wins from about six requests at once, but those are different models; if a small open model does the job, its own hosted price is the one to beat.

Long prompts push the GPU further behind. With 20,000 tokens in and 2,000 out, the kind of request a document-search feature sends, the same H100 tops out near 1,338 tokens a second. Fully loaded, that is about $1.12 per 1,000 such chats, against $0.48 at DeepInfra, which charges half as much for tokens read as for tokens written.

Where the lines cross

The break-even point is the monthly rent divided by the API price of one chat. It is reachable only if the GPU can serve that many chats at a speed users accept, which makes concurrency the real threshold.

Model on rented GPUs, rent per month	Compared with	Requests in flight to break even	Chats a month at that load
gpt-oss-120b, one B200, $4,365	DeepInfra, $0.207	about 96	about 21.1 million
gpt-oss-120b, one B200, $4,365	Together, Fireworks or Groq, $0.75	about 11	about 5.8 million
Llama 3.3 70B, two H100s, $3,927	DeepInfra, $0.42	about 170	about 9.4 million
Llama 3.3 70B, two H100s, $3,927	Together, $2.08	about 16	about 1.9 million
Llama 3.1 8B, one H100, $1,964	DeepInfra, $0.06	beyond NIM’s measured range	about 32.7 million
Llama 3.1 8B, one H100, $1,964	GPT-6 Luna, $0.60	about 6	about 3.3 million

Requests in flight are interpolated between measured points. Prices per 1,000 chats. The 8B break-even is reachable only at TensorRT-LLM’s flat-out ceiling.

Those crossings come from vLLM’s curve, which SemiAnalysis measured up to 128 requests at once on one B200. The same benchmark ran NVIDIA’s TensorRT-LLM on one B200 with 256 in flight: 14,725 tokens a second, with each user receiving about 59 tokens a second. That works out near $0.11 per 1,000 chats, a little over half of DeepInfra’s, and it takes about 38.7 million chats a month, spread evenly, to keep the card that busy.

For scale, an app with 1,000 chats a day sends about 30,000 a month. At DeepInfra that is a gpt-oss bill of $6.21, and at Together $22.50, against $4,365 for the B200 sitting in a data center waiting for them. The rented B200 starts to pay against the pricier hosts at about 190,000 chats a day, and against DeepInfra at about 700,000.

Uneven traffic moves the line further out

The table assumes requests arrive at the same rate at 3 a.m. as at noon. DynamoLLM, a paper from Microsoft’s Azure researchers and the University of Illinois, measured a week of traffic on an Azure conversation service whose peak load, in tokens per second, was 1.7 times its average, and a coding service at 2.8 times. A GPU has to be sized for the peak, so it averages well below it. We apply the same ratio to requests in flight.

On vLLM’s curve, size one B200 for a peak 1.7 times the average at the 128 requests where that benchmark stopped, and it averages about 75 requests in flight. Its best case is then roughly $0.24 for every 1,000 chats, still more than DeepInfra charges. With a coding service’s 2.8 times peak the best case is about $0.33. There is no TensorRT-LLM curve for one B200 below 256 at once, so the same test cannot be run on it.

A June paper by Chitral Patil makes the same point with its own measurements. On H100 hardware, it found that self-hosting at 1 to 10 requests a second costs 2.5 to 24 times what a full-load estimate says, and that calculators assuming a busy GPU misprice self-hosting, “most severely over-selling it for low-traffic workloads.” Patil also found that the cost calculators and studies it reviewed take full or near-full load as given.

Pay-per-second GPUs

Serverless GPU platforms bill only while a container runs, which removes the empty nights; the low-load penalty stays. An H100 on Modal costs $0.001097 a second, plus a small charge for CPU and memory. Serving Llama 3.1 8B to one user at a time takes about 4.5 seconds per chat, about $5 per 1,000 chats, over 80 times DeepInfra’s rate for that model.

Idle time and start-up add to that. Modal keeps a container running, and billed, for 60 seconds after its last request by default, so a single chat in a quiet hour costs about $0.07 on its own. Its docs say a container is ready once the model has loaded, which “can range from seconds to minutes.” RunPod’s serverless workers bill idle time too, and an endpoint left without traffic for seven days drops to zero workers until its owner raises the limit again. For a chat feature that has to answer when someone shows up, that wait is a product problem as well as a cost.

The gaming PC you already own

A card you already own changes the sums, because the rent drops to the electricity bill. An RTX 5090 is rated at 575 watts for the card alone. At the 18.31 cents per kilowatt-hour that American homes paid on average in July, running it at that full rating costs about 10.5 cents an hour, a ceiling we use for every load below.

A third-party test by QuanTuring, posted on NVIDIA’s developer forum on October 1, measured Llama 3.1 8B on one 5090 with short prompts and answers:

Server on one RTX 5090	1 request	8 at once	32 at once
NIM, FP8 (batching server built on vLLM)	151 tokens/s	1,099	3,817
Ollama, 4-bit model, tuned to 16 slots	172 tokens/s	175	755

One request at a time, the card uses as much as $0.19 of electricity for each million tokens written. DeepInfra charges $0.04 per million written, plus $0.02 per million read, so the same work costs $0.06 there. Under a batching server, eight requests at once bring the power cost down to about $0.03, below the API. Under Ollama, the total output stayed flat from one request to eight in this test, so the power cost per token barely moved.

That leaves out the card itself. Knoop and Holtmann worked out that a $2,000 RTX 5090 serving a million tokens a day would take 8,772 days to pay for itself against GPT-5 nano; at 30 million tokens a day they put it at about 292 days. It also leaves out uptime on a home connection and the risk of opening a port: Cisco found 1,139 Ollama servers reachable from the internet, 214 of them answering prompts with live models.

When self-hosting is worth it anyway

If cost is the reason, the tables above usually say wait. These reasons hold up better.

• A fine-tune you need to keep. OpenAI stopped organizations that had never run fine-tuning from starting jobs on May 7, 2026, and from January 6, 2027 existing customers cannot start new ones either; models already tuned keep serving until their base model is retired. Open weights you fine-tune stay runnable and retrainable for as long as you keep them.
• A model nobody can retire on you. In our ledger of 71 shutdown notices, the median warning was 182 days at OpenAI, 63 at Anthropic and 28 at Google. A model whose weights you hold is retired when you decide.
• Data that may not leave your machines. OpenAI does not train on API data unless a customer opts in, keeps abuse-monitoring logs for up to 30 days unless the law requires longer, and grants zero data retention only on approval. If a customer contract rules out any third party, the price comparison does not apply.
• A full GPU for days at a time. Large offline jobs that keep a card at its peak are where the rental rates can win. Check them against the half-price batch APIs first.

Running it is also a job. vLLM, the open serving engine behind several benchmarks here, published 81 security advisories in the past year, two of them critical, in a year that also saw 32 releases. A stopped RunPod Pod can come back with zero GPUs if capacity has changed while it was off. And prices keep falling under you: Epoch AI estimates that the cost of reaching a given level of AI performance has dropped about 13 times a year since 2023, so a long prepaid commitment locks in today’s price while the market keeps moving.

What to do before you rent a GPU

1. Compare against the same model’s cheapest host. Check DeepInfra, Together, Fireworks and Groq for the open model you would run. That is the price to beat.
2. Measure requests in flight. Log when each request starts and ends, and work out your average and peak concurrency. Requests a day hide the number that matters.
3. Do the division. Monthly rent divided by the API price of one of your requests gives the volume you need, spread evenly. Then check your peak fits on the GPU.
4. If you do self-host, use a batching server. vLLM, SGLang and NIM keep a GPU busy across users; in the RTX 5090 test above, Ollama’s output barely grew with more users. Keep it off the open internet and plan time for patches.
5. Keep the model swappable. Put the model name and endpoint in one config value, so moving between an API and your own server is a change of settings.
6. Send anything that can wait to a batch API. It halves the price at OpenAI, Anthropic and Google without renting anything.