How AI Inference Became a Multibillion-Dollar Business

Forty trillion tokens. A day. That's what one company in San Mateo says it pushes through its servers, every twenty-four hours. They don't build a single model of their own.

Wait, they build nothing and they move forty trillion of... anything?

Nothing. They just run everybody else's. And in July of twenty twenty-six, investors valued that for seventeen and a half billion dollars.

Okay. So who actually makes the money when an AI answers me?

That's the question. Not who trains it. Who ANSWERS you. Hold onto it, because the answer is not the name on the chat box.

I'd have said OpenAI, Anthropic, Google, done.

There's a whole industry wedged between them and you. And it starts in twenty twenty-three, in a lab at Berkeley, with a grad student named Woosuk Kwon staring at a GPU that is doing almost nothing... and costing a fortune.

Doing nothing how? It's a chatbot. It's busy.

Most of its memory is sitting empty. Reserved, unused, wasted. When a model generates text it keeps notes on everything it's already said... the key-value cache. The old way, you'd reserve one big block of memory per conversation, sized for the worst case. But most conversations are short.

So it's booking a conference room for forty people and three show up.

Exactly that. And Kwon's fix borrowed an idea that's been sitting in operating systems for decades. Virtual memory. Paging. Break the cache into small pages, hand them out as needed, let different requests share.

Hang on. That's it? An old operating-systems trick, pointed at chatbots?

That's genuinely it. The paper is called vLLM... Easy, Fast, and Cheap LLM Serving with PagedAttention.

Cheap is doing a lot of work in that title.

Cheap is the entire business. Pack more conversations onto the same chip, serve more people per dollar. It came out of Berkeley's Sky Computing Lab, open source, free. Today it's one of the most active open-source AI projects anywhere, contributors from more than two thousand organizations.

And Nvidia just... let that happen?

Nvidia has its own serving stack, TensorRT-LLM. But the techniques stopped being secret. Continuous batching, prefix caching, quantization, squeezing a model down to four or eight bits so it fits in less memory. All public.

Then where's the business? Everyone can read the recipe.

In the margin. If my stack serves thirty percent more requests per GPU than yours, I can charge less than you AND make more than you. Tiny efficiency gaps. Enormous volume. Which brings us back to San Mateo. Fireworks AI, founded twenty twenty-two by people out of the PyTorch team at Meta. October twenty twenty-five, they raise two hundred and fifty million at a four billion dollar valuation.

Four billion for running other people's models.

By May twenty twenty-six, reports they're talking to investors at fifteen. In July, they close it. One and a half billion dollars, at seventeen and a half.

Four to seventeen point five. In nine months.

And they said why. Past a billion dollars in annualized revenue, roughly five times year over year, serving customers like Cursor and the legal AI company Harvey. Hence the forty trillion tokens a day.

Who put the money in?

Index Ventures, TCV, others... and Nvidia.

The shovel company investing in the diggers. Never seen that before.

Then there's Together AI in San Francisco, the open-source side of the same trade. March twenty twenty-four, worth one and a quarter billion. February twenty twenty-five, three point three billion, with about a hundred million in revenue disclosed.

And now?

June twenty twenty-six. Eight hundred million dollar Series C at eight point three billion, led by Aramco Ventures. They said annual bookings passed one point one five billion in the last quarter.

Saudi oil money. Into token serving.

Energy money following compute. That's not an accident. And the category keeps widening... Replicate, Anyscale, and Fal, the multimodal hosting company, reportedly circling around eight billion this autumn, according to reporting in the past few weeks.

Let me push back though. Isn't this a race to zero? Everyone undercutting everyone on a commodity?

That's the real argument and nobody's settled it. Price per token keeps falling. But demand rises faster, because every agent, every coding tool, every search box burns tokens by the billion. So far, volume has beaten the price cuts.

So far.

So far. And a caution... these are private valuations, not audited profit. Fireworks' revenue claims are Fireworks'. Together's are Together's. Company statements, not filings.

But the structure is what gets me. Training is one giant spend, once. Inference is forever.

That's the sentence. Training is capital expenditure. Inference is a utility bill that never stops arriving. Utilities have never been glamorous. They've just been where the durable money lives.

So the labs get the headlines, and these people get the meter.

Which answers the question I opened with. Every time you type into a chat box, somebody is making a margin on the half-second you waited. The model is the famous part... the money is in the plumbing.