AI Labs Spend Millions on GPUs

Everyone quotes what it costs to train a frontier model.

Almost nobody quotes how efficiently the money was spent.

Here's the number that changes how you read every "$100 million training run" headline: on the best-engineered GPU clusters in the world, roughly 40% of the hardware's theoretical compute turns into actual model progress.

Not 100%. Not 90%. Forty.

And in this industry, 40% isn't a failure. It's the benchmark.

Where the millions actually go

Start with the bill. Epoch AI broke down what it cost to develop frontier models like GPT-4 and Gemini Ultra. Accelerator chips, servers and interconnect made up 47–67% of the total. Research staff took 29–49%. Energy, the thing everyone argues about, was just 2–6%. (Source: Epoch AI)

Those costs have grown about 2.4x a year since 2016, on track for billion-dollar training runs by 2027. (Source: Epoch AI)

Source: Epoch AI

So the GPU is the bill. Which makes the next question the one that matters.

How much of that GPU are you actually using?

The industry measures it with MFU, Model FLOPs Utilization. Google introduced it in the PaLM paper as a metric that:

"allows fair comparisons between training runs on different systems." — Google Research, PaLM paper

In plain English: of all the math the chip could do at full speed, what share went into training the model?

Here's the public scoreboard:

(Sources: PaLM; SemiAnalysis estimate; Llama 3; MegaScale)

Source: Meta, The Llama 3 Herd of Models

Sit with Meta's numbers for a second. One of the best ML infrastructure teams on earth, 16,384 H100s, and each GPU delivered about 380–430 TFLOPs against a peak near 1,000.

That's a 16,000-GPU cluster doing the useful math of roughly 6,500.

ByteDance's 55.2% is the public high-water mark. They wrote an entire systems paper on how they got there.

Why it's 40% and not 100%

The GPU isn't lazy. It's waiting.

• Waiting on the network. The model is split across thousands of chips, and every step they sync with each other. The math stalls while the bytes move.

• Waiting on memory. Operations like softmax and layer norm are limited by memory speed, not compute. The tensor cores sit idle.

• Waiting in line. With pipeline parallelism, some stages sit idle at the start and end of every batch.

• Doing work twice. Labs recompute activations to save memory. It's real work, but it isn't model progress. That's why PaLM's hardware utilization was 57.8% while its MFU was 46.2%.

• Falling over. In a 54-day snapshot of Llama 3 training, Meta logged 419 unexpected interruptions, roughly one every three hours. Faulty GPUs and HBM3 memory caused nearly half. (Source: SDxCentral)

Meta said the failure scenarios of its 16K-GPU training:

"surpass those of much larger CPU clusters that we have operated." — Meta, Llama 3 team

Source: Meta, The Llama 3 Herd of Models

Then there's the trap that catches most teams: the dashboard.

The "GPU utilization" number in nvidia-smi only tells you whether a kernel is running. Not how much work it's doing. Trainy found a foundation-model team sitting at 100% utilization while hitting about 20% MFU, and showed you can reach:

"100% GPU utilization by just reading/writing to memory while doing 0 computations." — Trainy

(Source: Trainy)

The utilization gauge tells you the engine is on. MFU tells you how far the car went.

Now the math that should be on every AI budget slide:

Effective cost of compute = sticker price ÷ MFU.

At $2 per H100-hour and 40% MFU, every hour of useful compute costs $5.

Apply that to the estimated GPT-4 run, ~$63 million of A100 time at ~34% MFU. Roughly two-thirds of that capacity never became model math.

At Llama scale, it cuts the other way too. A two-month run on 16,384 H100s at $2 an hour is about $47 million. Lift MFU from 40% to 41% and the same job finishes ~2.4% sooner. That one point is worth over $1 million.

That's why frontier labs pay systems engineers like research scientists.

So, do you really want to train your own LLM?

Every few months a board asks it. "Should we build our own model?"

Read the section above again. Meta, with one of the best infrastructure teams on earth, got about 40% out of its GPUs and lost a job to hardware failure every three hours. Your team will not beat that on its first run.

And here's the thing: training was never the problem you were trying to solve.

What most enterprises actually want is a model that knows their business: their policies, their products, their customers. You don't need to bake that into model weights. You need to hand it to the model at the right moment.

That's what RAG and CAG do.

RAG (retrieval-augmented generation) searches your documents at question time and passes the relevant pieces to the model. Your knowledge base can change every day, and the model is always reading the latest version.

CAG (cache-augmented generation) loads a smaller, stable knowledge set into the model's context once and caches it, so there's no retrieval step at all. The researchers who named it put it bluntly in their paper title:

"Don't Do RAG: When Cache-Augmented Generation is All You Need"

(Source: Chan et al., arXiv)

Neither one touches a GPU cluster. Neither one cares about MFU. Both can be running in weeks.

So climb the ladder one rung at a time, and stop at the first rung that works:

1. Prompt a frontier model. Most use cases end here.

2. CAG. A policy manual, a product catalog, a fixed set of SOPs. Small, stable knowledge that fits in context.

3. RAG. Large or fast-changing knowledge: contracts, tickets, case files, anything you'd otherwise retrain for every week.

4. Fine-tune. Only when you need a consistent format, tone or narrow skill that prompting can't hold. Now you're renting GPUs, so ask for MFU, not nvidia-smi percentages.

5. Train from scratch. Only if the model is your product, you own data nobody else can get, and you can budget for 40% efficiency and a failure every few hours.

Almost every enterprise we meet belongs on rungs 1 to 3. That's how we build at Symprio: on the frontier, not at it. Frontier labs have already paid the 60% tax. Let them.

Your edge isn't in the weights. It's in the knowledge you feed them.

Symprio builds AI products and agentic automation for regulated enterprises. If your board is asking whether to train your own model, talk to us →.

Model FLOPs Utilization (MFU): the share of a chip's peak compute that becomes model progress. On the best clusters in the world, 40% is a good day.