Estimating the Compute and Memory Needed for Language Models

Draft · 11 July 2026

I started looking at these numbers because I wanted to answer a much narrower question: if I have only ten minutes, how large can the model be, how many tokens can it see, and where does that put the run relative to the Chinchilla scaling rule? I also wanted some sense of what ten minutes on eight H100s means on my M2, either through MLX or on the CPU.

When we say that a language model is expensive, what exactly is expensive? There are at least three different questions hidden inside that statement:

1. How many arithmetic operations are required?
2. How much memory is required to store the model?
3. How many bytes must move through memory while producing tokens?

These questions give different limitations. A model may fit in memory but generate slowly. A processor may advertise a very large number of floating-point operations per second, but spend most of its time waiting for model weights to arrive from memory. We are going to build a first-order estimate that separates these limits.

The Quantities We Need

We will use the following quantities:

1. \(N\): the number of model parameters;
2. \(D\): the number of training tokens;
3. \(G\): the number of tokens generated during inference;
4. \(b\): the number of bytes used to store each parameter;
5. \(C\): the processor's arithmetic throughput, in FLOP/s;
6. \(B\): its memory bandwidth, in bytes/s;
7. \(M\): its available memory capacity, in bytes.

A FLOP is one floating-point operation. By the convention used here, a multiplication and an addition count as two FLOPs. The estimates below assume a dense decoder-style transformer: nearly all parameters are used for every token. Sparse mixture-of-experts models need a slightly different count because only some parameters are active for each token.

FLOPs and FLOP/s are easy to mix up. FLOPs measure an amount of work; FLOP/s measure how quickly a processor can do that work. Dividing one by the other gives time. The same equations apply whether the processor is a CPU or a GPU—the values of \(C\), \(B\), and \(M\) are what change.

Training Computation

During a forward pass, each parameter is used approximately once per token. Using a weight normally involves one multiplication and one addition, so the forward pass requires roughly

Training also has a backward pass. We need to calculate the gradient with respect to both the activations and the weights. As a first-order estimate, this takes twice the computation of the forward pass:

Putting \(\eqref{eq:forward_training}\) and \(\eqref{eq:backward_training}\) together gives the common training estimate:

This is a count of model FLOPs, not a hardware-counter reading. The \(6N\) convention counts the parameterised matrix multiplications in a standard dense transformer and excludes self-attention. PaLM gives the more complete per-token estimate

where \(L\) is the number of layers, \(H\) the number of attention heads, \(Q\) the head dimension, and \(T\) the sequence length. The attention term matters more for small models and long sequences. Weight sharing needs another correction: if a recurrent transformer applies the same saved block several times, \(N\) counts its parameters once even though the hardware executes the block several times.

For example, training a 7-billion-parameter model on 140 billion tokens gives

Dividing this number by a processor's advertised FLOP/s—at the same numerical precision and sparsity assumption—only gives an ideal lower bound on training time. Real training does not sustain peak arithmetic throughput continuously; data loading, communication, non-matrix operations, and imperfect utilisation all add time.

Parameter Golf in 10 Minutes

Parameter Golf gives us a concrete example. The competition asks for the best language model whose complete submitted artifact—training code plus compressed weights—is at most 16,000,000 bytes. Training must finish in under 600 seconds on eight NVIDIA H100 SXM GPUs. Models are evaluated on a held-out FineWeb split using bits per byte, so different tokenizers can still be compared.

The 16 MB limit is an artifact-storage limit, not a training-memory limit. Quantisation lets a submission place more parameters in the artifact, but training may still keep higher-precision weights, gradients, optimiser states, and activations. For example, one 35.95-million-parameter record used a roughly 16 MB submitted artifact but peaked at about 41.7 GB per GPU during training. Quantisation made the saved model small; it did not make the training job 16 MB.

Actual Competition Runs

The logs answer a different question from a specification sheet. A hardware specification gives a theoretical peak; a log gives observed tokens per second. An equation such as \(6N\) is the bridge between them. Keeping those three quantities separate avoids calling an equation-derived number a measurement.

Here, \(20N\) is the common shorthand taken from Chinchilla: roughly 20 training tokens for each parameter. For a 35.95M model, that would be about

The actual run saw 3.927B tokens, or about 5.5 times this amount. So the competition sits well on the small-model, data-heavy side of the Chinchilla rule. That is not an accident: the artifact limit prevents us from freely increasing \(N\), while the eight H100s can process billions of tokens in ten minutes. More aggressive quantisation lets competitors move toward a larger \(N\); a ternary submission, for example, fit 73.7M parameters into the same 16 MB limit.

The \(20N\) number is only a landmark. Chinchilla was fitted using much larger dense models, so extrapolating it down to tens of millions of parameters—or to recurrent and heavily quantised architectures—should not be treated as an exact optimum.

Which FineWeb?

There are three different dataset sizes that are easy to call “FineWeb”:

1. The complete public FineWeb corpus contains roughly 15 trillion GPT-2 tokens in the original paper (the current dataset card reports more than 18.5 trillion after later updates).
2. Parameter Golf uses a selected FineWeb export. With the baseline's 1,024-token SentencePiece tokenizer, the current manifest contains 19.47B training tokens across 195 shards.
3. A particular run may download only a prefix of those shards. The current default is 80 shards, or 8B tokenized training tokens.

No ten-minute competition run comes close to exhausting full FineWeb. The 3.927B-token 36M run is only about 0.026% of the original 15T-token corpus. Whether a run repeats data depends on the locally cached prefix, not on the size of full FineWeb. The historical baseline cached 2.5B tokens and processed 7.225B, so it made about 2.9 passes. The final 36M-class run cached 8B and processed 3.927B, so it used about half of one pass.

Raw token totals also change with the tokenizer. A 1,024-token vocabulary splits the same text into more tokens than an 8,192-token vocabulary. That is why the competition scores bits per source byte: bytes provide the common denominator when tokenizers differ.

Specs, Equations, and Local Equivalents

There are three different layers in the comparison:

1. Published specification: the theoretical arithmetic or bandwidth peak of the hardware.
2. Observed throughput: tokens per second recorded by a complete training program.
3. Equation-derived throughput: observed tokens per second multiplied by an assumed FLOP count per token.

Eight H100s: Specification Versus the Record

NVIDIA quotes 1,979 BF16 Tensor TFLOP/s for one H100 SXM with structured sparsity. The corresponding dense BF16 peak is about 989.5 TFLOP/s per GPU, or 7.92 PFLOP/s across eight GPUs. The Parameter Golf record does not claim the 2:4 structured sparsity needed for the doubled number, so the dense peak is the relevant comparison.

The record log gives the observed quantity directly:

If we now apply the simple \(6N\) convention, we derive—not independently measure—the following model-FLOPs rate:

Relative to the 7.92 PFLOP/s dense BF16 hardware peak, this simple normalisation is about 18%. That is a plausible end-to-end utilisation, but it is not proof that the hardware executed exactly 1.41 PFLOP/s: the agreement is partly constructed by the \(6N\) equation itself.

The record also shows why \(6N\) is only a landmark. It starts with 11 physical layers, then applies layers 3–5 two additional times after step 2,204. Recounting only those repeated linear layers raises the rough rate to about 1.8 PFLOP/s, or 23% of dense BF16 peak. Self-attention adds more work, but the exact amount depends on the packed document lengths, which are not recoverable from the summary throughput log. The clean statement is therefore: 6.55M tokens/s observed, 1.41 PFLOP/s under the simple \(6N\) normalisation.

The M2: What the Specification Does and Does Not Tell Us

The measured machine is an 8 GB MacBook Air with an M2, an 8-core CPU, and a 10-core GPU. Apple publishes 100 GB/s of unified-memory bandwidth, but it does not publish a directly comparable FP32 or BF16 GPU training-FLOP/s figure. The advertised 15.8 trillion operations per second belongs to the Neural Engine; these MLX training runs use the GPU, so that number is not the relevant ceiling.

This means the M2 arithmetic comparison must start from measured tokens per second. We can define a useful normalised quantity

and turn it around to project the token budget for a model of size \(N\) in time \(t\):

\(C_{6N}\) is useful for normalising runs, but it is derived from the observed token rate. It should not be confused with an independently measured M2 FLOP/s value.

The projected columns put every row onto the same hypothetical 35.95M dense model. The original token rates came from different model sizes, so these projections assume that their \(6N\)-equivalent rate transfers unchanged. That is useful for scale, not a promise about the exact Parameter Golf architecture.

The tiny 0.36M anchor could have extrapolated badly. The measurements linked from Memory to Train a Transformer provide the better check at approximately the right model size: the 41M dense model achieved about 2,400 tokens/s at \(B=1,T=2048\) and 4,900 tokens/s at \(B=4,T=512\). Under the common \(6N\) normalisation, that puts the 35.95M projection at roughly 8–16 days; the small-model anchor gives about 20 days. The CPU row remains an unbenchmarked assumption.

There is one more useful way to read the result. Combining \(D \approx 20N\) with \(6ND \approx C_{6N}t\) gives

This is the recipe for sizing a run on any machine: obtain a \(6N\)-equivalent rate from a short benchmark, multiply by the time budget, divide by 120, and take the square root. A peak specification should not be substituted unless the workload's sustained utilisation is also known. For a ten-minute run, the simple \(6N\) normalisation gives about 84M parameters for the eight-H100 record, about 1.7–2.4M for the measured M2 dense-transformer rates, and roughly 0.5–1.0M for the CPU assumption. Architecture-aware FLOP counts would move these values, but the scale remains useful: a ten-minute local experiment is better matched to a model of a few million parameters than to the competition's 35–40M models.

Inference Computation

Inference has a forward pass but no backward pass. Ignoring the context-dependent attention terms for now, one token therefore costs

Generating \(G\) tokens requires approximately

A 7-billion-parameter model consequently needs about 14 billion FLOPs for each generated token. If arithmetic were the only limitation, a processor with throughput \(C\) would have the following ceiling:

This looks encouraging on modern GPUs. It is also incomplete, because arithmetic cannot begin until the weights have reached the processor.

Storing the Model

If every parameter occupies \(b\) bytes, the weight storage is simply

These are decimal gigabytes and include only the weights. Quantised formats also store scales and other metadata, so a real 4-bit model is slightly larger than \(0.5N\) bytes. The runtime, temporary activations, and the KV cache require additional memory. A model whose weights exactly equal the advertised memory capacity does not, in practice, fit comfortably.

Training requires much more memory than inference. In one common mixed-precision Adam setup, each parameter may need a low-precision weight, a gradient, a full-precision master weight, and two full-precision optimiser states. That can reach roughly 16 bytes per parameter before activations are included. Equation \(\eqref{eq:weight_storage}\) should therefore be treated as an inference-weight estimate, not a complete training-memory estimate. The complete training estimate, including the activation terms, is derived and measured in Memory to Train a Transformer.

Transferring the Model for Every Token

During batch-one autoregressive generation, each new token passes through the whole model. The processor therefore needs to read nearly all \(Nb\) bytes of weights for each token. Generating \(G\) tokens moves approximately

If the memory system can deliver \(B\) bytes per second, the memory-bandwidth ceiling is

The actual generation rate cannot exceed either the arithmetic ceiling in \(\eqref{eq:compute_rate}\) or the memory ceiling in \(\eqref{eq:memory_rate}\):

This is the useful distinction. FLOPs tell us how much arithmetic the model asks for. Bytes tell us how quickly the hardware can feed that arithmetic. At FP16, the weight multiplications perform approximately \(2N\) FLOPs after reading \(2N\) bytes: only about one FLOP per byte. That is low arithmetic intensity, so batch-one generation is commonly limited by memory bandwidth rather than peak FLOP/s.

Batching changes the picture. If one copy of the weights can be reused across several sequences, more arithmetic is performed for every byte of weights read. Throughput improves, although the latency experienced by an individual sequence may not.

Apple M2 and NVIDIA H200

Consider two very different systems. Apple offered the M2 with up to 24 GB of unified memory and 100 GB/s of memory bandwidth; the machine measured in this article has 8 GB. With unified memory, both the CPU and GPU access the same pool. MLX takes advantage of this: an array does not need to be copied into separate GPU memory before a GPU operation can use it. NVIDIA's H200, by comparison, has 141 GB of HBM3e memory and 4.8 TB/s of memory bandwidth.

Applying \(\eqref{eq:memory_rate}\) gives the following ideal batch-one bandwidth ceilings. These are not benchmark predictions; they assume that the full advertised bandwidth is sustained and count only weight traffic.

†The 7B FP16 weights do not fit on the tested 8 GB M2. The 7.1-token/s figure is only the bandwidth ceiling for an M2 configuration with enough memory, before runtime overhead and KV-cache space are included.

*The 70B FP16 weights nominally fit in the H200's 141 GB, but leave only about 1 GB for everything else. A practical deployment needs more room for the KV cache, activations, and runtime allocations.

The table makes two different hardware limits visible. The tested 8 GB M2 rules out 7B FP16 as well as both 70B examples before we consider speed. For models that fit, quantisation reduces both storage and the number of weight bytes transferred per token. The H200 can hold larger models and move their weights much faster, but it is still not exempt from the same bandwidth calculation.

What the Estimate Misses

Like any back-of-the-envelope calculation, these equations leave out important details:

1. The \(6N\) training estimate assumes each saved parameter is applied approximately once per token. Weight-tied recurrent blocks require a count of block applications, not just unique stored parameters.
2. The \(2N\) inference estimate captures parameterised matrix multiplications, but attention also performs work that grows with the context length. Long contexts make the omitted terms more important.
3. The KV cache adds memory traffic and grows with context length, batch size, number of layers, and the number of key-value heads.
4. Quantisation adds scales, metadata, and dequantisation work. Four-bit weights do not produce a perfect fourfold speed-up over FP16.
5. Advertised FLOP/s depends on numerical precision and sometimes assumes structured sparsity. Even after matching those assumptions, kernel shape, framework overhead, thermal limits, and memory-access patterns reduce sustained performance.
6. Prompt processing (prefill) handles many tokens in parallel and has a different compute-to-memory balance from one-token-at-a-time decoding.

Still, the simple equations are useful. Start with \(6ND\) as the standard dense training baseline, \(Nb\) for inference-weight storage, \(2N\) FLOPs for each generated token, and \(Nb\) weight bytes transferred for each generated token. Then add the architecture-specific corrections and ask which hardware ceiling is lower. That tells us whether the first limitation is arithmetic, memory capacity, or memory bandwidth.

Sources and Notes

1. The \(6N\) FLOPs per training token convention is described in the PaLM paper, which separates it into \(2N\) forward-pass and \(4N\) backward-pass matrix-multiplication FLOPs. Its Appendix B gives the additional \(12LHQT\) self-attention term and defines model-FLOPs utilisation against theoretical peak throughput.
2. Apple lists an 8-core CPU, a configurable 10-core GPU, 100 GB/s of unified-memory bandwidth, and up to 24 GB of memory in its M2 announcement. The measured MacBook Air has the 10-core GPU and 8 GB of memory.
3. The MLX documentation explains how its arrays are shared by the Apple-silicon CPU and GPU.
4. NVIDIA's H100 specifications list 1,979 BF16 Tensor TFLOP/s with sparsity. NVIDIA's Hopper architecture table separates this from the approximately 1,000 dense BF16 TFLOP/s figure used in the comparison.
5. NVIDIA lists 141 GB of HBM3e and 4.8 TB/s of memory bandwidth in the H200 specifications.
6. The official Parameter Golf repository defines the 16 MB artifact and ten-minute eight-H100 rules. The baseline log provides the 17.06M-parameter and 7.225B-token figures.
7. The 35.95M-parameter comparison comes from the final Calib32 record submission. Its training script uses BF16 autocast and recurrently reapplies layers 3–5; a seed log records recurrence beginning at step 2,204 and training ending at step 4,997 after 599.6 seconds. The larger-model example is the official 73.7M-parameter ternary submission.
8. Hugging Face describes the size of full FineWeb. The challenge's tokenizer-specific counts are published in its dataset manifest.
9. The \(20N\) comparison is a shorthand based on the 70B-parameter, 1.4T-token example in the Chinchilla paper, not a claimed exact optimum for Parameter Golf-scale models.
10. The M2 anchor is from a local five-minute train_gpt_mlx.py run: 361,608 parameters, 8,428 steps, and 8,192 tokens per step.
11. The 10.6M, 41.2M, and 100.4M dense measurements are from the measured section of Memory to Train a Transformer: plain MLX modules, FP32, ordinary Adam, uncompiled training step, on an 8 GB M2. The batched rows use the same 41M shape \((10, 512, 8, 8{,}448)\) at \(B=4\) and \(T=512\); throughput at \(B=8\) was slightly lower, so \(B=4\) is reported as the saturated rate.