Estimating the Compute and Memory Needed for Language Models

Draft · 11 July 2026

I started looking at these numbers because I wanted to answer a much narrower question: if I have only ten minutes, how large can the model be, how many tokens can it see, and where does that put the run relative to the Chinchilla scaling rule? I also wanted some sense of what ten minutes on eight H100s means on my M2, either through MLX or on the CPU.

When we say that a language model is expensive, what exactly is expensive? There are at least three different questions hidden inside that statement:

1. How many arithmetic operations are required?
2. How much memory is required to store the model?
3. How many bytes must move through memory while producing tokens?

These questions give different limitations. A model may fit in memory but generate slowly. A processor may advertise a very large number of floating-point operations per second, but spend most of its time waiting for model weights to arrive from memory. We are going to build a first-order estimate that separates these limits.

The Quantities We Need

We will use the following quantities:

1. \(N\): the number of model parameters;
2. \(D\): the number of training tokens;
3. \(G\): the number of tokens generated during inference;
4. \(b\): the number of bytes used to store each parameter;
5. \(C\): the processor's arithmetic throughput, in FLOP/s;
6. \(B\): its memory bandwidth, in bytes/s;
7. \(M\): its available memory capacity, in bytes.

A FLOP is one floating-point operation. By the convention used here, a multiplication and an addition count as two FLOPs. The estimates below assume a dense decoder-style transformer: nearly all parameters are used for every token. Sparse mixture-of-experts models need a slightly different count because only some parameters are active for each token.

FLOPs and FLOP/s are easy to mix up. FLOPs measure an amount of work; FLOP/s measure how quickly a processor can do that work. Dividing one by the other gives time. The same equations apply whether the processor is a CPU or a GPU—the values of \(C\), \(B\), and \(M\) are what change.

Training Computation

During a forward pass, each parameter is used approximately once per token. Using a weight normally involves one multiplication and one addition, so the forward pass requires roughly

Training also has a backward pass. We need to calculate the gradient with respect to both the activations and the weights. As a first-order estimate, this takes twice the computation of the forward pass:

Putting \(\eqref{eq:forward_training}\) and \(\eqref{eq:backward_training}\) together gives the common training estimate:

For example, training a 7-billion-parameter model on 140 billion tokens gives

Dividing this number by a processor's advertised FLOP/s only gives an ideal lower bound on training time. Real training does not sustain peak arithmetic throughput continuously; data loading, communication, non-matrix operations, and imperfect utilisation all add time.

Parameter Golf in 10 Minutes

Parameter Golf gives us a concrete example. The competition asks for the best language model whose complete submitted artifact—training code plus compressed weights—is at most 16,000,000 bytes. Training must finish in under 600 seconds on eight NVIDIA H100 SXM GPUs. Models are evaluated on a held-out FineWeb split using bits per byte, so different tokenizers can still be compared.

The 16 MB limit is an artifact-storage limit, not a training-memory limit. Quantisation lets a submission place more parameters in the artifact, but training may still keep higher-precision weights, gradients, optimiser states, and activations. For example, one 35.95-million-parameter record used a roughly 16 MB submitted artifact but peaked at about 41.7 GB per GPU during training. Quantisation made the saved model small; it did not make the training job 16 MB.

Actual Competition Runs

The logs are more useful than advertised peak FLOP/s. They tell us how many tokens each complete implementation actually processed before the 600-second cutoff.

Here, \(20N\) is the common shorthand taken from Chinchilla: roughly 20 training tokens for each parameter. For a 35.95M model, that would be about

The actual run saw 3.927B tokens, or about 5.5 times this amount. So the competition sits well on the small-model, data-heavy side of the Chinchilla rule. That is not an accident: the artifact limit prevents us from freely increasing \(N\), while the eight H100s can process billions of tokens in ten minutes. More aggressive quantisation lets competitors move toward a larger \(N\); a ternary submission, for example, fit 73.7M parameters into the same 16 MB limit.

The \(20N\) number is only a landmark. Chinchilla was fitted using much larger dense models, so extrapolating it down to tens of millions of parameters—or to recurrent and heavily quantised architectures—should not be treated as an exact optimum.

Which FineWeb?

There are three different dataset sizes that are easy to call “FineWeb”:

1. The complete public FineWeb corpus contains roughly 15 trillion GPT-2 tokens in the original paper (the current dataset card reports more than 18.5 trillion after later updates).
2. Parameter Golf uses a selected FineWeb export. With the baseline's 1,024-token SentencePiece tokenizer, the current manifest contains 19.47B training tokens across 195 shards.
3. A particular run may download only a prefix of those shards. The current default is 80 shards, or 8B tokenized training tokens.

No ten-minute competition run comes close to exhausting full FineWeb. The 3.927B-token 36M run is only about 0.026% of the original 15T-token corpus. Whether a run repeats data depends on the locally cached prefix, not on the size of full FineWeb. The historical baseline cached 2.5B tokens and processed 7.225B, so it made about 2.9 passes. The final 36M-class run cached 8B and processed 3.927B, so it used about half of one pass.

Raw token totals also change with the tokenizer. A 1,024-token vocabulary splits the same text into more tokens than an 8,192-token vocabulary. That is why the competition scores bits per source byte: bytes provide the common denominator when tokenizers differ.

M2 and CPU Equivalents

We can turn \(6ND\) around. If a system sustains effective training throughput \(C_{\text{eff}}\), then the token budget in time \(t\) is

A local five-minute MLX run provides one measured anchor. A 361,608-parameter model processed 69.0M tokens in 300 seconds. Under the \(6ND\) convention, that is about 0.50 TFLOP/s of effective training throughput. This is an end-to-end measurement, not the M2's advertised peak.

If we naively hold that measured rate constant for a 35.95M dense model, the comparison looks like this:

*The H100 number is \(6ND/t\)-equivalent throughput, not a measured hardware counter. The record uses recurrence, quantisation-aware operations, and specialised kernels, so it is only a scaling landmark.

The most defensible conclusion is the order of magnitude: reproducing the amount of training in a ten-minute eight-H100 record is likely several days to several weeks on an M2, and roughly months on a CPU-only implementation. The CPU row is explicitly an assumed range; it is not a benchmark of this machine.

The 0.36M anchor could have extrapolated badly: a larger model may execute its matrix multiplications more efficiently, but it also needs smaller microbatches, gradient accumulation, and replacements for CUDA-only FlashAttention and Triton kernels. The measured 41M rows settle this at the right scale. While checking the memory estimate in Estimating Transformer Training Memory, we trained plain dense MLX transformers on an 8 GB M2 in FP32 with ordinary Adam and an uncompiled training step. Effective \(6ND\) throughput at \(B=1\) grew with model size—about 0.37 TFLOP/s at 10.6M parameters, 0.60 at 41.2M, and 0.86 at 100.4M—so larger matrices do run more efficiently. Batching helped further: the 41M model reached about 4,900 tokens per second at \(B=4\) and \(T=512\), or 1.19 TFLOP/s, and larger batches did not improve on that. At these measured rates, matching the record's 3.927B tokens takes 9 to 19 days—close to the original 19.7-day extrapolation and well short of more optimistic scenarios, at least for an unoptimised dense implementation. The record's recurrent, quantisation-aware architecture would still need its own benchmark.

There is one more useful way to read the result. Combining \(D \approx 20N\) with \(6ND \approx C_{\text{eff}}t\) gives

This is the recipe for sizing a run on any machine: estimate the effective training throughput \(C_{\text{eff}}\) (a short measured run beats any spec sheet), multiply by the time budget, divide by 120, and take the square root. For a ten-minute run, this crude calculation gives about 84M parameters for the measured eight-H100 compute budget, about 1.7–2.4M for the measured M2 dense-transformer rates, and roughly 0.5–1.0M for the CPU assumption. This explains why a ten-minute local experiment is better matched to a model of a few million parameters than to the competition's 35–40M models.

Inference Computation

Inference has a forward pass but no backward pass. Ignoring the context-dependent attention terms for now, one token therefore costs

Generating \(G\) tokens requires approximately

A 7-billion-parameter model consequently needs about 14 billion FLOPs for each generated token. If arithmetic were the only limitation, a processor with throughput \(C\) would have the following ceiling:

This looks encouraging on modern GPUs. It is also incomplete, because arithmetic cannot begin until the weights have reached the processor.

Storing the Model

If every parameter occupies \(b\) bytes, the weight storage is simply

These are decimal gigabytes and include only the weights. Quantised formats also store scales and other metadata, so a real 4-bit model is slightly larger than \(0.5N\) bytes. The runtime, temporary activations, and the KV cache require additional memory. A model whose weights exactly equal the advertised memory capacity does not, in practice, fit comfortably.

Training requires much more memory than inference. In one common mixed-precision Adam setup, each parameter may need a low-precision weight, a gradient, a full-precision master weight, and two full-precision optimiser states. That can reach roughly 16 bytes per parameter before activations are included. Equation \(\eqref{eq:weight_storage}\) should therefore be treated as an inference-weight estimate, not a complete training-memory estimate. The complete training estimate, including the activation terms, is derived and measured in Estimating Transformer Training Memory.

Transferring the Model for Every Token

During batch-one autoregressive generation, each new token passes through the whole model. The processor therefore needs to read nearly all \(Nb\) bytes of weights for each token. Generating \(G\) tokens moves approximately

If the memory system can deliver \(B\) bytes per second, the memory-bandwidth ceiling is

The actual generation rate cannot exceed either the arithmetic ceiling in \(\eqref{eq:compute_rate}\) or the memory ceiling in \(\eqref{eq:memory_rate}\):

This is the useful distinction. FLOPs tell us how much arithmetic the model asks for. Bytes tell us how quickly the hardware can feed that arithmetic. At FP16, the weight multiplications perform approximately \(2N\) FLOPs after reading \(2N\) bytes: only about one FLOP per byte. That is low arithmetic intensity, so batch-one generation is commonly limited by memory bandwidth rather than peak FLOP/s.

Batching changes the picture. If one copy of the weights can be reused across several sequences, more arithmetic is performed for every byte of weights read. Throughput improves, although the latency experienced by an individual sequence may not.

Apple M2 and NVIDIA H200

Consider two very different systems. Apple's M2 has up to 24 GB of unified memory and 100 GB/s of memory bandwidth. With unified memory, both the CPU and GPU access the same pool. MLX takes advantage of this: an array does not need to be copied into a separate GPU memory before a GPU operation can use it. NVIDIA's H200, by comparison, has 141 GB of HBM3e memory and 4.8 TB/s of memory bandwidth.

Applying \(\eqref{eq:memory_rate}\) gives the following ideal batch-one bandwidth ceilings. These are not benchmark predictions; they assume that the full advertised bandwidth is sustained and count only weight traffic.

*The 70B FP16 weights nominally fit in the H200's 141 GB, but leave only about 1 GB for everything else. A practical deployment needs more room for the KV cache, activations, and runtime allocations.

The table makes two different hardware limits visible. The M2's memory capacity rules out the 70B examples before we consider speed. For the models that do fit, quantisation reduces both storage and the number of weight bytes transferred per token. The H200 can hold larger models and move their weights much faster, but it is still not exempt from the same bandwidth calculation.

What the Estimate Misses

Like any back-of-the-envelope calculation, these equations leave out important details:

1. The \(2N\) inference estimate captures parameterised matrix multiplications, but attention also performs work that grows with the context length. Long contexts make the omitted terms more important.
2. The KV cache adds memory traffic and grows with context length, batch size, number of layers, and the number of key-value heads.
3. Quantisation adds scales, metadata, and dequantisation work. Four-bit weights do not produce a perfect fourfold speed-up over FP16.
4. Advertised FLOP/s and memory bandwidth are peak values. Kernel shape, framework overhead, thermal limits, and memory-access patterns reduce sustained performance.
5. Prompt processing (prefill) handles many tokens in parallel and has a different compute-to-memory balance from one-token-at-a-time decoding.

Still, the simple equations are useful. Start with \(6ND\) for training computation, \(Nb\) for inference-weight storage, \(2N\) FLOPs for each generated token, and \(Nb\) weight bytes transferred for each generated token. Then ask which hardware ceiling is lower. That tells us whether the first limitation is arithmetic, memory capacity, or memory bandwidth.

Sources and Notes

1. The \(6N\) FLOPs per training token convention is described in the PaLM paper, which separates it into \(2N\) forward-pass and \(4N\) backward-pass matrix-multiplication FLOPs.
2. Apple lists the M2 at 100 GB/s of unified-memory bandwidth and up to 24 GB of memory in its M2 announcement.
3. The MLX documentation explains how its arrays are shared by the Apple-silicon CPU and GPU.
4. NVIDIA lists 141 GB of HBM3e and 4.8 TB/s of memory bandwidth in the H200 specifications.
5. The official Parameter Golf repository defines the 16 MB artifact and ten-minute eight-H100 rules. The baseline log provides the 17.06M-parameter and 7.225B-token figures.
6. The 35.95M-parameter comparison comes from the final Calib32 record submission. The larger-model example is the official 73.7M-parameter ternary submission.
7. Hugging Face describes the size of full FineWeb. The challenge's tokenizer-specific counts are published in its dataset manifest.
8. The \(20N\) comparison is a shorthand based on the 70B-parameter, 1.4T-token example in the Chinchilla paper, not a claimed exact optimum for Parameter Golf-scale models.
9. The M2 anchor is from a local five-minute train_gpt_mlx.py run: 361,608 parameters, 8,428 steps, and 8,192 tokens per step.
10. The 10.6M, 41.2M, and 100.4M dense measurements are from the measured section of Estimating Transformer Training Memory: plain MLX modules, FP32, ordinary Adam, uncompiled training step, on an 8 GB M2. The batched rows use the same 41M shape \((10, 512, 8, 8{,}448)\) at \(B=4\) and \(T=512\); throughput at \(B=8\) was slightly lower, so \(B=4\) is reported as the saturated rate.