Compute & Moore's Law

From 2,300 transistors in the 1971 Intel 4004 to 100 billion+ in a modern AI accelerator, transistor counts have doubled roughly every 2–3 years. FLOPS per dollar of supercomputing have improved ~100× per decade for 70 years. AI training compute has been growing ~4–5× per year since 2010 — far faster than Moore's Law alone can deliver.

2,300
Transistors in Intel 4004 (1971)
100B+
Transistors in modern AI accelerators
~50yr
Years of Moore's-law-style scaling
4–5×/yr
AI training-compute growth (post-2010)

Key insights

📐

Moore's law is slowing, not stopping

From 1965 to ~2010, transistor density doubled every ~24 months (Moore's Law). Density growth has slowed to ~3 years per doubling since 2015 as feature sizes approach atomic limits. But chip performance hasn't slowed proportionally — gains now come from architectural changes (chiplets, 3D stacking, specialised accelerators), not pure scaling. Cost per transistor stopped falling around 2014; performance per dollar continues to improve but more slowly.

💡

AI compute is the new exponent

From 2010 to 2024, the compute used to train frontier AI models grew ~4–5× per year — far faster than Moore's Law (~1.4×/year). The gains came from parallel scaling (more GPUs in clusters) and architectural improvements (Transformer, mixed-precision training). GPT-4 training is estimated at ~2.1×10²⁵ FLOPs (Epoch AI says such estimates can be off by a factor of 5); Llama 3.1 405B at ~3.8×10²⁵; rumoured GPT-5/Llama-4-scale training at 10²⁶ and above.

🏭

Capex and energy are the new constraints

Top-end fabs (TSMC 3nm/2nm, Samsung 3nm) cost $20–30 billion each. EUV machines from ASML cost $200M per unit. Data-centre electricity demand from AI is doubling every 2 years. The constraint on compute scaling is shifting from chip-density physics to fab construction, machine availability, and grid interconnection — all of which scale slowly even with capital.

Transistor count per chip — landmark processors 1971–2024

Transistors per chip (log scale appropriate)

Show data table
Transistor count per chip — landmark processors 1971–2024
Transistors per chip (millions)
19710.0023
19850.275
19955.5
2005290
20155000
202054,200
202380,000
2024208,000

Key Finding: Approximate 100,000,000× increase over 50 years. Recent flagship AI accelerators reach 100+ billion transistors.

Training compute of notable AI models 2012–2024

Estimated training FLOPs, log scale appropriate

Show data table
Training compute of notable AI models 2012–2024
Training FLOPs (×10^20)
AlexNet (2012)0.005
GPT-3 (2020)3100
PaLM (2022)25,000
GPT-4 est. (2023)210,000
Llama 3.1 405B (2024)380,000

Key Finding: Frontier training compute has grown roughly 4–5× per year for 12 years — vastly outpacing Moore's Law alone.

Methodology & caveats

Moore's Law in its three meanings

(1) The original 1965 statement: transistor counts double every year. (2) The 1975 revision: every two years. (3) The marketing meaning: 'computers get exponentially better'. (1) and (2) refer to a specific count on a specific cost basis; (3) bundles density, frequency, parallelism and architecture. The first two have slowed; the third remains broadly true but at a slower rate.

FLOPS — peak vs sustained

Theoretical peak FLOPS = number of execution units × frequency × ops/cycle. Sustained FLOPS on real workloads is typically 30–70% of peak. AI training uses mixed precision (FP16/BF16/FP8) — quoted FLOPS often refer to the lowest-precision rate, which can be 4–8× higher than FP32 peak. Compare AI compute numbers carefully — apples and oranges are common.

Cost per FLOPS

The long-run improvement in FLOPS per dollar is about 100× per decade, but the pace depends heavily on the period and hardware. It has slowed in recent decades because of fixed costs (packaging, system integration, cooling). For AI-relevant FP16/BF16 compute on GPUs, the recent rate is closer to 10× per decade. Total spend on AI compute is rising far faster than unit costs are falling, which is why capex rather than price-performance now dominates the story.

How reliable are training-compute figures

Almost no lab publishes how much compute a frontier model used. Epoch AI estimates it from cluster size, training time and utilisation. It says most estimates are correct within a factor of 2, and within a factor of 5 for models such as GPT-4 where key numbers were not disclosed. The early points on this chart are less certain still, so the trend is more reliable than any single bar.

Transistor counts are not like-for-like

The newest bars count multi-die packages. Nvidia's 208 billion transistor Blackwell chip joins two dies, so it is not a single piece of silicon in the sense of the 1971 Intel 4004. That matters for reading Moore's Law: packaging now delivers part of the gain that used to come from shrinking. What would change the trend is the supply side, in advanced packaging, high-bandwidth memory and power, more than transistor physics.

Cite this page

GlobeChart. “Compute & Moore's Law.” Last updated 2026-10-31. https://globechart.com/technology/gpu-compute-history/

Please also credit the original publisher named under each chart: Wikipedia transistor-count compilation, manufacturer disclosures; Epoch AI 'Notable Models' database. Spotted an error? Tell us.