F10. The Transformer

Year: 2017
Zone: Artificial Intelligence
Band: outer
Tier: Built

“Attention Is All You Need” (2017) introduced the Transformer, the architecture behind today’s Large Language Models. Every token decides how much to look at every other token. Click a word to see where it looks.

ArchitectureTransformer, 2017
MechanismSelf-attention
Weights shownIllustrative

Portal

Illustrative self-attention · "it" looks at...

The5%

train55%

stopped10%

because6%

it12%

was5%

running4%

late3%

Illustrative weights, not from a real model. Real models run dozens of these heads in parallel per layer.

Illustrative next-token prediction

The museum train arrived at the

station58%

platform21%

gate9%

museum5%

future3%

Illustrative next-token odds, not real model output.

Loading portal...

Open in museum View on floor plan All exhibits

Sources

[TBD: source] ([TBD: source])

Portal

Sources

On the map