Home › Key concepts › Attention and the transformer

Key concept · Crawl (beginner) · the same on every platform

Attention and the transformer

Each word looks at every other word to decide what matters.

What it is

The transformer is the design behind modern language models. Its key part, attention, lets each token weigh every other token in the context to build its meaning: in 'the bank of the river', 'bank' attends to 'river'.

Why it matters

It explains why context matters so much, and why long contexts cost more (attention grows with length).

An example

In 'She paid the excess because it was cheaper', attention helps 'it' link to 'the excess'.

The common mistake

Assuming a model 'reads' a long document the way a person does, evenly from start to end.

Where you meet it on each platform

The idea stays the same; only the product name changes.

JobAWSAzureGoogle CloudDatabricksOpen source
Model APIcall a large language modelAmazon BedrockAzure OpenAI in Microsoft FoundryGemini on Vertex AIFoundation Model APIsOllama, or a free hosted open model

Related ideas

Where it shows up

Part of the Crawl level on these paths: LLM Engineering (how models work).