Home › Key concepts › Attention and the transformer
Key concept · Crawl (beginner) · the same on every platform
Attention and the transformer
Each word looks at every other word to decide what matters.
What it is
The transformer is the design behind modern language models. Its key part, attention, lets each token weigh every other token in the context to build its meaning: in 'the bank of the river', 'bank' attends to 'river'.
Why it matters
It explains why context matters so much, and why long contexts cost more (attention grows with length).
An example
In 'She paid the excess because it was cheaper', attention helps 'it' link to 'the excess'.
The common mistake
Assuming a model 'reads' a long document the way a person does, evenly from start to end.
Where you meet it on each platform
The idea stays the same; only the product name changes.
| Job | AWS | Azure | Google Cloud | Databricks | Open source |
|---|---|---|---|---|---|
| Model APIcall a large language model | Amazon Bedrock | Azure OpenAI in Microsoft Foundry | Gemini on Vertex AI | Foundation Model APIs | Ollama, or a free hosted open model |
Related ideas
Where it shows up
Part of the Crawl level on these paths: LLM Engineering (how models work).