A transformer is the neural-network architecture under every modern LLM. Its core idea, attention, lets the model weigh which other words in the text matter most when interpreting each word — capturing context and relationships at scale, in parallel.
The problem it solved
Earlier models read text word by word, which was slow and forgetful over long passages. The 2017 transformer processed the whole sequence at once and let every word look at every other word directly. That parallelism is what made training on internet-scale data practical.
Attention, in one line
For each word, attention asks: which other words should I pay attention to, and how much? "It" in a sentence learns to attend to the noun it refers to. Stack many attention layers and the model builds a rich, contextual understanding of meaning.
Why it scaled
Attention is highly parallel, so it maps well onto GPUs. Add more data, more parameters, and more compute, and quality kept improving — the scaling behaviour that produced GPT, Claude, Gemini, and the rest.
What operators actually need
You do not need the math to use LLMs well. But knowing that attention is the engine explains why context windows, token limits, and prompt structure matter: they all change what the model can attend to when it answers.
Frequently asked
What is a transformer in AI?
The neural-network architecture behind modern large language models, built on self-attention, which lets the model weigh relationships between all tokens in a sequence in parallel.
What is the attention mechanism?
A method that lets a model decide, for each word or token, how much to focus on every other token when interpreting it — capturing context and long-range relationships.
Why are transformers important?
Their parallel attention made it possible to train on internet-scale data and keep improving with more compute, which produced today's LLMs like GPT, Claude, and Gemini.