Attention, Explained Plainly

1Why this paper matters

In June 2017, eight researchers at Google published a paper with a confident title: Attention Is All You Need. It described a new design for a neural network, which they called the Transformer. Their goal was modest on paper. They wanted to translate sentences between English, German and French faster and more accurately than the systems of the time.

They succeeded, but that turned out to be the least important part. The design was so simple and so well suited to modern computer chips that researchers began using it for everything. Within a few years it powered Google Search's understanding of queries, then the GPT models, then the chat assistants many people now use every day. When you hear "large language model," you are hearing about a scaled-up descendant of this paper. The "T" in ChatGPT stands for Transformer.

This post walks through the paper from the ground up. You don't need a maths background. Each idea starts with an everyday picture and only then gets its proper name. If you want to click through the same ideas, the picture-book version and the live simulation cover the same ground.

2The problem with reading one word at a time

Before 2017, the best translation systems were built from recurrent neural networks, usually a kind called an LSTM. A recurrent network reads a sentence the way you might read with your finger on the page: one word, then the next, then the next. As it goes, it keeps a running summary of everything it has read so far, a bit like a mental note.

Picture a line of people passing a note. The first person reads the word "The," writes a summary, and hands it on. The second person reads "animal," updates the note, and hands it on. By the time the note reaches the eleventh person, it has been rewritten ten times. Details from the beginning have faded or been squeezed out to make room for newer information.

This caused two problems. The first was memory. When a word late in the sentence depended on a word near the start, the connection had to survive a long chain of rewrites, and it often didn't. The second was speed. Step eleven cannot begin until step ten has finished, so the work cannot be shared out across the thousands of small processors inside a modern graphics chip. Training was slow, and that limited how big these models could get.

3How a computer reads a word

Computers can't work with words directly, only with numbers. So the first job of any language model is to turn each word into a list of numbers. This list is called an embedding. In the paper, every word becomes a list of 512 numbers.

A helpful way to imagine an embedding is as a character card in a game. A card might score a word on qualities like "is it a living thing," "is it large," "is it a feeling" and "is it a place." "Cat" and "kitten" would have very similar cards. "Elephant" would share the living-thing score but differ on size. "Street" would look completely different.

The real numbers don't come with tidy labels like these. The model invents its own qualities during training, spreading meaning across all 512 numbers in ways people can only partly interpret. The useful property survives, though: words with similar meanings end up with similar lists. If you could squeeze those 512 numbers down to two and plot them on a page, related words would cluster together.

Before the paper's model reads a sentence, it breaks words into common pieces, a technique called byte-pair encoding, so that rare words can be built from familiar parts. The English and German vocabulary in the paper had about 37,000 of these pieces, each with its own embedding.

4Attention, the big idea

Read this sentence:

The animal didn't cross the street because it was too tired.

What does "it" mean? You knew instantly: the animal. Streets don't get tired. Now change one word:

The animal didn't cross the street because it was too wide.

Now "it" means the street. The word "it" hasn't changed, but its meaning depends entirely on the other words around it. To understand a word, you have to look at its context.

Attention is the mechanism that lets a model do this. When the model processes "it," it looks at every other word in the sentence and decides how much each one matters for understanding "it." Those decisions are expressed as shares that add up to 100 percent. In the "tired" sentence, a trained model might give about half its attention to "animal," a smaller share to "tired," and very little to the rest.

Here is the crucial part. The model then builds a new version of "it" by mixing in information from the words it paid attention to, in proportion to those shares. After this step, the model's representation of "it" contains a good deal of "animal." The word has absorbed its context.

And this happens for every word at the same time. "Animal" looks at the whole sentence, "street" looks at the whole sentence, "tired" looks at the whole sentence, all in one step. There's no chain of notes. The connection between the first word and the last is direct, no matter how far apart they are. Because the attention happens within a single sentence, the paper calls it self-attention.

5Question, label and content

How does a word actually decide where to look? The paper's answer is easiest to understand through something familiar: searching for a video.

You type a question into the search box, say "how to bake bread." Every video on the site has a title. The site compares your question with each title and gives each one a match score. "Bread baking for beginners" scores high. "Funny cat videos" scores near zero. Then you watch the content of the best matches.

Attention works the same way, using three lists of numbers made from each word's embedding:

• The query is the question a word is asking. For "it," something like: which noun could get tired?
• The key is the label a word offers to others. "Animal" might be labelled: a living thing, a noun.
• The value is the content a word hands over if someone pays attention to it, its actual meaning.

Each word's query, key and value are made by multiplying its embedding by three grids of numbers, called weight matrices, that the model learns during training. In the paper each of these is a list of 64 numbers.

To work out the attention for "it," the model compares the query of "it" with the key of every word. The comparison is a dot product: multiply the two lists number by number and add up the results. Similar lists give a large total, which means a strong match.

Those raw match scores are then turned into shares that add up to 100 percent, using a function called softmax. Softmax gives most of the share to the strongest matches and very little to weak ones. Finally, the model takes a weighted blend of every word's value, using those shares. That blend is the new, context-aware version of "it."

The paper packs all of this into one line:

There is one small detail worth understanding, the division by √d_k. Here d_k is the length of the query and key lists, 64, so the model divides every score by 8. Why? When you add up 64 products, the totals can get large. Large scores push softmax to an extreme where one word gets nearly 100 percent and every other word gets nearly zero. In that state the model receives almost no signal about how to improve, and learning stalls. Dividing by the square root of the list length keeps the scores in a comfortable range, however long the lists are. The paper's authors call their version scaled dot-product attention because of this step.

6Asking several questions at once

A single question can't capture everything a word needs to know. For "it," you might want to know who it refers to, how it is feeling, and what it did. These are different questions that point to different words.

So the Transformer runs several attention calculations side by side. Each one is called a head, and each head has its own query, key and value weights, so it learns to ask its own kind of question. One head might learn to link pronouns to the nouns they refer to. Another might learn to connect verbs with their objects. The paper's appendix shows real examples of trained heads doing exactly these kinds of jobs.

The paper uses 8 heads in every attention layer. Each head works with lists of 64 numbers instead of the full 512, so running 8 small heads costs about the same as running one big one. When all the heads have finished, their results are placed side by side and mixed together by one more learned grid of numbers. The paper calls this multi-head attention.

Did the extra heads matter? The authors tested it. With only one head, translation quality dropped noticeably. With far too many tiny heads, it dropped again. Eight was a good balance for their model size.

7Teaching it word order

Attention has a surprising blind spot. Because every word looks at every other word in the same way, the calculation has no idea which word came first. "Dog bites man" and "man bites dog" contain the same three words, so to plain attention they look like the same bag of words. For language, that is a serious problem.

The fix is called positional encoding. Before a sentence enters the model, every word gets a second list of 512 numbers that describes its seat in the sentence, and this list is added to the word's embedding. Now "dog" in position 1 and "dog" in position 3 have different starting numbers, and the model can tell them apart.

The paper builds these seat numbers from sine and cosine waves. Imagine a row of clocks, each ticking at a different speed. The fast clocks change a lot from one position to the next, which helps tell neighbouring words apart. The slow clocks barely move across a sentence, which helps tell the beginning from the end. Reading every clock at a given position gives that position a unique fingerprint.

The authors chose waves because the pattern for "five words later" is always the same simple shift, whichever position you start from. They hoped this would make it easy for the model to reason about relative distance. They also tried simply letting the model learn its own position numbers. Both approaches scored almost exactly the same, and they kept the waves because they might work for sentences longer than any seen during training.

8Putting the machine together

With attention and positional encoding in hand, we can assemble the full Transformer. Like earlier translation systems, it has two halves. The encoder reads the source sentence, say the French je suis étudiant, and produces a rich understanding of it. The decoder writes the translation, i am a student, one word at a time.

Inside one encoder layer

Each encoder layer has two parts. First comes multi-head self-attention, where every word gathers information from every other word. Then comes a small feed-forward network, which you can think of as a thinking box. Each word goes through the thinking box on its own, and it updates the word's numbers using what attention just gathered. In the paper the thinking box expands each word's 512 numbers to 2,048, applies a simple rule that zeroes out negative values, and shrinks them back to 512.

Two small additions keep this stable. Around each part there is a residual connection, a shortcut that adds the part's input back onto its output. That means each layer only has to learn an improvement rather than rebuild everything from scratch, which makes deep stacks much easier to train. After each shortcut comes layer normalization, which rescales each word's numbers so they stay in a sensible range.

Stacking the layers

The encoder repeats this layer six times. Each layer reads the output of the one below and adds more understanding. You can imagine "it" becoming richer as it rises: first it is just a pronoun, then a pronoun pointing backwards, then probably the animal, then the tired animal that didn't cross the road. The real layers store this as numbers rather than sentences, but the intuition of gradual refinement is right. The layers share the same design but each has its own learned numbers.

9How it writes a translation

The decoder is also six layers deep, and each of its layers has three parts instead of two.

The first is masked self-attention. The decoder writes one word at a time, so when it is choosing the third English word, it must not be able to see the fourth. During training the model is shown the whole correct translation at once, which would make cheating easy. To prevent it, the model hides every future word by setting its match score to minus infinity before softmax. After softmax, those hidden words get a share of exactly zero. Each word can only look at itself and the words before it.

The second part is cross-attention, the bridge between the two halves. Here the questions come from the decoder, the English being written, while the labels and content come from the encoder's final understanding of the French. This is how the decoder checks back with the source sentence. When it is about to write "student," it pays most of its attention to "étudiant." Older systems needed a separate alignment step to match words across languages; here the model learns those links on its own.

The third part is the same thinking box as in the encoder, again with shortcuts and normalization around each part.

At the very top, the decoder's final numbers go through one more step that gives a score to every word in the vocabulary. Softmax turns those scores into probabilities. The model picks a likely word, writes it down, feeds it back into the decoder as the newest input, and repeats. When it produces a special end-of-sentence marker, it stops.

In practice the paper didn't just take the single most likely word each time. It used beam search, which keeps the four most promising partial translations at every step and picks the best complete one at the end. This avoids getting stuck after one early choice that looked good but led somewhere awkward.

10How it learns

At the start of training, every number in the model is random, so its translations are nonsense. Training is a loop. The model reads a sentence pair, predicts each next word, and measures how far its predictions were from the correct words. That measure of error is called the loss. The model then nudges every one of its numbers a tiny amount in the direction that would have reduced the loss, a process called backpropagation. Then it does the same with the next batch of sentences, millions of times over.

The authors trained on standard translation datasets: about 4.5 million English–German sentence pairs and 36 million English–French sentence pairs. They used one machine with eight graphics chips. The smaller "base" model trained for about 12 hours, and the "big" model for three and a half days.

A few practical choices helped. The learning rate, which controls how large each nudge is, started small and grew for the first 4,000 steps, then slowly shrank. Starting gently stops the random early model from making wild changes. They also used dropout, which randomly switches off a tenth of the signals during training so the model can't rely too heavily on any one path. And they used label smoothing, which tells the model the correct word is 90 percent right rather than 100 percent, so it learns to stay slightly humble. That made one measure of confidence look worse, but it made the translations better.

11What happened when they tested it

Translation quality is usually measured with a score called BLEU, which compares a machine's output with professional human translations. Higher is better, and gains of even one point are considered significant. Here is how the Transformer did on the English-to-German test, compared with the strongest earlier systems.

Even the small Transformer beat every earlier system, including combinations of several models, while using a fraction of the computing power. The big model set a new record by more than two points. On English-to-French it also set a new record for a single model. The authors then tried it on a different task, analysing the grammatical structure of English sentences, and with almost no adjustments it matched or beat most specialised systems.

The authors also ran careful experiments, removing or changing one piece at a time to see what mattered. Bigger models did better. Removing dropout made results worse. A single attention head was clearly worse than eight. And, as mentioned, learned position numbers and sine waves performed almost identically.

12Why it changed everything

The real breakthrough was speed. A recurrent network has to take one step per word. A self-attention layer handles every word in one step, so the work can be spread across all the processors in a chip at once. That made it practical to train far larger models on far more text than before, and larger models trained on more text kept getting better.

Researchers soon realised they could use each half of the Transformer on its own:

• Just the reading half became BERT in 2018, which helped search engines understand what people mean.
• Just the writing half, with its no-peeking rule, became the GPT family. Scaled up enormously, this is the design behind today's chat assistants, including ChatGPT, Claude and Gemini.
• Both halves together live on in models like T5 and in modern translation tools.
• Beyond words, the same attention idea now reads images, speech, music and even protein structures.

The design does have a cost. Because every word compares itself with every other word, doubling the length of the input roughly quadruples the work. That is fine for sentences but expensive for whole books, and a great deal of research since 2017 has gone into making attention cheaper for long inputs. Modern models also tweak many details: different position schemes, normalization placed before each part instead of after, and more efficient thinking boxes. The core idea, every word deciding what to pay attention to, is unchanged.

13Glossary

Attention

Deciding how much to focus on each other word, then mixing in their information by those shares.

Self-attention

Attention where the words being looked at come from the same sentence.

Cross-attention

Attention where the decoder's questions look at the encoder's words.

Embedding

The list of numbers that represents a word. 512 numbers in the paper.

Query, key, value

The question a word asks, the label it offers, and the content it hands over.

Dot product

Multiply two lists number by number and add up the results; a quick similarity score.

Softmax

Turns any set of scores into positive shares that add up to 100 percent.

Head

One independent attention calculation. The paper uses 8 per layer.

Positional encoding

A wave-based fingerprint added to each word so the model knows word order.

Feed-forward network

The small "thinking box" each word passes through after attention.

Residual connection

A shortcut that adds a layer's input to its output, making deep models easier to train.

Layer normalization

Rescaling each word's numbers to keep them in a stable range.

Encoder and decoder

The half that reads the input and the half that writes the output.

Masking

Hiding future words from the decoder so it cannot peek at the answer.

Loss

A number measuring how wrong the model's predictions were.

BLEU

A translation quality score that compares output with human translations.

Keep going

Further reading: the original paper, Attention Is All You Need (arXiv:1706.03762), and Jay Alammar's The Illustrated Transformer, a classic visual walk-through.