Attention Is All You Need

Vaswani et al., NeurIPS 2017. Based on the NeurIPS publication and its arXiv record; where they differ, the map follows NeurIPS.

Click a heading to fold it · drag the dots or press Alt+arrows to reorder

Core claim: sequence transduction does not require recurrence or convolution

The Transformer relies on attention to connect positions in a sequence, making training more parallelizable while improving machine-translation quality.

1The problem with sequential models

Why the authors looked beyond recurrent and convolutional sequence models.

RNNs and LSTMs process sequence positions step by step. That dependency limits parallel training, especially for long sequences.
Convolutional alternatives allow more parallel work, but connecting distant positions requires paths that grow with sequence distance.
Attention already helped encoder–decoder models, but it was normally attached to a recurrent network.
The proposal: make attention the primary mechanism instead of an accessory.
2Transformer architecture

The familiar encoder–decoder shape remains, but its internal machinery changes.

The encoder and decoder each use a stack of six identical layers (§3.1).
Each encoder layer combines multi-head self-attention with a position-wise feed-forward network.
The decoder adds encoder–decoder attention and masks future positions so generation remains autoregressive.
Residual connections and layer normalization wrap the sublayers.
The base model uses:
model dimension dmodel = 512
feed-forward inner dimension dff = 2048
six encoder and six decoder layers
3Attention mechanism

Queries ask, keys match and values carry the information that is combined.

Attention(Q, K, V) = softmax(QKT / √dk)V
Dividing the dot products by √dk keeps large values from pushing softmax into regions with very small gradients.
Multi-head attention projects queries, keys and values into several learned representation spaces and runs attention in parallel.
Eight heads in the base model, 64 dimensions per head (§3.2.2).
Attention appears in three roles:
encoder self-attention
masked decoder self-attention
decoder attention over encoder outputs
4Order without recurrence

Attention alone does not know whether a token came first, last or somewhere in between.

The model adds positional encodings to token embeddings before the first encoder and decoder layers.
The chosen encoding uses sine and cosine waves at different frequencies, giving each position a structured signal.
Learned positional embeddings produced nearly identical results in the reported comparison.
Removing recurrence makes position information an explicit design choice.
5Training and efficiency

Parallel structure is valuable only if it translates into practical training gains.

Experiments use WMT 2014 English–German and English–French translation datasets.
Training ran on one machine with eight NVIDIA P100 GPUs (§5.2).
Base models: 100,000 steps, about 12 hours. Big models: 300,000 steps, about 3.5 days (§5.2).
The recipe also includes Adam, learning-rate warmup, residual dropout and label smoothing.
Self-attention reduces sequential operations per layer to a constant number, enabling substantially more parallel computation.
6Results and significance

What the 2017 experiments established — and what the title should not make us forget.

28.4 BLEU on WMT 2014 English–German, above previously reported systems including ensembles (Table 2).
41.0 BLEU on English–French in the NeurIPS version of Table 2, at a fraction of the training cost of the previous state of the art. The arXiv revision reports 41.8.
The architecture also generalized to English constituency parsing in both large- and limited-data settings.
The title is a thesis, not an inventory: embeddings, feed-forward networks, residuals and normalization all remain. Which of them are load-bearing?
The enduring idea is architectural: global dependency modeling can be parallel, direct and effective.
The problem with sequential models Transformer architecture Attention mechanism Order without recurrence Training and efficiency Results and significance