LLM Architect: How Transformers Work

LLM Architect: How Transformers Work

Modern large language models like GPT, BERT, and T5 all share one transformative idea: the Transformer. This architecture replaced older RNN and CNN methods with a single, powerful mechanism — attention. Transformers are flexible. They can read, summarize, translate, and even generate creative text with remarkable efficiency.

1. What Is Attention?

Attention allows the model to focus on the most important parts of a sentence. Each word looks at other words to understand context and relevance.

Every word is transformed into three special vectors:

• Q (Query) – what the word is looking for
• K (Key) – what the word represents
• V (Value) – the actual information the word carries

The model compares Queries (Q) with Keys (K) to calculate relevance, then mixes the Values (V) accordingly. A simple version of the formula looks like this:

Attention(Q, K, V) = softmax((Q × Kᵀ) / √dₖ) × V

Example: In “The cat chased the mouse because it was hungry,” attention helps the model know that “it” refers to “the cat,” not “the mouse.”

2. Encoder and Decoder

The Transformer consists of two main parts:

• Encoder – reads and understands input text
• Decoder – generates text based on the encoder's understanding

Three common types of Transformer models exist:

• Encoder-only (BERT): Reads the full text bidirectionally. Great for understanding language (classification, Q&A) but cannot generate text.
• Decoder-only (GPT): Reads left-to-right to predict the next word. Ideal for text generation, summarization, or chat.
• Encoder–Decoder (T5/BART): Encoder reads input, decoder produces output. Perfect for translation, summarization, and tasks where input and output differ.

Example: T5 translating “Hello” to Spanish — the encoder understands “Hello,” and the decoder outputs “Hola.”

3. Multi-Head Attention

Transformers don’t rely on just one attention mechanism. They use multi-head attention, where each head focuses on different relationships:

• One head tracks subjects of verbs
• Another head tracks long-range dependencies, like pronouns referring to distant nouns

Combining heads creates a richer understanding of text.

4. Real-World Examples

Encoder-Only Model (BERT)

Task: Question answering Input: “The Eiffel Tower is located in Paris. What city is the Eiffel Tower in?” Output: “Paris”

Key point: Encoder-only models understand text but don’t generate new sentences.

Decoder-Only Model (GPT)

Task: Story writing Input: “Once upon a time, there was a little cat named Whiskers. She loved to…” Output: “…play with yarn in the sunny garden every morning.”

Key point: Decoder-only models excel at generating fluent text.

Encoder–Decoder Model (T5/BART)

Task: Summarization Input: “The city council announced a new plan to reduce traffic in downtown areas. The plan includes new bike lanes, improved public transport, and restrictions on certain roads during peak hours. Citizens are encouraged to use alternative transport.” Output: “The city council plans to reduce downtown traffic by adding bike lanes, improving public transport, and restricting certain roads.”

Key point: Encoder-decoder models combine understanding and generation, perfect for translation and summarization.

5. Why Transformers Work

Attention enables the model to connect words, even across long distances. It also allows parallel computation, making training faster than sequential RNNs. Tasks like summarization, Q&A, generation, and translation all rely on this simple mechanism.

6. Step-by-Step: How a Transformer Reads a Sentence

Example sentence: “The cat chased the mouse.”

1. Tokenization: ["The", "cat", "chased", "the", "mouse"]
2. Embedding: Each word becomes a numerical vector
3. Creating Q, K, V: Each vector is transformed into Query, Key, and Value
4. Calculating Attention Scores: Queries compared to Keys to find important words
5. Weighting Values: Values are mixed based on attention scores
6. Multi-Head Attention: Different heads capture different relationships
7. Feed-Forward Layers: Refines the information further
8. Output: Encoder produces contextual representations; decoder generates text step by step

Result: The model understands “the cat chased the mouse” as a cat performing an action on a mouse.

7. Summary

• Encoder → understands input text (BERT)
• Decoder → generates new text (GPT)
• Encoder–Decoder → transforms one text into another (T5)

Transformers demonstrate that a single, elegant mechanism — attention — can enable models to understand and generate language at scale. Bottom line: attention is all you need.